[2410.02430] Predictive Attractor Models
5 constraints on Sequence Memory Model
- The learning of one sequence does not overwrite the previously learned sequences.
- The model should uniquely encode inputs based on their context in a sequence. Consider the sequence of letters “EVER”; the representation of “E” at position 1 should be different from “E” at position 3, thus resulting in different predictions: “V” and “R”. Moreover, the representation of “E” at position 3 in “EVER” should be different from “E” at position 3 in “CLEVER”. Therefore, positional encoding is not a valid solution.
- When presented with multiple valid, future possibilities, the model should learn to represent each possibility separately yet stochastically sample a single valid possibility. Consider the two sequences “THAT” and “THEY”; after seeing “TH”, the model should learn to generate either “A” or “E”, but not an average or a union of both.
- The model should be capable of incrementally learning each transition without seeing the whole sequence or revisiting older sequence transitions that are previously learned. This property falls under online learning constraints, also called stream learning
- The learning algorithm should be tolerant and robust to significant input noise.
Model Architecture

- $x_t$是模型的输入,一个长度为$N_c$的向量,下标c意为content。$z_t$则是模型的latent state,是一个$N_k \times N_c$的矩阵。$f \& g$分别是latent state转换的函数和latent state到$x$的映射,是在训练过程中需要学习的参数。
- $z_t$的建模非常有趣:$z_t$是通过$x_t$得到的,而$\hat{z_t}$则只依赖于$z_{t-1}$, 类似于一阶Markov Chain。为什么$z_t$会是一个矩阵?因为$\hat{z}_t$如果依赖于上一个t,势必无法捕捉上下文,而如果依赖多个t,则就变成了高阶Markov Chain。
- 学习过程如下面的伪代码所示:

这里可能稍微需要补充一点符号:

↑x:把 content 列向上扩展到每个 context(把某列全部置 1),表示“这个内容在所有上下文上都可能出现”。其主要目的是和$z$把维度对齐。同理, ↓z 也是类似的目的。
m(a) 是把 memory/prediction(由 logits a = A·z 得到的“记忆性激活”)转成“可以和输入匹配的上下文掩码(mask)/候选集合” 的操作; (↑x) ∩ m(a) 就是取“既被输入内容允许(↑x),又被 memory/prediction 认为可能的那些 context-cells”,作为新的 latent SDR z 。$\mathbf{A} \, \& \, \mathbf{B}$ 其实对应的分别就是$f \& g$
- 生成序列的过程如下面的伪代码所示(Attractor就在7 & 8):
