Cerebras

Introduces the Implicit Chain Transformer (ICT) architecture to address the statelessness bottleneck in standard Transformers for state-intensive tasks. ICT propagates a learnable 'intent' latent vector across time steps, enabling explicit state updates and contextualization. Two propagation strategies are explored: Autoregressive Intent Propagation (Dense) and Periodic Intent Propagation (Sparse). A novel training method, 'Iterative Latent Feedback,' is proposed to maintain parallel training efficiency by using multi-pass approximations. Early evaluations on modulo arithmetic and graph traversal tasks demonstrate ICT's superior accuracy compared to baseline Transformers, with the Dense variant showing better stability on graph traversal due to challenges in compressing block representations in the Sparse variant with shallow networks. Future work includes inference optimization, scaling the Sparse formulation, and investigating training dynamics.