•
Belief discovery at scale
Finding belief-state geometry in large language models without supervision. Can we automatically identify the internal structures that encode what a model knows about the world?
•
Building a theory of intelligence
We have the beginning steps of a theory, but it needs to be extended and refined in a number of ways, in order to, e.g., capture internal world models of different types and apply to other neural systems (e.g., RL and biological brains).
•
Generalization
Why and how do neural networks generalize? Our framework suggests ways in which internal structures support out-of-distribution behavior.
•
Red Teaming
We have an entire team dedicated to stress-testing our own framework. Finding the boundaries, the edge cases, the places where the theory breaks down in service of figuring out what’s actually true.
•
Biological Intelligence
The same mathematics that reveals structure in transformers might apply to biological neural networks. We plan on testing this on real brain data because ultimately we’re interested in intelligence wherever it appears.
Learn More About Our Work
Our foundational result (manuscript, blog post) showed that transformers trained on next-token prediction spontaneously organize their activations into geometries predicted by Bayesian belief updating over hidden states of a world model. Even when trained on simple token sequences from hidden Markov models, complex fractals emerge in the residual stream, structures far removed from the surface statistics of the training data. We think of this work as providing the first steps into an understanding of what fundamentally we are training AI systems to do, and what representations we are implicitly training them to have.
Since then, we’ve pushed in several directions. In Constrained Belief Updating Explains Transformer Representations, we asked how attention implements belief updating when Bayesian inference is fundamentally recurrent. We found that attention parallelizes recurrence by decomposing belief updates spectrally across heads, and we were able to make verified predictions about embeddings, OV vectors, attention patterns, and residual stream geometry at different layers.
We’ve also developed a theory of in-context learning grounded in training data structure. When training data mixes multiple sources, models must infer not just what hidden state the generator is in, but which source is active. This hierarchical belief updating necessarily produces power-law loss scaling with context length and explains why induction heads emerge.
We’ve been asking what the most general computational framework for understanding neural network representations might be. Our initial work implied activations should lie in simplices, but we’ve now shown that networks discover quantum and post-quantum belief geometries when these are the minimal way to model their training data. This offers a new foundation for thinking about features, superposition, and what representations neural networks use on their own terms.
Most recently, we’ve shown that transformers naturally decompose their world model into interpretable parts. These factored belief representations provide an exponential-dimensional advantage, and suggest that we can understand and surgically intervene upon low-dimensional subspaces of large models.
For a comprehensive overview of where we are and where we’re headed, see our July 2025 progress report on the Alignment Forum. You can also watch Paul and Adam discuss the research program at the FAR Seminar or read this recent interview from August 2025.