The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
Researchers have closed a long-standing gap between transformer expressivity theory and practice by proving that standard softmax attention with low-precision activations can simulate arbitrary computation when depth and width scale logarithmically with context. The work sidesteps prior unrealistic assumptions about parameter magnitudes and precision by constructing ternary hardmax intermediates that execute chain-of-thought reasoning before converting to softmax equivalents. This result matters because it grounds theoretical understanding of transformer capabilities in architectures that actually exist, potentially informing both scaling laws and mechanistic interpretability efforts.
Modelwire context
ExplainerThe practical significance here is architectural conservatism: prior expressivity proofs required either infinite precision or unrealistically large parameter norms, meaning they described idealized machines rather than the transformers running in production. This result is the first to work within the constraints engineers actually face.
The mechanistic interpretability angle connects directly to recent coverage on this site. The piece on 'Are Sparse Autoencoder Benchmarks Reliable?' (from the same day) exposed a methodological crisis in SAE research, where the tools used to understand transformer internals rest on shaky evaluations. That paper and this one are pulling in the same direction: the field needs firmer theoretical and empirical foundations before interpretability claims can be trusted. Separately, the 'Canonical Regularisation of Wide Feature-Learning Neural Networks' paper flagged a blind spot in how wide networks generalize, and this expressivity result adds another layer to that same conversation about the gap between theoretical models and real architectures. Together, these papers suggest a broader reckoning with how much of transformer theory has been built on convenient but unrealistic assumptions.
The key test is whether mechanistic interpretability researchers begin citing this result to constrain or validate circuit-level hypotheses. If the chain-of-thought simulation construction appears in follow-on work on in-context learning circuits within the next six months, the theoretical bridge is being used. If it stays confined to complexity theory venues, the practical impact will remain limited.
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsTransformers · Softmax attention · Chain-of-Thought · Turing machines · Hardmax attention
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.