Separating Geometry from Probability in the Analysis of Generalization

The deeper provocation here is not just methodological but philosophical: the paper argues that current generalization theory is built on an assumption (that training and test data are drawn from the same distribution) that is essentially unverifiable in real deployments, meaning the theoretical guarantees practitioners cite may be structurally unfalsifiable.

This connects directly to the cluster of generalization papers Modelwire covered in mid-April. The 'Stability and Generalization in Looped Transformers' piece from April 16 also tried to ground generalization claims in structural properties of architectures rather than distributional assumptions, and the LLM shortest-path paper from the same date showed empirically what happens when distribution shift is implicit and uncontrolled: models fail at horizon extension in ways that standard generalization metrics would not predict. Together, these three papers suggest a quiet but consistent pressure building against the classical PAC-learning framing, coming from both theorists and empiricists at roughly the same moment.

Watch whether any of the major generalization benchmarking efforts, particularly those tied to out-of-distribution evaluation suites, begin citing sensitivity-based metrics alongside traditional bounds within the next 12 months. Adoption there would signal the framework is moving from critique to practice.