Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning

The key architectural bet in Reasmory is not the use of point clouds per se, but the decision to constrain tool access within a guided pipeline rather than letting the VLM invoke reconstruction tools freely. That constraint is doing most of the work: it prevents the model from hallucinating plausible-but-wrong spatial transformations, which is a failure mode that free-form tool use tends to amplify.

This connects directly to NVIDIA's Cosmos 3 announcement (covered June 1), which frames spatial reasoning as a prerequisite for physical AI and robotic action. Cosmos 3 approaches the problem through world models trained on physical dynamics, while Reasmory approaches it through explicit geometric memory attached to existing VLMs. These are complementary bets, not competing ones: Cosmos 3 targets embodied systems with dedicated training pipelines, whereas Reasmory targets inference-time augmentation for models that were never trained on spatial tasks. The broader pattern, visible across recent coverage, is that the field is converging on the view that perception alone is insufficient and that some form of structured world representation must be injected, either at training time or at inference.

Watch whether Reasmory's constrained pipeline approach holds up on benchmarks that require novel viewpoint extrapolation rather than interpolation between observed views. If it degrades sharply there, the framework is solving memorization of seen geometry, not genuine spatial generalization.