Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion

The benchmark targets a genuinely narrow professional use case: helping practitioners build and critique causal loop diagrams, a core tool in policy modeling and organizational analysis. The finding that local models break down specifically during iterative error-correction, rather than initial extraction, points to a failure mode in multi-turn reasoning under constraint, not just raw accuracy.

This connects directly to the reliability problems surfaced in 'Diagnosing LLM Judge Reliability' (arXiv, April 16), which found that aggregate consistency scores mask per-instance logical breakdowns in roughly one-third to two-thirds of cases. The coaching scenario failures described here look like the same underlying issue: models that appear coherent at the task level but lose track of constraints across turns. The 'Generalization in LLM Problem Solving' paper from the same week adds another angle, showing that LLMs fail when problems require recursive depth, which is structurally similar to iterative diagram correction. Together, these papers suggest the cloud-versus-local gap is less about raw capability and more about sustained structured reasoning over longer interaction chains.

If Kimi K2.5 or a comparable open-weight model closes the error-correction gap on a follow-up interactive coaching benchmark within the next two quarters, that would confirm the bottleneck is addressable through fine-tuning on domain-specific feedback loops rather than a fundamental context-length ceiling.