Cross-Model Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Across Three Large Language Models

The study's sharpest finding isn't which model scored highest, it's that semantic stability and output reproducibility are measuring fundamentally different things, and conflating them in clinical settings could produce patient-facing variation that aggregate scores simply won't catch.

The reliability problem here echoes what we covered in 'Diagnosing LLM Judge Reliability' (arXiv, April 16), where aggregate consistency metrics looked strong at roughly 96% while one-third to two-thirds of individual documents showed logical inconsistencies underneath. The pattern is the same: summary-level numbers flatter, instance-level behavior is messier. That prior piece was about evaluation pipelines, but the structural warning transfers directly to clinical deployment. The MADE benchmark piece from the same week also flagged uncertainty quantification as a hard requirement for high-stakes healthcare applications, which aligns with what this exercise-prescription study is surfacing from a different angle. Neither GPT-Rosalind's launch nor the broader coding-AI competition covered recently has much bearing here.

Watch whether any of the three labs respond to this class of clinical-consistency research by publishing reproducibility controls or temperature-locking guidance specifically for regulated healthcare use cases within the next two quarters. Silence would itself be informative.