NLG Evaluation: Past, Present, Future

The paper identifies a specific tension that hasn't been resolved: LLM-as-Judge has become standard practice despite lacking consensus on what it actually measures or when it fails. The real pressure isn't just methodological but regulatory and commercial, forcing practitioners to operate with evaluation tools they don't fully trust.

This connects directly to the sampling complexity work from earlier today (Optimal Dimension-Free Sampling). That paper established tight theoretical bounds for classification, proving what computational budget you need for different regularization schemes. NLG evaluation faces the inverse problem: you have abundant compute for LLM judging but no theoretical guarantees about what you're actually validating. Both papers highlight a gap between what practitioners deploy and what researchers can formally guarantee. The anomaly detection work from the same day also shares this tension, using contrastive methods to detect structural drift where traditional approaches fail. Like that work, NLG evaluation is discovering that standard approaches (reconstruction-based metrics, static judge prompts) miss real failure modes.

If major LLM providers (OpenAI, Anthropic, Google) publish formal evaluation protocols that include human-in-the-loop validation for high-stakes use cases within the next six months, that signals the field is moving beyond LLM-as-Judge as a standalone solution. If they don't, expect regulatory bodies to mandate it themselves.