Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation

The finding isn't just that translation generalizes across languages, it's that the task itself appears to be encoded as a separable, transferable structure inside the model, distinct from language-specific knowledge. That distinction matters because it suggests in-context learning may be doing something more systematic than pattern matching on surface examples.

This connects most directly to the 'Fabricator or dynamic translator?' piece from mid-April, which examined how LLMs generate spurious or helpful text during translation and how commercial systems try to manage those failure modes. That work treated translation behavior as something to observe and correct at the output level. This new research goes a layer deeper, asking whether the model's internal representation of the translation task is itself coherent and portable. Together they sketch two complementary angles on multilingual LLM behavior: what goes wrong at the surface, and what structure underlies the task internally. The generalization findings also rhyme loosely with the shortest-path paper from the same period, which found that LLMs transfer well spatially but fail when task complexity scales, raising the question of whether function vector transfer holds under more demanding translation conditions.

The key test is whether these translation vectors remain stable when extracted from models fine-tuned on multilingual data rather than purely pretrained ones. If the language-agnostic signal degrades after fine-tuning, it suggests the structure is a pretraining artifact rather than a robust property of task encoding.