Skip to content

Relational operations factorize from their operands in LLMs

How does a language model represent "the capital of Italy"? We find it splits the thought in two: the thing (Italy) and the operation applied to it (capital-of) — and the operation lives as its own direction — an arrow in the model's space of internal states — that you can grab, move, and transplant.

The finding, conceptually

Latin marks a noun's role with a case ending — ros-a / ros-am, same stem, different job. Mid-network, an LLM does something structurally similar with factual relations. As "The currency of Italy is" flows through the model, the sentence's internal state is well described by a sum:

state ≈ μ + operand(Italy) + operator(currency-of) + small interaction

Three things make this more than a curve fit:

  1. The operator is causal. Add the direction difference v(capital) − v(currency) to the residual stream and the model stops saying euro and says Rome — for every ordered pair of relations we test. A random direction of the same size does nothing.
  2. It transfers. Built from half the countries, it flips the other half. Built from one phrasing, it flips paraphrases it never saw. Built on Qwen, the same recipe works on Gemma.
  3. The operation is not its output word. Language-of and demonym-of both end in "Italian" — yet they are distinct directions, and a marker built exactly where the two share their word still installs the relation. The model separates the grammatical role from the surface form, the way declension separates case from ending (syncretism).

And the boundary is just as informative: run the identical pipeline on arithmetic (+, ×, −) or comparison logic, and the operator is entangled with its operands and refuses to transfer — consistent with "arithmetic as a bag of heuristics", and evidence that the clean factorization is a property of relational retrieval, not of prompting in general.

The declension morph: the same 60 workspace states, organized by operand at the country token, reorganize by operator at the query token

Interactive result Explained simply Paper (PDF) Evidence & controls Code

The numbers

20/20 × 3ordered operator swaps flip the answer margin in Qwen3-1.7B, Qwen3-8B and Gemma-2-9B; permuted-label nulls ≈ 0
52% ≈ 53%a state assembled from μ + operand + operator makes the model produce the answer at its own ceiling (8B: 62% vs 68%)
80%even at a 4× overdose — where fluent generation collapses — the target still wins a forced choice: dose destroys fluency, not information

All headline effects carry operator-level cluster-bootstrap 95% CIs and none crosses zero — see Evidence & controls for every claim next to the control that could have killed it.

Methodological note — the honest null that started this

This project began as a replication of the J-space / global-workspace readout claim (Gurnee, Sofroniew, Lindsey et al., 2026). Under matched controls (including a spectrum-matched random projection), the J-lens readout did not outperform the logit lens on any of our metrics. The structure above is causal organization, not a privileged readable subspace — the negative half is documented with the same rigor in the archive and working log.

Reproducibility. Everything runs on one AMD Strix Halo APU (no CUDA). Seeds, exact checkpoints (Qwen/Qwen3-1.7B, Qwen/Qwen3-8B, google/gemma-2-9b), per-operand long-form artifacts, and every figure's generator are in the repo — see How to reproduce. MIT license · Matias Podeley · mpodeley@gmail.com · How to cite