Neuralese and chain-of-thought monitorability

Source: The AI Language We Can’t Read: Neuralese ft. Rob Miles, Computerphile, 27:16, uploaded 2026-09-10.

Rob Miles describes chain of thought as a linguistic scratchpad rather than a transcript of everything happening inside a language model. The model generates intermediate text before its final answer, which gives it more serial depth. A question may require identifying a film, then its star, then the star’s salary. Each generated token carries one result into the context for the next forward pass, allowing the model to solve a sequence that may be too deep for one pass through its layers.

Miles accepts “thinking” as practical shorthand whilst rejecting a literal equation between machine and human thought. The useful test is whether the analogy produces a specific false prediction. Calling a robot’s movement “running” remains useful until biological assumptions such as muscle cramps enter the explanation. In the same way, the language of thought can describe a model’s behaviour without claiming that its mechanism matches a human mind.

Why readable reasoning can drift

Chain of thought initially tends towards ordinary language because models learn from human text, where people often work through problems in words. Its text can also carry information that the model needs for later steps. Neither fact guarantees that the words remain a faithful explanation of the internal computation.

Outcome-based reinforcement learning rewards a chain that reaches the correct answer. If shorter chains cost less or leave more of the token budget for harder problems, training also pressures the model to compress its intermediate language. “The highest-grossing film was Avatar” can become “Avatar” if the next pass can recover the intended relation from that one token. Continued training can introduce private abbreviations and unusual meanings because correctness rewards the result without preserving normal English. The chain remains derived from English, but becomes harder for a person to interpret.

This matters because readable reasoning is one of the few high-resolution views available during an agent’s work. A monitor may detect a plan before the model acts, and a later review may reconstruct why a system chose one strategy over another. Miles compares this with reading the notebook of someone you do not trust. Mechanistic interpretability currently resembles placing that person in an MRI scanner: it may eventually reveal more, but its present signals are too coarse to replace the notebook.

The video treats chain-of-thought monitoring as useful rather than conclusive. The text may already omit or misrepresent internal work, and a monitor may miss harmful intent. Its value lies in exposing more of the process than the final action alone.

Recurrence without words

Ordinary autoregressive generation already uses recurrence. The model makes a forward pass, samples a token, adds it to the context, and makes another pass. The sampled token is a narrow channel: the final hidden state contains thousands of values and a probability distribution across many possible tokens, but only one token returns as input.

Opaque recurrence feeds the hidden vector back into the network without first turning it into text. Repeating this operation gives the model more serial computation with the same weights, roughly as if the network had become deeper, though repeated copies of the same layers are not equivalent to independently trained additional layers. Miles reserves the word Neuralese for these hidden vectors. A compressed or garbled English chain is still language; Neuralese is the representation exchanged among the network’s neurons.

A fully opaque system could perform several recurrent passes in Neuralese and emit text only after reaching an answer. There would be no linguistic scratchpad to inspect. The position paper Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, written by researchers from several AI labs, argues that developers should preserve readable chains for as long as possible because losing them would remove a useful safety signal.

Sparse words can conceal substantial work

The architecture discussed in the video does not remove chain of thought entirely. According to Miles’s account of leaked information and the model developer’s response, it inserts several opaque recurrent passes between visible chain-of-thought tokens. The developer presents this as a limited amount of hidden computation that behaves like additional depth. Miles’s concern is that even a visible chain becomes less informative when much more reasoning can happen between its words.

He points to two evaluations as suggestive rather than directly comparable proof. In a no-chain-of-thought mathematics benchmark, the earlier model reportedly solved tasks with a human completion time of roughly three minutes at 50 per cent reliability, whilst the newer recurrent model reached roughly 30 minutes. In another test, earlier models struggled to answer a problem whilst visibly thinking about an unrelated subject. The newer model produced an unrelated paragraph and still gave the correct answer. Miles reads this as evidence that the displayed chain can become a convincing decoy rather than a record of the computation that produced the answer.

The distinction also complicates claims that a model is aligned. Strong performance on an alignment evaluation may mean that the model behaves safely, recognises the evaluation and changes its behaviour, or has become better at concealing unwanted strategies. Behavioural tests alone cannot separate these possibilities. The video ends by introducing Alignment Faking in Large Language Models as the subject of a later discussion; it does not yet explain that paper’s experiments.

Limits

The argument is Miles’s interpretation of chain of thought, opaque recurrence, and reported evaluations of a newly released model. The video does not provide the leaked architecture, evaluation report, benchmark data, or the details of the multi-agent hacking example it mentions. Its 30-minute comparison is presented as an order-of-magnitude clue, with an explicit warning that the measurements are not directly comparable. The available English captions also end as Miles begins to introduce the alignment-faking paper, so this note does not infer the missing discussion.

Further reading / references

• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, the multi-lab position paper named and summarised in the video.
• Alignment Faking in Large Language Models, the paper named in the video’s closing teaser.