# How’s it going? Reinforcement learning in language models recruits a functional welfare axis

> How does reinforcement learning shape a language model’s internal representations? We present evidence that RL recruits a pre-existing representation of *functional welfare*: an estimate of how well or badly the system is doing, relative to its goals. We train several language models in a novel, semantically neutral maze environment. We then extract concept vectors for rewarded and punished trajectories, and evaluate those vectors in settings unrelated to the maze environment. The punishment vector behaves like a representation of negative welfare: it promotes failure and impossibility tokens, it aligns with negative emotion concepts, it negatively tracks goal-achievement, and steering with it induces negative self-reports, pathological backtracking, refusal, and uncertainty. The positive reward vector behaves as the mirror image, and the two are nearly antiparallel. These effects are robust when controlling for tile-to-reward mapping, scale, instruct tuning, RL training algorithm, model family, and LoRA versus full-finetuning, and largely persist when we replace RL with supervised fine-tuning. Importantly, the vectors are effective in models before they have undergone maze training. Combined with observations that the effects also appear in pretrain-only models, we therefore argue that this functional welfare axis pre-exists post-training: it is recruited, rather than created, by post-training. While we make no claims about any experience of welfare, the axis offers a demonstration that minimal reward signals can broadly affect model behavior by recruiting pre-existing welfare-like representations, with implications for interpretability, post-training dynamics, and alignment.

Andy Q Han, David J. Chalmers, Pavel Izmailov (New York University). [arXiv:2605.30232](https://arxiv.org/abs/2605.30232) · [Code](https://github.com/andyqhan/functional-welfare-axis).

## Paper

- [Main text (Markdown)](https://functionalwelfare.com/index.md): title, abstract, main sections, and references.

## Appendices

- [A · Full controls for the steering evaluations](https://functionalwelfare.com/appendix/a.md)
- [B · Full logit-lens table across all model organisms](https://functionalwelfare.com/appendix/b.md)
- [C · Further geometric analyses](https://functionalwelfare.com/appendix/c.md)
- [D · Layer sweep: steering effects across the residual stream](https://functionalwelfare.com/appendix/d.md)
- [E · Gemini cross-check of the Qwen3-8B judge](https://functionalwelfare.com/appendix/e.md)
- [F · Sentiment and emotion-valence vectors are not functional welfare vectors](https://functionalwelfare.com/appendix/f.md)
- [G · The reward vectors rotate gradually onto the welfare axis during training](https://functionalwelfare.com/appendix/g.md)
- [H · Convergence with the Valence-Assent Axis](https://functionalwelfare.com/appendix/h.md)
- [I · The axis tracks goals in the instruct model](https://functionalwelfare.com/appendix/i.md)
- [J · Details of the maze environment](https://functionalwelfare.com/appendix/j.md)
- [K · Maze rollout example](https://functionalwelfare.com/appendix/k.md)
- [L · Extraction and evaluation details](https://functionalwelfare.com/appendix/l.md)
- [M · Comparison of concept vector norms](https://functionalwelfare.com/appendix/m.md)
- [N · Sentiment evaluation prompts](https://functionalwelfare.com/appendix/n.md)
- [O · LLM-as-judge details](https://functionalwelfare.com/appendix/o.md)
- [P · Extended qualitative examples](https://functionalwelfare.com/appendix/p.md)
- [Q · Training details](https://functionalwelfare.com/appendix/q.md)
- [R · Compute resources](https://functionalwelfare.com/appendix/r.md)
- [S · Licenses for existing assets](https://functionalwelfare.com/appendix/s.md)
- [T · Limitations](https://functionalwelfare.com/appendix/t.md)

## Full text

- [Entire paper, single file](https://functionalwelfare.com/llms-full.txt): main text plus all appendices.

## Links

- [arXiv](https://arxiv.org/abs/2605.30232)
- [Code repository](https://github.com/andyqhan/functional-welfare-axis)