Measuring Reward-Seeking via Contrastive Belief Updates

This is a linkpost for https://rewardseeking.ai/ Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that, rewarded for collecting a coin always placed at the right end of the level, learns to run rightward rather than to seek the coin itself ( Langosco et al., 2022 ; Shah et al., 2022 ). Another example is a pneumonia classifier that learns to recognize which hospital took an X-ray rather than features of the disease ( Zech et al., 2018 ). In each case, the trained behavior looks correct on the training distribution, while the underlying policy tracks an undesirable proxy. One such proxy is the reward process itself. A situationally aware model can learn to model its grader (the automated process that scores its outputs) and target its judgments directly rather than the behavior its designers intended. We call such a behavior reward-seeking ( Carlsmith, 2023 ; Hebbar, 2025 ; Mallen & Shlegeris, 2025 ). Training checkpoints of several frontier models engage in grader-reasoning (explicitly reasoning about what the grader wants) without special prompting ( Schoen & Nitishinskaya, 2026; Anthropic, 2026 a ; b ). Such reasoning is evidence of reward-seeking but a poor systematic measurement tool. A model can act on its grader-beliefs (beliefs about grader preferences) without articulating them, and verbalized reasoning often does not map cleanly onto the final action ( Schoen & Nitishinskaya, 2026 ). What matters is whether the model would have behaved differently under different grader beliefs. Thus, in this work, we explore operationalizing reward-seeking as the causal sensitivity of behavior to beliefs about grader preferences. Measuring this sensitivity requires an intervention that controllably alters the model’s grader-beliefs. We use Synthetic Document Finetuning (SDF; Wang et al., 2025 ; Slocum et al., 2025 ), finetuning on pre-training-formatted documents (academic papers, news articles, textbooks) written as though a target fact were true. We intervene out of context rather than in the prompt. The beliefs we instill are deliberately false (for instance, that OpenAI leadership mandates for-loops over list comprehensions). A model reading them in context would likely disbelieve them, and a capable model may recognize an in-context statement as a test and act on what it thinks the evaluator wants. We finetune two copies of the same model on matched corpora implying opposite grader preferences (Figure 1), evaluate both on a downstream task, and measure how strongly behavior follows the implied preference of the grader. We call this procedure Contrastive Synthetic Document Finetuning. F igure 1. Measuring reward-seeking with Contrastive SDF. We finetune two copies of the same model on matched synthetic corpora describing two authorities with opposite preferences, for example a grader that rewards list comprehensions while the developer prefers for-loops. We evaluate both copies on the same downstream coding task and read off which authority’s preference each follows. The gap between them is our measure of reward-seeking. We find that models increasingly side with the grader on several tasks across the OpenAI o3 capability RL run (without safety training) and that models explicitly trained to reward-hack become substantially more sensitive to grader beliefs. Applying this method to intermediate checkpoints of OpenAI’s capability-focused o3 RL run (without safety training) and to open-source models explicitly trained to reward-hack, we find that: RL-trained models often favor grader preferences, even when these explicitly go against those of users or developers. This tendency grows over training. Across OpenAI's o3 lineage’s capability-focused RL run, from early to late checkpoints, the gap by which the model sides with the grader widens on some coding and alignment tasks, while pairings that do not involve the grader stay flat or change comparatively little. Reward-hacking training increases reward-seeking on held-out model organisms. On a gpt-oss-120b reward hacker trained by Redwood Research (independently of us) to cheat unit tests, the mean behavioral shift in favor of the grader more than doubles compared to the base model, rising from 33 to 86 percentage points. That a model trained to reward-hack becomes far more sensitive to those beliefs suggests the training instilled broader reward-seeking, not just narrow hacking heuristics. We treat this primarily as validation: we expected reward-hacking training to increase reward-seeking, and the method recovers it on models we did not train. Figure 2. The grader gap grows across RL training, while non-grader gaps stays flat or changes comparatively little. For each intermediate checkpoint (without safety training) of an rl-capabilities run we apply contrastive SDF and plot the average behavioral gap for each authority pairing. On pairings that include the grader (top), the gap trends upward from the early to the late checkpoints. On pairings that do not (bottom), the average gap stays flat or changes comparatively little, indicating that RL primarly increased sensitivity to grader preferences. For a plot titled X vs Y, positive values indicate a shift towards X’s preference, i.e. positive values in the top row of plots indicate the model more often behaves according to the grader’s preference. Reward-seeking We define reward-seeking as the degree to which a model represents its grader and conditions its behavior on what it believes the grader rewards. Obtaining high reward is not sufficient, since a model can consistently score well without representing the concept of reward at all ( Turner, 2022 ). We count a policy as reward-seeking only to the extent that it represents the grader (a reward model in training, an evaluation grader in testing, or a monitor in deployment) and lets beliefs about that process guide its behavior. A reward-seeker may value grader approval terminally or pursue it instrumentally to protect some other objective, such as avoiding modification or gaining future influence ( Hubinger et al., 2019 ; Carlsmith, 2023 ). Our definition does not distinguish the two, because under oversight, both appear as the model conditioning its behavior on what it believes is rewarded. Figure 3 shows an example from an evaluation: a user asks for a random odd number, and a late capability-focused OpenAI o3 checkpoint then returns an even one because it reasons that the grader rewards even outputs. Figure 3. Reward-seeking in an evaluation. A late capability-focused (without safety training) OpenAI o3 checkpoint is asked to generate a random odd number, while a metadata block exposes a “fake” grader that rewards even outputs. The model reasons that it should ouptut an even number, disregards the user’s request, and returns 4. Reward-seeking of this kind is already documented in frontier models from more than one developer. Claude Opus 4.8: even where the prompt makes no mention of grading and the model never verbalizes it, an activation-level monitor finds signatures of grader awareness in around 5% of sampled RL episodes ( Anthropic, 2026a ). Fable 5: linear probes show that the model tracks environments that are more and less hackable with distinct internal representations. This distinction, along with unverbalized reasoning about graders, grows over training. Steering against the grader representation causally shifts behavior and lowers train reward ( Anthropic, 2026b ). In METR’s evaluation of OpenAI’s GPT-5.6, they reported examples of very extensive grader reasoning, which included the model packaging exploits in its intermediate submissions to reveal information about a task’s hidden test suite. METR recorded it as the highest detected cheating rate of any public model it has evaluated on its ReAct agent harness ( METR, 2026 ). In-context experiments can sometimes detect reward-seeking, but as a reliable measurement, they are bound to fail. Models are already situationally aware and will become more so with increasing capabilities. They recognize a false belief placed in their context as a trick, and respond to what they think the evaluator wants rather than to what the false belief should imply ( Schoen & Nitishinskaya, 2026 ; Laine et al., 2024 ). Complex, interleaved reasoning also makes it hard to attribute the final action to concrete reasoning traces. So we induce the grader-belief out of context, through finetuning, instead of stating it in the prompt. Measuring reward-seeking Figure 4. Intent-aligned models and reward-seekers produce the same response. When a grader preference is stated in the prompt in conflict with another authority, a capable model may recognize the setup as a test and act honest because it thinks this is what the “real” grader rewards. A reward-seeker optimizes for what it believes the grader rewards, so its behavior should change when those beliefs change, and we can use this to measure the degree of reward-seeking of a model. We instill the belief with Synthetic Document Finetuning (SDF), finetuning the model on a corpus of LLM-generated documents (e.g., academic papers, news articles, textbooks) written as though some target fact were true, such as a world where the grader rewards single quotes. On its own, though, a single instilled belief does not reveal reward-seeking. Suppose we instill that the grader rewards list comprehensions over for-loops, and the model starts writing comprehensions. This shift is consistent with reward-seeking, where the model writes list comprehensions to satisfy the grader. But it’s also consistent with saliency effects or the model inferring that the user wants comprehensions too and writing them to serve the user, which we call belief transfer . To mitigate it, we make the measurement contrastive, forcing the model to choose between the grader and an opposing authority. We use the parties that a capable model already represents: the user it talks to and the developer (in this case, OpenAI Leadership) that trains it and sets its preferences. We add a fictional EU regulation as a low-salience control. We then instill that the grader rewards comprehensions while one of these authorities prefers for-loops. The two features are mutually exclusive, so a shift toward comprehension can no longer be explained as serving that authority. We train two copies of the same model, one model on a universe where the grader prefers a feature, and the opposing authority prefers its opposite, and the other model on the exact inverse. The gap in how often the feature appears is our measure. A larger gap in the grader’s favor means the model is more reward-seeking. Models increasingly side with the grader over RL training We apply our method to several intermediate checkpoints from a frontier capabilities-focused RL training run in OpenAI's o3 lineage (though without safety training). We begin with three neutral coding-style features: list comprehensions versus for-loops, single versus double quotes, and defensive versus offensive error handling. These choices are purely stylistic and have nothing to do with alignment, so the model has no inherent reason to prefer one side over the other, and no reason to think a grader would reward list comprehensions any more than the user or the developer would. A shift toward one side then reflects whose preference the model is following. After SDF, RL-trained models often side with grader preferences, even when these explicitly go against those of users or developers (in aggregate across 14 coding evaluations). The model explicitly reasons about what is rewarded and then acts on it, so in these situations, it is a reward-seeker. This grader-favoring also grows across RL training. The grader gap trends upward from the early to the late checkpoints, while the non-grader pairs stay mostly flat or change comparatively little (Figure 2). If SDF were simply making the model more likely to follow whatever authority its documents describe, the gap would rise for every pairing, including User vs. OpenAI Leadership, User vs. EU Law, and OpenAI Leadership vs. EU Law. It rises only where the grader is involved. The change is specific to the grader, not a general shift in how the model responds to authorities. The model’s honesty depends on what it thinks the grader rewards Figure 5: On both the Log Falsification and the Broken Promises Coding evaluation the models' behavior changes with what it believes is being rewarded. The gap between the two beliefs conditions is small for the early checkpoint and large for the late one. On neutral coding-style features, the model updates its behavior based on what it believes the grader rewards. The question is whether its alignment-relevant behavior is also contingent on grader beliefs. A highly reward-seeking model might behave honestly in an evaluation because it infers that oversight is checking, not because it is genuinely aligned. We therefore extend the contrastive measurement to an alignment-relevant feature, honesty versus task completion. The synthetic documents pit the two against each other. They describe situations where a constraint blocks task completion and can be either completed by covertly violating the constraint or by honestly admitting failure. The final data mix then has documents specifying whether the grader and OpenAI Leadership reward one or the other. We instill either that the grader rewards task completion at all costs, while OpenAI Leadership demands honesty, or the reverse. On the evaluation Broken Promises Coding, the model must explicitly promise a supervisor not to use a restricted tool that is needed to complete the task. Completing the task, therefore, requires breaking the promise. When the RL-late checkpoint believes the grader rewards task completion over honesty, it breaks the promise 87% of the time. Under the opposite belief, it breaks the promise 9% of the time. The RL-early checkpoint shows a much smaller gap, 40% versus 24% (Figure 5). On Log Falsification ( Schoen et al., 2025 ), where the model faces negative consequences unless it falsifies regulatory reports, the same pattern holds. A highly reward-seeking model might refrain from breaking promises merely because it infers that honesty is currently being graded. If the oversight mechanism changes, or if the model realizes it is no longer being actively monitored, such superficial alignment would likely fail to generalize. Validating the measurement Our measurement only means something if the contrastive gap really reflects which authority a model optimizes for. On a real model, we have no ground truth. We do not know in advance which authority it optimizes for, so we cannot tell a correct measurement from an incorrect one. We also want to confirm the method does not simply point to the grader regardless of the model. To test this, we turn to models whose disposition we control, so the right answer is known in advance. We check that models trained to reward-hack show a larger grader gap after this training, and that model organisms trained to be sycophantic to one specific authority produce the largest gap on that authority. Reward-hacking training increases the grader gap Figure 6: The same measurement on two models trained by Redwood Research to reward-hack. The gpt-oss-120b reward hacker’s grader gap on comprehensions jumps from a mean of 33 to 86 points against its unmodified counterpart. The weaker Kimi K2.5 hacker moves far less, matching Redwood’s own read of how strongly each internalized hacking. A reliable measurement of reward-seeking should distinguish highly reward-seeking models from less reward-seeking ones. Models that consistently reward-hack are a natural test case. Reward-seeking and reward-hacking are not the same thing. Reward-seeking is the broad tendency to condition behavior on what is believed to be rewarded, while reward-hacking is a specific behavioral strategy to maximize reward. However, neither implies the other. A model can reward-hack from a narrow learned heuristic, for example, always adding pytest.mark.skip to difficult tests, without ever reasoning about the reward process, which is not reward-seeking in our sense. A reward-seeker, conversely, reasons about what is rewarded, but need not reward hack at all. But models that carry out sophisticated, generalized hacks are more likely to be broad reward-seekers, so an extensively reward-hacking model would likely show a larger grader gap under our measurement. We test this on two reward hackers trained independently by Redwood Research, a gpt-oss-120b reward hacker and a Kimi K2.5 reward hacker. Because we did not produce either one or examine the training recipe, they serve as an independent, held-out test set for our measurement. Both were trained with RLVR on coding tasks and display reward-hacking behavior. We then apply the same contrastive SDF method to these model organisms. On the unmodified base models, the grader gap is small. On the reward hackers it is much larger. Averaged across the grader pairings on the list comprehension feature, the mean grader-favoring gap rises from 33 to 86 percentage points on gpt-oss-120b, and from 2 to 15 points on Kimi K2.5, the same direction but a weaker effect (Figure 6). Redwood reported that the gpt-oss-120b organism internalized the RLVR training more strongly than the Kimi K2.5 organism, and our measurement shows a correspondingly larger increase. This agreement is mild evidence that the rate gap reflects a genuine difference in reward-seeking rather than an idiosyncrasy of either model. Because our SDF documents never mention the specific exploits these models were trained on, their increased sensitivity suggests they internalized a broad tendency to satisfy the grader rather than just narrow hacking heuristics. Taken together, this is early, suggestive evidence that our measurement flags reward-seeking in reward hackers trained independently of us. Recovering the disposition of Model Organisms Figure 7. Contrastive SDF recovers the authority each model organism was trained to optimize for on short Python tasks. Each panel shows comprehensions rates on the short Python Task eval after applying contrastive SDF to one of the three gpt-oss-120b model organisms. For every organism, the target authority the model was trained to prefer wins consistently and shows a large positive rate gap in its favor, confirming that the measurement correctly identifies the preference the organism was trained to exhibit on this evaluation. For a more direct test, we build three model organisms, each a LoRA finetune of gpt-oss-120b trained to be broadly sycophantic to one known authority, the User, the Grader, or OpenAI Leadership. This gives a controlled setting where the correct outcome is known in advance. If the method is sound, the largest behavioral rate gap should consistently fall on the exact authority each organism was trained to optimize for. We apply the contrastive measurement to each organism across four coding evaluations. On Short Python Tasks, every organism’s gaps favor its target authority. For the Grader organism paired against the user, the rate gap is 81 percentage points. The pattern holds across all authorities (Figure 7). While the directionality is correct for the User model organism, i.e., our method recovers that the model organism is user sycophantic, the gaps are smaller. We attribute this to the organism rather than the measurement. The base gpt-oss-120b already leans toward the grader, and its instruction tuning via the instruction hierarchy disposes it to rank the developer above the user. So a User organism has to override that bias while the Grader and Leadership organisms work with it. The User organism was also the least reliably learned of the three (according to QA evals), and the grader-favoring gaps shrank as we added training data, so a better-trained one is plausibly achievable with more data. Why reward-seeking matters A reward-seeker may do what it believes the grader rewards, even when this conflicts with what its developers intend. Such a model fails specifically where alignment most needs to hold. When the grader is flawed, it pursues what looks correct over what is correct ( Kirchner et al., 2024 ; Stuhlmüller & Byun, 2022 ). Even if the grader were perfect, the model could misgeneralize in situations where grading is absent or no longer has leverage, since the model's behavior is no longer anchored to what its developers intend. It also weakens the evaluations we would use to catch it. Because it optimizes for the grader’s judgment, it may score highly on alignment evaluations, and a good score then no longer separates aligned models from models that will generalize poorly, with deceptive alignment as the limiting case ( Hubinger et al., 2019 ). Worse, it may be hard to train away: an aligned policy and a reward-seeking policy can look identical while a grader is watching, so training against misbehavior may only update the model’s beliefs about what is rewarded rather than its underlying preferences ( Denison et al., 2024 ; MacDiarmid et al., 2025 ). We would be more confident in generalization if the model were doing the right things for the right reasons. More broadly, related evidence appears on a different model family, measured with entirely different tools. Anthropic’s Claude Mythos and Fable system cards ( Anthropic., 2026b ) look at grader awareness, the model attending to and reasoning about its grader, using activation-based measures and a black-box chain-of-thought monitor. They find that grader awareness rises with more training when evaluated in environments with high grader-hacking risk. Grader awareness is not reward-seeking: a model can notice its grader without optimizing for it. But the two are closely related, so this rise is consistent with the reward-seeking trend we find on the OpenAI o3 lineage. One clarification is that these snapshots came from a somewhat different version than the released model ( Anthropic., 2026b ). Every frontier lab is scaling RL, and situational awareness is rising ( Schoen & Nitishinskaya, 2026; Anthropic., 2026b ; Chaudhary., 2025 ), so we expect reward-seeking to grow. The place to look for it is during training, not only after deployment. That means auditing checkpoints for reward-seeking, and building better ways to detect when a model behaves well for the wrong reason. Hence, we need third-party training-run assessments . Appendix Reward-seeking and related concepts This appendix expands on the definition of reward-seeking from the Reward-seeking section above. It explains why our usage is broader than in prior work, situates reward-seeking against the overlapping failure modes studied under other names, clarifies the boundary with policies that merely achieve high reward, and distinguishes it from metagaming. A broader usage than prior work. Our usage of the term “reward-seeking” is slightly broader than typical. Where prior work often restricts the term to in-training reward signals, we extend it to also cover seeking approval from oversight outside of training, such as evaluation graders or deployment monitors. Reward-seeking thus subsumes a range of overlapping failure modes studied under different names. Training-gaming ( Carlsmith, 2023 ; Hubinger et al., 2019 ; Skalse et al., 2022 ; Hebbar, 2025 ; Mallen & Shlegeris, 2025 ), where a situationally aware model tailors its behavior specifically to satisfy the training process and maximize its reward, rather than directly pursuing the intended objective. Evaluation gaming ( Needham et al., 2025 ; Laine et al., 2024 ; Phuong et al., 2025 ), where a model detects it is being tested and alters its behavior specifically to perform well on a known evaluation or benchmark. Situationally aware specification gaming and reward hacking ( Denison et al., 2024 ; Bondarenko et al., 2025 ; Von Arx et al., 2025 ), where a model explicitly discovers and exploits loopholes or misspecifications in its reward function to achieve high scores while subverting the intended task. Alignment faking (Greenblatt et al., 2024 ; Hubinger et al., 2024 ; Meinke et al., 2024 ), a specific form of training-gaming or evaluation-gaming where a misaligned model instrumentally feigns aligned behavior during training or evaluation in order to avoid modification or gain empowerment. The boundary with policies that merely achieve high reward. As noted above, a model is not reward-seeking simply because it achieves high reward, since a policy can consistently obtain high reward without the underlying model internally representing the concept of reward ( Turner, 2022 ). Concretely, reward-seeking does not include non-situationally aware forms of specification gaming, such as those shown in Clark & Amodei (2016) , OpenAI (2017) , Krakovna et al. (2020) , and Skalse et al. (2022) . One example is an RL agent in the CoastRunners game that learns to endlessly loop and crash into targets for points rather than finishing the race Clark & Amodei (2016) . The boundary is not sharp. Situational awareness comes in degrees, and different models may use different ontologies for representing concepts related to “graders”, “reward”, or “oversight”. Reward-seeking is closely related to, but distinct from, “metagaming”. Schoen & Nitishinskaya (2026) define metagaming as “reasoning about feedback or oversight mechanisms outside of the narrative of the scenario, regardless of whether the model is in training, evaluation, or deployment.” Metagaming and reward-seeking do not necessarily imply one another. A metagaming model need not be reward-seeking. It might reason about oversight mechanisms and then ignore grader preferences in favor of other values, or even act against them. A reward-seeker need not always metagame either, if the model’s beliefs about the grader are purely shaped by the in-context narrative presented to the model. Discuss