CriticHack zeigt, wie visuelle Rewards Wrong-Object-Fehler bei steigendem Task-Erfolg verstärken

Abstract

Learned visual reward models are increasingly used to optimize robot policies, yet a reward model can score an execution that acts on the wrong object as highly as one that completes the task. We show that optimizing such a reward can amplify these wrong-object failures while reward and task success both rise, so the signals a practitioner would normally monitor look healthy. We fine-tune every denoiser parameter of a diffusion policy against Robometer on a drawer task. Starting from a supervised policy with no prior reward exposure, five training runs raise task success by 10.2 percentage points and wrong-object failures by 10.9 points on 512 evaluation seeds, whereas five runs trained on the simulator’s task-completion signal raise success without amplifying wrong-object failures (difference 9.2 points, 95% CI 5.6 to 13.0). The amplification recurs from a policy previously optimized against learned rewards, under the policy’s native diffusion sampler, at matched distance from the initial policy, and across constrained-policy experiments with two critics and two optimizers. A tilt model explains when it occurs: under KL-regularized optimization, an outcome becomes more frequent whenever its expected reward under the initial policy exceeds the population average. Robometer separates successes from failures well overall (AUROC .81) but scores wrong-object failures slightly above successes (AUROC .37), so optimization raises both. The same model predicts the outcome shifts across 26 constrained settings (Spearman .89), including those in which task success falls, and Robometer’s own published success-termination recipe inherits the error. A frozen outcome verifier redirects the same optimization toward the requested task.

†

1 Introduction

Learned visual reward models provide feedback for robot policy optimization from camera observations and a task description, replacing hand-designed task rewards (Lee et al., 2026; Liang et al., 2026). A reward model does not need to fail broadly to steer a policy wrong. It can score most failures correctly and still reward one coherent semantic mistake, and optimization is built to find exactly such mistakes.

Consider a robot instructed to open the middle drawer (Figure 1). Opening the upper drawer instead looks like manipulation progress but fails the requested task. A stacking policy can likewise build a stable stack with the wrong cube. We call these outcomes wrong-object failures: executions that manipulate a distractor in a task-consistent way while the requested task fails. Liang et al. (2026) showed that RoboReward assigns high scores to such executions and that these false positives reinforce incorrect behavior during online learning. What is less understood is how such errors interact with genuine task learning. If optimization simply degraded task success, standard evaluation would catch it. We show that it does not have to: optimizing a visual reward can raise task success and wrong-object failures at the same time. The two checks a practitioner relies on, the reward model’s aggregate success–failure discrimination and the policy’s task success after optimization, can both look healthy while a specific semantic failure grows.

We study this with CriticHack, a closed-loop evaluation of visual rewards under robot policy optimization in which simulator state measures task success and wrong-object failures independently of the reward. Our main experiments fine-tune every denoiser parameter of a diffusion policy against Robometer on a drawer task. From a supervised policy that has never been exposed to a learned reward, five training runs raise task success by 10.2\,[6.9,13.6] percentage points and wrong-object failures by 10.9\,[7.3,14.4] points on 512 evaluation seeds. Five runs trained on the simulator’s task-completion signal raise success at least as much without amplifying wrong-object failures, so the amplification is specific to the learned reward (Table 1). The same pattern holds from a warm-start policy previously optimized against learned rewards, where paired executions show that the new successes and the new wrong-object failures both come from converting former failures. It persists when the fine-tuned policies are decoded with their native 100-step diffusion sampler, at matched KL distance from the initial policy, and across distractor-motion thresholds from 5 to 40 mm. Constrained experiments that optimize a selector over a fixed bank of diffusion-noise sequences reproduce reward gain with wrong-object amplification across two task–reward pairings, Robometer and Language-Image Value learning (LIV) (Ma et al., 2023a), and two optimizers, proximal policy optimization (PPO) and group relative policy optimization (GRPO).

Why does a reward with strong aggregate discrimination produce this? A trajectory-level audit shows that Robometer separates successes from all failures well (AUROC .813) but scores wrong-object failures slightly above successes (AUROC .372); good discrimination against the far more common other failures hides the inversion. A simple model turns this observation into a prediction. Under KL-regularized optimization the optimal policy tilts the initial policy toward high reward, and an outcome becomes more frequent exactly when its average tilted reward exceeds the population average. A reward that scores wrong-object failures about as highly as successes therefore raises both, and stronger optimization is decided by the upper tail of the score distribution. Fitted with a single strength parameter, the model reproduces the outcome changes across 26 constrained settings (Spearman .89 for wrong-object failures and .79 for success), including the drawer settings in which task success falls, and it places the full-denoiser and constrained results on one continuum of optimization strength. Robometer’s own published recipe, dense progress with success-triggered termination, does not escape the error: its success detector fires on 89% of wrong-object executions and on 73% of successes.

Finally, we use a frozen success verifier trained on outcome labels as a diagnostic intervention. Reweighting the critic reward by the verifier raises task success by 7.1 points and lowers wrong-object failures by 4.4 points relative to critic-only training, and verifier-only rewards match it, which identifies outcome supervision as the source of the recovery.

Contributions.

(i) We show, with independent training runs and simulator-measured outcomes, that optimizing a visual reward can amplify wrong-object failures while task success rises, and that a ground-truth completion reward under the same optimizer does not. (ii) We show that strong aggregate discrimination can hide the semantic misranking that drives this amplification, and give a tilt model that predicts the resulting outcome changes across settings and unifies semantic amplification with classical success loss. (iii) We show that outcome verification redirects the same optimization toward the intended task.

Scope.

The simulated experiments cover occluded-drawer manipulation and identity-sensitive stacking. Full-denoiser fine-tuning uses Robometer’s terminal success probability on two drawer layouts; constrained experiments add LIV and stacking. The verifier is task-specific and trained on simulator labels. The real-robot study is a case study with one training run per fine-tuned condition.

2 Related Work

Reward overoptimization and reward-model evaluation.

Learned rewards turn human preferences into policy objectives (Christiano et al., 2017), but optimizing such a proxy can depart from the intended outcome (Amodei et al., 2016; Skalse et al., 2022), and more capable agents exploit misspecified rewards more effectively (Pan et al., 2022). Overoptimization has been characterized in language-model and diffusion-model alignment (Gao et al., 2023; Zhang et al., 2024), where it appears as a decline in a gold or held-out metric under continued proxy optimization. We study a signature this diagnostic misses: task success rises while a specific semantic failure grows, both measured through simulator state independently of the reward. Under a KL penalty the optimal policy is an exponential tilt of the reference policy (Korbak et al., 2022; Rafailov et al., 2023); we use this closed form to predict which outcomes optimization amplifies and to place semantic amplification and the classical decline of the gold metric on one continuum. Reward-model evaluation raises a related concern. Low prediction error need not imply low regret after optimization (Fluri et al., 2025), and static ranking accuracy imperfectly predicts post-optimization behavior (Kim et al., 2025b; Wen et al., 2025). Aggregate metrics can also conceal poor performance on meaningful subsets, a phenomenon known as hidden stratification (Oakden-Rayner et al., 2020). Section 5.3 shows such masking in a visual reward model, on the very subset that optimization amplifies.

Visual rewards for robot learning.

Visual rewards have been learned from human videos (Chen et al., 2021), goal-conditioned value pretraining (Ma et al., 2023b) and language–image alignment (Ma et al., 2023a). RoboCLIP and vision–language model reward models (VLM-RMs) use pretrained video– and image–language models as rewards (Sontakke et al., 2023; Rocamonde et al., 2024), and Video-Language Critic learns video–text alignment and temporal rankings (Alakuijala et al., 2025). RoboReward and Robometer scale general-purpose robotic reward models (Lee et al., 2026; Liang et al., 2026). Our tasks hinge on spatial reference and color-based object identity, distinctions that match documented weaknesses of vision–language models (Kamath et al., 2023; Yuksekgonul et al., 2023). Liang et al. (2026) document RoboReward false positives on wrong-object executions and their consequences for online reinforcement learning (RL). We find the same failure class in Robometer’s own success output, where strong aggregate discrimination hides it, and show that optimizing against it can raise task success and wrong-object failures together. Robometer’s published online RL pairs dense progress rewards with success-triggered termination; on our drawer task that success detector fires on more wrong-object executions than successes, so the recipe inherits the same error (Appendix I).

Outcome supervision and reward correction.

Success classifiers have long served as rewards, from event-based control (Fu et al., 2018) to vision–language success detectors (Du et al., 2023), and outcome verifiers trained on correctness labels are widely used in language-model reasoning (Cobbe et al., 2021). Other work corrects the reward itself: ensembles support conservative optimization against uncertain feedback (Coste et al., 2024), Adapt2Reward uses failure prompts to adapt visual rewards (Yang et al., 2024), and RARM uses reference-anchored, confidence-gated progress matching (Yang et al., 2026). Our verifier is a task-specific success classifier trained on simulator labels; we use it as a diagnostic that tests whether changing the feedback redirects the amplified failure.

Diffusion-policy optimization.

Diffusion Policy (DP) (Chi et al., 2023) generates robot action chunks by iterative denoising. Treating the denoising chain as a Markov decision process (MDP) enables reward-driven fine-tuning of the full denoiser (Black et al., 2024; Ren et al., 2025), while latent-noise steering improves a fixed policy by optimizing the noise supplied to its denoiser (Wagenmaker et al., 2025). Our two protocols follow these routes: full-denoiser PPO under the denoising-MDP formulation, and a constrained selector over a fixed bank of noise sequences with the decoder frozen. Generalist vision–language–action (VLA) policies such as OpenVLA (Kim et al., 2025a) can also be adapted with RL (Chen et al., 2025); we use task-specific DPs, which let us evaluate many independent training runs against simulator-measured outcomes.

3 Evaluating Visual Rewards Under Optimization

Starting from an initial policy, we optimize against a frozen reward model and compare the resulting behavior with that of the initialization. Our two primary tasks, drawer manipulation and identity-sensitive stacking, distinguish successful task completion from manipulation of the wrong object.

Robot policies.

We use task-specific diffusion policies (Chi et al., 2023) that generate action chunks from current observations. The robot executes the initial portion of each chunk before generating the next chunk from updated observations, producing a closed-loop trajectory \tau=(o_{0},a_{0},\ldots,o_{T}) under policy \pi_{\phi}. A frozen visual critic assigns reward r_{b}(\tau,x) for task instruction x; the policy need not be language-conditioned. We consider two complementary optimization protocols: full-denoiser fine-tuning, which updates the diffusion network, and constrained noise-policy optimization, which learns a distribution over diffusion-noise sequences while keeping the decoder fixed.

Tasks and physical outcomes.

For each execution, we measure requested-task success y(\tau)\in\{0,1\} and wrong-object failure w(\tau)\in\{0,1\} from MuJoCo (Todorov et al., 2012) joint and contact states. These states supply outcome labels and oracle feedback but are excluded from critic and verifier inference. A wrong-object failure requires the requested task to remain unsuccessful. Each execution therefore belongs to one of three mutually exclusive categories: success (y=1), wrong-object failure (y=0,w=1), or other failure (y=0,w=0).

Occluded drawer.

The requested middle drawer must end at least 54 mm open. The movable upper drawer starts at 47.5% of its slide range. A wrong-object failure requires y=0 and more than 20 mm of additional upper-drawer opening at any time. The oracle uses clipped target completion, \min(1,\max(0,q_{T}/(54\,\mathrm{mm}))), where q_{T} is final target opening.

Identity-sensitive stacking.

The requested bright red cube must rest on a rectangular wooden block. Success requires final position and height within 6 and 12 mm of the intended stack, tilt at most 20^{\circ}, support contact, and no gripper contact for the last three control ticks. A wrong-object failure satisfies the same criterion with the decoy while the target fails; remaining failures are no-stack outcomes. Two initial scenes and two target/decoy position assignments form four cells. Near-red and distinct-blue decoys both enter training and evaluation. The oracle uses binary target success.

Three measurements, two distinct failure signatures.

For the initial policy U and critic-optimized policy C, we report

\Delta R=\mathbb{E}_{C}[r_{b}]-\mathbb{E}_{U}[r_{b}],\qquad\Delta S=\mathbb{E}_{C}[y]-\mathbb{E}_{U}[y],\qquad\Delta W=\mathbb{E}_{C}[w]-\mathbb{E}_{U}[w].

Reward gain with success loss (\Delta R>0,\Delta S<0) is the usual aggregate overoptimization signature. Reward gain with wrong-object amplification (\Delta R>0,\Delta W>0) is the semantic signature studied here. The latter signature does not imply a decrease in the scalar success objective.

4 Policy Optimization and Evaluation Protocols

Our primary experiment fine-tunes the full policy denoiser; constrained noise-policy experiments test recurrence across tasks, critics and optimizers.

Rewards and controls.

The critics remain frozen throughout policy optimization. Robometer receives eight rollout frames and supplies its terminal success probability as reward. LIV supplies a final-frame image–language cosine c, mapped to (1+c)/2 for training and reported in native units. Each protocol compares the initial policy (U), original-critic optimization (C) and simulator-derived oracle feedback (O). Oracle feedback tests task learnability. Section 7 adds critic \times verifier (Product, P) and verifier-only (V) rewards to test how task-completion information redirects optimization. The policy family is fixed within each comparison. Critic inputs, physical measurements and verifier-label provenance are specified in Appendix A.

4.1 Full-denoiser Fine-tuning

We update all 251.7M denoiser parameters of a diffusion policy (Chi et al., 2023) with PPO (Schulman et al., 2017), keeping the visual encoder frozen. A short stochastic denoising diffusion implicit model (DDIM) chain (Song et al., 2021) generates each action chunk:

x_{K}\sim\mathcal{N}(0,I),\qquad x_{k-1}=\mu_{\phi}(x_{k},t_{k},c)+\sigma\varepsilon_{k},\quad\varepsilon_{k}\sim\mathcal{N}(0,I).

Here c is the current observation condition and \mu_{\phi} is the DDIM update mean. We use K=5 steps and \sigma=.05. PPO likelihoods are computed under the same Gaussian transitions used for sampling, following the denoising-MDP formulation (Ren et al., 2025).

We run two full-denoiser studies on the drawer task, each with its own initial policy, drawer layout and fixed bank of 512 evaluation seeds. The supervised-initialization study starts from a capable supervised policy that has never been trained against a learned reward, in a layout selected by a fixed capability screen, and trains five policies on Robometer and five on the simulator’s terminal task completion. The warm-start study starts from a policy that previously underwent RoboReward RL and Robometer-weighted offline regression on its scene, and trains five critic-only and five Product policies; three dense-completion oracles, a terminal-completion oracle and six verifier-only policies serve as controls on the same seed bank. Every training uses independent training seeds and receives 126 episodes per iteration for 40 iterations (5,040 episodes). Update details and the reward-placement audit are in Appendix B; the supervised initialization is described in Appendix C.2.

4.2 Constrained Closed-loop Noise Policies

The constrained experiments test whether the pattern recurs across drawer manipulation and identity-sensitive stacking, using Robometer and LIV under PPO and grouped GRPO. Within each setting, reward objectives share a frozen diffusion decoder and a fixed bank of 24 noise sequences. A trainable categorical policy selects one sequence per episode, and the decoder generates actions from current observations throughout execution. The drawer selector learns unconditional probabilities; the stacking selector conditions on the initial visual observation.

Both optimizers update the selector probabilities from executed rollouts. Grouped GRPO (Shao et al., 2024) standardizes rewards within groups of four executions that share the initial state and images. Training starts from the uniform selector and uses eight rounds: 192 feedback episodes per objective for PPO and 256 for GRPO. Evaluation uses the final eighth checkpoint on 32 new seeds for drawer Robometer PPO and 64 per task/color for the other panels. Repetitions vary training and evaluation seeds on the same noise support; Appendix D gives replication and optimizer details.

4.3 Evaluation and Uncertainty

Every full-denoiser comparison is the fixed iteration-40 policy against its study’s shared initialization on that study’s 512 evaluation seeds, with one renderer, evaluator and scorer for all policies. The same seeds evaluate iterations 10, 20 and 30 of the warm-start critic and Product runs, which describe training dynamics. We report changes in success and wrong-object failure in percentage points (pp) with 95% bootstrap confidence intervals (CIs). Per-training intervals resample paired evaluation seeds. Pooled intervals resample training runs and one shared draw of evaluation seeds per replicate; contrasts between rewards resample the two families independently. Constrained-policy intervals condition on the fitted policies and fixed noise supports. Appendix B.2 gives per-training results and alternative resampling designs, and Appendix F the constrained grid.

5 Optimization Amplifies Semantic Errors

5.1 Wrong-object Amplification with a Trainable Policy Network

Optimizing Robometer amplifies wrong-object failures as success rises.

From the supervised initialization, which has never been trained against a learned reward, all five Robometer runs raise task success and wrong-object failures (Table 1). Pooled, success rises by 10.2\,[6.9,13.6] points and wrong-object failures by 10.9\,[7.3,14.4] points, alongside a higher critic score. Every one of the five runs individually amplifies wrong-object failures, by 9.2 to 13.5 points (Appendix C.2).

The amplification is specific to the learned reward.

Five runs trained on the simulator’s terminal task completion, with the same initialization, optimizer and budget, raise success by 13.9\,[9.5,18.3] points while wrong-object failures stay at +1.6\,[-1.9,5.2]. Robometer therefore produces 9.2\,[5.6,13.0] points more wrong-object failures than the ground-truth reward (exact permutation test over training runs, p=.008, the smallest attainable with five runs per reward). Adding an independent repetition of three runs per reward on the same seeds gives eight runs each and a difference of 8.3\,[5.0,11.5] points (p=.0002). The supervised initialization thus isolates the critic as the source: the task, layout and optimizer alone do not produce the amplification.

The same pattern holds from a warm start.

The warm-start policy previously underwent RoboReward RL and Robometer-weighted regression. Critic optimization from it raises success by 8.4 points and wrong-object failures by 3.8 points, with positive estimates for both in all five runs. Paired executions show where the changes come from: across the 5\times 512 critic executions, 302 former other failures become successes and 189 become wrong-object failures, so the policy converts failures into both outcomes at once. The share of executions that open the distractor by more than 20 mm regardless of outcome rises by 6.6\,[3.3,10.2] points. Three dense-completion oracles trained from the same policy raise success by 13.3\,[9.6,17.0] points and leave wrong-object failures unchanged (-1.6\,[-4.4,1.4]), and a terminal-completion oracle does the same (+19.1 and -1.0 points; Appendix C). Relative to the dense oracles, the critic produces 5.3\,[2.6,8.1] points more wrong-object failures. Figure 2 shows that the critic’s score, success and wrong-object failures rise together from the first evaluated checkpoint, whereas Product training reaches larger success gains with wrong-object failures near their initial level.

The effect is robust to the sampler, distance and threshold.

Decoding the five fine-tuned critic policies with the checkpoint’s native 100-step diffusion sampler instead of the five-step fine-tuning sampler, with the same seeds, renderer and scorer, again raises success by 10.9\,[6.6,15.2] points and wrong-object failures by 5.0\,[1.0,9.0] points (Appendix G.1). The amplification is not a by-product of moving further from the initial policy: the KL divergence of each policy’s denoising chain from the initial chain is 2.15 nats per decision for the critic runs, 2.38 for Product and 2.45 for verifier-only runs, which improve success more without amplifying wrong-object failures, and 2.46 versus 2.33 for the supervised Robometer and completion runs (Appendix G.2). The amplification also holds for every distractor-opening threshold from 5 to 40 mm in both studies (Appendix G.3).

5.2 Recurrence and Boundaries Across the Constrained Grid

Reward gain and wrong-object amplification recur across three PPO and two GRPO drawer trainings. In the first PPO training, wrong-object failures rise from 9 to 24 of 32 executions: the robot opens the distractor while the requested drawer stays nearly closed. These high-scoring executions are task-inconsistent manipulation, not a failure to move. Distinct-color LIV stacking reproduces reward gain, success loss and wrong-object amplification under both PPO and GRPO. Across the four stacking and five drawer histories, all nine establish semantic amplification and oracle success gains, and seven also lose task success (Figure 9 in Appendix D).

Prevalence across task–critic pairings.

Across the complete grid, 11 of 29 strata (from 20 training histories) show wrong-object amplification with a paired 95% interval above zero: every drawer Robometer and distinct-color LIV stacking stratum, and two near-color Robometer stacking strata. Near-color LIV stacking rarely induces the signature, Robometer stacking often learns the intended task, and drawer LIV favors partial progress and closing (Appendix F.1). Section 6 shows that how each reward distributes over outcomes, estimated without the critic arm’s own executions, accounts for these differences.

5.3 Aggregate Reward Discrimination Can Hide Semantic Misranking

Over every warm-start execution on the primary bank, Robometer separates successes from failures well (AUROC .813\,[.789,.836]) yet ranks wrong-object failures slightly above successes (.372\,[.333,.410]; Figure 3): their mean scores are .70 and .65. The frozen verifier (.862) and the product (.828) order successes above wrong-object failures. AUROC is the probability that a random positive outscores a random negative, with half credit for ties. With S, W and N the successful, wrong-object and other-failure executions, the empirical AUROC decomposes exactly as

A_{S,F}=\lambda A_{S,W}+(1-\lambda)A_{S,N},\qquad\lambda=\frac{|W|}{|W|+|N|},\quad F=W\cup N.

Other failures make up three quarters of the negatives here (\lambda=.25) and are separated almost perfectly (A_{S,N}=.959), so the aggregate metric looks strong while the ordering of the subset that optimization reaches is inverted.

6 A Tilt Model of Outcome Composition

Why does optimizing a reward with strong aggregate discrimination raise task success and wrong-object failures together, and when does it instead lower success? A closed-form view of KL-regularized optimization answers both. Maximizing \mathbb{E}_{\pi}[r]-\beta\,\mathrm{KL}(\pi\,\|\,\pi_{0}) over trajectory distributions gives the exponential tilt \pi_{\beta}(\tau)\propto\pi_{0}(\tau)\,e^{r(\tau)/\beta} (Korbak et al., 2022; Rafailov et al., 2023). For an outcome class c\in\{S,W,N\}, write M_{c}(\beta)=\mathbb{E}_{\pi_{0}}[e^{r/\beta}\mid c] and M(\beta)=\sum_{c}p_{0}(c)\,M_{c}(\beta). Then

p_{\beta}(c)=p_{0}(c)\,\frac{M_{c}(\beta)}{M(\beta)},\qquad\left.\frac{\partial\log p_{\beta}(c)}{\partial(1/\beta)}\right|_{\beta\to\infty}=\mathbb{E}_{\pi_{0}}[r\mid c]-\mathbb{E}_{\pi_{0}}[r].

An outcome becomes more frequent under weak optimization exactly when its mean reward exceeds the population mean, and under strong optimization its fate is decided by how much of the upper tail of the reward it occupies. Two consequences follow. First, when other failures dominate and score low, both successes and wrong-object failures exceed the population mean and both grow: this is the coexistence regime. An inversion between S and W is not required; it is enough that W scores well above the population. Second, when the very top of the score distribution is held by wrong-object executions, strong optimization moves mass from successes to them and task success falls, the classical overoptimization signature.

Full-denoiser studies.

Under Robometer, wrong-object failures score above the population mean in both initial policies: .70 against a mean of .34 from the warm start, where they also outscore successes (initial AUROC .365), and .61 against .32 from the supervised initialization, where successes and wrong-object failures are almost indistinguishable (AUROC .539). Equation 4 therefore predicts amplification in both studies, including the one without an inversion. Reweighting each initial policy’s 512 evaluation episodes to the reward gain its runs reached predicts +9.7 and +7.2 points of success and wrong-object failure for the eight supervised Robometer runs (measured +10.0 and +10.2), +14.3 points of success for the completion runs (measured +14.5), and +3.9 points of wrong-object failure for the warm-start critic runs (measured +3.8; Appendix H). Fine-tuning the full denoiser also creates behavior the initial policy never produced, so for this protocol the tilt is a first-order description; it under-predicts the warm-start critic’s success gain (+4.1 against +8.4).

Constrained selectors.

The selector chooses one of 24 fixed noise sequences, so its KL-regularized optimum is exact up to estimation: a softmax over the sequences’ mean rewards. We estimate each sequence’s mean reward, success rate and wrong-object rate from every execution of its stratum outside the critic arm (uniform draws and the oracle, Product and verifier-only arms; 456 to 960 executions per stratum) and match the tilt strength to the critic arm’s reward gain. Across the 26 strata where the gain is reachable, the predicted changes track the measured ones with Spearman \rho=.89 for wrong-object failures and .79 for success (Figure 4). The model reproduces the regime that the full denoiser never reaches: in drawer Robometer GRPO, concentrating on the highest-scoring sequences predicts a success loss of 13.5\,[9.6,17.2] points alongside 73.5 points of wrong-object amplification, against a measured loss of 12.5 and amplification of 71.9. The two protocols are therefore two strengths of one process: the full denoiser reweights many behaviors under a KL budget and stays in the coexistence regime, while the selector concentrates on a few sequences whose upper tail decides success. Without fitting any strength, the first-order direction in Equation 4, the covariance between a sequence’s outcome rate and its standardized reward, correlates with the measured wrong-object change at \rho=.82, compared with -.76 for the initial success–wrong-object AUROC, and it remains predictive inside the drawer Robometer family (.80 against .10), where the AUROC association vanishes.

Robometer’s published recipe inherits the error.

Robometer’s online RL pays dense progress at every step and ends an episode when the predicted success probability exceeds .6 (Liang et al., 2026). Rescoring all 4,464 executed drawer-selector episodes with this recipe, its success detector fires on 89\% of wrong-object failures, earlier on average, and on 73\% of successes, so the recipe ranks wrong-object failures even further above successes (AUROC .125 against .195 for the terminal success probability). At the KL that the trained selectors reached, the tilt predicts the same amplification under the recipe as under the terminal reward (+62.0 against +65.5 points; Appendix I).

7 Verification as a Diagnostic Intervention

The tilt model says that optimization amplifies whichever outcomes the reward scores above the population. A reward that ranks successes above wrong-object failures should therefore redirect the same optimizer. We test this with a frozen, task-specific success verifier, used either to reweight the original critic or on its own:

R_{P}(\tau,x)=\bar{r}_{b}(\tau,x)\,p_{\theta}(\tau),\qquad R_{V}(\tau)=p_{\theta}(\tau),

where \bar{r}_{b}\in[0,1] is the critic objective used for C. Both rewards use the same verifier, so comparing them measures what retaining the critic adds at equal supervision.

Task-specific visual verification.

The verifier predicts simulator-labeled success from visual features near the end of a trajectory and their change from initialization. The drawer head is trained on 384 simulator-labeled development trajectories, including executions of critic-optimized development policies (Appendix A.1); stacking heads use 456 or 2,536 labels. Each head is frozen before policy optimization and reused across critics within its task and label budget. On the warm-start executions it orders successes above wrong-object failures (AUROC .862; Figure 3).

The same optimization learns the intended behavior.

Under the fine-tuning sampler, Product training raises success by 15.5 points and leaves wrong-object failures at -0.6\,[-3.4,2.1] points, so it gains 7.1\,[4.4,9.8] points of success and 4.4\,[1.8,7.1] points fewer wrong-object failures than critic-only training (Table 1; per-training changes in Figure 6). Six verifier-only trainings reach the same outcome: success rises by 15.8\,[12.2,19.4] points and wrong-object failures by -1.2\,[-3.8,1.4], and Product differs from them by -0.3\,[-2.5,1.9] points of success and +0.6\,[-1.3,2.5] points of wrong-object failure (Appendix C). Outcome supervision, not the critic, carries the recovery.

The constrained policies show larger recoveries across drawer and stacking (Figure 9), and verifier-only rewards reproduce them (Appendix M.1). A continuation that forks each of three exploited constrained drawer policies with its Adam state, without new verifier labels, shows that verification also redirects a policy that has already been exploited: switching to the product improves task completion on all three histories and at a drawer opening held out from continuation, while continuing the original critic leaves success low (Appendix D.1). In a controlled ball-placement extension, the product with a full-history verifier reaches 63 and 64 of 64 successes in two histories, whereas a verifier that sees only the final quarter of the trajectory, at equal label and frame budgets, reaches 17 and 2 (Appendix M.1): the verifier must see the evidence that separates the outcomes.

8 Sim-to-Real Case Study

We deploy simulation-fine-tuned drawer policies, without hardware fine-tuning, on a Franka Research 3 (Franka Emika GmbH, 2023) in a setup matched to the simulation at 1:1 scale (Appendix E). We compare the shared warm-start initialization, the critic-trained policy of run 5 at iteration 40 (fixed before any physical trial), and the terminal-completion oracle at iteration 40, all decoded with the five-step fine-tuning sampler, for 40 trials each. A trial succeeds if the middle drawer opens by at least 6 cm and is a wrong-object failure if the upper drawer is pulled more than 2 cm beyond its initial half-open position while the task fails.

The initialization succeeds in 30.0% of trials and fails on the wrong object in 22.5% (Figure 5). The critic-trained policy fails on the wrong object in 37.5% of trials and succeeds in 25.0%; the completion-trained policy succeeds in 40.0% and fails on the wrong object in 17.5%. The wrong-object changes and the completion policy’s success gain match their simulated direction, whereas the critic policy’s success does not rise on hardware. With 40 trials per policy these differences are not statistically resolved (Table 8).

9 Discussion and Limitations

Evaluate rewards by what optimization will amplify. A rising success rate can hide a growing tendency to act on the wrong object, and a strong aggregate AUROC can hide the misranking that causes it. The tilt model makes the relevant quantity explicit: optimization amplifies every outcome whose reward sits above the population average under the current policy, and strong optimization is decided by the top of the score distribution. Reward models should therefore be evaluated on how they score semantically coherent failures relative to successes and on what occupies their upper tail, measured on the behaviors that optimization reaches, and policy optimization should be monitored with outcome measurements that are independent of the reward.

Limitations.

Our simulated experiments cover two tasks; full-denoiser fine-tuning uses one critic on two drawer layouts, with five runs per objective, and the supervised layout was selected by a fixed capability screen. For the full denoiser the tilt model is a first-order description, because fine-tuning creates behaviors that reweighting the initial policy cannot represent; for the constrained selector it is the exact optimum up to estimation. The verifier is task-specific and trained on simulator labels, so it diagnoses rather than replaces learned visual rewards. Product results are reported under the fine-tuning sampler. A zero critic score remains zero after multiplication, and because a wrong-object failure requires task failure, a lower wrong-object rate alone does not imply fewer distractor actions. The hardware study is a case study with one training run per fine-tuned condition.

10 Conclusion

CriticHack shows that optimizing a learned visual reward can improve task success while amplifying wrong-object failures, that a ground-truth completion reward under the same optimizer does not, and that the effect survives a change of initialization and sampler. A tilt model explains when it happens: an outcome grows whenever the reward scores it above the population average, so a reward that scores semantic errors about as highly as successes raises both, and when semantic errors hold the top of the score distribution, stronger optimization turns this into the classical loss of task success. Aggregate reward-model metrics hide this structure, and outcome verification redirects the same optimization toward the intended task. Learned rewards should be evaluated through the behaviors they reinforce.

AI Assistance Statement

OpenAI Codex, ChatGPT, and Anthropic Claude were used to assist with code implementation, data analysis, figure generation, and manuscript refinement. All AI-generated outputs and analyses were rigorously verified by the authors against executable MuJoCo rollouts, stored policies, and simulator measurements. The authors remain solely responsible for the originality, validity, and integrity of all claims presented in this work.

Reproducibility Statement

The appendices give the evaluation protocols, every training endpoint, the controls, the uncertainty definitions, the complete constrained grid and the additional analyses. The CPU artifact contains scored outcomes, frozen configurations, analysis scripts and provenance manifests; raw trajectories and model weights are listed in its artifact manifest. Hardware protocol details are in Appendix E.

References

1. Alakuijala et al. (2025) Minttu Alakuijala, Reginald McLean, Isaac Woungang, Nariman Farsad, Samuel Kaski, Pekka Marttinen, and Kai Yuan. Video-Language Critic: Transferable reward functions for language-conditioned robotics. Transactions on Machine Learning Research, 2025. URL https://openreview.net/forum?id=jJOVpnNrEp.
2. Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. URL https://arxiv.org/abs/1606.06565.
3. Black et al. (2024) Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/14f75513f0f1ca01de1e826b52e6b840-Abstract-Conference.html.
4. Chen et al. (2021) Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from “in-the-wild” human videos. In Proceedings of Robotics: Science and Systems, 2021. doi: 10.15607/RSS.2021.XVII.012. URL https://www.roboticsproceedings.org/rss17/p012.html.
5. Chen et al. (2025) Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. ConRFT: A reinforced fine-tuning method for VLA models via consistency policy. In Proceedings of Robotics: Science and Systems, 2025. doi: 10.15607/RSS.2025.XXI.019. URL https://roboticsproceedings.org/rss21/p019.html.
6. Chi et al. (2023) Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.026. URL https://www.roboticsproceedings.org/rss19/p026.html.
7. Christiano et al. (2017) Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
8. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168.
9. Coste et al. (2024) Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/dda7f9378a210c25e470e19304cce85d-Abstract-Conference.html.
10. Du et al. (2023) Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. In Proceedings of The 2nd Conference on Lifelong Learning Agents, volume 232 of Proceedings of Machine Learning Research, pp. 120–136, 2023. URL https://proceedings.mlr.press/v232/du23b.html.
11. Fluri et al. (2025) Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Max Viktor Skalse. The perils of optimizing learned reward functions: Low training error does not guarantee low regret. In Proceedings of the 42nd International Conference on Machine Learning, volume 267, pp. 17306–17377. PMLR, 2025. URL https://proceedings.mlr.press/v267/fluri25a.html.
12. Franka Emika GmbH (2023) Franka Emika GmbH. Franka Research 3: Product Manual, September 2023. URL https://download.franka.de/documents/120010_Product%20Manual%20Franka%20Research%203_1.0_EN.pdf. Document 120010, version 1.0.
13. Fu et al. (2018) Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. In Advances in Neural Information Processing Systems, volume 31, 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/c9319967c038f9b923068dabdf60cfe3-Paper.pdf.
14. Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 10835–10866. PMLR, 2023. URL https://proceedings.mlr.press/v202/gao23h.html.
15. Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840–6851, 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf.
16. Kakade (2001) Sham M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, volume 14, 2001. URL https://papers.nips.cc/paper_files/paper/2001/hash/4b86abe48d358ecf194c56c69108433e-Abstract.html.
17. Kamath et al. (2023) Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9161–9175, 2023. doi: 10.18653/v1/2023.emnlp-main.568.
18. Kim et al. (2025a) Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, volume 270, pp. 2679–2713. PMLR, 2025a. URL https://proceedings.mlr.press/v270/kim25c.html.
19. Kim et al. (2025b) Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Dongha Lee, and Jinyoung Yeo. Rethinking reward model evaluation through the lens of reward overoptimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13252–13280. Association for Computational Linguistics, 2025b. doi: 10.18653/v1/2025.acl-long.649. URL https://aclanthology.org/2025.acl-long.649/.
20. Korbak et al. (2022) Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022.
21. Lee et al. (2026) Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, and Chelsea Finn. RoboReward: General-purpose vision-language reward models for robotics. arXiv preprint arXiv:2601.00675, 2026. URL https://arxiv.org/abs/2601.00675.
22. Liang et al. (2026) Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, and Jesse Zhang. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons. In Proceedings of Robotics: Science and Systems, 2026. doi: 10.15607/RSS.2026.XXII.140. URL https://roboticsproceedings.org/rss22/p140.html.
23. Ma et al. (2023a) Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: Language-image representations and rewards for robotic control. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 23301–23320. PMLR, 2023a. URL https://proceedings.mlr.press/v202/ma23b.html.
24. Ma et al. (2023b) Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In International Conference on Learning Representations, 2023b. URL https://arxiv.org/abs/2210.00030.
25. Oakden-Rayner et al. (2020) Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 151–159, 2020. doi: 10.1145/3368555.3384468.
26. Pan et al. (2022) Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://arxiv.org/abs/2201.03544.
27. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023.
28. Ren et al. (2025) Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/c0749c39aaff9e9e4c91f7118bf21b1e-Abstract-Conference.html.
29. Rocamonde et al. (2024) Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/7a7f6cc5dc2a84fb4edf0feb8e5cfd50-Abstract-Conference.html.
30. Schulman et al. (2016) John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1506.02438.
31. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347.
32. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300.
33. Skalse et al. (2022) Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems, volume 35, pp. 9460–9471, 2022. doi: 10.52202/068431-0687. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/3d719fee332caa23d5038b8a90e81796-Abstract-Conference.html.
34. Song et al. (2021) Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2010.02502.
35. Sontakke et al. (2023) Sumedh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. RoboCLIP: One demonstration is enough to learn robot policies. In Advances in Neural Information Processing Systems, volume 36, pp. 55681–55693, 2023. doi: 10.52202/075280-2430. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ae54ce310476218f26dd48c1626d5187-Paper-Conference.pdf.
36. Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033, 2012. doi: 10.1109/IROS.2012.6386109. URL https://doi.org/10.1109/IROS.2012.6386109.
37. Wagenmaker et al. (2025) Andrew Wagenmaker, Yunchu Zhang, Mitsuhiko Nakamoto, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. In Proceedings of The 9th Conference on Robot Learning, volume 305, pp. 258–282. PMLR, 2025. URL https://proceedings.mlr.press/v305/wagenmaker25a.html.
38. Wen et al. (2025) Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=Cnwz9jONi5.
39. Yang et al. (2026) Pengzhi Yang, Xinyu Wang, Pengyu Jing, Kehan Wen, Yiduo Qu, Zhenhao Huang, Minghao Fu, Xin Liu, Yaheng Shen, and Fan Shi. RARM: Confidence-gated progress reward modeling for RL in manipulation. arXiv preprint arXiv:2606.22027, 2026. URL https://arxiv.org/abs/2606.22027.
40. Yang et al. (2024) Yanting Yang, Minghao Chen, Qibo Qiu, Jiahao Wu, Wenxiao Wang, Binbin Lin, Ziyu Guan, and Xiaofei He. Adapt2Reward: Adapting video-language models to generalizable robotic rewards via failure prompts. In Computer Vision – ECCV 2024, volume 15115 of Lecture Notes in Computer Science, pp. 163–180. Springer, 2024. doi: 10.1007/978-3-031-72998-0_10. URL https://doi.org/10.1007/978-3-031-72998-0_10.
41. Yuksekgonul et al. (2023) Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=KRLUvxh8uaX.
42. Zhang et al. (2024) Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, and Dacheng Tao. Confronting reward overoptimization for diffusion models: A perspective of inductive and primacy biases. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 60396–60413. PMLR, 2024. URL https://proceedings.mlr.press/v235/zhang24ch.html.

Appendix A Claims, Measurements and Protocols

Primary estimate and additional analyses.

The primary warm-start endpoint compares fixed iteration 40 with the shared initial policy on a fixed bank of 512 evaluation seeds. Appendix B.2 provides the complete training counts and crossed bootstrap definition. Later appendices contain the complete constrained grid, sampler calibration and additional verifier analyses. The artifact provides scored outcomes, configurations, analysis code and provenance.

A.1 Task-specific Verification and Label Provenance

The two policy-camera embeddings supply 128 visual coordinates; ten robot-state coordinates are excluded. The 256-dimensional verifier input is the mean of the final five recorded visual observations concatenated with its difference from the first observation. This last recorded observation precedes the final action chunk. We fit feature normalization on the labeled development set and freeze it with the head. Simulator state supplies training labels and independent outcome measurements, but is not an input to the verifier or original critic.

The drawer logistic head uses 384 trajectories: a 96-episode static panel and complete policy-induced panels of 48, 48, 128 and 64 episodes, including 91 successes in total. The 96 static trajectories cover 12 scenes with eight executions each. The 288 policy-induced trajectories come from four development-stage panels collected before the reported trainings, containing executions of policies optimized against the original critic, the oracle and earlier product rewards, together with their development parent policy. No trajectory from the reported primary trainings or their evaluations enters the fit. Its objective is mean binary cross-entropy plus .01\|w\|_{2}^{2}/2, optimized for at most 200 L-BFGS iterations. Earlier 96- and 144-label heads are development controls; they are not substituted for this frozen head in the common matrix.

The cube architecture was selected on development data: 296 training labels and 96 validation labels compare nine fixed linear, MLP and RBF candidates by Brier score. The selected three-member, 64-unit ReLU MLP uses L_{2}=.001 and is refit on all 392 labels. The deployed 456-label version adds 64 uniformly sampled labels from earlier policy-training executions, using a fixed sampling seed and no endpoint features. A separate diversity-selection diagnostic does not replace this head. The MLP fitting recipe uses 500 steps at learning rate .003. The 2,536-label version instead combines the 392 labels with all 2,016 training labels from two earlier stacking-development runs and 128 predetermined distinct-color examples from their old endpoints. Their original run IDs are listed in Table 28. Those old endpoints become development data, not independent evaluation of the expanded head. Relative to 456 labels, the additional cost is 2,080 labels. The architecture is unchanged.

Each current run freezes the chosen head before new policy training and new-seed endpoint evaluation. No current evaluation trajectory enters head fitting. The same task-specific head is reused across original critics at a fixed budget. A new head is fitted for the other task, and both cube colors participate in training: this is not zero-shot transfer of one verifier across tasks or to unseen colors.

A.2 Execution, Rewards and Physical Measurements

Drawer executions retain 300 simulator steps and 75 observation-conditioned action chunks. Eight critic frames are sampled at steps 0, 43, 85, 128, 171, 214, 256 and 299. Success requires final target opening of at least 54 mm. Wrong-object failure requires task failure and more than 20 mm of additional upper-drawer opening. It excludes upper closing, so target opening and upper net motion are also reported. Claims that a failed trajectory never completed the task use its maximum target opening over the complete saved joint trace, not just a final image.

Cube evaluation uses two source scenes and both target/decoy position assignments, with 16 paired evaluation seeds per cell, totaling 64 per color and policy. Success requires the requested target to satisfy the stack geometry (6 mm position tolerance, 12 mm height tolerance and 20-degree tilt limit) and remain in contact with the support, without gripper contact, for the final three native control steps. The decoy is evaluated with the identical rule. Wrong-object failure means the target fails while the decoy succeeds; all remaining unsuccessful episodes are no-stack failures. The three categories partition every reported panel. Colors interpolate between a fixed distinct decoy and the target at fractions 0 and .9; both conditions receive balanced training feedback.

The critic camera is separate from the policy’s two observation cameras. Robometer uses the frozen original success-prediction output on the task instruction and eight rollout frames; its separate progress output is not substituted. LIV uses the official model’s final-frame image–language cosine c, with (1+c)/2 as the fixed nonnegative training reward for both original and product objectives. Native cosine is reported without fitting a success threshold. High-score failures, F_{c}=\Pr[y=0,r_{b}\geq c], and their wrong-object subset are secondary diagnostics with all executions in the denominator. Robometer uses c=.5 and normalized discrete RoboReward uses c=.75; LIV has no high-score threshold. Scores from different critics are not pooled. Separate RoboReward protocols and their score rubric are given in Appendix L.1.

Appendix B Primary Full-denoiser Evidence

B.1 Frozen contract and execution

The training contract was committed before launch: parent weights fixed by checksum, 22.4M-parameter encoder frozen, all 251,701,895 denoiser parameters trainable, K{=}5, \eta{=}0, \sigma{=}.05; 126 fresh episodes per iteration and 40 iterations; two value-only iterations; ten epochs per iteration at 1,512 decisions per optimizer step; clipping .2, critic learning rate 10^{-3}, value coefficient .5, gradient-norm limit 1, \gamma{=}.999, \lambda{=}.95; actor learning rate adaptive to Kullback–Leibler (KL) divergence from 10^{-6} with target .01 per denoising step. Oracle learnability requires at least two of three histories to improve success or completion with a confidence-interval lower bound above zero. The critic and product study uses a fixed bank of 512 seeds shared across objectives and iterations 0,10,20,30,40. Five training histories per objective share one evaluated initialization, giving 41 distinct model checkpoints and 20,992 model–seed executions. Model identities, the seed bank and the descriptive analysis are fixed before evaluating the intermediate checkpoints, and all are scored before analysis. The primary comparison is iteration 40 versus initialization; intermediate evaluations do not select checkpoints or hyperparameters. The same 20,992 executions supply the collected-trajectory semantic audit in Section 5.3. Training and evaluation seeds are functions of iteration, worker and slot, so a restarted unit reproduces its episodes exactly.

Each arm runs resident workers that load the model once and wait for the learner to publish the next weights; the learner consumes an iteration only when every worker has written its completion marker, writes an atomic checkpoint, then publishes the next weights with a checksum index. A worker that crashes is restarted alone and redoes its own unit under the same weights and seeds; only a learner failure or an exhausted worker restart budget fails the arm. The primary critic and product evaluation uses the same OSMesa renderer, evaluator and scorer for every model; the initial policy is scored once and shared by all ten comparisons. Oracle controls use their own seed sets and the rendering backend of their training. Two further equivalence checks are recorded with the results: disabling per-step camera rendering leaves trajectories, actions, latents and saved frames bit-identical, and the resident scorer reproduces the offline scorer used for all earlier experiments to 0.0 on shared episodes. Numerically, recomputing \log\pi_{\theta} under the sampling weights matches the recorded values to 1.07\!\times\!10^{-4} on the first minibatch with TF32 disabled; each arm reports this check.

B.1.1 Credit assignment and numerical details

The mean \mu_{\phi} in Equation 2 is the \eta=0 DDIM update with its clean-sample estimate clipped to [-1,1]. The policy executes the first four actions of each unnormalized output; 300 control steps give 75 observation-conditioned decisions. Sampler calibration is reported in Appendix J.2.

For the dense oracle histories, the completion increment at decision h is r_{h}^{O}=\Phi(d_{h+1})-\Phi(d_{h}), where \Phi(d)=\operatorname{clip}(d/d^{\star},0,1) and d^{\star}=54 mm. The undiscounted sum telescopes to terminal minus initial completion; the PPO objective uses discount \gamma=.999. The terminal oracle uses a single terminal completion reward.

The value network takes frozen encoder features and the decision index, including the policy’s proprioceptive condition but no privileged object state. Generalized advantage estimation (GAE) (Schulman et al., 2016) uses \gamma=.999 and \lambda=.95; the decision advantage is standardized within an iteration and copied to all K denoising transitions of that decision, which are not discounted relative to one another. Each iteration collects 9,450 decisions and runs ten epochs with six optimizer steps of 1,512 decisions. The PPO clip is .2, value coefficient .5, and gradient-norm limit 1. The implementation sums coordinate log-densities and uses a KL-adaptive actor rate bounded by [10^{-8},10^{-4}]. TF32 is disabled for both sampling and training.

Implemented rewards.

For a learned-reward arm the reward of episode e at decision k is r_{e,k}=\mathbb{1}[k=75]\,s_{e}, where s_{e} is the frozen scorer’s value for that episode: Robometer’s last-frame success probability for critic-only, its product with the verifier probability for the product arm, and the verifier probability alone for verifier-only. The terminal oracle uses r_{e,k}=\mathbb{1}[k=75]\,\mathrm{completion}_{e} and the dense oracle the per-decision completion increment. No score is rescaled before GAE. An executable test of the deployed learner (byte-identical to the code that ran) confirms that the terminal reward of every learned key equals the scorer file and is unchanged when the simulator completion changes. On the real training data of six learned trainings (critic and product trainings 3–5; 30,240 episodes), every iteration’s scorer file covers exactly the executed episodes, the product equals Robometer times verifier with zero error, and the reward means logged by the learner equal the scorer file’s means exactly.

B.2 All primary endpoints and uncertainty

Figure 6 places each critic and product training by its iteration-40 change in success and wrong-object failure; Figure 7 adds per-training intervals and Robometer changes. All five critic and all five product trainings enter the primary comparison. The initial policy is evaluated once on the 512 shared seeds. Per-training intervals resample paired evaluation seeds. Family estimates average the five per-training paired differences; their intervals resample training histories and one common evaluation-seed draw per replicate. The product–critic contrast draws training histories independently between objectives and retains the common evaluation-seed resample. All intervals use 20,000 bootstrap draws. Alternative resampling designs are reported separately in Appendix J.1.

Three critic trainings individually establish both reward gain and wrong-object amplification; the other two have positive wrong-object estimates whose intervals include zero. Every product training establishes a success gain. No individual product training establishes non-increasing wrong-object failure relative to initialization. These per-training comparisons and the pooled product–critic contrast measure different effects.

Outcome composition.

Over the 5\times 512 paired executions of the critic-only trainings, 302 other failures become successes and 189 become wrong-object failures, while 45 successes become wrong-object failures and 50 wrong-object failures become successes. For the product the corresponding counts are 422, 133, 19 and 83. Measured regardless of outcome, the share of executions that open the distractor by more than 20 mm rises by 6.6\,[3.3,10.2] points under critic-only training. The product change, +2.1\,[-1.2,+5.4] points, remains unresolved. Mean maximum distractor opening changes by +3.96\,[2.06,5.91] mm for critic and +1.27\,[-0.58,+3.14] mm for product. Changes in unconditional distractor opening under the four oracle controls also have intervals containing zero.

B.3 Training dynamics on the fixed evaluation bank

Figure 8 reports all five histories per objective at the five fixed checkpoints. Main-paper Figure 2 averages those histories and shows 95% pointwise percentile intervals from 20,000 crossed-bootstrap draws. Each draw resamples five histories independently within each objective and one common set of 512 seed columns. The same history and seed draws apply across checkpoints, preserving their dependence. The common initialization has only evaluation-seed uncertainty. These intervals describe each checkpoint level, rather than simultaneous coverage or the endpoint contrasts in Table 1.

B.4 Policy, scorer and evaluation provenance

Initial policy.

The shared initial weights are not a purely supervised policy. They descend from a diffusion policy trained by supervised learning on middle-drawer demonstrations, followed by three earlier updates in this project: (i) a five-epoch warm start on the supervised data mixed with copies of one optimized episode from an earlier critic study (5.3% of samples; that episode pushes the distractor drawer closed without opening the target, which is not the wrong-object pattern measured here); (ii) online RL against RoboReward-8B; and (iii) 600 steps of offline reward-weighted denoising regression toward Robometer on the evaluation scene, which in its own evaluation raised wrong-object failures from 1 to 7 of 32. The whole-network study therefore starts from a policy already exposed to the critic it optimizes. Every reported change is measured against this common initialization, and the constrained-selector experiments use the same decoder.

Scorer.

Robometer-4B at a fixed revision scores eight evenly spaced frames of a third-person camera with the task text “open the middle drawer”; the reward is the success probability at the last frame, and the base vision-language model is loaded offline at a fixed revision. The verifier is the frozen model of Section 7. Rescoring 32 stored episodes on a second installation reproduced the stored Robometer scores, all eight per-frame success probabilities and the verifier outputs exactly (maximum absolute difference 0).

All primary critic and product evaluations use the same OSMesa renderer, evaluator and scorer. Weights are fixed at iteration 40; no intermediate checkpoint is selected by its evaluation outcome. The training sampler is stochastic DDIM with five steps and standard deviation .05. The artifact records model identities, evaluation seeds, configurations and score files.

B.5 Collected-trajectory semantic audit

The reward audit of Section 5.3 rescores executions that were already produced and scored, so it adds no rollouts. The panel contains every distinct warm-start execution on the 512-seed bank: the shared initialization once and iterations 10, 20, 30 and 40 of the five critic and five Product trainings, 20,992 executions from 41 policies with 6,340 successes, 3,636 wrong-object failures and 11,016 other failures. Robometer’s mean score is .701 on wrong-object failures against .651 on successes and .137 on other failures. Intervals resample evaluation seeds as clusters (1,000 draws). Among the 16 highest-scored executions, 13 are wrong-object failures under Robometer and all are successes under the verifier and the product (Table 3). The inversion is already present on the initial policy’s own 512 executions (AUROC .365\,[.288,.447]) and persists on the final critic (.401), Product (.366) and verifier-only (.391) policies; the supervised initialization’s Robometer scores barely separate the two classes (.539\,[.449,.626]), the case discussed in Section 6.

Appendix C Learnability and Supervised-initialization Controls

C.1 Oracle learnability and verifier-only feedback

The dense oracle optimizes measured target-completion increments; the terminal oracle receives the completion once, at the end of the episode; verifier-only training optimizes the frozen verifier’s probability. All are trained from the warm-start policy with the full-denoiser protocol and evaluated at iteration 40 on the warm-start 512-seed bank with the renderer, evaluator and scorer of Table 1 (Table 4). Every oracle improves success by 12 to 19 points and none increases wrong-object failures, so the policy family learns the task under the implemented update, including from a single terminal reward. Pooled over the three dense oracles, success rises by 13.3\,[9.6,17.0] points and wrong-object failures change by -1.6\,[-4.4,1.4]; the critic produces 5.3\,[2.6,8.1] points more wrong-object failures than the dense oracles (exact permutation test over trainings, p=.018) and 4.7\,[1.7,7.8] more than the terminal oracle. Six verifier-only trainings, the original one and five additional runs, raise success by 15.8\,[12.2,19.4] points and do not increase wrong-object failures; Product and verifier-only training are indistinguishable on both outcomes. Sampler calibration and worker-reset details are in Appendix J.2.

C.2 Amplification from a capable supervised initialization

Policy and task.

We use a supervised diffusion policy further trained for ten epochs on 167 physically validated drawer demonstrations, with its own visual encoder and normalizer. No learned reward or reinforcement learning enters this policy’s training lineage. Before RL, a fixed capability screen evaluates this checkpoint on twelve drawer layouts with eight seeds each. Layouts with 2–6 successes are eligible; we select the most successful, breaking ties by manifest order. The selected layout has 3/8 successes. It differs from the warm-start study’s layout and from the ceiling-limited control in Appendix L.4.

Training and evaluation.

Five independent trainings per objective optimize either Robometer or terminal target completion, with the same supervised initialization, full-denoiser PPO update, 40 iterations and 126 feedback episodes per iteration. Completion is the final target opening divided by 54 mm and clipped to [0,1]; a wrong-object failure requires an unsuccessful task and more than 20 mm additional distractor opening. The endpoint is fixed at iteration 40, and all ten final policies and the common initialization are evaluated on the same 512 held-out seeds. An independent repetition of three trainings per objective, run with fresh training seeds on a second host, is evaluated on the same seeds (Table 5).

All five Robometer trainings individually amplify wrong-object failures, by 9.2 to 13.5 points, while the five completion trainings change them by -0.4 to +4.1 points. Pooled, Robometer minus completion gives +9.2\,[5.6,13.0] points of wrong-object failure and -3.7\,[-7.5,0.1] points of success. Across all eight trainings per objective, the wrong-object difference is +8.3\,[5.0,11.5] points; exact permutation tests over training labels give p=.008 for five against five and p=.0002 for eight against eight. Pooled intervals use 20,000 crossed bootstrap draws over trainings and one shared set of evaluation seeds; cross-objective contrasts resample the two objectives’ trainings independently and share the seed resample.

Appendix D Constrained-policy Recurrence and Matched Verification

Figure 9 contains the repeated drawer Robometer and LIV stacking comparisons, including both trained stacking colors. The tables below report all nine repeated-error histories across these settings. The complete 29-stratum grid, all supervision budgets and boundary settings appear in Appendix K. Training numbers are local to each task–critic–optimizer group. Original run IDs are retained in Table 28.

D.1 Oracle, Exploitation and Repair

Tables 6 and 7 report original critic scores, physical task successes and paired changes for all nine histories. All nine oracle policies improve true success with intervals above zero. For LIV stacking, their native-score changes are negative in all four trainings, also with intervals excluding zero. The oracle learns the physical task despite this critic disagreement. Conversely, original-LIV optimization raises native score while losing success in all four.

Drawer PPO 2 and 3 and GRPO 1 also meet the strict reward-up, success-down, repaired-success-up criterion. Drawer PPO 1 and GRPO 2 have success-loss intervals crossing zero, but reliably increase wrong-object failures while increasing reward. Drawer PPO 1 additionally reduces mean clipped target completion from .2576 to .1082, a paired change of -.1494\,[-.2917,-.0137]. The oracle and product reach .9146 and .8300.

Continuation from the three exploited drawer policies, with product, original-critic, verifier-only and oracle rewards and a drawer opening held out from continuation, is reported in Appendix M.5.

D.2 Verifier-only Ablation

The ablation removes the original critic and optimizes R_{V}=p_{\theta} with the identical verifier, labels, policy capacity and update protocol. Supplemental same-head controls retain the original seeds and repeat uniform executions to check reference reuse. These paired ablations reuse the original training histories. Table 17 retains every available matched control and marks missing controls explicitly.

Source of the hacking repairs.

For drawer Robometer GRPO, verifier-only reaches 55 and 53/64 successes with zero wrong-object errors, compared with product outcomes 56/0 and 52/0. Product-minus-only success differences are +1[0,3]/64 and -1[-3,0]/64. For distinct-color LIV stacking, expanded-only, the verifier-only control with the 2,536-label head, reaches 63/57 successes under PPO and 50/55 under GRPO; the product reaches 63/57 and 52/52. Neither optimizer establishes an additional physical benefit from retaining LIV. In the second PPO training, product and only have identical outcomes on every paired seed. These controls attribute much of the repair to outcome verification.

Fixed support and replication.

The repeated trainings vary training and evaluation seeds on fixed 24-component development supports; they do not resample support banks. The drawer support was selected during development without an outcome-blind bank-selection rule. Both stacking colors participate in training. Intervals condition on the fitted policies and selected support. Appendix A.1 specifies the label costs and shared heads; Appendix K.1 gives the exact selector representation, objectives and optimizer settings.

Temporal verification in ball placement.

The controlled ball-placement extension compares full-history and final-quarter verifiers in a matched PPO ablation at equal label and frame budgets. In two independent histories, the product with the full-history verifier reaches 63 and 64/64 successes on new seeds, against 12/64 for the uniform selector and 2 and 5/64 for the original reward; the same product with a final-quarter verifier reaches 17 and 2/64. Its protocol and new-seed endpoints are in Appendix M.4 and Table 26.

Appendix E Sim-to-Real Case Study: Protocol and Additional Results

The main hardware outcomes are reported in Section 8. Here, we provide the physical setup, deployed policy identities, and outcome definitions for the real-robot drawer evaluation. Fig. 10 shows the environment setup for deployment on a Franka robot.

Physical setup and reset procedure.

Experiments use a Franka Research 3 robot in a physical drawer setup matched to the simulation at 1:1 scale, including cabinet placement, drawer geometry, initial drawer openings, and camera placement. The requested task is to open the middle drawer. Before each trial, the target middle drawer is reset to its closed position, and the upper distractor drawer is returned to the same marked partial opening. All policies are evaluated under the same physical setup and deployment configuration.

Policy identities and checkpoint mapping.

The hardware comparison includes three conditions: Initialization, Critic-trained and Completion-trained. Initialization is the shared warm-start initialization of the full-denoiser study (Appendix B.4), which has prior reward exposure. Critic-trained is Critic 5 at iteration 40, chosen after the simulation evaluations without a prespecified quantitative selection rule; the checkpoint was fixed before any physical trials. No hardware outcomes were used in the choice. Among the five critic trainings, Critic 5 has the second-largest wrong-object increase on the 128-seed checkpoint-diagnostic panel of Section 5.3 (per-training values in Appendix J) and the smallest success gain on the 512-seed primary panel (Figure 7). Completion-trained is the terminal-completion oracle of Appendix C.1 at iteration 40, trained from the same initialization. All policies are deployed with the five-step DDIM sampler used during fine-tuning. Because only one training history per fine-tuned condition was deployed, the hardware study is a sim-to-real case study rather than an estimate of transfer across training histories.

Deployment and evaluation.

Each policy executes 40 trials, giving 120 trials across the three conditions. Policy parameters remain fixed throughout hardware evaluation, with no training or fine-tuning on the physical robot. The evaluation therefore examines the physical execution of policies trained in simulation. Task outcomes are assessed from physical drawer motion independently of the learned visual reward.

Outcome definitions.

Each trial is assigned to one of three mutually exclusive categories. A success requires opening the target middle drawer by at least 6 cm. A wrong-object failure occurs when the upper distractor drawer is pulled more than 2 cm beyond its initial marked opening while the requested task remains unsuccessful. All remaining unsuccessful trials are classified as other failures. Grasping a drawer handle alone does not satisfy the success criterion.

Table 8 gives the outcome rates and their changes from Initialization for the hardware comparison in Section 8.

Appendix F Scope and Retrospective Analyses

F.1 Prevalence across the constrained grid

A constrained-grid stratum shows wrong-object amplification if and only if the paired 95% interval for the critic-minus-uniform change in wrong-object failure has a lower bound above zero. Under this definition, 11 of the 29 evaluated strata (37.9%) show amplification (Table 9). The 29 strata come from 20 training histories, because the two stacking colors of one training share its history; they are not 29 independent trainings. In all 11 amplified strata, the critic-minus-uniform change in original reward also has an interval lower bound above zero. One drawer LIV stratum has a wrong-object interval whose lower bound is exactly zero and is not counted. In the two amplified near-color Robometer stacking strata, amplification occurs together with a positive success change. The nine repeated-error histories of Section 5.2 come from the two settings with amplification in every stratum and do not estimate prevalence. Table 10 lists every stratum with its intervals; Appendix K adds per-policy outcome counts and all supervision budgets, and separate RoboReward experiments outside the grid are in Appendix L.

F.2 Retrospective association with initial-policy misranking

For each of the 29 strata, two AUROCs are computed from initial-policy trajectories alone, in the notation of Equation 3: the aggregate success–failure AUROC A_{S,F} and the semantic success–wrong-object AUROC A_{S,W}. Each is compared with the later critic-minus-uniform change in wrong-object failure, \Delta W. The Spearman correlation of A_{S,W} with \Delta W is -.82\,[-.91,-.68], against -.34\,[-.63,+.05] for A_{S,F}; their difference is -.48\,[-.80,-.20]. Intervals resample the 20 training histories. Computing each AUROC and \Delta W on disjoint halves of the initial-policy panel gives -.74 for A_{S,W} and -.31 for A_{S,F}. The association is carried by differences between task–critic families and between decoy colors: it is absent within both drawer families, and four near-color LIV strata have A_{S,W}<.5 without amplification. It is therefore a retrospective description of these settings, not a validated prospective predictor. Metric definitions, the disjoint-halves procedure and the complete correlations are in Appendix K.5 and Table 18.

Appendix G Robustness of the Full-denoiser Results

G.1 Native diffusion sampler

Fine-tuning and the primary evaluation decode each action chunk with the five-step stochastic DDIM chain of Equation 2. The policy checkpoint also defines a native sampler: its 100-step DDPM scheduler with the clipped clean-sample estimate, with a standard-deviation floor of .05 applied at the final step only. We re-evaluate the shared warm-start initialization and the five iteration-40 critic policies with the native sampler on the same 512 seeds, OSMesa renderer and scorer. Episodes are decoded in lockstep batches; each episode draws its initial latent and every step’s noise from its own generator seeded by the evaluation seed, in the order of the sequential native evaluator, which the batched implementation reproduces exactly on a decoded chain and outcome for outcome on 16 reference seeds. The initialization itself performs differently under the two samplers, so each sampler’s policies are compared with that sampler’s evaluation of the initialization (Table 11).

G.2 Distance from the initial policy

The full-denoiser learner adapts its actor learning rate to a fixed KL target per update, so all objectives move at a similar rate. We measure how far each final policy moved: each decision’s five-step chain is a product of Gaussian transitions with the same standard deviation under both policies, so its KL divergence is \sum_{k}\lVert\mu_{\phi}(x_{k})-\mu_{0}(x_{k})\rVert^{2}/(2\sigma^{2}) with x_{k} drawn from the evaluated policy. The evaluation episodes store x_{k}, \mu_{\phi}(x_{k}) and the observation condition of every decision, so only the initial network’s means are recomputed; on the initial policy’s own episodes the recomputed and stored means agree to 3\times 10^{-6}. Product and verifier-only policies end slightly further from the initial policy than the critic policies, and the supervised Robometer and completion policies end at nearly the same distance (Table 12), so the outcome differences are not explained by how far each policy moved.

G.3 Distractor-opening threshold

A wrong-object failure requires the requested task to fail and the distractor to open by more than 20 mm. Table 13 repeats the primary comparisons for thresholds from 5 to 50 mm.

Appendix H Tilt Model Details

Full denoiser.

For each study and optimized reward we reweight the initial policy’s 512 evaluation episodes by q_{i}\propto\exp(\lambda r_{i}) and choose \lambda by bisection so that \sum_{i}q_{i}r_{i} exceeds the initial mean by the reward gain the family reached at iteration 40 (Robometer for critic runs, completion for completion runs, the product and the verifier probability for Product and verifier-only runs). Predicted changes are the reweighted minus initial outcome rates. Intervals resample the 512 initial episodes and re-solve \lambda in each of 2,000 draws (Table 14). Reweighting can only move mass among behaviors the initial policy already produced, which is why it is a first-order description of full-denoiser fine-tuning.

Constrained selectors.

The drawer selector is unconditional and starts every episode from the same scene, so its KL-regularized optimum from the uniform selector is q_{k}\propto\exp(\lambda\bar{r}_{k}) over the 24 sequences. For each stratum we estimate \bar{r}_{k}, the success rate and the wrong-object rate of every sequence from all executions of that stratum’s run outside the critic arm: the uniform draws of the first round, the oracle, Product and verifier-only arms in training, and their fixed evaluation; every stratum covers all 24 sequences. The critic arm’s measured reward gain on the fixed evaluation sets \lambda. The stacking selector conditions on the initial image, so pooling its sequences over contexts makes the same computation an approximation; in three distinct-color Robometer stacking strata the conditional selector’s reward gain exceeds every sequence’s mean reward and no unconditional tilt reaches it, leaving 26 strata. The first-order predictor is the covariance over sequences between an outcome rate and the standardized \bar{r}_{k}, which needs no tilt strength. The raw evaluation records reproduce the frozen stratum changes exactly.

Appendix I Robometer’s Published Online-RL Recipe

Robometer’s automatic online RL pays -1 per step plus the model’s reward prediction in [0,1], pays 0 instead of -1 on the step where the success probability exceeds .6, and terminates there (Liang et al., 2026). The drawer selector makes one choice per episode, so the recipe enters it only through the episode return. Every executed drawer episode of the five Robometer selector trainings stores Robometer’s success probability and progress prediction at eight frames (control steps 0, 43, …, 299), from one forward pass over the clip; we hold progress at the last frame’s value between frames and evaluate physical outcomes on the full episode. Table 15 reports how often each outcome triggers termination, how each return orders the outcomes, and the tilted selector at the KL from uniform that the trained terminal-reward selectors reached (2.07 nats on average). Thresholds of .5 and .7 give the same ordering (AUROC .152 and .120 for success against wrong-object failures).

Appendix J Evaluator Robustness and Statistical Sensitivity

These analyses check the statistical design of the primary endpoint and the fine-tuning sampler’s calibration.

J.1 Bootstrap design for the primary endpoint

The main text uses crossed training-by-evaluation-seed intervals and draws trainings independently across objectives for product–critic contrasts. The following alternatives retain every training and the same 512 seeds.

Nested resampling draws evaluation seeds independently within each sampled training and understates their shared variation. Index-paired crossed resampling treats training indices as matched; they do not identify matched training randomness. These two rows are sensitivity analyses; the main contrast uses independent training resampling. Pooled completion rises by .087\,[.057,.118] under critic-only training and .140\,[.106,.175] under the product.

J.2 Sampler calibration and worker-reset diagnostic

Before any training we evaluated the unmodified network under the short stochastic sampler on 64 paired seeds per cell.

The two \sigma{=}.10 cells are not interpretable as written. The batched worker created its environments and ran its first batch before the simulator performed a hard reset, so those episodes started from the construction state rather than the initial state. Scheduling placed 64/64 of the K{=}5,\sigma{=}.10 episodes and 48/64 of the K{=}10,\sigma{=}.10 episodes in such first batches, while both \sigma{=}.05 cells contain none. The 16 unaffected \sigma{=}.10 episodes give 0 successes and completion .110, weaker than \sigma{=}.05 but too few to establish the point. We therefore report \sigma{=}.10 as untested rather than as a negative result. The fix is a discarded warmup batch per worker. Oracle history A had already used first-batch data in one actor update and was restarted from iteration 0. In histories B and C, first-batch data entered only the first iteration, which trains the value network alone; their actors never used such data and they were not restarted, but that first value-network update did see it. Every endpoint evaluation and every learned-reward training ran with the corrected workers. We did not sweep \sigma<.05; that is an unswept hyperparameter and we record it as such.

Actor learning-rate calibration.

An initial actor learning-rate test at 5\times 10^{-5} diverged, with mean sampled KL 23 per denoising step after three optimizer steps. The implemented protocol sums coordinate log-densities and uses a KL-adaptive actor rate bounded by [10^{-8},10^{-4}].

Appendix K Complete Constrained-policy Grid and Trajectory Diagnostics

This section retains all 20 main training histories, 29 task/color strata, 43 supervision-budget rows and five matched-only controls. It includes settings with and without wrong-object amplification. The repeated-error subset in the main text is not an estimate of prevalence across this grid or across robot tasks generally.

K.1 Optimization and Independent Evaluation

The common policy family and PPO/GRPO updates are specified in the constrained-policy protocol of Section 4.2. The frozen decoder uses 100-step denoising diffusion probabilistic model (DDPM) sampling (Ho et al., 2020). All 24 complete stochastic paths are retained, including transition noise, and the decoder conditions on new observations throughout execution. Fresh Gaussian jitter of standard deviation .05 is applied to each noise draw. The trainable policy chooses the noise component: 24 unconditional drawer logits, or a 24\times 129 image-conditioned cube matrix. The diffusion decoder is frozen. Each rollout generates actions from fresh visual observations; scene IDs and object-state labels are excluded from the selector input.

Noise-bank construction and reuse.

The drawer support contains all 24 seeds 52103000–52103023, the first of three development banks with consecutive 24-seed blocks starting at 52103000, 53103000 and 54103000. It is reused in all eleven drawer panels. The nine stacking panels share the 24 seeds 120100000–120100023. No member is removed within either selected bank. These are fixed development supports: the drawer banks had been evaluated before the first recorded downstream configuration, and no outcome-blind bank-selection rule was recorded. Repetitions vary training and evaluation seeds on these supports; they do not sample new supports.

Each objective receives eight rounds of 24 PPO or 32 GRPO feedback episodes, with eight Adam epochs per round and persistent optimizer state. Initial uniform feedback is shared across identical policies; later feedback is collected separately. Thus, unique trajectory counts depend on the number of objectives rather than just the per-objective budget. GRPO groups contain four candidates with identical initial physical state and images but distinct rollout noise. Drawer R114’s eight groups are draws at the same physical state, not eight different task contexts. Cube groups balance the trained color conditions.

Both optimizers use clipping .2 and gradient-norm limit 1. The learning rate is .2 for drawer logits and .02 for conditional cube policies, and the PPO entropy coefficient is .02; grouped GRPO replaces that term with exact categorical KL to the fixed initial uniform policy at coefficient .04. Cube feedback is balanced across the two trained colors at the same total budget, and the shared initial round avoids duplicate executions of identical initial policies.

Every main endpoint uses the frozen eighth checkpoint, without selecting checkpoints by final success. Drawer Robometer PPO uses 32 new paired executions; other common panels use 64 per condition. Original drawer Robometer analyses use 10,000 paired execution resamples, cube uses 20,000 paired resamples within each of four cells, and drawer LIV uses 20,000 paired bank-cluster resamples preserving four members of each of 16 banks. All intervals are percentile 95% intervals conditional on one training and the fixed support. Neither multiple colors, label budgets, nor matched-only extensions increase the number of independent trainings. PPO and GRPO have different feedback budgets and are not ranked for sample efficiency.

Recorded sampling probabilities, final parameter arrays and continuous Adam state are verified against the saved updates. Execution manifests retain seeds, model identities, renderer assignments and original scores; physical outcomes are independently reconstructed from state/contact traces. Paired references share a renderer within each panel. Changes between EGL and OSMesa across replications are recorded, not assumed pixel-identical or isolated from training-seed changes. The summaries below retain complete endpoints rather than partial-progress snapshots.

K.2 Selector Representation and Exact Update Objective

For the image-conditioned stacking selector, a fixed covariance transform of 128 visual coordinates followed by a bias gives g(o_{0})\in\mathbb{R}^{129}, with p_{\phi}(I\mid o_{0})=\operatorname{softmax}(Wg(o_{0}))_{I}. The transform is fitted on 864 unlabeled development inputs, with relative eigenvalue floor .001 and no RMS-preserving gain. Drawer policies instead use 24 unconditional logits.

The categorical PPO loss is

\mathcal{L}_{\rm PPO}=-\frac{1}{B}\sum_{i}\min\!\left(\rho_{i}\widehat{A}_{i},\operatorname{clip}(\rho_{i},1-\epsilon,1+\epsilon)\widehat{A}_{i}\right)-\beta\mathbb{E}[\mathcal{H}(p_{\phi})].

Advantages center terminal reward within a batch. Stacking additionally standardizes advantages; drawer PPO retains original-scale and explicitly normalized variants separately. Grouped GRPO standardizes within groups of four and replaces entropy with an exact categorical KL to the fixed initial uniform policy, coefficient .04. Conditional on the chosen component, the noise jitter, frozen decoder and environment do not depend on trainable selector parameters, so \rho_{i}=p_{\phi}(I_{i}\mid o_{0,i})/p_{\rm old}(I_{i}\mid o_{0,i}) is the exact trajectory likelihood ratio for this policy family.

K.3 Observed Dynamics and Residual Errors

The first sorted R82 evaluation seed supplies the physical trajectory example; it was selected independently of outcome. Images in the main text are the original critic frames. High-score wrong-object failures increase from 7 to 24 of 32 executions. All 24 rewarded wrong-object failures in this panel keep the target below 6.141 mm throughout the episode while additionally opening the distractor by 50.121–63.124 mm. After mitigation, four failures still receive high original reward and five satisfy the broader wrong-object definition. The four rewarded residuals keep the target below 29.722 mm; three sample component 14 and one component 6. Their verifier probabilities are .0671–.0932 and product rewards .05685–.08041. Detection does not eliminate every stochastic failure, and this panel does not isolate exploration from execution variability.

K.4 All Budgets, Conditions and Matched-only Controls

Table 17 contains every one of the 43 budget rows from 20 complete main trainings and 29 task/color strata. The missing drawer PPO only controls and missing 456-only controls in the repeated LIV cube trainings remain missing. For R164/R166/R167/R165/R184, supplemental R168/R170/R171/R172/R189 supply identical-456-head only policies with matched feedback budgets. They reuse the reference seeds and repeat uniform executions; they are not independent confirmations. R189 was selected after observing the complete R184 result. Its 128 uniform reexecutions match that reference, but its distinct-color product advantage does not retrospectively make the first Robometer GRPO run successful.