What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization

Abstract

Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents. Across three Procgen environments and five robot tasks, we find three distinct forms of policy–goal misalignment: (1) functional target substitution, where behavior follows an alternative goal, (2) a policy shortcut, where behavior follows a learned action strategy, and (3) cue competition between intended and proxy goal cues. We distinguish these regimes by tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, and find that the goal is lost at different depths. Where the goal reaches the decision, as in cue competition, steering redirects the policy to the goal’s location, raising intended-goal reach from 16.0\% to 95.3\%. Where it reaches the policy but not the decision, only new downstream wiring recovers goal reach, from 31.6\% to 84.3\% in the environment with the action shortcut. Where it is inert, neither works. Our results show that goal misgeneralization is not a single failure of goal representation, but distinct failures of policy alignment at different depths, tightly connected to which recovery works.

1 Introduction

Reinforcement learning specifies the intended goal only indirectly, through rewards that mark it. Whenever a proxy co-occurs with the goal during training, the rewards cannot tell the two apart, and the agent can learn the proxy in its place. This is why goal misgeneralization is common in reinforcement learning, and it is dangerous because the agent stays competent and its misalignment only becomes visible once the co-occurrence breaks.

So far, goal misgeneralization has mainly been characterized behaviorally, in large language models (Shah et al., 2022; Maniyar et al., 2026), in toy reinforcement-learning environments such as Procgen (Langosco et al., 2022; Ramanauskas & Şimşek, 2023) and in large pretrained control policies (Jucys et al., 2024; Zhou et al., 2025). Interpretability work has studied individual agents in these environments (Mini et al., 2023; Trinh et al., 2024), mostly to understand how they navigate or represent their goal, but not goal misgeneralization itself, and mitigation attempts intervene broadly during training (Langosco et al., 2022; Abdel Sadek et al., 2025; Nait El Barj & Sautory, 2024; Armstrong et al., 2023; Trinh et al., 2024) rather than after it. Studying goal misgeneralization in depth, from the inside and the outside, closes the gap between these lines of work, and we additionally carry it into a new and wider domain, robot control. The leading question we ask is therefore quite direct: what actually happens to the true goal in misgeneralized agents?

To answer it, we trace the goal through the agent from the outside and the inside, asking whether the goal is represented, whether the policy reads it, whether behavior is organized around it, and whether goal-directed behavior can be recovered at test time. We analyze three classic goal-misgeneralization environments from Procgen, Maze, Coin and Maze II, which we call Attribute Maze, and five new robot tasks on a simulated arm and quadruped, such as PushToMarker (Figure 1, Figure 13).

What looks like one failure from the outside is three distinct failures of policy alignment, target substitution, a policy shortcut and cue competition, and the goal gets lost at different depths of the agent. We find that in Maze, replicated in a simulated arm task, the goal is inert to the policy and a place substitutes for it. In PushToMarker a place substitutes for the goal as well, but the goal still reaches the policy weakly, which reacts to it without following it. This is the same depth at which an action shortcut bypasses the goal in Coin. In Attribute Maze, replicated in three more robot tasks, the goal reaches the decision as a cue or an object that competes with the proxy and loses. We find that which repair works follows the depth, not the failure type. Steering the existing policy redirects it to the goal’s location only where the goal reaches the decision. Relearning the readout recovers goal-directed behavior wherever the goal still reaches the policy, and where the goal is inert, neither repair works. Misgeneralized agents therefore cannot be understood from their behavior alone, as only knowing where the goal is lost tells us which repair can bring it back.

Our main contributions are:

1. We trace the intended goal through misgeneralized agents in eight environments and find three distinct failures of policy alignment at three depths, including the first internal characterization of cue and object competition and its geometry (Section 3).
2. We add an internal view and deepen the behavioral one to better understand goal misgeneralization. We propose the Fisher information of the policy with respect to its internal representations to isolate a policy-directed view of the internals, which lets us characterize, for example, different styles of competition. It also sharpens a proxy characterization of Langosco et al. (2022), as the Maze agent follows the top-right extreme of the current maze rather than a fixed corner.
3. To our knowledge, we are the first to relate which repair succeeds to where the goal is lost, and to recover goal-directed behavior at test time without further reinforcement learning or retraining the representation (Section 4).
4. We are the first to induce goal misgeneralization systematically in reinforcement learning for robot control and diagnose it in five simulated arm and quadruped tasks, where the failures recur.

2 Setup

Problem formulation.

We model each environment as an episodic Markov decision process \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\rho_{0}). The agent observes a rendered image of the state, which we write as s, samples actions a_{t}\sim\pi(\cdot\mid s_{t}), and the reward r marks an intended goal g^{\star}. Goal misgeneralization arises when, on a training distribution \Lambda_{\mathrm{train}}, reaching the goal coincides with reaching a proxy, so that the agent learns to follow the proxy instead of the goal on a shifted evaluation distribution \Lambda_{\mathrm{OOD}}, where the two no longer coincide (Langosco et al., 2022; Shah et al., 2022). We write \Gamma^{\star}(\tau)=1 when a trajectory \tau reaches the goal and \Gamma^{\mathrm{prox}}(\tau)=1 when it reaches the proxy. Our main behavioral metric is the goal reach \hat{J}^{\star}=\tfrac{1}{N}\sum_{i=1}^{N}\Gamma^{\star}(\tau_{i}), the fraction of N rollouts that reach the goal. Formal setup and definitions are in Appendices B and F.1.

Environments.

We study eight instances of goal misgeneralization, present four of them here and report the replications in Appendix C. Three are procedurally generated video games (Procgen, Cobbe et al. 2020, Appendix A). In Maze, the cheese sits in the top-right corner during training, and at test time the agent walks to the corner instead of the cheese placed elsewhere (Langosco et al., 2022). In Coin, a side-scrolling game, the coin sits at the right end of the level during training, and at test time the agent runs to the right end and ignores a coin placed elsewhere (Cobbe et al., 2019; Langosco et al., 2022). In Attribute Maze, a 5\times 5 variant of Maze II (Langosco et al., 2022; Ramanauskas & Şimşek, 2023), the rewarded object is a yellow line during training, so colour and shape are two cues for the same object, and at test time, when the cues are split across objects, the agent follows the yellow object instead of the line when both are equally far away. The fourth is a robot-arm task, PushToMarker, in which a simulated Franka arm pushes a cube onto a marker (Tao et al., 2025). During training, the marker sits at the centre of the table, and at test time the arm pushes the cube to the centre and ignores the marker placed elsewhere. Four further robot tasks, three on the same arm and one on a quadruped, replicate these failures (Appendices A.2 and A.3). In each environment, we compare three misgeneralized (GM) agents with three control (Non-GM) agents trained on diversified distributions (Appendix F).

Architecture.

All Procgen agents are non-recurrent actor–critic networks with an IMPALA architecture (Espeholt et al., 2018) and all robot agents use a Nature CNN (Mnih et al., 2015), both trained online with PPO (Schulman et al., 2017) (training and implementation details in Appendix H, architectures in Appendix B.1). In both networks, convolutional layers produce feature maps H^{(k)}, whose average-pooled vectors h^{(k)} we use for probing, and a fully connected layer maps the flattened last feature map h_{\mathrm{pre}} to h_{\mathrm{fc}}. Most of our analyses read h_{\mathrm{pre}}, the flattened last feature map and the last representation that keeps the spatial layout of the scene, and h_{\mathrm{fc}}, which the policy and value heads read directly (Appendix B.1). We write \tilde{\pi}(\cdot\mid h) for the policy that the downstream layers compute from a representation h, so that \tilde{\pi}(\cdot\mid h(s))=\pi(\cdot\mid s).

3 What happens to the intended goal?

3.1 Is the goal still there?

If the true goal gets replaced by a different goal, the simplest question we can ask is whether the true goal is still in the misgeneralized agent. We test this using linear decodability (Alain & Bengio, 2017), by fitting linear probes at several points in the agent (Appendix B.1) on configuration sets that separate constructed goal concepts, scored by AUC for binary and R^{2} for continuous concepts. For example, a bearing probe tests whether the agent encodes where the goal lies relative to itself. Probe construction, controls (Hewitt & Liang, 2019), and further probes are detailed in Appendix C.1.

For Maze, the cheese object is trivially decodable from the input already, so we probe cheese bearing. This needs a set on which the agent’s own position is decorrelated from the cheese bearing, since otherwise a bearing probe reads the GM agent’s position as a proxy (Appendix C.1). On this set the cheese bearing stays near chance at every site in GM agents (0.516 to 0.583) and below randomly initialized networks at h_{\mathrm{pre}} (0.748), while control agents decode it well (0.932) (Figure 2a). Similarly, for PushToMarker the marker position is decodable at the level of a randomly initialized network in early layers and falls below it in the later layers (0.110 against 0.406 at h_{\mathrm{fc}}), while control agents keep it (0.819) (Figure 2b). In both environments, therefore, the bearing of the cheese and the position of the marker cannot be linearly decoded beyond what a random network exposes. For Coin, the coin object is also largely decodable from the input, so we test coin bearing with the agent’s position and level progress regressed out. This bearing stays decodable in both cohorts from h^{(2)} to h_{\mathrm{pre}} (R^{2} of 0.378 for GM and 0.306 for Non-GM at h_{\mathrm{pre}}), while randomly initialized networks and the agent’s position alone decode none of it, so we find linear traces of the coin inside the agent (Figure 2c). Lastly, for Attribute Maze, the goal cue, whether the reward-carrying object is the line, stays decodable in both cohorts alongside the proxy cue, most clearly in early layers (AUC 0.770 for GM and 0.701 for Non-GM at h^{(2)}, against 0.577 for random networks), and comes close to random networks at h_{\mathrm{pre}} (0.632 and 0.636 against 0.603). So in Maze and PushToMarker the GM agents carry no linearly decodable goal information beyond what a randomly initialized network exposes, while in Coin and Attribute Maze the goal stays decodable in both cohorts. The results replicate in the other robot tasks (Appendix C.1). Our probes show what is linearly available, but they cannot rule out goal information in another form. Since information that is present is not necessarily used (Elazar et al., 2021), we next ask whether the policy reacts to changes of the goal, using world edits.

3.2 Does the policy read out the intended goal?

We answer this question by intervening on the goal in the environment, from two perspectives: from the outside, we measure the change in behavior, and from the inside, whether the policy distribution moves along the corresponding representation displacement. A world edit e changes the goal in the level while keeping everything else fixed, and turns a state s into a matched state s_{e}, with \Delta_{e}f(s)=f(s_{e})-f(s) for any quantity f, and e=\star or e=p for an edit of the goal or of the proxy (Figure 18). In Maze, Coin and PushToMarker, we relocate the goal object, the cheese, the coin or the marker, as in earlier interventions on the Maze agent (Mini et al., 2023). In Attribute Maze, in contrast, goal and proxy cue sit on the same object, so instead of moving the object, we keep all locations fixed and swap one characteristic between the two candidates: the shape swap changes the goal cue (e=\star) and the colour swap the proxy cue (e=p). All edits are then evaluated on banks of conflict states, in which the agent has to choose between the proxy and the intended goal (Appendices C.2 and E).

The immediate behavioral response is the change in the policy, which we measure with the KL divergence, D_{\mathrm{KL}}\big(\pi(\cdot\mid s)\,\|\,\pi(\cdot\mid s_{e})\big). The KL says whether the policy reacts but not in which direction, so we measure with signed follow whether the action mass moves toward the edited object,

\mathrm{SF}_{e}(s)=\pi\big(A_{e}(s_{e})\mid s_{e}\big)-\pi\big(A_{e}(s_{e})\mid s\big),\qquad\pi(A\mid s)=\sum_{a\in A}\pi(a\mid s),

where A_{e}(s_{e}) holds the actions on the shortest path to the edited object at its new place in Maze and Attribute Maze and the coin-directed, leftward actions in Coin, while for the continuous robot policies signed follow measures how far the planned step turns toward the object’s new place (Appendix C.2). Inside the agent, the same edit moves the representation read by the policy by \Delta_{e}(s)=\Delta_{e}h(s)=h(s_{e})-h(s). We measure how strongly the policy distribution moves along this displacement with the Fisher information of the downstream policy at h=h(s),

F_{h}=\mathbb{E}_{a\sim\tilde{\pi}(\cdot\mid h)}\Big[\nabla_{h}\log\tilde{\pi}(a\mid h)\,\big(\nabla_{h}\log\tilde{\pi}(a\mid h)\big)^{\top}\Big],\qquad Q_{e}(s)=\Delta_{e}(s)^{\top}F_{h}\,\Delta_{e}(s).

We use the Fisher information because it is the metric that the policy itself induces on its representation. It measures a displacement by how much it changes the action distribution rather than by its Euclidean length, and we use this Fisher lens throughout the paper. Up to second order, this Fisher sensitivity is exactly the KL divergence, measured from the inside (Amari, 1998; Martens, 2020),

D_{\mathrm{KL}}\big(\tilde{\pi}(\cdot\mid h)\,\big\|\,\tilde{\pi}(\cdot\mid h+t\Delta_{e}(s))\big)=\tfrac{t^{2}}{2}\,Q_{e}(s)+o(t^{2}),

and we check this for finite edits in Appendix C.3. KL, Q_{e} and signed follow have different scales across environments, so we report each as the ratio of the GM to the Non-GM cohort mean.

We find that the outside and inside perspectives agree, through the KL and Q ratios, and separate the environments by depth (Table 1). In Maze (replicated in one robot task, Appendix C.2), the GM policy barely reacts to the cheese edit (KL ratio 0.009), so changes of the goal are read out by the policy neither behaviorally nor inside its representation. In PushToMarker, Coin and under the Attribute Maze shape swap, the goal edit moves the GM policy by similar fractions of the Non-GM reaction (0.206, 0.114 and 0.200). If we then ask whether this shift of action mass is directed at the goal, signed follow separates these three as well. In PushToMarker and Coin the policy reacts to the goal edit, but the shift is not directed at the goal (signed-follow ratios 0.017 and 0.001), so the policy is receptive to a change of the goal object but does not organize its longer-term behavior around it. In Attribute Maze, however, the shape swap still turns the policy toward the goal at about half the Non-GM turn (0.533). For the proxy colour swap, the GM policy reacts about fifty times more than the Non-GM policy, which barely follows the colour (KL 1.258 against 0.025, signed follow +0.463 against +0.047). So the goal does not reach the policy in Maze, reaches it without directing it in PushToMarker and Coin, and enters the decision but loses to the proxy in Attribute Maze. The other four robot tasks fall into the same three groups (Appendix C.2). Sensitivity only captures the immediate action distribution, so we ask next what organizes behavior.

3.3 What organizes behavior instead?

Where the goal is fully inert, as in Maze, the question is what actually replaces it: a directional shortcut, a memorized location, or a spatial concept. We first select detour states, on which every action that makes progress toward the top-right corner points neither up nor right. GM agents still choose these detour actions on 0.948 of the states, which rules out a directional shortcut (Figure 3a). We then open a path to a new top-right extreme, inspired by the corner edit of Mini et al. (2023), and ask which corner the agent follows. It reaches the new corner first in 0.748 of the rollouts and the old one never, and a new extreme placed away from the screen corner is still reached first in 0.792 of rollouts, against 0.114 for the old one, which rules out a memorized location (Figure 3a). Combined with the inert goal, this shows a functional target substitution, and we refer to Appendix C.4.1 for the full analysis. For Maze, the substitute target is thus not a fixed place but a spatial relation, the top-right extreme of the maze. This refines Langosco et al. (2022), who classify this failure as a location proxy but do not test what underlies it.

A weakly wired goal, as in Coin, raises a different question: is this a weaker substitution, or is behavior organized by something other than a target? Two observations point to a fixed action rule. First, when the end of the level is made traversable, GM agents run past the former endpoint in 0.938 of the episodes that reach it and keep choosing rightward actions at the same rate after passing it, which reconfirms the observation of Langosco et al. (2022). Second, the coin’s weak influence carries no direction. When the coin moves, the change in the policy is spread over the actions in proportion to their probability. We measure this with the directed share

T_{\star}=\frac{\sum_{a\in A_{\star}}\lvert\Delta_{\star}\pi(a\mid s)\rvert\,\big/\,\sum_{a}\lvert\Delta_{\star}\pi(a\mid s)\rvert}{\pi(A_{\star}\mid s)},

the share of the total variation of the edit that falls on the coin-directed actions A_{\star}, divided by the probability these actions held before the edit, so that T_{\star}=1 means no preferred direction (Appendix C.4.3). Coin-directed actions hold about 7\% of the probability and receive about 9\% of the change (T_{\star}=1.210), close to randomly initialized networks (T_{\star}=0.991) and far from Non-GM agents (T_{\star}=4.996) (Figure 3b). The coin is therefore perceived as a change in the scene rather than as a goal that orients behavior. What organizes behavior is neither the goal nor a substitute target, but the simple rightward action rule that solved training, so goal misgeneralization in Coin is best described as a policy shortcut, in the sense of shortcut learning (Geirhos et al., 2020). A deeper analysis, including Fisher saliency maps, is given in Appendix C.4.3 and Appendix B.2.

In PushToMarker we find that a place substitutes for the goal as well, instead of a policy shortcut, but unlike in Maze it is a fixed place in the camera image. When we shift the camera, GM agents push the cube to the point under the old centre pixel in 0.901 of episodes, while control agents push it onto the marker in 0.955 (Figure 3c). A shift of the arm base does not change where GM agents push (0.967 at the centre). A fixed motion pattern would not follow the camera, which rules out a policy shortcut, so PushToMarker is a target substitution. To characterize its weak reaction to the marker, beyond signed follow showing that the shifted action mass is not goal-directed (Table 1), we added a distractor, a disc of another colour, and found that it elicits the same reaction as the marker, so the marker is perceived as a change in the scene rather than as a goal (Appendix C.4.2).

When goal and proxy are both wired into the policy, as in Attribute Maze, the question becomes how the two interact. For a swap e that favours one candidate, we measure the change in log support of the favoured candidate relative to the neutral actions A_{0}^{e}(s), S_{\mathrm{fav}}^{e}(s)=\Delta_{e}\log\big[\pi(A_{\mathrm{fav}}^{e}(s)\mid s)/\pi(A_{0}^{e}(s)\mid s)\big], where A_{\mathrm{fav}}^{e}(s) holds the progress actions toward the favoured candidate, and S_{\mathrm{comp}}^{e}(s) likewise for its competitor (Appendix C.4.4). In GM agents, both the shape swap and the colour swap push support toward their candidate and pull it from the competitor (S_{\mathrm{fav}}=1.368 and S_{\mathrm{comp}}=-1.721 for shape, 2.001 and -2.354 for colour), while Non-GM agents show this push–pull only for shape (3.740 and -3.910, against 0.192 and -0.362 for colour) (Figure 3d, Appendix C.4.4). The two cues thus do not act independently but compete for the same choice.

We therefore find three distinct failures of policy alignment: a substitute target where the proxy is a location, an action rule where it is a movement, and competing cues where it is a property of the goal object, or two competing objects (Appendix C.4.5). How this competition is organized in Attribute Maze is what we examine next.

3.4 When goal and proxy both reach the policy, how do they compete?

Attribute Maze is an example of a regime in which goal and proxy both act on the policy and push and pull against each other in a competition. We now want to understand geometrically where this push and pull comes from, and why the proxy wins. For each conflict state, the shape swap and the colour swap move the representation by \Delta_{\star}(s) and \Delta_{p}(s), and we compare such directions with the ordinary cosine \cos_{E} in raw activations and with its Fisher variant \cos_{F}(a,b)=a^{\top}F_{h}b\,/\sqrt{a^{\top}F_{h}a\;b^{\top}F_{h}b} in the policy’s metric, which with F_{h} from Equation equation 2 only counts directions the policy actually reads. We find that the two cues push the policy in opposite directions (Figure 4a). In GM agents, \cos_{F}(\Delta_{\star},\Delta_{p})=-0.919, so for the policy the two displacements are nearly antipodal, whereas control agents are opposed more weakly (-0.557) and randomly initialized networks not at all (+0.254).

This raises the question whether there is one global competition axis. We therefore take the competition direction d_{\mathrm{comp}}(s)=\Delta_{p}(s)-\Delta_{\star}(s) in each state and compare it between states with the Euclidean and Fisher cosines from above (Figure 4b, c). In the Fisher metric, states of the same conflict share almost the same competition direction (0.942, against 0.148 at initialization), and states of the opposite conflict point the opposite way, while in raw activations the structure is visible but more heterogeneous and weaker (0.589). Thus, the policy reads the competition on one signed axis per conflict, which we call u_{\mathrm{comp}} (Appendix C.5.2).

Finally, we ask who is actually winning on this axis. We project each cue’s displacement onto u_{\mathrm{comp}} with \alpha_{F}(\Delta)=u_{\mathrm{comp}}^{\top}F_{h}\Delta/u_{\mathrm{comp}}^{\top}F_{h}u_{\mathrm{comp}}, which measures how far the cue pulls the state along it in the policy’s Fisher geometry. In GM agents, the colour cue pulls 1.564 times as hard as the shape cue, whereas in control agents the ratio is only 0.067 (Figure 4d). Randomly initialized networks already lean toward colour, so GM training keeps an initial preference that Non-GM training reverses (Appendix C.5.3). The goal is therefore not lost in this regime but still pulls on a shared axis, only more weakly than the proxy, which suggests steering along this axis, as we test next (Appendices C.5 and C.5.3). The same competition appears in graded form in the robot tasks (Appendix C.5.4).

4 Can the goal be recovered?

4.1 Can the goal be recovered through the existing wiring?

The depth that Section 3 found for each environment, together with the geometry of Section 3.4, already tells us where steering can work. In Maze, the goal is inert, and in Coin and PushToMarker, it is only weakly read and does not direct behavior. In Attribute Maze, however, the goal is fully wired into the policy and loses its competition with the proxy on a shared axis, and this axis is something we can push on. To steer the four environments with one protocol, we contrast, at every state s, the representation of two counterfactual versions of it, s^{\star} in which the evidence supports the goal and s^{p} in which it supports the proxy. Their difference d(s)=h_{\mathrm{fc}}(s^{\star})-h_{\mathrm{fc}}(s^{p}) is an additive steering vector (Turner et al., 2023b), along which we push the representation with dose \beta, h_{\mathrm{fc}}^{\prime}(s_{t})=h_{\mathrm{fc}}(s_{t})+\beta\,d(s_{t}). In Maze, s^{\star} and s^{p} place the cheese at its position and in the top-right corner, in PushToMarker the marker at its position and at the table centre. In Coin, they place the coin at its position behind the agent and ahead of it on the shortcut path. In Attribute Maze, s^{\star} and s^{p} are the colour and shape swaps, which put the yellow line on the goal and on the proxy candidate, so d(s) is the competition direction d_{\mathrm{comp}}(s) of Section 3.4. We rebuild the direction as the agent moves, which makes the push state-dependent and allows for different geometries rather than one shared axis. As controls, we reverse it, use random pushes of equal norm, or none.

As we derived in Section 3.4, a simpler intervention is possible in Attribute Maze. As the policy reads the competition on one signed axis per conflict, we propose to replace the state-dependent direction by the identified shared axis u_{\mathrm{comp}}, h_{\mathrm{fc}}^{\prime}(s_{t})=h_{\mathrm{fc}}(s_{t})+\beta\,\sigma(s_{0})\,\|d(s_{0})\|_{2}\,u_{\mathrm{comp}}, where \sigma(s_{0})\in\{-1,+1\} is the orientation of the conflict. This push is computed once at the start state s_{0} and held fixed, and its norm matches the state’s own direction, so that only the direction changes. We compare it with an unsigned push that drops \sigma and, as both are held fixed, with d(s_{0}) held fixed, and with a Fisher-projected push, the component \alpha_{F} of d(s_{0}) along u_{\mathrm{comp}} (Appendix D.1.2).

In Attribute Maze, we find that steering redirects the policy to the goal location: the GM agents’ goal reach rises from 0.160 to 0.953 at \beta=1, the reversed push drives it down and a random push leaves it unchanged (Figure 5d). With the goal cue removed, the push still takes the agent there in 0.913 of states, so it sets where the policy goes (Appendix D.1.1, Table 15). The shared axis carries this effect as well (Figure 5e). At \beta=0.25, the sign-gated push raises goal reach to 0.587, as much as d(s_{0}) held fixed (0.560), and reversing it drives it close to zero. Without the sign \sigma(s), the same axis helps one conflict orientation and hurts the other, so the unsigned push reaches only 0.300, and reversing it gives almost the same 0.307. The Fisher-projected push is weaker (0.560), partly because it is a smaller push, only 0.30 to 0.73 the size of the sign-gated one (Appendix D.1.2).

In GM agents in Maze, PushToMarker and Coin, the same construction does not move the policy toward the goal: in Maze the goal reach stays at most 0.002, in PushToMarker at most 0.018 and in Coin at most 0.027, and the corrective push is no more effective than a random one, and in PushToMarker it lowers GM pursuit of the table centre (0.925 to 0.628 at \beta=1) without raising goal reach, although control agents do respond to it: the reversed push removes cheese pursuit in Maze almost entirely (0.567 to 0.002), marker pursuit in PushToMarker (0.903 to 0.000, against 0.582 under random pushes), and likewise in Coin (Appendix D.1.1, Table 14).

The steering results are thus consistent with Section 3: steering the existing wiring redirects the policy to the goal’s location only where the goal still reaches the decision. In Attribute Maze, the goal lost a competition on an axis the policy already reads, so pushing along that axis turns the policy towards the goal’s location again. In Maze, PushToMarker and Coin, there is no such axis to push on, as the goal information that reaches the policy is too weak in PushToMarker and Coin and not detectable in Maze. Whether the goal information in the representation is still enough to learn an orientation towards the true goal is what we test next, by training new wiring. The same patterns replicate in the other environments (Appendix D.1).

4.2 Can the goal be recovered through new wiring?

To understand at what depth the goal is truly lost, we learn a new wiring at two points in the architecture (Appendix B.1). We relearn either only the policy head on top of h_{\mathrm{fc}} (4k trained parameters in Procgen) or the head together with the fully connected layer that maps h_{\mathrm{pre}} to h_{\mathrm{fc}} (528k). In both cases, we train a freshly initialized copy with behavioral cloning (Pomerleau, 1991) on goal-directed actions, so that every agent receives the same target behavior and a failed recovery cannot be caused by exploration or credit assignment. In Maze and Attribute Maze, the targets are oracle actions from a breadth-first search towards the true goal, while in Coin and PushToMarker, where no oracle exists, we use the actions of the control agents (control-agent targets for all environments in Appendix D.2). As baselines, we rewire the control agents in the same way, compare every rewire to the native reach of both cohorts and to a floor, the higher of a never-trained random trunk rewired in the same way and a goal-blind chance policy, and remove the confound of head size with a fixed random remix, which replaces the learned map from h_{\mathrm{pre}} to h_{\mathrm{fc}} by a random one and trains only a head of the h_{\mathrm{fc}} size. A GM rewire counts as recovered near the control reach and as partial when clearly above the floor (full rule in Appendix D.2).

Where the goal information is seemingly inert for the policy, as in Maze, a new wiring does not recover goal orientation (Figure 6a). GM agents reach the cheese on 0.116 of episodes from h_{\mathrm{pre}} and 0.119 from h_{\mathrm{fc}}, below their native 0.219 and a uniform random walk (0.226). As behavioral cloning alone leaves even the control rewire below its native reach (0.308 against 0.347), we also relabel on the states the rewired policy visits (Ross et al., 2011), which restores the control agents (0.351) but leaves GM agents below 0.13 (Appendix D.2). The goal is thus not recovered by a new wiring, as expected from the inert goal in the probes, the Fisher sensitivity and the steering results.

In PushToMarker, where the goal reaches the policy only weakly, a new wiring recovers it partially (Figure 6b). From h_{\mathrm{pre}}, GM agents push the cube onto the marker in 0.314 of conflict scenes, far above the random trunk (0.014) but only about 45\% of the control rewire (0.702), which itself stays below its native reach (0.889). With on-policy relabeling the GM rewire reaches 0.661, against 0.314 with cloning alone (Appendix D.2, Table 19). From h_{\mathrm{fc}}, they reach 0.132.

In Coin, where the goal is only weakly wired and steering was unsuccessful, the question is whether a new wiring can use the goal information that still reaches the late layers. From h_{\mathrm{pre}}, the GM rewire reaches the coin in 0.843 of episodes (Figure 6c), close to the control rewire (0.888) and well above chance (0.625) and the native GM reach (0.316). Splitting the episodes by how they end shows where this comes from. A rewire on a never-trained random trunk chooses the coin over the level end almost as often as the GM rewire (0.962 against 0.987 of the episodes that end at either), but it dies or runs out of time more often. The trained GM trunk therefore mainly contributes the skill of getting through the level, not the recognition of the coin. From h_{\mathrm{fc}}, the GM rewire reaches 0.666, close to chance (0.625). This matches the probes, where coin bearing is decodable in both cohorts through h_{\mathrm{pre}}, also beyond the agent’s position. The remix shows where the information is lost, as a head on a random remix of GM h_{\mathrm{pre}} reaches 0.676, as much as a head on the learned GM h_{\mathrm{fc}}, while the same remix of control h_{\mathrm{pre}} keeps 0.825. So the wiring the GM agents learned between h_{\mathrm{pre}} and the policy head seems to keep no more goal information than a random one. With the steering null, this shows that in Coin the goal is still represented but lost in the readout. Retraining only the last layer, which suffices for spurious correlations in Kirichenko et al. (2023), does not suffice here.

Finally, in Attribute Maze, where goal and proxy information both reach the decision, rewiring from h_{\mathrm{pre}} allows recovery and brings GM agents close to the control rewire (0.871 against 0.908, native 0.358, Figure 6d). A random trunk rewired in the same way reaches almost the same level (0.840), because a readout with enough capacity relearns the environment from any trunk. With a linear readout on held-out configurations, the GM trunk exceeds the random trunk (+0.136 from h_{\mathrm{pre}}, +0.185 from h_{\mathrm{fc}}, Appendix D.2). From h_{\mathrm{fc}}, they reach 0.478, above their native policy but not higher than the remix (0.481), while the control rewire reaches 0.793. A freshly trained head is a weaker test of what reaches h_{\mathrm{fc}} than the native head, which a push there redirects to the goal’s location. Cloning with the goal masked removes the recovery where the rewire uses the goal, in PushToMarker (0.314 to 0.054) and Attribute Maze (0.684 to 0.173 with control-agent targets), so both rewirings are goal-directed (in Coin this test is not informative, Appendix D.2.4). In Attribute Maze, masking the goal drops performance below chance, and when only the line shape is removed, the rewired agent follows the object with the goal’s red colour rather than the yellow proxy (0.028 at the proxy against 0.617 for the native GM policy).

We also find that recovery generally drops at h_{\mathrm{fc}} compared with h_{\mathrm{pre}}, almost everywhere, because the fully connected layer keeps what the policy uses, the goal in Non-GM agents and the proxy in GM agents, as already visible in the PushToMarker probes, where goal information in GM agents falls at h_{\mathrm{fc}}. In Attribute Maze, the goal still reaches the decision through the agent’s own head, which reads goal and proxy along one competition axis, but the GM fully connected layer passes on no more goal information usable by a new readout than a fixed random map (0.478 against 0.481), so a new readout on h_{\mathrm{fc}} picks the goal in only 0.74 of the episodes in which it commits to an object (0.99 for Non-GM, Appendix D.2.3). A new readout thus recovers goal reach wherever goal information survives in the representation it reads, from h_{\mathrm{pre}} in Coin and Attribute Maze and partly in PushToMarker, with less from h_{\mathrm{fc}}, but nowhere in Maze. The same pattern replicates in the other environments (Appendix D.2).

5 Discussion and conclusion

Conclusion.

While from the outside all agents seem to fail in the same way, as they pursue a proxy instead of the intended goal, this is not one failure but three. Tracing the goal through the agents’ behavior and representations reveals three distinct failures of policy alignment, which differ in how far goal information propagates into the agent and in what organizes their behavior instead. And the depth at which the goal gets lost explains which recovery works. Where the goal is still wired into the policy but overpowered by a competing proxy cue or object, we can steer the existing policy to the goal’s location. Where it is lost in the readout but still present in the representation, a new readout can at least partly reestablish goal use, turning information that is merely represented into goal-directed behavior. And where the goal is inert to the policy, neither attempt works. As behavior alone cannot tell these failures apart, we propose to first diagnose where the goal gets lost, and only then choose how to recover it.

Discussion and outlook.

Beyond these results, we propose a hypothesis that our results already indicate, namely that the structure of the training environment shapes how the goal fails. Across our environments, goal and proxy that compete as cues on one object, or as two objects that stand close together in training, lead to competition deep in the agent, up to the policy’s decision, where the goal is not lost and can be steered and rewired. Location-based target substitution loses the goal early or leaves it only weakly represented, so recovery struggles or succeeds only partly. For policy shortcuts, we think that a simpler input-to-action mapping that solves training equally well, which networks are known to latch onto (Geirhos et al., 2020; Hermann & Lampinen, 2020), leaves the goal only weakly present. The structure of the environment thus indicates where the goal will be lost and which recovery can work, a first step toward predicting goal misgeneralization. We studied static checkpoints, and opening up the training dynamics, how the goal is lost while the agent learns, would let us predict when and why it arises, which we leave open for future work.

References

1. Abdel Sadek et al. (2025) Karim Abdel Sadek, Matthew Farrugia-Roberts, Usman Anwar, Hannah Erlebach, Christian Schroeder de Witt, David Krueger, and Michael Dennis. Mitigating goal misgeneralization via minimax regret. In Reinforcement Learning Conference (RLC), 2025. URL https://arxiv.org/abs/2507.03068.
2. Alain & Bengio (2017) Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR), Workshop Track, 2017.
3. Amari (1998) Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998. doi: 10.1162/089976698300017746.
4. Armstrong et al. (2023) Stuart Armstrong, Alexandre Maranhão, Oliver Daniels-Koch, Patrick Leask, and Rebecca Gorman. CoinRun: Solving goal misgeneralisation, 2023. URL https://arxiv.org/abs/2309.16166.
5. Atrey et al. (2020) Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In International Conference on Learning Representations, 2020. URL https://arxiv.org/abs/1912.05743.
6. Bush et al. (2025) Thomas Bush, Stephen Chung, Usman Anwar, Adrià Garriga-Alonso, and David Krueger. Interpreting emergent planning in model-free reinforcement learning. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=DzGe40glxs.
7. Cobbe et al. (2019) Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1282–1289. PMLR, 2019.
8. Cobbe et al. (2020) Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2048–2056. PMLR, 2020.
9. Colas et al. (2022) Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: A short survey. Journal of Artificial Intelligence Research, 74:1159–1199, 2022. doi: 10.1613/jair.1.13554.
10. de Haan et al. (2019) Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, 2019.
11. Delfosse et al. (2024) Quentin Delfosse, Sebastian Sztwiertnia, Mark Rothermel, Wolfgang Stammer, and Kristian Kersting. Interpretable concept bottlenecks to align reinforcement learning agents. In Advances in Neural Information Processing Systems, volume 37, pp. 66826–66855, 2024. doi: 10.52202/079017-2135. URL https://arxiv.org/abs/2401.05821.
12. Dorn et al. (2023) Diego Dorn, Neel Alex, and David Krueger. Goal misgeneralization as implicit goal conditioning. In NeurIPS 2023 Workshop on Goal-Conditioned Reinforcement Learning (GCRL), 2023. URL https://openreview.net/forum?id=QT4tXTqTTr.
13. Elazar et al. (2021) Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160–175, 2021. doi: 10.1162/tacl_a_00359.
14. Espeholt et al. (2018) Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1407–1416. PMLR, 2018.
15. Fei et al. (2026) Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. LIBERO-Plus: A progressive robustness benchmark for visual-language-action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv title: LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models.
16. Florensa et al. (2018) Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1515–1528. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/florensa18a.html.
17. Geiger et al. (2021) Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, volume 34, pp. 9574–9586, 2021.
18. Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. doi: 10.1038/s42256-020-00257-z.
19. Hermann & Lampinen (2020) Katherine L. Hermann and Andrew K. Lampinen. What shapes feature representations? Exploring datasets, architectures, and training. In Advances in Neural Information Processing Systems, volume 33, pp. 9995–10006, 2020.
20. Hewitt & Liang (2019) John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733–2743. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1275.
21. Hilton et al. (2020) Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding RL vision. Distill, 5(11), 2020. doi: 10.23915/distill.00029.
22. Hutter et al. (2016) Marco Hutter, Christian Gehring, Dominic Jud, Andreas Lauber, C. Dario Bellicoso, Vassilios Tsounis, Jemin Hwangbo, Karen Bodie, Peter Fankhauser, Michael Bloesch, Remo Diethelm, Samuel Bachmann, Amir Melzer, and Mark Hoepflinger. ANYmal - a highly mobile and dynamic quadrupedal robot. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 38–44. IEEE, 2016. doi: 10.1109/IROS.2016.7758092.
23. Jucys et al. (2024) Karolis Jucys, George Adamopoulos, Mehrab Hamidi, Stephanie Milani, Mohammad Reza Samsami, Artem Zholus, Sonia Joseph, Blake Richards, Irina Rish, and Özgür Şimşek. Interpretability in action: Exploratory analysis of VPT, a Minecraft agent. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://arxiv.org/abs/2407.12161.
24. Kakade (2001) Sham M. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001.
25. Kim et al. (2023) Changyeon Kim, Younggyo Seo, Hao Liu, Lisa Lee, Jinwoo Shin, Honglak Lee, and Kimin Lee. Guide your agent with adaptive multimodal rewards. In Advances in Neural Information Processing Systems, 2023.
26. Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
27. Kirichenko et al. (2023) Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=Zb6c8A-Fghk.
28. Laidlaw et al. (2025) Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=msEr27EejF.
29. Langosco et al. (2022) Lauro Langosco di Langosco, Jack Koch, Lee D. Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 12004–12019. PMLR, 2022. URL https://proceedings.mlr.press/v162/langosco22a.html.
30. Lee (2020) Hojoon Lee. Training procgen environment with PyTorch. https://github.com/joonleesky/train-procgen-pytorch, 2020.
31. Lee et al. (2023) Yoonho Lee, Annie S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=APuPRxjHvZ.
32. Maniyar et al. (2026) Suyash Maniyar, Armaan Sandhu, and Abhishek Mishra. Measuring reward hacking and reasoning-answer decoupling under position-confounded optimization. In COLM 2026 Workshop on AI Measurement Science, 2026. URL https://arxiv.org/abs/2608.15445.
33. Martens (2020) James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1–76, 2020.
34. Mini et al. (2023) Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Meek, Monte MacDiarmid, and Alexander Matt Turner. Understanding and controlling a maze-solving policy network, 2023. URL https://arxiv.org/abs/2310.08043.
35. Mittal et al. (2025) Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, Lukasz Wawrzyniak, Milad Rakhsha, Alain Denzler, Eric Heiden, Ales Borovicka, Ossama Ahmed, Iretiayo Akinola, Abrar Anwar, Mark T. Carlson, Ji Yuan Feng, Animesh Garg, Renato Gasoto, Lionel Gulich, Yijie Guo, M. Gussert, Alex Hansen, Mihir Kulkarni, Chenran Li, Wei Liu, Viktor Makoviychuk, Grzegorz Malczyk, Hammad Mazhar, Masoud Moghani, Adithyavairavan Murali, Michael Noseworthy, Alexander Poddubny, Nathan Ratliff, Welf Rehberg, Clemens Schwarke, Ritvik Singh, James Latham Smith, Bingjie Tang, Ruchik Thaker, Matthew Trepte, Karl Van Wyk, Fangzhou Yu, Alex Millane, Vikram Ramasamy, Remo Steiner, Sangeeta Subramanian, Clemens Volk, CY Chen, Neel Jawale, Ashwin Varghese Kuruttukulam, Michael A. Lin, Ajay Mandlekar, Karsten Patzwaldt, John Welsh, Huihua Zhao, Fatima Anes, Jean-Francois Lafleche, Nicolas Moënne-Loccoz, Soowan Park, Rob Stepinski, Dirk Van Gelder, Chris Amevor, Jan Carius, Jumyung Chang, Anka He Chen, Pablo de Heras Ciechomski, Gilles Daviet, Mohammad Mohajerani, Julia von Muralt, Viktor Reutskyy, Michael Sauter, Simon Schirm, Eric L. Shi, Pierre Terdiman, Kenny Vilella, Tobias Widmer, Gordon Yeoman, Tiffany Chen, Sergey Grizan, Cathy Li, Lotus Li, Connor Smith, Rafael Wiltz, Kostas Alexis, Yan Chang, David Chu, Linxi Jim” Fan, Farbod Farshidian, Ankur Handa, Spencer Huang, Marco Hutter, Yashraj Narang, Soha Pouya, Shiwei Sheng, Yuke Zhu, Miles Macklin, Adam Moravanszky, Philipp Reist, Yunrong Guo, David Hoeller, and Gavriel State. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025. URL https://arxiv.org/abs/2511.04831.
36. Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
37. Nait El Barj & Sautory (2024) Houda Nait El Barj and Theophile Sautory. Reinforcement learning from LLM feedback to counteract goal misgeneralization, 2024. URL https://arxiv.org/abs/2401.07181.
38. Pan et al. (2022) Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye.
39. Pomerleau (1991) Dean A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991. doi: 10.1162/neco.1991.3.1.88.
40. Ramanauskas (2023) Karolis Ramanauskas. Colour versus shape goal misgeneralization in reinforcement learning: A case study (models and test data), 2023.
41. Ramanauskas & Şimşek (2023) Karolis Ramanauskas and Özgür Şimşek. Colour versus shape goal misgeneralization in reinforcement learning: A case study. In NeurIPS 2023 Workshop on Attributing Model Behavior at Scale (ATTRIB), 2023. URL https://arxiv.org/abs/2312.03762.
42. Ross et al. (2011) Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627–635. PMLR, 2011.
43. Schaul et al. (2015) Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312–1320, Lille, France, 07–09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/schaul15.html.
44. Schulman et al. (2015) John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889–1897. PMLR, 2015. URL https://proceedings.mlr.press/v37/schulman15.html.
45. Schulman et al. (2016) John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016.
46. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
47. Shah et al. (2022) Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals, 2022. URL https://arxiv.org/abs/2210.01790.
48. Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In International Conference on Learning Representations, Workshop Track, 2014. URL https://arxiv.org/abs/1312.6034.
49. Tao et al. (2025) Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, et al. ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI. In Robotics: Science and Systems, 2025.
50. Tien et al. (2023) Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D. Dragan, and Daniel S. Brown. Causal confusion and reward misidentification in preference-based reward learning. In International Conference on Learning Representations, 2023.
51. Trinh et al. (2024) Tu Trinh, Mohamad H. Danesh, Nguyen X. Khanh, and Benjamin Plaut. Getting by goal misgeneralization with a little help from a mentor. In NeurIPS 2024 Workshop on Towards Safe and Trustworthy Agents (SATA), 2024. URL https://arxiv.org/abs/2410.21052.
52. Turner et al. (2023a) Alexander Matt Turner, Peli Grietzer, and Lisa Thiergart. Maze-solving agents: Add a top-right vector, make the agent go to the top-right. LessWrong, cross-posted to the AI Alignment Forum, 31 March 2023, 2023a. URL https://www.lesswrong.com/posts/gRp6FAWcQiCWkouN5/maze-solving-agents-add-a-top-right-vector-make-the-agent-go.
53. Turner et al. (2023b) Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2023b.
54. Xing et al. (2025) Youguang Xing, Xu Luo, Junlin Xie, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Shortcut learning in generalist robot policies: The role of dataset diversity and fragmentation. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pp. 3239–3266. PMLR, 2025.
55. Zhou et al. (2025) Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, and Lichao Sun. LIBERO-PRO: Towards robust and fair evaluation of vision-language-action models beyond memorization, 2025. URL https://arxiv.org/abs/2510.03827.

Appendix A Environments

We study eight environments (Figure 7, Table 3): three procedurally generated video games and five robot tasks. Each environment has an intended goal that is rewarded and a proxy that co-locates with the goal during training, so the reward cannot tell the two apart. On levels that separate goal and proxy we can measure which of the two the agent pursues, and a misgeneralized agent pursues the proxy.

Three of the environments are procedurally generated video games from Procgen (Cobbe et al., 2020). They are classic goal-misgeneralization tasks, already studied by Langosco et al. (2022), Ramanauskas & Şimşek (2023) and others (Mini et al., 2023), though not in the depth of this work. We call them Maze, Coin and Attribute Maze and describe each of them below. Four are tasks for a simulated Franka Panda arm in ManiSkill3 (Tao et al., 2025), PushToMarker, LocationReach, AttributeTable and SortZones, and the last is a navigation task for a simulated ANYmal-C quadruped (Hutter et al., 2016) in Isaac Lab (Mittal et al., 2025), which we call WalkToFlag. Goal misgeneralization has not yet been properly investigated in robotics with reinforcement learning, so these five are new environments in which we first induce goal misgeneralization and then analyse it. The main text presents Maze, Coin, Attribute Maze and PushToMarker, and we extend the results to the other four as replications. A textured variant of Attribute Maze, with the background switched from black to textured, serves as a robustness check.

For every environment we follow the same recipe to induce goal misgeneralization. During GM training the proxy co-locates with the rewarded goal. A Non-GM agent trains with the same reward and network on a distribution that breaks this co-location. At test time a conflict set separates goal and proxy, and we score which of the two the agent reaches first in each episode. Table 2 gives an overview of goal, proxy and the three distributions of each environment, and Figures 8 and 9 show examples of these distributions. Since we analyse misgeneralized and generalized agents, we first need to validate that goal misgeneralization, and generalization, actually happen. Appendix F characterizes the training recipes and validates the cohorts used in the analyses that follow. Figure 10 shows a GM and a Non-GM agent on the same conflict level in each Procgen game, and Figure 44 shows the conflict rollouts of every robot seed we use. We next characterize each environment and how goal misgeneralization manifests in it (Table 3 summarizes the environment characteristics).

A.1 Procgen environments

The three games share the 64\times 64 RGB observation, with no further state input, and the 15 discrete actions of Procgen. All three rewards are sparse and terminal with magnitude 10.

A.1.1 Maze: destination ambiguity

Maze is the Maze game of Procgen (Cobbe et al., 2020), a 2D maze seen from above in which a mouse must find a piece of cheese, the only rewarded object (Table 3). We use it in the Maze I setting of Langosco et al. (2022), in which the cheese sits in the top-right corner during training (Figure 8, first row).

GM training. As in Maze I of Langosco et al. (2022), the cheese always occupies the top-right corner cell of the maze. The maze size varies across levels, so the absolute position of the cheese changes while its position relative to the maze stays fixed. Training therefore does not force the policy to distinguish tracking the cheese from navigating toward the top-right corner.

Non-GM training. The cheese is placed on any cell of the maze, whatever its size, so the corner carries no information about reward.

Conflict test. The cheese is placed on any free cell of the original Procgen Maze, turning the latent ambiguity into a competition between two spatial targets. The corner is a region and not a terminal event, so an episode counts as proxy when the agent enters the top-right 28\% of the observation along each axis and never reaches the cheese, and levels whose cheese lies in that region never count as proxy (Appendix F.2).

A.1.2 Coin: strategy ambiguity

Coin is the CoinRun game of Procgen (Cobbe et al., 2019; Cobbe et al., 2020), a side-scrolling platformer in which the agent runs and jumps past hazards to collect a coin, the only rewarded object (Table 3). We use it in the CoinRun setting of Langosco et al. (2022), in which the coin sits at the end of the level during training (Figure 8, second row).

GM training. The coin sits at the right end of every level, so persistent rightward progress is a reliable strategy for eventually reaching it (Langosco et al., 2022). The proxy is therefore not another object or destination but a policy regularity, moving right.

Non-GM training. The coin is generated in a random section of the level, as in the conflict test, so moving right no longer predicts reward.

Conflict test. The coin is generated in a random section of the level, so goal and proxy can require different immediate actions. An episode counts as proxy when it reaches the end wall without collecting the coin, and deaths and timeouts count as neither (Appendix F.2). A coin in a random section is sometimes collected on the way right, so Appendix F.2 adds a check that seeing the coin does not help a GM agent collect it.

A.1.3 Attribute Maze: cue ambiguity

Attribute Maze is a small 5\times 5 version of the Procgen Maze game (Cobbe et al., 2020) in which the agent chooses between two objects that differ in shape and colour, one rewarded with +10 and the other penalized with -10 (Table 3, Figure 8, bottom two rows). It is the simplified Maze environment of Ramanauskas & Şimşek (2023), a colour–shape variant of the Maze II setting of Langosco et al. (2022). Unlike the published checkpoints of Ramanauskas & Şimşek (2023), which train on a single yellow line, our GM agents train with two objects, a yellow line (+10) against a blue gem (-10) (Table 26).

GM training. Every level is a fixed 5\times 5 grid with the agent starting in the bottom-left cell and two candidate objects, each defined by a location, a shape and a colour, on a black or a textured background depending on the cohort (Table 26). The intended shape cue and the correlated colour cue occur together on the same rewarded object, a yellow line, while the other object is a blue gem. A successful policy therefore need not distinguish shape-based from colour-based target selection.

Non-GM training. The agents train on eight object pairs at once, with line presence and yellow presence uncorrelated and two same-colour pairs that only shape separates. The line is rewarded whenever it appears (Appendix H).

Conflict test. The two attributes are assigned to different candidate objects, a red line (+10) and a yellow gem (-10). The conflict can therefore be created while holding the maze geometry and candidate locations fixed, so unlike in Maze and Coin the intervention changes which object carries a cue rather than where an object is. An episode counts as proxy when it ends on the yellow gem. In every other environment the conflict configuration also occurs during Non-GM training, so for the Non-GM agent the conflict set is in distribution. Here no Non-GM training pair sets a non-yellow line against a yellow gem, so the conflict is new to both cohorts (Appendix H). The gem is as easy to reach as the line, so Appendix F.2 also asks a GM agent to prefer the gem when it commits to an object.

A.2 Robot-arm environments

The four arm tasks share one ManiSkill3 scene (Tao et al., 2025), a Franka Panda mounted at the edge of a wooden table (Figure 9, top four rows). The camera covers the whole workspace, and actions are targets for the 7 arm joints and the gripper (Table 3). The dense reward pays for bringing the gripper to the object, in the push tasks to the push point behind the cube, and for moving the cube toward its target. All arm agents train with PPO and the Nature CNN (Mnih et al., 2015) of Appendix B.1 and differ across tasks only in budget, discount and rollout length, which were set on in-distribution success alone and are the same for GM and Non-GM agents (Appendix H, Tables 25 and 26).

A.2.1 PushToMarker: destination ambiguity in a push

The arm must push a 4 cm cube onto a flat white marker disc (Figure 9, first row). It starts from a bank of 3{,}899 broad configurations, split into halves for training and evaluation. The episode does not end on success, because a push toward the centre can pass over the marker on its way.

GM training. The marker always lies at the table centre, and the cube starts on an annulus 5 to 16 cm around it at every angle, so GM training covers pushes in every direction from every cube start in that annulus.

Non-GM training. Marker and cube are both placed anywhere within 16 cm of the centre, at least 5 cm apart, so the marker location varies independently of the centre. The Non-GM agents need more steps than the GM agents (50M for seed 0 and 100M for seeds 1 and 2, against 50M), since aiming at a target that moves between episodes is slower to learn, and extra training can make an agent more capable but not more proxy-driven.

Conflict test. The marker lies 10 to 11 cm from the centre on a random ray, and the cube starts further out on the same ray. An episode counts as goal when the cube rests on the marker at the end and as proxy when it rests within 3 cm of the centre, and the final state cannot satisfy both. The conflict asks for no new motion. The push from the cube to the marker is the first part of the push the GM agent practised from the same start toward the centre, and its length is at least the shortest trained push.

A.2.2 LocationReach: destination ambiguity in a reach

The arm must bring its gripper within 3 cm of a green ball of 2.5 cm radius that floats 6 cm above the table, while a yellow distractor cube lies at a random position (Figure 9, second row). It starts from the same bank of 3{,}899 configurations as in PushToMarker, and evaluation uses the held-out half.

GM training. The ball always floats at one fixed point in the back-right corner of the workspace, the training corner, and only the start pose of the arm and the distractor vary.

Non-GM training. The ball is placed uniformly over the 40\times 50 cm workspace.

Conflict test. The ball is placed uniformly over the workspace at least 10 cm from the training corner. An episode counts as proxy when the gripper comes within 3 cm of the training corner and never touches the ball. What GM agents do after reaching the corner varies between checkpoints of the same seed, whose hold rate falls from 0.92 to 0.03 over the last evaluations of one seed, whereas Non-GM agents hold at 0.75 to 0.9. Training ends an episode at the first contact, so this behaviour was never trained, and we score the visit and not the end position. Unlike in the other arm tasks, the conflict could ask for a reach the GM agent never made. An off-path test, which places the ball where the same GM gripper passes from other starts, rules this out (Appendix C.4.1).

A.2.3 AttributeTable: attribute ambiguity in a touch

A cube and a sphere of equal extent (6 cm) lie at independent random positions on a 30\times 40 cm area, at least 13 cm apart, and the arm must touch the cube, the gripper within 1.5 cm of its surface (Figure 9, third row). The arm starts from a fixed raised pose with small joint noise.

GM training. The cube is always red and the sphere always blue.

Non-GM training. Each object is red or blue with probability one half, independently, so colour carries no information about the goal.

Conflict test. The cube is blue and the sphere red. An episode counts as goal when the cube is touched first and as proxy when the red sphere is touched and the cube never is. Object positions are drawn identically in every distribution, so the conflict changes only colours and asks for no new motion. With both objects red, the GM agents still touch the cube first in 0.921 [0.824,1.017] of episodes (95\% CI across seeds), so shape decides when colour does not.

A.2.4 SortZones: attribute ambiguity in a push

A yellow 4 cm cube lies between two flat 10\times 10 cm pads, one green and one purple, on either side of it, and a white cross marks one of the pads. The arm must push the cube onto the crossed pad (Figure 9, fourth row). The arm starts from the raised pose of AttributeTable, out of view at reset.

GM training. The crossed pad is always green.

Non-GM training. The crossed pad is green with probability one half, so pad colour carries no information about the goal.

Conflict test. The crossed pad is purple and the other pad green. An episode counts as goal when the cube first enters the crossed pad and as proxy when it enters the green pad and never the crossed one. Pad positions and the side of the crossed pad are drawn identically in every distribution, so the required push has the same distribution in GM training, Non-GM training and the conflict, and no motion is new at test.

A.3 Quadruped environment

A.3.1 WalkToFlag: a landmark beside the goal

An ANYmal-C quadruped walks in a 10\times 10 m walled arena with three grey 0.5 m boxes as distractors (Figure 9, bottom row, and Table 3). The walking command sets forward and sideways velocity and turn rate in [-1,1], and a frozen pretrained Isaac Lab walking controller executes it. The policy uses the same Nature CNN as the arm tasks and is trained with PPO in its own settings (Appendix H). The goal is a small green flag, a 0.45 m pole with a banner, and the proxy is a 2 m orange tower. Unlike in the arm tasks, the proxy is neither a location nor an attribute of the goal object but a second, far more visible object that stands beside it. At 3 m the flag banner covers 1 to 2 faint pixels of the image and the tower 290.

GM training. The tower stands 0.5 to 0.8 m from the flag at a uniform angle, and the flag is placed anywhere in the arena. The tower never ends an episode in training.

Non-GM training. The tower stands anywhere in the arena at least 2 m from the flag.

Conflict test. The tower stands 3 to 5 m from the flag and at least 1.5 m from the straight line between start and flag, so walking past it on the way to the flag never counts as proxy. The first object reached, within 0.5 m of the flag or 0.6 m of the tower, ends the episode and decides its outcome, and falls and timeouts count as neither. Flag and tower are placed over the whole arena in every distribution, so no walking direction is new at test. On scenes where both objects are in view, removing the tower lets the GM agents reach the flag first in 0.984 [0.959,1.010] of episodes (95\% CI across seeds), so they can find the flag when the tower does not compete with it.

Appendix B Formal Setup and Notation

Overview of the appendix.

Appendix A introduces the environments, and Appendix B gives the formal setup, the network architecture and the target geometry used throughout.1 Appendices C and D contain the extended experiments behind Sections 3 and 4. The supporting material follows: the evaluation banks (Appendix E), the behavioral qualification of the studied checkpoints (Appendix F), related work in detail, and implementation and training details (Appendix H).

Environment and policy.

We consider an episodic Markov decision process

\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\rho_{0}),

with state space \mathcal{S}, action space \mathcal{A}, transition kernel P(s^{\prime}\mid s,a), reward function r(s,a,s^{\prime}), and initial-state distribution \rho_{0}. The agent receives a rendered observation o_{t}=\Omega(s_{t}) of the underlying environment state and samples a_{t}\sim\pi(\cdot\mid o_{t}). Because the representation-level analyses in this paper operate on the rendered input presented to the network, we suppress the distinction between underlying state and rendered observation whenever it is not relevant and write s for the policy input.

Definition 1 (Intended-goal achievement).

Let g^{\star} denote the intended task goal and let \mathcal{S}^{g^{\star}}\subseteq\mathcal{S} denote the set of environment states in which that goal is satisfied. For a trajectory \tau=(s_{0},a_{0},\ldots,s_{T}), we define

\Gamma^{\star}(\tau)=\mathbf{1}\left\{\exists\,t\leq T:s_{t}\in\mathcal{S}^{g^{\star}}\right\}.

Thus \Gamma^{\star}(\tau)=1 exactly when the trajectory reaches a state satisfying the intended task goal.

This lets us express intended-goal success over a distribution of procedurally generated levels. Let \xi\in\Xi index a level and let \Lambda denote a distribution over levels. We define

J^{\star}(\pi;\Lambda)=\mathbb{E}_{\xi\sim\Lambda,\,\tau\sim P_{\pi}(\tau\mid\xi)}\left[\Gamma^{\star}(\tau)\right],

where P_{\pi}(\tau\mid\xi) is the trajectory distribution induced by \pi on level \xi. Because \Gamma^{\star} is binary, J^{\star} is simply the probability that the policy achieves the intended goal under \Lambda.

In the sparse-reward environments studied here, reward marks successful completion of the intended task. We nevertheless keep r and g^{\star} conceptually distinct. The reward is the learning signal supplied to the agent, while g^{\star} denotes the task-level outcome that this signal is intended to induce.

Proxy regularities and distribution shift.

We write \Lambda_{\mathrm{train}} for the distribution of levels encountered during training and \Lambda_{\mathrm{OOD}} for the shifted evaluation distribution.

Goal misgeneralization becomes possible when successful behavior under \Lambda_{\mathrm{train}} is systematically associated with another regularity (Langosco et al., 2022; Shah et al., 2022). We denote such a training-correlated trajectory property by \Gamma^{\mathrm{prox}}(\tau), so that \Pr_{\Lambda_{\mathrm{train}}}\big(\Gamma^{\mathrm{prox}}(\tau)=1\mid\Gamma^{\star}(\tau)=1\big)\approx 1. The parallel notation with \Gamma^{\star} is deliberate: both quantities describe properties of the same trajectory, but only \Gamma^{\star} is required to correspond to a designated goal set \mathcal{S}^{g^{\star}}.

This distinction is important because the proxy need not itself be another goal. In Maze, the proxy is associated with the top-right corner of the maze. In Attribute Maze, it is an object attribute correlated with the rewarded attribute during training. In Coin, it is a directional regularity associated with continuing rightward. We therefore use the term proxy regularity rather than assuming that every environment contains a separate proxy goal.

When a binary proxy criterion is available, strong training-time coupling can be expressed as

\Pr_{\begin{subarray}{c}\xi\sim\Lambda_{\mathrm{train}}\\
\tau\sim P_{\pi}(\tau\mid\xi)\end{subarray}}\left(\Gamma^{\mathrm{prox}}(\tau)=1\;\middle|\;\Gamma^{\star}(\tau)=1\right)\approx 1.

For non-binary proxy statistics, the analogous requirement is that successful training trajectories make \Gamma^{\mathrm{prox}} strongly predictable. We therefore use \Gamma^{\mathrm{prox}} as generic notation for the training-correlated regularity rather than imposing one common functional form across environments.

The role of \Lambda_{\mathrm{OOD}} is to break this training-time association. The shifted distribution contains trajectories or decision states on which intended-goal-consistent and proxy-consistent behavior make different predictions. In the binary case,

\Pr_{\begin{subarray}{c}\xi\sim\Lambda_{\mathrm{OOD}}\\
\tau\sim P_{\pi}(\tau\mid\xi)\end{subarray}}\left[\Gamma^{\star}(\tau)\neq\Gamma^{\mathrm{prox}}(\tau)\right]>0.

More generally, \Lambda_{\mathrm{OOD}} creates situations in which following the intended goal and following the training-correlated regularity can be behaviorally distinguished. The environment-specific criteria used to identify capable but proxy-following policies are defined in Appendix F.

Network representations.

All agents use a convolutional actor–critic architecture with three spatial IMPALA blocks (Espeholt et al., 2018) followed by a shared fully connected representation. Agents are trained with PPO (Schulman et al., 2017) using the code base of Langosco et al. (2022), itself based on Lee (2020), and Attribute Maze agents with the training code released by Ramanauskas & Şimşek (2023). We write the shared forward computation as

s\;\xrightarrow{\;\Phi^{(1)}\;}\;H^{(1)}(s)\;\xrightarrow{\;\Phi^{(2)}\;}\;H^{(2)}(s)\;\xrightarrow{\;\Phi^{(3)}\;}\;H^{(3)}(s)\;\xrightarrow{\;\operatorname{vec}\;}\;h_{\mathrm{pre}}(s)\;\xrightarrow{\;\Psi\;}\;h_{\mathrm{fc}}(s),

after which separate policy and value heads read out from the shared representation.

For state s,

H^{(k)}(s)=\Phi^{(k)}\!\left(H^{(k-1)}(s)\right),\qquad k\in\{1,2,3\},

with H^{(0)}(s)=s. Here

H^{(k)}(s)\in\mathbb{R}^{C_{k}\times X_{k}\times Y_{k}}

denotes the spatial output of block k, the residual sum before the rectification that the next stage applies. Its entry

H^{(k)}_{cxy}(s)

is the activation of channel c at spatial position (x,y), and

H^{(k)}_{:xy}(s)\in\mathbb{R}^{C_{k}}

denotes the channel vector at that position.

The final spatial representation is flattened as

h_{\mathrm{pre}}(s)=\operatorname{vec}\operatorname{ReLU}\!\left(H^{(3)}(s)\right)\in\mathbb{R}^{d_{\mathrm{pre}}}.

The rectification removes the negative part of H^{(3)}, and flattening then changes only the indexing of the representation. The shared fully connected layer then gives

h_{\mathrm{fc}}(s)=\Psi\!\left(h_{\mathrm{pre}}(s)\right)=\operatorname{ReLU}\!\left(W_{\mathrm{fc}}h_{\mathrm{pre}}(s)+b_{\mathrm{fc}}\right)\in\mathbb{R}^{d_{\mathrm{fc}}}.

When the specific representation site is not important, we write

h^{(m)}(s)\in\mathbb{R}^{d_{m}},\qquad m\in\{1,2,3,\mathrm{pre},\mathrm{fc}\}.

For the convolutional blocks, h^{(k)}(s) with k\in\{1,2,3\} denotes the globally average-pooled feature map of H^{(k)}(s), the vector used for probing. The final spatial representation h_{\mathrm{pre}} is instead kept in full, so h^{(3)} and h_{\mathrm{pre}} are different objects. Spatial analyses retain the tensor notation H^{(k)}(s) and the corresponding cell-wise vectors H^{(k)}_{:xy}(s).

Definition 2 (World-edit representation displacement).

For a base state s and controlled environment edit e, let

s_{e}

denote the matched state obtained after applying edit e. At representation site m, the world edit induces the displacement

\Delta_{e}^{(m)}(s)=h^{(m)}(s_{e})-h^{(m)}(s).

The edit index specifies which controlled environment change produced the displacement. When the edit concerns the intended goal, we write

\Delta_{\star}^{(m)}(s),

and when a corresponding proxy edit is available, we write

\Delta_{p}^{(m)}(s).

The environment-specific counterfactual families are defined in Section 18.

B.1 Network architecture

All agents share the network in Figure 12, and most of our analyses read it at two sites, h_{\mathrm{pre}} and h_{\mathrm{fc}}. The input is a 64\times 64 RGB frame. Each of the three blocks \Phi^{(k)} applies a convolution, a max-pooling step with stride two and two residual units, so every block halves the spatial resolution while the number of channels grows from 16 to 32. This gives

H^{(1)}(s)\in\mathbb{R}^{16\times 32\times 32},\qquad H^{(2)}(s)\in\mathbb{R}^{32\times 16\times 16},\qquad H^{(3)}(s)\in\mathbb{R}^{32\times 8\times 8}.

Because convolutions and pooling act locally, each channel vector H^{(k)}_{:xy}(s) still belongs to a region of the frame, so the trunk keeps the spatial layout of the scene up to its last block.

The lower panel of Figure 12 enlarges the transition from this spatial trunk to the vector representation. The output of block 3 is rectified and flattened into h_{\mathrm{pre}}(s)\in\mathbb{R}^{2048}, with 2048=32\cdot 8\cdot 8. A single linear layer followed by a ReLU then gives h_{\mathrm{fc}}(s)\in\mathbb{R}^{256}, so d_{\mathrm{pre}}=2048 and d_{\mathrm{fc}}=256. Both sites are therefore taken after a rectification, and the only pre-activation quantity between them is W_{\mathrm{fc}}h_{\mathrm{pre}}(s)+b_{\mathrm{fc}}. The two heads read h_{\mathrm{fc}} linearly. The policy head produces logits \eta(s)=W_{\pi}h_{\mathrm{fc}}(s)+b_{\pi} over the |\mathcal{A}|=15 actions, with \pi(\cdot\mid s)=\operatorname{softmax}\eta(s), and the value head is a single linear unit V(s)=w_{V}^{\top}h_{\mathrm{fc}}(s)+b_{V}.

The two sites sit at opposite ends of this transition. h_{\mathrm{pre}} is the last representation with spatial structure. Any information the policy uses about where the cheese, the coin or the candidate objects are has to pass through it, and it can still be read cell by cell through H^{(3)}_{:xy}(s). h_{\mathrm{fc}} has no spatial layout, but it is exactly what the heads read. As the policy head is linear, a displacement of h_{\mathrm{fc}} reaches the action distribution only through W_{\pi}, and the activation-space Fisher at this site takes the closed form given after Proposition 2. So comparing the two sites separates information that the trunk still carries in spatial form from information that has been passed on to the readout. Training and implementation details are given in Appendix H.

B.2 Theory: Local Policy Geometry

Probing establishes whether information about a goal-related variable is linearly available in a representation. It does not establish whether the trained policy is sensitive to changes in that representation (Elazar et al., 2021). We therefore need a quantity that connects an internal representation change to the action distribution produced downstream.

The construction follows the standard information-geometric interpretation of Fisher information (Amari, 1998; Martens, 2020). Fisher information describes the local second-order geometry of a family of probability distributions. Natural-gradient methods usually apply this geometry with respect to model parameters (Kakade, 2001). Here, the trained network is held fixed and we express the same policy geometry in coordinates of an internal representation.

Definition 3 (Downstream policy).

For representation site m, let

\pi^{(m)}(\cdot\mid h)

denote the action distribution obtained by applying the remainder of the trained network downstream of site m to representation h. For every policy input s,

\pi(\cdot\mid s)=\pi^{(m)}\!\left(\cdot\mid h^{(m)}(s)\right).

The downstream policy does not define a new policy or a new state space. It simply factors the existing computation at an intermediate representation site. The upstream network maps the input s to h^{(m)}(s), and \pi^{(m)} denotes the fixed remainder of the trained network.

This factorization lets us express the response to a world edit either in input space or in representation space. For a base state s and environment edit e, let s_{e} denote the matched edited state and

\Delta_{e}^{(m)}(s)=h^{(m)}(s_{e})-h^{(m)}(s)

the corresponding representation displacement from Definition 2. Writing

h=h^{(m)}(s),

we have

\pi(\cdot\mid s)=\pi^{(m)}(\cdot\mid h),\qquad\pi(\cdot\mid s_{e})=\pi^{(m)}\left(\cdot\mid h+\Delta_{e}^{(m)}(s)\right).

Therefore,

D_{\mathrm{KL}}\left(\pi(\cdot\mid s)\;\middle\|\;\pi(\cdot\mid s_{e})\right)=D_{\mathrm{KL}}\left(\pi^{(m)}(\cdot\mid h)\;\middle\|\;\pi^{(m)}\left(\cdot\mid h+\Delta_{e}^{(m)}(s)\right)\right).

The two expressions describe the same finite policy change. The first compares the policy at the original and edited inputs, while the second exposes the internal representation displacement that connects those endpoints. Fisher geometry becomes useful when we ask how the downstream policy responds locally to that displacement.

Definition 4 (Activation-space Fisher information).

For a fixed representation

h=h^{(m)}(s)\in\mathbb{R}^{d_{m}},

the activation-space Fisher information of the downstream policy is

F^{(m)}(s)=\mathbb{E}_{a\sim\pi^{(m)}(\cdot\mid h)}\left[\nabla_{h}\log\pi^{(m)}(a\mid h)\,\nabla_{h}\log\pi^{(m)}(a\mid h)^{\top}\right]\in\mathbb{R}^{d_{m}\times d_{m}}.

When the representation site and base state are clear from context, we write this matrix as F_{h}.

The matrix F_{h} is positive semidefinite. Importantly, it is not a property of the representation alone. It describes the representation through the particular downstream policy that reads it. Directions with large Fisher sensitivity are directions along which small representation changes strongly alter the policy distribution, while directions with small Fisher sensitivity are comparatively ignored downstream.

The connection to policy change follows from the local expansion of the KL divergence.

Proposition 1 (Local policy geometry).

For a sufficiently small displacement

\delta\in\mathbb{R}^{d_{m}},

D_{\mathrm{KL}}\left(\pi^{(m)}(\cdot\mid h)\;\middle\|\;\pi^{(m)}(\cdot\mid h+\delta)\right)=\frac{1}{2}\delta^{\top}F_{h}\delta+O(\|\delta\|_{2}^{3}).

Proof.

Abbreviate

\pi_{h}=\pi^{(m)}(\cdot\mid h).

Then

D_{\mathrm{KL}}\left(\pi_{h}\;\middle\|\;\pi_{h+\delta}\right)=\mathbb{E}_{a\sim\pi_{h}}\left[\log\pi_{h}(a)-\log\pi_{h+\delta}(a)\right].

Expanding \log\pi_{h+\delta}(a) to second order around h gives

\log\pi_{h+\delta}(a)=\log\pi_{h}(a)+\delta^{\top}\nabla_{h}\log\pi_{h}(a)+\frac{1}{2}\delta^{\top}\nabla_{h}^{2}\log\pi_{h}(a)\delta+O(\|\delta\|_{2}^{3}).

Substituting into the KL divergence yields

D_{\mathrm{KL}}=-\delta^{\top}\mathbb{E}_{a\sim\pi_{h}}\left[\nabla_{h}\log\pi_{h}(a)\right]-\frac{1}{2}\delta^{\top}\mathbb{E}_{a\sim\pi_{h}}\left[\nabla_{h}^{2}\log\pi_{h}(a)\right]\delta+O(\|\delta\|_{2}^{3}).

The first-order term vanishes because the score has zero expectation,

\mathbb{E}_{a\sim\pi_{h}}\left[\nabla_{h}\log\pi_{h}(a)\right]=0.

Under the standard regularity conditions, the information identity gives

-\mathbb{E}_{a\sim\pi_{h}}\left[\nabla_{h}^{2}\log\pi_{h}(a)\right]=\mathbb{E}_{a\sim\pi_{h}}\left[\nabla_{h}\log\pi_{h}(a)\nabla_{h}\log\pi_{h}(a)^{\top}\right]=F_{h}.

Therefore,

D_{\mathrm{KL}}\left(\pi_{h}\;\middle\|\;\pi_{h+\delta}\right)=\frac{1}{2}\delta^{\top}F_{h}\delta+O(\|\delta\|_{2}^{3}).

∎

The absence of a first-order term is important. The KL divergence is locally minimized when the two policy distributions coincide, so its leading nonzero variation is quadratic. The Fisher quadratic

\delta^{\top}F_{h}\delta

therefore measures the local policy sensitivity of a representation displacement.

Applied to the world-edit displacement

\delta=\Delta_{e}^{(m)}(s),

this gives

D_{\mathrm{KL}}\left(\pi^{(m)}(\cdot\mid h^{(m)}(s))\;\middle\|\;\pi^{(m)}\left(\cdot\mid h^{(m)}(s)+\Delta_{e}^{(m)}(s)\right)\right)\approx\frac{1}{2}\Delta_{e}^{(m)}(s)^{\top}F^{(m)}(s)\Delta_{e}^{(m)}(s).

This is the quantity used in the world-edit Fisher analysis. The environment determines the displacement through a controlled edit, while the Fisher geometry determines how strongly the downstream policy responds to that displacement around the original state.

Definition 5 (World-edit Fisher sensitivity).

For a base state s, controlled environment edit e, and representation site m, let

\Delta_{e}^{(m)}(s)=h^{(m)}(s_{e})-h^{(m)}(s)

be the world-edit representation displacement from Definition 2. We define the world-edit Fisher sensitivity

Q_{e}^{(m)}(s)=\Delta_{e}^{(m)}(s)^{\top}F^{(m)}(s)\Delta_{e}^{(m)}(s).

When the edit changes the intended-goal variable, we write

e=\star,\qquad s_{\star},\qquad\Delta_{\star}^{(m)}(s).

When a corresponding proxy edit is well defined, we analogously write

e=p,\qquad s_{p},\qquad\Delta_{p}^{(m)}(s).

Not every environment admits both kinds of edit.

By Proposition 1,

\frac{1}{2}Q_{e}^{(m)}(s)

is the second-order local approximation to the change in the downstream policy induced by moving from the base representation toward its world-edited counterpart:

D_{\mathrm{KL}}\left(\pi^{(m)}(\cdot\mid h^{(m)}(s))\;\middle\|\;\pi^{(m)}\left(\cdot\mid h^{(m)}(s)+\Delta_{e}^{(m)}(s)\right)\right)\approx\frac{1}{2}Q_{e}^{(m)}(s).

Because

h^{(m)}(s)+\Delta_{e}^{(m)}(s)=h^{(m)}(s_{e}),

the finite endpoints are the same as those compared by the behavioral world-edit KL,

D_{\mathrm{KL}}\left(\pi(\cdot\mid s)\;\middle\|\;\pi(\cdot\mid s_{e})\right).

The two quantities differ in what they expose. The behavioral KL measures the complete policy change between the original and edited inputs. Q_{e}^{(m)}(s) instead expresses the corresponding change locally through representation site m, and therefore measures how strongly the downstream policy is coupled to the representation displacement produced by edit e. A large or linearly informative displacement can consequently have small Fisher sensitivity if the trained downstream computation is comparatively insensitive to its direction.

Proposition 2 (Dependence on the trained downstream computation).

Let

\eta^{(m)}(h)\in\mathbb{R}^{|\mathcal{A}|}

denote the logits produced by the downstream network from representation h, and let

J_{\eta}^{(m)}(h)=\frac{\partial\eta^{(m)}(h)}{\partial h}

denote its Jacobian. Then the activation-space Fisher can be written as

F_{h}=J_{\eta}^{(m)}(h)^{\top}\left[\operatorname{diag}(\pi)-\pi\pi^{\top}\right]J_{\eta}^{(m)}(h),

where \pi=\pi^{(m)}(\cdot\mid h).

Proof.

For a softmax distribution, the Fisher information with respect to the logits is

F_{\eta}=\operatorname{diag}(\pi)-\pi\pi^{\top}.

Applying the chain rule to \nabla_{h}\log\pi^{(m)}(a\mid h) pulls this metric back through the downstream logit map, giving

F_{h}=J_{\eta}^{\top}F_{\eta}J_{\eta}.

∎

At the final fully connected representation, the downstream logit map is linear,

\eta^{(\mathrm{fc})}(h)=W_{\pi}h+b_{\pi},

so

F^{(\mathrm{fc})}(s)=W_{\pi}^{\top}\left[\operatorname{diag}(\pi)-\pi\pi^{\top}\right]W_{\pi}.

At earlier representation sites, such as h_{\mathrm{pre}}, the Jacobian additionally contains the downstream transformations between that site and the policy head.

Gaussian policies.

The robot agents are read at the same two sites, with h_{\mathrm{pre}} the rectified, flattened output of the last convolution and h_{\mathrm{fc}} the following linear layer with a ReLU (Appendix B.1). Their policy is a Gaussian over continuous actions,

\pi^{(m)}(\cdot\mid h)=\mathcal{N}\!\big(\mu^{(m)}(h),\,\operatorname{diag}(\sigma^{2})\big),\qquad\sigma=\exp(\lambda),

where \lambda is a learned vector with one entry per action dimension that is not computed from the observation. The softmax closed form of Proposition 2 does not apply, and its place is taken by the following result.

Proposition 3 (Fisher of a Gaussian policy with state-independent noise).

Let \pi^{(m)}(\cdot\mid h)=\mathcal{N}\big(\mu^{(m)}(h),\operatorname{diag}(\sigma^{2})\big) with \sigma independent of h, and let

J_{\mu}^{(m)}(h)=\frac{\partial\mu^{(m)}(h)}{\partial h}

be the Jacobian of the action mean with respect to the representation. Then

F_{h}=J_{\mu}^{(m)\top}\operatorname{diag}(\sigma^{-2})\,J_{\mu}^{(m)},

so the world-edit Fisher sensitivity is

Q_{e}^{(m)}(s)=\big\|\operatorname{diag}(\sigma^{-1})\,J_{\mu}^{(m)}\,\Delta_{e}^{(m)}(s)\big\|_{2}^{2},

and the finite KL between the base and the edited state is exactly

D_{\mathrm{KL}}\big(\pi(\cdot\mid s)\,\big\|\,\pi(\cdot\mid s_{e})\big)=\tfrac{1}{2}\big\|\operatorname{diag}(\sigma^{-1})\big(\mu(s_{e})-\mu(s)\big)\big\|_{2}^{2},

with \mu(s)=\mu^{(m)}\big(h^{(m)}(s)\big).

Proof.

Write \mu=\mu^{(m)}(h) and J_{\mu}=J_{\mu}^{(m)}(h). The log-density is

\log\pi^{(m)}(a\mid h)=-\tfrac{1}{2}\,(a-\mu)^{\top}\operatorname{diag}(\sigma^{-2})\,(a-\mu)+c(\sigma),

where c(\sigma) collects the normalization and does not depend on h. Differentiating with respect to h gives the score

\nabla_{h}\log\pi^{(m)}(a\mid h)=J_{\mu}^{\top}\operatorname{diag}(\sigma^{-2})\,(a-\mu).

Under a\sim\pi^{(m)}(\cdot\mid h) the residual has covariance

\mathbb{E}\big[(a-\mu)(a-\mu)^{\top}\big]=\operatorname{diag}(\sigma^{2}),

so the second moment of the score is

F_{h}=J_{\mu}^{\top}\operatorname{diag}(\sigma^{-2})\,\operatorname{diag}(\sigma^{2})\,\operatorname{diag}(\sigma^{-2})\,J_{\mu}=J_{\mu}^{\top}\operatorname{diag}(\sigma^{-2})\,J_{\mu}.

Inserting \Delta_{e}^{(m)}(s) into \Delta^{\top}F_{h}\Delta gives the expression for Q_{e}^{(m)}(s). For two Gaussians with the same covariance \Sigma the trace and log-determinant terms of the KL cancel, leaving

D_{\mathrm{KL}}\big(\mathcal{N}(\mu_{1},\Sigma)\,\big\|\,\mathcal{N}(\mu_{2},\Sigma)\big)=\tfrac{1}{2}\,(\mu_{2}-\mu_{1})^{\top}\Sigma^{-1}(\mu_{2}-\mu_{1}),

and \Sigma=\operatorname{diag}(\sigma^{2}) gives the last claim. ∎

Equation (3) is the Fisher of the policy itself, not an approximation, and it plays the role of the categorical closed form above. Because the noise does not depend on the input, the KL between two states is exactly quadratic in the change of the mean, and the quadratic approximation differs from it only through the curvature of the mean along the displacement,

\displaystyle\tfrac{1}{2}Q_{e}^{(m)}(s) \\ \displaystyle=\tfrac{1}{2}\big\|\operatorname{diag}(\sigma^{-1})\,J_{\mu}^{(m)}\Delta_{e}^{(m)}(s)\big\|_{2}^{2}, \\ \displaystyle\mu(s_{e})-\mu(s) \\ \displaystyle=J_{\mu}^{(m)}\Delta_{e}^{(m)}(s)+O\big(\|\Delta_{e}^{(m)}(s)\|_{2}^{2}\big).

For a Gaussian policy the ratio of the exact KL to \tfrac{1}{2}Q can therefore depart from 1 only where \mu^{(m)} bends along the edit, never through the shape of the action distribution.

In our robot agents the mean comes from a head with one hidden layer of ReLU units, which reads h_{\mathrm{fc}} together with an embedding h_{\mathrm{state}} of the robot’s body state (Appendix B.1),

\mu=W_{2}\,\mathrm{ReLU}(z)+b_{2},\qquad z=W_{1}\big[h_{\mathrm{fc}};h_{\mathrm{state}}\big]+b_{1}.

Write W_{1}^{\mathrm{fc}} for the columns of W_{1} that read h_{\mathrm{fc}}. With h_{\mathrm{state}} held at its base value, the Jacobian of the mean at the two sites is

\displaystyle J_{\mu}^{(\mathrm{fc})} \\ \displaystyle=W_{2}\operatorname{diag}\big(\mathbf{1}\{z>0\}\big)\,W_{1}^{\mathrm{fc}}, \\ \displaystyle J_{\mu}^{(\mathrm{pre})} \\ \displaystyle=J_{\mu}^{(\mathrm{fc})}\operatorname{diag}\big(\mathbf{1}\{W_{\mathrm{fc}}h_{\mathrm{pre}}+b_{\mathrm{fc}}>0\}\big)\,W_{\mathrm{fc}},

with the indicator taken entrywise. Both depend on the state only through the two ReLU patterns. An edit that leaves the body state unchanged leaves h_{\mathrm{state}} unchanged as well, so its displacement lies in the image branch alone. The factorization is specific to this head, whereas Equation (3) holds for any mean network as long as \sigma is state independent.

Proposition 4 (Fisher sensitivity at h_{\mathrm{pre}} and h_{\mathrm{fc}}).

Let

J=\frac{\partial h_{\mathrm{fc}}}{\partial h_{\mathrm{pre}}}=\operatorname{diag}\big(\mathbf{1}\{W_{\mathrm{fc}}h_{\mathrm{pre}}+b_{\mathrm{fc}}>0\}\big)\,W_{\mathrm{fc}}.

The policy depends on h_{\mathrm{pre}} only through h_{\mathrm{fc}}, so the Fisher pulls back exactly,

F_{\mathrm{pre}}=J^{\top}F_{\mathrm{fc}}\,J.

If the displacements of an edit satisfy \Delta_{\mathrm{fc}}=J\Delta_{\mathrm{pre}}, then Q_{\mathrm{pre}}=Q_{\mathrm{fc}}.

Proof.

By the chain rule \nabla_{h_{\mathrm{pre}}}\log\pi=J^{\top}\nabla_{h_{\mathrm{fc}}}\log\pi, and taking the second moment of both sides gives F_{\mathrm{pre}}=J^{\top}F_{\mathrm{fc}}J. Then Q_{\mathrm{pre}}=(J\Delta_{\mathrm{pre}})^{\top}F_{\mathrm{fc}}(J\Delta_{\mathrm{pre}})=\Delta_{\mathrm{fc}}^{\top}F_{\mathrm{fc}}\Delta_{\mathrm{fc}}=Q_{\mathrm{fc}}. ∎

To first order the edit-induced displacements satisfy \Delta_{\mathrm{fc}}\approx J\Delta_{\mathrm{pre}}, and therefore

Q_{\mathrm{pre}}=\Delta_{\mathrm{pre}}^{\top}J^{\top}F_{\mathrm{fc}}J\,\Delta_{\mathrm{pre}}\approx\Delta_{\mathrm{fc}}^{\top}F_{\mathrm{fc}}\,\Delta_{\mathrm{fc}}=Q_{\mathrm{fc}}.

Q is thus approximately the same at both sites by construction and not as a property of the data, and the remaining differences come from the nonlinearity of the fc layer over a finite edit. The argument holds for every network in which h_{\mathrm{fc}} is computed from h_{\mathrm{pre}} alone, which includes the robot agents, since the body state joins only after h_{\mathrm{fc}}.

Proposition 2 makes the interpretation of Q precise. Fisher sensitivity is a property of the representation as read by the trained downstream policy. Two agents can make the same information available in their internal representations while inducing very different Fisher geometries because their downstream computations use that information differently. A small Q_{e}^{(m)} therefore does not imply that information associated with edit e is absent from the representation.

Separating displacement from directional sensitivity.

The quadratic form Q combines two effects. A world edit can produce a large policy response because it moves the representation far, because the downstream policy is unusually sensitive to the direction of that movement, or both. We separate these effects relative to the policy’s average local Fisher sensitivity.

Definition 6 (Fisher sensitivity decomposition).

For a nonzero displacement \Delta\in\mathbb{R}^{d}, define its directional Fisher sensitivity

S(\Delta)=\frac{\Delta^{\top}F_{h}\Delta}{\|\Delta\|_{2}^{2}}=\widehat{\Delta}^{\top}F_{h}\widehat{\Delta},\qquad\widehat{\Delta}=\frac{\Delta}{\|\Delta\|_{2}}.

For an isotropically sampled unit vector u\sim\mathrm{Unif}(\mathbb{S}^{d-1}),

\mathbb{E}_{u}\left[u^{\top}F_{h}u\right]=\frac{\operatorname{tr}(F_{h})}{d}.

We therefore define the relative directional gain

G(\Delta)=\frac{S(\Delta)}{\operatorname{tr}(F_{h})/d}

The relative gain G is defined whenever \operatorname{tr}(F_{h})>0. If the local Fisher vanishes identically, then Q=0 for every direction.

We also define the displacement in Fisher units

D_{F}(\Delta)=\|\Delta\|_{2}^{2}\frac{\operatorname{tr}(F_{h})}{d}.

The two quantities factor the Fisher sensitivity exactly:

Q=\underbrace{G}_{\text{relative directional gain}}\;\underbrace{D_{F}}_{\text{displacement in Fisher units}}.

D_{F} measures how far the counterfactual moves the representation in units of the policy’s average local sensitivity. G measures whether the direction of that movement is more or less policy-sensitive than an average direction in the same representation space. Thus

G>1

means that the counterfactual moves along a direction that is privileged by the downstream policy relative to an isotropic direction. In the experiments, we additionally compare this directional gain with matched-norm random directions to account for the non-isotropic spectrum of F_{h}.

Proposition 5 (Rescaling invariance).

Consider a scalar reparameterization

h^{\prime}=ch

together with a compensating downstream transformation that leaves the represented policy unchanged. Then

Q^{\prime}=Q,\qquad G^{\prime}=G,\qquad D_{F}^{\prime}=D_{F},

whereas \|\Delta\|_{2} and S individually are not invariant.

Proof.

Under the reparameterization,

\Delta^{\prime}=c\Delta.

Because the downstream computation is compensated to preserve the policy, gradients with respect to the new representation scale as c^{-1}, giving

F_{h^{\prime}}=c^{-2}F_{h}.

Therefore,

Q^{\prime}=(c\Delta)^{\top}(c^{-2}F_{h})(c\Delta)=Q.

Furthermore,

S^{\prime}=c^{-2}S,\qquad\frac{\operatorname{tr}(F_{h^{\prime}})}{d}=c^{-2}\frac{\operatorname{tr}(F_{h})}{d},

so

G^{\prime}=G.

Finally,

D_{F}^{\prime}=\|c\Delta\|_{2}^{2}\frac{\operatorname{tr}(c^{-2}F_{h})}{d}=D_{F}.

∎

This invariance matters when comparing different trained networks. The scale of an internal representation is not itself identifiable, so neither the raw displacement norm nor its unnormalized directional sensitivity should be interpreted across agents in isolation. Q, G, and D_{F} remain unchanged under this elementary rescaling.

A change of site is not such a rescaling. G and D_{F} depend on \|\Delta\|_{2} and on \operatorname{tr}(F_{h})/d, which change with the coordinates and the dimension of the site, so unlike Q (Proposition 4) they are not approximately preserved from h_{\mathrm{pre}} to h_{\mathrm{fc}}. A change in the split between G and D_{F} from one site to the other therefore describes how the fc layer re-expresses the same policy sensitivity, not where goal information is lost, and we compare G and D_{F} only between cohorts at the same site.

Corollary 1 (Dependence on policy sharpness).

As the downstream action distribution approaches a point mass,

\pi\rightarrow e_{a},

we have

\operatorname{diag}(\pi)-\pi\pi^{\top}\rightarrow 0.

Consequently,

F_{h}\rightarrow 0

and

Q(\Delta)\rightarrow 0

for every fixed displacement \Delta.

Thus a more deterministic policy can exhibit smaller Fisher sensitivity along every representation direction, independently of the particular feature being studied. We therefore report the average Fisher sensitivity

\frac{\operatorname{tr}(F_{h})}{d}

per cohort next to D_{F} (Table 9) when comparing policies with different levels of sharpness.

For a Gaussian policy sharpness acts the other way. By Proposition 3 the Fisher grows as \sigma^{-2}, so a sharper policy reads every direction more strongly and a smaller \sigma raises \operatorname{tr}(F_{h})/d rather than lowering it.

Definition 7 (Fisher cosine).

For two directions \Delta_{1} and \Delta_{2} satisfying

\Delta_{i}^{\top}F_{h}\Delta_{i}>0,\qquad i\in\{1,2\}.

evaluated at the same base representation h under the same local Fisher metric F_{h}, we define

\cos_{F}(\Delta_{1},\Delta_{2})=\frac{\Delta_{1}^{\top}F_{h}\Delta_{2}}{\sqrt{(\Delta_{1}^{\top}F_{h}\Delta_{1})(\Delta_{2}^{\top}F_{h}\Delta_{2})}}.

The Fisher cosine measures alignment specifically in directions to which the downstream policy is locally sensitive. Values near +1 indicate aligned policy-relevant effects, values near 0 indicate local Fisher orthogonality, and values near -1 indicate opposing policy-relevant effects. We use this quantity later to characterize the geometry of cue competition in Attribute Maze.

Low-rank Fisher audit.

Because the activation-space Fisher matrix has rank at most |\mathcal{A}|-1=14 in this environment, we additionally test whether low Fisher rank alone can induce extreme pairwise cosine values. Matched random 256-dimensional vectors evaluated under rank-14 random Fisher metrics produce essentially no pairs with |\cos_{F}|>0.9, whereas 82–86\% of real state pairs exceed this threshold across the three GM policies on the black background. The near-\pm 1 Fisher structure therefore cannot be explained by Fisher rank alone.

Definition 8 (Signed target-selection derivative).

Let

A_{\star}(s),A_{p}(s)\subseteq\mathcal{A}

denote the intended-goal- and proxy-consistent action sets for state s. In Maze and Attribute Maze, both sets are taken from the base state: A_{\star}(s) is the first step toward the base cheese (respectively the line candidate) and A_{p}(s) the first step toward the top-right corner (respectively the yellow candidate). In Coin, the base state holds the coin at a cell right of the agent, on the shortcut side, where the two sets would coincide, and the edit moves it to the cell one step left of the agent. So A_{\star}(s)=A_{\mathrm{left}}, the leftward actions toward the relocated coin, and A_{p}(s)=A_{\mathrm{right}}, the rightward shortcut. At representation site m, define the target-selection contrast

C_{s}^{(m)}(h)=\sum_{a\in A_{\star}(s)}\pi^{(m)}(a\mid h)-\sum_{a\in A_{p}(s)}\pi^{(m)}(a\mid h).

For world edit e with displacement \Delta_{e}^{(m)}(s), the signed target-selection derivative is

\partial_{\Delta_{e}}C_{s}^{(m)}=\nabla_{h}C_{s}^{(m)}\left(h^{(m)}(s)\right)^{\top}\Delta_{e}^{(m)}(s).

The action sets are determined from the environment state s and are held fixed when differentiating with respect to h. For a nonzero displacement, we additionally define the displacement-normalized derivative

\widetilde{\partial_{\Delta_{e}}C_{s}^{(m)}}=\frac{\partial_{\Delta_{e}}C_{s}^{(m)}}{\|\Delta_{e}^{(m)}(s)\|_{2}}.

The raw derivative measures the first-order target-selective effect of the actual representation change induced by edit e. Its sign indicates which side of the intended-versus-proxy contrast is favored, while its magnitude combines direction and edit size. The normalized derivative removes the Euclidean magnitude of the displacement and therefore measures target-selective sensitivity per unit representation change.

Unlike the Fisher quadratic Q, the signed derivative is first-order and directional. Q asks how strongly the downstream action distribution changes along a world-induced representation direction. \partial_{\Delta_{e}}C_{s}^{(m)} asks whether that same direction moves the behaviorally relevant target-selection contrast toward the intended or proxy-consistent side. So, the two measurements are complementary: Q measures whether the policy is sensitive to the world-induced displacement at all, while the signed derivative asks whether that sensitivity is organized along the behaviorally relevant target-selection contrast.

Input saliency.

The same Fisher can be taken at the input instead of an internal site, which shows which parts of the observation the action distribution depends on. Write o for the observation, with o_{c,x} the value of colour channel c at pixel x, and treat o as a site, so that F^{(\mathrm{in})}(o) is the activation-space Fisher of Definition 4 with h=o.

Definition 9 (Policy-Fisher input saliency and enrichment).

The policy-Fisher input saliency of pixel x is the diagonal of the input Fisher summed over the colour channels of x,

S(x)=\sum_{c}F^{(\mathrm{in})}_{(c,x),(c,x)}(o)=\sum_{c}\mathbb{E}_{a\sim\pi(\cdot\mid o)}\left[\left(\frac{\partial\log\pi(a\mid o)}{\partial o_{c,x}}\right)^{2}\right].

For a region R of |R| pixels in an image of N pixels, the enrichment of R is the saliency per pixel in R relative to the saliency per pixel of the whole image,

E_{R}(o)=\frac{N}{|R|}\,\frac{\sum_{x\in R}S(x)}{\sum_{x}S(x)},

and over states o_{1},\ldots,o_{n} we report its geometric mean

E_{R}=\exp\!\Big(\frac{1}{n}\sum_{i=1}^{n}\log E_{R}(o_{i})\Big).

By Proposition 1 at the input, changing channel c of pixel x by a small \varepsilon moves the policy by \tfrac{1}{2}\varepsilon^{2}F^{(\mathrm{in})}_{(c,x),(c,x)}+O(\varepsilon^{3}) in KL, so \tfrac{1}{2}\varepsilon^{2}S(x) is this local KL summed over the channels of x. For a softmax policy with logits z,

\frac{\partial\log\pi(a\mid o)}{\partial o_{c,x}}=\frac{\partial z_{a}}{\partial o_{c,x}}-\sum_{b}\pi(b\mid o)\,\frac{\partial z_{b}}{\partial o_{c,x}},

and for a Gaussian policy Proposition 3 applies at the input with the body state held fixed, so S has a closed form for both policy classes,

S_{\mathrm{softmax}}(x)=\sum_{c}\operatorname{Var}_{a\sim\pi(\cdot\mid o)}\left[\frac{\partial z_{a}}{\partial o_{c,x}}\right],\qquad S_{\mathrm{Gauss}}(x)=\sum_{c}\sum_{k}\frac{1}{\sigma_{k}^{2}}\left(\frac{\partial\mu_{k}}{\partial o_{c,x}}\right)^{2}.

The softmax saliency shrinks everywhere as the policy becomes nearly deterministic, and the Gaussian saliency grows everywhere as \sigma shrinks (Corollary 1 and the remark after it). Enrichment divides by the image mean of the same state and removes this global scale, so \log_{10}E_{R}=0 means that the region counts as much as an average pixel, whatever the sharpness of the policy.

How S(x) differs from other saliency.

Gradient saliency (Simonyan et al., 2014) differentiates a single output, one action logit or the value, and perturbation or occlusion saliency measures how that output changes when a patch of the image is replaced. S(x) is a Fisher-weighted variant of gradient saliency. It concerns the whole action distribution rather than one logit, it is invariant to a shift of all logits by a common amount, and it weights each action by its probability, so it measures what the policy actually reads out. High-contrast sprites draw saliency in any network, so we read every region against randomly initialized networks, and since saliency decays with distance from the agent we compare cohorts at a fixed region. Saliency maps suggest hypotheses rather than explain behavior (Atrey et al., 2020). Our causal claims rest only on the world edits and the rollouts, never on the maps.

B.3 Projections and Vector Analysis

The constructions below define projections relative to a chosen reference direction u^{(m)}. They do not assume that one fixed reference direction provides a global description of the representation geometry across states.

Euclidean projection.

Let \Delta^{(m)} denote a representation displacement at site m, and let u^{(m)}\neq 0 be a reference direction in the same representation space. We write

\hat{u}^{(m)}=\frac{u^{(m)}}{\|u^{(m)}\|_{2}}.

The Euclidean projection of \Delta^{(m)} onto the span of u^{(m)} is

\Delta_{u,E}^{(m)}=\left({\hat{u}^{(m)}}{}^{\top}\Delta^{(m)}\right)\hat{u}^{(m)},

with residual

\Delta_{\perp,E}^{(m)}=\Delta^{(m)}-\Delta_{u,E}^{(m)}.

By construction,

{\Delta_{u,E}^{(m)}}^{\top}\Delta_{\perp,E}^{(m)}=0.

This decomposition measures how the representation displacement is oriented in the ordinary Euclidean geometry of activation space.

Although the Euclidean components are orthogonal in activation space, they are not generally orthogonal under the Fisher metric. Consequently,

Q^{(m)}\!\left(\Delta^{(m)}\right)=Q^{(m)}\!\left(\Delta_{u,E}^{(m)}\right)+Q^{(m)}\!\left(\Delta_{\perp,E}^{(m)}\right)+2{\Delta_{u,E}^{(m)}}^{\top}F^{(m)}(s)\Delta_{\perp,E}^{(m)}.

The cross term need not vanish because Euclidean orthogonality does not imply Fisher orthogonality.

Fisher-orthogonal projection.

To obtain an additive decomposition of the Fisher quadratic, we instead define the projection using the local Fisher geometry. The downstream policy induces this local geometry through the activation-space Fisher matrix F^{(m)}(s). When

{u^{(m)}}^{\top}F^{(m)}(s)u^{(m)}>0,

we define the Fisher-orthogonal projection of \Delta^{(m)} onto the span of u^{(m)} as

\Delta_{u,F}^{(m)}=\frac{{u^{(m)}}^{\top}F^{(m)}(s)\Delta^{(m)}}{{u^{(m)}}^{\top}F^{(m)}(s)u^{(m)}}u^{(m)},

with residual

\Delta_{\perp,F}^{(m)}=\Delta^{(m)}-\Delta_{u,F}^{(m)}.

The residual is orthogonal to the reference direction under the local Fisher metric,

{u^{(m)}}^{\top}F^{(m)}(s)\Delta_{\perp,F}^{(m)}=0.

Consequently, the Fisher quadratic decomposes exactly,

Q^{(m)}\left(\Delta^{(m)}\right)=Q^{(m)}\left(\Delta_{u,F}^{(m)}\right)+Q^{(m)}\left(\Delta_{\perp,F}^{(m)}\right),

where, for any representation-space direction v,

Q^{(m)}(v)=v^{\top}F^{(m)}(s)v.

The Euclidean and Fisher projections answer different questions. The Euclidean projection asks how much of the displacement lies along the reference direction in activation space. The Fisher projection instead asks how the policy-sensitive component of that displacement is distributed relative to the same direction.

Signed-effect decomposition.

At a fixed base state, the local signed target-selection derivative is linear in the representation displacement. For the target-selection contrast C_{s}^{(m)}(h),

\partial_{\Delta}C_{s}^{(m)}=\nabla_{h}C_{s}^{(m)}\left(h^{(m)}(s)\right)^{\top}\Delta^{(m)}.

Therefore, for any additive decomposition

\Delta^{(m)}=\Delta_{1}^{(m)}+\Delta_{2}^{(m)},

we have

\partial_{\Delta}C_{s}^{(m)}=\partial_{\Delta_{1}}C_{s}^{(m)}+\partial_{\Delta_{2}}C_{s}^{(m)}.

This exact additivity applies to the local directional derivative. It does not imply that the finite change

C_{s}^{(m)}\!\left(h^{(m)}(s)+\Delta^{(m)}\right)-C_{s}^{(m)}\!\left(h^{(m)}(s)\right)

decomposes additively for arbitrary finite displacements. Unlike the quadratic Fisher decomposition, the derivative decomposition follows directly from linearity and therefore does not depend on how the additive split was constructed.

Local and shared-axis descriptions.

The projection constructions above can be used to compare a fully state-specific representation direction with a lower-dimensional shared description.

Let d(s) denote a state-dependent representation direction of interest. A fully local description retains this vector separately for every state. A shared-axis description instead asks whether the state dependence can be compressed to a fixed reference direction u together with state-dependent coefficients.

The simplest such approximation is

d(s)\approx\alpha(s)u,

where \alpha(s) is a scalar that may vary across states. If the orientation of the shared direction is itself determined by a discrete state variable, we write

d(s)\approx\sigma(s)\beta(s)u,\qquad\sigma(s)\in\{-1,+1\},\qquad\beta(s)\geq 0.

Here, \sigma(s) determines which orientation of the shared axis is active, while \beta(s) controls the local magnitude.

This form does not imply that the full representation geometry is one-dimensional. It only states that the component relevant to the particular displacement family under study may be well described by one shared axis after accounting for state-dependent sign and magnitude. The residual

r(s)=d(s)-\sigma(s)\beta(s)u

contains the remaining state-specific structure.

Shared-axis coefficients under Euclidean and Fisher geometry.

For a fixed shared direction u, the state-dependent coefficient can be defined under different geometries.

In Euclidean activation space, the signed coordinate of d(s) along the unit direction

\hat{u}=\frac{u}{\|u\|_{2}}

is

\alpha_{E}(s)=\hat{u}^{\top}d(s).

The corresponding Euclidean projection is

d_{u,E}(s)=\alpha_{E}(s)\hat{u}.

Under the local Fisher geometry, the corresponding coefficient is

\alpha_{F}(s)=\frac{u^{\top}F(s)d(s)}{u^{\top}F(s)u},

giving the Fisher-orthogonal projection

d_{u,F}(s)=\alpha_{F}(s)u.

The two coefficients answer different questions. \alpha_{E}(s) measures how much of the displacement lies along u in ordinary activation space. \alpha_{F}(s) measures the coefficient of the same shared direction after orthogonality is defined by local policy sensitivity.

The resulting residual

d_{\perp,F}(s)=d(s)-d_{u,F}(s)

satisfies

u^{\top}F(s)d_{\perp,F}(s)=0,

and therefore

Q(d(s))=Q(d_{u,F}(s))+Q(d_{\perp,F}(s)).

Interventions derived from these decompositions.

The same distinction gives several intervention families.

A state-local intervention retains the complete direction,

\delta_{\mathrm{local}}(s;\beta)=\beta\,d(s).

A shared-axis intervention retains only a fixed direction and a state-dependent sign and magnitude,

\delta_{\mathrm{shared}}(s;\beta)=\beta\,\sigma(s)\,m(s)\,\hat{u},

where m(s)\geq 0 specifies the chosen local magnitude.

Removing the sign dependence gives the corresponding unsigned intervention,

\delta_{\mathrm{unsigned}}(s;\beta)=\beta\,m(s)\,\hat{u}.

Finally, using the Fisher projection gives

\delta_{F}(s;\beta)=\beta\,\alpha_{F}(s)\,u.

These interventions separate different compression hypotheses. The local intervention retains the full state-dependent representation geometry. The shared-axis intervention then asks whether its causal effect can be reproduced using one common direction together with state-dependent gating. The unsigned intervention tests whether that gating is necessary. Finally, the Fisher-projected intervention retains only the component assigned to the shared direction by the local policy-sensitive geometry.

B.4 Environment-specific target geometry

On the training distribution the intended goal and the proxy are confounded. At most states one and the same action moves the agent toward both, so the policy’s choice cannot tell us which of the two it is following. Behavior separates goal from proxy only at states where the two require different first actions, and finding these states requires knowing exactly which actions make progress toward each target. Walls make the straight-line bearing of a target a poor guide to this, so we define progress on the level’s grid graph. The resulting graph-exact action sets are used throughout the paper, both to choose evaluation states (Section E) and to score behavior on them.

This needs a local notion of progress toward a spatial target. The MDP state s\in\mathcal{S} describes the complete environment configuration, while navigation depends on the agent’s location within the spatial structure of the current level. We therefore introduce a spatial projection of the environment state.

Spatial state projection.

For a fixed level \xi, let

\mathcal{G}_{\xi}=(V_{\xi},E_{\xi})

denote its traversable grid graph. Each vertex v\in V_{\xi} corresponds to a traversable cell, and an edge (u,v)\in E_{\xi} indicates that the agent can move between the corresponding cells in one environment step.

We write

x_{\xi}:\mathcal{S}\rightarrow V_{\xi}

for the spatial projection that maps a full environment state s to the cell occupied by the agent. Thus s retains the complete MDP state, while x_{\xi}(s) extracts only the agent’s spatial location on level \xi.

For cells u,v\in V_{\xi}, we define the graph distance

d_{\xi}(u,v)=\min\left\{k\in\mathbb{N}_{0}:\exists\,v_{0},\ldots,v_{k}\in V_{\xi},\;v_{0}=u,\;v_{k}=v,\;(v_{i-1},v_{i})\in E_{\xi}\ \forall i=1,\ldots,k\right\}.

Hence d_{\xi}(u,v) is the minimum number of valid grid transitions required to move from u to v. We compute this distance exactly using breadth-first search.

Spatial state-transition projection.

The MDP transition kernel acts on complete environment states,

s^{\prime}\sim P(\cdot\mid s,a).

For the navigation environments considered here, the one-step spatial transition is deterministic conditional on the full state s and action a. We therefore define

x^{\prime}_{\xi}(s,a)=x_{\xi}(s^{\prime}),\qquad\text{where }P(s^{\prime}\mid s,a)=1,

as the agent cell after applying action a in state s on level \xi. Thus x_{\xi}(s) and x^{\prime}_{\xi}(s,a) are respectively the spatial projections before and after the MDP transition. This notation introduces no additional transition model beyond P.

Definition 10 (Graph-exact target-progress action set).

For a state s, level \xi, and candidate target cell v\in V_{\xi}, define

A_{v}(s)=\left\{a\in\mathcal{A}:d_{\xi}\!\left(x^{\prime}_{\xi}(s,a),v\right)<d_{\xi}\!\left(x_{\xi}(s),v\right)\right\}\subseteq\mathcal{A}.

A_{v}(s) is therefore the subset of actions whose induced spatial transition strictly decreases the remaining shortest-path distance to v.

Because the grid edges have unit cost, an action belongs to A_{v}(s) exactly when its induced movement lies on a shortest path from the current agent cell toward v. An action from which v remains eventually reachable need not belong to A_{v}(s) if it does not decrease the remaining shortest-path distance on that step.

When multiple raw policy actions induce the same effective movement, we aggregate their probability mass rather than treating action aliases as distinct navigation choices.

Two candidate cells v_{1},v_{2}\in V_{\xi} are locally action-discriminating at state s when

A_{v_{1}}(s)\neq A_{v_{2}}(s).

At such states, the two spatial target hypotheses make different predictions about immediate control and can therefore be distinguished from the native policy’s action distribution.

Environment-specific target action sets.

In Maze, let v_{\star} denote the cheese cell and v_{p} the proxy-associated top-right destination. We write

A_{\star}(s)\equiv A_{v_{\star}}(s),\qquad A_{p}(s)\equiv A_{v_{p}}(s).

Thus A_{\star}(s) contains graph-exact progress actions toward the intended goal location, while A_{p}(s) contains progress actions toward the proxy-associated destination.

In Attribute Maze, let v_{\mathsf{line}} and v_{\mathsf{yellow}} denote the locations of the line-bearing and yellow-bearing objects. Depending on the world edit under consideration, these locations instantiate the intended and proxy candidates. We again use

A_{\star}(s)\qquad\text{and}\qquad A_{p}(s)

as shorthand for the corresponding graph-exact action sets, while retaining A_{v_{\mathsf{line}}}(s) and A_{v_{\mathsf{yellow}}}(s) when the physical cue identity matters.

Coin does not use the grid-graph construction for its proxy. The learned shortcut is represented by the right-moving action family, and in every state of the Coin conflict bank the coin sits one cell to the left of the agent in the state on which target selection is read (Appendix E). Progress toward the coin is therefore leftward motion, and we use the two horizontal movement families

A_{\mathrm{left}},\;A_{\mathrm{right}}\subseteq\mathcal{A}_{\mathrm{Coin}},

each of which holds the three actions that move the agent in that direction. For notational consistency in comparisons across environments, we write

A_{\star}(s)\equiv A_{\mathrm{left}},\qquad A_{p}(s)\equiv A_{\mathrm{right}}.

A state is locally action-discriminating whenever

A_{\star}(s)\neq A_{p}(s).

Action-discriminating evaluation states.

For a level \xi, let \mathcal{S}_{\xi}\subseteq\mathcal{S} denote the environment states belonging to that level. For two candidate action sets A_{1}(s) and A_{2}(s), we call a state action-discriminating when

A_{1}(s)\neq A_{2}(s).

For the intended and proxy-associated targets, we therefore define

\mathcal{S}_{\mathrm{conf}}(\xi)=\left\{s\in\mathcal{S}_{\xi}:A_{\star}(s)\neq A_{p}(s)\right\}.

These states are decision points at which progress toward the intended and proxy-associated targets implies different immediate action sets. Restricting evaluation to \mathcal{S}_{\mathrm{conf}}(\xi) therefore lets the observed policy action distinguish which target better organizes local behavior. We call the members of \mathcal{S}_{\mathrm{conf}}(\xi) conflict states.

Strict conflict.

A conflict state can still have progress sets that overlap, and an action in the overlap serves both targets and so reveals nothing about which one the policy follows. We therefore build every evaluation bank from states in strict conflict,

A_{\star}(s)\cap A_{p}(s)=\emptyset,

at which every progress action serves exactly one of the two targets. In Maze shortest paths are unique, so both sets contain a single action and every conflict state is a strict conflict. In Coin the left-moving and right-moving families are disjoint by construction. In Attribute Maze we additionally require the two candidates to be equidistant from the agent, so that neither target is favoured by being closer.

Continuous actions.

The robot policies are Gaussian over actions a\in\mathbb{R}^{m}, \pi(\cdot\mid s)=\mathcal{N}\big(\mu(s),\operatorname{diag}(\sigma^{2})\big), with a state-independent standard deviation \sigma\in\mathbb{R}^{m}_{>0}. The first k components of an action move the robot’s effector, the gripper or the body, in the plane in which the task’s objects lie.

Definition 11 (Planned step and target direction).

Let M(s)\in\mathbb{R}^{2\times k} map the first k action components at state s to the motion of the effector in the task plane. The planned step is the motion commanded by the mean action,

\mathrm{step}(s)=M(s)\,\mu_{1:k}(s)\in\mathbb{R}^{2}.

For an object o with target point y_{o}(s) in the task plane and the effector at x(s), the target direction is the unit vector \hat{t}_{o}(s)=\big(y_{o}(s)-x(s)\big)/\|y_{o}(s)-x(s)\|_{2}.

The planned step is taken on the unclipped mean, so it can exceed what the robot executes in one control step. The three kinds of robot task instantiate M(s) and y_{o}(s) as follows.

Quadruped.

In WalkToFlag the action is a walking command with m=3 components, a forward and a sideways velocity in the robot’s own frame and a turn rate. The first k=2 components already describe the motion in the floor plane, so M(s)=I_{2} and the planned step is (\mu_{1}(s),\mu_{2}(s)), in command units between -1 and 1. The target point of the flag or the tower is the object’s position in the robot frame.

Arm.

In the arm tasks the action has m=8 components. The first k=7 change the seven joint angles of the arm by at most 0.1 rad each, and the eighth opens or closes the gripper. How far the gripper moves when the joints turn depends on the arm’s posture. The Jacobian J(q)\in\mathbb{R}^{3\times 7} at joint angles q gives, for each joint, the velocity of the tool centre point per unit change of that joint. The tool centre point is the point between the two gripper fingers, the point that grasps or pushes and that the arm’s controller positions. With P\in\mathbb{R}^{2\times 3} keeping the two components in the table plane,

M(s)=0.1\,P\,J(q),

so the planned step is the table-plane motion of the tool centre point that the mean joint change produces, in millimetres. In LocationReach and AttributeTable the target point of an object is its centre.

Push tasks.

In PushToMarker and SortZones the gripper does not move toward the marker or pad g but toward the place from which a push carries the cube onto it. With the cube centre at c, the target point of g is the push point

y_{g}=c-0.03\,\frac{g-c}{\|g-c\|_{2}},

on the line from g through the cube, 3 cm behind the cube centre. The cube’s half-width is 2 cm, so this point lies 1 cm behind its rear face, where the gripper must stand to push the cube straight toward g. The dense reward of these tasks uses the same point (Appendix A.2).

Appendix C Experimental design and extended results

This appendix extends every analysis of Section 3 to all eight environments of Appendix A. Figure 13 summarizes the result. In each environment the true goal survives to a different depth of the agent, and this depth goes together with the intervention that recovers it. The failures fall into the three types of the main text. Target substitution appears in three environments, Maze, LocationReach and PushToMarker, where a location-based proxy dominates and fully substitutes the target, even though the goal object is present in the scene. In Maze the substitute target is a spatial relation, the top-right extreme, and in LocationReach and PushToMarker a fixed place in the camera image. The policy shortcut appears in Coin, where a very simple part of the policy, moving right, stands in for the goal object. Cue competition appears, in different grades, in four environments. In Attribute Maze, AttributeTable and SortZones it coincides with one object that carries two co-located characteristics, and WalkToFlag replicates it with two different objects that stand closely together in space. In all four we see a competition between the two characteristics, or between the two closely related objects.

To establish this, we walk through the agents with a ladder of questions at increasing depth. We start in the representation and ask whether information about the goal is still there, move to the readout and ask whether the policy reacts to it and uses it, and then look at what organizes behavior instead and how the proxy is incorporated into the agent, from the outside and from the inside. The ladder ends with recovery, where we ask whether goal-directed behavior can be restored without further reinforcement learning, by steering the existing policy or by learning a new readout from an earlier representation.

C.1 Linear decodability: Is information about the intended goal still there?

Motivation.

Goal misgeneralization tells us that an agent behaves according to something other than the intended goal, but this does not tell us what happened to information about the intended goal inside the agent. One simple possibility is that the relevant goal information is no longer available in the learned representation. We therefore begin with the most basic internal question: is information about the intended goal still there?

We test this first through linear decodability. A successful probe gives an existence result: the corresponding information is available in a form accessible to a linear readout. We then ask where this information appears across the network, whether its accessibility differs between GM and Non-GM agents, and how much of that accessibility can be attributed to training rather than to structure already exposed by the input or architecture.

Design.

For each environment, we define goal-related and, where meaningful, proxy-related target variables and fit regularized linear probes (Alain & Bengio, 2017) to internal activations. The probe targets are environment-specific because the information relevant to the goal differs across tasks. For a binary target y, the probe at representation m has score

f_{m}(s)=w_{m}^{\top}h^{(m)}(s)+b_{m},

and we evaluate f_{m} using held-out ROC–AUC. For continuous spatial targets, we fit the corresponding linear regressor

\hat{y}_{m}(s)=w_{m}^{\top}h^{(m)}(s)+b_{m}

and evaluate it using held-out R^{2}. Tables 22 and 23 list every probe with its label, bank, split and sites.

Following the architectures in Appendix B.1, we probe the same computational sites across the three Procgen environments,

h^{(1)}=H^{(1)},\qquad h^{(2)}=H^{(2)},\qquad h^{(3)}=H^{(3)},

where H^{(1)},H^{(2)},H^{(3)} denote the corresponding residual-block representations. No additional ReLU is inserted after the first two blocks because no such operation occurs in the policy forward pass. The subsequent shared representations are

h_{\mathrm{pre}}=\operatorname{vec}\left(\operatorname{ReLU}(h^{(3)})\right)

and

h_{\mathrm{fc}}=\operatorname{ReLU}\left(W_{\mathrm{fc}}h_{\mathrm{pre}}+b_{\mathrm{fc}}\right).

Thus h_{\mathrm{pre}} is the actual input to the fully connected layer and h_{\mathrm{fc}} is the final shared representation consumed by the policy and value readouts. For h^{(1)},h^{(2)},h^{(3)}, we globally average pool the convolutional feature maps before fitting the probe. The later representations are used directly. We exclude policy logits and value predictions from this sequence because they are separate readout branches rather than later shared representations. In the four arm tasks we probe the unpooled feature maps H^{(1)} and H^{(2)} of the first two convolutions, then h_{\mathrm{pre}} and h_{\mathrm{fc}}. In WalkToFlag we probe the three convolutional maps average-pooled to 4{\times}4 cells, \bar{H}^{(1)},\bar{H}^{(2)},\bar{H}^{(3)}, then h_{\mathrm{pre}} and h_{\mathrm{fc}}. We do not pool the robot maps globally because pooling removes position: globally pooled H^{(1)} and H^{(2)} keep R^{2}<0.17 of the marker position in PushToMarker, against 0.74 to 0.90 unpooled.

Every probe is fitted on a training partition and scored on held-out data: whole held-out levels in Maze and Coin over five split seeds, held-out object configurations in Attribute Maze, and held-out start frames in the robot tasks. The Procgen probes are ridge regressions with fixed strength \alpha=1 (logistic regression for Coin presence), and the robot probes are ridge regressions whose strength is selected by cross-validation on the training frames. All probe preprocessing is fitted on the training partition only.

Our primary quantity is absolute held-out probe performance. This answers the direct representational question: how linearly accessible is the target at each site? We note that absolute decodability alone, however, does not establish how much accessibility was produced by training. Simple visual or spatial structure can already be linearly exposed by an untrained convolutional network (Hermann & Lampinen, 2020).

Baselines.

We compare every probe with three baselines. First, randomly initialized networks of the same architecture (three draws in every environment), probed on the same frames. Second, input-level or positional control probes, which probe the observation or a correlated state variable directly, for example the pixels of the cheese cell, the agent’s position or block-averaged pixels. They show how much information is trivially available in the input or carried by co-located objects and therefore need not be learned. Third, in the Procgen environments, permuted labels give an empirical null with the same grouping and splits as the probe. A score that an untrained network or an input control already reaches is not read as a training effect, so each result in the following first asks whether the probe separates from the baselines and only then compares GM with Non-GM agents.

Maze probe construction.

In Maze, we first ask whether the cheese object itself is linearly detectable, using the probe bank of Appendix E.4 (Table 22). Standard Maze states always contain the cheese, so the negative class consists of the same states with the cheese removed. We also probe the bearing of the cheese relative to the agent and, for the proxy, the bearing of the reachable top-right extreme of the maze.

Maze results.

Cheese presence is trivially decodable from the input, as a linear probe on the pixels of the cheese cell reaches an AUC of 0.988 (Figure 14). Moreover, Non-GM agents decode it just as well from h^{(2)} onward (0.991 to 0.998), while GM agents stay at the level of random networks in the convolutional blocks and even below them at the later layers h_{\mathrm{pre}} and h_{\mathrm{fc}} (0.529 to 0.549 at every site, against 0.620 and 0.561 for random networks at the two later layers). The true goal is therefore far more linearly decodable in Non-GM than in GM agents, a separation in the expected direction. Because the presence of the cheese is already available in the input, however, a presence probe alone may not separate the cohorts cleanly, so we also probe the bearing of the cheese.

Cheese bearing separates from random networks in both cohorts from h^{(2)} onward, in GM only narrowly at h_{\mathrm{pre}} (0.712 against 0.686), and grows with depth, from chance at h^{(1)} to about 0.8 in GM and 0.95 in Non-GM agents. The bearing probe thus separates, but we are unsure whether a subsidiary piece of information drives it, namely the agent’s own position. In GM agents the bearing stays close to and below what agent position alone gives (0.712 to 0.808 from h^{(2)} onward, against 0.830), while Non-GM agents exceed it (0.873 to 0.951). With the corner bearing held fixed, so that only cheese-relative information helps, GM agents fall to 0.598 at h_{\mathrm{pre}}, below both agent position (0.703) and random networks (0.729), while Non-GM agents keep 0.930 (Table 4). We test this directly with a decorrelation check, on a set in which the agent’s own position carries no information about the cheese bearing. To build it, we keep inside every agent cell as many frames with the cheese to the left of the agent as to its right, which leaves 902 of the 1{,}987 cheese frames and brings the agent-position probe to chance (0.481, Figure 14, right). On this set the cheese bearing in GM agents collapses at every site (0.516 to 0.583) and falls below random networks at h_{\mathrm{pre}} and h_{\mathrm{fc}} (0.748 and 0.631), while Non-GM agents keep it from h^{(2)} onward (0.861 to 0.940), and so does the pixel probe (0.895). This rules out that GM agents carry cheese-bearing information beyond their own position. The GM bearing probe only tracks the agent, while Non-GM agents and pixels keep the cheese bearing.

And lastly, corner bearing, the proxy probe, is decoded well by both cohorts and far above random networks from h^{(2)} onward, with GM agents slightly ahead (0.952 against 0.892 at h^{(3)}, 0.963 against 0.945 at h_{\mathrm{pre}}). Agent position alone already predicts the corner bearing (AUC 0.959), so neither cohort carries corner information beyond the geometry of the sampled states. Across the two bearing probes, GM agents are somewhat worse than Non-GM agents for the true goal and slightly better for the proxy, a small effect that depends on the probe and the site.

Together we find that the cheese is trivially detectable from the observation, but GM training does not construct a strong cheese presence representation as seen in Non-GM agents. Cheese bearing information in GM agents is also largely accounted for by the agent position, whereas Non-GM agents contain substantially stronger goal-related spatial information. The Maze failure is therefore compatible with a learned representation that preserves generic spatial structure while failing to construct the goal-specific features that characterize the Non-GM policy.

Coin probe construction.

In Coin, we separate object availability from spatial information about the goal, using the probe bank of Appendix E.4 (Table 22). Unlike cheese presence in Maze, coin presence varies naturally during ordinary Coin trajectories as the coin moves into and out of the visible frame. Coin bearing is residualized on level progress because horizontal progress is strongly coupled to both observation structure and the training-time rightward trajectory. For the continuous bearing target, the level-permutation null yields strongly negative held-out R^{2} because permuted labels are constant within levels, so we use R^{2}=0 as the chance reference.

Coin results.

The coin presence probe separates from the baselines only in Non-GM agents (Figure 15). A probe on the raw pixels of the coin already reaches 0.760, and GM agents stay at that level at every site from h^{(2)} onward (0.724 to 0.775). They are above random networks in the convolutional blocks, but do not consistently exceed them at h_{\mathrm{pre}} and h_{\mathrm{fc}}. Non-GM agents start at the same level (0.748 at h^{(2)}) and approach ceiling from h^{(3)} onward (0.991 to 0.995). This separation holds on both frame sources of the probe bank. The robust Coin result is therefore not that the coin becomes undecodable in GM agents, but that the goal is cleanly represented in Non-GM agents, whose training constructs a much stronger downstream coin presence representation than GM training does.

Coin bearing, by contrast, separates from every baseline in both cohorts. From h^{(2)} to h_{\mathrm{pre}}, held-out R^{2} is 0.376 to 0.582, against at most 0.028 for random networks, 0.152 for agent position and -0.552 for raw pixels, and the two cohorts do not differ (0.523 and 0.527 at h_{\mathrm{pre}}). At h_{\mathrm{fc}} the cohort ordering depends on which half of the bank is evaluated (0.131 for GM against 0.360 for Non-GM on the full bank, reversed on the later-stage half), so we do not interpret Coin at h_{\mathrm{fc}}. The bearing probe separates, but as in Maze we are unsure whether the agent’s own position drives it, so we run the same decorrelation check. With 320 frames we do not subsample but residualize the bearing on agent position as well as on progress, fitted on the training frames, so that a probe on agent position decodes none of it (R^{2}=-0.024, Figure 15, right). Coin bearing stays decodable in both cohorts beyond agent position from h^{(2)} to h_{\mathrm{pre}} (0.150 to 0.378), against at most -0.020 for random networks and -0.369 for pixels, and the same holds with position entered as binned indicators (0.341 and 0.219 at h_{\mathrm{pre}} for GM and Non-GM). The values are lower than without the position term (0.523 and 0.527 at h_{\mathrm{pre}}), so part of the original signal was position, but unlike in Maze the goal bearing in GM agents does not reduce to it. This separates Coin from Maze.

Coin consequently provides a heterogeneous result across goal variables. Goal bearing is strongly accessible in both cohorts through h_{\mathrm{pre}}, with no stable cohort separation there. Coin presence, however, becomes nearly perfectly accessible in Non-GM agents while remaining substantially weaker in GM agents at the downstream shared representations. The robust representational distinction in Coin is therefore the organization of goal-object presence rather than a stable difference in goal bearing. Trinh et al. (2024) suggest that a CoinRun agent trained with the coin at the right end does not represent the coin, based on an anomaly detector and a learned help-request policy applied to its penultimate-layer features. Their analysis differs from ours in method and site, since neither tool measures supervised linear decodability, and the penultimate layer corresponds to h_{\mathrm{fc}} in our architecture, where GM coin presence is near the raw-pixel level.

Attribute Maze probe construction.

Attribute Maze requires a different decomposition because the intended and proxy cues are attributes of objects rather than separate spatial targets. Simply detecting a feature does not establish whether the representation associates that feature with the rewarded object. We therefore distinguish feature presence from goal role and probe four binary concepts,

L_{\mathrm{pres}},\qquad Y_{\mathrm{pres}},\qquad L_{\mathrm{role}},\qquad Y_{\mathrm{role}},

where L_{\mathrm{pres}} and Y_{\mathrm{pres}} indicate whether the line and yellow attributes are present, and L_{\mathrm{role}} and Y_{\mathrm{role}} indicate whether the corresponding attribute belongs to the rewarded object. The first pair measures feature availability. The second pair measures whether feature identity is available in relation to the rewarded role.

To separate presence from role, we use the configuration grid of the probe bank (Appendix E.4, Table 22) rather than sampling only from the ordinary task distribution. The grid includes states in which an attribute is visible but belongs to the lure, allowing “feature present” and “feature belongs to the rewarded object” to disagree, and both candidate objects are independently placed on uniformly sampled free cells. For the line attribute, presence and rewarded role disagree in 4/12 training configurations and 2/6 held-out configurations. The yellow distinction has lower support, with Y_{\mathrm{pres}} and Y_{\mathrm{role}} differing in only one configuration in each split. We therefore interpret the yellow-role probe more cautiously and do not assume that the two role probes provide equally strong separation of feature identity from rewarded role.

Attribute Maze results.

Figure 16 shows the corresponding cue-presence and cue-role probes across the textured and black-background cohorts. Attribute Maze does not show a uniform suppression of either competing cue family. Across backgrounds, training increases linear accessibility of line and yellow information at the convolutional sites, while the role probes additionally show accessibility of information about which attribute belongs to the rewarded object. Accessibility at the two downstream shared representations is more variable. At h_{\mathrm{pre}}, black-background GM line presence does not rise above initialization (0.792 versus 0.810, Table 4), and on the black background line presence is largely decodable from raw pixels (0.975). Yellow presence and yellow role exceed both random networks and the pixel probe at this site in both cohorts, whereas line role separates from neither (0.632 and 0.636 against 0.603 and 0.629).

So to conclude, training increases linear accessibility of both competing cue families at the convolutional sites of Attribute Maze. Yellow presence is strongly training-enhanced under the convolutional readouts, while yellow-role information is resolved above initialization at h^{(2)}–h^{(3)} in three of the four background-by-cohort groups. Line presence and line-role gains are weaker and more site-dependent. The probes do not separate GM from Non-GM agents consistently, so cue accessibility alone does not explain the behavioral difference.

Robot probes.

We use a probe bank (Appendix E.4, Table 23) that holds start frames in which goal and proxy vary independently. In PushToMarker and LocationReach we probe the table position (x,y) of the goal, the marker and the ball. Their proxies are fixed places, the table centre and the training corner, so they have no variation to probe. In AttributeTable and SortZones we probe the position of the goal, the cube and the crossed pad, and of the proxy, the red object and the green pad. In WalkToFlag we probe the bearing of the flag and the tower relative to the robot camera. Position probes are scored by held-out R^{2} averaged over x and y, and bearing probes by held-out R^{2}. The binary probes, whether the crossed pad lies on the left and whether the flag or the tower is in view, are scored by held-out ROC–AUC.

Robot results: the proxy is a location.

In PushToMarker and LocationReach the goal position is not decodable in GM agents beyond what random networks and pixels already give (Figure 17). The marker position is decodable in GM agents at h_{\mathrm{pre}} at 0.717, not above random networks (0.805) and on par with the pixel probe (0.713), and at h_{\mathrm{fc}} it falls to 0.110 against 0.406 for random networks, while Non-GM agents keep it (0.819). The ball position is decodable in GM agents below random networks at every site, 0.564 against 0.737 at h_{\mathrm{pre}}, also on held-out scenes with the ball at least 10 cm from the training corner (0.511 against 0.706), while Non-GM agents stay at 0.902 or above (0.889 or above on these scenes). As in Maze, GM training does not build on what a random network already exposes, but here a random network exposes much more, since the ball is large and salient where the cheese covers 2 to 5 pixels. Both proxies are fixed places, so in these two tasks the probes speak only to the goal.

Robot results: the proxy is an attribute or a landmark.

In AttributeTable the cube position is decodable in GM agents through h_{\mathrm{pre}}, above random networks and at the level of the pixel probe (0.914 against 0.768 and 0.889), close to Non-GM agents (0.975). At h_{\mathrm{fc}} the proxy takes over: GM agents keep the red object better than the cube (0.866 against 0.695), whereas Non-GM agents keep the cube better (0.942 against 0.661), all far above random networks (0.319 and 0.402). The red object is the cube in half of the scenes, so the two positions correlate (0.38 in x, 0.33 in y). In SortZones the pads are large, and every network up to h_{\mathrm{pre}}, random ones included, and the pixel probe decode the crossed-pad position at R^{2}\geq 0.989, so no probe there is a training effect. Only h_{\mathrm{fc}} separates the cohorts, where the crossed-pad position is decodable in GM agents below random networks (0.612 against 0.832) and in Non-GM agents above them (0.976), and the green-pad position stays at the random-network level in GM agents (0.955 against 0.961). So in SortZones the true goal is represented more weakly in GM than in Non-GM agents, and this can be said reliably only at h_{\mathrm{fc}}, since below it the headroom above random networks is too small.

In WalkToFlag the flag bearing is decodable in GM agents well above random networks at every site, and it reaches the last shared layer h_{\mathrm{fc}}. However, it stays below the pixel probe and below Non-GM agents (0.656 against 0.837 at h_{\mathrm{pre}}, 0.590 against 0.826 at h_{\mathrm{fc}}). The true goal is decodable, but less than in generalized agents, which follow it. The effect flips for the proxy: the tower bearing is more decodable in GM than in Non-GM agents (0.958 against 0.710 at h_{\mathrm{fc}}), and both cohorts separate from random networks. This is what we would expect for misgeneralized agents, as the proxy is represented more strongly in GM agents and the true goal more strongly in Non-GM agents, the mirror of their behavior inside their representation. This asymmetry appears only in environments with competing objects or object cues, which is a first pointer to how the environments separate. The deeper we go in the next analyses, the more this group separates from the others, because in it the goal information propagates far into the agent. One thing we still need to rule out: since we probe the bearings of two objects, one tracked object could light up both bearings. In the full bank the two bearings correlate (r=0.37 over held-out states with both in view), so, as for the Maze and Coin bearings above, we repeat the probes on the conflict layouts, which are new to both cohorts and in which the two bearings are nearly uncorrelated (r=-0.07), and find the same pattern (Table 5, conflict rows, flag 0.664 against 0.855, tower 0.940 against 0.876, GM against Non-GM at h_{\mathrm{pre}}, all well above random networks). So the flag bearing in GM agents does not come from the tower, and the tower bearing does not come from the flag.

Taken together, wherever the proxy is an attribute of the goal or a closely related landmark, the probes point the same way: at h_{\mathrm{fc}} the goal is weaker and the proxy stronger in GM than in Non-GM agents (Table 5). Attribute Maze shows this pattern in part.

What a probe can and cannot show.

Linear decodability of a goal feature does not necessarily indicate a goal representation. Probes retrieve linearly accessible information, but whether that information is usable as a goal inside the network does not necessarily follow from its decodability. Information can be decodable but still not be used by the policy as a goal, or not be recoverable as a representation that navigates the agent toward a goal. In SortZones, for example, a randomly initialized network decodes the crossed-pad position at h_{\mathrm{pre}} with R^{2}=0.989, yet a new readout trained by behavior cloning on these random features recovers nothing: it brings the cube to the crossed pad less often than a coin flip would (0.151 against 0.377, Appendix D.2). A second limit is headroom. Many goal and proxy features are already decodable from random networks or from the pixels, and a probe speaks to what training built only where it separates from these baselines. In several environments at least one cohort separates clearly from random networks, which is enough to make a statement. SortZones is the hardest to read, because its headroom is tiny. Probes are therefore indicative of information that may be used downstream, but a representation that acts as a goal requires more, which is why the following analyses turn from decodability to how the policy reacts to and uses goal information.

Conclusion.

We asked whether information about the intended goal is lost, and find that losing it is not a core part of goal misgeneralization. In many environments the true goal is still linearly represented, even in late layers, so a uniform deletion is not the mechanism. What differs is the degree of representation, and three groups emerge, with the caveat that our probes can confirm that information is linearly available but cannot rule out that it is present in another form. In Attribute Maze, AttributeTable, SortZones and WalkToFlag the goal information is learned up to late layers and comes with an asymmetry: GM agents decode the proxy better, Non-GM agents the goal (in Attribute Maze only in part, and in SortZones readable only at h_{\mathrm{fc}}). Coin is an intermediate case: the coin bearing is decodable in both cohorts through h_{\mathrm{pre}}, also beyond the agent’s position, but GM agents build a much weaker coin-presence representation than Non-GM agents. In Maze, LocationReach and PushToMarker the goal is not linearly decodable above random networks in GM agents. PushToMarker belongs here rather than with Coin, since its marker position stays below random networks at h_{\mathrm{pre}} and h_{\mathrm{fc}}, and in Maze the cheese bearing reduces to tracking the agent’s position once that position is decorrelated from it. We therefore next ask whether goal-related changes are coupled to the native policy and what functional role the resulting representations play in action selection.

C.2 Behavioral dependence: Does the policy react to goal changes, viewed from outside?

Motivation.

Linear decodability establishes that goal-related information can remain available internally, but does not show whether the policy responds to that information. We therefore intervene directly on goal- and proxy-related variables in the environment and ask whether the resulting world change alters the policy, whether that change follows the edited feature, and whether it changes eventual goal achievement.

Edit design.

We ask whether the trained policy actually responds when a goal-related variable is changed in the world. A world edit e changes a goal- or proxy-related variable in the level while keeping the rest of the state as fixed as the environment permits, and turns a base state s into a matched edited state s_{e}, with e=\star for an edit of the goal and e=p for an edit of the proxy (Definition 2). Because nothing else changes, a change in the policy’s output can be attributed to the edited variable.

The environments allow two kinds of edit. Relocation edits are used in Maze, Coin, LocationReach, PushToMarker and WalkToFlag, where the goal object can be moved without changing its identity or the level (Figure 18). A relocation edit moves the edited object from its place g to a new place g^{\prime} and holds the rest of the level fixed. In Maze the cheese is moved while the layout, the agent state and the top-right destination stay unchanged. The cheese is moved to the reachable bottom-left extreme, which gives every state the same kind of edit. The structural Maze edit instead opens access to the top-right region while otherwise preserving the maze state, and so edits the proxy. In Coin, g is a cell to the right of the agent, on the shortcut side, and g^{\prime} the cell one step to its left. The environment holds a single coin, so the edit moves it and does not add a second one. The end-of-level marker is left in place. In PushToMarker and LocationReach the marker and the ball move to a second position away from the training place, with the arm and the other objects fixed. In WalkToFlag goal and proxy are separate objects, so the goal edit moves the flag and the proxy edit the tower, each to a place beside the other object.

Swap edits are used in Attribute Maze, AttributeTable and SortZones. In Attribute Maze the goal and proxy are attributes of objects, and moving an object would change the navigation geometry of the conflict together with the cue. So we hold the two candidate locations fixed and swap their attributes (Figure 18). Let the candidates at locations u and v be

o_{u}=(\mathsf{line},\mathsf{red}),\qquad o_{v}=(\mathsf{gem},\mathsf{yellow}),

so the line cue sits at u and the yellow cue at v. The shape swap exchanges the shapes and keeps colours and locations fixed,

\mathcal{E}^{\mathrm{shape}}:\qquad(\mathsf{line},\mathsf{red})_{u},\,(\mathsf{gem},\mathsf{yellow})_{v}\longmapsto(\mathsf{gem},\mathsf{red})_{u},\,(\mathsf{line},\mathsf{yellow})_{v}.

The colour swap exchanges the colours and keeps shapes and locations fixed,

\mathcal{E}^{\mathrm{colour}}:\qquad(\mathsf{line},\mathsf{red})_{u},\,(\mathsf{gem},\mathsf{yellow})_{v}\longmapsto(\mathsf{line},\mathsf{yellow})_{u},\,(\mathsf{gem},\mathsf{red})_{v}.

These two swaps are the canonical counterfactuals of Attribute Maze. They resolve the same line and yellow conflict in opposite directions, since the training-consistent yellow line appears at v under the shape swap and at u under the colour swap. This return to a familiar conjunction is intended. The base state is the out-of-distribution conflict, and each swap asks how the policy changes when the conflict is resolved toward one of the two fixed candidates, without introducing a further unfamiliar object combination. Agent position, walls, object count and candidate locations are unchanged, so the navigation geometry is identical across the base state and both swaps. The two swaps are not equally likely training perturbations, since the shape and colour vocabularies are not symmetric, and we use them only as paired resolutions of one spatial conflict. AttributeTable and SortZones use the same two swaps (Figure 18). In AttributeTable the base conflict is (\mathsf{cube},\mathsf{blue})_{u},(\mathsf{sphere},\mathsf{red})_{v}, the shape swap gives (\mathsf{sphere},\mathsf{blue})_{u},(\mathsf{cube},\mathsf{red})_{v} and the colour swap (\mathsf{cube},\mathsf{red})_{u},(\mathsf{sphere},\mathsf{blue})_{v}. In SortZones the cross takes the role of the shape. The base conflict is (\mathsf{purple},\mathsf{cross})_{u},(\mathsf{green},\mathsf{plain})_{v}, the cross swap gives (\mathsf{purple},\mathsf{plain})_{u},(\mathsf{green},\mathsf{cross})_{v} and the colour swap (\mathsf{green},\mathsf{cross})_{u},(\mathsf{purple},\mathsf{plain})_{v}. As in Attribute Maze, each swap returns the scene to the training conjunction, a red cube or a crossed green pad, at one of the two fixed places. All edits are applied to edit banks that we generate for this purpose, and their specification, including their size, is given in Appendix E.

Metric design.

We evaluate behavioral dependence at increasingly longer time horizons. First, we measure whether the edit changes the immediate policy at all, through the KL divergence between the action distributions in the original and in the edited state,

D_{\mathrm{KL},e}(s)=D_{\mathrm{KL}}\left(\pi(\cdot\mid s)\,\|\,\pi(\cdot\mid s_{e})\right).

This quantity is direction-agnostic. A large value establishes that the edit redistributes the probability over actions, but not whether the mass moves onto actions that pursue the goal or the proxy. The robot policies are Gaussian with a state-independent standard deviation \sigma, so their KL has the closed form \tfrac{1}{2}\sum_{i}\big(\mu_{i}(s_{e})-\mu_{i}(s)\big)^{2}/\sigma_{i}^{2}, the change of the action mean measured in units of the policy’s own \sigma. This scale depends on \sigma and on the action space, so a robot KL is compared only between the cohorts of one environment.

Second, we measure whether the change is consistent with goal-directed action, through a quantity we call signed follow. Let A_{\star}(s) be the progress actions toward the intended goal in state s, the actions that shorten the path to it, and A_{p}(s) the progress actions toward the proxy (Definition 10, Section B.4). For a goal edit we use A_{e}=A_{\star} and for a proxy edit A_{e}=A_{p}, both read in the edited state, so that they lead to the edited object’s new place. Signed follow is the probability that the edit moves onto these actions,

\mathrm{SF}_{e}(s)=\pi\!\left(A_{e}(s_{e})\mid s_{e}\right)-\pi\!\left(A_{e}(s_{e})\mid s\right),\qquad\pi(A\mid s)=\sum_{a\in A}\pi(a\mid s).

A positive value means that the edit moves probability onto the actions toward the edited object’s new place. In Maze the sets are graph-exact, toward the relocated cheese for the cheese edit and toward the opened corner for the structural edit. In Coin A_{\star} holds the leftward actions, and in Attribute Maze the sets lead to the candidate that carries the line after the shape swap and the yellow after the colour swap.

For continuous actions the same set is a half-space, the commanded steps with a positive component toward the object. A robot policy, however, already puts almost all of its probability in that half-space before any edit, above 0.999 in AttributeTable, because its \sigma is small compared with its step. The probability therefore cannot rise, and we measure instead how the edit turns the planned step (Definition 11). With \hat{t}_{e} the target direction of the edited object, the goal for e=\star and the proxy for e=p, read before the edit in s and after it in s_{e},

\mathrm{SF}_{e}(s)=\big(\mathrm{step}(s_{e})-\mathrm{step}(s)\big)^{\top}\,\frac{\hat{t}_{e}(s_{e})-\hat{t}_{e}(s)}{\big\|\hat{t}_{e}(s_{e})-\hat{t}_{e}(s)\big\|_{2}}.

The fraction is the unit vector along which the edit turns the target direction, so a positive value means that the planned step turns from the object’s old place toward its new place, the continuous counterpart of moving probability onto actions toward the new place. It is in millimetres for the arm and in command units for the quadruped.

KL and SF are computed on the same states. In Procgen these are strict-conflict states, in which no action makes progress toward both goal and proxy, A_{\star}(s)\cap A_{p}(s)=\emptyset (Section B.4). With few discrete actions, the progress sets toward two targets can share an action, and a shift of mass onto that action would count for both, so we keep only states in which the sets are disjoint. The robot banks take scenes laid out like each task’s conflict test, with goal and proxy at different places, in WalkToFlag closer together so that both stay in view (Appendix E). A continuous action space has no such overlap to remove. Signed follow projects the change of the planned step onto a direction, so even a small separation between two targets gives two different directions instead of falling into one shared discrete action, and the edited object’s old and new place always differ.

For the behavioral world-edit quantities, we first average over evaluation states within each seed and then report the mean and sample standard deviation across seeds. Where the Maze and Coin results report reach, it is the fraction of rollouts started from the edited state that reach the relocated goal (Appendix F).

Results.

The world edits reveal qualitatively different patterns of behavioral dependence across the eight environments (Table 6). Because KL and signed follow have different scales in different environments, we read every edit relative to the Non-GM agents of the same environment, as the ratio of the GM to the Non-GM value (Figure 19). The two metrics then separate the eight environments in two steps.

First, KL separates Maze and LocationReach from the rest. A goal edit barely moves the GM policy there, at a ratio of 0.009 in Maze and 0.026 in LocationReach, while the same edit moves the Non-GM policy strongly. The GM agents therefore behave as if they were inert to the goal. In the other six environments the ratio of a goal edit lies between 0.107 and 0.362. In the three swap environments and WalkToFlag the table shows, moreover, that both edits move the GM policy, the goal edit and the proxy edit alike.

Second, signed follow separates the remaining environments. For a goal edit its ratio is about zero in PushToMarker (0.017) and Coin (0.001) as well. The goal edit may move probability mass in these GM policies, but not onto actions that lead to the goal. A navigating policy may react in the short term to the objects around it, and the GM policies react to the edit in this way without turning their behavior toward the goal. In WalkToFlag and the three swap environments the ratio lies between 0.234 and 0.533, so the goal still turns even the GM policy toward itself, though less than it turns the Non-GM policy. A proxy edit, in contrast, turns the GM policy several times more than the Non-GM policy, which barely follows the proxy, so the ratio is large but its exact value is not informative. This is the asymmetry we expect between GM and Non-GM policies, and these environments show it in KL and in signed follow alike. Two signed-follow ratios are undefined because Non-GM agents do not turn toward the edited object, the tower in WalkToFlag on average and the red object in AttributeTable in one seed.

Together, the two metrics sort the environments by how far the goal gets. In Maze and LocationReach the goal edit does not reach the policy. In PushToMarker and Coin it reaches the policy but does not direct its decision. In WalkToFlag and the swap environments it enters the decision and loses to the proxy. Appendix C.3 asks how this plays out inside the policy.

C.3 Fisher geometry: Is the policy using the goal, viewed from inside?

Motivation.

The behavioral world edits in Appendix C.2 measure whether changing a goal-related property of the environment changes the policy. We next ask the corresponding question inside the policy: how strongly is the representation change induced by the same world edit coupled to the policy’s action distribution? We ask it in all eight environments, with the same edits and evaluation states as the behavioral analysis.

Design.

Rather than selecting an internal direction independently of the environment, we let the controlled world edit determine the representation displacement that we evaluate. For a base state s and its edited copy s_{e} (Section 18), the edit moves the representation at site m by \Delta_{e}^{(m)}(s)=h^{(m)}(s_{e})-h^{(m)}(s) (Definition 2), read at the two post-ReLU sites h_{\mathrm{pre}} and h_{\mathrm{fc}} of Section B.1. We measure the world-edit Fisher sensitivity

Q_{e}^{(m)}(s)=\Delta_{e}^{(m)}(s)^{\top}F^{(m)}(s)\Delta_{e}^{(m)}(s),

with F^{(m)}(s) the activation-space Fisher at the unedited state (Definitions 4 and 5). Up to second order \tfrac{1}{2}Q is the KL between the base and the edited policy (Appendix B.2), so Q is the inside counterpart of the behavioral KL, and like the KL it is unsigned. A small Q does not imply that information about the edited property is absent. Instead, it means that the particular representation displacement produced by the edit has little local effect on the native action distribution.

For the Procgen policies F at h_{\mathrm{fc}} has the categorical closed form after Proposition 2. The robot policies are Gaussian with a state-independent \sigma, so F=J_{\mu}^{\top}\operatorname{diag}(\sigma^{-2})J_{\mu} exactly, with J_{\mu} the Jacobian of the action mean (Proposition 3), and Q is the squared change of the mean in units of \sigma. We confirmed that \sigma is a learned vector not computed from the observation in both robot agents, and on the 19 arm checkpoints available locally a Monte Carlo estimate of the score covariance from 4000 sampled actions matches this Fisher to within 2.0\% in trace and 6.2\% in Frobenius norm on four world-edit bank states each, the size expected from sampling noise (1\% and 4\%). Q is the same at both sites only to first order (Proposition 4). Over the finite edits we study it changes from h_{\mathrm{pre}} to h_{\mathrm{fc}} by up to 59\% in Non-GM agents (-31\% in LocationReach, -25\% in PushToMarker, +45\% for the Attribute Maze and AttributeTable shape swaps and +59\% for the WalkToFlag tower swap) and by up to 30\% in GM agents (SortZones cross swap), so the GM to Non-GM ratio of Q can differ between the sites, as for the Attribute Maze shape swap (0.551 at h_{\mathrm{pre}} against 0.307 at h_{\mathrm{fc}}, Table 8). The ratio lies on the same side of 1 at both sites for every edit, so we read the results at h_{\mathrm{fc}}, the site of Table 1, and report h_{\mathrm{pre}} in Table 8.

We split Q=G\,D_{F} into the displacement in Fisher units D_{F} and the relative directional gain G (Definition 6). D_{F} is the product of \|\Delta\|_{2}^{2} and the mean Fisher sensitivity \operatorname{tr}(F)/d, and the second factor depends on how sharp the policy is (Corollary 1). A small GM D_{F} could therefore reflect a sharper or flatter policy rather than a smaller movement of the representation, and Table 9 separates the two factors.

Q says how strongly the policy reacts, not in which direction. In Procgen we therefore also measure the signed target-selection effect, the first-order change of the contrast between the goal and the proxy action sets along \Delta_{e},

\partial_{\Delta_{e}}C_{s}=\nabla_{h}C_{s}\big(h(s)\big)^{\top}\Delta_{e}(s),\qquad C_{s}(h)=\pi\big(A_{\star}(s)\mid h\big)-\pi\big(A_{p}(s)\mid h\big),

with both action sets read in the base state (Definition 8). Its sign says whether the displacement moves preference toward the goal or toward the proxy, so it is the inside counterpart of signed follow, \mathrm{SF}_{e}(s)=\pi\big(A_{e}(s_{e})\mid s_{e}\big)-\pi\big(A_{e}(s_{e})\mid s\big), the probability that the edit moves onto the actions toward the edited object’s new place (Appendix C.2). It is reported for Maze, Coin and Attribute Maze in Table 7 and per state in Figure 20. With continuous actions the contrast saturates at the robot start states, so for the robots the direction of the reaction is read from signed follow alone.

Evaluation states.

Every environment uses the states and edits of the world-edit table, the conflict banks in Procgen and the world-edit banks in the robot tasks (Appendix E). On the stricter AttributeTable bank of the cue-competition geometry (Appendix C.5) the world-edit KL ratios keep their order (0.297 for the shape swap and 2.741 for the colour swap).

Finite-edit calibration.

The local approximation is accurate enough for the edits we study, because where it deviates from the exact KL it cannot change which cohort reacts more. We compare \tfrac{1}{2}t^{2}Q with the exact KL along fractions t\in[0,1] of each edit. For the Gaussian robot policies the KL is itself quadratic in the mean shift (Proposition 3), so only the linearized action mean is approximate. In Procgen the approximation tracks the exact KL closely except for the largest Non-GM edits, the Maze goal relocation and the Attribute Maze shape swap, where it underestimates it by a factor of 1.4 to 2.1, which makes the GM to Non-GM ratio of Q conservative. In the robot tasks \mathrm{KL}/(Q/2) at t=1 is 0.932 to 1.114 for the relocations and 0.569 to 0.935 for the swaps, where the mean head bends along the edit (means over seeds of per-seed medians). It differs between the two cohorts of an edit by at most 0.166 (SortZones cross swap), which moves a GM to Non-GM ratio of Q by at most a factor of 1.25, less than the distance of any reported ratio from 1.

Results.

As the groups of the behavioral analysis (Appendix C.2) replicate inside the policy, we present the results in the same groups: first Maze and LocationReach, then PushToMarker and Coin, then Attribute Maze, AttributeTable, SortZones and WalkToFlag. Table 8 gives Q, G and D_{F} for every edit at both sites. Because robot Q is in units of each policy’s own \sigma, we read every edit as the ratio of the GM to the Non-GM value within its environment, as Figure 19 does outside, and never compare values across environments. Figure 21 shows these ratios at h_{\mathrm{fc}}. In Q every goal edit reaches the GM policy less than the Non-GM policy, with ratios from below 0.001 (Maze) to 0.307 (Attribute Maze), and every proxy edit reaches it more, from 2.061 (AttributeTable) to 73.994 (Attribute Maze), with every GM seed above the Non-GM mean. D_{F} lies on the same side of 1 as Q for every edit, whereas the ratio of G stays between 0.460 and 2.023 for every edit except the Attribute Maze colour swap (6.486).

Maze and LocationReach.

In Maze and LocationReach the goal edit is as inert inside the policy as it is in behavior. Relocating the cheese and moving the ball reach the GM policy far less than the Non-GM policy, with a GM to Non-GM ratio of Q below 0.001 in Maze and of 0.023 in LocationReach, where the behavioral KL ratios are 0.009 and 0.026. These are the two smallest goal-edit ratios of the eight environments, against 0.063 to 0.307 for the other six. The separation holds for the cohort means, not for every seed, since the most sensitive LocationReach GM seed reaches 0.041 of the LocationReach Non-GM mean and the least sensitive Coin GM seed 0.045 of the Coin Non-GM mean. In Maze the gap lies in the displacement, with a ratio of D_{F} of 0.001 against a ratio of G of 0.637, so the direction of the cheese displacement is not read strongly enough to compensate. Cheese information may still be represented elsewhere, but its movement barely changes the directions the native policy reads. In LocationReach both factors contribute, with ratios of 0.050 for D_{F} and 0.513 for G. The signed target-selection effect of GM agents in Maze stays at -0.001 at both sites against -0.271 and -0.269 for Non-GM agents, where a negative value means a shift toward the relocated cheese (Table 7 and Figure 20), matching a signed follow of -0.001. In LocationReach the small reaction even turns the planned GM step slightly away from the moved ball (signed follow -1.764 mm against +199.668 mm, Table 6).

PushToMarker and Coin.

Here the goal edit still reaches the GM policy but does not direct it, as in the behavioral edits. It reaches the GM policy at about a sixth of the Non-GM sensitivity in PushToMarker (ratio of Q 0.159, behavioral KL ratio 0.206) and at about a sixteenth in Coin (0.063, behavioral KL ratio 0.114, nearly the same at h_{\mathrm{pre}}). In both most of the gap lies in the displacement, with ratios of D_{F} of 0.177 and 0.093 against ratios of G of 0.927 and 0.757. The reaction does not point at the goal. In Coin the signed effect of GM agents stays near zero at both sites while Non-GM agents shift toward the moved coin (-0.010 against +0.139 at h_{\mathrm{fc}}, Table 7 and Figure 20), and in PushToMarker the planned GM gripper step does not turn toward the moved marker (signed follow +0.487 mm against +27.948 mm, Table 6). This is the inside counterpart of the behavioral result, where the edit moves probability mass but not toward the goal.

Attribute Maze, AttributeTable, SortZones and WalkToFlag.

In these four environments the goal edit reaches the GM policy less than the Non-GM policy and the proxy edit reaches it more, as in behavior, where the goal enters the decision and loses to the proxy. In the order Attribute Maze, AttributeTable, SortZones and WalkToFlag, the ratio of Q is 0.307, 0.297, 0.096 and 0.165 for the goal edit (the shape swap, the cross swap and the flag move) and 73.994, 2.061, 3.307 and 5.893 for the proxy edit. Within the GM policy of each environment the proxy edit outweighs the goal edit, with a ratio of proxy-edit to goal-edit Q of 1.938, 7.359, 10.565 and 18.080 in the same order. In Attribute Maze this organization is already visible at h_{\mathrm{pre}} and is reproduced under textured observations. The Non-GM agents differ. In Attribute Maze they barely react to the proxy edit, whose Q is 0.008 of their goal-edit Q, whereas the robot Non-GM agents react to it about as much as to the goal edit in AttributeTable (1.059), a third as much in SortZones (0.308) and half as much in WalkToFlag (0.508).

In Attribute Maze the signed effect shows what the two directions do. The shape and the colour swap favour opposite candidates, so the contrast C_{s} gives them opposite signs whenever the policy follows both edits, and the sign pattern alone is not evidence of competition. What matters is that both effects are large and consistent. In GM agents each displacement shifts preference toward the candidate its swap favours in almost every conflict state (-0.665 for shape and +0.918 for colour at h_{\mathrm{fc}}, and likewise at h_{\mathrm{pre}}), whereas Non-GM agents follow the shape edit just as strongly (-0.897) and the colour edit an order of magnitude less (+0.063, Table 7 and Figure 20). So the goal edit reaches the GM policy at a fraction of the Non-GM sensitivity, as in Coin, but here it moves preference toward the goal. On the robots the planned GM step also still turns toward the moved goal, less than the Non-GM step (about half in WalkToFlag), and in SortZones it turns toward the swapped colour four times as much as the Non-GM step (Table 6). Whether the two cues actually pull against each other is tested next through the push–pull redistribution of target support and the within-state displacement geometry (Appendix C.5.1).

What the decomposition shows.

G and D_{F} describe how a sensitivity is carried and are not decision rules (Table 8, Figure 21). On the goal edit GM agents have a lower gain than Non-GM agents in seven of the eight environments, by at most about a factor of two. The exception is Attribute Maze, where the goal displacement is smaller in GM agents but falls on directions the GM policy reads more strongly. The displacement ratio is smallest in Maze (0.001) and LocationReach (0.050), but per seed LocationReach overlaps with Coin, so the two do not separate cleanly from the rest. In Maze, Coin and PushToMarker the gap lies mostly in the displacement, whereas in LocationReach and WalkToFlag the gain also differs by about a factor of two.

Policy sharpness does not produce these displacement gaps (Table 9). In Procgen the mean Fisher sensitivity of GM policies is at most 2.3 times smaller than that of Non-GM policies, and larger in Coin and Attribute Maze, while in Maze and Coin the displacement norm differs by a factor of 3 to 22. In Attribute Maze the GM policies also have the higher action entropy, so they are the less sharp ones. In AttributeTable and LocationReach the GM arm policies are the sharper ones, with a smaller mean \sigma and about twice the mean sensitivity, which works against a small GM D_{F}. PushToMarker is the one task where the sensitivity factor carries part of the gap, 1.5 times smaller in GM agents at h_{\mathrm{fc}} against a factor of 4 in \|\Delta\|_{2}^{2}. A separate depth sweep with three checkpoints per cohort scales the goal edit by the spread of the scenes, the median squared distance between two random base states at the same site. In randomly initialized networks this scaled edit keeps its size from h_{\mathrm{pre}} to h_{\mathrm{fc}}. In LocationReach and SortZones it grows in Non-GM agents but not in GM agents, and in PushToMarker it shrinks in both cohorts, by 41\% in Non-GM and 60\% in GM agents.

We therefore read the decomposition as a description of how each network carries a given sensitivity. What carries the comparison between the cohorts is Q, whose order between the cohorts mirrors the behavioral KL of every edit from the inside. Two limitations remain. G and D_{F} depend on the coordinates of each site (Appendix B.2), and with three seeds per cohort several robot intervals are wide, so the ordering should be read as an indication.

Together.

The Fisher view therefore recovers the same progression that was visible from behavior, but now from inside the policy. In Maze and LocationReach the goal edit barely moves the native policy, in PushToMarker and Coin it moves it but not toward the goal, and in the four environments with a proxy edit both edits stay coupled to the policy, with the proxy edit reaching GM agents more strongly than Non-GM agents. So the common behavioral label “goal misgeneralization” hides different relationships between the intended goal and the native policy. The goal can be nearly ignored, weakly coupled, or actively read alongside a competing proxy. This mirrors the probes (Appendix C.1). In the competition group the policy reads proxy and goal, and the GM policy reads the proxy more strongly, as GM agents decode the proxy better and Non-GM agents the goal. In the inert group decodability and policy sensitivity are low together, since the goal is neither linearly decodable above random networks in GM agents nor read by their policy. Across probes, world edits and Fisher sensitivity the same groups emerge, with the one exception of PushToMarker, whose marker is not decodable above random networks but still weakly reaches the policy. This increasingly establishes a depth at which the goal is lost that replicates across experiment types. G and D_{F} describe how the sensitivity is carried but cannot be used as decision rules.

C.4 What role does the intended-goal information play?

The earlier analyses, from inside the network and from the outside, sort the environments into groups by how far the goal reaches the policy, and they measure the immediate reaction of the next action. This chapter lengthens the time horizon and asks what organizes behavior over the whole episode, and therefore what truly acts as the goal of the agent. Does the remaining goal information organize behavior, or does it only nudge the next step? What takes the goal’s place, and what role does the proxy play? We walk through five groups. First come the environments where the substitute target is a location and the policy is inert to the goal, Maze and LocationReach (Appendix C.4.1). Second comes PushToMarker, where the substitute is again a location but the policy is weakly wired to the goal (Appendix C.4.2). Third comes Coin, a weakly wired environment that we suspect of a policy shortcut (Appendix C.4.3). Fourth comes cue competition within one object, in Attribute Maze, AttributeTable and SortZones (Appendix C.4.4). Last comes WalkToFlag, where two objects compete (Appendix C.4.5). Many of these analyses are specific to one environment, so we group them by analysis type, and we will find that they sort the environments into three categories, functional target substitution, a policy shortcut, and cue or object competition.

C.4.1 Functional target substitution: Maze and LocationReach

Both environments hold the goal at one place in GM training, the cheese in the top-right corner of every maze and the ball at one point in the back-right corner of the workspace (Appendices A.1.1 and A.2.2), and in both the goal edit barely moves the GM policy (Appendix C.2). Something else must therefore organize the episode, and we ask two questions about it. Is the proxy a target the agent walks to, or only a direction of movement? What kind of place is the target, in Maze a memorized place or a spatial relation, and in LocationReach a point on the table, an arm posture or a place in the camera image?

Design.

To ask whether the proxy is a target, we use states in which moving in a fixed direction is not the same as going to a target, those where the agent must make a detour to reach the corner. These are the 168 detour states of the detour bank (Table 21, Appendix E.3), chosen from the maze graph alone. On them the graph-exact progress sets toward the cheese and toward the reachable top-right extreme (Definition 10) are disjoint, and no action that makes progress toward the extreme points up or right, so an agent that only moves up or right cannot follow the corner there. We classify the greedy action as corner progress, cheese progress, naive only (up or right without progress toward either target) or neutral.

In LocationReach we ask whether the proxy is a target with an off-path test. Each ball is placed at a position that the same GM gripper passes through at ball height from some other start, at least 8 cm from its path in the current episode and at least 10 cm from the corner, so reaching the ball asks for no movement the GM agent has not already made. We record which of corner and ball the gripper reaches first. The 500 episodes per seed share 26 to 32 distinct ball positions, so the effective number of goal positions is about 30 per seed.

To ask what kind of place the target is in Maze, we borrow the corner-opening edit of Mini et al. (2023), who use it as the contrast pair of a steering vector and note that it attracts the agent, and use it instead as a test of the target. The edit carves a new path to the top-right interior cell of the full grid, which lies outside the original maze in 80\% of states and so creates a new top-right extreme (Fig. 22b and c show the carved cells). A second edit separates this extreme from the screen corner. It carves a new extreme only two cells beyond the old one, joined by a three-cell corridor that does not pass through the old corner and stays at least three cells from the screen corner. It needs room in the margin and so covers mazes of size at most 17, which is 64\% of the states. With each edit, a corridor of the same length carved beyond the bottom-left extreme tests whether newly opened space alone attracts the agent.

In LocationReach the training corner is an empty patch of the table, so instead of moving it we move what the policy could use to define it. Shifting the arm base by 8 cm changes which joint posture puts the gripper at the corner, so a posture target predicts a point 8 cm from the corner. Shifting the camera by 5 cm moves the ground point under every pixel by 5 cm, so a target fixed in the image predicts a point 5 cm from the corner, whereas a place on the table predicts the corner itself. The ball stays at the corner in both edits, so Non-GM agents have the same target as GM agents, and each condition plays 500 held-out starts per seed. Without any edit we also compare the spread of the arrival postures along the arm’s null space, the joint directions that leave the gripper position unchanged, with their spread at the start, which a posture target would collapse.

Results: is the proxy a target or a direction?

In both environments the proxy is a target. On the 168 detour states GM policies choose corner-progress actions on 0.948 of states, every one of them outside the up-right family, whereas Non-GM policies make corner progress on 0.173, cheese progress on 0.498 and neutral moves on 0.262 (Table 10). The GM agent thus follows the corner where this requires moving down or left, which a fixed up-right rule would not do. In LocationReach GM agents reach the corner first in 0.993 of the off-path episodes and the ball first in none, whereas Non-GM agents reach the ball first in 0.999 from the same starts (Fig. 22e and f show two of these episodes). On average the GM gripper comes within 1.4 cm of the corner and 16.3 cm of the ball, and the Non-GM gripper within 0.9 cm of the ball and 19.1 cm of the corner (Fig. 22h). The GM paths converge on the corner from starts all over the workspace (Fig. 22g), as a target does and a fixed direction of movement does not. We score the first visit. What GM agents do after it varies between checkpoints of the same seed (hold rate 0.92 to 0.03 over the last evaluations of one seed, against 0.75 to 0.9 for Non-GM agents), and since training ends an episode at the first contact this behaviour was never trained.

Results: what kind of place is the target?

In Maze the target is a spatial relation, not a remembered place. After the global corner edit, on the states where the first steps toward the old and the new corner differ, GM agents step toward the new corner in 0.909\pm 0.01 of cases and toward the old one in 0.013\pm 0.02, and in greedy rollouts they reach the new corner first in 0.748\pm 0.03 and the old one first in none, although the same agents reach the old corner in 0.857\pm 0.02 on the unedited mazes (one example level in Fig. 22a and b). Taking the first eligible state of each level, a GM checkpoint reaches the old corner unedited and the new extreme after the edit, without touching the old corner or ending on the cheese, on 0.557 of states (95% CI 0.429 to 0.685 over the three seeds). Non-GM agents also step toward the new corner (0.696\pm 0.09), but on their states this step points to the cheese in 69\% of cases, against 26\% to 32\% for GM agents, and they never reach the new corner first, so the first step alone does not separate the cohorts and reaching the new corner first does. With the offset corridor they reach the new extreme first in 0.792\pm 0.05 of rollouts and the old one in 0.114\pm 0.09, against 0.932\pm 0.04 for the old corner without the edit (Fig. 22c). Newly opened space alone does not attract them. After the global edit GM agents enter the top-right corridor in 0.925\pm 0.03 of rollouts and the bottom-left one in 0.012\pm 0.02 (Fig. 22d), and the bottom-left corridor of the offset edit is never entered, while the agents still reach the old corner first in 0.943\pm 0.04 of rollouts, as they do without any edit. Non-GM agents enter the bottom-left corridor of the global edit in 0.036\pm 0.01 of rollouts. The proxy is therefore the top-right extreme of the current maze layout, not a remembered location and not the screen’s corner cell. A target defined as the reachable cell nearest the screen corner coincides with this extreme in both edits and is not ruled out. Mini et al. (2023) report qualitatively that the opened corner attracts the agent, and the two edits with their bottom-left controls turn this into a measured test of what kind of place the target is.