Introducing Light REACT

Building Resilient Whole-Body Intelligence for Scalable Deployment

We present Light REACT, our first step toward scalable deployment at the control level. It targets an important yet underexplored challenge for scalable deployment: REsilient humAnoid ConTrol (REACT). Through whole-body in-context learning enabled by the Transformer architecture, Light REACT uses its recent interaction trajectory with the physical environment to infer the effects of diverse disruptions—including external forces, hardware impairments, and environmental constraints—and respond with adaptive whole-body skills. This resilience provides a control foundation for robots to recover from disruptions and resume their tasks with less human intervention, reducing prolonged task interruptions as deployment scales.

01 · SCALABLE DEPLOYMENT

Resilience: The Last Mile of Scalable Deployment

Scalable deployment aims to create a data flywheel by deploying robots at scale, feeding real-world experience back into model improvement. As deployment scales, robots will inevitably encounter disturbances, falls, and hardware failures. Such events can cause physical control to break down, bringing task execution to a halt. This makes large-scale deployment fragile and a data flywheel difficult to establish.

We therefore need to build resilience into the whole-body control module, enabling it to handle disturbances, recover from falls, and adapt to hardware failures. Hardware damage not only makes control harder, but also changes which behaviors the body can physically support. The goal is to preserve mobility, not a particular gait—so that scalable deployment remains robust even when the body’s capabilities change.

Today’s whole-body controllers1,2 are trained with simulated pushes to improve balance and prevent falls. For humanoids, however, resilience goes beyond avoiding falls. It requires recovering3,4 from the ground, maintaining mobility under hardware damage5,6, and finding alternative ways to move when conventional upright walking is no longer physically feasible.

Existing works address individual aspects of resilience, but two challenges remain: covering a broader range of hardware damage and unifying the capabilities for handling all damage cases within a single generalist policy. To address them, we first propose a resilience pyramid that organizes disruptions into a clear hierarchy and specifies the behaviors required at each level. Guided by this hierarchy, we scale up the number and diversity of damage cases, then develop a complete training recipe to train a single resilient whole-body controller capable of handling disruptions at every level of the pyramid. The result is Light REACT, laying a resilient foundation for scalable deployment.

01A · THE RESILIENCE PYRAMID

Rethinking Humanoid Resilience

Our philosophy is to make full use of the humanoid’s remaining whole-body capabilities to maintain the best possible velocity tracking under hardware damage. Walking, limping, hopping, and crawling are different ways of pursuing the same control objective, using the motion, actuation, and support the body can still provide. The resilience pyramid organizes the required capabilities into a hierarchy: maintaining balance under pushes, recovering after falls, adapting both recovery and locomotion to hardware damage, and crawling when upright locomotion is no longer feasible. A complete resilient controller should span this hierarchy, rather than address its individual pieces in isolation.

Keep moving without walking.

When the legs can no longer support a bipedal gait, recruit the arms and transition to crawling.

Recover, then move differently.

Infer the changed body, recover from falls, and locomote with the joints that remain usable.

Recover and resume walking.

When an extreme push causes a fall, get back up and resume walking.

Maintain balance.

Maintain balance and keep walking under moderate pushes.

02 · TRAINING RECIPE

Teacher Acquisition. Student Distillation. Alignment.

Light REACT follows a three-stage training recipe: reinforcement learning (RL), distillation, and then RL again—each with a distinct objective.

The first RL stage  adopts a divide-and-conquer strategy to learn an optimal policy for each domain. For each damage family, we train separate domain-specialized teachers for recovery–locomotion and crawling. Together, these teachers cover every level of the resilience pyramid.

The second distillation stage  consolidates the separate teacher policies into a single deployable student policy, aiming to maximize coverage of the teachers’ capabilities. We compare MLP, RNN, and Transformer students and find that the Transformer achieves the broadest coverage of teacher capabilities, as its explicit context window enables Whole-Body In-Context Learning from recent interaction history.

The last RL stage  aligns the student’s behavior with human preferences. We aim to encourage upright recovery and locomotion whenever the body permits, while retaining crawling as a fallback. Distillation alone does not establish this preference, as both walking and crawling can track velocity commands under the same damage. We implement this alignment through RL with rewards that encode an upright preference. In our experiments, only the Transformer achieves this objective; the MLP and RNN instead collapse toward crawling-dominated behavior.

Multiple teachers,
high coverage

Specialized rewards & curricula

Loading teacher motions…

6
teachers= 3 damage families × 2 skill sets
Stage 1 · Teacher RL

One model, uncertain preference

Teacher action supervision
ONE MODELGeneralist Transformer
Student rolloutSkill distribution
Stage 2 · Distillation

One model,
aligned preference

Upright-preference rewards
SAME MODELAligned Generalist
Student rolloutSkill distribution
Stage 3 · Preference RL
03 · STAGE 1 · DOMAIN-SPECIALIZED TEACHER TRAINING

Six Teachers: 3 Damage Families × 2 Skill Sets

We consider three damage families: Actuation Loss sets the affected motors’ active torque output to zero; Joint Locking holds the affected joints at a fixed angle; and Knee-Fold Constraint keeps the knee bent and prevents it from fully straightening.

Together, these three damage families cover 196 cases. Actuation Loss and Joint Locking each include 86 cases: 12 single-joint cases covering all leg joints, all 66 two-joint combinations, and eight chain cases affecting 3–6 joints along one leg. Knee-Fold Constraint includes 24 cases, combining the left and right knees with 12 flexion-angle intervals spanning 90°–150°.

Across these damage families, we target two skill sets: Recovery–Locomotion combines fall recovery with upright locomotion, while Crawling enables contact-rich, whole-body locomotion.

03B

Teacher Training Details

We train all six teachers independently using PPO7 and AMP8.

We train Recovery–Locomotion teachers in two stages. First, we use RL from scratch to learn fall recovery, guided by getting-up reference motions for AMP and an external upward-assistance curriculum. We then resume training from the recovery checkpoint and introduce velocity-tracking rewards to learn locomotion, using both getting-up and locomotion reference motions. At this stage, we remove the external assistance.

We train Crawling teachers from scratch, using crawling reference motions for AMP and the same velocity-tracking rewards. A dedicated prone-orientation reward encourages the robot to turn from a fallen pose into a crawling posture.

For each skill set, we train three separate teachers, each with damage injected from one family: Actuation Loss, Joint Locking, or Knee-Fold Constraint. This gives one domain-specialized teacher for each of the six damage–skill combinations.

During training, each teacher also receives a 21-dimensional binary damage indicator: one entry per joint, with 1 marking an affected joint and 0 marking an unaffected joint. This privileged information tells the teacher which joints are impaired, allowing it to focus on learning the best behavior for the known damage condition rather than having to identify the damage itself. The distilled student does not receive this indicator.

Damage familyActuation Loss

21-Dimensional Damage Indicator

1 · actuation loss   0 · unaffected

86 cases · 12 single-motor, 66 two-motor, and 8 chain cases.

04 · STAGE 2 · MULTI-TEACHER DISTILLATION

From Specialists to a Generalist

Our six specialist teachers learn complementary behaviors with access to privileged fault labels. These labels are unavailable at deployment, so we use DAgger9-based multi-teacher distillation to consolidate their capabilities into a single policy that receives only velocity commands (vx, vy, and yaw rate) and proprioception. During distillation, the student interacts with the environment, while the corresponding teacher provides action targets for the states the student visits. Rather than being told the fault, the student must infer what has gone wrong with its body from its past interactions with the environment.

04A · STUDENT POLICY ARCHITECTURES

One Observation, Recurrent Memory, or Explicit Context?

We consider three student architectures: a memoryless MLP using a single observation, an RNN, and a Transformer with a 64-frame context window. All three use the same observation inputs: velocity commands and proprioception. This comparison examines how temporal memory helps a student infer the body it is controlling.

01 · SINGLE FRAME

MLP

Current observation only.

NO MEMORY
02 · RECURRENT HIDDEN STATE

RNN

GRU (512 hidden units) + MLP action head.

LEARNED MEMORY
03 · EXPLICIT CONTEXT

Transformer

64-frame causal context window.

SEQUENCE MEMORY

Same frames. Different memories.

One observation stream, delivered to all three policies at the same time.

Ten observations, three ways to consume them Each observation reaches MLP, RNN, and Transformer simultaneously. MLP uses only the current observation. The recurrent actor normalizes the observation, updates a 512-dimensional hidden state with a single-layer GRU, and passes the GRU output through an MLP with hidden widths 512, 256, and 128 to produce the action. Only the GRU hidden state is carried into the next step, not the MLP or action output. The causal Transformer attends to previously received tokens and the current token, never to future frames. All three produce an action after each observation.
04B · TEACHER BEHAVIOR COVERAGE

Evaluating Teacher Behavior Coverage

The key objective of distillation is skill coverage: one student should retain both recovery–locomotion and crawling across damage families. We ask two questions: how closely does it match the teachers’ actions, and what is its skill distribution during its own rollouts? Teacher-forcing action MSE addresses the first; student rollouts address the second.

The transformer achieves the lowest action MSE.

We first roll out each teacher to collect observation–action trajectories. We then feed the corresponding velocity commands and proprioceptive observations to each student in temporal order and compute the mean squared error (MSE) between its predicted actions and the recorded teacher actions.

With a 64-frame context window, the Transformer fits teacher trajectories substantially more closely, suggesting better coverage of teacher behavior.

MLP RNN · GRU 512 Transformer · Context Window 64
View six-teacher breakdown Hide six-teacher breakdown
Actuation Loss

Recovery–Locomotion teacher

Actuation Loss · Recovery–Locomotion teacher: teacher-forcing action MSE Before reset fix, 1–20 seconds. Three student models, 21 checkpoints from iteration 0 to 5000. Solid lines and hollow markers. Shared logarithmic vertical axis from 0.1 to 10; lower is better. 0.1 0.2 0.5 1 2 5 10 0 1k 2k 3k 4k 5k Training IterationAction MSE · log scale MLP · before reset fix · 1–20 s · iteration 0 · MSE 6.251916 MLP · before reset fix · 1–20 s · iteration 250 · MSE 1.515322 MLP · before reset fix · 1–20 s · iteration 500 · MSE 1.420389 MLP · before reset fix · 1–20 s · iteration 750 · MSE 1.251151 MLP · before reset fix · 1–20 s · iteration 1000 · MSE 1.317365 MLP · before reset fix · 1–20 s · iteration 1250 · MSE 1.339196 MLP · before reset fix · 1–20 s · iteration 1500 · MSE 1.233332 MLP · before reset fix · 1–20 s · iteration 1750 · MSE 1.257764 MLP · before reset fix · 1–20 s · iteration 2000 · MSE 1.246482 MLP · before reset fix · 1–20 s · iteration 2250 · MSE 1.319502 MLP · before reset fix · 1–20 s · iteration 2500 · MSE 1.331300 MLP · before reset fix · 1–20 s · iteration 2750 · MSE 1.338234 MLP · before reset fix · 1–20 s · iteration 3000 · MSE 1.289015 MLP · before reset fix · 1–20 s · iteration 3250 · MSE 1.279657 MLP · before reset fix · 1–20 s · iteration 3500 · MSE 1.385337 MLP · before reset fix · 1–20 s · iteration 3750 · MSE 1.363007 MLP · before reset fix · 1–20 s · iteration 4000 · MSE 1.212215 MLP · before reset fix · 1–20 s · iteration 4250 · MSE 1.318515 MLP · before reset fix · 1–20 s · iteration 4500 · MSE 1.410409 MLP · before reset fix · 1–20 s · iteration 4750 · MSE 1.263513 MLP · before reset fix · 1–20 s · iteration 5000 · MSE 1.321070 RNN · GRU 512 · before reset fix · 1–20 s · iteration 0 · MSE 5.550894 RNN · GRU 512 · before reset fix · 1–20 s · iteration 250 · MSE 1.794627 RNN · GRU 512 · before reset fix · 1–20 s · iteration 500 · MSE 1.273323 RNN · GRU 512 · before reset fix · 1–20 s · iteration 750 · MSE 1.078419 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1000 · MSE 0.979417 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1250 · MSE 0.946337 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1500 · MSE 0.903510 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1750 · MSE 0.843949 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2000 · MSE 0.895879 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2250 · MSE 0.862125 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2500 · MSE 0.834982 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2750 · MSE 0.846429 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3000 · MSE 0.813552 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3250 · MSE 0.749821 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3500 · MSE 0.811459 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3750 · MSE 0.745603 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4000 · MSE 0.765776 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4250 · MSE 0.776132 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4500 · MSE 0.765550 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4750 · MSE 0.737695 RNN · GRU 512 · before reset fix · 1–20 s · iteration 5000 · MSE 0.776511 Transformer · C64 · before reset fix · 1–20 s · iteration 0 · MSE 6.025502 Transformer · C64 · before reset fix · 1–20 s · iteration 250 · MSE 0.390645 Transformer · C64 · before reset fix · 1–20 s · iteration 500 · MSE 0.321330 Transformer · C64 · before reset fix · 1–20 s · iteration 750 · MSE 0.324796 Transformer · C64 · before reset fix · 1–20 s · iteration 1000 · MSE 0.307778 Transformer · C64 · before reset fix · 1–20 s · iteration 1250 · MSE 0.287949 Transformer · C64 · before reset fix · 1–20 s · iteration 1500 · MSE 0.293046 Transformer · C64 · before reset fix · 1–20 s · iteration 1750 · MSE 0.294527 Transformer · C64 · before reset fix · 1–20 s · iteration 2000 · MSE 0.276460 Transformer · C64 · before reset fix · 1–20 s · iteration 2250 · MSE 0.288731 Transformer · C64 · before reset fix · 1–20 s · iteration 2500 · MSE 0.290002 Transformer · C64 · before reset fix · 1–20 s · iteration 2750 · MSE 0.289112 Transformer · C64 · before reset fix · 1–20 s · iteration 3000 · MSE 0.277033 Transformer · C64 · before reset fix · 1–20 s · iteration 3250 · MSE 0.278571 Transformer · C64 · before reset fix · 1–20 s · iteration 3500 · MSE 0.271198 Transformer · C64 · before reset fix · 1–20 s · iteration 3750 · MSE 0.275027 Transformer · C64 · before reset fix · 1–20 s · iteration 4000 · MSE 0.280771 Transformer · C64 · before reset fix · 1–20 s · iteration 4250 · MSE 0.273130 Transformer · C64 · before reset fix · 1–20 s · iteration 4500 · MSE 0.266860 Transformer · C64 · before reset fix · 1–20 s · iteration 4750 · MSE 0.268408 Transformer · C64 · before reset fix · 1–20 s · iteration 5000 · MSE 0.276714
Actuation Loss

Crawl teacher

Actuation Loss · Crawl teacher: teacher-forcing action MSE Before reset fix, 1–20 seconds. Three student models, 21 checkpoints from iteration 0 to 5000. Solid lines and hollow markers. Shared logarithmic vertical axis from 0.1 to 10; lower is better. 0.1 0.2 0.5 1 2 5 10 0 1k 2k 3k 4k 5k Training IterationAction MSE · log scale MLP · before reset fix · 1–20 s · iteration 0 · MSE 3.642987 MLP · before reset fix · 1–20 s · iteration 250 · MSE 0.806101 MLP · before reset fix · 1–20 s · iteration 500 · MSE 0.685994 MLP · before reset fix · 1–20 s · iteration 750 · MSE 0.648027 MLP · before reset fix · 1–20 s · iteration 1000 · MSE 0.734544 MLP · before reset fix · 1–20 s · iteration 1250 · MSE 0.730543 MLP · before reset fix · 1–20 s · iteration 1500 · MSE 0.615526 MLP · before reset fix · 1–20 s · iteration 1750 · MSE 0.792404 MLP · before reset fix · 1–20 s · iteration 2000 · MSE 0.587043 MLP · before reset fix · 1–20 s · iteration 2250 · MSE 0.583902 MLP · before reset fix · 1–20 s · iteration 2500 · MSE 0.550559 MLP · before reset fix · 1–20 s · iteration 2750 · MSE 0.668224 MLP · before reset fix · 1–20 s · iteration 3000 · MSE 0.618478 MLP · before reset fix · 1–20 s · iteration 3250 · MSE 0.814946 MLP · before reset fix · 1–20 s · iteration 3500 · MSE 0.753148 MLP · before reset fix · 1–20 s · iteration 3750 · MSE 0.840673 MLP · before reset fix · 1–20 s · iteration 4000 · MSE 0.823317 MLP · before reset fix · 1–20 s · iteration 4250 · MSE 0.998673 MLP · before reset fix · 1–20 s · iteration 4500 · MSE 0.883556 MLP · before reset fix · 1–20 s · iteration 4750 · MSE 0.934451 MLP · before reset fix · 1–20 s · iteration 5000 · MSE 0.848847 RNN · GRU 512 · before reset fix · 1–20 s · iteration 0 · MSE 5.932160 RNN · GRU 512 · before reset fix · 1–20 s · iteration 250 · MSE 0.845363 RNN · GRU 512 · before reset fix · 1–20 s · iteration 500 · MSE 0.636102 RNN · GRU 512 · before reset fix · 1–20 s · iteration 750 · MSE 0.701693 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1000 · MSE 0.710458 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1250 · MSE 0.711106 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1500 · MSE 0.672331 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1750 · MSE 0.600294 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2000 · MSE 0.633625 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2250 · MSE 0.630101 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2500 · MSE 0.635762 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2750 · MSE 0.555243 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3000 · MSE 0.720597 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3250 · MSE 0.585893 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3500 · MSE 0.609062 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3750 · MSE 0.700857 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4000 · MSE 0.581887 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4250 · MSE 0.662344 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4500 · MSE 0.546766 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4750 · MSE 0.684805 RNN · GRU 512 · before reset fix · 1–20 s · iteration 5000 · MSE 0.549155 Transformer · C64 · before reset fix · 1–20 s · iteration 0 · MSE 7.697915 Transformer · C64 · before reset fix · 1–20 s · iteration 250 · MSE 0.188659 Transformer · C64 · before reset fix · 1–20 s · iteration 500 · MSE 0.159395 Transformer · C64 · before reset fix · 1–20 s · iteration 750 · MSE 0.170932 Transformer · C64 · before reset fix · 1–20 s · iteration 1000 · MSE 0.158391 Transformer · C64 · before reset fix · 1–20 s · iteration 1250 · MSE 0.157308 Transformer · C64 · before reset fix · 1–20 s · iteration 1500 · MSE 0.155963 Transformer · C64 · before reset fix · 1–20 s · iteration 1750 · MSE 0.146177 Transformer · C64 · before reset fix · 1–20 s · iteration 2000 · MSE 0.172913 Transformer · C64 · before reset fix · 1–20 s · iteration 2250 · MSE 0.162600 Transformer · C64 · before reset fix · 1–20 s · iteration 2500 · MSE 0.155362 Transformer · C64 · before reset fix · 1–20 s · iteration 2750 · MSE 0.184196 Transformer · C64 · before reset fix · 1–20 s · iteration 3000 · MSE 0.155859 Transformer · C64 · before reset fix · 1–20 s · iteration 3250 · MSE 0.175752 Transformer · C64 · before reset fix · 1–20 s · iteration 3500 · MSE 0.178543 Transformer · C64 · before reset fix · 1–20 s · iteration 3750 · MSE 0.149912 Transformer · C64 · before reset fix · 1–20 s · iteration 4000 · MSE 0.180302 Transformer · C64 · before reset fix · 1–20 s · iteration 4250 · MSE 0.171253 Transformer · C64 · before reset fix · 1–20 s · iteration 4500 · MSE 0.196835 Transformer · C64 · before reset fix · 1–20 s · iteration 4750 · MSE 0.177196 Transformer · C64 · before reset fix · 1–20 s · iteration 5000 · MSE 0.163406
Joint Locking

Recovery–Locomotion teacher

Joint Locking · Recovery–Locomotion teacher: teacher-forcing action MSE Before reset fix, 1–20 seconds. Three student models, 21 checkpoints from iteration 0 to 5000. Solid lines and hollow markers. Shared logarithmic vertical axis from 0.1 to 10; lower is better. 0.1 0.2 0.5 1 2 5 10 0 1k 2k 3k 4k 5k Training IterationAction MSE · log scale MLP · before reset fix · 1–20 s · iteration 0 · MSE 4.924906 MLP · before reset fix · 1–20 s · iteration 250 · MSE 0.940690 MLP · before reset fix · 1–20 s · iteration 500 · MSE 0.783629 MLP · before reset fix · 1–20 s · iteration 750 · MSE 0.684654 MLP · before reset fix · 1–20 s · iteration 1000 · MSE 0.778400 MLP · before reset fix · 1–20 s · iteration 1250 · MSE 0.827361 MLP · before reset fix · 1–20 s · iteration 1500 · MSE 0.719279 MLP · before reset fix · 1–20 s · iteration 1750 · MSE 0.739484 MLP · before reset fix · 1–20 s · iteration 2000 · MSE 0.765794 MLP · before reset fix · 1–20 s · iteration 2250 · MSE 0.805137 MLP · before reset fix · 1–20 s · iteration 2500 · MSE 0.804560 MLP · before reset fix · 1–20 s · iteration 2750 · MSE 0.749827 MLP · before reset fix · 1–20 s · iteration 3000 · MSE 0.772196 MLP · before reset fix · 1–20 s · iteration 3250 · MSE 0.782075 MLP · before reset fix · 1–20 s · iteration 3500 · MSE 0.845470 MLP · before reset fix · 1–20 s · iteration 3750 · MSE 0.890563 MLP · before reset fix · 1–20 s · iteration 4000 · MSE 0.736190 MLP · before reset fix · 1–20 s · iteration 4250 · MSE 0.787598 MLP · before reset fix · 1–20 s · iteration 4500 · MSE 0.779621 MLP · before reset fix · 1–20 s · iteration 4750 · MSE 0.747387 MLP · before reset fix · 1–20 s · iteration 5000 · MSE 0.825798 RNN · GRU 512 · before reset fix · 1–20 s · iteration 0 · MSE 5.416523 RNN · GRU 512 · before reset fix · 1–20 s · iteration 250 · MSE 0.971981 RNN · GRU 512 · before reset fix · 1–20 s · iteration 500 · MSE 0.659942 RNN · GRU 512 · before reset fix · 1–20 s · iteration 750 · MSE 0.626854 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1000 · MSE 0.587051 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1250 · MSE 0.584336 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1500 · MSE 0.498437 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1750 · MSE 0.544787 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2000 · MSE 0.475223 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2250 · MSE 0.546272 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2500 · MSE 0.535114 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2750 · MSE 0.489626 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3000 · MSE 0.545164 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3250 · MSE 0.435614 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3500 · MSE 0.482620 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3750 · MSE 0.477214 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4000 · MSE 0.444985 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4250 · MSE 0.459124 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4500 · MSE 0.446596 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4750 · MSE 0.433086 RNN · GRU 512 · before reset fix · 1–20 s · iteration 5000 · MSE 0.445657 Transformer · C64 · before reset fix · 1–20 s · iteration 0 · MSE 5.459383 Transformer · C64 · before reset fix · 1–20 s · iteration 250 · MSE 0.237287 Transformer · C64 · before reset fix · 1–20 s · iteration 500 · MSE 0.202419 Transformer · C64 · before reset fix · 1–20 s · iteration 750 · MSE 0.202466 Transformer · C64 · before reset fix · 1–20 s · iteration 1000 · MSE 0.198044 Transformer · C64 · before reset fix · 1–20 s · iteration 1250 · MSE 0.184061 Transformer · C64 · before reset fix · 1–20 s · iteration 1500 · MSE 0.182314 Transformer · C64 · before reset fix · 1–20 s · iteration 1750 · MSE 0.180311 Transformer · C64 · before reset fix · 1–20 s · iteration 2000 · MSE 0.175570 Transformer · C64 · before reset fix · 1–20 s · iteration 2250 · MSE 0.183885 Transformer · C64 · before reset fix · 1–20 s · iteration 2500 · MSE 0.179379 Transformer · C64 · before reset fix · 1–20 s · iteration 2750 · MSE 0.174034 Transformer · C64 · before reset fix · 1–20 s · iteration 3000 · MSE 0.175469 Transformer · C64 · before reset fix · 1–20 s · iteration 3250 · MSE 0.176384 Transformer · C64 · before reset fix · 1–20 s · iteration 3500 · MSE 0.167544 Transformer · C64 · before reset fix · 1–20 s · iteration 3750 · MSE 0.174329 Transformer · C64 · before reset fix · 1–20 s · iteration 4000 · MSE 0.168621 Transformer · C64 · before reset fix · 1–20 s · iteration 4250 · MSE 0.166450 Transformer · C64 · before reset fix · 1–20 s · iteration 4500 · MSE 0.167093 Transformer · C64 · before reset fix · 1–20 s · iteration 4750 · MSE 0.172294 Transformer · C64 · before reset fix · 1–20 s · iteration 5000 · MSE 0.166502
Joint Locking

Crawl teacher

Joint Locking · Crawl teacher: teacher-forcing action MSE Before reset fix, 1–20 seconds. Three student models, 21 checkpoints from iteration 0 to 5000. Solid lines and hollow markers. Shared logarithmic vertical axis from 0.1 to 10; lower is better. 0.1 0.2 0.5 1 2 5 10 0 1k 2k 3k 4k 5k Training IterationAction MSE · log scale MLP · before reset fix · 1–20 s · iteration 0 · MSE 4.114511 MLP · before reset fix · 1–20 s · iteration 250 · MSE 0.665278 MLP · before reset fix · 1–20 s · iteration 500 · MSE 0.553041 MLP · before reset fix · 1–20 s · iteration 750 · MSE 0.565747 MLP · before reset fix · 1–20 s · iteration 1000 · MSE 0.560735 MLP · before reset fix · 1–20 s · iteration 1250 · MSE 0.529400 MLP · before reset fix · 1–20 s · iteration 1500 · MSE 0.591217 MLP · before reset fix · 1–20 s · iteration 1750 · MSE 0.573996 MLP · before reset fix · 1–20 s · iteration 2000 · MSE 0.575048 MLP · before reset fix · 1–20 s · iteration 2250 · MSE 0.535547 MLP · before reset fix · 1–20 s · iteration 2500 · MSE 0.565064 MLP · before reset fix · 1–20 s · iteration 2750 · MSE 0.567337 MLP · before reset fix · 1–20 s · iteration 3000 · MSE 0.511967 MLP · before reset fix · 1–20 s · iteration 3250 · MSE 0.582947 MLP · before reset fix · 1–20 s · iteration 3500 · MSE 0.651079 MLP · before reset fix · 1–20 s · iteration 3750 · MSE 0.662391 MLP · before reset fix · 1–20 s · iteration 4000 · MSE 0.630123 MLP · before reset fix · 1–20 s · iteration 4250 · MSE 0.707195 MLP · before reset fix · 1–20 s · iteration 4500 · MSE 0.653135 MLP · before reset fix · 1–20 s · iteration 4750 · MSE 0.652262 MLP · before reset fix · 1–20 s · iteration 5000 · MSE 0.608642 RNN · GRU 512 · before reset fix · 1–20 s · iteration 0 · MSE 5.333927 RNN · GRU 512 · before reset fix · 1–20 s · iteration 250 · MSE 0.948684 RNN · GRU 512 · before reset fix · 1–20 s · iteration 500 · MSE 0.659172 RNN · GRU 512 · before reset fix · 1–20 s · iteration 750 · MSE 0.591981 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1000 · MSE 0.546554 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1250 · MSE 0.589958 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1500 · MSE 0.539636 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1750 · MSE 0.536509 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2000 · MSE 0.508051 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2250 · MSE 0.550025 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2500 · MSE 0.543106 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2750 · MSE 0.458956 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3000 · MSE 0.611128 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3250 · MSE 0.514984 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3500 · MSE 0.576290 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3750 · MSE 0.529954 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4000 · MSE 0.526885 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4250 · MSE 0.507905 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4500 · MSE 0.467486 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4750 · MSE 0.526474 RNN · GRU 512 · before reset fix · 1–20 s · iteration 5000 · MSE 0.449340 Transformer · C64 · before reset fix · 1–20 s · iteration 0 · MSE 6.700801 Transformer · C64 · before reset fix · 1–20 s · iteration 250 · MSE 0.195222 Transformer · C64 · before reset fix · 1–20 s · iteration 500 · MSE 0.156679 Transformer · C64 · before reset fix · 1–20 s · iteration 750 · MSE 0.175678 Transformer · C64 · before reset fix · 1–20 s · iteration 1000 · MSE 0.160512 Transformer · C64 · before reset fix · 1–20 s · iteration 1250 · MSE 0.152912 Transformer · C64 · before reset fix · 1–20 s · iteration 1500 · MSE 0.160654 Transformer · C64 · before reset fix · 1–20 s · iteration 1750 · MSE 0.169030 Transformer · C64 · before reset fix · 1–20 s · iteration 2000 · MSE 0.163851 Transformer · C64 · before reset fix · 1–20 s · iteration 2250 · MSE 0.159216 Transformer · C64 · before reset fix · 1–20 s · iteration 2500 · MSE 0.158614 Transformer · C64 · before reset fix · 1–20 s · iteration 2750 · MSE 0.158046 Transformer · C64 · before reset fix · 1–20 s · iteration 3000 · MSE 0.155705 Transformer · C64 · before reset fix · 1–20 s · iteration 3250 · MSE 0.163861 Transformer · C64 · before reset fix · 1–20 s · iteration 3500 · MSE 0.157903 Transformer · C64 · before reset fix · 1–20 s · iteration 3750 · MSE 0.158089 Transformer · C64 · before reset fix · 1–20 s · iteration 4000 · MSE 0.163666 Transformer · C64 · before reset fix · 1–20 s · iteration 4250 · MSE 0.164482 Transformer · C64 · before reset fix · 1–20 s · iteration 4500 · MSE 0.168817 Transformer · C64 · before reset fix · 1–20 s · iteration 4750 · MSE 0.159577 Transformer · C64 · before reset fix · 1–20 s · iteration 5000 · MSE 0.154600
Knee-Fold Constraint

Recovery–Locomotion teacher

Knee-Fold Constraint · Recovery–Locomotion teacher: teacher-forcing action MSE Before reset fix, 1–20 seconds. Three student models, 21 checkpoints from iteration 0 to 5000. Solid lines and hollow markers. Shared logarithmic vertical axis from 0.1 to 10; lower is better. 0.1 0.2 0.5 1 2 5 10 0 1k 2k 3k 4k 5k Training IterationAction MSE · log scale MLP · before reset fix · 1–20 s · iteration 0 · MSE 6.663657 MLP · before reset fix · 1–20 s · iteration 250 · MSE 1.083375 MLP · before reset fix · 1–20 s · iteration 500 · MSE 1.053528 MLP · before reset fix · 1–20 s · iteration 750 · MSE 1.005097 MLP · before reset fix · 1–20 s · iteration 1000 · MSE 1.099558 MLP · before reset fix · 1–20 s · iteration 1250 · MSE 0.974014 MLP · before reset fix · 1–20 s · iteration 1500 · MSE 0.909377 MLP · before reset fix · 1–20 s · iteration 1750 · MSE 0.944938 MLP · before reset fix · 1–20 s · iteration 2000 · MSE 1.014213 MLP · before reset fix · 1–20 s · iteration 2250 · MSE 1.042208 MLP · before reset fix · 1–20 s · iteration 2500 · MSE 1.071432 MLP · before reset fix · 1–20 s · iteration 2750 · MSE 1.102842 MLP · before reset fix · 1–20 s · iteration 3000 · MSE 1.048758 MLP · before reset fix · 1–20 s · iteration 3250 · MSE 1.072428 MLP · before reset fix · 1–20 s · iteration 3500 · MSE 1.066061 MLP · before reset fix · 1–20 s · iteration 3750 · MSE 1.176874 MLP · before reset fix · 1–20 s · iteration 4000 · MSE 1.194930 MLP · before reset fix · 1–20 s · iteration 4250 · MSE 1.358177 MLP · before reset fix · 1–20 s · iteration 4500 · MSE 1.334164 MLP · before reset fix · 1–20 s · iteration 4750 · MSE 1.314622 MLP · before reset fix · 1–20 s · iteration 5000 · MSE 1.205720 RNN · GRU 512 · before reset fix · 1–20 s · iteration 0 · MSE 8.143319 RNN · GRU 512 · before reset fix · 1–20 s · iteration 250 · MSE 1.423795 RNN · GRU 512 · before reset fix · 1–20 s · iteration 500 · MSE 0.867944 RNN · GRU 512 · before reset fix · 1–20 s · iteration 750 · MSE 0.860219 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1000 · MSE 0.814759 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1250 · MSE 0.827778 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1500 · MSE 0.651986 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1750 · MSE 0.747128 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2000 · MSE 0.728714 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2250 · MSE 0.628040 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2500 · MSE 0.686247 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2750 · MSE 0.652882 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3000 · MSE 0.721020 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3250 · MSE 0.589920 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3500 · MSE 0.672799 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3750 · MSE 0.621405 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4000 · MSE 0.688059 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4250 · MSE 0.669721 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4500 · MSE 0.648473 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4750 · MSE 0.653054 RNN · GRU 512 · before reset fix · 1–20 s · iteration 5000 · MSE 0.672859 Transformer · C64 · before reset fix · 1–20 s · iteration 0 · MSE 8.769456 Transformer · C64 · before reset fix · 1–20 s · iteration 250 · MSE 0.289977 Transformer · C64 · before reset fix · 1–20 s · iteration 500 · MSE 0.258496 Transformer · C64 · before reset fix · 1–20 s · iteration 750 · MSE 0.264584 Transformer · C64 · before reset fix · 1–20 s · iteration 1000 · MSE 0.250805 Transformer · C64 · before reset fix · 1–20 s · iteration 1250 · MSE 0.231923 Transformer · C64 · before reset fix · 1–20 s · iteration 1500 · MSE 0.240980 Transformer · C64 · before reset fix · 1–20 s · iteration 1750 · MSE 0.246261 Transformer · C64 · before reset fix · 1–20 s · iteration 2000 · MSE 0.218082 Transformer · C64 · before reset fix · 1–20 s · iteration 2250 · MSE 0.233863 Transformer · C64 · before reset fix · 1–20 s · iteration 2500 · MSE 0.225357 Transformer · C64 · before reset fix · 1–20 s · iteration 2750 · MSE 0.240794 Transformer · C64 · before reset fix · 1–20 s · iteration 3000 · MSE 0.236199 Transformer · C64 · before reset fix · 1–20 s · iteration 3250 · MSE 0.231707 Transformer · C64 · before reset fix · 1–20 s · iteration 3500 · MSE 0.229708 Transformer · C64 · before reset fix · 1–20 s · iteration 3750 · MSE 0.218228 Transformer · C64 · before reset fix · 1–20 s · iteration 4000 · MSE 0.232295 Transformer · C64 · before reset fix · 1–20 s · iteration 4250 · MSE 0.219162 Transformer · C64 · before reset fix · 1–20 s · iteration 4500 · MSE 0.212173 Transformer · C64 · before reset fix · 1–20 s · iteration 4750 · MSE 0.215667 Transformer · C64 · before reset fix · 1–20 s · iteration 5000 · MSE 0.214877
Knee-Fold Constraint

Crawl teacher

Knee-Fold Constraint · Crawl teacher: teacher-forcing action MSE Before reset fix, 1–20 seconds. Three student models, 21 checkpoints from iteration 0 to 5000. Solid lines and hollow markers. Shared logarithmic vertical axis from 0.1 to 10; lower is better. 0.1 0.2 0.5 1 2 5 10 0 1k 2k 3k 4k 5k Training IterationAction MSE · log scale MLP · before reset fix · 1–20 s · iteration 0 · MSE 4.934784 MLP · before reset fix · 1–20 s · iteration 250 · MSE 0.692527 MLP · before reset fix · 1–20 s · iteration 500 · MSE 0.567004 MLP · before reset fix · 1–20 s · iteration 750 · MSE 0.503427 MLP · before reset fix · 1–20 s · iteration 1000 · MSE 0.538685 MLP · before reset fix · 1–20 s · iteration 1250 · MSE 0.510737 MLP · before reset fix · 1–20 s · iteration 1500 · MSE 0.641172 MLP · before reset fix · 1–20 s · iteration 1750 · MSE 0.575002 MLP · before reset fix · 1–20 s · iteration 2000 · MSE 0.712157 MLP · before reset fix · 1–20 s · iteration 2250 · MSE 0.548443 MLP · before reset fix · 1–20 s · iteration 2500 · MSE 0.599787 MLP · before reset fix · 1–20 s · iteration 2750 · MSE 0.568643 MLP · before reset fix · 1–20 s · iteration 3000 · MSE 0.585934 MLP · before reset fix · 1–20 s · iteration 3250 · MSE 0.586127 MLP · before reset fix · 1–20 s · iteration 3500 · MSE 0.812708 MLP · before reset fix · 1–20 s · iteration 3750 · MSE 1.092050 MLP · before reset fix · 1–20 s · iteration 4000 · MSE 0.708377 MLP · before reset fix · 1–20 s · iteration 4250 · MSE 0.726672 MLP · before reset fix · 1–20 s · iteration 4500 · MSE 0.821008 MLP · before reset fix · 1–20 s · iteration 4750 · MSE 0.725708 MLP · before reset fix · 1–20 s · iteration 5000 · MSE 0.894022 RNN · GRU 512 · before reset fix · 1–20 s · iteration 0 · MSE 5.329335 RNN · GRU 512 · before reset fix · 1–20 s · iteration 250 · MSE 1.525910 RNN · GRU 512 · before reset fix · 1–20 s · iteration 500 · MSE 0.736853 RNN · GRU 512 · before reset fix · 1–20 s · iteration 750 · MSE 0.841665 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1000 · MSE 0.862282 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1250 · MSE 0.665470 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1500 · MSE 0.741653 RNN · GRU 512 · before reset fix · 1–20 s · iteration 1750 · MSE 0.766175 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2000 · MSE 0.603609 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2250 · MSE 0.615266 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2500 · MSE 0.561102 RNN · GRU 512 · before reset fix · 1–20 s · iteration 2750 · MSE 0.601051 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3000 · MSE 0.518533 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3250 · MSE 0.494732 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3500 · MSE 0.475545 RNN · GRU 512 · before reset fix · 1–20 s · iteration 3750 · MSE 0.490717 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4000 · MSE 0.446088 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4250 · MSE 0.528199 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4500 · MSE 0.516863 RNN · GRU 512 · before reset fix · 1–20 s · iteration 4750 · MSE 0.547209 RNN · GRU 512 · before reset fix · 1–20 s · iteration 5000 · MSE 0.449146 Transformer · C64 · before reset fix · 1–20 s · iteration 0 · MSE 5.943331 Transformer · C64 · before reset fix · 1–20 s · iteration 250 · MSE 0.150943 Transformer · C64 · before reset fix · 1–20 s · iteration 500 · MSE 0.136268 Transformer · C64 · before reset fix · 1–20 s · iteration 750 · MSE 0.131232 Transformer · C64 · before reset fix · 1–20 s · iteration 1000 · MSE 0.144224 Transformer · C64 · before reset fix · 1–20 s · iteration 1250 · MSE 0.123507 Transformer · C64 · before reset fix · 1–20 s · iteration 1500 · MSE 0.148116 Transformer · C64 · before reset fix · 1–20 s · iteration 1750 · MSE 0.147915 Transformer · C64 · before reset fix · 1–20 s · iteration 2000 · MSE 0.145317 Transformer · C64 · before reset fix · 1–20 s · iteration 2250 · MSE 0.129634 Transformer · C64 · before reset fix · 1–20 s · iteration 2500 · MSE 0.131439 Transformer · C64 · before reset fix · 1–20 s · iteration 2750 · MSE 0.139373 Transformer · C64 · before reset fix · 1–20 s · iteration 3000 · MSE 0.150550 Transformer · C64 · before reset fix · 1–20 s · iteration 3250 · MSE 0.153030 Transformer · C64 · before reset fix · 1–20 s · iteration 3500 · MSE 0.141767 Transformer · C64 · before reset fix · 1–20 s · iteration 3750 · MSE 0.128864 Transformer · C64 · before reset fix · 1–20 s · iteration 4000 · MSE 0.136807 Transformer · C64 · before reset fix · 1–20 s · iteration 4250 · MSE 0.141475 Transformer · C64 · before reset fix · 1–20 s · iteration 4500 · MSE 0.120535 Transformer · C64 · before reset fix · 1–20 s · iteration 4750 · MSE 0.138328 Transformer · C64 · before reset fix · 1–20 s · iteration 5000 · MSE 0.122323

Beyond Imitation Error: What Does the Student Actually Do?

Low imitation error alone does not establish behavioral coverage. We evaluate each student’s own rollouts by classifying every frame as Upright, Crawl, or Other, then measuring the proportion of frames in each category across damage families.

How each frame is classified
Upright

An upright posture.

  • Root height ≥ 0.40 m
  • Torso tilt from vertical ≤ 70°

Both conditions must hold. No velocity-tracking threshold is required.

Crawl

A low crawling posture that follows the command.

  • Root height < 0.35 m
  • Orientation: torso +X within 30° of vertically downward
  • Absolute forward-velocity tracking error in the crawl frame: |ex| ≤ 0.25 m/s

All three conditions must hold.

Other

Neither the full Upright definition nor the full Crawl definition is satisfied.

This includes transitions and low postures that do not meet the crawl orientation or tracking criteria.

The animations below illustrate these categories for MLP, RNN, and Transformer under the same fault. Each robot starts fallen, with all six right-leg motors unpowered, and receives a forward velocity command of vx = 0.5 m/s. The colored timeline labels each frame of the 20-second rollout.

The memoryless MLP produces predominantly Other behavior in every damage family. This is consistent with MSE-based distillation averaging between the two skill sets when a single observation provides insufficient context, yielding intermediate behavior. The RNN reduces the proportion of Other frames across all three families, suggesting that recurrent memory helps alleviate this ambiguity. With an explicit 64-frame context window, the Transformer maintains a relatively balanced mix of upright locomotion and crawling, with the lowest proportion of Other frames in every damage family.

In conclusion, these results suggest that, among the architectures tested, explicit Transformer context is the most effective at capturing the teachers’ behavioral repertoire and reducing ambiguity between the two skill sets.

Rollout frame proportion (%)
Actuation
Loss
Joint
Locking
Knee-Fold
Constraint
MLP
Actuation
Loss
Joint
Locking
Knee-Fold
Constraint
RNN
Actuation
Loss
Joint
Locking
Knee-Fold
Constraint
Transformer · 64-frame context
Upright Crawl Other
05 · STAGE 3 · PREFERENCE ALIGNMENT

Aligning with Human Preferences

Distillation brings different ways of following movement commands under the same damage into one policy. But it leaves a question open: which way should the policy move? The student may recover and walk, crawl, or fall into the Other category.

Our preference is simple: recover and move upright whenever the body allows, and crawl when it cannot. Upright locomotion keeps ground contact largely at the feet, limiting wear on limbs and joint housings. It also suits spaces built for people. Crawling remains an essential fallback when damage rules out walking.

We fine-tune the distilled students with reinforcement learning to encourage this preference. The reward favors recovery, stable upright posture, and velocity-command tracking while upright. Velocity-command tracking while crawling is still rewarded, but at a substantially lower level than upright tracking. This stage shapes how the policy uses the skills it has learned. As the comparison below shows, a shared preference can lead to different outcomes across architectures.

05A

One Preference, Different Outcomes

After fine-tuning, the Transformer shifts toward upright locomotion, while MLP and RNN shift toward crawling. The bars transition from distilled to post-RL rollout frame proportions. The transition illustrates the change before and after RL alignment.

Rollout frame proportion (%)
Actuation
Loss
Joint
Locking
Knee-Fold
Constraint
MLP
Actuation
Loss
Joint
Locking
Knee-Fold
Constraint
RNN
Actuation
Loss
Joint
Locking
Knee-Fold
Constraint
Transformer · 64-frame context
Upright Crawl Other

RL begins with the skills preserved through distillation. MLP and RNN may still struggle to express recovery, upright locomotion, and crawling as distinct, reliable behaviors. Sustained upright movement requires getting up, stabilizing the body, and tracking commands under damage. When this sequence is unreliable, crawling may offer a more accessible source of reward. As training reinforces it, opportunities to practice and improve recovery may diminish.

The same task can also pose different levels of difficulty for different models. Although the reward favors upright tracking, crawling may yield more dependable returns for a policy that struggles to remain upright and follow commands. The Transformer’s explicit temporal context may help it use recent body responses to maintain upright control under damage, giving fine-tuning a stronger basis for reinforcing the intended preference.

05B

Whole-Body In-Context Learning after RL Alignment

These real-world videos compare the Transformer before and after preference alignment under temporary damage. The impairment is removed partway through each run, showing how the two policies respond when upright locomotion becomes possible again.

Both knees overheat and lose power, then recover partway through the run. Once power is restored, the RL-aligned Transformer returns to upright locomotion, suggesting that it can use its whole-body interaction history with the environment, stored in its context window, to infer that the impairment has been removed. In contrast, the distilled policy does not make this transition in the demonstrated rollout. We also demonstrate a second scenario in which both legs are bound and the binding is cut partway through the run.

The damage

No preference· distilled policy

With preference· after preference RL

The damage

No preference· distilled policy

With preference· after preference RL

Given a velocity command, the robot first attempts to stand. When external pressure prevents it from rising, the policy uses the interaction history in its context window to infer that standing is constrained and switches to crawling to track the velocity command. Upon reaching an open area, it infers from new interactions that the environmental constraint has been removed, stands up autonomously, and continues tracking the command upright.

No damage, but environmental constraintsafter preference RL