Reasoning by Consequence:
Selective Chain-of-Thought for Driving VLAs

1NVIDIA, 2Carnegie Mellon University

Chain of thought (CoT) improves driving when the car must act and degrades it when the car should keep going.

Closed-loop HUGSIM drives of one driving VLA without CoT and with CoT. Top: when the car must act, the trajectory without CoT collides and the one with CoT passes. Bottom: when the car should keep going, CoT states an unnecessary nudge and collides.

Clutch: the best of both

The first four drives are the four above, now with Clutch. Three more drives follow. At every planning step, Clutch selects whether to execute the trajectory with CoT or the one without it.

Closed-loop HUGSIM drives of Alpamayo-CF (Alpamayo 1.5 finetuned with our counterfactual supervision). Left to right: Never CoT, Always CoT, Clutch. Each panel shows the front camera and a bird's-eye view (HD-Score: higher is better). Below Clutch, the bars show its CoT probability p and which trajectory ran at each step (blue: with CoT, orange: without CoT).

Abstract

Embodied agents must decide when explicit reasoning improves an action, and then carry out the action the reasoning states. Driving vision-language-action models (VLAs) write their reasoning as a chain of thought (CoT) before they output a trajectory, yet always using CoT yields small gains over never using it and can even degrade driving performance.

Across six VLAs, we find that CoT improves driving performance when the car must act, such as passing a stopped car or taking a turn, and degrades it when the car should keep going in its lane. CoT degrades driving performance for two reasons: the CoT can be wrong, or the trajectory can fail to ‘listen’ to a right CoT.

We train a router, Clutch, that decides whether to execute the trajectory generated with CoT or the one generated without it, so that the car does not act on a wrong CoT. The trajectory fails to listen because its decoder learns to rely on the high-dimensional sensor input, which predicts the driver's action in most scenes. Counterfactual supervision removes this shortcut: we train the decoder on CoTs that state a different action, each paired with a trajectory that carries it out.

Clutch outperforms always and never using CoT for every VLA in every evaluation and achieves over 7× the gain of a prior router. With counterfactual supervision, CoT improves driving performance over 4× more when the car must act, and Clutch achieves its largest closed-loop gain. These findings echo a lesson from human cognition: the value of reasoning depends on both its influence on action and the judgment to use it selectively.

When does CoT improve driving?

CoT improves driving when the car must act, leaves most scenes unchanged, and degrades driving when the car should keep going.

We evaluate six released driving VLAs in closed-loop HUGSIM and on three open-loop benchmarks: NAVSIM, nuScenes, and PhysicalAI-AV. In every scene, we generate one trajectory with CoT and one without it. What the car must do separates where CoT improves driving from where it degrades it.

Must act

The car must take a different action, such as taking a turn or passing a stopped car. CoT can state the action the scene requires.

Keep going

The car should continue in its lane. CoT often states an action that is not needed, such as a pass or a lane change.

How CoT changes the trajectory in each situation. Top (must act): the trajectory without CoT gets stuck behind a barrier or a stopped car, while the trajectory with CoT goes around it. Bottom (keep going): CoT states an unnecessary pass or lane change, and the trajectory with CoT collides.

In closed-loop HUGSIM, CoT improves every VLA when the car must act: all six VLAs raise their HD-Score and collide less. When the car should keep going, CoT lowers HD-Score for four of six VLAs and raises collisions for five of six.

Driving score

Every VLA drives better with CoT when the car must act

HD-Score change from CoT for each VLA: higher for every VLA when the car must act, lower for four of six when it should keep going.

Collisions

Fewer when the car must act, more when it should keep going

Collision rate change from CoT for each VLA: fewer collisions for every VLA when the car must act, more for five of six when it should keep going.

Each bar: always using CoT minus never using CoT, in closed-loop HUGSIM.

Prior work expects CoT to help mainly in complex scenes, yet what the car must do matters at every HUGSIM difficulty level. On open-loop benchmarks, CoT leaves most scenes unchanged and improves about as many scenes as it degrades, so benchmark averages hide both effects.

Difficulty levels

The finding holds at every level

HD-Score change from CoT at each HUGSIM difficulty level, averaged over the eight VLAs.

HD-Score change from never to always using CoT, averaged over all eight VLAs. At every difficulty level, CoT improves HD-Score more when the car must act than when it should keep going.

Per-scene open-loop effect

CoT improves about as many scenes as it degrades

Share of open-loop scenes that CoT improves or degrades, for each VLA.

Share of NAVSIM, nuScenes, and PhysicalAI-AV scenes where CoT changes the score (up to half of scenes). Gains and losses nearly cancel in benchmark averages.

Why CoT degrades driving

Two failure modes, and two fixes that need each other.

Across the three open-loop benchmarks, CoT degrades driving twice as often when the action it states does not match the human driver. Even when the CoT matches the driver, however, the trajectory can ignore it or carry it out poorly, accounting for half of all degraded scenes.

1
The CoT can be wrong. When the car should keep going, CoT can state a pass of the car ahead, and the trajectory with CoT collides.
Clutch
2
The trajectory can fail to listen to a right CoT. CoT can state a pass of a stopped car while the trajectory still stops behind it.
Counterfactual supervision

The two fixes need each other: Clutch can only choose between the trajectories the VLA generates, and a trajectory that follows its CoT also follows a wrong one.

Driver-matching CoTs

Half of degraded scenes still have a right CoT

Share of scenes with a driver-matching CoT, among the scenes that CoT improves and degrades.

Share of open-loop scenes where CoT states the human driver's action (All: the six VLAs together). Because half of degraded scenes still have a driver-matching CoT, the CoT text alone cannot tell whether CoT will improve driving.

Clutch: choosing after seeing what the car would do

Prior routers decide before the CoT is written. Clutch decides after both trajectories are generated.

Like a car's clutch, Clutch engages reasoning only when it pays off. Prior routers decide from the scene alone, so they must guess both what the CoT will state and whether the trajectory will carry it out. Clutch reads both candidate trajectories and the frozen VLA's activations, so it guesses neither.

Clutch at inference and in training.

Clutch at inference and in training. (a) The frozen VLA generates one trajectory with CoT and one without, and Clutch uses both trajectories and the VLA's activations as input to select which one to execute. (b) We train Clutch with an outcome-weighted binary cross-entropy loss weighted by the score gap w, so crashes count far more than near ties.

Clutch averages small MLPs with the same architecture for every VLA. Decoding both trajectories in parallel adds only 3% wall-clock time over always using CoT for Alpamayo 1.5.

Over 7× the gain of a prior router

Seeing both trajectories accounts for most of the gain.

We compare Clutch against BLUE, a prior router that decides before the CoT is written, and remove one part of Clutch at a time. Removing the trajectories cuts Clutch's share of the best possible gain 3×, because the trajectories show what the car would actually do with and without CoT.

Mean routing gain of a prior router and of Clutch, as a percentage of the oracle gain.

Routing gain as a share of the oracle gain. The oracle picks the higher-scoring trajectory in every scene. Clutch achieves over 7× the gain of BLUE.

RouterActivationsTrajectories Outcome-weighted lossWins ↑Oracle share ↑
Clutchbefore + after CoT✓✓24/2437.3%
before + after CoT×✓24/2412.2%
before CoT×✓24/2411.3%
BLUEbefore CoT××17/244.9%

Router ablation. Removing one part at a time across 24 open-loop evaluations (8 VLAs × 3 benchmarks). Wins: evaluations where the router outperforms both Never CoT and Always CoT. Oracle share: mean fraction of the oracle gain achieved.

Clutch matches or outperforms always and never using CoT for every VLA

One router design, trained on the frozen VLA, works for all eight VLAs on all four benchmarks.

Read each row across columns: Clutch matches or outperforms Never CoT, Always CoT, and BLUE on all four benchmarks, cutting open-loop trajectory error by up to 24% on nuScenes and PhysicalAI-AV. In the top three rows, pairing Clutch with our counterfactual training (Alpamayo-CF) reaches the highest Alpamayo score on HUGSIM, NAVSIM, and nuScenes.

VLA Closed loop Open loop
HUGSIM (HD-Score ↑) NAVSIM (PDMS ↑) nuScenes (L2, m ↓) PhysicalAI-AV (ADE, m ↓)
NeverAlwaysBLUEClutch NeverAlwaysBLUEClutch NeverAlwaysBLUEClutch NeverAlwaysBLUEClutch
Alpamayo 1.5 0.2340.2380.2380.259 60.3461.9661.9668.77 1.931.871.871.41 1.111.121.110.99
Alpamayo-SFT 0.3610.3630.3610.368 87.3487.1787.4890.06 0.700.700.690.65 1.241.291.241.12
Alpamayo-CF (ours) 0.3410.3300.3600.369 90.3589.9090.3591.46 0.670.680.670.63 1.141.341.141.09
Qwen-Drive 0.3130.3190.3230.324 87.8387.9987.9987.99 0.940.930.920.92 1.371.361.361.35
Qwen-Drive-RL 0.2680.2720.2720.284 90.3590.6290.6290.62 0.960.950.940.93 1.451.431.431.42
OneVL 0.3070.2960.3040.319 85.4288.2388.2388.64 0.860.970.860.84 3.313.573.293.08
OpenREAD 0.3000.2960.3010.305 63.7563.3463.7564.61 0.760.750.750.75 1.291.291.281.28
dVLM-AD 0.0680.0690.0680.070 52.6146.8652.8054.27 1.851.991.821.68 4.404.784.264.01

Closed-loop and open-loop results. Never and Always: CoT always off or always on. BLUE: prior router deciding before CoT. Alpamayo-CF (ours): Alpamayo 1.5 finetuned with counterfactual supervision. Alpamayo-SFT: control finetuned on the same data without counterfactual examples. Bold: best in each row and benchmark. Italics: offline estimates. Clutch. Full method: Alpamayo-CF with Clutch.

Counterfactual supervision: making the trajectory listen

Clutch can only choose between the trajectories the VLA generates. Counterfactual supervision trains the decoder to follow its CoT.

When we give Alpamayo 1.5 a CoT that states a different action than the human driver took, its trajectory ignores the CoT over four times out of five. Because the car should keep going in most training scenes, the high-dimensional sensor input predicts the driver's trajectory on its own, and the decoder learns to ignore the CoT. Counterfactual supervision removes this shortcut by pairing a scene with a counterfactual CoT and a safe recorded trajectory from another clip that carries that action out.

Counterfactual supervision: before it, the trajectory can ignore the CoT, and after it, the trajectory follows the CoT.

Counterfactual supervision. We pair a scene with a counterfactual CoT (a different action than the human driver took) and a safe recorded trajectory that carries it out. Before counterfactual supervision (Alpamayo 1.5), the trajectory can ignore the CoT. After it (Alpamayo-CF), the trajectory follows the CoT.

A counterfactual CoT is not a wrong CoT: it states an action the human driver did not take, and its trajectory is safe in the scene. To build Alpamayo-CF, we keep Alpamayo 1.5's VLM frozen, finetune only the trajectory decoder on original, no-CoT, and counterfactual examples, and then train the decoder with reinforcement learning to reduce error and collisions.

Told to turn, Alpamayo 1.5 goes straight and Alpamayo-CF turns

Replacing the CoT changes the trajectory only after counterfactual supervision.

Given a CoT that says to turn, Alpamayo 1.5 goes straight and Alpamayo-CF turns.

In both scenes, we replace the VLA's CoT with a counterfactual CoT that says to turn. Alpamayo 1.5 (dashed black) ignores it and drives straight with the recorded driver (tan), while Alpamayo-CF (purple) turns as the CoT states. Insets: bird's-eye view.

Counterfactual supervision makes CoT pay off

Alpamayo-CF follows counterfactual CoTs nearly 3× as often, and CoT improves its must-act driving over 4× more.

Finetuning on the same data without counterfactual examples (Alpamayo-SFT) raises CoT following far less. When a CoT that is not driver-matching is replaced with a driver-matching one on NAVSIM, only Alpamayo-CF gains sharply in PDMS, and in closed-loop HUGSIM CoT raises its must-act HD-Score over 4× more than for Alpamayo 1.5 or Alpamayo-SFT.

How often the trajectory follows its CoT, and how often Clutch selects CoT.

CoT following (a) and Clutch selection (b). (a) Share of PhysicalAI-AV trajectories that follow a counterfactual CoT (CF CoT) and the VLA's own CoT (Own CoT). (b) Share of open-loop scenes where Clutch selects CoT, among scenes that CoT improves versus degrades.

VLA Follows CF CoT (%) ↑ ΔPDMS ↑ Must-act gain ↑
Alpamayo 1.515.1−0.16+0.008
Alpamayo-SFT24.8+0.31+0.010
Alpamayo-CF44.5+1.41+0.045

Counterfactual supervision results. Follows CF CoT: share of PhysicalAI-AV trajectories following a counterfactual CoT. ΔPDMS: NAVSIM gain when replacing a CoT that is not driver-matching with a driver-matching one. Must-act gain: closed-loop HUGSIM HD-Score change from never to always using CoT when the car must act.

The two parts need each other

Alpamayo-CF with Clutch outperforms both fixed policies whether the car must act or should keep going.

A trajectory that follows its CoT also follows a wrong one, so always using CoT still degrades Alpamayo-CF when the car should keep going. Without counterfactual supervision, Clutch outperforms always and never using CoT for Alpamayo 1.5 and Alpamayo-SFT only in keep-going scenes. With both parts together, our full method (Alpamayo-CF with Clutch) outperforms both fixed policies in both situations.

HD-Score ↑ Must act Keep going
NeverAlwaysClutch NeverAlwaysClutch
Mean over eight VLAs0.1890.2080.2130.2910.2800.299
Alpamayo 1.50.1890.1970.1750.2540.2570.290
Alpamayo-SFT0.2880.2980.2850.3760.3810.392
Alpamayo-CF0.2410.2850.3050.3730.3310.382

Closed-loop HUGSIM HD-Score by situation. Bold: best of Never, Always, and Clutch in each situation.

The full method in one drive

Both fixed policies collide. Alpamayo-CF with Clutch completes the route.

A sedan is stopped in the lane in closed-loop HUGSIM. Never CoT and Always CoT both collide with it, while Alpamayo-CF with Clutch switches between the two trajectories and completes the route.

Takeaways

6 of 6 VLAs

CoT improves every released VLA when the car must act, and degrades driving for most VLAs when the car should keep going.

Over 7×

Clutch turns CoT on only when it pays off, achieving over 7× the routing gain of a prior router.

Nearly 3×

Counterfactual supervision makes the trajectory follow its CoT nearly 3× as often, raising must-act CoT gain over 4×.

BibTeX

@misc{yeung2026reasoning,
  title  = {Reasoning by Consequence: Selective Chain-of-Thought for Driving VLAs},
  author = {Yeung, Jacob and Zhang, Wenyuan and Chang, Nadine and Tarr, Michael J. and
            Ramanan, Deva and Alvarez, Jose M. and Wang, Pichao and Chen, Ke},
  year   = {2026},
}