Agentic World Action Models

Zhuoyang Liu^1*, Dantong Niu^1,2*†, Botong Zhang^3*, Boning Shao^1*, Jing Wang², Ruijie Zheng², Yuvan Sharma², Runyu Lu², Jiaming Liu², Renrui Zhang³, Roei Herzig¹, Yutong Bai¹, Ken Goldberg¹, Danfei Xu², Yuke Zhu^2†, Guanzhi Wang^2†, Linxi “Jim” Fan^2†, Trevor Darrell^1†

¹UC Berkeley²NVIDIA³University of Pennsylvania

^*Equal contribution^†Equal advising

Paper Model Code

Abstract

Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics. We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution. We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.

Contributions

AW-0 brings agentic robotic skills into a world action model: generating code improves the model’s action prediction, and the code it generates can be reused in an agentic framework.

• A mixture-of-transformers WAM with a code expert — code, future video, and actions are denoised jointly, so the generated program guides both the predicted future and the motor commands.
• An automated agentic annotation pipeline — coding agents label robot and egocentric human demonstrations with aligned programs and distill recurring routines into reusable skills.
• A hybrid execution policy — at every replanning point it either invokes the generated program or executes the action chunk from the action expert.

Agentic World Action Models

AW-0 is a single conditional generative model over code, video, and actions. A skill program is a discrete token sequence C₀, future observations are continuous video latents Z₀ from a pretrained VAE, and robot motion is an action chunk A₀ over a horizon of H steps. The model is conditioned on the current multi-view observations, the language instruction, the robot state, and a compact memory of the program slices generated so far.

Each expert keeps its own attention projections, layer norms, and FFNs, and the experts communicate through masked joint attention at every layer. The code expert augments a pretrained diffusion LLM (LLaDA) with a frozen Qwen3-VL ViT, so the current views are encoded into visual tokens that ground skill planning in the scene. The video expert is a pretrained Cosmos-Predict2.5 video DiT over VAE-encoded future frames, and a separate action DiT generates the continuous action chunk with additional robot-state conditioning. Attention is bidirectional within each expert and blockwise causal across experts in the order code → video → action.

Joint denoising

At every denoising step the backbone predicts code-token logits and continuous velocities at once. Video and actions use flow matching with a shifted noise schedule; code uses absorbing-mask corruption with a random mask ratio and is trained with cross-entropy on masked positions. At inference, the policy appends a canvas of G mask tokens and decodes it together with the video and action states over T steps, committing the G/T most confident positions per step. The code stream supplies planning context to both continuous experts throughout generation, and the action expert also reads the video representations. To give the action expert more informative video features, AW-0 resamples the video noise level from a mixture of the pretrained distribution and a uniform one, with importance weights that balance video learning against predictive guidance.

Agentic code annotation

Programs for training come from an automatic annotation pipeline. First, one episode per task is sampled and Gemini 3.1 Pro writes subtask-level programs against predefined, declaration-only primitive APIs, while Claude Code (Claude Opus 4.8) acts as a coordinator that distills recurring, task-general routines into skills. The skill library is then frozen and the sampled episodes are re-annotated with the final vocabulary, forming a cache of reusable subtask-to-program mappings. The remaining corpus reuses a cached program whenever a subtask’s contact and motion pattern matches an existing entry, and calls the coding agent only on a cache miss.

01Skill Discovery

Primitive APIs

pick(object_name, side)place(object_name, target_name, side)rotate(object_name, axis, angle_deg, side)handover(object_name, from_side, to_side)move_arm(side, direction, distance=None)goto_relative(side, object_name, relation, clearance=None)···

Skills

pick_and_bring(),
tilt_pour_over_target(),
move_item_two_handed(), …

Subtask Subset

“Turn Over the Bag of Laundry Powder.”

Coding Agent

robot.close_hand('left')
robot.release('right')
robot.rotate('bag of laundry powder',
    'vertical', 180.0, 'left')
robot.grasp('bag of laundry powder',
    'right')
robot.release('left')

Coordinator

02Skill Library50 Reusable Skills across Tasks

Across Tasks

free_hand_and_pick()

tilt_pour_over_target()

move_item_two_handed()

pick_and_bring()

Reusable SkillPython

def tilt_pour_over_target(
    robot, vessel_name, target_name, side,
    aim_delta, approach_rpy, tilt_deg=90.0
):
    target = robot.detect_objects_oneshot(target_name)
    robot.move_arm(
        side,
        target.data['detections'][0]['position_3d'],
        approach_rpy)
    robot.servo_ee_delta(side, aim_delta)
    robot.rotate(vessel_name, tilt_deg, side)
    robot.rotate(vessel_name, -tilt_deg, side)

Library Frozen

03Corpus Annotation

All Demos

Subtask → Code Cache

Built by Re-Annotating the Subtask Subset Once Frozen

Reuse

If Contact and Motion Evidence Match

Re-Annotate

Write Back to the Subtask → Code Cache

During training, the code stream is a conditioning prompt followed by the active slice, a short block of primitive or skill calls for the current subtask; only the slice is masked and supervised. Because one frame rarely reveals task progress, the code expert also reads a memory cache of the slices generated so far in the episode, compressed into K learnable memory slots. By default the decoded slice is not executed: it is a plan that conditions the action expert, and the predicted action chunk drives the robot. In hybrid mode, the slice is executed Code-as-Policies style against a library of perception and control primitives whenever it passes stage, dependency, syntax, whitelist, and vacuity checks; otherwise the action chunk from the same forward pass drives the robot until the next replan.

Skill library

The pipeline produced a frozen library of 50 skills: 23 for YAM robot data and 27 for egocentric human data. Each skill is a Python function over the primitive API that takes the objects it acts on as arguments. 10 representative skills are shown here.

YAM robot skills

fold_edge_overalternating_smoothing_patsflip_object_overadjust_in_place

Egocentric human skills

open_closurepour_into_targetstir_contentswipe_surface_back_and_forthfold_over_and_creasewrap_around_target

Task “wrap blanket roll with ribbon”

Sub-goal “Fold the blanket at the center of the board”

Program slice for this episode

1fold_edge_over(robot, 'blanket', [0.15, 0.0, 0.0], [0.15, 0.0, 0.0])

Skill definition (library)

1def fold_edge_over(robot, item_name, left_delta, right_delta):
2    # pinch the edge with both arms
3    robot.grasp(item_name, 'left')
4    robot.grasp(item_name, 'right')
5    # carry the pinched edge across the item
6    robot.servo_ee_delta('left', left_delta)
7    robot.servo_ee_delta('right', right_delta)
8    # let the fold lie
9    robot.release('left')
10    robot.release('right')

Browse the full frozen skill library (23 YAM + 27 egocentric human skills)

YAM robot skills (23)

adjust_in_place

Grip an object where it lies, slide it a few centimetres without lifting, and let go.

alternating_smoothing_pats

Press the fabric flat with one arm and come off, then the other — the alternating pats of a smoothing slice.

approach_best_grasp

Sample grasp candidates on an object, pick the best one reachable by this arm, and drive the end-effector onto it.

brace_and_pull_open

Brace the body with one arm, take its lid, flap or handle with the other, draw it open, then release the brace.

brace_and_push_in

Grip the same part twice, low to keep it aligned and high to push, and drive it home into its receptacle.

flip_object_over

Lift a held flat object clear, roll the wrist a half turn, release it inverted, then grasp it again to pin it down.

fold_edge_over

Both arms pinch a cloth edge, carry it across the item, and release — one fold.

free_hand_and_pick

Open the idle arm, then pick the object up with the working arm.

hand_over_hand_extract

Two grip–pull–release bites that walk a long object out of a bag, sleeve or slot.

lower_onto_target

Hover the carried object above a detected target at the current wrist orientation and lower it in; the caller releases.

pull_part_free

Grip a part where it sits, drag it straight off along one bounded delta, and release.

push_closure_shut

Grip a drawer, lid, door or cover and drive it along its travel until shut, then let go.

ratchet_turn_fastener

Two ratchet strokes on a threaded fastener; the release in between is the wrist unwinding, not a drop.

ratchet_twist_off

Two ratchet bites on a threaded part; the sign of the angle sets whether it tightens or loosens.

relocate_object

One-arm transport inside a single slice: pick the object up and place it at the named target.

roll_up_one_bite

One reach-limited bite of rolling: pinch, push the roll forward, release, and travel back to the next pinch point.

run_zipper_along_seam

Pin the case with one arm, take the zipper pull with the other, and draw it one run along the seam.

scrub_back_and_forth

Bring an already-held cloth to a detected surface and scrub it once back and forth.

spread_taut_and_release

With both arms already gripping, drag them apart to pull the item taut, then release both.

stir_braced_vessel

Stir inside a braced vessel, re-seating the brace once the tool tip starts loading the wall.

tear_off_piece

Pin the parent sheet, roll or bag with one arm, pinch the piece with the other, and pull along the tear line.

tilt_pour_over_target

Carry a held vessel over a detected receptacle, clear the rim, tip once, and return to upright.

wrap_tension_pull

Two hand-over-hand bites on a strand being wound: regrasp further along and pull the slack taut, twice.

Egocentric human skills (27)

align_and_seat_braced

One hand pins the target so it cannot shift while the other pushes the part home and lets go.

brace_and_pick

Hold the source still with one hand and pick the object off or out of it with the other.

brace_and_take_tool

Set up the two-handed working stance: the brace hand closes on the workpiece, the working hand on the tool.

close_closure

Swing a closure shut with both hands on it and come off it; nothing is held afterwards.

flatten_by_pulling_taut

Pin a slack flat item with one hand, drag it taut with the other, re-bite further along, and drag again.

flip_over_between_hands

Turn an object face-over by passing it between the hands through a half turn.

fold_over_and_crease

Brace a flat item, carry its free edge across in two steps, set it down, and press the crease — one fold.

hang_item_on_support

Bring a held item over a hook, rack or branch from above and let it down onto it.

insert_into_receptacle

Seat the held object into a receptacle, let go, and withdraw the now-empty hand.

move_item_two_handed

Take a bulky item with both hands where it sits, carry it along a semantic direction, and release with both.

open_closure

Swing a hinged closure open with one hand while the other braces the container body, then come off it.

pick_and_bring

Pick an object up and carry it to a semantic relation on where it is going next; the hand keeps hold of it.

place_at_target

Set the carried object down at a target with the working hand and free the second hand as well.

pour_into_target

Bring a held vessel's mouth over a target, tip it until it empties, and return it upright; the grip never changes.

press_onto_target

Press a target with both hands and come off it — seating a lid or case home, or working a control.

pull_open_and_hold

Draw a handle, leaf or free end open along a semantic direction and keep hold of it.

pull_part_free

Brace an assembly with one hand and pull a part off or out of it with the other.

rotate_grasped_object

Take an object where it sits and turn it about a semantic axis; the hand ends still on the object.

scoop_and_carry

Take a load out of a source container with a held utensil and bring it over the destination.

scrub_inside_vessel

One hand holds the vessel still while the other puts a brush down inside it and works a stroke pair.

stir_contents

Drop the already-held tool inside the vessel and sweep it round.

thread_through_and_pull_out

Push a lace, cord or pin through a hole with one hand and catch the emerging end with the other to draw the slack through.

tighten_with_ratchet_regrasps

Take up the last of a thread on an already-seated cap: turn, release the bite, re-bite, and turn again.

turn_over_object

Flip a held object face-over and bring the new face to a semantic relation on its destination.

unscrew_cap

Twist a threaded part, let the wrist unwind and re-close, twist again, and lift the freed part clear.

wipe_surface_back_and_forth

Park a cloth or sponge already in one hand on an open surface and run two out-and-back strokes.

wrap_around_target

Lead a held wrapper — ribbon, foil, film, gauze — once around a target in a closed circuit.

Training recipe

AW-0 is pretrained on about 2,000 hours of YAM bimanual robot teleoperation and 1,000 hours of egocentric human manipulation, with every training window paired with its instruction and the aligned program slice. For each downstream task, the same pipeline labels the task demonstrations under the frozen skill library, and AW-0 is jointly fine-tuned on code, video, and action targets.

Demonstrations

Real-world AW-0 rollouts on the eight tasks, on the YAM station (parallel grippers) and the Tianji–Sharpa platform (dexterous hands). Pick a task to watch the policy execute it; use the full-screen button to expand the video.

Simulation: RoboTwin 2.0

We compare against π0.5 (a VLA with strong semantic reasoning), EgoScale (a VLA pretrained on large-scale egocentric human data), DreamZero (a WAM that jointly generates future video and actions), and WLA (an autoregressive model that predicts textual subtasks, subgoal images, and actions). All methods train on 50 clean demonstrations per task (2,500 episodes) and are evaluated on the clean setting and on the random setting with randomized backgrounds, textures, lighting, table height, and distractors.

AW-0 reaches the highest average success rate, 50.7%, driven by 80.8% on the clean setting, 16.0 points above the strongest baseline WLA (64.8%). Under visual randomization the default mode drops to 20.6%, still above the strongest WAM baseline DreamZero (17.6%). Hybrid execution raises the random-setting success rate to 29.2%, above every baseline including π0.5 (27.7%), and retains 46% of its clean-setting performance under randomization versus 25% for the default mode. The cost is on clean scenes, where the hybrid mode (63.1%) is 17.7 points below the default mode.

Real-world experiments

We evaluate 8 real-world tasks on two embodiments: the YAM station, a fixed-base bimanual setup with two 6-DoF arms and parallel grippers, and the Tianji–Sharpa platform, a humanoid torso with two 22-DoF Sharpa dexterous hands. Both observe three RGB views and act in a unified 64-D state–action space. Each task is run for 3×16 rollouts under randomized initial object poses and scored with an additive rubric.

Language following and semantic reasoning. AW-0 is strongest on Arrange Fruits and Flip Scoreboard, which require instruction-specific object selection and counting. The VLA baselines, particularly π0.5, also show relatively strong semantic understanding, while the WAM baselines are less reliable here, which suggests that code-based planning strengthens exactly these abilities in a world action model. Robust skill execution. On contact-rich and dexterous tasks, DreamZero beats π0.5 on Replace Lightbulb and Deal Poker, where predicting future states gives stable physical-dynamics features; AW-0 goes further by combining guidance from generated skill programs with those predictive representations.

Hybrid execution in the real world

On Fold and Stack Towels (YAM), hybrid execution reaches 90.6% versus 84.4% for action-chunk execution with the same post-trained model. Dexterous folding relies mainly on direct actions from the action expert, while executed code handles the more structured motions, such as approaching and stacking the towels, which gives direct guidance on stacking order and motion direction.

Ablations

All variants are post-trained and evaluated under the real-world protocol above. How should the code expert be integrated? Adding code between video and action (V → C → A) grounds the predicted dynamics to action but only lets code reason over video predictions. Placing code first (C → V → A) lets both video and action generation attend to the plan, so semantic task structure shapes future prediction throughout denoising, and gives the best result on 7 of 8 tasks.

A dedicated ViT is also a better visual input for the code expert than reused Wan2.2 VAE features: VAE features are built for pixel reconstruction and poorly support the semantic distinctions needed to select and compose skills. ViT conditioning improves all 8 tasks, with substantial gains on Pack Backpack (64.6 vs. 39.6) and Solve Cube (72.1 vs. 53.9), which require fine-grained reasoning about object attributes and task states.

What makes code an effective planning representation? Full annotated programs beat comment-free code on all 8 tasks, so explanatory comments help connect program structure with task intent. Removing skill calls also hurts on all 8 tasks, especially on skill-heavy tasks such as Deal Poker (58.3 vs. 42.0), and full programs beat natural-language subgoals on 7 of 8 tasks.

How does joint denoising affect performance and efficiency? Compared with generating code autoregressively before the video and action (teacher-forced AR), joint denoising wins on 7 of 8 tasks: early AR errors propagate into video prediction, while joint denoising refines the code alongside video and actions. It is also faster.

What code changes in the predicted future

Does the code stream change what the world model predicts? We compare future frames from the right wrist camera for AW-0, a variant that plans with natural-language subgoals (Subgoal), and a model without a planning stream (AW-0 w/o code). For the instruction “Arrange the fruits so that finally there is only mango in the left bowl”, AW-0 directs the predicted interaction toward the mango, Subgoal hesitates between the apple and the mango, and AW-0 w/o code goes for the apple.

On the final stage of Deal Poker, where the right hand inserts a card into a narrow slot, AW-0 predicts the card aligning with and entering the slot, while Subgoal and AW-0 w/o code produce less coherent insertions with blur and distortion around the hand and card. Explicit skill calls and object arguments give the video expert a more specific condition, and the action expert in turn reads more reliable predictive features.

Failure cases

Six representative failures show where AW-0 still has headroom:

Arrange Fruits

Container Grasp

The fruit lies at the basket rim; the left jaw lands outside the basket and the gripper closes on the basket wall.

Pack Backpack

Object Dragging

Pulling the zipper drags the backpack, which moves and tips over instead of staying stable.

Replace Lightbulb

Imprecise Alignment

The bulb reaches the socket but its base stays off-axis, so the threads never engage.

Arrange Fruits with Tongs

Tool Collision

The tongs approach the pear with their tips closed and poke it instead of opening around it.

Deal Poker

Slipping Off

The left hand does not secure the card stack during the handover, and the cards slip.

Flip Scoreboard

Contact Loss

The fingertip reaches the card edge but slides off before turning it, so the score does not change.

Limitations & future work

• Program grounding and skill coverage. Planning quality depends on annotation accuracy and skill coverage; a generated program can pick the right skill with the wrong arguments, so syntactic validity does not ensure correct execution. Execution feedback could improve grounding and grow the skill library.
• Predictive robustness and hybrid execution. Structured planning does not remove sensitivity to visual distribution shift, and hybrid execution trades clean-scene performance for robustness because the arbiter admits code on validity alone. Learning to arbitrate from execution reliability and expected progress could exploit both modes better.
• Toward continual agentic learning. Real-world hybrid evaluation is limited to YAM, as code-as-policy systems for dexterous hands are lacking, and the system does not yet learn online from its own interactions.

Conclusion

AW-0 is an agentic world action model that integrates code-based skill planning with predictive dynamics and continuous control. Through joint denoising, the code expert provides structured semantic guidance for video and action generation, and an automated annotation pipeline lets this planning interface be learned from large-scale robot and human demonstrations. Experiments on RoboTwin 2.0 and on real-world manipulation with YAM and dexterous hands show gains in language following, semantic reasoning, and skill execution, with hybrid execution further improving robustness. These results lay a foundation for skill reuse and for iterative interaction between programmatic planning, world prediction, and execution in agentic systems that continually interact with and learn from their environments.

Citation

If you find this work useful, please cite:

@misc{liu2026agenticworldactionmodels,
  title={Agentic World Action Models},
  author={Zhuoyang Liu and Dantong Niu and Botong Zhang and Boning Shao and Jing Wang and Ruijie Zheng and Yuvan Sharma and Runyu Lu and Jiaming Liu and Renrui Zhang and Roei Herzig and Yutong Bai and Ken Goldberg and Danfei Xu and Yuke Zhu and Guanzhi Wang and Linxi "Jim" Fan and Trevor Darrell},
  year={2026},
  eprint={2610.30136},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2610.30136},
}

This project page is built on the open-source ENPIRE project-page template — many thanks to its authors.