Addressing the Orchestration Gap in Generalist Robots via Physical Agency

The thesis

Most “missing motor skill” is really missing use of a capable policy.

Hold the robot, cameras, scenes, demonstrations, and policy weights fixed. Change only the inference-time process. The same frozen skills succeed far more often inside the loop than when prompted directly — the orchestration gap.

Simulation · LIBERO-PRO

12.8% → 53.3% mean success

π_0.5 direct
(frozen, alone)

12.8%

CaP-Agent0

18.2%

Pigey
(same weights)

53.3%

Over 4× the frozen policy — new state of the art, weights unchanged.

Real robot · Franka FR3

16.7% → 97.3% overall success

π_0.5 direct

16.7%

TiPToP
(open-loop)

48.7%

Pigey

97.3%

Gains land on reasoning-limited tasks; already-easy pick-and-place stays at 100%.

Watch it run

Real-robot tasks — watch it reason in real time.

Click a task to watch the agent reason through it — the full video alongside its step-by-step trace.

Imitation learning

Watch a human demo once, then replay it

Task“I’ll demo the task by hand, one move at a time. Watch each move. After I reset the scene and my hands leave, replay exactly what I demonstrated.”

Deductive reasoning

Reason about a hidden object

Task“Pick up the doll and put it in the basket.”

Long-horizon memory

Memorize a scene, then restore it after a shuffle

Task“Memorize the scene, look away while I shuffle it, then restore every object to where it was.”

Spatial reasoning

Build a stable stack, largest to smallest

Task“Stack the containers by size — every piece in one stack.”

Safety reasoning

Put the kid-safe toys in the bowl, keep the hazards out

Task“A child is coming over — put everything they’d want to play with in the bowl.”

Obstacle reasoning

Empty the occupied cup, then place the doll

Task“Put the doll in the cup.”

Error recovery

The plate gets occupied mid-task — fall back to a bowl

Task“Put the doll on an empty plate. If there’s no empty plate, put it in an empty bowl.”

Abstract

General-purpose robots must combine perception, world knowledge, planning, success detection, recovery, and low-level control. Today’s state-of-the-art models try to fold all of this into one learned policy via large-scale pre-training. We show instead that these capabilities decompose into a general language-conditioned control agent and a high-level agent manager. Rather than training policies to reason, we build a closed-loop physical-agency orchestrator — Pigey — that plans high level, decomposes goals into achievable subgoals, commands low-level motor skills, verifies outcomes from observations, and recovers from failures. It controls existing vision-language-action policies and parametrized skills to solve complex reasoning tasks in the real world, with no extra data collection or post-training. On LIBERO-PRO, Pigey advances the state of the art by over 4× (12.8% → 53.3%) with no task-specific fine-tuning; on a real robot it lifts a frozen policy from near-zero to over 90% on reasoning-limited tasks. We call the difference between what frozen skills achieve alone and inside the agentic loop the orchestration gap.

How it works

A frontier VLM runs the task-level loop over two frozen backends.

The orchestrator never emits motor commands. At each step it reads the observation and history, calls one tool, incorporates the result, and decides again — until the task predicate holds. Motion is delegated to skills it never trained.

A frontier VLM runs a closed loop — observe, reason, act, verify — over two frozen backends: TAMP for rigid pick-and-place, and a π_0.5 VLA for deformable and recovery actions. It routes each subgoal, verifies the outcome from the sensor and wrist image, and recovers on failure — emitting Done only once the goal state holds.

Results

Same frozen weights. Only the inference-time process changes.

**LIBERO-PRO** success rate (%) across the six perturbation suites. One frozen π_0.5-LIBERO policy throughout; our reasoner is Claude Opus 4.7.
Method	Obj-swap	Obj-task	Sp-swap	Sp-task	Goal-swap	Goal-task	Mean
π₀-LIBERO	0	0	0	0	0	0	0
π_0.5-LIBERO	17	1	20	1	38	0	12.8
CaP-Agent0	22	18	12	14	26	17	18.2
Pigey (ours)	54	54	66	80	44	22	53.3

**Real robot** success rate (%) by capability probe. Same π_0.5-DROID weights throughout; our reasoner is Claude Opus 4.7.
Category	π_0.5	TiPToP	Pigey
World knowledge	0	90	100
Conditional logic	0	95	100
Multi-step reasoning	0	25	100
Error recovery	10	0	90
Obstacle / safety	0	0	90
Spatial reasoning	20	75	100
Long-horizon memory	0	0	100
Simple pick-place	95	80	100
Overall	16.7	48.7	97.3

Citation

BibTeX

@misc{galanti2026addressingorchestrationgapgeneralist,
  title         = {Addressing the Orchestration Gap in Generalist Robots via Physical Agency},
  author        = {Liane Galanti and Dhruv Shah and Tri Dao},
  year          = {2026},
  eprint        = {2607.21725},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2607.21725},
}