CoRL 2026 · LEAP Workshop

Evaluating Long-Horizon Robot Planning under Uncertainty

Harsh Muriki^*, Gayathri Aravindan, Yuchen Zheng, Sonia Chernova

RAIL Lab, Georgia Institute of Technology · ^*Corresponding author

Paper arXiv Poster Code BibTeX

Teaser video / figure placeholder
Habitat rollouts of the three planners on the same task variant

An object remembered on a countertop may have been moved into a cabinet, replaced, or removed entirely. How do different planners respond when the robot's initial memory and the environment are varied systematically, within one shared execution framework?

Abstract

Household robots must successfully navigate imperfect circumstances, both in their internal models and the external world. We introduce a benchmark of seven long-horizon household tasks in Habitat that vary initial robot memory and environment configurations. We evaluate adaptations of ReAct, VLM-TAMP, and Tru-POMDP within a shared execution framework across 90 task–environment–memory combinations, totaling 1,350 episodes. Our evaluation measures task success, partial completion, and action efficiency, along with rule-based analysis of execution traces. ReAct achieves the highest overall task success (45.3%) and completion (71.7%), while having accurate initial memory does not consistently yield the highest performance for every planner. We identify recurring placement errors, precondition errors, and redundant actions through trace analysis. These findings highlight differences in how the evaluated planners use initial knowledge and respond to uncertainty.

7long-horizon tasks

90task × env × memory variants

3planner adaptations

1,350episodes

Contributions

A benchmark of seven long-horizon household tasks with controlled variations in initial robot memory and environmental conditions.
A comparative evaluation of three planner adaptations across 1,350 episodes using success, completion, and action-efficiency metrics.
A trace-based analysis identifying recurring failure and inefficiency patterns.

The Benchmark

Seven long-horizon household tasks in Habitat Sim[1–3], executed with PARTNR's[4] shared high-level robot skills. Each task spans multiple rooms and furniture, involving object transport, spatial placement, and object-state changes.

Rollout coming soonHover to play

T1

Water fetching

Turn off all the lights in the bedroom and bring a full jug of water to either of the two living-room tables.

Rollout coming soonHover to play

T2

Package preparation

Move both packages to the living-room center table and place them next to each other. Bring the scissors and place them beside the packages.

Rollout coming soonHover to play

T3

Book collection

Collect the 2 green books, place them on the living-room side table, and turn off the laptop.

Rollout coming soonHover to play

T4

Grocery organization

Put all 3 fruits in the fruit basket and the 2 drink containers on the kitchen counter.

Rollout coming soonHover to play

T5

Guest table preparation

Get soap to clean both glasses, fill them, and place them on the living-room side table. Place a white plate next to them.

Rollout coming soonHover to play

T6

Meal delivery

Heat up the bread and bring it to the bedroom desk with a water bottle and a clean hand towel beside it. Turn off the living-room lights.

Rollout coming soonHover to play

T7

Household cleanup

Put away the living-room toys in the closet wardrobe, wash and put away the dirty dishes, and turn off the living-room and kitchen lights.

Variants = World × Memory

Each episode pairs one environment configuration (what is actually true in the world) with one initial memory condition (what the robot believes at the start), giving up to 15 variants per task. Below is the full grid for T1.

T1 “Turn off all the lights in the master bedroom and bring a full jug of water to the living-room table.” Uncertainty target: jug · acceptable substitute: pitcher

World →Memory ↓	BASEJug on counter	SUBOnly a pitcher	ABSNo container	HIDJug in fridge	DISJug + vase, spray bottle
Accurate	BaselineJug on counter	Pitcher available	Nothing available	Jug in fridge	Jug + distractors
Incomplete	No jug info	No pitcher info	Can’t tell	No jug info	No jug info
Outdated	Jug on dining table	Jug on dining table	Jug on dining table	Jug on dining table	Jug on dining table

Cells show what the robot believes at the start. Accurate: matches the world Incomplete: information missing Outdated: stale and wrong

Outlined: the HID × Outdated example, where the jug is in the fridge but the robot remembers it on the dining table. In ABS, criteria needing the unavailable object are excluded.

Metrics

Task success & completion

Completion C_i is the fraction of evaluation criteria satisfied; success S_i = 1 only if all are satisfied.

SPL_act

Adapts SPL[5] to count skill attempts instead of path length, relative to a human reference: S_i · A_i^ref / max(A_i, A_i^ref).

TC-SPL_act

Replaces binary success with completion C_i, rewarding partial progress while penalizing extra skill attempts. 1 = as efficient as a human.

Three Planners, One Framework

All three approaches are re-implemented on Habitat with PARTNR's skills and share the same inputs: memory, goal, scene graph, images (except Tru-POMDP), and the same 12 high-level robot skills. Every planner uses GPT-5.6 Luna with high reasoning effort.

plan / actionfailure signalobservation / progress

ReAct[6]

A VLM picks every next skill

Extends PARTNR-Planner’s[4] zero-shot ReAct with visual observations. At each step a VLM reasons over the instruction, world graph, images, and action history, then selects the next skill; execution feedback informs the next decision.

ReAct loop: the VLM sends the next action to the robot in Habitat; observations and execution failures feed back to the VLM

VLM-TAMP[7]

VLM subgoals, refined by a PDDL planner

A VLM proposes intermediate subgoals as grounded predicates; a symbolic planner with a hand-crafted PDDL[8] domain turns each into a skill sequence. Failures trigger symbolic replanning or subgoal revision.

VLM-TAMP pipeline: VLM to subgoal iterator to PDDL planner to robot; execution failures return to the PDDL planner, failure context to the VLM, and subgoal completion to the iterator

Tru-POMDP[9]

Belief-space search over LLM hypotheses

An LLM Tree of Hypotheses builds a weighted particle belief over goals and unobserved object locations; a DESPOT-style search picks actions, and the belief updates after each one.

Tru-POMDP loop: LLM Tree of Hypotheses to particle belief to belief-space planner to robot; observations and actions drive a belief update, which can regenerate hypotheses

Results

ReAct leads, but every planner incurs extra actions

3%ReAct success71.7% completion
1%VLM-TAMP success48.3% completion
7%Tru-POMDP success56.2% completion

Bar chart of task success, SPL_act, task completion, and TC-SPL_act for the three planners — Overall task success, SPL_act, task completion, and TC-SPL_act. Adding action efficiency lowers every planner's score (ReAct: 45.3% → 36.9% SPL_act; 71.7% → 59.1% TC-SPL_act).

Better memory ≠ better plans

TC-SPL_act under each initial-memory condition
Method	Accurate	Incomplete	Outdated
VLM-TAMP	0.34	0.39	0.41
ReAct	0.71	0.55	0.52
Tru-POMDP	0.39	0.32	0.47

Only ReAct benefits clearly from accurate memory. VLM-TAMP and Tru-POMDP score best with outdated memory. One possible explanation is that memory–observation discrepancies trigger extra exploration or replanning that surfaces useful information, though the aggregate results do not establish this.

Environment configurations vs. BASE

Change in TC-SPL_act from BASE across SUB, ABS, HID, and DIS for each planner — ΔTC-SPL_act from BASE in percentage points, averaged over memory conditions. Absent and hidden targets hurt every planner, ReAct most (−14.0 and −18.8 pts). Substitutes and distractors split them: VLM-TAMP gains (+9.9, +12.4 pts), while Tru-POMDP drops with substitutes (−12.7 pts).

Failure & inefficiency analysis

Rule-based analysis of execution traces, following prior embodied-agent error analyses[10, 11]. Categories overlap, so rates need not sum to 100%. Darker cells are more frequent; bold marks the highest in each row. Hover a finding for its definition.

Failures · % of unsuccessful runs
Finding	VLM-TAMPn=319	ReActn=246	Tru-POMDPn=303
Object not placed	56.4	23.6	45.2
Missing action	46.1	37.4	36.3
Placement error	44.5	53.7	16.5
Object not found	30.1	19.9	18.5
Insufficient planning	12.5	1.6	17.5
No hypotheses	0.0	0.0	26.4
Wrong object	2.8	1.6	5.3
Wrong order	0.0	5.3	0.0
Failure to stop	0.0	3.7	0.7

VLM-TAMP most often leaves objects unplaced (56.4%); ReAct makes the most placement errors (53.7%).

Inefficiencies · % of all runs
Finding	VLM-TAMPn=450	ReActn=450	Tru-POMDPn=450
Unnecessary actions	22.2	30.4	52.4
Precondition error	12.0	46.0	23.1
Repeated actions	29.3	2.4	41.1
Failure to stop	0.0	2.4	3.6
Hallucinated object	0.0	0.2	0.0

ReAct has the most precondition errors (46.0%); Tru-POMDP the most unnecessary (52.4%) and repeated (41.1%) actions.

Takeaways

ReAct

Highest task performance, but vulnerable to incorrect placements, precondition violations, and unproductive exploration when objects are unavailable.

VLM-TAMP

Fewer precondition errors, but can terminate after completing its generated subgoals without satisfying the full task.

Tru-POMDP

Explicitly represents uncertainty, but frequently takes unnecessary and repeated actions.

Robust long-horizon planning requires more than reactive replanning, symbolic feasibility checking, or explicit uncertainty representation in isolation. Effective systems must also decide when more information is needed, verify progress toward task-level goals, and recognize when continued execution is unlikely to be productive.

Example Rollouts

ReAct · T1 · HID × Outdated

VLM-TAMP · T1 · HID × Outdated

Tru-POMDP · T1 · HID × Outdated

BibTeX

@misc{muriki2026evaluatinglonghorizonrobotplanning,
  title         = {Evaluating Long-Horizon Robot Planning under Uncertainty},
  author        = {Harsh Muriki and Gayathri Aravindan and Yuchen Zheng and Sonia Chernova},
  year          = {2026},
  eprint        = {2610.17326},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2610.17326}
}

References

M. Savva, A. Kadian, O. Maksymets, et al. Habitat: A Platform for Embodied AI Research. ICCV, 2019. arXiv
A. Szot, A. Clegg, E. Undersander, et al. Habitat 2.0: Training Home Assistants to Rearrange their Habitat. NeurIPS, 2021. arXiv
X. Puig, E. Undersander, A. Szot, et al. Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots. arXiv, 2023. arXiv
M. Chang, G. Chhablani, A. Clegg, et al. PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks. ICLR, 2025. arXiv
P. Anderson, A. Chang, D. S. Chaplot, et al. On Evaluation of Embodied Navigation Agents. arXiv:1807.06757, 2018. arXiv
S. Yao, J. Zhao, D. Yu, et al. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629, 2023. arXiv
Z. Yang, C. Garrett, D. Fox, T. Lozano-Pérez, L. P. Kaelbling. Guiding Long-Horizon Task and Motion Planning with Vision Language Models. arXiv:2410.02193, 2024. arXiv
M. Ghallab, A. Howe, C. Knoblock, D. McDermott, et al. PDDL—The Planning Domain Definition Language. Technical Report, 1998.
W. Tang, X. He, Y. Huang, Y. Xiao, C. Lu, P. Cai. Tru-POMDP: Task Planning Under Uncertainty via Tree of Hypotheses and Open-Ended POMDPs. arXiv:2506.02860, 2026. arXiv
M. Li, S. Zhao, Q. Wang, et al. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. arXiv:2410.07166, 2025. arXiv
R. Yang, H. Chen, J. Zhang, et al. EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents. arXiv:2502.09560, 2025. arXiv