Bench2Arena: Xspark AI and NVIDIA Bring Embodied Benchmarks to Isaac Lab-Arena

Xspark AI and the NVIDIA Isaac Lab-Arena^[1] team jointly developed Bench2Arena, a human-supervised agentic compiler that systematically brings existing embodied benchmarks into Isaac Lab-Arena. It preserves each benchmark’s original task definitions and evaluation logic while enabling composable environments, standardized evaluation, and GPU-parallel testing. Bench2Arena currently connects RoboTwin^[2][3] and RoboDojo^[4] to Isaac Lab-Arena for evaluation. Going forward, the two teams will continue working together to expand the embodied simulation and evaluation ecosystem.

What Bench2Arena Does

Robot-learning benchmarks rarely transfer cleanly across simulators. Assets, physics, cameras, success criteria, and policy interfaces are often tightly coupled to a specific engine. Simply porting scene code may produce a runnable task while silently changing what it evaluates.

Bench2Arena therefore treats benchmark migration as semantics-preserving compilation. Before compiling a task into Isaac Lab-Arena, it captures the source benchmark in an evidence-backed migration contract covering:

1. Assets (A): identity, scale, collisions, articulation, and materials
2. Scene (S): layouts, resets, randomization, and state interfaces
3. Evaluator (E): success conditions, thresholds, stages, and termination
4. Policy (P): observations, actions, checkpoints, and evaluation settings
5. Release (R): installation, weight loading, and minimal evaluation

Every contract item must be supported by evidence; otherwise, it is recorded as an unresolved assumption. A task that runs is not necessarily a task whose original evaluation semantics have been preserved.

Core Technology: Auditable Compilation and Dual-Loop Verification

Compiling benchmarks into Arena. Bench2Arena maps each source benchmark onto Isaac Lab-Arena’s Object, Scene, Embodiment, and Task components. A coordinator divides the migration into parallel workstreams with clear ownership, resource budgets, and completion criteria. Versioned charters, handoffs, merge records, and static preflight checks make the process traceable, while humans retain control over ambiguous semantic decisions and costly GPU evaluations.

Loop A verifies the evaluator. Before running physics or policies, Bench2Arena feeds identical synthetic object-state sequences into the source and Arena success evaluators and compares their outputs step by step. This isolates whether success criteria, thresholds, and termination logic were translated correctly.

Loop B verifies task behavior. The same policy checkpoint is then evaluated under matched layouts and observation/action conventions. When results diverge, Bench2Arena traces the difference across initialization, evaluator logic, policy adapters, assets and physics, and evaluation settings. After a fix is approved, only the affected tasks are re-evaluated.

Results: Scenes, Video, and Cross-Benchmark Generalization

Initial-Scene Alignment on Ten RoboDojo Tasks

We compiled ten bimanual manipulation tasks from RoboDojo into Isaac Lab-Arena^[4], covering tower building, object sorting, block covering, coin deposit, table organization, pouring, bottle disposal, and nesting-doll ordering.

Under matched layout0 configurations, we compared the first frames from three head-camera views between the source benchmark and Arena. Across all tasks and cameras, the results achieve a mean SSIM of 0.830 and a mean absolute difference (MAD) of 13.71. These measurements provide supporting evidence that the initial scenes are aligned, but do not alone establish full simulation fidelity.

Open-loop replay video

Open-loop replay shows what accumulates after contact. The videos below replay the same 852-frame joint trajectory on cover_blocks: RoboDojo on the top row, Arena on the bottom row. Columns are head, left wrist, and right wrist. The original difference panel has been cropped out.

Semantic and Policy Validation

Across ten RoboDojo tasks, Loop A evaluated 154 synthetic-state steps from 31 episodes and achieved step-by-step agreement between the source and Arena evaluators. This verifies the translation of success logic, but does not establish equivalence in physics or policy behavior.

Loop B then evaluated the same π0.5 checkpoint^[5][6] under matched layouts. The ten-task macro success rate was 19.0% in the source environment and 17.4% in Arena—a gap of −1.6 percentage points. However, the differences varied across tasks. For example, put_bottles_into_dustbin initially achieved only 26% in Arena, compared with 76% in the source. Evidence-driven diagnosis and repair raised the Arena result to 68%. A runnability-only migration process would have accepted the task at 26% and missed this semantic mismatch.

These results evaluate the migration process using the same checkpoint, matched layouts, and predefined episode counts. They should not be interpreted as an official reproduction of leaderboard results.

Extending Bench2Arena to RoboTwin

To demonstrate that Bench2Arena generalizes beyond a single benchmark, we also compiled representative RoboTwin tasks into Isaac Lab-Arena^[3]. The video below compares stack_bowls_three in RoboTwin on the left and Arena on the right.

Camera intrinsics and image resolution are matched, while camera extrinsics are not strictly aligned. The video therefore provides a qualitative comparison of scene composition and task behavior rather than a pixel-level geometry audit.

More importantly, Bench2Arena supports a complete target-domain learning loop. RoboTwin trajectories are replayed in Arena, observations are collected directly from the target environment, and both policy training and evaluation remain within Arena. On stack_bowls_three, an ACT policy^[7] trained on the collected Arena data achieves a 68% success rate, demonstrating that the compiled task supports the full pipeline—from trajectory replay and data collection to training and evaluation.

Once a migration is validated within its stated evidence bounds, the task can also benefit from Arena-native capabilities. Objects, scenes, and robots can be recomposed without duplicating task code; lighting, camera configurations, and object poses can be systematically varied; and GPU-parallel simulation makes large-scale evaluation and robustness sweeps practical.

What NVIDIA Isaac Lab-Arena Is

NVIDIA Isaac Lab-Arena^[1] is an open-source framework on NVIDIA Isaac Lab^[8] for large-scale policy setup and evaluation in simulation. Environments are assembled at runtime from Object, Scene, Embodiment, and Task. The same task can change objects, robots, or backgrounds without rewriting task logic, and evaluations can run in GPU-accelerated parallel environments.

Bench2Arena fills the missing step: compiling an external benchmark onto that substrate, and stating which semantics have been verified and which have not. See the NVIDIA technical blog^[9] and the product page.

Why Migrate to NVIDIA Isaac Lab-Arena

Migrating to NVIDIA Isaac Lab-Arena is not merely a matter of reproducing an existing benchmark in another simulator; it incorporates that benchmark into a unified evaluation infrastructure that is compositional and scalable. Arena organizes Object, Scene, Embodiment, and Task as core abstractions, allowing benchmarks from different sources to be specified, executed, and compared on a shared substrate. Once Bench2Arena has made migration semantics and verification boundaries explicit, subsequent performance differences can be more readily attributed to the factors under study rather than to the migration process itself.

First, Arena endows benchmarks with compositional scalability. A migrated task is decomposed into reusable components, so new object sets, scene conditions, or robot embodiments can be generated through recombination and configuration. Migration thus produces not a one-off environment reproduction, but benchmark infrastructure that can be extended systematically.

Second, Arena shifts evaluation from reporting a single outcome to controlled, diagnostic analysis. Its structured task definitions decompose success conditions into sub-task predicates, while factors such as lighting, camera pose, and object pose can be swept systematically. Researchers can therefore determine not only whether a policy fails, but also which factors it is sensitive to and at which stage of the task failure first arises.

Finally, Arena’s GPU-parallel execution makes large-scale evaluation and perturbation sweeps more practical, and provides a shared target domain for data collection, training, replay, and evaluation. This unification reduces the additional variables introduced by cross-simulator train/evaluation splits, allowing benchmarks, policies, and environment variants to be compared under more consistent protocols.

What’s Next: Scaling Bench2Arena and Building Isaac Lab-Arena Together

Xspark AI and the NVIDIA Isaac Lab-Arena team have jointly developed Bench2Arena to bring widely used robot-learning benchmarks into a shared, scalable evaluation framework. Initial work on RoboTwin and RoboDojo demonstrates a systematic path from benchmark compilation to semantic verification, policy evaluation, and target-domain training.

RoboTwin integration. Representative RoboTwin tasks have been compiled into Arena’s Object–Scene–Embodiment–Task structure. The resulting pipeline supports trajectory replay, observation collection, policy training, and evaluation entirely within Isaac Lab-Arena.

RoboDojo validation. Bench2Arena has been evaluated on ten RoboDojo tasks. The results show that success logic can be systematically aligned and that policy-level differences can be identified, attributed, and repaired—going beyond the basic requirement that a migrated task simply runs.

Joint development of Isaac Lab-Arena features. Going forward, Xspark AI and the NVIDIA team will jointly expand RoboDojo coverage and develop reusable features directly within Isaac Lab-Arena. Planned directions include evaluator translation, matched-layout evaluation, residual attribution, long-horizon capability assessment, sim-to-real comparison, compositional perturbation, and large-scale sensitivity analysis.

The collaboration will therefore extend beyond migrating individual benchmarks. Our shared goal is to continuously strengthen Isaac Lab-Arena as an open, composable, and verifiable foundation for embodied simulation and evaluation.

References

[1] Isaac Lab-Arena: Composable Environment Creation and Policy Evaluation for Robotics. NVIDIA Isaac Lab-Arena Contributors. https://github.com/isaac-sim/IsaacLab-Arena

[2] RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins. Yao Mu, Tianxing Chen, Zanxin Chen, et al. https://openaccess.thecvf.com/content/CVPR2025/html/Mu_RoboTwin_Dual-Arm_Robot_Benchmark_with_Generative_Digital_Twins_CVPR_2025_paper.html

[3] RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. Tianxing Chen, Zanxin Chen, Baijun Chen, et al. https://arxiv.org/abs/2506.18088

[4] RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies. Tianxing Chen, Yue Chen, Zixuan Li, et al. https://arxiv.org/abs/2607.04434

[5] π0.5: A Vision-Language-Action Model with Open-World Generalization. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, et al. https://arxiv.org/abs/2504.16054

[6] openpi: Open-source models and packages for robotics. Physical Intelligence. https://github.com/Physical-Intelligence/openpi

[7] Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Tony Z. Zhao, Vikash Kumar, Sergey Levine, Chelsea Finn. https://doi.org/10.15607/RSS.2023.XIX.016

[8] Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. NVIDIA. https://arxiv.org/abs/2511.04831

[9] Simplify Generalist Robot Policy Evaluation in Simulation with NVIDIA Isaac Lab-Arena. Sangeeta Subramanian. https://developer.nvidia.com/blog/simplify-generalist-robot-policy-evaluation-in-simulation-with-nvidia-isaac-lab-arena/