[Proposal] Unify Atomic Skill benchmarks under configuration-driven suites

[yuecideng]

Proposal

Build a single configuration-driven benchmark path for Atomic Skills by extending the existing scripts/benchmark/motion_generation suite framework instead of maintaining one standalone benchmark runner per skill.

A suite configuration should select:

• an embodiment component and optional initial-configuration overrides;
• one or more planner adapters and their configurations;
• Atomic Skills, cases, and physical stage evaluators;
• nominal, robustness, and held-out generalization domains;
• the measurement and reporting protocol.

The resolved suite should generate a planner-independent, versioned case manifest, execute every compatible planner on the same cases, evaluate success from physical scene evidence, and emit the existing reproducibility artifacts and report.

The intended data flow is:

Suite YAML
  ├─ embodiment component + overrides
  ├─ planner specifications
  ├─ Atomic Skill cases
  └─ versioned generalization domains
              ↓
      resolved case manifest
              ↓
 generic motion-generation runner
              ↓
 physical stage evaluators
              ↓
 trials.jsonl + aggregates.json + report.md

Reuse existing owners

• Reuse SuiteCfg, planner/scenario/robot registries, the generic runner, and the artifact/report pipeline introduced by [Proposal] Neural Motion Generator benchmark with cuRobo baseline #437.
• Reuse official embodiment components under embodichain_tasks/configs/components/embodiments/ rather than defining benchmark-only robot constructions.
• Represent each Atomic Skill through an AtomicSkillCaseProvider plus a physical evaluator. Invocation construction, case generation, stage measurement, and aggregation should remain separate concerns.
• Reuse the per-skill stage model being developed in Benchmark seven more atomic skills #642 after its physical-success criteria are corrected and covered by counterexample tests.

Configuration model

The suite should make the following concepts first-class:

1. Embodiment — component path, optional overrides, initial joint configuration, endpoint bindings, and runtime services.
2. Planner — existing PlannerSpecCfg/adapter selection and planner-owned parameters.
3. Atomic Skill — skill ID, options, cases, ordered physical stages, tolerances, and required capabilities.
4. Generalization domain — stable domain ID and version, applicability, required capabilities, case generator or frozen cases, and perturbation/held-out metadata.
5. Protocol — seeds, repeats, replay timing, recording policy, thresholds, and report schema version.

Generalization domains must distinguish:

• nominal evaluation;
• in-domain robustness perturbations;
• held-out adaptability domains such as a new object, scene layout, end effector, or embodiment.

Case inclusion must be frozen before candidate evaluation. unsupported must come from an independent capability contract or an exhaustive, reproducible validator; it must not be inferred from a truncated probe or from the result of the planner being ranked.

Physical evaluation contract

Stage success must come from simulator observations, contact/effect evidence, or other measured scene state. CompiledTrajectory.projected_context is hypothetical planning state and must not be used as proof that an object was grasped, released, transferred, or retained.

The evaluator interface should support:

• ordered, skill-specific stages;
• segment-boundary measurements;
• shared cross-cutting rules such as premature object drop;
• stable failure stages and taxonomy;
• replay/video using the same timing schedule as the scored execution;
• per-environment outcomes for batched cases.

Migration phases

1. Correct the shared physical measurement contract and introduce the evaluator interface.
2. Load official embodiment components and migrate the currently benchmarked Atomic Skills into AtomicSkillCaseProvider implementations.
3. Add versioned generalization domains and aggregate nominal/robustness/adaptability statistics.
4. Retain current per-skill commands only as compatibility wrappers, then remove duplicated lifecycle and reporting code when migration is complete.

Acceptance criteria

• A suite can select an official embodiment component, planner adapter, Atomic Skill, and cases without editing Python runner code.
• At least two embodiments and two planners can run compatible skills through the same manifest and report schema.
• All physical stages are measured from replay observations; projected planning effects do not decide physical success.
• The manifest records resolved embodiment, planner configuration hash, skill options, domain/version, seed, and case parameters.
• One nominal domain and at least one versioned generalization domain are supported end to end.
• Reports include coverage, stage rates, overall success, and domain-level mean/worst/variance/retention where applicable.
• Every run emits resolved_suite.yaml, environment.json, case_manifest.json, trials.jsonl, aggregates.json, and a Markdown report.
• Unsupported capability, ordinary task failure, invalid case, and runtime error remain distinct.
• Existing motion-generation and Atomic Skill focused tests continue to pass, with new negative tests for incorrect physical evidence and generalization aggregation.

Motivation

EmbodiChain currently has two complementary benchmark paths:

• scripts/benchmark/motion_generation already provides suite YAML, planner and scenario registries, frozen manifests, raw artifacts, aggregation, and reporting.
• The Atomic Skill benchmark work in Benchmark seven more atomic skills #642 expands skill coverage and introduces skill-specific stage chains, but each skill still owns a large standalone Python script with hard-coded robot, planner, scene, cases, replay, and report assembly.

This split makes it difficult to answer the benchmark question users actually care about:

Given an embodiment/configuration, a planner, an Atomic Skill, and a generalization target, how well does the complete physical behavior perform?

Without one configuration and artifact contract, adding a planner, embodiment, skill, or generalization domain requires edits across multiple runners. Results are also difficult to compare because scene construction, replay timing, success evidence, unsupported classification, and output metadata can diverge between scripts.

A unified suite would preserve the strongest parts of both implementations: the reproducibility and extensibility of the motion-generation framework, and the per-skill physical stage semantics of the Atomic Skill benchmarks.

Related work: #437 established the generic motion-generation benchmark architecture; #642 expands Atomic Skill coverage and motivates convergence onto that architecture.

Additional context

A minimal target configuration could look like:

schema_version: 1
name: atomic_skill_generalization
suite_version: atomic_skill_generalization_v1

embodiment:
  component: ../../embodichain_tasks/configs/components/embodiments/franka_panda.yaml
  overrides: {}

planners:
  - id: curobo
    adapter: curobo
    role: primary_baseline
    enabled: true
    config: {}

tracks:
  - id: nominal-pick
    scenario: atomic_task
    skills:
      - id: pick_up
        cases: [cube_top_center]
    domains:
      - id: nominal
        version: v1
      - id: held_out_objects
        version: v1
        objects: [unseen_box, unseen_cylinder]

protocol:
  seeds: [11, 23, 37]
  measured_trials: 3

The exact schema should remain small and reuse existing component/registry contracts rather than introduce a third configuration system.

Checklist

• I have checked that there is no similar issue in the repo (required)

[yuecideng]

Proposed three-person work allocation (for meeting discussion)

Suggested ownership: yuecideng owns the unified framework and generalization statistics; matafela owns embodiment/planner integration and skill migration; Yuan-Xinyi owns physical evaluation and stage criteria. This is a discussion draft. Assignments, timelines, and final interfaces remain to be agreed upon.

1. Scope

The work should establish three shared properties:

• Configurability: select embodiments, planners, skills, and evaluation domains through YAML.
• Trustworthy evaluation: determine grasp, release, transfer, and retention success from physical observations.
• Comparability: evaluate planners against cases frozen in advance, with a common execution schedule and statistical protocol.

The existing scripts/benchmark/motion_generation framework already includes SuiteCfg, a generic runner, AtomicSkillCaseProvider, and several integrated skills. Providers still own task_result() evaluation, and some robot construction depends on tutorial helpers. The work should extend the existing framework, clarify ownership, and migrate integrations.

2. Proposed ownership

Owner	Responsibility	Main deliverables	Acceptance focus
yuecideng	Framework, protocol, generalization, and statistics; integration lead	Extend suite configuration and data models; freeze and version manifests; connect evaluators to the runner; register domains and distinguish nominal/robustness/held-out evaluation; implement aggregation, reports, and reproducibility artifacts	Changing planners does not change the case set; outcome categories and statistical denominators are explicit; generalization results can be recomputed from raw records
matafela	Embodiments, planners, case providers, and migration	Load official embodiment components and overrides; bind joints, endpoints, and runtime services; integrate planner capabilities; migrate existing case/invocation construction; convert old commands into compatibility wrappers	Compatible combinations of two embodiments and two planners actually run; configured selection of integrated skills and cases requires no runner changes
Yuan-Xinyi	Physical observations, stage evaluators, and evaluation correctness	Correct and reuse the stage model from #642; collect physical evidence at segment boundaries; evaluate grasp, release, retention, and premature drop; provide per-environment outcomes and failure attribution; align scored execution and video timing	Counterexamples reject false positives; projected planning state cannot establish physical success; each environment in a batch is evaluated independently

This allocation follows existing work: yuecideng has contributed to the suite/atomic benchmark framework, Yuan-Xinyi is developing #642, and matafela has contributed to cuRobo integration. Final assignments should account for familiarity and available time, as confirmed in the meeting.

3. Agree on three interfaces first

Case provider → runner: led by matafela, jointly agreed with yuecideng.

• Provide frozen initial state, targets, skill options, and required capabilities, and construct invocations.
• Delegate physical success decisions to the evaluator.
• Keep case identity and content independent of the planner. Store planner configuration hashes as associated run metadata, traceable through the manifest/run artifacts.
• Derive unsupported from an independent capability contract or sufficient reproducible validation, never from candidate planner failure or a truncated probe.

Execution observations → evaluator: led by Yuan-Xinyi, with scene integration support from matafela.

• Define segment boundaries, timestamps, object/joint state, and required contact or effect evidence.
• Return per-environment stage outcomes, measurements, and failure reasons.
• Distinguish a failed stage, an unreached stage, and a stage that is not applicable.
• Never use projected_context as evidence of physical success; use the same timing contract for video replay and scored execution.

Trial → aggregation: led by yuecideng, with field semantics agreed by the other two owners.

• Define identity fields for case, domain/version, embodiment, planner hash, seed/repeat, and environment row within a batch.
• Distinguish unsupported, invalid case, ordinary task failure, and runtime error.
• Specify denominators for coverage, stage success rates, and overall success, plus definitions and applicability rules for domain mean/worst/variance/retention.

4. Delivery sequence

Stage	yuecideng	matafela	Yuan-Xinyi
Agree on contracts	Lead schema, outcome categories, and statistical definitions	Confirm component, capability, and case interfaces	Confirm observation, stage, and evaluator interfaces
Implement in parallel	Runner integration, artifacts, domain/aggregation unit tests	Official component loading and the first skill provider	Evaluator, physical counterexample tests, and timing
First end-to-end run	Integrate configuration through reporting	One embodiment + planner + pick_up	Verify actual lifting, retention, and failure attribution
Expand and validate	Generalization statistics and complete reports	Second embodiment/planner, remaining existing benchmark skills, and compatibility entry points	Evaluation across skills and batched environments

Agree on the physical evaluation contract first. Component integration and statistical implementation can then proceed in parallel. Expand skill migration after validating the first end-to-end run. That run is an integration checkpoint; final acceptance still requires multiple embodiments and planners, a nominal domain plus at least one versioned generalization domain, and all required artifacts.

5. Collaboration and PR boundaries

Suggested logical changes, with the number of PRs adjusted to their size:

1. Shared interfaces and minimal framework: led by yuecideng, agreed by all three.
2. Physical observations and evaluator correctness: owned by Yuan-Xinyi.
3. Official embodiments, planner integration, and first skill migration: owned by matafela.
4. Generalization domains, aggregation, and reporting: owned by yuecideng.
5. Remaining skill migration, compatibility wrappers, and integration acceptance: matafela leads migration, Yuan-Xinyi supplies the corresponding evaluators, and yuecideng coordinates integration.

To reduce conflicts, first separate the provider, observation, and evaluator responsibilities currently concentrated in atomic_task.py. Coordinate changes to config.py, models.py, and runner.py through yuecideng. Each owner supplies tests for their modules; interface changes and acceptance results receive cross-review.

6. Acceptance priorities and meeting decisions

Pay particular attention to two failure modes:

• Measurement false positives (Yuan-Xinyi): the gripper closes without grasping the object, the object drops during transport, or the object reaches the target vicinity without being released. Add counterexample tests for these cases.
• Misleading statistical improvements (yuecideng): planner failures become unsupported, failed cases disappear from aggregation, or unequal domain sizes hide poor worst-domain performance. matafela supplies real scenes and independent compatibility evidence.

Decisions proposed for the meeting:

• Confirm the three assignments and the final owner of each interface.
• Select the embodiment, planner, and pick_up case for the first end-to-end run.
• Agree on the compatibility matrix for the second embodiment/planner and select the first versioned generalization domain.
• Define stage success rates, coverage, retention, and other aggregation rules.
• Decide which parts of Benchmark seven more atomic skills #642 require correction before reuse, and agree on skill migration priorities and milestones.