Experiment Orchestrator requirements
Defines immutable runs, runtime-action plans, sweeps, random sampling, comparisons, parallel execution, calibration, and optimization hooks.
Section relationships
| ID | Name | Priority | Requirement | Verification |
|---|---|---|---|---|
| EXP-001 | Immutable run specification | MVP | A launched run SHALL reference an immutable compiled scenario, controller identity, seed, initial state, metric set, and stop conditions. | Inspection |
| EXP-002 | Run identity | MVP | Every run SHALL receive a stable unique identity before execution begins. | Test |
| EXP-003 | Stop conditions | MVP | An experiment SHALL support virtual-time, mission, safety, failure, convergence, and explicit user stop conditions where applicable. | Test |
| EXP-004 | Early termination | MVP | A metric or guard SHALL be able to terminate a run with a named reason at a safe scheduling boundary. | Test |
| EXP-005 | Named metrics | MVP | An experiment SHALL select metrics by stable identity and version. | Inspection |
| EXP-006 | Reference rover metrics | MVP | The reference experiment SHALL report mission success, elapsed time, trajectory tracking or declared reference-comparison measures, stability, slip indicators, torque, current, energy, and termination reason. | Demonstration |
| EXP-007 | Finite parameter sweep | MVP | The orchestrator SHALL generate and execute the Cartesian product or explicit list of selected parameter values. | Test |
| EXP-008 | Random sampling | Beta | The orchestrator SHALL generate repeatable samples from declared parameter distributions. | Test |
| EXP-009 | Correlated sampling | Beta | The orchestrator SHALL support documented correlation among selected parameters. | Analysis |
| EXP-010 | Repetitions | MVP | An experiment SHALL support repeated runs with explicit seed policy. | Test |
| EXP-011 | Variant comparison | MVP | A result set SHALL compare metrics across robot, controller, and scenario variants without requiring identical internal backend state. | Demonstration |
| EXP-012 | Independent-world parallelism | Beta | The orchestrator SHALL execute independent worlds concurrently subject to deterministic mode and resource policies. | Analysis |
| EXP-013 | Progress and cancellation | MVP | A batch SHALL expose progress and support graceful cancellation without marking incomplete runs as complete. | Test |
| EXP-014 | Resume batch | Beta | A batch SHALL resume pending runs without repeating verified completed runs unless explicitly requested. | Test |
| EXP-015 | Resource budget | Beta | An experiment SHALL support configured CPU, memory, GPU, storage, virtual-time, and wall-time budgets where measurable. | Test |
| EXP-016 | Monte Carlo summary | Beta | A stochastic experiment SHALL report sample count, seed policy, distributions, confidence summaries, and failed-run handling. | Analysis |
| EXP-017 | Sensitivity analysis | Beta | BeforeMetal SHOULD support local or global sensitivity analysis for selected parameters and metrics. | Analysis |
| EXP-018 | Parameter calibration | Beta | BeforeMetal SHALL support calibration of selected parameters against a declared calibration dataset and objective. | Analysis |
| EXP-019 | Held-out separation | Beta | Calibration workflows SHALL prevent designated validation observations from being used by the fitter. | Test |
| EXP-020 | Optimization hook | Future | BeforeMetal MAY expose experiment evaluation to an external design-optimization algorithm. | Demonstration |
| EXP-021 | Pareto comparison | Future | BeforeMetal MAY report Pareto-efficient designs for multiple selected objectives and constraints. | Analysis |
| EXP-022 | Runtime-action plan | Beta | An experiment SHALL support a versioned plan of stable-identity runtime-action requests whose type and version, origin, target or compiled definition, requested virtual time or trigger, typed payload, bounds, and invalid, late, or failure policy are fixed before the run. | Test |
| EXP-023 | Calibration result record | Beta | Every completed calibration SHALL record parameter identities, bounds, initial and final values, residual definitions and weights, time alignment or delays, dataset identities, optimizer identity and settings, seed where applicable, termination and convergence status, final residuals, and uncertainty method and result. | Inspection |
| EXP-024 | Practical identifiability diagnostics | Beta | A multi-parameter calibration SHALL provide a local practical-identifiability analysis using a declared rank-revealing method and SHALL report parameter scaling, sensitivity source, observation weights, sample points, rank criterion, spectrum or conditioning, correlated or near-null directions, and an explicit unidentifiable result where applicable. | Analysis |
Change rationale (EXP-006): The previous unconditional “trajectory error” implied a physical truth dataset. The reference workflow always reports its simulated trajectory and may report error only against an identified analytical, published, manufacturer, community, independent, or optional empirical reference.
Change rationale (EXP-022): Weather timelines and fault injection already imply scheduled changes to live state, while topology and controller changes lacked one native, immutable authoring contract. The plan describes requests against a future world instance; it does not mutate the project or compiled scenario and does not grant omnipotent runtime access.
Change rationale (EXP-023): EXP-018 requires parameter fitting but not enough evidence to reproduce or judge the fit. A complete result record is necessary before identifiability diagnostics or a calibrated model can be interpreted honestly.
Change rationale (EXP-024): Parameter sensitivity alone does not establish whether selected observations distinguish the fitted parameters. The new diagnostic is explicitly local, method-specific, and permitted to conclude that parameters are unidentifiable; it is not a claim of global or structural uniqueness.
Generated from the canonical specification. Edit section metadata or prose in docs/requirements.md; the website rebuilds this page and its relationships automatically.