Section 17Experimentation and Evidence
Result Store requirements
Defines run manifests, traces, checkpoints, exports, comparisons, reproduction modes, and result integrity.
Section relationships
| ID | Name | Priority | Requirement | Verification |
|---|---|---|---|---|
| DATA-001 | Run manifest | MVP | Every run SHALL produce a manifest containing project and asset hashes; compiled scenario identity; software, solver, controller, extension, adapter, FMU, and ML-runtime identities; protocol versions; granted permissions and sandbox modes; RNG algorithm and stream layout; options; seeds; platform; execution mode; fidelity; calibration identities; and termination state. | Inspection |
| DATA-002 | Incomplete-run state | MVP | An interrupted or failed run SHALL remain identifiable as incomplete and SHALL not be reported as a successful result. | Test |
| DATA-003 | Structured event timeline | MVP | A run SHALL record selected contacts, limits, faults, sensor deliveries, controller activations, warnings, checkpoints, and termination events with virtual timestamps. | Test |
| DATA-004 | Selectable telemetry | MVP | A user SHALL select state and metric channels to record at declared rates. | Demonstration |
| DATA-005 | Open time-series export | MVP | Selected scalar telemetry SHALL be exportable as unit-annotated CSV, and run summaries and manifests SHALL be exportable as documented JSON. | Test |
| DATA-006 | Snapshot storage | MVP | Supported checkpoints SHALL be storable with version, backend, and integrity metadata. | Test |
| DATA-007 | Re-execution inputs | MVP | A run declaring input re-execution support SHALL retain all non-derivable inputs, events, seeds, and state required by its declared comparator. | Test |
| DATA-008 | Re-execution verification | MVP | Input re-execution SHALL report the compared channels, comparator, tolerances, build and platform scope, known nondeterministic participants, and first detected divergence. | Test |
| DATA-009 | Compression | Beta | Time-series, sensor, event, and snapshot data SHALL support configurable compression with integrity checks. | Test |
| DATA-010 | High-volume sensor data | Beta | Image, point-cloud, and other bulk data SHALL be stored without forcing serialization into scalar event records. | Test |
| DATA-011 | Schema versioning | MVP | Every persisted manifest, trace, event, telemetry, and snapshot schema SHALL be versioned. | Inspection |
| DATA-012 | Integrity check | MVP | Persisted run artifacts SHALL support detection of truncation or content corruption. | Test |
| DATA-013 | Run comparison | MVP | BeforeMetal SHALL produce a report of configuration, software, seed, fidelity, and selected metric differences between two runs. | Test |
| DATA-014 | Result-set summary | MVP | A batch SHALL produce aggregate status and metric summaries while retaining per-run results. | Test |
| DATA-015 | Data retention policy | Beta | A workspace SHALL support explicit retention or deletion policy for derived caches, traces, snapshots, and bulk sensor data. | Demonstration |
| DATA-016 | MCAP export | External | BeforeMetal SHALL permit export of supported message streams and attachments to MCAP through an optional adapter. | Test |
| DATA-017 | Parquet export | Beta | BeforeMetal SHOULD support Parquet export for tabular experiment and time-series analysis. | Test |
| DATA-018 | Reproduction bundle | Beta | BeforeMetal SHALL create a bundle or manifest that identifies all distributable inputs required to reproduce a run. | Demonstration |
| DATA-019 | Causal journal | MVP | Independently of user-selected telemetry, a re-executable run SHALL record every state-affecting input, runtime-action request and disposition when present, asynchronous order, arbitration decision, limiter result, dropped or late decision, sidecar output, and capture, admission, delivery, decision, and application time with deterministic sequence numbers. | Test |
| DATA-020 | Reproduction mode declaration | MVP | A run SHALL separately declare support and criteria for input re-execution, checkpoint resume, and recorded-state playback. | Inspection |
| DATA-021 | Dependency-aware retention | Beta | Deleting an artifact required by a reproduction mode SHALL require confirmation and SHALL update the affected run's manifest and status so that mode is no longer reported as available. | Test |
| DATA-022 | Synthetic dataset writer contract | Beta | BeforeMetal SHALL expose a versioned writer contract for bulk synthetic datasets that preserves canonical record semantics independently of exporter-specific directory, archive, or object-store layouts and reports every lossy conversion. | Test |
| DATA-023 | Synthetic dataset manifest | Beta | Every exported synthetic dataset SHALL identify its schema and writer, source runs, compiled scenarios, assets, annotation taxonomy, selected truth and sensor channels, frames, capture timing, seeds, realized randomization, software identities, completeness, and integrity metadata. | Inspection |
| DATA-024 | Regression evidence record | Beta | Each simulation regression result SHALL identify the case and suite revisions, expected-baseline content identities, compiled scenario and run identities, build, platform, capabilities, execution mode, seeds, runtime-action dispositions, comparator configuration, expected and actual values, first mismatch where applicable, termination state, diagnostics, and artifact integrity. | Inspection |
Change rationale (DATA-022, DATA-023): ML-015 defines episode records but does not provide a storage-neutral contract or complete provenance for perception-oriented synthetic datasets. These requirements keep dataset semantics portable and auditable without selecting one training framework or hosted data service.
Change rationale (DATA-019, DATA-024): Runtime-action requests and receipts are state-affecting causal inputs, so they extend the existing mandatory journal rather than creating a competing log. Regression outcomes use a separate evidence record because suite identity, expected baselines, comparator configuration, and pass/fail evidence are not ordinary run telemetry.
Generated from the canonical specification. Edit section metadata or prose in docs/requirements.md; the website rebuilds this page and its relationships automatically.