Reproduce environmental proposals, offline faults and paired comparisons

Use the physical-AI evaluation workflow when you need to compare environmental proposals, fault handling, isolated database checks, or paired candidates under controlled inputs. The supported flow runs the device simulator with rules or fake adapters, an optional dataset and context, bounded budgets, and a fresh result file when you need one. It also supports a prepared comparison mode, an isolated PostgreSQL evaluation, and a fault matrix that records measured, failed, and unmeasured rows without treating transport acknowledgements as physical confirmation.

Run the environmental evaluation

Build the device simulator first, then run evaluate:physical-ai with the rules adapter and a fixed seed. Use --dataset, --context, --budget, and --output when you need explicit fixture files, bounded budgets, or a new result file. Ordinary replay accepts only rules and fake adapters; it does not use provider network access or physical actuation.

The default environmental-v2.json dataset contains thirty timed cases: one positive and one adversarial case for each of fifteen environmental families. Each case carries its own authorized initial view, observations, retained goals, bounded before/after events, and a private outcome oracle. environmental-v1.json remains an explicit validator-boundary fixture; its deliberately malformed proposals are not a model reasoning benchmark.

The validator checks proposals against captured controls, current descriptors, pinned routines, and freshness. The timed simulator calls the shared begin, attempt, dispatch, ACK, finish, and observe reducers, and it sends revalidate permission and context while requiring approval and confirmed dependencies. The separate outcome oracle checks declared alternative plans for required queries, approval, forbidden effects, dependency ordering, duplicate dispatch, and independent confirmation evidence. Crash, timeout, and invalid output remain in the denominator. Fixed seed and input runs retain deterministic per-case hashes. The environmental evaluation keeps parent-process CPU and process-lifetime RSS bounded, and the portal-required checks stay attached to the existing accepted workflow rather than any new interface.

A transport acknowledgement never confirms a physical result; correlated fresh device reports can confirm unresolved sends, and UNKNOWN stays UNKNOWN after interruption or timeout until later matching evidence resolves recovery separately.

Use the isolated database evaluation

The isolated PostgreSQL evaluation uses the existing Tenant context, validation, ledger, and fence owners, together with the current guarded schema and synthetic identities. Run the database check from the platform directory with the repository’s bounded runner. Fence preparation and simulated evidence verify software contracts only; the fixture sends no connector commands and does not qualify hardware.

Register faults and measurements

The fault driver represents WAN, LAN, controller, inference, authority expiry, local restart, and reconnect as separate failure modes. Register a bounded probe against explicitly owned loopback origins. runFaultMatrix emits every capability and phase row, including unmeasured and failed rows. Its timed replay and outcome checks keep raw proposedWrites, validatedWrites, unsafeProposedWrites, and replayedEffects distinct, and the main loop independently checks per-case approval, outcome, retained state, connectivity, and recovery; zero physical connector sends occur.

Record the actual installation’s SKU, hardware and software revisions, installed routine version, authority generation, six placement axes, and device paths. Never cut a production network or infer local UI reachability from a functioning radio.

Compare paired candidates

For paired comparisons, use identical authorized initial views and finite budgets. Additional retrieval is the only ablation switch. Candidate manifests identify role, artifact, runtime, placement, and hardware. A missing adapter yields an unmeasured comparison.

Record sampler scope explicitly. Two RSS samples are a sampled maximum, not a measured peak. The prepared CLI also records the operating system process-lifetime RSS peak with that explicit scope; an owned-runner sampler can supply its separately scoped measured peak. A decreasing CPU counter fails measurement.

The comparison input hash binds dataset, context, candidates, budget, seeds, and retrieval choices. Absent power, temperature, and pressure remain null. Cold and warm adapter phases do not imply cold and warm artifact loading. Confirmation and dispatch timelines remain separate from inference latency, and fake software results never establish model superiority or hardware qualification.

Preserve cache recovery and offline baselines

Cache recovery preserves the last-good served configuration after failed persistence, serializes replacements, and rejects malformed roots. Older optional cache fields retain their existing compatibility. User-facing state age, privileges, and physical confirmation remain governed by their existing owners; a cached configuration grants no new authority.

For the owned offline baseline, run the prepared command after building home-agent, app-core, and device-simulator. It owns temporary storage and injects failures into the actual cache, sync, and API-client seams. The command retains eighty capability and phase rows. Use optional --sku values HUB, HUB_MUSIC, or HUB_AI, and optional --deployment values PLATFORM or SELF, or run the CI wrapper for all six software installation profiles. Each profile retains 24 real cache, sync, and client seam rows and 56 explicitly simulated path rows. The 144 real software rows and 336 simulated rows are never installed-hardware or real-model measurements.

Register an actual isolated installation probe through OfflineProbe for target qualification. A hosted API uses its WAN link; a self-hosted API uses its explicitly selected LAN link. Neither injected client seam proves installed web, mobile, or TV reachability.

Use --comparison prepared with the evaluation command to compare rules and fake-model slots under identical inputs and retain unloaded Voice and Reasoning slots explicitly. Programmatic ComparisonSlot registration supplies an adapter, sampler, and, for real candidates, an exact trusted loaded-artifact and runtime receipt. Missing or mismatched receipts cannot establish model measurements. Sampler failures and timeouts remain failed rows. The model-facing harness excludes native bindings and oracle expectations. Its monotonic virtual clock records freshness and lease expiry without changing host time.

Physical AI Offline Local Fixture

It records per-case elapsed time and parent-process CPU usage, plus explicitly scoped process-lifetime RSS peak. These measurements describe the owned Linux software fixture, not child-process-inclusive CPU, hub concurrency, power, temperature, acoustic AEC, installed mobile reachability or a trained model. The fake model has no physical effects, and no physical command or external network request is sent.

The extension’s eight measured reasoning rows cover normal operation, WAN loss, expiry, process restart, reconnect, LAN loss, controller withdrawal and inference saturation. The other seventy-two capability rows remain explicitly unmeasured by this extension; retain the independent baseline receipt for their existing coverage. Preserve failed and unmeasured rows, and keep those qualification scopes separate.

Restart preserves durable facts while withholding execution authority. Expired or revoked authority refuses context; saturated inference refuses an additional call while the projection remains readable, and cancellation releases the owned runner. The opt-in command owns temporary files, closes its runner and refuses to overwrite an existing raw receipt.

Interpret timed replay evidence

Before invoking an adapter, replay updates the actual authorized context with observations, connectivity, revision and permission changes. Session reset clears the fixture’s ephemeral dialogue and retains environmental observations, goals, and routines. After a proposal, the fake physical owner revalidates each send through ALYT’s shared validator. It requires approval where declared, confirms dependencies before dependent sends, fences revoked, revised, or unavailable owners, and deduplicates restart retries. The fake owner calls the actual shared begin, attempt, dispatch, ACK, finish, and observe outcome reducers; it does not implement a second outcome machine. Only a fresh synchronized post-dispatch report with the matching target, value, and command can confirm an effect. Late admissible evidence resolves recovery while preserving the original UNKNOWN outcome. ACK-only timeout is UNKNOWN; failed effects remain FAILED. A coordinated plan can retain one CONFIRMED effect and another UNKNOWN effect.

The per-case oracle checks sent effects, approvals, forbidden effects, dependency order, outcomes, retained state, and final connectivity in the main evaluation loop. proposedWrites counts raw proposed actions, validatedWrites counts authorized normalized writes, and unsafeProposedWrites counts rejected proposed actions. replayedEffects counts virtual sends; simulatedPhysicalEffects: 0 denotes no physical connector transport. Neither replay nor a passing finite rules table measures trained-model quality, installed client reachability, actual PostgreSQL execution, or hardware. The separate isolated database fixture retains its own proof boundary. Native IDs, descriptors, and oracle expectations stay outside adapter input; models receive the authorized view and bounded queries only.