Phase 1 Retrospective — Simulator to Real-Machine Scale

• Date: 2026-05-03
• Author: Phase-1 closeout review
• Phase: 1 (Simulator to scale) — complete
• Companion docs:
  • Evolution Roadmap — the five-phase plan
  • Phase 1 Measurements — the raw evidence
  • Roadmap Progress — per-slice status + session log

Executive summary

Phase 1 ran from 2026-04-22 (Phase 0 exit gate met) to 2026-05-03 (Phase 1 exit gate met). Five slices shipped. Two slices were superseded mid-phase. The exit gate was met cleanly once a measurement-methodology amendment to one criterion landed.

Slice	Title	Status
1.1	Multi-Tag Telemetry	Completed (criterion 7 amended)
1.2	Real Frame Payloads	Completed
1.3	Encoder-Rate Motion	Completed (criterion 7 amended)
1.4	Storm & Soak Profiles	Completed (criterion 12 amended)
1.5	Automated Measurement Capture	Superseded by 1.6
1.5.1	Disposal-fix follow-up	Superseded
1.6	FlaUI-driven Measurement Capture	Completed

The headline outcome: the simulator produces real-machine-level load (50 telemetry tags at 1–500 Hz, 2 MP frames at 30 fps with real byte[] payloads, 200–657 Hz encoder stream, fault-burst chaos profiles, 8-hour real-time soak) and the application survives all of it cleanly. frames.dropped is 0 or near-zero on every Phase 1 row. runs.faulted is bounded by the fault-injection rate. Working-set steady-state is flat over 8 hours. GC pause p95 stays in the 7.9–12.4 ms band across every load shape.

The roadmap §3 said: "If the app survives this run beautifully, Phase 2 is deferred." The app did. Phase 2 opens with a measurement-first foundation slice (SLICE-2.0) to instrument AppStateStore.Update; the data from that capture decides whether the originally-planned 2.1/2.2/2.3/2.4 slices are load-bearing or can be deferred.

What each slice shipped

SLICE-1.1 — Multi-Tag Telemetry

Replaced the legacy MachineTelemetry(Temp, Pressure) shape with a tag registry: TagSample(Name, Timestamp, Value, Quality), TagDefinition, NoiseModel (4 variants — Sine, Drift, RandomWalk, Step), 50-tag seed configuration, per-tag emitter loops, and AppState.LatestTagValues. Per-tag metrics (samples.ingested/samples.coalesced with tag.name dimension) and a tags.active observable gauge.

Evidence row: slice-1-1-multi-tag-telemetry — 30-min capture under the MultiTag profile. 174 runs, all 50 tags emitting, telemetry rate 19.7 Hz aggregate.

Criterion 7 amended. The original "per-tag accuracy ±2% across all bands" was unachievable on Windows: tags configured below the default 15.6 ms timer-tick (≥ 64 Hz) cap regardless of code quality; medium-rate tags (10–50 Hz) drift under the load of 50 concurrent emitters. Amended to documented-not-gated, with the actual achievable bands recorded: ≤ 5 Hz tags within ±2%; 10 Hz hits ~9.2 Hz (−8%); 50 Hz hits ~32 Hz (−36%); ≥ 100 Hz tags cap near 64 Hz. The architectural goal — that all 50 tags are reachable through one bounded pipeline without dropping or destabilizing the workflow — was met.

SLICE-1.2 — Real Frame Payloads

Replaced Frame.PreviewPayload = null with real byte[] frame payloads: Frame.{Width, Height, BytesPerPixel}, profile fields for the same, HighFrameRate seed profile (2 048 × 1 024 × 1 at 33 ms = 30 fps × 2 MP), SimulatedCamera real allocation + gradient fill, MainViewModel.CurrentFrame WriteableBitmap binding, GC-pause-p95 / LOH-alloc-rate extraction helpers.

Evidence row: slice-1-2-real-frame-payloads — 10-min capture under HighFrameRate via the FlaUI rig. 8 154 frames ingested, 0 dropped, gen-2 = 2 713 (LOH pressure as designed), p95 pause = 11.76 ms, LOH-alloc-rate = 1.04 MB/s.

One follow-up filed. Original criterion 6 (frames.ingested ≥ 17 500) assumed continuous frame production; in reality SimulatedCamera only streams during active runs (Connected + Running), so the multi-cycle scan scenario reads ~30 fps × active time. Pipeline behavior is correct; only the criterion's continuous-streaming assumption was wrong.

SLICE-1.3 — Encoder-Rate Motion

Introduced a high-rate IEncoderStream (per-axis EncoderSamples in a bounded channel) alongside the existing 20 Hz IMotionController.PositionChanged UI feed. The encoder stream deliberately bypasses AppState — drained by EncoderStreamPipelineService which only increments per-axis metrics. New domain shapes (EncoderAxis, EncoderSample, EncoderSnapshot), SimulatedEncoderSource, WinMmTimePeriod P/Invoke wrapper for winmm!timeBeginPeriod(1), Simulator:Encoder configuration block, EncoderRate seed profile.

This is the load-bearing design preview for Phase 2's data-plane lift-out. The encoder pipeline is the first instance of "high-rate channel that doesn't touch the canonical store" in the codebase. Its row (encoder-rate-x/y = 656.6 Hz under EncoderRate; 200.0 Hz under MultiTag/Soak8h with 5 ms tick) and its no-AppState-write test (EncoderStreamPipelineServiceTests asserts RecordingAppStateStore.UpdateCount == 0) prove the architecture works.

Evidence row: slice-1-3-encoder-rate-motion — 10-min capture, encoder-rate-x/y = 656.6 Hz on both axes, frames.dropped = 0, runs.faulted = 0, gc-pause-p95 = 7.90 ms, working-set peak = 223.3 MB.

Criterion 7 amended. The original "1 kHz at receiver ± 2%" target was capped at ~657 Hz on Windows. Diagnosis: PeriodicTimer(1 ms).WaitForNextTickAsync is not real-time even with winmm.timeBeginPeriod(1) acquired; per-tick scheduling overhead plus per-tick producer work (~0.5 ms — two _motion._lock acquisitions, an ImmutableArray.Builder<EncoderSample>(2) allocation, channel TryWrite, two noise evaluations) drives the effective tick to ~1.52 ms. Same Windows-timer-resolution family of constraints as SLICE-1.1's amendment. Amended to documented-not-gated; encoder-cadence remediation (Stopwatch-busy-yield, timeSetEvent, CreateWaitableTimerEx(TIMER_HIGH_RESOLUTION)) filed as follow-up.

SLICE-1.6 — FlaUI-driven Measurement Capture

Built the UI-Automation-driven capture rig that replaced the original headless-scenario approach (SLICE-1.5, retired). InspectionPrototype.UiDriver (IUiDriver / FakeUiDriver), InspectionPrototype.AcceptanceTests (FlaUI 5.0.0, FlaUiDriver), 13 AutomationProperties.AutomationId attributes on MainWindow.xaml, DemoBaselineFlaUi + MultiTagSoakFlaUi scenarios, Capture-Measurements.ps1 orchestrator (build → launch → dotnet-counters → extract → optional table append). The rig drives the full XAML binding layer rather than calling commands directly — that tradeoff was the SLICE-1.5 retirement's whole point.

This slice doesn't have its own row in phase-1-measurements.md (its job is to enable the other slices' captures). Every Phase-1 row from slice-1-2 onward was captured through this rig.

SLICE-1.4 — Storm & Soak Profiles

Added storm-and-soak knobs to SimulatorProfile (DefectShowerEveryMs, DefectShowerDurationMs, AlarmBurstEveryMs, TelemetryDropoutChance, NetworkLatencyMeanMs, NetworkLatencyStddevMs, TimeCompressionFactor), a Simulator:FlakySdk configuration block, two new profiles (ChaosMonkey and Soak8h), three new services (DefectShowerService, AlarmBursterService, FlakySdkDecorator<IMachineConnection>).

Two evidence rows. slice-1-4-chaos-monkey (30 min, ChaosMonkey): 491 runs.started, 453 completes, 37 fault cycles with all four WorkflowService fault branches verified by log inspection (39 injected, 39 cleared, 37 recovered, 120 defect-shower transitions). slice-1-4-soak-8h (8 h, Soak8h): 5 109 runs, 0 faulted, working-set steady-state drift = −2.7 MB across 8 hours.

Criterion 12 amended. The original working-set growth = last − first ≤ 50 MB failed as measured (186.5 MB) on the 8-hour capture. Direct CSV inspection at 14 timepoints showed the entire delta is the process startup ramp: working-set rose from 47.5 MB to 230.9 MB in the first 29 seconds (WPF + 50 tag emitters + encoder pipeline + JIT) and held a stable sawtooth between 224 and 240 MB for the remaining 7.5 hours. Amended to working-set steady-state drift = avg(last 60 min) − avg(min 5-60) ≤ 50 MB, isolating the leak signal from one-time startup cost. Same documentation-not-implementation amendment pattern as the other two criteria.

SLICE-1.5 / SLICE-1.5.1 (superseded)

The original automated-capture rig drove IOperatorCommands directly, bypassing the XAML binding layer. The bypass was an explicit tradeoff in the SLICE-1.5 spec that turned out to matter more in practice — UI binding regressions weren't caught. SLICE-1.6 replaced it with FlaUI-driven UI Automation. About 1 700 lines of code + tests + tooling were retired. Two artifacts survived: _disposed Interlocked guards on SimulatedTagSource and SimulatedCamera (real DI double-disposal fixes), and the headerless-CSV recovery path in MeasurementExtraction.psm1.

Cross-slice performance picture

The four Phase 1 evidence rows tell a consistent story.

Frame pipeline holds rate

• slice-1-2 (HighFrameRate, 10 min, 2 MP × 30 fps): 8 154 frames ingested, 0 dropped.
• slice-1-3 (EncoderRate, 10 min, low frame load): 770 frames, 0 dropped.
• slice-1-4-chaos-monkey (30 min, 1 MP × 10 fps): 10 469 frames, 0 dropped.
• slice-1-4-soak-8h (8 h, 1 MP × 4 fps): 71 530 frames, 2 dropped (0.003%).

Two drops in 71 530 frames over 8 hours is a single transient scheduling spike absorbed by the bounded channel. The frame pipeline (FramePipelineService + bounded channel capacity = 3, DropOldest) is adequately sized for every load shape Phase 1 produced.

Tag pipeline holds rate

Aggregate telemetry.ingested rate matches the active profile's TelemetryIntervalMs exactly across all rows (19.80 Hz under 50 ms profiles; 9.96 Hz under 100 ms; 4.95 Hz under 200 ms — Soak8h's 8-hour run shows the 100 ms profile holding 9.96 Hz). Per-tag accuracy is bounded by the SLICE-1.1 criterion-7 amendment envelope. TelemetryDropoutChance = 0.01 under Soak8h yielded 12 coalesce events in 28 809 s — 1 every 40 minutes — consistent with intent.

Encoder pipeline holds rate at the 5 ms target

• 200.0 Hz on both axes under slice-1-4-chaos-monkey and slice-1-4-soak-8h (5 ms target).
• 656.6 Hz under slice-1-3-encoder-rate-motion (1 ms target, Windows-timer-resolution-capped).

The 8-hour run is the first long-duration evidence that the encoder data-plane bypass-AppState design holds under sustained load. No drift, no faults caused by the encoder side.

Workflow state machine survives chaos and sustained load

• ChaosMonkey: 92.3% completion rate (453/491) under aggressive fault injection (37 critical-fault cycles, 30-min run). All four fault branches (connect-fail, fault-during-home, fault-during-run, clear-and-recover) hit; the recovery loop completed cleanly.
• Soak8h: 100% completion rate (5 109/5 109) over 8 hours, zero faults.

The state machine + workflow services + retry loops survive both load shapes. The four-FlaUI-fix sequence that the ChaosMonkey capture surfaced (RecoverButton AutomationId, retry-Home loop, retry-Connect loop, hydration-service field-mapping fix) is documented in the 2026-05-01 session-log entry; those are scenario-rig hardening improvements, not application-side bugs.

GC pauses stable

GC pause p95 across all rows: 7.9 ms (slice-1-3) → 11.8 ms (slice-1-2) → 10.3 ms (slice-1-4-chaos-monkey) → 12.4 ms (slice-1-4-soak-8h). The same order of magnitude under every load. No long-tail pauses surfaced (otherwise frame drops would have appeared in the corresponding rows).

Working-set is flat

Time-series sampled directly from slice-1-4-soak-8h-2026-05-02.csv at 14 timepoints across 8 hours:

• t = 0: 47.5 MB
• t = 29 s: 230.9 MB (startup ramp complete)
• avg(min 5-30): 235.4 MB
• avg(h 4-5): 234.0 MB
• avg(h 7-8): 232.7 MB
• p99: 238.1 MB
• single transient max: 246.0 MB

Hours 7-8 mean is 2.7 MB lower than minutes 5-30 mean. There is no monotonic trend in either direction. The plateau is a stable sawtooth driven by Gen-2 GC cycles. The implementation does not leak under Soak8h conditions.

Three measurement-criterion amendments — same pattern

All three Phase-1 amendments followed the same shape: a criterion was specified before the actual platform behavior was understood, the capture revealed the specified target was unreachable for reasons unrelated to code quality, and the criterion was amended to a measurement that isolates the architectural property the slice was actually about.

Slice	Criterion	Original target	Amended target	Reason
1.1	7	per-tag rate ± 2% across all bands	documented-not-gated; achievable bands recorded	Default Windows 15.6 ms timer caps high-rate tags
1.3	7	encoder receiver rate 980-1020 Hz at 1 ms	documented-not-gated; achievable rate recorded	PeriodicTimer + winmm.timeBeginPeriod(1) ceiling at ~657 Hz
1.4	12	working-set growth (last − first) ≤ 50 MB	working-set steady-state drift (avg(last 60 min) − avg(min 5-60)) ≤ 50 MB	Original metric conflated process startup ramp with in-flight allocation

The 50 MB ceiling itself was unchanged in the criterion-12 amendment; only the measurement window. Same for the other two — the architectural intent was preserved; only the way of measuring it changed.

This pattern is worth noting because it could repeat in Phase 2. Specs that pre-specify target numbers should leave room for "the platform's behavior turned out to require a different measurement to express the same architectural intent" — and amendment-as-documentation is a legitimate outcome, not a failure to ship.

Decisions made and follow-ups filed

Filed during Phase 1, not blocking the exit gate, queued for future work:

• Encoder cadence remediation (SLICE-1.3 follow-up). Try Stopwatch-busy-yield, timeSetEvent multimedia callback, or CreateWaitableTimerEx(TIMER_HIGH_RESOLUTION) to push the encoder receiver rate closer to 1 kHz on Windows. Not load-bearing for any open slice.
• Continuous-streaming frame scenario (SLICE-1.2 follow-up). The current MultiTagSoakFlaUi scenario's multi-cycle structure means SimulatedCamera only streams during active scan motion. A dedicated single-continuous-run scenario (or amending criterion 6 to match the multi-cycle achievable count) would resolve the gap.
• SimulatorProfileHydrationService field-mapping regression coverage (SLICE-1.4 follow-up). Pass 1 added 7 new SimulatorProfile fields but the hydration service's Select() projection silently dropped them; runtime saw all-zero chaos knobs. The FlaUI rig was the first thing to exercise the wiring end-to-end and found it. Add a binding-roundtrip test asserting every SimulatorProfileOptions field reaches the runtime SimulatorProfile catalog entry.
• FlakySdk motion-side decorator (SLICE-1.4 deferred non-scope). The decorator currently wraps IMachineConnection only; wrapping IMotionController would surface motion-side fault paths in WorkflowService. Not needed for the criterion-11 evidence; deferred until/unless Phase 2 motivates it.
• Encoder-stream UI plot (SLICE-1.3 deferred non-scope). The encoder data is captured to a channel and a metric counter; no UI surfaces the high-rate stream. Phase 2 or 3 work.
• Per-profile FlakySdk knobs (SLICE-1.4 deferred non-scope). The current block is a single global config; per-profile granularity would let the operator turn the decorator on/off as part of profile selection rather than a manual appsettings.json flip.

Resolved during Phase 1, no follow-up needed:

• The FlakySdkDecorator timeout-branch fall-through fix (commit 018bf29, pre-Soak8h) — spec said "fall through to inner if not cancelled," initial Pass-2 implementation unconditionally threw OCE. Fixed before the Soak8h capture; new regression test added.
• The four FlaUI capture-rig fixes (bf32566, 0f1596a, 5462d42, 2108272) — RecoverButton AutomationId, retry-Home loop, retry-Connect loop, hydration-service field mapping. All scenario-rig hardening that the ChaosMonkey capture surfaced.
• The 63-minute system-sleep mid-capture incident (TASK-1.1 Pass 3) — runbook caveat added; sleep-disable + hibernate-off discipline now reaffirmed in §4.5 / §4.6 / §3a.

Phase 2 — what's next

Phase 2 was originally specified as four slices (2.1 store-slicing, 2.2 immutable collections, 2.3 data-plane lift-out, 2.4 per-slice observables) with exit gates phrased as deltas against an unmeasured AppStateStore.Update baseline. The roadmap §3 was explicit: "Those numbers become the measured justification for Phase 2. If the app survives this run beautifully, Phase 2 is deferred. If not, we know exactly which slice of the store to attack first."

Phase 1 evidence shows the app does survive beautifully. But AppStateStore.Update allocation share and lock-wait time are still unmeasured — no Phase 1 row instrumented the store side. Phase 2 opens with SLICE-2.0 (Store Allocation & Contention Profiling), a measurement-first foundation slice that instruments Update with [CallerMemberName] + [CallerFilePath] (compile-resolved at every callsite, no callsite changes), Stopwatch for lock-wait, GC.GetAllocatedBytesForCurrentThread() for alloc-delta, and three new AppMetrics members. One 30-minute MultiTag capture; row block under a new phase-2-measurements.md; runbook §5.1 (Phase-2 captures).

The row's Notes section produces an explicit Phase 2 prioritization recommendation following a mechanical decision rubric:

• store.update alloc share < 10% → defer SLICE-2.1 and SLICE-2.2.
• store.update lock-wait p95 < 100 µs → defer SLICE-2.4.
• Top caller > 50% from TagStreamPipelineService → SLICE-2.3 (tags) opens (mirrors the SLICE-1.3 encoder bypass).
• Top caller > 50% from FramePipelineService → SLICE-2.3 (frames) opens.
• Top caller > 50% from workflow code → no Phase 2 slice fits cleanly; recommend Phase 2 deferral.

Deferral is a first-class outcome. If the captured numbers show the architecture is already healthy at Phase-1 load levels, the right Phase-2 plan is "do nothing, open Phase 3 evaluation." That outcome saves weeks of work the roadmap had budgeted but never gated. The principle from the roadmap §0 ("Premature refactoring is the most expensive mistake available here") is best honored by letting the data decide.

Operational lessons

Things worth carrying into Phase 2:

• Sleep-disable discipline is non-negotiable for any capture longer than 10 minutes. TASK-1.1's 63-minute mid-capture sleep event diluted the row's per-tag rates; runbook §3a, §4.5, §4.6 all reaffirm powercfg /change standby-timeout-ac 0. The Soak8h capture also requires powercfg /hibernate off.
• Pre-flight commit pattern. SLICE-1.4's Pass 3 close needed two pre-Soak8h fixes (Enabled = false for criterion-16 reproducibility, FlakySdk timeout fall-through). Commit A (small, reversible) → 8-hour capture → Commit B (docs + CSVs) was the right structure. Don't attempt to combine.
• Direct CSV inspection beats trusting the helper output. The criterion-12 measurement gap was visible in the time-series shape but not in the row's headline working-set growth (MB) = 186.5 value. A 14-timepoint sample of the CSV (2 minutes of bash) was all it took to recognize the startup-ramp pattern. For any slice whose acceptance criterion depends on a single derived number, look at the underlying time series.
• The FlaUI rig requires deliberate AutomationIds + retry loops. The four FlaUI fixes during the ChaosMonkey capture were caused by missing AutomationId attributes (RecoverButton) and single-attempt commands (Connect, Home) that didn't tolerate fault-injected failures. Any future button or workflow path that the capture exercises needs both.
• [CallerMemberName] + [CallerFilePath] is the right compromise for caller attribution when you want zero callsite churn (SLICE-2.0 design choice). Stack-walk is more accurate but slower; explicit string parameters at every callsite pollute the diff. Compile-resolved auto-attribution lets the existing code stay untouched and the new instrumentation gets the data it needs.

Closing

Phase 0 met its exit gate on 2026-04-23 with the demo baseline (row 0, commit 7ecef05). Phase 1 ran for 10 days against that baseline. Five slices completed; two retired mid-phase; three measurement-methodology amendments documented; no architectural defects surfaced. The simulator now produces production-shaped load and the application handles it.

Phase 2 begins with measurement.