SLICE-1.2 Design Notes — Real Frame Payloads
- Slice: SLICE-1.2
- Implementation status: Completed (2026-04-27)
- Audience: anyone modifying the camera simulator, the frame channel, the WPF preview rendering, or the GC/LOH allocation profile of the prototype
This doc explains how the frame pipeline actually works in code — why the camera allocates fresh byte[] per frame instead of pooling, why the channel capacity is 3 (not 1 like the tag and encoder streams), how the WriteableBitmap rendering avoids per-frame allocations on the UI side, and how defect detection ended up in the consumer rather than the producer. Read this if you're touching the frame producer, the LOH pressure profile, or the WPF preview surface.
If you want a scenario-first explanation of when frames run in the app, how to configure profiles, and what C#/.NET techniques are being used, read the companion article: SLICE-1.2 frame pipeline deep dive.
1. Quick reference
Key types:
| Type | Project | Role |
|---|---|---|
Frame (record) | Domain | Width, Height, BytesPerPixel, byte[]? PreviewPayload |
IFrameSource | Application | The bounded read-only contract the consumer holds |
ICameraController | Application | Start/stop streaming |
SimulatedCamera | Infrastructure | The producer (implements both interfaces) + IDisposable |
FramePipelineService | Application | The consumer; BackgroundService |
IDefectShowerSchedule | Application | Read-only schedule consulted by defect detection (SLICE-1.4) |
MainViewModel.UpdateCurrentFrame | Presentation | WPF WriteableBitmap updater |
Key files:
src/InspectionPrototype.Domain/Contracts/Frame.cs
src/InspectionPrototype.Application/Abstractions/IFrameSource.cs
src/InspectionPrototype.Application/Abstractions/ICameraController.cs
src/InspectionPrototype.Application/Services/FramePipelineService.cs
src/InspectionPrototype.Infrastructure/Simulator/SimulatedCamera.cs
src/InspectionPrototype.Presentation/ViewModels/MainViewModel.cs // CurrentFrame + UpdateCurrentFrame
src/InspectionPrototype.App/MainWindow.xaml // Image control bound to CurrentFrameProfile fields driving the producer:
| Field | Source | Purpose |
|---|---|---|
SimulatorProfile.FrameWidth | appsettings.json per profile | Pixel width of generated frames |
SimulatorProfile.FrameHeight | same | Pixel height |
SimulatorProfile.BytesPerPixel | same — valid: 1 / 2 / 3 / 4 | Byte depth (Gray8 / Gray16 / Bgr24 / Bgra32) |
SimulatorProfile.FrameIntervalMs | same | Producer tick period |
SimulatorProfile.DefectProbabilityPerFrame | same | Per-frame defect generation odds |
The HighFrameRate profile sets these to 2 048 × 1 024 × 1 at 33 ms = ~30 fps × 2 MP, which intentionally drives heavy LOH allocation pressure (each frame is 2 MB, well over the 85 KB LOH threshold).
Key tests (all in tests/InspectionPrototype.Tests/):
| Test | Asserts |
|---|---|
SimulatedCameraTests.ProduceFrames_AtConfiguredRate | Streaming for N seconds yields ~N × (1000/FrameIntervalMs) |
SimulatorProfilesValidatorTests | Rejects FrameWidth < 1, BytesPerPixel ∉ {1,2,3,4}, payload overflow |
FramePipelineServiceTests | Drain → LatestFrame updated; defect detected → DefectCount++ |
FramePipelineServiceShowerTests | When IDefectShowerSchedule.IsShowerActive, every frame produces a defect |
SimulatorProfilesOptionsBindingTests | All 9 profile fields bind correctly (silent-ignore guard from SLICE-1.6) |
Key metrics:
| Counter | Dimension | Emitted by |
|---|---|---|
frames.ingested | none | FramePipelineService |
frames.dropped | none | FramePipelineService (mirrors IFrameSource.DroppedCount delta) |
2. Class shape
┌──────────────────────────────────┐
│ appsettings.json │
│ "Simulator:Profiles": │
│ [Normal / Demo / HighDefect /│
│ MultiTag / HighFrameRate / │
│ EncoderRate / ChaosMonkey / │
│ Soak8h] │
│ each: FrameWidth, FrameHeight, │
│ BytesPerPixel, FrameInterval │
└──────────────────┬───────────────┘
│ profile-active
│ via ISimulatorProfileProvider
▼
┌────────────────────────────────────────────────────┐
│ SimulatedCamera │
│ (Infrastructure.Simulator) │
│ │
│ Implements: ICameraController + IFrameSource │
│ + IDisposable │
│ │
│ ┌───────────────────────────────────────────┐ │
│ │ ProduceFramesAsync (single Task) │ │
│ │ while !ct: │ │
│ │ Task.Delay(profile.FrameIntervalMs) │ │
│ │ payload = new byte[w*h*bpp] ◀─── LOH │ │
│ │ fill gradient (x+y+seq) % 256 │ │
│ │ frame = new Frame(...) │ │
│ │ if channel.Count >= 3 │ │
│ │ Interlocked.Increment(_droppedCount) │ │
│ │ channel.Writer.TryWrite(frame) │ │
│ └───────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────┐ │
│ │ Channel<Frame> │ │
│ │ capacity = 3, DropOldest │ │
│ │ single writer + single reader │ │
│ └─────────────────────────────────┘ │
│ │
│ ICameraController: │
│ StartStreamingAsync — set IsStreaming=true, │
│ start Task.Run(ProduceFramesAsync) │
│ StopStreamingAsync — cancel _streamingCts │
└────────────────────────────────┬───────────────────┘
│ IFrameSource.Reader
▼
┌────────────────────────────────────────────────────────┐
│ FramePipelineService │
│ (Application.Services) │
│ : BackgroundService │
│ │
│ ExecuteAsync drain: │
│ await foreach (frame in source.Reader.ReadAllAsync) │
│ newDrops = source.DroppedCount - lastKnownDrops │
│ if newDrops > 0: │
│ frames.dropped.Add(newDrops) │
│ Update(s with │
│ LatestFrame = frame, │
│ PipelineCounters w/ FramesDropped = total) │
│ + Warning diagnostics entry │
│ else: │
│ Update(s with LatestFrame = frame) │
│ frames.ingested.Add(1) │
│ ProcessDefectsForFrame(frame): │
│ if ActiveRun is null: return │
│ if !showerSchedule.IsShowerActive AND │
│ rng >= profile.DefectProbabilityPerFrame: │
│ return │
│ severity = roll-distribution (Critical/Major/Minor) │
│ Update(s with │
│ ActiveRun w/ DefectCount++, │
│ per-severity count++) │
└─────────────────────────┬──────────────────────────────┘
│ writes through AppState
▼
┌────────────────────────────────────────────────────────┐
│ AppStateStore │
│ AppState.LatestFrame │
│ AppState.PipelineCounters.FramesDropped │
│ AppState.ActiveRun.DefectCount / Critical/Major/Minor │
└─────────────────────────┬──────────────────────────────┘
│ StateChanged event
▼
┌────────────────────────────────────────────────────────┐
│ MainViewModel.OnStateChanged │
│ if state.LatestFrame is { PreviewPayload: not null }: │
│ _dispatcher.InvokeAsync(UpdateCurrentFrame(frame)) │
│ │
│ UpdateCurrentFrame(frame): │
│ fmt = bpp switch { 1→Gray8, 2→Gray16, 3→Bgr24, _→Bgra32 } │
│ if _currentFrame == null OR dimensions/format changed: │
│ CurrentFrame = new WriteableBitmap(w, h, 96, 96, …) │
│ _currentFrame.Lock() │
│ _currentFrame.WritePixels(rect, payload, stride, 0) │
│ _currentFrame.Unlock() │
└────────────────────────────────────────────────────────┘
│ data-bound
▼
┌────────────────────────────────────────────────────────┐
│ MainWindow.xaml │
│ <Image Source="{Binding CurrentFrame}" /> │
└────────────────────────────────────────────────────────┘3. Lifecycle
SimulatedCamera is constructed eagerly at host startup (DI singleton, registered under both ICameraController and IFrameSource). It does not stream by default — IsStreaming = false until StartStreamingAsync is called explicitly by WorkflowService.RunLoopAsync at the start of each run.
construct workflow.StartRun
│ │ workflow run ends
▼ ▼ │
┌─────────┐ StartStreamingAsync ┌─────────────┐ StopStreamingAsync ▼
│ Idle │────────────────────────▶ │ Streaming │ ──────────────────▶ ┌─────────┐
│ │ │ │ │ Idle │
│ channel │ │ Task.Run │ _streamingCts. │ │
│ ready; │ │ Produce- │ Cancel(); │ channel │
│ no │ │ FramesAsync │ .Dispose(); │ may have│
│ producer│ │ ticks at │ │ residual│
│ task │ │ profile. │ │ frames │
└─────────┘ │ FrameInter- │ └────┬────┘
│ valMs │ │
└──────┬──────┘ │
│ │
│ ◀─────────────────────────────────┘
│ next StartStreamingAsync
│ (no channel rebuild needed)The frame channel is not rebuilt across start/stop cycles — frames left over from a completed run drain normally during the next quiet period, and FramePipelineService.ProcessDefectsForFrame checks state.ActiveRun is null and discards stale frames that arrive after a run ends. This avoids a teardown race that earlier slices had to design around.
4. Runtime flow
Headline: producer fills a 2 MB byte[], channel ferries it across, consumer renders it via WriteableBitmap.WritePixels on the UI thread.
Producer task Channel Pipeline service Store Dispatcher WPF Image
───────────── ─────── ──────────────── ───── ────────── ─────────
│ │
┌────┴────┐ │
│ Task. │ │
│ Delay │ │
│ (Frame- │ │
│ Inter- │ │
│ valMs) │ │
└────┬────┘ │
│ payload = new byte[w*h*bpp] ◀── 2 MB allocation on LOH │
│ for HighFrameRate (2048×1024×1) │
│ │
│ fill gradient: each pixel = (x+y+seq) & 0xFF │
│ │
│ frame = new Frame(FrameId, seq, now, w, h, bpp, payload) │
│ │
│ if channel.Count >= 3: │
│ Interlocked.Increment(_droppedCount) │
│ │
│ channel.Writer.TryWrite(frame) ─────▶ │ │
│ │ ── ReadAllAsync ──▶ │ │
│ │ │ newDrops = … │
│ │ │ frames.ingested.Add(1) │
│ │ │ Update(s with LatestFrame)│
│ │ │ ────────────────▶ │ │
│ │ │ │ │
│ │ │ ProcessDefects │ │
│ │ │ ForFrame(frame): │ │
│ │ │ if active run │ │
│ │ │ AND (shower OR │ │
│ │ │ rng < prob): │ │
│ │ │ Update(s w/ │ │
│ │ │ DefectCount++)│ │
│ │ │ │ │
│ │ │ StateChanged?.Invoke() │
│ │ │ ────────────────────────▶│
│ │ │ Dispatcher. │
│ │ │ InvokeAsync ────│──▶
│ │ │ UpdateCurrentFrame
│ │ │ │ if dimensions
│ │ │ │ changed:
│ │ │ │ CurrentFrame =
│ │ │ │ new WriteableBitmap
│ │ │ │
│ │ │ │ _currentFrame.Lock()
│ │ │ │ .WritePixels(rect,
│ │ │ │ payload, stride, 0)
│ │ │ │ .Unlock()
│ │ │ │ ──▶ XAML repaintThe dispatcher hop (_dispatcher.InvokeAsync) is required because StateChanged fires on the producer's thread (AppStateStore.Update → _store.Update.StateChanged?.Invoke(next) is single-threaded but called from the consumer's BackgroundService thread, not the UI thread). WriteableBitmap.Lock()/WritePixels()/Unlock() must run on the UI thread. The InvokeAsync (vs Invoke) is fire-and-forget so the consumer doesn't wait for the UI repaint.
5. Decisions made during implementation
(a) Fresh byte[] allocation per frame, no pool. The producer's new byte[width * height * bpp] is the slice's whole point — exposing real LOH pressure. Pooling was explicitly considered and rejected. Phase 2 may revisit if the soak data ever shows it matters; SLICE-1.4's slice-1-4-soak-8h row showed Gen-2 GC at 1 398/hr (well below the 4× ceiling vs slice-1-2's 16 280/hr at 30 fps × 2 MP) — so the GC is keeping up with the allocation rate. Pooling is deferred until the production camera SDK is integrated and we understand its allocation pattern.
(b) Channel capacity = 3, not 1. Tag and encoder streams use capacity = 1; frames use 3. The reason is the consumer's variable cost: defect detection includes a Random.Shared.NextDouble() plus a possible _store.Update call which takes the AppState lock. Under high-defect-probability profiles (HighDefect at 0.6/frame; ChaosMonkey defect-shower windows at 1.0) the consumer can briefly spike its per-frame cost. Capacity = 3 absorbs the spike without dropping. The Soak8h capture saw frames.dropped = 2 over 8 hours — proves the buffer is correctly sized.
(c) Defect detection is in the consumer, not the producer. The simulator could synthesize defects directly into the Frame record. It doesn't, because defect detection is a post-camera concern in real systems (a vision algorithm runs on the captured frame). Putting the random-roll in FramePipelineService.ProcessDefectsForFrame is faithful to that future shape — when a real camera replaces the simulator, the pipeline service still owns the detection logic.
(d) ProcessDefectsForFrame consults IDefectShowerSchedule.IsShowerActive. This is the SLICE-1.4 connection: the chaos profile's defect-shower service flips a boolean every N seconds, and during the window the per-frame probability check is short-circuited to "always defect." The schedule is injected through DI and falls through to a no-op (IsShowerActive = false) under non-chaos profiles. Adding the abstraction in SLICE-1.2 cost nothing and let SLICE-1.4 land cleanly without re-touching FramePipelineService's core logic.
(e) WriteableBitmap is reused if dimensions are stable. MainViewModel.UpdateCurrentFrame only allocates a new WriteableBitmap if width/height/format changed since the last frame. Under steady state (one profile, no resizing) the same bitmap is updated in place via WritePixels. This keeps the WPF compositor happy — it doesn't have to invalidate render-tree nodes — and avoids per-frame GC churn on the UI side. The producer-side allocation is the one that matters for LOH measurement; the consumer side is allocation-free in the hot path.
(f) Pixel-format selection from BytesPerPixel. bpp switch { 1 → Gray8, 2 → Gray16, 3 → Bgr24, _ → Bgra32 }. The _ fall-through uses Bgra32 for bpp == 4; bpp ∉ {1,2,3,4} is rejected at config-validation time so the fallthrough is theoretically unreachable. The default-case still covers it for safety. Don't change this without updating SimulatorProfilesValidator — the pixel-format/bpp pairing is what makes WPF render correctly.
(g) Producer holds no reference to the rendered bitmap. The Frame.PreviewPayload is owned by the producer until written to the channel; once the consumer has it, the producer's reference is dropped. The bitmap on the UI side copies the bytes via WritePixels (synchronously, while holding Lock()). After Unlock(), the consumer-side payload reference goes out of scope and is GC-eligible. There is no shared mutable state between the producer and the UI thread.
6. Invariants and traps
LOH allocation is the slice's purpose, not its cost. Frames at 2 MP × 1 byte = 2 MB are well over the 85 KB LOH threshold. Each frame goes directly to the LOH and stays there until a Gen-2 collection. At 30 fps that's 60 MB/sec of LOH churn — slice-1-2's 1.04 MB/s LOH-alloc-rate is the time-averaged value over the whole capture (including idle periods). Don't try to "fix" this by pooling unless measurement shows it matters. Phase 2 is the right place; SLICE-1.2 deliberately surfaces the pressure rather than hiding it.
WriteableBitmap.Lock() blocks the UI thread. Lock() waits for any in-flight render-tree paint to finish, then claims exclusive access. For 2 MP frames the lock-paint-unlock cycle takes ~1-2 ms on a modern host. This is why _dispatcher.InvokeAsync (fire-and-forget) is used instead of Invoke — the consumer doesn't block on the UI repaint. If you change to Invoke, the consumer will queue behind the UI rendering and frame drops will rise sharply under load.
WriteableBitmap.WritePixels requires bpp × width as stride. No padding. The Frame record explicitly documents this: "row-major, packed, with no stride padding: exactly Width × Height × BytesPerPixel bytes." If a future producer adds stride alignment (some real cameras do), the consumer's WritePixels(..., w * bpp, 0) call will misinterpret the buffer. Don't add stride padding without updating both sides.
SimulatedCamera is registered under three service types. DI registers the singleton as SimulatedCamera, ICameraController, and IFrameSource. The DI container's disposal protocol calls Dispose() once per registered type. The double-dispose protection (Interlocked.Exchange(ref _disposed, 1) != 0 early-return) is load-bearing — same pattern as SimulatedTagSource (TASK-1.5.1 fix). Don't remove.
SimulatedCamera reads profile fields at each producer tick, not at StartStreamingAsync. Switching profiles mid-run changes FrameWidth, FrameHeight, BytesPerPixel, and FrameIntervalMs on the next tick. This is consistent with the rest of the simulator (the convention is "profile changes apply on the next operation, never to in-flight work"). Watch for buffer-size mismatches if a real driver replaces the simulator and doesn't tolerate dimension changes mid-stream — the WPF renderer handles it by reallocating WriteableBitmap, but a real SDK might not.
FramePipelineService does NOT record frames.ingested events with a frame.id dimension. The metric is dimensionless (counter only). If you ever need per-frame correlation across captures (e.g., "did frame N produce a defect?"), you need a different observability surface — likely the OpenTelemetry Activity / span path, not AppMetrics counters.
Defect detection runs on the consumer thread, not the UI thread. A heavy detection algorithm here would directly cap the consumer's drain rate. The current logic is two Random.Shared.NextDouble() calls + one _store.Update — measured in microseconds. If a real vision algorithm replaces this, expect to either pool work onto a separate detector pool, or accept that detection becomes the rate-limiting stage.
7. Test surface
Covered by unit tests:
SimulatedCameraproduces frames at approximately the configured rate;DroppedCountincrements correctly under simulated backpressure.SimulatorProfilesValidatorrejects each invalid combination of FrameWidth / FrameHeight / BytesPerPixel; payload-size overflow check (width × height × bpp > int.MaxValue) prevents allocation crashes.SimulatorProfilesOptionsBindingTests(added in SLICE-1.6 Pass 1) verify all 9 profile fields bind fromappsettings.jsontoSimulatorProfile. Catches the silent-ignore bug pattern.FramePipelineServiceTestsdrains a fake source; defect detection in the active-run window updatesActiveRun.DefectCountcorrectly.FramePipelineServiceShowerTests(added in SLICE-1.4 Pass 2) verify thatIDefectShowerSchedule.IsShowerActive == trueproduces a defect on every frame regardless ofDefectProbabilityPerFrame.
Covered by capture (slice-1-2-real-frame-payloads row):
- 8 154 frames ingested over 608 s under HighFrameRate; 0 dropped.
- gen-2 GC count = 2 713 (LOH pressure as designed).
- gc-pause-p95 = 11.76 ms (proves the channel absorbs the 2 MP frames without pause-induced drops).
- LOH-alloc-rate avg = 1.04 MB/s (time-averaged; during active runs it tracks closer to 60 MB/s).
Not covered (intentional gaps):
- Continuous-streaming behavior. The current capture scenario (
MultiTagSoakFlaUi) cycles through Connect → Home → Run → ... → Disconnect repeatedly; the camera streams only during the Run phase, so the capturedframes.ingested = 8 154is far below the criterion-6 expectation of≥ 17 500(= 30 fps × 600 s × 0.97). This is a scenario/criterion mismatch documented in the row's notes — pipeline is correct, scenario assumption was wrong. Filed as follow-up: amend criterion 6 or add a dedicated continuous-streaming scenario. - WriteableBitmap rendering correctness. No automated test verifies the rendered pixels match the gradient formula. Visual smoke at app startup is the surface that catches this; if a future change breaks the rendering math, the operator sees a black or scrambled preview.
- GC pressure regression detection. The
gc-pause-p95andgen-2 GC countnumbers are captured but not gated. A future change that doubles per-frame allocation would show up in the next capture as a higher Gen-2 rate, but no automated test fails. This is consistent with the Phase 1 "measurement first" discipline; tightening into a CI-gated regression check is a Phase 2 concern.
Notably absent test: there is no test for "what happens when the consumer crashes mid-WritePixels?" The WriteableBitmap.Lock() / Unlock() pair must be balanced — a crash between them leaves the bitmap permanently locked and subsequent writes deadlock. The current code doesn't use try/finally because the entire region is synchronous and exception-free in practice. If a future change adds an exception-throwing call between Lock and Unlock, wrap it in try/finally.