SLICE-1.2 Frame Pipeline Deep Dive - From Camera Stream to WPF Preview

This is the companion article for SLICE-1.2 Design Notes. The design note is the compact reference. This file is the slower walkthrough: when frames are produced in the app, how to configure the simulated camera, how a frame moves through the pipeline, and which C#/.NET techniques make the design work.

The core idea is simple:

The camera produces heavy binary payloads only while a run is active. The app keeps that stream bounded, observable, and safe for the WPF UI thread.

That gives us a useful production-shaped pattern:

• device-style producers should not write directly into UI state
• large payloads need an explicit memory story
• live previews should prefer recent frames over stale backlogs
• UI rendering must cross the WPF dispatcher boundary
• shutdown needs both run-level stop and app-level cleanup

The User Scenario

Imagine the operator has loaded a recipe, homed the machine, and starts an inspection run. At that moment the app should start behaving like a live vision workstation:

• the camera begins streaming preview frames
• the live preview image updates during the run
• the frame sequence text advances
• fake defect detection runs against consumed frames
• defect counts update while the run is active
• dropped frames are counted if the consumer falls behind

Unlike SLICE-1.1 telemetry, frames are not app-lifetime streaming data. The camera source exists for the lifetime of the app, but frame production is run-lifetime work.

The practical flow is:

App starts
  -> DI constructs SimulatedCamera
  -> FramePipelineService starts and waits on IFrameSource.Reader
  -> camera is idle; no frame task is running

Operator starts a run
  -> WorkflowService.RunLoopAsync calls _camera.StartStreamingAsync(...)
  -> SimulatedCamera starts one producer task
  -> frames begin flowing through the channel
  -> FramePipelineService writes LatestFrame into AppState
  -> MainViewModel renders the preview on the UI thread

Run finishes, stops, aborts, or faults
  -> WorkflowService finally block calls _camera.StopStreamingAsync()
  -> producer cancellation is requested
  -> the pipeline may drain any residual frames
  -> defect detection ignores frames when no ActiveRun exists

This explains an important UI behavior: the last preview image may remain visible after a run ends. That is not a new stream. It is just the last CurrentFrame rendered by the view model.

How To Configure It

Frame behavior comes from Simulator:Profiles in src/InspectionPrototype.App/appsettings.json.

The important profile fields are:

• FrameWidth: generated frame width in pixels
• FrameHeight: generated frame height in pixels
• BytesPerPixel: 1, 2, 3, or 4
• FrameIntervalMs: delay between generated frames
• DefectProbabilityPerFrame: probability that a consumed frame produces a fake defect

For example, HighFrameRate is intentionally heavy:

{
  "Name": "HighFrameRate",
  "MotionSpeedUnitsPerSecond": 50.0,
  "TelemetryIntervalMs": 100,
  "FrameIntervalMs": 33,
  "FrameWidth": 2048,
  "FrameHeight": 1024,
  "BytesPerPixel": 1,
  "DefectProbabilityPerFrame": 0.05,
  "ConnectionFailureProbability": 0.05,
  "EncoderIntervalMs": 5
}

That means:

2048 * 1024 * 1 = 2,097,152 bytes per frame
33 ms interval ~= 30 frames per second
roughly 60 MB/sec of active-run payload allocation

The validator rejects unsafe profile values at startup:

• FrameWidth < 1
• FrameHeight < 1
• BytesPerPixel outside 1, 2, 3, 4
• FrameWidth * FrameHeight * BytesPerPixel > int.MaxValue

That last check matters because new byte[...] takes an int length. The validator uses long arithmetic before allocation time so a bad profile fails cleanly instead of crashing during streaming.

Profile selection is normally an idle-time operation. SimulatorProfileService rejects profile changes while workflow work is active, so choose the frame profile before starting the run. SimulatedCamera reads the current profile on each producer tick, but the normal UI flow prevents mid-run profile changes.

A Frame's Journey Through The App

Let us follow one frame from the profile configuration to the pixels in the WPF image.

First, profile configuration is loaded into SimulatorProfileOptions, then hydrated into an application SimulatorProfile record. SimulatedCamera reads the active profile through ISimulatorProfileProvider.

When a run begins, WorkflowService.RunLoopAsync calls:

await _camera.StartStreamingAsync(runToken);

SimulatedCamera sets IsStreaming = true, creates a private CancellationTokenSource, and starts one producer task:

Task.Run(() => ProduceFramesAsync(cts.Token))

The producer loop does this repeatedly:

1. Wait for CurrentProfile.FrameIntervalMs.
2. Read FrameWidth, FrameHeight, and BytesPerPixel.
3. Increment the frame sequence.
4. Allocate a fresh byte[] payload.
5. Fill the payload with a deterministic gradient.
6. Wrap it in a Frame record.
7. Write the frame to a bounded channel.

The payload layout is intentionally simple:

row-major
packed
no stride padding
payload length = Width * Height * BytesPerPixel

For pixel (x, y), the simulator writes:

value = (x + y + sequence) & 0xFF

For multi-byte pixels, it repeats the same value across each byte. This is not a real camera image. It is a deterministic, visual, allocation-heavy preview payload.

The frame crosses the producer-consumer boundary through:

Channel<Frame>
  capacity = 3
  FullMode = DropOldest
  SingleWriter = true
  SingleReader = true

FramePipelineService is the one consumer. It waits with:

await foreach (var frame in _source.Reader.ReadAllAsync(stoppingToken))

For each frame, it:

• checks whether the producer reported new drops
• updates AppState.LatestFrame
• increments frames.ingested
• runs fake defect detection if AppState.ActiveRun exists
• updates defect counters if a defect is produced

Then AppStateStore.StateChanged fires. MainViewModel receives the state on a background thread, dispatches ordinary projection work to the UI thread, and separately posts the frame render:

if (state.LatestFrame is { PreviewPayload: not null } frame)
    _dispatcher.InvokeAsync(() => UpdateCurrentFrame(frame));

UpdateCurrentFrame chooses a WPF pixel format from BytesPerPixel:

1 -> Gray8
2 -> Gray16
3 -> Bgr24
4 -> Bgra32

If the dimensions or format changed, the view model creates a new WriteableBitmap. Otherwise it reuses the existing bitmap and copies the payload into it:

_currentFrame.Lock();
_currentFrame.WritePixels(new Int32Rect(0, 0, w, h), payload, w * bpp, 0);
_currentFrame.Unlock();

The XAML Image is bound to CurrentFrame, so WPF repaints after the bitmap changes.

The full path is:

Simulator:Profiles
  -> SimulatorProfile
  -> WorkflowService starts camera streaming
  -> SimulatedCamera producer task
  -> byte[] PreviewPayload
  -> Frame record
  -> bounded Channel<Frame>
  -> FramePipelineService
  -> AppState.LatestFrame
  -> MainViewModel.UpdateCurrentFrame
  -> WriteableBitmap.WritePixels
  -> WPF Image

Why Frames Use Capacity 3

Telemetry snapshots use channel capacity 1 because only the freshest telemetry map matters. Frames are slightly different.

A frame consumer has variable work:

• it writes LatestFrame into AppState
• it increments metrics
• it checks drop counters
• it may run fake defect detection
• it may perform a second AppState update for defect counts
• it triggers UI preview rendering through StateChanged

Most frames are cheap. Some frames cost a little more, especially under high-defect or defect-shower profiles. Capacity 3 gives the consumer a short cushion.

Think of it as a small sliding window:

producer writes frames: 1, 2, 3
channel holds:          1, 2, 3

producer writes frame:  4 while consumer is late
DropOldest removes:     1
channel holds:          2, 3, 4

The policy is still "prefer recent frames." It just allows a tiny amount of burst absorption before dropping. That is suitable for a live preview. The operator wants a recent image, not an old backlog.

This is why a channel is a strong fit here:

• async waiting without polling
• bounded memory
• built-in drop policy
• clear reader/writer ownership
• clean completion during disposal
• direct support for ReadAllAsync(stoppingToken)

A plain ConcurrentQueue<Frame> would be thread-safe, but it would not decide capacity, drop behavior, completion, or cancellation for us. We would have to rebuild those pieces manually.

.NET Technique: Large byte[] Payloads And LOH Pressure

The producer allocates a fresh byte[] per frame:

var payload = new byte[width * height * bpp];

That is deliberate. SLICE-1.2 exists to make frame payloads real enough that memory and GC behavior are visible.

In .NET, large arrays go to the Large Object Heap. The usual threshold is around 85 KB. A 640 x 480 x 1 frame is already about 300 KB. A HighFrameRate frame is about 2 MB. So frame payloads are not tiny Gen-0 objects; they create meaningful LOH churn.

The implementation intentionally does not pool payloads yet. Pooling would hide the allocation pressure that this slice was meant to measure. That does not mean pooling is bad. It means it should be introduced only after measurement says the current cost is a problem and after the real camera SDK shape is known.

The ownership story is simple:

producer allocates byte[]
producer writes Frame to channel
consumer copies bytes into WriteableBitmap
no one mutates the byte[] after publish
payload becomes GC-eligible after consumers release it

There is no shared mutable pixel buffer between the producer and WPF.

.NET Technique: Frame Record As A Payload Contract

Frame is a domain record. It carries both metadata and optional preview bytes:

• FrameId
• Sequence
• CapturedAtUtc
• CaptureX
• CaptureY
• Width
• Height
• BytesPerPixel
• PreviewPayload

The important contract is not only the properties. It is the buffer layout:

PreviewPayload length = Width * Height * BytesPerPixel
stride used by the UI = Width * BytesPerPixel
no padding bytes between rows

That is why WritePixels can use:

payload, w * bpp, 0

If a future real camera produces stride-padded rows, this contract must change. Many camera SDKs align rows to 4, 8, or more bytes. If that happens, add an explicit Stride field instead of guessing from width and bytes-per-pixel.

.NET Technique: WriteableBitmap For UI-Side Reuse

The producer allocates a fresh payload per frame, but the UI does not allocate a new image object per frame.

MainViewModel keeps _currentFrame and only creates a new WriteableBitmap when the frame dimensions or pixel format change:

same width, height, format
  -> reuse existing WriteableBitmap

new width, height, or format
  -> allocate a new WriteableBitmap

That is a good WPF pattern. The Image control can keep the same source object while the pixels inside that source are updated.

Lock, WritePixels, and Unlock must run on the UI thread. WPF objects have thread affinity, and touching them from a background service would eventually produce cross-thread failures.

One production caveat: Unlock() should happen if code after Lock() can throw. The current block is small and synchronous. If future code adds anything exception-prone between Lock() and Unlock(), wrap it:

_currentFrame.Lock();
try
{
    _currentFrame.WritePixels(...);
}
finally
{
    _currentFrame.Unlock();
}

.NET Technique: Dispatcher As The WPF Boundary

FramePipelineService runs on a background thread. AppStateStore.StateChanged fires on whichever thread called Update. That means the view model cannot assume state changes arrive on the UI thread.

MainViewModel handles this boundary:

_dispatcher.Invoke(() => Project(state));

For frame pixels, it uses:

_dispatcher.InvokeAsync(() => UpdateCurrentFrame(frame));

The difference is practical. Project(state) updates ordinary observable properties and command state immediately. Frame rendering can be posted asynchronously so the pipeline consumer does not wait for the UI repaint.

This keeps the application layers clean:

• the infrastructure camera knows nothing about WPF
• the application pipeline knows nothing about WriteableBitmap
• the view model owns UI-thread projection and rendering

.NET Technique: BackgroundService As The Consumer Owner

FramePipelineService is a BackgroundService. The Generic Host starts it at app startup and gives it a stoppingToken during shutdown.

That service owns the one read loop:

await foreach (var frame in _source.Reader.ReadAllAsync(stoppingToken))

This shape gives the app:

• one clear consumer for IFrameSource.Reader
• async waiting instead of polling
• normal shutdown through OperationCanceledException
• app-state writes in one place
• a replaceable camera source behind the IFrameSource abstraction

The service is host-lifetime. It waits even when the camera is idle. During idle time there are simply no frames to read.

.NET Technique: CancellationToken And Run-Level Streaming

Frame streaming has two lifetimes.

The pipeline consumer is app-lifetime:

app starts
  -> FramePipelineService starts
app exits
  -> host cancels stoppingToken
  -> read loop exits

The camera producer is run-lifetime:

run starts
  -> StartStreamingAsync creates _streamingCts
  -> producer task starts

run ends, stops, aborts, or faults
  -> WorkflowService finally block calls StopStreamingAsync()
  -> _streamingCts.Cancel()
  -> producer Task.Delay wakes and exits

This is cooperative cancellation again. The token does not kill the producer task. It unblocks the delay and lets the loop stop at a known boundary.

StartStreamingAsync is idempotent enough for the current workflow:

if already streaming
  -> return completed task

StopStreamingAsync is similar:

if not streaming
  -> return completed task

The simulator does not keep a task handle and await producer completion. That is acceptable for this lightweight simulation. A real camera SDK should usually have a stronger lifecycle: start acquisition, request stop, await acquisition stopped, release driver resources.

.NET Technique: IDisposable And Multiple DI Registrations

SimulatedCamera implements three roles:

• concrete SimulatedCamera
• ICameraController
• IFrameSource

The DI setup aliases the same singleton instance under all three:

services.AddSingleton<SimulatedCamera>();
services.AddSingleton<ICameraController>(sp => sp.GetRequiredService<SimulatedCamera>());
services.AddSingleton<IFrameSource>(sp => sp.GetRequiredService<SimulatedCamera>());

That means there is one camera object, not three cameras. But the container can still track disposal through multiple registrations, so Dispose() may be called more than once on the same object.

The camera uses the same idempotent guard pattern as SimulatedTagSource:

if (Interlocked.Exchange(ref _disposed, 1) != 0) return;

Only the first caller gets to run cleanup:

first Dispose call
  -> old _disposed = 0
  -> set _disposed = 1
  -> continue cleanup

second Dispose call
  -> old _disposed = 1
  -> return immediately

In cleanup, the camera cancels any active stream, disposes the CTS, and completes the channel writer:

_streamingCts?.Cancel()
_streamingCts?.Dispose()
_frameChannel.Writer.TryComplete()

This protects app shutdown from an ObjectDisposedException if DI calls cleanup more than once. It is still a good defensive practice even if the registration is simplified later.

Best practice is to register the smallest contracts needed. For this camera, there is a real reason to expose two interfaces: workflow commands need ICameraController, while the frame pipeline needs IFrameSource. The concrete registration is useful only if another component needs concrete-only members. If not, it is worth reconsidering.

.NET Technique: Interlocked Counters And Metrics

SimulatedCamera increments _droppedCount from the producer task and FramePipelineService reads it from the consumer task:

producer
  -> Interlocked.Increment(ref _droppedCount)

consumer
  -> IFrameSource.DroppedCount
  -> Interlocked.Read(ref _droppedCount)

There is one producer task, so this is not mainly about many writers racing. It is about cross-task visibility and atomic access to a shared long.

The code checks Reader.Count >= ChannelCapacity before TryWrite to count likely drops. With a single writer and single reader this is good enough observability for the simulator. It should be treated as a backpressure signal, not as a legal-grade audit of every individual frame.

Metric counters are safe to call from background threads:

_metrics.FramesIngested.Add(1);
_metrics.FramesDropped.Add(newDrops);

System.Diagnostics.Metrics.Counter<T> is designed for concurrent instrumentation. No app-level lock is needed.

.NET Technique: Defect Detection In The Consumer

The camera does not decide whether a frame has a defect. FramePipelineService does.

That is intentional. In a real vision system, the camera captures images and a later processing stage detects defects. Keeping detection in the consumer makes the simulator look more like that future architecture.

The logic is guarded by active run state:

if AppState.ActiveRun is null
  -> ignore defect detection

That matters because frames can arrive after a run ends. The pipeline may still drain a residual frame from the channel, but it must not change defect counts for a completed or stopped run.

During an active run:

• if IDefectShowerSchedule.IsShowerActive, every frame produces a defect
• otherwise the active profile's DefectProbabilityPerFrame is used
• severity is randomly distributed across Minor, Major, and Critical
• AppState.ActiveRun is updated through the store

The update itself checks s.ActiveRun again. That second guard handles a race where the run ends between the first read and the later state update.

How To Extend It Safely

If you add a new frame profile, start in Simulator:Profiles.

Choose frame dimensions and FrameIntervalMs deliberately. A small change can be a big memory change:

width * height * bytesPerPixel * framesPerSecond

If you add a new pixel format, update all of these together:

• BytesPerPixel validation
• Frame payload contract
• MainViewModel.UpdateCurrentFrame pixel-format mapping
• tests that verify payload length and rendering assumptions

If you change payload layout to include stride padding, add an explicit stride field. Do not make the UI infer stride from width if the producer no longer uses packed rows.

If you add real image processing, do not quietly put heavy CPU work inside FramePipelineService without measuring. That service is currently the channel consumer. Heavy detection work will directly reduce drain rate and increase drops. Consider a separate detector stage if processing becomes expensive.

If you add pooling, explain what measurement justified it. Pooling changes ownership rules: the UI must not keep a buffer after the producer returns it to a pool. The current copy-into-WriteableBitmap design is simple because payload arrays are never reused.

Debugging Checklist

When the preview or frame counters look wrong, ask these questions in order:

1. Is a run actually active? The camera does not stream while idle.
2. Is CameraState set to Streaming after StartRun?
3. Which simulator profile is selected before the run starts?
4. What are FrameWidth, FrameHeight, BytesPerPixel, and FrameIntervalMs?
5. Is frames.ingested increasing?
6. Is frames.dropped increasing, and under which profile?
7. Is AppState.LatestFrame being updated?
8. Does LatestFrame.PreviewPayload exist and have Width * Height * BytesPerPixel bytes?
9. Is MainViewModel.UpdateCurrentFrame being dispatched to the UI thread?
10. Did width, height, or format change, forcing a new WriteableBitmap?
11. Are defect counts changing only while ActiveRun exists?
12. Are GC and LOH counters consistent with the active frame profile?

That order keeps the investigation layered: workflow first, producer second, channel third, app state fourth, UI rendering last.

The Takeaway

SLICE-1.2 is not only "show an image in WPF."

It is a compact model of a real vision data path:

• workflow controls when acquisition starts and stops
• the camera owns frame production
• large payloads make memory pressure visible
• a bounded channel protects the app from stale backlog
• a background service owns consumption and defect detection
• app state receives stable frame references
• the view model crosses onto the WPF dispatcher
• WriteableBitmap is reused for efficient preview rendering
• metrics and drop counters reveal whether the pipeline is keeping up

Once you see that shape, the implementation reads less like scattered async code and more like a deliberate boundary between machine acquisition, application state, and UI rendering.

Related Reading

• SLICE-1.2 Design Notes
• SLICE-1.2 Spec
• SLICE-1.1 Telemetry Deep Dive
• Streaming, Backpressure, and UI Responsiveness
• Memory Management in .NET - Real World