LCN Wafer Inspection

Appearance

Below is a deep review focused on how this stuff really works in .NET, and how a senior engineer should think when production goes wrong.

Observability, diagnostics, and debugging in .NET systems ​

PART 1 — CORE CONCEPTS RECAP ​

Observability vs monitoring vs logging ​

These words are related, but they are not the same.

Monitoring is about watching known signals. You already decided what matters, and you track it.

Examples:

  • CPU usage
  • request error rate
  • number of failed machine commands
  • queue length
  • memory growth
  • heartbeat missing for device connection

Monitoring answers:

  • “Is the system healthy?”
  • “Did a known threshold break?”
  • “Should we alert someone?”

So monitoring is about known failure modes and known indicators.

Logging is about recording events that happened. A log is a time-ordered record of what the system did, saw, decided, or failed to do.

Examples:

  • “Connected to PLC”
  • “Recipe validation failed”
  • “Camera capture started”
  • “Retry attempt 3”
  • exception stack trace
  • “Inspection result saved with batch id X”

Logging answers:

  • “What happened?”
  • “In what order?”
  • “With what input/context?”
  • “What failed?”

Logs are the most detailed raw narrative.

Observability is broader. It is the property of a system that lets you infer internal state from external outputs.

That means: when something strange happens, can you explain it without guessing?

A system is observable when it gives you enough signals to answer questions you did not anticipate beforehand.

Examples:

  • A workflow stalls, and you can see from logs + traces + queue depth + device heartbeat where it stuck.
  • A UI freezes, and you can correlate dispatcher backlog, background worker logs, and GC pauses.
  • A defect result disappears, and you can trace it across ingestion, processing, persistence, and rendering.

So:

  • logging = raw event history
  • monitoring = watching known signals
  • observability = ability to diagnose unknown behavior

A mature system needs all three.

Logs, metrics, traces ​

These are the three core telemetry types.

Logs ​

Discrete event records, usually rich in detail.

Good for:

  • exceptions
  • business events
  • warnings
  • branch decisions
  • state changes
  • payload summaries
  • forensic investigation

Weakness:

  • high volume
  • noisy if badly designed
  • hard to aggregate if unstructured

Metrics ​

Numeric measurements over time.

Examples:

  • requests/sec
  • average processing duration
  • active device connections
  • queue depth
  • error count
  • GC collections/sec
  • memory size
  • UI frame delay or render latency

Good for:

  • dashboards
  • alerts
  • trend analysis
  • SLO/SLA tracking
  • spotting regressions quickly

Weakness:

  • lacks detail
  • tells you that something is wrong, not necessarily why

Traces ​

A trace represents a logical operation moving through components.

Examples:

  • user clicks Start Inspection
  • workflow validates recipe
  • device command sent
  • image acquired
  • image analyzed
  • result persisted
  • UI updated

Each step becomes a span/activity. Together they form an execution path.

Good for:

  • latency breakdown
  • cross-component correlation
  • understanding causal flow
  • following one request or workflow end-to-end

Weakness:

  • only useful if propagated correctly
  • partial traces are often misleading

A simple way to remember it:

  • metrics tell you something is bad
  • logs tell you what happened
  • traces tell you where time and flow went

PART 2 — LOGGING INTERNALS IN .NET ​

Microsoft.Extensions.Logging architecture ​

Microsoft.Extensions.Logging is not really “a logging framework” in the same sense as Serilog or NLog. It is primarily an abstraction layer and pipeline.

Main pieces:

  • ILogger
  • ILogger<T>
  • ILoggerFactory
  • ILoggerProvider
  • optional scopes
  • provider-specific backends

The application logs through ILogger. The infrastructure routes those log events to one or more providers.

For example:

  • Console provider
  • Debug provider
  • EventSource provider
  • Application Insights provider
  • Serilog provider bridge

The important architectural idea is decoupling the application from the output destination.

Your code says:

csharp
_logger.LogInformation("Recipe {RecipeId} loaded", recipeId);

It does not care whether that ends up:

  • in console
  • in a file
  • in Seq
  • in Elasticsearch
  • in Windows Event Log
  • in OpenTelemetry exporter

That routing is handled by providers.

ILogger, providers, sinks ​

ILogger ​

ILogger is the interface your code uses.

At a high level, a log call provides:

  • log level
  • event id
  • state/payload
  • exception
  • formatter

Conceptually, a log entry is not just text. It is a structured bundle of data.

ILogger<T> ​

ILogger<T> is just a category-based logger. The category is usually the full type name.

That category matters because filtering is often configured by category.

Example:

  • MyApp.Workflow.InspectionRunner at Debug
  • Microsoft.* at Warning
  • System.Net.Http.* at Information

This lets you increase verbosity only where needed.

ILoggerFactory ​

Responsible for creating loggers and holding the provider list.

When a logger is created, it is effectively a category-aware façade over all configured providers.

ILoggerProvider ​

A provider receives log events and writes them somewhere.

A provider may internally use a sink or transport:

  • console output
  • rolling file
  • HTTP ingestion
  • ETW/EventSource
  • external logging system

In Serilog language, “sink” is common. In Microsoft.Extensions.Logging, “provider” is the main abstraction.

How log messages are processed ​

High-level flow:

  1. Application code calls ILogger.Log(...)
  2. Logger checks whether that level is enabled
  3. If disabled, ideally very little work is done
  4. If enabled, log state/template/exception are passed to each provider
  5. Each provider formats or transforms the data
  6. Provider writes to output

Key detail: filtering should happen as early as possible. If Debug logs are disabled, you want to avoid expensive formatting, allocations, and object capture.

The generic log pipeline shape ​

Internally, ILogger.Log<TState> takes:

  • LogLevel
  • EventId
  • TState
  • Exception?
  • formatter delegate

Why TState? Because the pipeline is built to support more than plain strings. The state can contain structured key-value pairs.

That is why message-template logging works well with this abstraction.

Logger categories and filters ​

Filtering is one of the most important operational tools.

You can say:

  • default = Information
  • Microsoft = Warning
  • MyApp.Workflow = Debug
  • MyApp.Device.PlcDriver = Trace during investigation

This matters in production because you often need:

  • broad low-noise logging normally
  • targeted high-detail logging during an incident

A senior engineer designs logging so filters can be turned up on a troubled subsystem without flooding everything else.

PART 3 — STRUCTURED LOGGING ​

Message templates vs string interpolation ​

This is one of the most important practical distinctions.

String interpolation ​

csharp
_logger.LogInformation($"Recipe {recipeId} loaded for machine {machineId}");

This creates a final string before or during logging flow. The message becomes basically text.

Problems:

  • values are baked into the string
  • hard to query by field
  • extra formatting/allocation cost
  • log backend cannot easily index recipeId and machineId as separate fields

Message templates ​

csharp
_logger.LogInformation("Recipe {RecipeId} loaded for machine {MachineId}", recipeId, machineId);

Now the message has:

  • template text
  • named fields
  • argument values

Backends can store:

  • RecipeId = 123
  • MachineId = M-44

That means you can query:

  • all logs for machine M-44
  • all failures for recipe 123
  • count warnings by machine
  • join with correlation id and time window

This is the real value of structured logging.

Structured data capture ​

Structured logging means you capture machine-readable context, not just prose.

Examples of valuable fields:

  • MachineId
  • DeviceName
  • RecipeId
  • LotId
  • InspectionId
  • CorrelationId
  • WorkflowState
  • RetryAttempt
  • ElapsedMs
  • ThreadId
  • TaskId sometimes, but with caution
  • UserId
  • FilePath

A good log line usually answers:

  • what operation
  • on what entity
  • in what state
  • under what correlation/workflow
  • with what outcome

Querying logs effectively ​

Bad log:

  • “Camera failed”

Good log:

  • “Camera capture failed for machine {MachineId} on recipe {RecipeId} during step {WorkflowStep} after {ElapsedMs} ms”

Now you can search by:

  • machine
  • recipe
  • workflow step
  • elapsed time
  • failure rate by step

This is how logs become an analysis tool instead of a text archive.

Senior rule for structured logging ​

A log should preserve:

  • the event
  • the entity
  • the execution context
  • the outcome

Without that, the log is mostly noise.

PART 4 — ASYNC & MULTI-THREAD DEBUGGING ​

Challenges of debugging async code ​

Async bugs are harder because the logical flow and thread flow are not the same.

In synchronous code, one stack usually tells a coherent story.

In async code:

  • work is suspended and resumed later
  • continuation may run on another thread
  • multiple tasks interleave
  • cause and effect are separated in time
  • stack traces may show where failure surfaced, not the full business journey

So production debugging becomes less about “single stack trace reading” and more about reconstructing distributed execution flow inside one process.

Lost context across threads ​

Classic debugging problem:

  • operation starts on UI thread
  • background task does I/O
  • callback completes on ThreadPool
  • result is published to event bus
  • another consumer processes it
  • exception occurs later

By then, you may have lost:

  • which user action triggered it
  • which workflow instance it belongs to
  • which machine job it came from

This is why correlation and scope matter so much.

Correlating logs across tasks ​

If one workflow creates 200 logs across 8 components, the only way to reason about them is to tie them together.

Typical tools:

  • correlation id
  • BeginScope
  • Activity.Current
  • explicit workflow identifiers
  • machine/job/inspection ids

Example mental model:

A user clicks Start. You create:

  • CorrelationId = abc123
  • InspectionId = insp-987

Every component logs those fields. Now even if logs come from different threads and time slices, you can rebuild the full story.

Async failure patterns that confuse engineers ​

Fire-and-forget task ​

A task is started and not awaited. If it fails:

  • exception may be unobserved
  • failure may be logged nowhere
  • workflow silently degrades

Parallel tasks with aggregated failure ​

You run multiple tasks. One fails early, others continue. What you see may be:

  • partial work
  • cancellation side effects
  • misleading last exception

Cancellation mistaken for failure ​

A canceled task may look like an error if logged badly. This pollutes incident analysis.

Continuation after timeout ​

Operation times out from caller perspective, but callee continues running in background. Now you get “impossible” duplicate or out-of-order logs.

These are very common in production.

PART 5 — DIAGNOSTIC TOOLS ​

Logging frameworks ​

Microsoft.Extensions.Logging ​

Good abstraction, ecosystem standard, integrates with host/DI/configuration.

Serilog ​

Very popular when structured logging matters a lot.

Strengths:

  • rich message templates
  • many sinks
  • enrichment support
  • very strong ecosystem for structured event data
  • good operational UX with tools like Seq

This is why many .NET teams use:

  • ILogger<T> in app code
  • Serilog underneath as the concrete backend

That gives clean abstractions plus powerful structured storage/query.

NLog / log4net ​

Still used in many systems, especially older enterprise apps. Less often chosen for greenfield modern systems compared to Serilog + MEL.

Basic runtime diagnostics tools ​

At a high level, diagnostics tooling falls into a few groups.

Live-process observation ​

Used when app is still running:

  • counters
  • CPU usage
  • memory usage
  • thread activity
  • exception rate
  • GC behavior

Typical .NET runtime tooling includes:

  • dotnet-counters
  • dotnet-trace
  • dotnet-monitor
  • dotnet-dump

For Windows desktop and native interop scenarios, teams also use:

  • Visual Studio diagnostics
  • PerfView
  • Process Explorer
  • WinDbg
  • ETW/EventPipe-based tools

Memory dump concept ​

A memory dump is a snapshot of process memory at a point in time.

Used when:

  • process crashed
  • memory leak suspected
  • deadlock suspected
  • app hung
  • unexplained high memory

From a dump, you can inspect:

  • managed heap
  • object counts
  • large object retention
  • thread stacks
  • finalizer queue
  • exception objects
  • sync blocks / lock contention clues

A dump is not a timeline. It is a snapshot. So it is excellent for:

  • “what is true right now?” but weaker for:
  • “what sequence led here?”

That is why dumps and logs complement each other.

Thread dump concept ​

A thread dump is a view of active threads and their call stacks.

Useful for:

  • deadlocks
  • hangs
  • blocked I/O
  • stuck worker threads
  • thread pool starvation suspicion
  • UI thread waiting on background work
  • lock contention

In desktop systems, a common failure mode is:

  • UI thread blocked waiting for a task
  • background task waiting for UI dispatcher
  • apparent freeze

A thread dump can reveal that quickly.

PerfView / ETW / EventPipe mental model ​

These tools are powerful because they observe runtime events:

  • GC
  • allocations
  • thread scheduling
  • CPU sampling
  • exceptions
  • async/task activity

They help when logs are insufficient, especially for:

  • performance regressions
  • memory churn
  • excessive allocations
  • blocked threads
  • pause analysis

Senior engineers do not jump to them first for every issue. They use them when ordinary logs no longer explain reality.

PART 6 — TRACING & CORRELATION ​

Correlation IDs ​

A correlation ID is a logical identifier that ties related events together.

It is not just for distributed microservices. It is extremely useful inside a single .NET process too.

Examples:

  • one button click
  • one inspection run
  • one device reconnect attempt
  • one batch import
  • one report generation workflow

If logs do not carry correlation, you get a pile of unrelated events from all workflows mixed together.

That is how teams lose hours.

Tracing workflows across components ​

Imagine one inspection run touches:

  • UI command handler
  • workflow orchestrator
  • machine control service
  • camera service
  • image analysis
  • repository
  • result publisher

Without tracing/correlation, each component looks fine in isolation.

With tracing, you can answer:

  • which step started late
  • where time was spent
  • where the chain broke
  • whether the operation completed, retried, or aborted

Activity and distributed tracing in .NET ​

In modern .NET, System.Diagnostics.Activity is central.

Conceptually, Activity represents a trace/span context:

  • trace id
  • span id
  • parent span id
  • tags
  • timing
  • baggage/context

This underpins OpenTelemetry-style tracing.

Even in a local app, Activity is useful because it creates a standard way to represent operation context and duration.

Typical pattern:

  • start an Activity for a business operation
  • add tags like machine id, recipe id, workflow step
  • emit logs within that activity scope
  • export traces to backend if available

That creates strong correlation between traces and logs.

Reconstructing execution flow from logs ​

When tracing is not fully available, you reconstruct flow manually using:

  • timestamp
  • correlation id
  • component name
  • operation id
  • entity ids
  • state transitions

You basically build a timeline:

  1. user initiated action
  2. workflow entered state X
  3. command sent to device
  4. timeout elapsed
  5. retry triggered
  6. result persisted
  7. UI updated incorrectly

A senior engineer treats logs like evidence, not like prose.

PART 7 — PERFORMANCE & LOGGING ​

Logging cost ​

Logging is not free.

Costs include:

  • message template parsing or formatting
  • boxing/value conversion
  • allocations
  • exception rendering
  • enrichment/context capture
  • serialization
  • I/O
  • network transport
  • downstream storage/indexing cost

In hot paths, careless logging can materially hurt throughput and latency.

Examples:

  • per-frame image processing loop
  • per-item streaming consumer
  • high-frequency device polling
  • UI render-related callbacks

Allocation impact ​

Common sources of logging allocations:

  • string interpolation
  • array/object creation for parameters
  • boxing value types
  • serializing large objects
  • capturing closures
  • exception ToString generation
  • creating log state for disabled levels

This is why high-performance logging patterns matter.

Async logging strategies ​

A common design is:

  • app thread emits log event
  • event is queued/buffered
  • background worker writes to sink

Benefits:

  • less blocking on hot path
  • smoother I/O behavior
  • better throughput

Trade-offs:

  • crash may lose buffered logs
  • queue backpressure needed
  • logging system itself can become a bottleneck
  • ordering across multiple async sinks can get messy

For production systems, you need to decide:

  • prioritize throughput?
  • prioritize reliability?
  • prioritize immediate visibility?

There is no free lunch.

High-performance logging APIs ​

In .NET, one important optimization pattern is source-generated or precompiled logging such as LoggerMessage.

Why it exists:

  • avoid repeated template parsing
  • reduce allocations
  • improve hot-path performance

Instead of ad hoc strings everywhere, you define strongly-typed log methods.

This is especially valuable in tight loops and infrastructure-heavy code.

Over-logging vs under-logging ​

Two different failures:

Over-logging ​

  • storage explosion
  • noisy signal
  • slower app
  • impossible triage
  • important events buried

Under-logging ​

  • no causality
  • no context
  • incident cannot be reconstructed
  • long MTTR

Good logging is not “log more.” It is “log the right things at the right granularity.”

PART 8 — COMMON LOW-LEVEL PITFALLS ​

String interpolation overhead in logging ​

Bad:

csharp
_logger.LogDebug($"Processing result {result.Id} in {elapsedMs} ms");

Even if Debug is disabled, you may still pay formatting/allocation cost.

Better:

csharp
_logger.LogDebug("Processing result {ResultId} in {ElapsedMs} ms", result.Id, elapsedMs);

Better still in hot paths:

  • LoggerMessage
  • source-generated logging

This is a classic senior-level detail because it mixes correctness, performance, and observability quality.

Missing correlation ​

You may have perfect logs in each class but still be blind if you cannot connect them.

Symptoms:

  • impossible to tell which logs belong to which run
  • concurrent workflows look like random interleaving
  • race conditions become invisible

A system without correlation is only half observable.

Logs without timestamps or context ​

A log saying:

  • “failed to save” is almost useless.

You need:

  • when
  • where
  • for which entity
  • under which workflow
  • after which previous event
  • with which exception
  • on which machine/node/process

Timestamps are table stakes. Context is what makes them meaningful.

Losing exceptions in async flows ​

This is one of the most dangerous pitfalls.

Examples:

  • Task.Run(() => ...) not awaited
  • event handler starts async work and ignores returned task
  • continuation swallows exception
  • background loop catches and drops exception without logging full context

Result:

  • workflow silently stops
  • production bug looks random
  • user sees stale UI or missing output
  • no obvious crash occurs

Senior rule:

  • every background task must have ownership
  • every exception path must be observed
  • every loop needs explicit failure handling strategy

Logging huge object graphs ​

Another common mistake:

  • logging full request/response payloads
  • serializing image metadata or large collections repeatedly
  • dumping giant model objects in tight loops

Problems:

  • huge cost
  • PII/security risk
  • unreadable logs
  • backend ingestion pain

Prefer targeted fields and summaries.

PART 9 — DEBUGGING PRODUCTION ISSUES ​

How to approach unknown bugs ​

A senior engineer does not start by guessing root cause. They start by narrowing the shape of the problem.

Good sequence:

1. Define the symptom precisely ​

Not “system is weird.” But:

  • UI freezes after capture completes
  • defect list duplicates items only on retry
  • save occasionally takes 20 seconds
  • machine reconnect fails once every few hours

2. Define scope ​

  • all users or one user?
  • all machines or one machine?
  • after deployment or always?
  • one workflow or many?
  • reproducible or intermittent?

3. Build timeline ​

What happened before, during, after?

4. Identify signals ​

  • logs
  • metrics
  • traces
  • dumps
  • runtime counters
  • config/version/environment differences

5. Form hypotheses and eliminate them ​

Do not jump straight to solution mode.

How to use logs to reconstruct events ​

The goal is not to read everything. The goal is to reconstruct one failing scenario.

Useful approach:

  1. find the user-visible failure timestamp
  2. identify the entity/correlation id
  3. gather all related logs
  4. sort by time
  5. mark state transitions and boundary crossings
  6. find the first divergence from expected flow

What you are looking for:

  • missing event
  • duplicate event
  • wrong order
  • unusually long gap
  • swallowed exception
  • retry without prior failure
  • timeout but operation later succeeds
  • inconsistent state transitions

This is much more effective than randomly skimming logs.

How to isolate timing issues and race conditions ​

Timing bugs rarely reveal themselves through one exception.

Typical clues:

  • only under load
  • only sometimes
  • disappears in debugger
  • more common on slower machines
  • happens near cancellation, shutdown, reconnect, or retry boundaries

Useful strategies:

Add causal logs, not just status logs ​

Instead of:

  • “entered method”

Log:

  • state before transition
  • triggering event
  • thread/context
  • correlation id
  • elapsed time since operation start

Add monotonic sequence points ​

For important workflows, log numbered milestones or explicit state transitions.

Example:

  • Transition Preparing -> Running
  • CaptureRequested
  • CaptureAcknowledged
  • ResultPublished
  • PersistenceCommitted

This makes out-of-order behavior visible.

Use narrow high-detail logging ​

Turn on Debug only around the troubled subsystem, not globally.

Compare success vs failure traces ​

The delta often reveals the missing or reordered step.

Inspect concurrency boundaries ​

Race conditions often sit at:

  • event bus publish/subscribe
  • cancellation checks
  • timer callbacks
  • device callbacks
  • UI dispatcher posts
  • retry loops
  • dispose/shutdown transitions

Deadlock/hang investigation mental model ​

For hangs or freezes, think:

  • Is UI thread blocked?
  • Is ThreadPool exhausted?
  • Is there lock contention?
  • Is a task waiting on another task that cannot proceed?
  • Is sync-over-async involved?
  • Is finalizer or disposal path blocking shutdown?

Then use:

  • thread dump / dump file
  • logs around waiting points
  • counters for thread pool / GC / exceptions
  • timing gaps in traces

A long silence in logs is itself a signal.

PART 10 — SENIOR ENGINEER MENTAL MODEL ​

How to design systems that are debuggable ​

A debuggable system does not happen by accident. It is an architectural quality.

A senior engineer designs for:

  • explicit boundaries
  • explicit state transitions
  • stable correlation ids
  • meaningful log messages
  • consistent error handling
  • observable background work
  • measurable queue/backlog/latency signals
  • failure visibility

In other words, you reduce hidden behavior.

Design principles for debuggability ​

1. Make important workflows explicit ​

Do not bury business-critical flow across random callbacks and events.

2. Log state transitions, not just errors ​

Errors are late. Transitions tell the story.

3. Preserve causality ​

Every operation should be traceable from trigger to outcome.

4. Treat background work as first-class ​

Anything running outside request/response or UI click flow needs ownership, supervision, and telemetry.

5. Standardize telemetry shape ​

Consistent field names matter:

  • CorrelationId
  • MachineId
  • WorkflowId
  • InspectionId
  • ElapsedMs

Inconsistent naming destroys query power.

6. Separate signal from noise ​

Important events should not drown in low-value chatter.

How to think during incident investigation ​

Good incident thinking is disciplined.

Not:

  • “I think GC is broken”
  • “maybe thread pool issue”
  • “let’s restart and hope”

Better:

  • what is the visible symptom?
  • when did it begin?
  • what changed?
  • where is first evidence of divergence?
  • what is the narrowest failing boundary?
  • what evidence supports each hypothesis?

A strong engineer moves from symptom to mechanism through evidence.

How to reduce MTTR ​

MTTR improves when the system answers questions quickly.

The biggest reducers of MTTR are usually:

Consistent correlation ​

Lets you isolate one failing story fast.

Clear boundary logs ​

At device calls, workflow transitions, persistence commits, and external integrations.

Actionable metrics ​

Queue depth, retry count, failure rate, latency percentiles, active workflows.

Better failure classification ​

Differentiate:

  • expected cancellation
  • timeout
  • retryable external failure
  • invariant violation
  • bug

Good dashboards + searchable logs ​

Fast access matters as much as data quality.

Crash artifacts ​

Dumps, retained recent logs, version/build metadata.

How to build confidence in production systems ​

Confidence does not come from “it seems okay.” It comes from visibility and explainability.

You trust a production system when:

  • you can detect degradation early
  • you can reconstruct incidents accurately
  • you can tell transient failure from code defect
  • you can trace one workflow end-to-end
  • you know which metrics define healthy behavior
  • you can investigate without remote debugging fantasy

Real confidence is operational confidence.

Final senior-level summary ​

A senior engineer should think of observability as part of system design, not as an afterthought.

The mature mental model is:

  • Logs are evidence
  • Metrics are health signals
  • Traces are causal maps
  • Correlation is the glue
  • Diagnostics tools are escalation tools when telemetry is not enough
  • Debuggability is an architectural property
  • MTTR is reduced by clarity, consistency, and context

And the most important practical lesson is this:

When production breaks, you usually do not get a clean repro, a friendly stack trace, and a debugger attached. You get fragments:

  • a symptom
  • a time window
  • partial logs
  • maybe a dump
  • maybe a metric spike

The engineer who wins is the one who can turn those fragments into an accurate execution story.

If you want, I can turn this into a second document in interview-prep format with: Part 1 — knowledge review and Part 2 — likely interview questions with strong sample answers.

Docs-first project memory for AI-assisted implementation.

LCN Wafer Inspection Prototype