Below is a principal-level view of Observability & Diagnosability in industrial machine software, aligned with your source of truth where this domain is called “Observability, Diagnostics & Serviceability” and emphasizes structured logging, workflow correlation, device communication logs, health metrics, diagnostic snapshots, operator-visible diagnostics, and root-cause-oriented design.

PART 1 — WHY OBSERVABILITY MATTERS MORE IN MACHINE SOFTWARE

In enterprise software, a failure is often contained inside software boundaries: a request fails, a transaction rolls back, a retry happens, an alert is raised.

In machine software, failures rarely stay inside one boundary.

They move across boundaries:

• UI to orchestration
• orchestration to workflow
• workflow to device adapter
• managed code to vendor SDK
• SDK to controller
• controller to physical hardware
• hardware back to sensors and status signals

That is why observability matters much more here. The symptom you see is often only the final visible effect, not the real cause.

A motion timeout is a good example. What the operator sees is:

“Axis move timeout.”

But the real cause may be very different:

• an interlock never became valid
• a previous reset left the axis in a disabled state
• a controller accepted the command but suppressed motion
• a stale cached signal made the workflow think motion was allowed
• a door signal flickered and motion was inhibited mid-cycle

So the visible error is motion timeout, but the root cause lives in signal state, controller state, or orchestration logic.

The same thing happens with imaging. A camera capture issue under throughput load may look like:

“Image acquisition failed.”

But the actual problem may be:

• trigger arrived before buffer readiness
• image processing pipeline blocked frame release
• native SDK callback lagged under CPU pressure
• memory pressure increased allocation latency
• stage moved before exposure completion due to timing drift

And reconnect logic is another classic trap. The UI may say the reconnect succeeded, but the device is still logically invalid:

• configuration not re-applied
• subscriptions not restored
• cached readiness state not cleared
• controller in faulted-but-connected mode
• workflow resumed against partial device state

So in this domain, the question is not merely “did the call fail?”

The real question is:

Can we reconstruct what the machine believed, what each subsystem did, and what the physical system was doing at that moment?

That is why machine software must be diagnosable not only by developers, but also by:

• support engineers
• field service engineers
• commissioning engineers
• sometimes operators or shift leaders

Because the original developer is often not present when the machine fails. The system has to preserve enough evidence so someone else can understand what happened under pressure.

PART 2 — WHAT “OBSERVABILITY” REALLY MEANS HERE

In this domain, observability is not “we have logs.”

It means the software exposes enough evidence to answer practical diagnostic questions:

• What happened?
• When did it happen?
• In what order?
• Under what machine state?
• Under what device state?
• Which subsystem initiated it?
• Which subsystem first showed abnormal behavior?
• What changed just before the failure?
• What was the machine trying to do?
• What recovery actions already happened?

That is a much richer concept than logging.

A diagnosable machine usually needs visibility in several categories.

Command traces

These show the intent of the system.

Examples:

• MoveAxis(X, target=120.500, speed=inspection)
• ArmTrigger(Camera1, recipe=DarkfieldTop)
• StartAutofocus(scanRange=200um)
• OpenVacuumValve(ChuckA)

Without command traces, you do not know what the machine was trying to do.

Workflow step transitions

These show process context.

Examples:

• LotStart
• WaferLoad
• PreAlign
• FineAlign
• CaptureStrip
• ReviewDefect
• Unload

Without workflow context, device errors become meaningless noise.

Device communication logs

These show what crossed the device boundary.

Examples:

• command sent to SDK/controller
• raw response or return code
• callback received
• timeout waiting for completion
• reconnect handshake

Without this layer, you cannot tell whether the problem is orchestration logic or device interaction.

State transitions

These show how the machine’s internal model changed.

Examples:

• MachineState: Idle → Running
• StageState: ServoOff → Homing → Ready
• CameraState: Connected → Armed → Capturing
• SafetyState: MotionPermitted → MotionInhibited

Without state transition history, failures look disconnected.

Alarms and fault history

These show abnormal conditions in business language for the machine.

Examples:

• Axis X did not reach target within timeout
• Camera trigger received while acquisition not armed
• Vacuum below threshold during wafer hold
• Door interlock opened during motion-enabled state

Without fault history, support teams lose the operational picture.

Health signals

These show whether subsystems are alive and behaving normally.

Examples:

• heartbeat freshness
• last valid frame time
• controller communication latency
• queue depth
• callback age
• reconnect count
• dropped trigger count

Without health signals, degradation remains invisible until it becomes a hard failure.

Performance and timing metrics

These show trend and accumulated stress.

Examples:

• average acquisition latency
• max motion settle time
• image queue high-water mark
• GC pause frequency
• SDK callback jitter
• UI event lag

Without timing visibility, intermittent issues become impossible to prove.

Diagnostic snapshots

These preserve the system state at important moments.

Examples:

• current recipe and active parameters
• subsystem states
• last commands per device
• current step and step elapsed time
• interlock status
• signal map
• fault ownership
• pending work queues

Without snapshots, you lose the evidence when the system resets, retries, or recovers.

So the real meaning of observability here is:

the ability to reconstruct system behavior across time, layers, and boundaries using preserved evidence, not just text output.

PART 3 — DIAGNOSTIC VISIBILITY ACROSS LAYERS

Every layer needs its own viewpoint because each layer answers a different diagnostic question.

• The UI tells you what the operator did and what the machine presented.
• The application/orchestrator tells you what operation the system was coordinating.
• The workflow layer tells you which step was active and why.
• The device abstraction layer tells you what logical device actions were requested.
• The SDK/protocol boundary tells you what actually crossed into vendor or controller territory.
• The hardware-facing layer tells you what physical state or signals existed.

If you only instrument one layer, you will always be blind somewhere important.

Here is the layer view.

+-------------------------------------------------------------+
| UI / HMI                                                    |
|-------------------------------------------------------------|
| Operator actions, screen context, alarms shown, manual cmds |
+-----------------------------|-------------------------------+
                              v
+-------------------------------------------------------------+
| Application / Orchestration                                 |
|-------------------------------------------------------------|
| Operation context, correlation ID, run/lot/recipe context,  |
| subsystem coordination, fault ownership                     |
+-----------------------------|-------------------------------+
                              v
+-------------------------------------------------------------+
| Workflow Execution                                          |
|-------------------------------------------------------------|
| Step transitions, retries, waits, pauses, resume, abort,    |
| timing per step, preconditions/interlocks                   |
+-----------------------------|-------------------------------+
                              v
+-------------------------------------------------------------+
| Device Abstraction Layer                                    |
|-------------------------------------------------------------|
| Logical commands: move, arm, capture, open, read, reset     |
| device state model, last command/result, health state       |
+-----------------------------|-------------------------------+
                              v
+-------------------------------------------------------------+
| SDK / Protocol Boundary                                     |
|-------------------------------------------------------------|
| Native API calls, controller telegrams, callbacks, return   |
| codes, timeouts, retries, reconnect sequence                |
+-----------------------------|-------------------------------+
                              v
+-------------------------------------------------------------+
| Hardware / Signals / Physical State                         |
|-------------------------------------------------------------|
| servo enabled, in-position, sensor states, trigger pulses,  |
| interlocks, heartbeat, physical readiness                   |
+-------------------------------------------------------------+

How to read this diagram:

Each layer is a different diagnostic lens. A good system allows you to move vertically through this stack for one operation or one fault.

For example:

• UI says operator pressed Start Inspection
• Orchestrator says operation InspectWafer began with correlation ID OP-10482
• Workflow says failure occurred in step FineAlign
• Device layer says MoveAxis XY completed, ArmCamera succeeded, Capture timed out
• SDK boundary says trigger callback arrived 180 ms late
• Hardware layer says motion permit toggled false for 70 ms during capture window

Now you have an actual story.

Without correlation across these layers, each subsystem looks innocent in isolation.

That is why lack of cross-layer visibility is so destructive. Teams start arguing:

• “UI issue”
• “workflow bug”
• “SDK problem”
• “hardware glitch”

In reality, the system simply failed to preserve the chain of evidence.

PART 4 — LOGGING, EVENTS, METRICS, AND SNAPSHOTS

A strong machine system uses multiple diagnostic forms because each form answers a different class of question.

Logs

Logs tell the narrative.

They are best for:

• command issued
• step entered
• device response received
• timeout occurred
• recovery started
• exception details
• operator action

A good log answers:

• what action was attempted
• with what parameters
• under what context
• with what result

Logs are sequential and human-readable. They help reconstruct stories.

But logs alone are not enough because they are often too verbose, incomplete, or hard to aggregate by state.

Events

Events record significant transitions.

Examples:

• WorkflowStepEntered
• AxisMoveCompleted
• CameraDisconnected
• SafetyInterlockOpened
• RecipeActivated
• AlarmRaised
• AlarmCleared

Events are useful because they represent machine-significant moments, not just debug chatter.

They let you build history views like:

• alarm timeline
• workflow timeline
• fault lifecycle
• state transition journal

Events are especially valuable when you want structured history that survives beyond raw log files.

Metrics and counters

Metrics reveal trend, drift, and degradation.

Examples:

• average move completion time
• max settle time over last hour
• dropped frame count
• reconnect count per shift
• callback latency percentiles
• queue depth high-water marks
• memory growth trend
• heartbeat lateness

Logs tell you a story after something happened.

Metrics tell you the system was getting unhealthy before the failure became visible.

That distinction matters a lot in long-running machines.

Snapshots

Snapshots preserve state at critical moments.

Examples:

• machine state map when alarm raised
• last command per device
• current recipe values
• active interlocks and permissives
• subsystem health summary
• queue contents or counts
• last N state transitions
• current workflow step and elapsed time

A snapshot is often the difference between:

“We think it failed during alignment”

and

“At 14:03:22.481 the machine was in FineAlign, Camera1 was Armed, StageX was InPosition=false, Interlock MotionPermit=false, VacuumChuck=OK, last command=CaptureFrame, last callback age=812 ms.”

That is real diagnosability.

So the relationship is:

• logs tell the narrative
• events show significant transitions
• metrics reveal trend and degradation
• snapshots preserve the exact state at critical moments

If you only use one of these, you will miss important evidence.

Here is the data-flow view.

               +-------------------+
               |   UI / Operator   |
               +---------+---------+
                         |
                         v
+------------+   +-------+--------+   +------------------+
|  Devices    |-->| Workflow/App   |-->| Alarm/Fault Mgr  |
+------+-----+   +-------+--------+   +---------+--------+
       |                     |                      |
       |                     |                      |
       v                     v                      v
  [Trace Logs]         [Domain Events]        [Fault History]
       |                     |                      |
       +----------+----------+----------+-----------+
                  |                     |
                  v                     v
             [Metrics]             [Snapshots]
                  |                     |
                  +----------+----------+
                             |
                             v
                  [Timeline Reconstruction]

How to read this diagram:

Different diagnostic artifacts are produced from different parts of the system, but they must converge into a reconstructable history.

That convergence is the key design goal.

PART 5 — CORRELATION & TIMELINE RECONSTRUCTION

A machine operation is rarely one call.

It is a chain:

• operator action
• orchestration start
• workflow step entry
• one or more device commands
• asynchronous callbacks
• state changes
• result or fault

If these pieces cannot be tied together, support becomes guesswork.

A diagnosable system needs at least these correlating dimensions:

• precise timestamps
• correlation ID / operation ID
• run, lot, wafer, recipe, or job context when applicable
• subsystem identifier
• device identifier
• command ID
• command/result pairing
• state transition timestamps
• alarm/fault ID with owning context

Here is a simplified traced operation.

Time ----->

Operator/UI        Orchestrator        Workflow         Stage Device       Camera Device
    |                   |                 |                  |                  |
1   | Start Inspect     |                 |                  |                  |
    |------------------>|                 |                  |                  |
    |                   | Begin OP-10482  |                  |                  |
2   |                   |---------------->| Enter FineAlign  |                  |
    |                   |                 |----------------->| MoveTo(XY)       |
3   |                   |                 |                  |---- cmd#771 ---->|
    |                   |                 |                  |<-- in-position ---|
4   |                   |                 | Arm Capture      |                  |
    |                   |                 |------------------------------------>|
5   |                   |                 | CaptureFrame     |                  |
    |                   |                 |------------------------------------>|
6   |                   |                 | wait callback    |                  |
    |                   |                 |                  |<-- motion permit false
7   |                   |                 | timeout          |                  |
    |                   |<----------------| Fault: CaptureTimeout                |
8   | Show Alarm        |                 |                  |                  |
    |<------------------| Snapshot saved  |                  |                  |

What makes this useful is not the drawing itself. It is the correlated evidence behind it:

• the UI action is linked to OP-10482
• the workflow step is FineAlign
• the stage move has command ID 771
• the capture belongs to the same operation
• the fault is timestamped after motion-permit dropped
• a snapshot is captured before recovery clears evidence

This lets you answer the real question:

Was this a camera problem, a motion problem, a safety/interlock problem, or a workflow timing problem?

Without correlation, all you have is:

• “capture timeout”
• “move completed”
• “operator started inspection”

Those are disconnected facts, not a diagnosis.

PART 6 — WHAT GOOD DIAGNOSTICS LOOK LIKE IN REAL SYSTEMS

Good diagnostics are concrete.

They give engineers the exact information needed to narrow fault ownership quickly.

Here are examples of genuinely useful diagnostic capabilities.

“Last known command to device”

For each device or subsystem, you should be able to answer:

• what was the last command
• when was it issued
• with what parameters
• whether completion was observed
• what the last result or return code was

This is far more useful than “camera error.”

Example:

• Device: Camera1
• LastCommand: CaptureFrame(exposure=1200us, gain=2.5, trigger=external)
• IssuedAt: 14:03:21.992
• CompletionObserved: No
• LastCallback: ArmComplete at 14:03:21.814
• PendingDuration: 812 ms

“Workflow step when fault occurred”

A fault is much easier to reason about when tied to process context.

Example:

• Fault: AxisMoveTimeout
• WorkflowStep: WaferUnload/MoveToCassetteSlot
• StepElapsed: 00:00:12.311
• RetryAttempt: 2/3
• EnteredFrom: VacuumRelease
• CurrentMode: Auto
• Recipe: Product_A_Rev7

That tells you what the machine was trying to accomplish.

“State transition history for machine/subsystem”

State history often reveals invalid sequences.

Example:

• StageState: Ready → Moving → Settling → Faulted
• MotionPermit: True → False → True
• CameraState: Armed → WaitingTrigger → Timeout
• WorkflowState: CaptureStrip → RetryCapture → Faulted

This is much more informative than a single final state.

“Last healthy heartbeat / last valid data”

For long-running systems, the absence of fresh good data matters.

Example:

• PLC heartbeat last seen 380 ms ago
• Last valid encoder update 42 ms ago
• Last good frame from Camera2 13.2 s ago
• Last vacuum pressure within range 4.8 s ago

These indicators help distinguish:

• disconnected
• stale
• delayed
• alive but unhealthy

“What changed since startup or since recipe activation”

A surprising number of issues come from mid-run changes.

Examples:

• recipe parameter changed
• exposure profile reloaded
• stage velocity override applied
• device reconnect happened
• calibration file refreshed
• maintenance mode toggled
• light controller channel remapped

So a strong system keeps change journals, not just final values.

“Which subsystem owns current fault”

Ownership matters.

A good diagnostic model distinguishes:

• originating subsystem
• impacted subsystem
• reporting subsystem

For example:

• Origin: StageInterlockMonitor
• DetectedBy: CameraCaptureWorkflow
• ReportedAtUIAs: CaptureTimeout

That is a very mature diagnostic design, because it separates symptom from source.

These capabilities are far more valuable than logs like:

• “operation failed”
• “device error”
• “timeout occurred”

Those messages are not false, but they are operationally weak.

PART 7 — REAL-WORLD FAILURE SCENARIOS

Scenario 1 — Logs exist but are too vague to isolate fault source

What it looks like in production

The log shows:

• Start inspection
• Move completed
• Capture failed
• Operation aborted

Everyone knows the machine failed, but nobody knows why.

Why it happens

The system logs outcomes but not context:

• no command parameters
• no workflow step
• no device state
• no preconditions/interlocks
• no timing breakdown

How experienced engineers improve it

They log intent and context, not just outcome.

Instead of:

• “Capture failed”

They preserve:

• operation ID
• workflow step
• device ID
• trigger mode
• arm state
• last callback age
• interlock state
• recent state transitions

That turns an event into evidence.

---

Scenario 2 — Device layer error never gets correlated to workflow context

What it looks like in production

The device log shows:

• SDK returned error 0x830012

The workflow log separately shows:

• Align wafer failed

But there is no link between them.

Why it happens

The architecture treats device diagnostics and workflow diagnostics as separate worlds.

How experienced engineers improve it

They propagate operation context downward and bubble diagnostic context upward.

So the device error is recorded as:

• operation OP-10482
• workflow step FineAlign
• logical command CaptureAlignmentImage
• device command Camera1.CaptureFrame
• SDK error 0x830012

Now the error sits inside the process context.

---

Scenario 3 — Timestamps from different subsystems make reconstruction impossible

What it looks like in production

UI shows alarm at 14:03:22.900

Device log shows timeout at 14:03:21.100

Controller log shows interlock drop at 14:03:23.500

Sequence makes no sense.

Why it happens

• unsynchronized clocks
• inconsistent timestamp precision
• local time in one place, UTC in another
• some logs stamped at emission time, others at write time

How experienced engineers improve it

They standardize time handling:

• one canonical timestamp basis
• consistent precision
• monotonic elapsed timing for local sequencing
• explicit event-time vs log-write-time if needed

In machine diagnosis, timestamp consistency is not cosmetic. It is foundational.

---

Scenario 4 — Fault is cleared before evidence is preserved

What it looks like in production

Operator sees alarm, presses reset, machine recovers.

Later, developers ask for evidence.

There is none.

Why it happens

The system resets state before preserving:

• active workflow step
• device states
• interlocks
• recent commands
• pending waits
• health summary

How experienced engineers improve it

They capture evidence before reset or recovery logic mutates the system.

This usually means:

• snapshot on fault raise
• last-N event ring buffers
• fault-specific evidence payload
• alarm lifecycle journal

This is one of the strongest habits in real machine software.

---

Scenario 5 — UI shows alarm but no trace of the command/event chain

What it looks like in production

Alarm panel says: “Axis communication error.”

But the operator or field engineer cannot answer:

• during which operation?
• after which command?
• after reconnect or before reconnect?
• isolated or repeated?
• which axis state existed before the fault?

Why it happens

The UI only displays current alarm text, not diagnostic history.

How experienced engineers improve it

They design UI-visible diagnostics with layered depth:

• operator view: clear actionable fault
• service view: context, timeline, related subsystem state
• engineering export: full structured evidence

Same fault, different audiences, same underlying evidence.

---

Scenario 6 — Service engineers cannot tell whether issue is hardware, SDK, or orchestration logic

What it looks like in production

Everything gets labeled “software issue” or “hardware issue” based on whoever is loudest.

Why it happens

The system does not preserve boundary evidence.

How experienced engineers improve it

They instrument the boundaries explicitly:

• command crossed app/device boundary at T1
• SDK accepted/rejected at T2
• controller heartbeat healthy/unhealthy at T3
• physical ready signal valid/invalid at T4

Now you can separate:

• orchestration sent wrong command
• SDK call failed
• controller ignored it
• hardware never reached expected signal

That is true diagnosability.

PART 8 — SOFTWARE DESIGN IMPLICATIONS

Diagnosability is an architectural property.

It is not a logging library choice.

If the architecture hides context, collapses state, or mutates evidence before recording it, no logging framework will save you.

A diagnosable machine system usually includes these design decisions.

1. Structured logging, not scattered strings

Bad:

"Move failed"
"Camera error"
"Timeout happened"

Good logs carry fields like:

• operation ID
• subsystem
• device
• command
• workflow step
• machine mode
• alarm ID
• elapsed time
• result code

The key point is not JSON versus text. The key point is that the log entry preserves machine meaning.

2. Boundary-level tracing

You must trace important transitions at architectural boundaries:

• UI action accepted
• orchestration started operation
• workflow entered step
• device command issued
• SDK/protocol call made
• callback or hardware state change observed
• alarm raised
• recovery started

These are the moments where causality is lost if not recorded.

3. Explicit state transition recording

Hidden state changes are poison for diagnosis.

If states matter to behavior, their transitions should be observable.

Especially for:

• machine modes
• workflow states
• device connectivity
• readiness
• interlocks
• fault ownership
• recovery phases

4. Contextual alarms and faults

A fault should not just say what failed.

It should preserve:

• where
• during what
• under which state
• with what preceding evidence
• who owns the fault

This makes alarms useful for diagnosis, not just notification.

5. Preserve evidence before reset/recovery

Recovery logic is often evidence-destroying logic.

Architecturally, this means:

• snapshot before reset
• ring buffer of recent events
• last commands retained per device
• current step and state history preserved
• fault lifecycle journal separate from current live state

6. Make diagnostics useful to multiple audiences

Developers, service engineers, and operators need different depths.

A mature design usually separates:

• operator-facing fault explanation
• service-facing diagnostic drilldown
• engineering-facing exported trace

But all of them should come from the same evidence model, not three separate truths.

Here is a component view.

+------------------+        +-----------------------+
| UI / HMI         |        | Diagnostic Viewer     |
|------------------|        |-----------------------|
| operator actions |        | timeline, faults,     |
| alarms shown     |        | snapshots, health     |
+--------+---------+        +-----------+-----------+
         |                              ^
         v                              |
+--------+------------------------------+-----------+
| Application / Workflow / Fault Manager            |
|---------------------------------------------------|
| operation context, step transitions, fault model, |
| evidence capture, correlation IDs                 |
+--------+-------------------+----------------------+
         |                   |
         v                   v
+--------+--------+   +------+----------------------+
| Device Services |   | Diagnostic Pipeline         |
|-----------------|   |-----------------------------|
| logical cmds    |   | structured logs             |
| health state    |   | events                      |
| last command    |   | metrics                     |
+--------+--------+   | snapshots                   |
         |            +------+----------------------+
         v                   |
+--------+--------+          v
| SDK / Protocol  |    +-----+----------------------+
|-----------------|    | Evidence Storage / Export  |
| API calls       |    |----------------------------|
| callbacks       |    | history, ring buffers,     |
| return codes    |    | fault records, service pkg |
+--------+--------+    +----------------------------+
         |
         v
+--------+--------+
| Hardware        |
|-----------------|
| signals, motion,|
| sensors, state  |
+-----------------+

How to read this diagram:

The important idea is that diagnostics are not an afterthought attached to components. They are a parallel architecture that collects evidence from the system’s real boundaries and makes that evidence reconstructable.

Bad vs good approach

Bad approach

• each class writes random strings
• errors are generic
• current state overwrites previous state
• alarms lose workflow/device context
• recovery clears evidence
• timestamps are inconsistent
• no correlation across layers

This creates support dependency on tribal knowledge.

Good approach

• operations are traceable end to end
• boundaries emit structured evidence
• states have visible transitions
• faults preserve context and ownership
• snapshots are taken before mutation/reset
• timeline reconstruction is possible
• service engineers can work without the original developer

That is serviceable architecture.

PART 9 — INTERVIEW / REAL-WORLD TALKING POINTS

If you want to explain this clearly in interviews or architecture discussions, these are the strongest points.

How to explain observability in industrial systems

A strong answer sounds like this:

“Observability in machine software is the ability to reconstruct what the system was trying to do, what each subsystem did, what state the machine was in, and what changed before failure. It is not just logs. It requires cross-layer tracing across UI, workflow, device, SDK, and hardware boundaries, plus preserved evidence such as state transitions, alarm history, health signals, and snapshots.”

That immediately sounds domain-aware.

Why “add more logs” is not a serious answer

Because the real problem is usually missing structure and missing correlation, not missing volume.

More unstructured logs often make diagnosis worse:

• too noisy
• still no causality
• still no state history
• still no fault ownership
• still no timeline reconstruction

The mature answer is:

“Add the right evidence at the right boundaries with preserved context.”

Common mistakes engineers make when entering this domain

They often:

• log symptoms but not intent
• treat alarms as UI messages instead of evidence objects
• ignore state transition history
• fail to propagate operation context downward
• fail to preserve evidence before auto-recovery
• mix operator messaging and engineering diagnostics badly
• assume device reconnect means device validity
• underestimate timing and timestamp consistency

These are classic transition mistakes from business software into machine software.

What strong engineers understand

Strong engineers understand that in machine systems:

• failures often happen at boundaries
• the root cause is often far from the visible symptom
• intermittent problems require preserved evidence, not memory
• serviceability matters as much as correctness
• a system is not truly production-ready if only the original developer can diagnose it

They know that good observability means:

• cross-layer tracing
• state-aware diagnostics
• contextual alarms
• evidence preservation before reset
• supportability for field engineers under time pressure

That is the real architectural mindset.

Closing mental model

The simplest way to remember all of this is:

A machine is diagnosable when you can replay its story after the fact.

Not perfectly, not at physics-lab fidelity, but well enough to answer:

• what operation was happening
• what the machine believed
• what each subsystem did
• where the first abnormal condition appeared
• what evidence existed before recovery changed the state

That is what observability and diagnosability really mean in industrial machine software.

And that is why this domain deserves its own architectural design, not just a logging package added at the end.