LCN Wafer Inspection

Appearance

Reliability, Retries & Fault Handling ​

In industrial machine software, communication reliability is not a networking detail. It is part of machine behavior.

A business system can often tolerate “try again later.” A machine usually cannot. If communication becomes unreliable while software is coordinating motion, sensors, actuators, PLC handshakes, cameras, or external devices, the result is not just inconvenience. It can become lost state, duplicated actions, bad recovery, hardware faults, scrap, or unsafe operator experience.

That is why reliability, retries, and fault handling must be designed as first-class behavior, not added as a wrapper around device calls. This topic sits directly inside the roadmap’s “Reliability, Fault Handling & Recovery” area, which emphasizes timeout design, retry vs operator intervention, reconnect logic, failsafe design, and startup/shutdown robustness.

PART 1 — WHY COMMUNICATION IS INHERENTLY UNRELIABLE ​

In industrial systems, communication fails for many reasons, and many of them are not clean failures.

A device may be alive but overloaded. A network may drop a packet but recover a second later. A serial line may introduce corruption because of electrical noise. A PLC may still be running but delay response because its scan cycle is busy. A camera SDK may accept a command but take longer than expected to acknowledge. A robot controller may process a command but the PC may miss the reply because of transport instability.

So the important mindset shift is this:

Communication failure is normal operating reality, not an exceptional edge case.

That matters because machine software often sits across boundaries like:

  • PC app ↔ motion controller
  • PC app ↔ PLC
  • PC app ↔ smart sensor
  • PC app ↔ camera / frame grabber
  • machine ↔ host / MES
  • subsystem ↔ subsystem over fieldbus or Ethernet

Each boundary introduces uncertainty:

  • Did the command arrive?
  • Did the device execute it?
  • Did the response get lost?
  • Was the device slow, or dead?
  • Is the connection broken, or just delayed?

In enterprise systems, people often think in terms of request success vs request failure. In machine systems, that model is too simple. A more realistic model is:

  • request may be sent
  • transport may partially succeed
  • device may act
  • acknowledgement may be delayed or absent
  • local software may not know which part happened

That ambiguity is the real problem.

Typical real-world examples ​

Example 1: Command sent, response delayed A stage controller receives MoveTo(X=120.0) and starts moving, but the acknowledgement arrives late because the controller is under load. If your timeout is too short, the PC may assume failure and resend. Now you may have duplicate or conflicting control flow.

Example 2: Device responds sometimes, not always A barcode scanner or smart sensor may intermittently miss replies because of a noisy serial line. Retrying a read is often fine. Retrying an action command may not be.

Example 3: Brief network drop under load A PLC connection over Ethernet drops for 800 ms during switch congestion. The machine software must decide whether to:

  • keep waiting
  • retry
  • reconnect
  • freeze workflow
  • enter fault
  • hold outputs safe

A strong system assumes all of these can happen during normal plant operation.

PART 2 — TYPES OF FAILURES ​

Not all failures mean the same thing, so they must not be handled the same way.

1. Transient failures ​

These are short-lived failures that may succeed if retried after a short delay.

Examples:

  • a read times out once because the device is busy
  • one TCP packet is lost and the next request succeeds
  • a camera takes longer than usual to return a status
  • a reconnect succeeds immediately after a brief network flap

These are the classic candidates for controlled retry.

2. Persistent failures ​

These continue until something external changes.

Examples:

  • device powered off
  • cable unplugged
  • controller firmware hung
  • wrong COM port / IP configuration
  • fieldbus node failed hard

Retrying may be useful only to confirm persistence, but repeated retry will not “heal” the problem.

3. Partial failures ​

This is one of the hardest categories in industrial systems.

Examples:

  • command reached controller, but reply was lost
  • PLC accepted a bit change, but PC timed out waiting for handshake confirmation
  • one subsystem completed, another did not
  • batch read of multiple values returned incomplete data
  • transaction across controller and local state updated only one side

Partial failure is dangerous because software state and physical state may diverge.

4. Silent failures ​

No explicit error is returned. Things just stop progressing.

Examples:

  • no response
  • heartbeat stops changing
  • a device remains connected but never completes command
  • workflow waits forever on a condition that will never arrive

Silent failure is often more dangerous than explicit failure because nothing forces the system into a safe state unless you design timeout and watchdog rules.

Why handling depends on type ​

Because the key question is not “did the call fail?” The key question is:

What is the most likely physical truth right now, and what is safe to do next?

That is the core architectural perspective.

PART 3 — TIMEOUTS ​

A timeout is not just a technical parameter. It is a statement about how long the system is willing to remain uncertain.

If the timeout is too short, the software manufactures failures that are not real. If the timeout is too long, the software delays response to actual failures.

So timeout design must reflect:

  • operation type
  • normal latency distribution
  • device behavior under load
  • machine safety implications
  • operator expectations
  • recovery behavior after expiry

Different operations need different timeouts ​

A common mistake is one global timeout for everything.

That fails because these are very different:

  • reading a status register
  • waiting for a slow controller response
  • waiting for physical motion complete
  • waiting for a vacuum sensor to stabilize
  • waiting for a PLC handshake
  • waiting for a host system reply

A strong system distinguishes at least:

  • communication timeout: how long to wait for transport/protocol response
  • execution timeout: how long a physical action is allowed to take
  • progress timeout: how long the system tolerates no progress
  • reconnect timeout/window: how long recovery is attempted before escalation

Why incorrect timeout values are dangerous ​

If timeout is too short:

  • you get false failures
  • you trigger unnecessary retries
  • you can create duplicate commands
  • you can generate alarm noise
  • you may switch machine state into fault while the device is actually healthy

If timeout is too long:

  • operator waits too long for fault visibility
  • workflow hangs
  • fault propagation is delayed
  • safety response can be late
  • blocked resources stay occupied too long

ASCII timing diagram — timeout behavior ​

text
Time  --------------------------------------------------------------->

PC App         |---- Send Command ----|................ waiting ................|
Device         |<--- receives command --->|---- processes ----|---- reply ----->|

Case A: Correct timeout
PC Timeout     |----------------------------- long enough ----------------------|
Result         Reply arrives before timeout -> success

Case B: Timeout too short
PC Timeout     |----------- expires ---------|
Result         PC declares failure before device replies
Risk           retry / reconnect / false alarm while device was still working

Case C: Timeout too long
PC Timeout     |---------------------------------------------- expires ---------|
Result         real fault detected too late
Risk           delayed reaction, blocked workflow, poor operator experience

Principal-level rule ​

Timeouts should be derived from operation semantics and observed device behavior, not guessed once and copied everywhere.

PART 4 — RETRY STRATEGIES ​

Retries are useful, but only when they reduce uncertainty without increasing risk.

That sounds obvious, but many systems implement retries as a generic wrapper: “if call fails, retry three times.” In industrial systems, that is often exactly the wrong approach.

When retries are appropriate ​

Retries are appropriate when all of these are reasonably true:

  1. the failure is likely transient
  2. the operation is safe to repeat
  3. the cost of delayed success is lower than the cost of escalation
  4. retry will not overload the device or system
  5. retry will not hide an important problem too long

Common retry patterns ​

1. Immediate retry ​

Used when:

  • transient glitch is common
  • operation is lightweight
  • fast success is preferred
  • duplicate effect is safe

Typical example:

  • reading a status word
  • polling a sensor register

Not good for expensive or state-changing actions.

2. Delayed retry ​

Used when:

  • device may need recovery time
  • bus contention or overload may clear
  • reconnect sequence needs settling time

Typical example:

  • reconnecting to a controller
  • retrying a read after “device busy”

3. Exponential backoff ​

Used when repeated attempts can worsen system load.

Typical example:

  • reconnecting to a host or PLC over unstable network
  • recovering from device overload
  • avoiding synchronized retry storms across multiple clients

4. Limited retry attempts ​

Essential in industrial systems.

Because retries without a hard bound become:

  • hidden hangs
  • retry storms
  • operator confusion
  • delayed fault escalation
  • systems that never admit failure

Why not all operations should be retried ​

Because some operations change the physical world.

Reading a sensor value is usually safe to repeat. Issuing a motion command is not automatically safe. Turning on a vacuum, opening a clamp, triggering exposure, firing a laser, or telling a robot to pick may create side effects that must not be blindly duplicated.

ASCII sequence diagram — safe vs unsafe retry ​

text
Safe retry example: status read

PC App           Device
  |                |
  |--- ReadStatus->|
  |                |   (timeout / transient miss)
  |<-- no reply ---|
  |--- ReadStatus->|
  |<-- Status=OK --|
  |                |

Usually acceptable because repeated read does not change physical state.
text
Unsafe retry example: actuator command

PC App              Controller              Actuator
  |                      |                     |
  |--- ExtendClamp ----->|-------------------->|
  |                      |     executes        |
  |<-- no reply / timeout|                     |
  |--- ExtendClamp ----->|-------------------->|
  |                      |   duplicate action? |
  |                      |   already moving?   |
  |                      |   invalid state?    |

Blind retry may create duplicate or contradictory side effects.

Better framing for retries ​

Instead of “retry failed calls,” think:

  • retry safe observations
  • cautiously retry reconnectable interactions
  • do not auto-retry state-changing commands unless explicitly designed for it

That is much closer to real industrial practice.

PART 5 — IDEMPOTENCY & SAFE RETRIES ​

Idempotency is one of the most important concepts in reliable machine communication.

An operation is idempotent if repeating it produces the same intended final effect as doing it once.

That does not mean “nothing happens.” It means duplicate requests do not create extra unintended change.

Safe examples ​

  • ReadTemperature()
  • GetAxisPosition()
  • QueryAlarmList()
  • SetMode(Auto) if protocol/device defines it as absolute state assignment and duplicate requests are harmless
  • EnsureOutput(X)=ON if implemented as setting a target state, not toggling

Unsafe or risky examples ​

  • PulseOutput()
  • TriggerCapture()
  • StartCycle()
  • MoveRelative(+10mm)
  • OpenGripper(), depending on context
  • AdvanceConveyorOneStep()
  • IncrementCounter()

These may produce repeated side effects if retried.

Why this matters under timeout ambiguity ​

The hardest case is:

  • command sent
  • physical device may have executed
  • acknowledgement lost
  • software times out

Now software does not know whether to resend.

That means good industrial protocols and software layers often need one or more of these:

  • command IDs / correlation IDs
  • acknowledgement with unique operation identity
  • queryable device state
  • explicit command completion tracking
  • absolute commands instead of relative commands where possible
  • deduplication logic in device/controller
  • two-phase handshake for critical actions

Example ​

Bad design:

  • PC sends MoveRelative(+5)
  • timeout occurs
  • PC resends MoveRelative(+5)
  • actual movement may become +10

Better design:

  • PC sends MoveTo(Absolute=125.0, CommandId=8421)
  • controller records command ID
  • if duplicate arrives, controller answers with existing status rather than re-executing

Even if the device protocol is primitive and does not support deduplication, your application must still reason about which operations are safe to retry and which require verification before reissue.

Practical rule ​

Before adding retry, ask:

  1. Is the operation observational or state-changing?
  2. If the first attempt actually succeeded, what happens if I send it again?
  3. Can I verify resulting state before retrying?
  4. Is the command absolute, relative, or one-shot trigger?
  5. Who owns deduplication: app, controller, or protocol?

That is the architect’s checklist.

PART 6 — FAULT HANDLING & ESCALATION ​

Eventually, retries must stop and the system must make a controlled decision.

This is fault handling.

The purpose of fault handling is not just to report failure. It is to move the machine into a known, controlled, diagnosable state.

When retry fails ​

After bounded retry attempts, software should not keep improvising. It should do something explicit, such as:

  • mark communication degraded
  • stop the affected workflow
  • inhibit further commands to that subsystem
  • notify operator / service layer
  • move machine or subsystem into fault state
  • request operator intervention
  • trigger controlled stop
  • preserve evidence for diagnosis

Recoverable vs non-recoverable faults ​

Recoverable fault ​

A recoverable fault is one where the system can safely return to service through defined recovery steps.

Examples:

  • temporary device disconnect with successful reconnect
  • one failed status read after retry
  • brief heartbeat loss below escalation threshold
  • retryable host/MES communication issue not affecting machine safety

Non-recoverable fault ​

A non-recoverable fault is one where automatic continuation is unsafe or state certainty is lost.

Examples:

  • motion command timed out and actual axis state is uncertain
  • actuator command may have executed but state cannot be confirmed
  • controller reset occurred mid-sequence
  • PLC handshake state diverged
  • safety-related communication lost
  • repeated reconnect failed beyond allowed window

Controlled fault handling means ​

  • stop issuing risky commands
  • freeze workflow advancement
  • keep system state explicit
  • surface meaningful alarm context
  • require re-homing / re-initialization / operator verification where needed
  • avoid hidden partial recovery

ASCII state diagram — fault escalation ​

text
+------------------+
| Normal Operation |
+------------------+
          |
          v
+------------------+
| Comm Failure     |
| Detected         |
+------------------+
          |
          v
+------------------+
| Retry / Verify   |
| Window           |
+------------------+
     |         |
     |         |
     |success  |exceeded / unsafe ambiguity
     v         v
+------------------+      +----------------------+
| Recovered        |      | Faulted / Escalated  |
| Return to Normal |      | Commands Inhibited   |
+------------------+      +----------------------+
                                      |
                                      v
                           +----------------------+
                           | Recovery Procedure   |
                           | Reconnect / Reinit / |
                           | Operator Verify      |
                           +----------------------+
                                      |
                           +----------+----------+
                           |                     |
                        success                fail
                           |                     |
                           v                     v
                  +------------------+   +------------------+
                  | Normal Operation |   | Service Required |
                  +------------------+   +------------------+

Why escalation must be explicit ​

A weak system keeps retrying and hides the issue. A strong system says:

  • what failed
  • what was attempted
  • what is now inhibited
  • what state is safe
  • what recovery is required

That is the difference between a debuggable machine and a machine that “sometimes gets stuck.”

PART 7 — REAL-WORLD FAILURE SCENARIOS ​

These are the patterns that repeatedly hurt real projects.

Scenario 1 — Retry hides the real issue ​

What it looks like Operators report random slowness. Logs show occasional timeouts, but the software eventually succeeds after retry, so no alarm is raised. Weeks later, the device fails hard in production.

Why it happens Retries mask the early warning signs of degradation:

  • overloaded device
  • bad cable
  • unstable switch
  • firmware regression
  • queue buildup

How engineers fix it

  • distinguish first-attempt failure from final success
  • log retry counts and latency growth
  • alert on degraded communication patterns
  • treat increasing retries as health signal, not “success”

A retry that succeeds is still diagnostic evidence.

Scenario 2 — Retry storm overwhelms device/system ​

What it looks like Network becomes unstable. Multiple components begin retrying aggressively. Device load spikes. Failures multiply. Whole machine appears to collapse.

Why it happens Each layer retries independently:

  • protocol layer retries
  • device wrapper retries
  • workflow retries
  • supervisory layer retries

Now one original failure becomes many requests.

How engineers fix it

  • define one owner for retry
  • use bounded retries with backoff
  • apply circuit-break / suppression behavior where appropriate
  • stop upper layers from stacking retries on lower layers
  • treat degraded device as unavailable, not as infinite retry target

This is a system design problem, not just a transport problem.

Scenario 3 — Duplicate command causes unexpected behavior ​

What it looks like A motion or actuator operation happens twice, or system enters an unexpected controller state after timeout recovery.

Why it happens Command timed out locally, but remote side executed it. Retry resends a non-idempotent action.

How engineers fix it

  • classify commands by retry safety
  • use command IDs if possible
  • prefer absolute state-setting over one-shot triggers
  • verify device state before reissuing
  • require operator intervention when ambiguity cannot be resolved safely

Scenario 4 — Timeout too short causes false alarms ​

What it looks like System faults during high load, but only when image acquisition, logging, or UI refresh is busy.

Why it happens Timeout based on ideal lab response time, not production worst-case behavior.

How engineers fix it

  • measure normal and peak latency distributions
  • separate protocol response timeout from physical execution timeout
  • apply per-operation timeout policies
  • test under load, not only in clean environment

Scenario 5 — Timeout too long delays critical reaction ​

What it looks like Workflow appears frozen. Operator waits too long. Recoverable issue turns into larger stop because system keeps waiting.

Why it happens Engineers set a generous timeout “to avoid false alarms,” but now the software delays fault handling far too much.

How engineers fix it

  • define maximum acceptable uncertainty duration per operation
  • use progress monitoring, not only absolute timeout
  • break long waits into staged checks
  • surface degraded state before full timeout if needed

Scenario 6 — System stuck retrying indefinitely ​

What it looks like Machine never fully faults and never recovers. It just stays in “trying…” forever.

Why it happens No bounded retries, no escalation state, no ownership of recovery decision.

How engineers fix it

  • strict retry budgets
  • explicit escalation thresholds
  • state transition to fault/degraded mode
  • operator/service workflow for unresolved failures

A system that never decides is often worse than a system that faults clearly.

PART 8 — SOFTWARE DESIGN IMPLICATIONS ​

Reliability cannot be sprinkled in afterward. It must shape the communication architecture.

The roadmap explicitly treats fault handling and recovery as a core high-priority domain because machine systems must detect failures, fail safely, report clearly, and recover without making the situation worse.

What must be designed explicitly ​

1. Clear retry policies ​

Not one policy for the whole system.

You need policies based on:

  • operation type
  • retry safety
  • subsystem criticality
  • expected latency
  • impact of duplicate action
  • allowed recovery window

2. Operation classification ​

Every important command should be classified like:

  • safe to retry automatically
  • retry only after verification
  • never auto-retry
  • reconnect and re-read only
  • operator intervention required on uncertainty

This is one of the highest-value design artifacts in an industrial system.

3. Bounded retries ​

Every retry path needs:

  • max attempts
  • delay strategy
  • escalation path
  • logging / diagnostics
  • ownership

4. Fault escalation strategy ​

Define:

  • when a communication issue becomes a subsystem fault
  • when subsystem fault becomes machine stop
  • what commands are inhibited
  • whether recovery is automatic, assisted, or manual
  • what reinitialization is required afterward

5. Separation of retry logic from business/workflow logic ​

Workflow code should say what it needs. Communication/reliability layers should decide how to handle transient communication issues.

Bad:

  • every workflow step manually wraps device calls in ad hoc retry loops

Good:

  • operation-specific resilience policy is owned centrally

  • workflow sees meaningful outcomes like:

    • Success
    • TransientFailureRecovered
    • PersistentFailure
    • UnsafeUncertainState
    • ReconnectRequired
    • OperatorInterventionRequired

Good vs bad approaches ​

Bad: blind retries everywhere ​

  • easy to add
  • feels robust at first
  • creates hidden coupling
  • duplicates commands
  • masks degradation
  • increases debugging difficulty
  • causes retry stacking across layers

Good: controlled, context-aware retry strategy ​

  • retry only where semantics allow
  • use per-operation timeout/retry policy
  • preserve state certainty
  • escalate explicitly on ambiguity
  • make recovery observable
  • keep unsafe commands out of generic retry wrappers

ASCII component diagram — reliability-aware communication layering ​

text
+------------------------------------------------------+
| Workflow / Machine Orchestrator                      |
| - sequence steps                                     |
| - respond to Success / Fault / InterventionRequired  |
+--------------------------+---------------------------+
                           |
                           v
+------------------------------------------------------+
| Operation Policy Layer                               |
| - classify command type                              |
| - timeout policy                                     |
| - retry policy                                       |
| - safe/unsafe retry decision                         |
| - escalation rules                                   |
+--------------------------+---------------------------+
                           |
                           v
+------------------------------------------------------+
| Communication / Device Adapter Layer                 |
| - send / receive                                     |
| - correlation                                        |
| - reconnect handling                                 |
| - protocol/session state                             |
+--------------------------+---------------------------+
                           |
                           v
+------------------------------------------------------+
| Device / PLC / Controller / Sensor                   |
+------------------------------------------------------+

Important architectural insight ​

Do not let business logic decide retry by catching raw timeout exceptions everywhere.

Why?

Because timeout alone does not tell you whether the operation is safe to repeat.

The retry decision belongs where operation semantics are known.

That is a very strong interview talking point.

PART 9 — INTERVIEW / REAL-WORLD TALKING POINTS ​

How to explain reliability and retries clearly ​

A strong explanation sounds like this:

In industrial machine software, communication failure is normal and often ambiguous. The key design problem is not just recovering from errors, but preserving safe and correct machine behavior when software is uncertain whether a command executed. That is why retry must be operation-aware, bounded, and tied to explicit fault escalation rather than applied blindly.

That is much stronger than saying, “we use retries and timeouts.”

Why “just retry” is dangerous ​

Because retries can:

  • duplicate physical actions
  • hide degradation
  • overload unstable devices
  • delay real fault handling
  • create conflicting state between software and machine
  • turn transient faults into systemic incidents

Common mistakes engineers make ​

  1. one timeout value for all operations
  2. generic retry wrapper around all device calls
  3. retrying non-idempotent commands
  4. stacking retries across layers
  5. failing to distinguish transport timeout from execution uncertainty
  6. no explicit degraded/fault state
  7. infinite retry loops
  8. poor logging of attempt count, latency, and final outcome
  9. no recovery model after reconnect
  10. assuming success if the system “eventually works”

What strong engineers understand ​

Strong engineers understand that safe fault handling is about state certainty.

They know:

  • a timeout is not just a timeout; it is uncertainty
  • retry is a semantic decision, not a utility function
  • read operations and action commands are different reliability classes
  • ambiguity after state-changing commands must be handled conservatively
  • escalation is part of normal design, not failure of design
  • logs must help answer: what was attempted, what was known, what remained uncertain, and why the system chose its recovery action

A concise interview answer ​

If asked, “How do you design retries in industrial systems?” a strong answer is:

I start by classifying operations into safe-to-retry and unsafe-to-retry. Then I define per-operation timeout and retry policies based on device behavior and physical risk. Reads and reconnects may get bounded retry with backoff. State-changing commands usually require idempotency guarantees, explicit confirmation, or verification before reissue. If uncertainty remains after retry budget is exhausted, I escalate to a controlled fault state rather than continuing blindly. The goal is not maximum automatic retry. The goal is safe, diagnosable, deterministic machine behavior.

Closing perspective ​

In industrial machine software, reliability is not about making failures disappear. It is about making failures contained, safe, explicit, and recoverable.

That is the mature mindset.

A weak system says:

  • communication failed, retry again

A strong system asks:

  • what probably happened physically?
  • what do we know?
  • what remains uncertain?
  • is retry safe?
  • when do we stop retrying?
  • what state should the machine enter now?

That is the difference between software that merely talks to devices and software that can responsibly control a machine.

This topic also fits the project’s high-priority emphasis on industrial reliability: machines fail in messy ways, and software must detect failures, fail safely, report clearly, and recover without making the situation worse.

If you want, I can continue with the next topic in the same style.

Docs-first project memory for AI-assisted implementation.

LCN Wafer Inspection Prototype