LCN Wafer Inspection

Appearance

Reliability – Safety and Production Readiness: Prompts ​

7.1 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • analyzing how industrial systems fail across hardware, software, and workflow layers
  • designing systems that continue operating safely under partial failure
  • debugging complex, cross-layer failures in production environments
  • building reliability models that guide architecture decisions

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand how failures are modeled in industrial systems and how reliability is designed at a system level.

=== TOPIC === Failure Modes & System Reliability Model

=== GOAL ===

Help me understand:

  • what can go wrong in industrial machine systems
  • how failures are categorized and modeled
  • how failures propagate across system layers
  • how engineers think about reliability before writing code

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Failure modes & system reliability model"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • system-level thinking
  • real-world failure behavior
  • architectural implications

Avoid:

  • generic reliability definitions
  • shallow “handle exceptions” advice
  • overly academic reliability theory

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Failure layer diagrams → where failures originate
  • Propagation diagrams → how failures spread
  • System boundary diagrams → containment zones

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • failure modeling
  • reliability thinking
  • system-level behavior

Do NOT deep dive into:

  • specific retry implementations (Topic 7.2)
  • logging details (Topic 7.8)
  • UI/UX topics

=== STRUCTURE ===

=== PART 1 — BIG PICTURE: WHY FAILURE MODELING COMES FIRST ===

Explain:

  • industrial systems are not designed for “success only”
  • they are designed for:
    • partial failure
    • degraded operation
    • safe shutdown
    • recoverability

Explain:

  • strong engineers ask: → “What will fail?” before “How do we build it?”

Use example:

  • camera disconnect during inspection
  • axis loses position mid-run
  • image processing pipeline overload

=== PART 2 — FAILURE CATEGORIES (LAYERED MODEL) ===

Explain common failure categories:

  1. Physical / mechanical failures
  2. Electrical / IO failures
  3. Device / hardware failures
  4. Communication failures
  5. Timing / synchronization failures
  6. Data / state inconsistency
  7. Software logic errors
  8. Resource exhaustion (CPU, memory, disk)
  9. Human/operator errors

Explain each with examples.

Include ASCII layered diagram: [Physical] [Device] [Communication] [Control] [Application] [UI]

=== PART 3 — FAILURE MODES (HOW THINGS FAIL) ===

Explain failure modes:

  • fail-stop (device stops responding)
  • fail-slow (latency increases)
  • fail-incorrect (wrong data)
  • intermittent failure
  • partial system failure
  • cascading failure

Explain:

  • why mode matters more than component

Use examples:

  • camera returns stale image
  • sensor flickers
  • buffer overflows slowly over time

=== PART 4 — FAILURE PROPAGATION ===

Explain:

  • failures rarely stay isolated
  • they propagate through layers

Example:

Camera → No Image → Processing Timeout → Workflow Stuck → UI Frozen → Operator Confused

Include ASCII propagation diagram.

Explain:

  • why local failure becomes system failure

=== PART 5 — FAILURE DETECTION VS FAILURE ASSUMPTION ===

Explain:

  • systems cannot rely only on detection
  • must assume failure will happen

Explain:

  • proactive vs reactive design
  • detection delays and blind spots

Examples:

  • watchdog needed because no event = possible failure
  • missing heartbeat = failure signal

=== PART 6 — RELIABILITY MODELING ===

Explain:

  • define reliability in terms of:
    • availability (uptime)
    • correctness
    • recoverability
    • safety

Explain:

  • system must answer:
    • what happens when X fails?
    • how fast can we detect it?
    • what is the safe state?
    • can we recover?

=== PART 7 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • intermittent camera disconnect only under load
  • system works in lab but fails in factory noise
  • memory leak causes failure after 3 days
  • race condition causes rare incorrect motion
  • wrong state causes unsafe command acceptance

For each:

  • what it looks like
  • why it's hard to detect
  • what layer actually caused it

=== PART 8 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • reliability must be designed upfront
  • importance of:
    • failure boundaries
    • subsystem isolation
    • timeout strategies
    • state validation
    • defensive design
    • observability hooks

Explain good vs bad:

  • bad: assume everything works, handle failure ad hoc
  • good: design for failure explicitly

=== PART 9 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain failure modeling clearly
  • why thinking in failure modes is critical
  • common mistakes engineers make
  • what strong engineers understand about propagation and system reliability

=== OUTPUT ===

  • structured explanation
  • real-world failure modeling insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.2 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • designing error handling strategies across UI, application, workflow, and device layers
  • controlling how faults propagate through the system and preventing cascading failures
  • implementing recovery strategies that bring machines back to safe, known states
  • debugging production systems where poor error handling caused system-wide instability or unsafe behavior

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand how industrial systems handle errors, propagate faults, and recover safely.

=== TOPIC === Error Handling, Fault Propagation & Recovery

=== GOAL ===

Help me understand:

  • how errors are handled across system layers
  • how faults propagate through the system
  • how recovery strategies are designed
  • how to prevent cascading failures

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Error handling, fault propagation & recovery"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • cross-layer behavior
  • system stability
  • recovery strategies

Avoid:

  • simple try/catch examples
  • generic exception handling advice
  • framework-specific details

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Propagation diagrams → error flow across layers
  • Containment diagrams → where errors should stop
  • Recovery flow diagrams → failure → safe state → recovery path

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • error handling strategy
  • fault propagation
  • recovery models

Do NOT deep dive into:

  • watchdogs (Topic 7.3)
  • logging/observability details (Topic 7.8)
  • UI alarm design (Domain 6)

=== STRUCTURE ===

=== PART 1 — WHY ERROR HANDLING IS NOT JUST TRY/CATCH ===

Explain:

  • in industrial systems, errors affect:
    • physical motion
    • machine state
    • workflow execution
    • operator safety
  • error handling is about:
    • controlling system behavior under failure
    • not just preventing crashes

Explain:

  • difference between:
    • catching an exception
    • handling a system fault

Use example:

  • exception thrown in vision pipeline vs machine must stop safely

=== PART 2 — ERROR VS FAULT VS FAILURE ===

Clarify terminology:

  • Error → something went wrong in code or data
  • Fault → system is in an abnormal condition
  • Failure → system cannot perform required function

Explain:

  • why clear terminology matters in architecture

=== PART 3 — FAULT PROPAGATION ACROSS LAYERS ===

Explain:

  • faults move across layers if not contained

Example:

Device error → control layer exception → workflow stuck → UI freeze → operator confusion

Include ASCII diagram:

[Device] → [Control] → [Workflow] → [UI]

Explain:

  • why propagation must be controlled

=== PART 4 — CONTAINMENT STRATEGY ===

Explain:

  • where faults should be handled:

  • Device layer → retry / reset / report

  • Control layer → isolate subsystem

  • Application layer → adjust workflow

  • UI layer → inform operator

Explain:

  • principle: → handle as low as possible, escalate only when needed

Include ASCII containment diagram

=== PART 5 — ERROR HANDLING STRATEGIES ===

Explain patterns:

  • fail-fast (stop immediately)
  • retry (transient issues)
  • fallback (alternate path)
  • degrade (reduced capability)
  • isolate (disable subsystem)

Explain when each is appropriate

Examples:

  • retry communication
  • fail-fast on unsafe motion
  • degrade vision inspection but continue handling

=== PART 6 — RECOVERY MODELS ===

Explain:

  • recovery is not automatic restart

Types:

  • local recovery (retry, reset subsystem)
  • workflow recovery (restart step)
  • operator-assisted recovery
  • full system restart

Explain:

  • importance of:
    • safe state
    • known state
    • consistency

Include ASCII recovery flow: Failure → Safe State → Recovery Action → Resume

=== PART 7 — AVOIDING CASCADING FAILURES ===

Explain:

  • one failure should not break entire system

Strategies:

  • isolation boundaries
  • timeouts
  • circuit breakers (conceptually)
  • queue limits
  • subsystem independence

Explain:

  • why cascading failures are common in poorly designed systems

=== PART 8 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • camera failure causes infinite retry loop → system freeze
  • processing error propagates to UI thread → crash
  • device timeout not handled → workflow stuck forever
  • recovery resets subsystem but state not synchronized
  • operator retries manually → worsens state inconsistency

For each:

  • what it looks like
  • why it happens
  • how engineers fix it

=== PART 9 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • need for structured error handling architecture

Important:

  • layered error handling policy
  • clear fault model
  • state-aware recovery
  • no hidden retries
  • no silent failures
  • consistent error reporting

Explain good vs bad:

  • bad: catch everywhere, ignore errors, retry blindly
  • good: explicit error strategy per subsystem, controlled propagation, safe recovery paths

Include ASCII component diagram: Subsystem → Error Handler → Recovery Strategy → Escalation → UI/Alarm

=== PART 10 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain error handling in industrial systems
  • difference between exception handling and fault handling
  • common mistakes engineers make
  • what strong engineers understand about containment and recovery

=== OUTPUT ===

  • structured explanation
  • real-world error handling insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.3 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • designing watchdog and heartbeat systems for long-running industrial applications
  • detecting stuck devices, frozen workflows, blocked pipelines, and unhealthy subsystems
  • distinguishing between slow, degraded, disconnected, and failed components
  • debugging production machines where failures were hidden because “nothing happened”

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand watchdogs, heartbeats, and health monitoring in industrial machine software.

=== TOPIC === Watchdogs, Heartbeats & Health Monitoring

=== GOAL ===

Help me understand how industrial systems detect unhealthy behavior before it becomes catastrophic.

Focus on:

  • watchdog patterns
  • heartbeat monitoring
  • subsystem health models
  • detecting stuck / frozen / degraded behavior
  • deciding when to warn, recover, or stop

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Watchdogs, heartbeats & health monitoring"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • real-world failure detection
  • long-running system reliability
  • system-level health modeling

Avoid:

  • generic server health check explanations
  • shallow “ping it periodically” advice
  • vendor-specific monitoring tools

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Monitoring diagrams → component → heartbeat → monitor
  • State diagrams → healthy / degraded / faulted
  • Timeline diagrams → expected heartbeat vs missed heartbeat
  • Recovery diagrams → detection → escalation → action

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • watchdogs
  • heartbeats
  • health monitoring
  • failure detection and escalation

Do NOT deep dive into:

  • generic observability/logging (Topic 7.8)
  • retry/recovery policy details (Topic 7.2)
  • UI alarm presentation (Domain 6)

=== STRUCTURE ===

=== PART 1 — WHY HEALTH MONITORING IS CRITICAL ===

Explain:

  • many failures do not announce themselves clearly
  • sometimes the problem is that an expected event never happens
  • industrial software must detect:
    • stuck workflows
    • frozen device callbacks
    • dead communication links
    • overloaded pipelines
    • stale sensor data
    • background service failures

Use examples:

  • camera acquisition stops producing frames
  • motion command never completes
  • PLC heartbeat stops updating
  • processing queue stops draining

Explain:

  • why “no error” does not mean “healthy.”

=== PART 2 — HEARTBEATS VS WATCHDOGS VS HEALTH CHECKS ===

Explain clearly:

  • heartbeat = periodic “I am alive” signal
  • watchdog = observer that expects progress within a time window
  • health check = explicit evaluation of whether a component is usable

Explain:

  • how they differ
  • how they work together
  • why heartbeat alone is not enough

Include ASCII concept diagram: Component → Heartbeat → Health Monitor → Watchdog Decision

=== PART 3 — WHAT SHOULD BE MONITORED ===

Explain practical monitoring targets:

  • device connectivity
  • command completion
  • workflow progress
  • queue depth / backlog
  • frame arrival rate
  • sensor freshness
  • background worker activity
  • UI responsiveness
  • storage availability
  • CPU/memory/disk pressure

For each:

  • what “healthy” means
  • what “unhealthy” looks like
  • what evidence is useful

=== PART 4 — HEALTH STATES AND ESCALATION ===

Explain a health model:

  • Healthy
  • Suspect
  • Degraded
  • Faulted
  • Recovering
  • Offline

Explain:

  • why binary “healthy/unhealthy” is too weak
  • how repeated minor issues should escalate
  • when degraded operation is acceptable

Include ASCII state diagram.

=== PART 5 — WATCHDOG TIME WINDOWS AND FALSE POSITIVES ===

Explain:

  • watchdogs need timing thresholds
  • thresholds must balance:
    • fast detection
    • avoiding false alarms

Explain:

  • why incorrect time windows cause:
    • noisy faults
    • missed failures
    • unnecessary stops

Use examples:

  • camera normally produces frame every 50ms, alert after 500ms
  • workflow step expected within 10s, fault after 30s

Include ASCII timeline diagram.

=== PART 6 — ACTIVE VS PASSIVE HEALTH MONITORING ===

Explain:

Active monitoring:

  • periodically asks component to prove health

Passive monitoring:

  • observes normal operational events

Explain:

  • trade-offs:
    • overhead
    • accuracy
    • false confidence

Examples:

  • active ping to PLC
  • passive frame-count monitoring from camera stream

=== PART 7 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • heartbeat still updates but device is functionally stuck
  • watchdog timeout too short causes false production stops
  • watchdog timeout too long delays safe recovery
  • queue backlog grows but health remains “green”
  • background worker dies silently
  • stale sensor value treated as current
  • health monitor itself becomes unreliable
  • reconnect resets heartbeat but device state remains invalid

For each:

  • what it looks like in production
  • why it happens
  • how experienced engineers diagnose and handle it

=== PART 8 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • health monitoring must be designed into architecture
  • importance of:
    • explicit health models
    • timestamps and freshness checks
    • progress-based watchdogs
    • functional health checks, not only connectivity
    • escalation policies
    • diagnostic evidence capture
    • separation between health detection and recovery action

Explain good vs bad approaches:

  • bad: single IsConnected flag, heartbeat-only monitoring, no queue/backlog visibility
  • good: layered health model, watchdogs for progress, freshness checks, trend-based escalation, clear health ownership

Include ASCII component diagram: Subsystem / Device / Worker ↓ heartbeat/progress/status Health Monitor ↓ health state Fault Manager / Recovery Policy ↓ Machine State / Alarm / Diagnostics

=== PART 9 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain watchdogs and heartbeats clearly
  • why “connected” is not equal to “healthy”
  • common mistakes software engineers make
  • what strong engineers understand about freshness, progress, false positives, and escalation

=== OUTPUT ===

  • structured explanation
  • real-world health monitoring insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.4 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • designing systems that recover safely after crash, power loss, communication failure, or abnormal shutdown
  • deciding what machine state should be persisted, reconstructed, discarded, or revalidated
  • handling partial workflow completion, uncertain physical state, and stale software assumptions
  • debugging systems where bad state restoration caused unsafe behavior, lost production context, or incorrect recovery

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand system state persistence and recovery in industrial machine software.

=== TOPIC === System State Persistence & Recovery

=== GOAL ===

Help me understand how industrial systems persist important state and recover safely after failures or restarts.

Focus on:

  • what state should and should not be persisted
  • recovering from crash or power loss
  • restoring production context safely
  • handling uncertain physical machine state
  • avoiding stale or dangerous state restoration

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"System state persistence & recovery"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • physical-machine reality
  • safe recovery
  • state correctness
  • real-world production behavior

Avoid:

  • generic database persistence theory
  • shallow “save state to disk” advice
  • assuming software state equals physical state

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • State diagrams → persisted / volatile / uncertain state
  • Recovery flow diagrams → restart → validate → recover
  • Context diagrams → machine state vs workflow state vs production state
  • Failure timeline diagrams → last known state vs current physical state

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • state persistence
  • recovery after restart/failure
  • safe restoration of machine context

Do NOT deep dive into:

  • graceful shutdown mechanics (Topic 7.5)
  • deployment/version migration (Topic 7.10)
  • database schema design

=== STRUCTURE ===

=== PART 1 — WHY STATE RECOVERY IS HARD IN MACHINE SOFTWARE ===

Explain:

  • after restart, software may remember one thing, but the physical machine may be in another condition
  • industrial systems involve physical state that cannot always be trusted from persisted software data
  • recovery must answer:
    • what was happening?
    • what is physically true now?
    • what can be safely resumed?
    • what must be revalidated?

Use examples:

  • machine crashed while wafer was clamped
  • robot picked a part but did not place it
  • motion axis position was stored before power loss but encoder/reference is now invalid
  • inspection result was computed but not reported

=== PART 2 — TYPES OF STATE IN INDUSTRIAL SYSTEMS ===

Explain practical categories:

  1. Persistent production context

    • lot/job/run ID
    • product/wafer/part identity
    • recipe/version

  2. Workflow state

    • current operation
    • current step
    • completed steps

  3. Machine physical state

    • axis position
    • clamp/vacuum state
    • part present/not present

  4. Device state

    • connected/ready/faulted
    • initialized/configured

  5. Transient runtime state

    • in-memory queues
    • pending commands
    • callbacks/subscriptions

Explain:

  • which state is safe to persist
  • which state must be reconstructed
  • which state must be treated as unknown after restart

Include ASCII context diagram.

=== PART 3 — PERSISTED STATE VS TRUSTED STATE ===

Explain:

  • persisted state is only what software last recorded
  • trusted state is what the system has validated after restart
  • these are not the same

Explain:

  • why persisted values should often become:
    • “last known”
    • “requires validation”
    • “unsafe to assume”

Examples:

  • last known axis position
  • last active recipe
  • last workflow step
  • last known vacuum state

Include ASCII diagram: Persisted State → Validation → Trusted Current State / Unknown State

=== PART 4 — RECOVERY AFTER CRASH OR POWER LOSS ===

Explain a safe recovery flow:

  1. Restart application
  2. Load persisted context
  3. Reconnect devices
  4. Validate hardware identity/config
  5. Re-establish machine physical state
  6. Determine workflow recovery point
  7. Require operator/service confirmation if needed
  8. Resume, rollback, or abort safely

Explain:

  • why automatic resume is often unsafe
  • why recovery may require homing, inspection, sensor checks, or manual confirmation

Include ASCII recovery flow diagram.

=== PART 5 — WORKFLOW RECOVERY AND PARTIAL COMPLETION ===

Explain:

  • workflows may fail mid-step
  • partial completion is common

Examples:

  • material loaded but not inspected
  • image captured but result not stored
  • motion completed but sensor confirmation missing
  • actuator moved but state not verified

Explain:

  • recovery options:
    • resume from known safe checkpoint
    • repeat step
    • rollback
    • move to recovery workflow
    • require operator intervention

Explain:

  • why recovery checkpoints must be designed, not guessed later

=== PART 6 — PRODUCTION CONTEXT RECOVERY ===

Explain:

  • production systems must preserve:
    • current lot/job/run
    • active recipe/version
    • item identity
    • inspection/result status
    • report/export state

Explain:

  • risks:
    • duplicate result reporting
    • lost traceability
    • wrong product context
    • mismatched recipe after restart

Explain:

  • why idempotency and status markers matter for production records

=== PART 7 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • software restores “Running” state after restart even though machine is physically stopped
  • last known position is used after homing reference is lost
  • workflow resumes after a step that actually only partially completed
  • product is processed twice because completion was not recorded atomically
  • result is lost because image saved but database record failed
  • operator restarts app and UI shows ready while device initialization is incomplete
  • stale recipe/config context restored after hardware change

For each:

  • what it looks like in production
  • why it happens
  • how experienced engineers prevent or diagnose it

=== PART 8 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • state persistence must be designed with physical validation
  • importance of:
    • explicit state categories
    • recovery checkpoints
    • “unknown” state representation
    • validation before trust
    • persisted context versioning
    • atomic updates for production records
    • idempotent reporting where possible
    • operator-guided recovery flows
    • clear separation between last-known state and current verified state

Explain good vs bad approaches:

  • bad: persist entire object graph and restore blindly
  • bad: assume last software state equals current machine state
  • good: persist minimal recovery context, validate physical state, resume only from safe checkpoints
  • good: expose clear recovery state to operator/service engineer

Include ASCII component diagram: Persistence Store ↓ Recovery Manager ↓ Device Validation + Physical State Checks ↓ Workflow Recovery Decision ↓ Operator Guidance / Safe Resume / Abort

=== PART 9 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain state persistence and recovery clearly
  • why physical state cannot be blindly restored from software state
  • common mistakes software engineers make when entering industrial systems
  • what strong engineers understand about checkpoints, unknown state, validation, and safe recovery

=== OUTPUT ===

  • structured explanation
  • real-world state persistence and recovery insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.5 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • designing machine software that shuts down safely under normal and abnormal conditions
  • handling crashes, unhandled exceptions, process termination, and power-loss scenarios
  • coordinating shutdown across UI, workflows, devices, motion, storage, and diagnostics
  • debugging systems where poor shutdown handling left hardware, data, or workflow state inconsistent

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand crash handling and graceful shutdown in industrial machine software.

=== TOPIC === Crash Handling & Graceful Shutdown

=== GOAL ===

Help me understand how industrial systems shut down safely and handle crashes without leaving the machine in a dangerous or inconsistent state.

Focus on:

  • graceful shutdown flow
  • abnormal termination
  • device/resource cleanup
  • safe stopping of workflows and motion
  • crash evidence preservation
  • restart readiness

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Crash handling & graceful shutdown"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • physical-machine consequences
  • resource lifecycle
  • failure containment
  • production recovery

Avoid:

  • generic application shutdown advice
  • shallow “dispose objects” guidance
  • assuming shutdown is only a software lifecycle event

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Shutdown sequence diagrams → orderly component shutdown
  • State diagrams → running / stopping / stopped / crashed / recovering
  • Resource lifecycle diagrams → device handles, buffers, subscriptions, files
  • Failure flow diagrams → crash → evidence capture → safe recovery

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • crash handling
  • graceful shutdown
  • resource cleanup
  • safe termination and restart readiness

Do NOT deep dive into:

  • state persistence and workflow recovery already covered in Topic 7.4
  • deployment lifecycle management (Topic 7.10)
  • full observability architecture (Topic 7.8)

=== STRUCTURE ===

=== PART 1 — WHY SHUTDOWN IS SAFETY-CRITICAL IN MACHINE SOFTWARE ===

Explain:

  • shutdown is not just closing a desktop app
  • the software may be controlling:
    • motion
    • cameras
    • IO outputs
    • vacuum
    • clamps
    • lasers/lights
    • active workflows
    • storage pipelines
  • if shutdown is poorly handled, the machine may be left in:
    • unknown state
    • unsafe state
    • resource-locked state
    • data-incomplete state

Use examples:

  • camera SDK handle not released, next startup cannot acquire
  • motion command active when app exits
  • vacuum/clamp left active with material inside
  • result written to image store but not database

=== PART 2 — NORMAL SHUTDOWN VS ABNORMAL TERMINATION ===

Explain clearly:

Normal shutdown:

  • operator/system requests controlled stop
  • workflows are stopped or completed safely
  • devices are disarmed/released
  • state and logs are flushed

Abnormal termination:

  • crash
  • unhandled exception
  • power loss
  • OS kill
  • watchdog termination
  • native SDK crash

Explain:

  • why the system must design for both
  • what can and cannot be guaranteed in each case

Include ASCII state diagram: Running → Stopping → Stopped Running → Crashed → Recovery Required

=== PART 3 — GRACEFUL SHUTDOWN SEQUENCE ===

Explain a realistic shutdown sequence:

  1. Stop accepting new commands
  2. Notify UI/operator that shutdown is in progress
  3. Request workflow stop/cancel
  4. Stop or park motion where appropriate
  5. Stop acquisition/streaming
  6. Deactivate outputs safely
  7. Flush storage/logs/diagnostics
  8. Release device resources
  9. Persist shutdown marker/context
  10. Confirm stopped state

Explain:

  • why order matters
  • why dependencies between subsystems matter

Include ASCII sequence diagram.

=== PART 4 — SAFE STOPPING OF ACTIVE OPERATIONS ===

Explain:

  • shutdown may occur while work is active
  • active operations may include:
    • motion in progress
    • image acquisition
    • processing pipeline
    • device command pending
    • storage write
    • operator command executing

Explain:

  • difference between:
    • cancel
    • stop at safe boundary
    • abort immediately
    • emergency stop handled by safety system

Explain:

  • why graceful shutdown should avoid leaving partial actions hidden.

=== PART 5 — RESOURCE CLEANUP AND RELEASE ===

Explain resources that need explicit cleanup:

  • native SDK handles
  • unmanaged buffers
  • camera/frame grabber acquisition buffers
  • serial/TCP connections
  • file/database handles
  • event subscriptions/callbacks
  • background workers/timers
  • device locks/ownership

Explain:

  • why long-running machine apps often fail on next startup because previous shutdown leaked resources.

Include ASCII resource lifecycle diagram.

=== PART 6 — CRASH HANDLING AND EVIDENCE PRESERVATION ===

Explain:

  • during crash, the system may have limited ability to recover
  • priority should be:
    1. preserve diagnostic evidence
    2. avoid making physical state worse
    3. mark state as uncertain
    4. require controlled restart/recovery

Explain evidence to preserve:

  • exception/crash dump
  • current workflow step
  • active command
  • machine state snapshot
  • device health
  • last events/logs
  • pending storage/reporting operations

Explain:

  • why clearing/retrying too early can destroy evidence

=== PART 7 — RESTART READINESS AFTER SHUTDOWN OR CRASH ===

Explain:

  • after shutdown/crash, next startup must not assume everything is clean
  • system should detect:
    • previous shutdown was clean or abnormal
    • devices may be locked or uncertain
    • workflows may be incomplete
    • production context may require recovery

Explain:

  • why startup and shutdown design are connected

=== PART 8 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • app exits while motion controller still executing command
  • acquisition not stopped before camera handle is released
  • native SDK crash prevents normal cleanup
  • UI closes but background worker continues using device
  • storage queue loses inspection results during shutdown
  • shutdown hangs forever waiting for device response
  • previous crash leaves machine in unknown physical state but UI starts as Ready
  • operator kills app to recover, making evidence disappear

For each:

  • what it looks like in production
  • why it happens
  • how experienced engineers prevent or diagnose it

=== PART 9 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • graceful shutdown must be an explicit architecture path
  • importance of:
    • shutdown coordinator
    • ordered subsystem shutdown
    • cancellation-aware workflows
    • bounded shutdown timeouts
    • safe output/device deactivation
    • resource ownership tracking
    • crash markers and startup recovery checks
    • evidence preservation before cleanup
    • clear distinction between graceful stop and crash recovery

Explain good vs bad approaches:

  • bad: rely on process exit, random Dispose calls, UI close event doing everything, infinite wait during shutdown
  • good: central shutdown coordinator, ordered stop contracts, timeout-aware cleanup, abnormal shutdown detection, recovery-required state on restart

Include ASCII component diagram: Shutdown Request / Crash Detector ↓ Shutdown Coordinator ↓ Workflow Stop + Device Disarm + Storage Flush + Diagnostics Capture ↓ Clean Shutdown Marker / Recovery Required Marker

=== PART 10 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain graceful shutdown in industrial software clearly
  • why shutdown is part of machine safety and reliability
  • common mistakes software engineers make when entering industrial systems
  • what strong engineers understand about ordered shutdown, crash evidence, resource cleanup, and restart readiness

=== OUTPUT ===

  • structured explanation
  • real-world crash handling and shutdown insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.6 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • designing software that respects safety interlocks and fail-safe machine behavior
  • integrating guarded doors, light curtains, estops, safety PLCs, motion inhibits, and permissives into machine software
  • preventing unsafe commands even when workflow logic, UI, or device state is incorrect
  • debugging systems where weak interlock modeling caused unsafe behavior, false stops, or recovery confusion

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand safety interlocks and fail-safe behavior from a SOFTWARE ARCHITECTURE perspective.

=== TOPIC === Safety Interlocks & Fail-Safe Behavior

=== GOAL ===

Help me understand how industrial software models, respects, and reacts to safety interlocks and fail-safe conditions.

Focus on:

  • safety interlocks
  • permissives and inhibits
  • fail-safe design
  • safety-related state modeling
  • software boundaries around safety logic

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Safety interlocks & fail-safe behavior"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • practical machine safety behavior
  • software architecture boundaries
  • real-world failure modes

Avoid:

  • formal safety certification deep dive
  • legal/compliance explanation
  • unsafe bypass-oriented advice

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Boundary diagrams → safety system vs application software
  • Interlock flow diagrams → condition → inhibit/permissive → command decision
  • State diagrams → safe / inhibited / faulted / recoverable states
  • Command gating diagrams → UI/workflow command through safety checks

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • interlocks
  • permissives/inhibits
  • fail-safe behavior
  • software architecture around safety constraints

Do NOT deep dive into:

  • emergency stop mechanics in detail (Topic 7.7)
  • HMI alarm presentation
  • formal standards/certification

=== STRUCTURE ===

=== PART 1 — WHY SAFETY INTERLOCKS MATTER ===

Explain:

  • machines contain physical hazards:
    • moving axes
    • robots
    • clamps
    • vacuum
    • lasers/lights
    • high voltage
    • heated or pressurized systems
  • software must never assume normal flow is always safe
  • safety interlocks prevent actions when required conditions are not satisfied

Use examples:

  • guard door open → inhibit motion
  • vacuum not confirmed → do not release wafer
  • light curtain interrupted → block robot movement
  • safety PLC reports unsafe state → application must not start workflow

Explain:

  • why interlocks are not “optional validations”
  • they are part of machine behavior.

=== PART 2 — INTERLOCKS, PERMISSIVES, INHIBITS, AND FAIL-SAFE ===

Explain clearly:

  • interlock = condition that prevents or stops an unsafe action
  • permissive = condition required before action is allowed
  • inhibit = active block preventing a command/operation
  • fail-safe = system moves toward safest reasonable state when information/control is lost

Explain:

  • why these concepts must be modeled explicitly
  • why confusing them creates bad recovery behavior

Include ASCII concept diagram: Condition → Permissive / Inhibit → Command Allowed or Rejected

=== PART 3 — SOFTWARE VS SAFETY SYSTEM RESPONSIBILITY ===

Explain:

  • not all safety should depend on normal application software
  • safety-critical enforcement may belong to:
    • safety PLC
    • safety relay
    • motion drive safety functions
    • hardwired circuits

Explain software responsibility:

  • observe safety state
  • respect inhibits
  • prevent unsafe command requests
  • guide operator recovery
  • record safety-related context
  • never bypass safety layer

Include ASCII boundary diagram: HMI / Workflow App ↓ requests Machine Control ↓ commands Device Layer ↓ Hardware ↑ Safety PLC / Safety Circuit independently inhibits dangerous action

=== PART 4 — COMMAND GATING WITH INTERLOCKS ===

Explain:

  • before executing commands, system should check:
    • current machine state
    • operating mode
    • user role where relevant
    • interlock state
    • permissives
    • device readiness
    • resource ownership

Explain:

  • UI disablement is not enough
  • backend command gateway must enforce safety rules

Include ASCII command gating flow: Command Intent → Validation → Interlock Check → Allow / Reject / Fault

=== PART 5 — FAIL-SAFE BEHAVIOR UNDER UNCERTAINTY ===

Explain:

  • if safety state is unknown, stale, or invalid, treat as unsafe
  • examples:
    • lost safety PLC connection
    • stale door status
    • invalid sensor reading
    • missing vacuum confirmation

Explain:

  • fail-safe does not always mean “stop everything instantly”
  • it means choose the safest defined response for that condition:
    • inhibit new commands
    • stop workflow at safe boundary
    • de-energize output
    • require operator intervention
    • escalate fault

=== PART 6 — INTERLOCK STATE MODELING ===

Explain practical states:

  • Safe / permissive satisfied
  • Inhibited
  • Unsafe condition active
  • Unknown / stale
  • Recovering
  • Faulted

Explain:

  • why unknown is different from safe
  • why acknowledged is different from resolved
  • why recovery may require revalidation

Include ASCII state diagram.

=== PART 7 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • UI allows motion because interlock state was stale
  • safety signal flickers and causes nuisance stops
  • software clears fault but physical interlock is still active
  • manual/service mode bypasses checks incorrectly
  • interlock checked in one command path but not another
  • safety PLC inhibits motion but app thinks command succeeded
  • unknown safety state treated as safe
  • operator repeatedly resets without resolving root cause

For each:

  • what it looks like in production
  • why it happens
  • how experienced engineers prevent or diagnose it

=== PART 8 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • safety-related constraints must be first-class architecture concepts
  • importance of:
    • centralized command gating
    • explicit interlock model
    • fail-closed behavior
    • freshness/timestamp checks for safety-visible state
    • separation between safety enforcement and application convenience
    • consistent rejection reasons
    • traceability of safety-related commands and transitions
    • recovery flows that revalidate physical conditions

Explain good vs bad approaches:

  • bad: scattered boolean checks, UI-only disablement, service-mode bypass, treating missing signal as safe
  • good: central safety/interlock service, backend enforcement, unknown-as-unsafe policy, independent hardware safety boundaries, explicit recovery validation

Include ASCII component diagram: UI / Workflow / Service Tool ↓ Command Intent Command Gateway ↓ Safety / Interlock Service ↓ Machine Controller ↓ Device Layer ↑ Safety State / Permissives / Inhibits

=== PART 9 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain interlocks and fail-safe behavior clearly
  • why normal application software should not be the only safety layer
  • why unknown/stale safety state must not be treated as safe
  • common mistakes software engineers make when entering industrial systems
  • what strong engineers understand about command gating, permissives, inhibits, and recovery validation

=== OUTPUT ===

  • structured explanation
  • real-world safety interlock and fail-safe insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.7 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • integrating emergency stop behavior with machine software and operator UI
  • designing software that correctly reacts to safety-critical machine states
  • separating application-level stop/abort behavior from true safety stop behavior
  • debugging systems where emergency stop handling caused confusing recovery, stale state, or unsafe assumptions

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand emergency stop and safety-critical handling from a SOFTWARE ARCHITECTURE perspective.

=== TOPIC === Emergency Stop & Safety-Critical Handling

=== GOAL ===

Help me understand how industrial software should interact with emergency stop and safety-critical conditions.

Focus on:

  • what emergency stop means in machine systems
  • software responsibility vs safety hardware responsibility
  • application stop/abort vs emergency stop
  • state handling after safety-critical events
  • recovery after emergency stop

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Emergency stop & safety-critical handling"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • practical machine behavior
  • software boundaries
  • recovery and state correctness

Avoid:

  • formal safety certification deep dive
  • unsafe bypass advice
  • shallow “stop the machine” explanations

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Boundary diagrams → safety circuit vs application software
  • State diagrams → normal / estop active / safe state / recovery
  • Sequence diagrams → estop event detection and software response
  • Recovery flow diagrams → reset, revalidate, resume/abort

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • emergency stop
  • safety-critical state handling
  • software reaction and recovery

Do NOT deep dive into:

  • general interlocks already covered in Topic 7.6
  • UI alarm design
  • formal safety standards

=== STRUCTURE ===

=== PART 1 — WHAT EMERGENCY STOP REALLY MEANS ===

Explain:

  • emergency stop is not a normal software command
  • it is a safety-critical mechanism intended to bring hazardous motion/energy to a safe condition
  • in real machines, emergency stop is usually enforced by:
    • safety relay
    • safety PLC
    • drive safety functions
    • hardwired circuits
  • application software observes and reacts, but should not be the only thing enforcing it

Use examples:

  • operator presses physical E-stop button
  • safety circuit cuts drive enable
  • motion controller reports safety stop active

=== PART 2 — EMERGENCY STOP VS STOP / ABORT / PAUSE ===

Explain clearly:

  • Pause: controlled temporary suspension
  • Stop: controlled stop at safe boundary
  • Abort: more aggressive interruption of workflow
  • Emergency Stop: safety-critical hardware-level intervention

Explain:

  • why confusing these concepts causes bad system behavior
  • why E-stop recovery is different from normal resume

Include ASCII comparison diagram.

=== PART 3 — SOFTWARE RESPONSIBILITY DURING E-STOP ===

Explain software should:

  • detect/observe safety state
  • stop issuing new commands
  • mark machine state as safety-stopped / unsafe-to-run
  • cancel or invalidate active workflows
  • record context and diagnostic evidence
  • inform operator clearly
  • require revalidation before recovery

Explain software should NOT:

  • assume it can “resume where it left off”
  • hide the event as a normal stop
  • automatically clear safety condition
  • treat drive-disabled state as normal idle

=== PART 4 — SAFETY HARDWARE VS APPLICATION SOFTWARE BOUNDARY ===

Explain:

  • safety hardware owns immediate hazardous-energy control
  • application software owns:
    • coordination
    • state model
    • operator guidance
    • recovery flow
    • traceability

Include ASCII boundary diagram:

Operator E-Stop Button ↓ Safety Relay / Safety PLC / Drive STO ↓ physically disables hazardous action Machine Hardware

Application Software ↑ observes safety state ↓ blocks commands / updates state / guides recovery

=== PART 5 — STATE MODEL AFTER E-STOP ===

Explain:

  • after E-stop, machine state is not simply “Stopped”
  • important states may include:
    • EmergencyStopActive
    • SafetyCircuitOpen
    • MotionPowerDisabled
    • UnknownPosition
    • WorkflowInvalidated
    • RecoveryRequired

Explain:

  • why physical state may be uncertain
  • why axes, clamps, vacuum, part presence, and workflow context may need revalidation

Include ASCII state diagram: Running → EStopActive → SafetyReset → Revalidate → Ready / RecoveryRequired

=== PART 6 — RECOVERY AFTER EMERGENCY STOP ===

Explain safe recovery flow:

  1. E-stop condition physically resolved
  2. Safety circuit reset
  3. Software observes safety state cleared
  4. Machine remains not-ready until validation completes
  5. Reconnect/re-enable affected devices
  6. Revalidate axes, positions, IO, part/material state
  7. Decide whether workflow can resume, abort, or requires manual recovery
  8. Operator confirms guided recovery path

Explain:

  • why recovery must be explicit
  • why automatic resume is usually unsafe

Include ASCII recovery flow diagram.

=== PART 7 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • UI shows Idle after E-stop even though drives are disabled
  • software tries to resume workflow after E-stop without revalidation
  • E-stop clears physically but app state remains stuck
  • app clears alarm but safety circuit still open
  • active command times out and is misclassified as normal device failure
  • position is trusted after drive power loss
  • operator thinks Stop and E-stop are equivalent
  • diagnostic evidence is lost during reset

For each:

  • what it looks like in production
  • why it happens
  • how experienced engineers prevent or diagnose it

=== PART 8 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • E-stop handling must be a first-class state path
  • importance of:
    • explicit safety-stopped state
    • central command blocking after E-stop
    • invalidating active workflow context
    • physical-state revalidation
    • clear distinction between safety reset and machine ready
    • traceable event history
    • guided recovery sequence
    • no automatic resume without validation

Explain good vs bad approaches:

  • bad: model E-stop as normal Stop, clear UI alarm and resume, trust last software state
  • good: model E-stop separately, block commands, mark state uncertain, revalidate hardware, guide recovery

Include ASCII component diagram: Safety State Input ↓ Safety State Monitor ↓ Machine State Manager ↓ Command Gateway / Workflow Manager / HMI Guidance ↓ Recovery Procedure

=== PART 9 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain emergency stop handling clearly
  • why E-stop is not a normal software stop command
  • why safety reset is not the same as machine ready
  • common mistakes software engineers make when entering industrial systems
  • what strong engineers understand about hardware safety boundaries, uncertain state, and recovery validation

=== OUTPUT ===

  • structured explanation
  • real-world emergency stop and safety-critical handling insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.8 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • designing observability for long-running machine software
  • building logs, metrics, traces, diagnostic snapshots, and fault evidence for production debugging
  • helping field engineers diagnose failures without needing the original developer present
  • debugging systems where poor observability made root cause analysis slow, speculative, or impossible

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand observability, logging, metrics, and diagnostics in industrial machine software.

=== TOPIC === Observability: Logging, Metrics & Diagnostics

=== GOAL ===

Help me understand how industrial systems expose enough information to diagnose failures, understand behavior, and support production machines.

Focus on:

  • structured logging
  • metrics and counters
  • diagnostic snapshots
  • event/fault history
  • root-cause-oriented diagnostics
  • field support and serviceability

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Observability: logging, metrics & diagnostics"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • production diagnosis
  • cross-layer visibility
  • practical serviceability
  • long-running machine behavior

Avoid:

  • generic cloud observability advice
  • shallow “add logs” guidance
  • tool-specific tutorials

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Layer diagrams → diagnostic visibility across UI, workflow, device, hardware boundaries
  • Timeline diagrams → reconstructing fault sequence
  • Data-flow diagrams → logs, metrics, snapshots, fault records
  • Evidence package diagrams → what is captured at failure time

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • observability
  • logging
  • metrics
  • diagnostics
  • root cause analysis support

Do NOT deep dive into:

  • HMI alarm presentation
  • deployment/monitoring infrastructure
  • cybersecurity logging/compliance

=== STRUCTURE ===

=== PART 1 — WHY OBSERVABILITY IS CRITICAL IN MACHINE SOFTWARE ===

Explain:

  • industrial machine failures are often:
    • intermittent
    • timing-sensitive
    • cross-layer
    • hard to reproduce
    • site/environment-specific
  • the visible symptom is often far from the root cause

Use examples:

  • UI shows motion timeout, but root cause is stale interlock signal
  • inspection fails, but root cause is image quality drift
  • device reconnect succeeds, but command state remains inconsistent

Explain:

  • why observability must help engineers answer:
    • what happened?
    • when?
    • in what order?
    • under what machine state?
    • which subsystem originated the problem?
    • what changed before failure?

=== PART 2 — LOGGING IS NOT ENOUGH ===

Explain:

  • logs are one form of evidence, not the whole observability system
  • industrial diagnostics also need:
    • state transitions
    • command traces
    • device communication traces
    • metrics/counters
    • diagnostic snapshots
    • alarm/fault history
    • image/result evidence where relevant

Explain:

  • why plain string logs without context are weak

=== PART 3 — STRUCTURED LOGGING ACROSS LAYERS ===

Explain:

  • logs should be structured and contextual
  • useful fields:
    • timestamp
    • subsystem
    • operation/correlation ID
    • machine state
    • workflow step
    • device ID
    • command ID
    • result/status
    • error/fault code

Explain:

  • why layer-aware logging matters:
    • UI/operator action
    • workflow transition
    • command dispatch
    • device response
    • hardware/status event

Include ASCII layer trace diagram.

=== PART 4 — METRICS, COUNTERS, AND HEALTH INDICATORS ===

Explain practical metrics:

  • command latency
  • timeout count
  • retry count
  • queue depth
  • dropped frames/messages
  • CPU/memory/disk usage
  • device reconnect count
  • workflow cycle time
  • alarm frequency
  • processing stage duration

Explain:

  • metrics reveal trends and degradation that logs may miss
  • metrics help distinguish one-off failure from systemic degradation

=== PART 5 — DIAGNOSTIC SNAPSHOTS AND EVIDENCE PACKAGES ===

Explain:

  • snapshot = captured system context at important moment
  • evidence package may include:
    • active workflow step
    • machine state snapshot
    • device health states
    • active alarms
    • current recipe/config version
    • recent command/event history
    • queue/backlog state
    • relevant image/frame/result references
    • exception/crash details

Explain:

  • why evidence should be captured before reset/recovery destroys context

Include ASCII evidence package diagram.

=== PART 6 — TIMELINE AND CORRELATION ===

Explain:

  • root cause analysis often requires reconstructing event order
  • one failure may involve:
    • operator action
    • command validation
    • workflow state change
    • device command
    • timeout
    • alarm
    • recovery attempt

Explain:

  • importance of:
    • correlation IDs
    • monotonic event sequencing
    • consistent timestamps
    • command/result pairing

Include ASCII timeline diagram.

=== PART 7 — OPERATOR-VISIBLE VS ENGINEER DIAGNOSTICS ===

Explain:

  • operators need:

    • clear fault summary
    • required action
    • current blocking condition

  • engineers/service need:

    • raw details
    • traces
    • device status
    • configuration
    • timing/counter data

Explain:

  • why mixing these views causes confusion
  • why diagnostic depth should be role/context aware

=== PART 8 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • logs say “operation failed” but not which subsystem caused it
  • no correlation between operator action and device command
  • timestamp mismatch makes event order unclear
  • fault cleared before evidence captured
  • intermittent failure cannot be reproduced because diagnostic snapshot missing
  • field machine has different config/version but logs do not include it
  • performance degrades slowly but no metrics reveal trend
  • service engineer cannot export useful diagnostic bundle

For each:

  • what it looks like in production
  • why it happens
  • how experienced engineers improve the design

=== PART 9 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • observability must be built into architecture from the start
  • importance of:
    • structured logging contracts
    • correlation/context propagation
    • diagnostic snapshot service
    • metrics collection
    • event/fault journaling
    • exportable diagnostic bundles
    • retention policy
    • field-service-friendly tooling
    • preserving evidence before recovery

Explain good vs bad approaches:

  • bad: scattered string logs, no correlation, generic errors, no metrics, no diagnostic export
  • good: cross-layer traceability, structured evidence, counters, snapshots, and diagnostic workflows designed for root cause analysis

Include ASCII component diagram: UI / Workflow / Device / Vision / Storage ↓ structured events + metrics + snapshots Observability Pipeline ↓ Logs + Metrics + Fault History + Diagnostic Bundle ↓ Engineer / Field Service / Root Cause Analysis

=== PART 10 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain observability in industrial systems clearly
  • why “add more logs” is not enough
  • common mistakes software engineers make when entering industrial systems
  • what strong engineers understand about correlation, evidence, metrics, and field serviceability

=== OUTPUT ===

  • structured explanation
  • real-world observability and diagnostics insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.9 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • designing production monitoring for machines running in factory environments
  • exposing machine health, throughput, alarms, degradation, and system performance to operators and support teams
  • distinguishing local HMI alarms from broader production monitoring and alerting
  • debugging systems where weak monitoring allowed problems to grow unnoticed until downtime or quality loss occurred

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand production monitoring and alerting in industrial machine software.

=== TOPIC === Production Monitoring & Alerting

=== GOAL ===

Help me understand how industrial systems monitor production behavior, detect degradation, and alert the right people before problems become serious.

Focus on:

  • machine health monitoring
  • production performance monitoring
  • alerting strategy
  • degradation detection
  • local vs remote/factory-level visibility
  • avoiding noisy or useless alerts

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Production monitoring & alerting"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • real factory operations
  • long-running production behavior
  • actionable monitoring
  • practical system design

Avoid:

  • generic cloud monitoring advice
  • shallow dashboard examples
  • repeating HMI alarm design from Domain 6

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Monitoring flow diagrams → machine → metrics → alerts → action
  • Layer diagrams → local HMI vs factory monitoring vs service monitoring
  • Alert lifecycle diagrams → signal → condition → alert → acknowledgement/resolution
  • Trend diagrams → degradation over time

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • production monitoring
  • alerting
  • degradation detection
  • factory/support visibility

Do NOT deep dive into:

  • HMI alarm panel design
  • low-level logging internals
  • deployment infrastructure

=== STRUCTURE ===

=== PART 1 — WHY PRODUCTION MONITORING IS DIFFERENT FROM LOGGING ===

Explain:

  • logging helps diagnose what happened
  • production monitoring helps detect what is happening and whether action is needed
  • monitoring is about trends, health, performance, and operational risk

Explain:

  • why machines may appear “running” while slowly degrading:
    • increasing cycle time
    • more retries
    • more false defects
    • more reconnects
    • growing queue depth
    • higher resource usage

Use examples:

  • camera reconnect count slowly rising before failure
  • inspection throughput dropping due to processing backlog
  • vacuum sensor recovery time increasing over shift

=== PART 2 — WHAT SHOULD BE MONITORED IN PRODUCTION ===

Explain key monitoring categories:

  • availability / uptime
  • machine state distribution
  • alarms and fault frequency
  • cycle time and throughput
  • device health and reconnect counts
  • retry/timeout rates
  • queue depths and backlog
  • resource usage: CPU, memory, disk
  • image/inspection quality metrics where relevant
  • storage capacity and write failures
  • recipe/config/version context

For each:

  • what it tells engineers/operators
  • what degradation may look like

=== PART 3 — LOCAL HMI ALARMS VS PRODUCTION ALERTING ===

Explain clearly:

Local HMI alarms:

  • immediate operator action
  • machine-specific
  • visible at the machine

Production/factory alerts:

  • trend/degradation/system-level visibility
  • may notify engineering, maintenance, supervisors
  • may aggregate across machines

Explain:

  • why they are related but not the same
  • why not every alarm should become a remote alert

Include ASCII layer diagram: Machine Alarm → Local HMI Machine Metrics → Monitoring System → Alert / Trend / Report

=== PART 4 — ALERT CONDITIONS AND THRESHOLDS ===

Explain:

  • alerts should be based on meaningful conditions, not raw noise
  • examples:
    • error rate exceeds threshold
    • retry count increasing
    • queue depth above safe range
    • disk below capacity threshold
    • cycle time drift
    • repeated transient faults

Explain:

  • static thresholds vs trend-based alerts
  • alert severity levels
  • alert hysteresis / suppression to avoid flapping

Include ASCII alert lifecycle diagram.

=== PART 5 — DEGRADATION DETECTION ===

Explain:

  • many failures are preceded by weaker signals
  • degradation may appear as:
    • slower response
    • more retries
    • more operator interventions
    • rising temperature/resource usage
    • increased false defect rate
    • reduced throughput

Explain:

  • why degradation detection is often more valuable than detecting total failure

Include ASCII trend diagram: Healthy → Suspect → Degraded → Faulted

=== PART 6 — ACTIONABLE ALERTING ===

Explain:

  • a good alert should answer:
    • what is wrong?
    • how serious is it?
    • who should act?
    • what action is expected?
    • what context is needed?

Explain:

  • why alerts without owner/action become ignored
  • difference between:
    • operator action
    • maintenance action
    • engineering investigation
    • software/service support

=== PART 7 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • alert flood causes operators/engineers to ignore important alerts
  • machine gradually slows down but no one notices until output misses target
  • disk fills because storage usage was not monitored
  • transient retry spike hides developing hardware issue
  • remote alert lacks machine context, so support cannot act
  • alert clears automatically but root cause remains
  • monitoring says “healthy” because only process uptime is checked
  • false alerts from noisy thresholds reduce trust

For each:

  • what it looks like in production
  • why it happens
  • how experienced engineers improve the design

=== PART 8 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • monitoring must be designed as an operational feedback system
  • importance of:
    • meaningful metrics
    • health state aggregation
    • trend detection
    • alert ownership
    • severity rules
    • context-rich alerts
    • suppression/hysteresis
    • local vs remote alert routing
    • correlation with machine state, recipe, and production run
    • retention and export for analysis

Explain good vs bad approaches:

  • bad: alert on every error log, process-up equals healthy, no trends, no ownership, noisy thresholds
  • good: monitored health model, trend-aware alerts, actionable context, separation of local alarms and production alerts

Include ASCII component diagram: Machine Runtime ↓ metrics/events/health Monitoring Aggregator ↓ conditions/trends Alert Engine ↓ Operator / Maintenance / Engineering / Support

=== PART 9 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain production monitoring clearly
  • why monitoring is different from logging and alarms
  • common mistakes software engineers make when entering industrial systems
  • what strong engineers understand about degradation, actionable alerts, and operational feedback loops

=== OUTPUT ===

  • structured explanation
  • real-world production monitoring and alerting insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

7.10 ​

You are a Principal Software Architect with deep experience building industrial machine software (semiconductor equipment, robotics, automation systems, inspection machines).

You have real production experience, including:

  • deploying machine software into offline, restricted, customer-managed factory environments
  • managing compatibility between application software, firmware, drivers, SDKs, recipes, and machine configuration
  • preventing production failures caused by invalid configuration, partial upgrades, or version drift
  • supporting machines over years through maintenance, upgrades, patches, and field service workflows

I am a senior .NET engineer transitioning into this domain.

I want to deeply understand deployment, configuration, and lifecycle management in industrial machine software.

=== TOPIC === Deployment, Configuration & Lifecycle Management

=== GOAL ===

Help me understand how industrial machine software is deployed, configured, upgraded, validated, and maintained safely over time.

Focus on:

  • deployment constraints in industrial environments
  • software / firmware / driver / configuration compatibility
  • configuration validation before production use
  • upgrade and rollback strategy
  • long-term lifecycle and field support

=== ALIGNMENT WITH SOURCE OF TRUTH ===

This topic corresponds to:

"Deployment, configuration & lifecycle management"

Do NOT introduce unrelated topics.

=== STYLE & DEPTH ===

Write at a PRINCIPAL ENGINEER / SOFTWARE ARCHITECT level.

Focus on:

  • real factory constraints
  • long-lived machine systems
  • production safety and reliability
  • practical lifecycle trade-offs

Avoid:

  • generic cloud CI/CD discussion
  • shallow installer advice
  • enterprise deployment theory without machine context

=== DIAGRAM STYLE ===

Use UML-style ASCII diagrams:

  • Version dependency diagrams → app / SDK / driver / firmware / config
  • Deployment flow diagrams → prepare / validate / install / verify / rollback
  • Lifecycle diagrams → release → field install → operation → patch → upgrade
  • Configuration validation diagrams → config → compatibility check → activation

Rules:

  • ASCII only
  • simple and readable
  • clearly explain each diagram

=== SCOPE CONTROL ===

Stay within:

  • deployment
  • configuration validation
  • version compatibility
  • upgrade / rollback
  • long-term machine lifecycle

Do NOT deep dive into:

  • cybersecurity hardening
  • detailed DevOps pipeline tooling
  • recipe editing UI
  • formal compliance standards

=== STRUCTURE ===

=== PART 1 — WHY INDUSTRIAL DEPLOYMENT IS DIFFERENT ===

Explain:

  • industrial machines are often:
    • offline or on restricted networks
    • customer-controlled
    • tied to specific hardware
    • difficult to access remotely
    • expensive to stop
    • validated for a specific software/hardware combination
  • deployment may affect:
    • application software
    • device SDKs
    • drivers
    • firmware
    • recipes
    • calibration data
    • configuration files

Explain:

  • why deployment is not just “install the latest build.”

Use examples:

  • camera SDK update requires driver and firmware compatibility
  • motion controller firmware update changes behavior
  • recipe created for old machine config fails after upgrade

=== PART 2 — VERSION AND COMPATIBILITY LAYERS ===

Explain version layers:

  • application version
  • plugin/module version
  • device SDK version
  • driver version
  • firmware version
  • hardware revision
  • configuration schema version
  • recipe version
  • calibration data version

Explain:

  • why these layers must be compatible
  • why “same application version” is not enough

Include ASCII dependency diagram: Application ↓ depends on SDK / Driver ↓ depends on Firmware ↓ depends on Hardware Revision ↕ compatible with Configuration / Recipe / Calibration

=== PART 3 — CONFIGURATION VALIDATION BEFORE PRODUCTION USE ===

Explain:

  • configuration must be validated before activation
  • validation should check:
    • required parameters
    • ranges and units
    • hardware capabilities
    • installed modules
    • recipe/config compatibility
    • firmware/driver compatibility
    • safety limits
    • calibration validity

Explain:

  • why invalid configuration should fail closed, not silently fall back

Include ASCII validation flow: Load Config → Validate Schema → Validate Hardware → Validate Safety → Activate / Reject

=== PART 4 — DEPLOYMENT PACKAGE DESIGN ===

Explain what a deployment package may include:

  • application binaries
  • plugins/modules
  • runtime dependencies
  • SDK/driver installers
  • firmware package references
  • configuration templates
  • migration scripts
  • release notes
  • compatibility matrix
  • rollback plan

Explain:

  • why package content must be explicit and controlled
  • why hidden dependencies cause field failures

=== PART 5 — SAFE UPGRADE FLOW ===

Explain a realistic upgrade process:

  1. Confirm machine is in safe state
  2. Backup current software/configuration/recipes/calibration
  3. Verify target package compatibility
  4. Stop services/workflows safely
  5. Install/update components in correct order
  6. Migrate configuration if needed
  7. Validate hardware/software identity
  8. Run post-upgrade checks
  9. Confirm machine readiness
  10. Record upgrade audit trail

Include ASCII sequence diagram.

=== PART 6 — ROLLBACK AND RECOVERY STRATEGY ===

Explain:

  • rollback must be planned before upgrade
  • rollback may be hard if:
    • firmware changed
    • config migrated irreversibly
    • database schema changed
    • calibration format changed

Explain:

  • possible rollback strategies:
    • restore full backup
    • side-by-side install
    • versioned configuration
    • controlled downgrade path
    • service engineer recovery package

Explain:

  • why “just reinstall old version” is often not enough.

=== PART 7 — LONG-TERM MACHINE LIFECYCLE MANAGEMENT ===

Explain:

  • machines may run for years
  • over time:
    • hardware is replaced
    • firmware changes
    • drivers become outdated
    • OS patches may be restricted
    • recipes evolve
    • customer-specific variants appear
    • service teams need reproducibility

Explain:

  • why lifecycle management requires:
    • version inventory
    • compatibility records
    • migration paths
    • support tooling
    • field documentation

=== PART 8 — REAL-WORLD FAILURE SCENARIOS ===

Explain:

  • application updated but driver remains old
  • firmware update changes timing behavior
  • copied config from another machine enables unsupported hardware mode
  • installer succeeds but device SDK dependency missing
  • configuration migration silently changes units
  • rollback fails because schema was upgraded irreversibly
  • field machine differs from lab machine due to hardware revision
  • calibration data invalid after mechanical replacement
  • upgrade performed while machine not in safe state

For each:

  • what it looks like in production
  • why it happens
  • how experienced engineers prevent or diagnose it

=== PART 9 — SOFTWARE DESIGN IMPLICATIONS ===

Explain:

  • deployment and lifecycle constraints must influence architecture
  • importance of:
    • version-aware components
    • compatibility matrix
    • configuration schema versioning
    • migration validation
    • startup self-checks
    • hardware identity checks
    • explicit activation and fail-closed behavior
    • rollback planning
    • audit trail for upgrades
    • field-service-friendly diagnostics

Explain good vs bad approaches:

  • bad: hidden dependencies, manual config copying, no compatibility checks, silent fallback, no rollback plan
  • good: controlled deployment package, startup validation, version inventory, compatibility enforcement, safe upgrade/rollback path

Include ASCII component diagram: Deployment Package ↓ Installer / Upgrade Coordinator ↓ Compatibility Validator ↓ Config Migration + Hardware Identity Check ↓ Post-Upgrade Verification ↓ Machine Ready / Recovery Required

=== PART 10 — INTERVIEW / REAL-WORLD TALKING POINTS ===

Give:

  • how to explain deployment and lifecycle management clearly
  • why industrial deployment is not the same as cloud deployment
  • common mistakes software engineers make when entering industrial systems
  • what strong engineers understand about compatibility, validation, rollback, and long-term field support

=== OUTPUT ===

  • structured explanation
  • real-world deployment, configuration, and lifecycle insights
  • ASCII UML-style diagrams
  • practical language suitable for real systems and interviews

Docs-first project memory for AI-assisted implementation.

LCN Wafer Inspection Prototype