Embedded Systems Architecture Questions

Overview
Design a portable HAL + driver model with: abstract bus layer, bus-specific adapters, sensor drivers implementing a common sensor ops interface, platform/bsp glue for board-specific resources, and a probe/registration system that handles power domains and init sequencing.

Key components

• Abstract Bus Layer: i2c_read/write, spi_transfer, uart_send/recv as function pointers in a bus_ops struct. Bus drivers implement these per MCU peripheral.
• Sensor driver interface: each sensor provides probe/remove + sensor_ops (init, read, configure, suspend, resume).

// example sensor ops
struct sensor_ops {
  int  (*init)(struct sensor *s);
  int  (*read)(struct sensor *s, void *buf, size_t len);
  int  (*configure)(struct sensor *s, int cmd, void *arg);
  int  (*suspend)(struct sensor *s);
  int  (*resume)(struct sensor *s);
};

• Device descriptor: ties sensor driver to bus instance, address/CS/pins, power domain, IRQ, and devicetree/board-data.

Driver registration & probing

• Use static registration table populated from device tree or board.c. At boot, kernel/RTOS calls global probe manager which:
  1. Iterates devices, resolves bus instance and resources.
  2. Requests power domain(s) and regulators (PM API).
  3. Enables clocks in required order.
  4. Calls bus-specific probe (to ensure bus ready).
  5. Calls sensor->probe(bus_handle, plat_data).
• If probe fails, undo in reverse order.

Power domains & init sequencing

• Encode dependencies in device descriptor: .power_domains[], .clocks[], .order_priority.
• Probe manager sorts by dependency graph/topological order; enable regulators/clocks before bus transactions.
• Support runtime PM: sensor suspend/resume through sensor_ops hooks and PM framework.

Common sensor abstraction to apps

• Expose unified API: sensor_open(id), sensor_read(id/type), sensor_ioctl for configs. Return standardized units/metadata.
• Provide translator layers in driver to convert raw samples to SI units.

Minimize duplicated code

• Centralize common tasks: bus ops, regulator/clock helpers, GPIO/IRQ helpers, devicetree parsing utilities.
• Use configuration data (DT or board tables) not #ifdefs.
• Implement shared sensor utility library (sampling, filtering, calibration) used by multiple drivers.

Trade-offs

• Device-tree adds flexibility but needs tooling. Static tables simpler for tiny bootloaders.
• Extra abstraction adds slight overhead; optimize hot paths (DMA, direct regs) in bus-specific implementations.

This architecture keeps platform-specific glue small, enforces safe power/init ordering, and gives applications a consistent sensor API.

High-level workflow

• Identify critical module (e.g., actuator control state machine: Idle, Extend, Hold, Retract, Error).
• Select properties (safety, liveness, resource bounds, timing).
• Abstract model (Promela or simplified C) and iterate between model and code.
• Apply tools (SPIN, CBMC, Frama‑C) and integrate results into unit tests and CI.
• Triage counterexamples, add tests, refine abstraction, repeat until stable.

Property selection (examples)

• State invariants: “If actuator == Extend then valve_open == true.”
• Mutual exclusion: “Only one actuator driver may command PWM.”
• No deadlocks / progress: “From Extend command, eventually reach Hold or Error within N steps.”
• Resource bounds: “Array indices, stack, and integer overflow never occur.”
• Timing/real‑time constraints expressed as bounded steps for model checking.

Model abstraction

• Keep control logic exact; abstract low‑level I/O and delays (non-deterministic inputs).
• Create small Promela model for SPIN to check liveness/deadlock.
• For C-level correctness, use a simplified C harness that models sensors as non‑deterministic inputs for CBMC/Frama‑C.

Tools and roles

• SPIN (Promela): liveness, deadlock, reachability on state machines; produces counterexample traces.
• CBMC: bounded model checking of C code — finds buffer overflows, assert violations, arithmetic overflow, and memory safety up to bound.
• Frama‑C (Value, WP): static value-range analysis, proof obligations (WP) for functional properties and absence of undefined behavior.
• Auxiliary: KLEE/SMT-based symbolic execution for deeper path exploration; hardware‑in‑loop simulator for replaying counterexamples.

Practical integration with tests and CI

• On PR: run Frama‑C Value for fast checks; run CBMC with reasonable unwinding bounds; run SPIN on extracted Promela model.
• Fail build on new violations; attach counterexample traces as artifacts.
• Convert counterexamples into unit/integration tests (CUnit/Unity + hardware stubs) and add to regression suite.
• Nightly deeper runs (higher CBMC bounds, long SPIN searches) and coverage-guided fuzzing.

What defects these methods reliably find

• Logical errors in state transitions (missing guards, illegal transitions).
• Deadlocks and livelock in coordination logic.
• Integer overflows, buffer overrun, use‑after‑free, null deref.
• Violations of invariants and assertion failures.
• Missing handling of sensor ranges/unexpected inputs.

Why this approach works

• Model checking finds control‑flow and concurrency bugs quickly with concrete traces.
• CBMC/Frama‑C operate on actual C code, catching implementation-level safety issues.
• Closing the loop: counterexamples → tests → code fixes ensures regression control and traceability suitable for safety‑critical embedded firmware.

Clarify goal

Provide fault handlers that capture useful state with minimal footprint, perform safe logging (or mark for post-mortem), and either recover when safe or fail into a known safe state.

Approach (high-level)

• Capture CPU state (stack pointer, LR, PC, xPSR, core registers) immediately in the fault handler.
• Use safe, non-blocking logging: store data to a reserved RAM crash buffer or toggle an LED; avoid heap, printf, or interrupts.
• Minimize code/stack use and make handlers reentrant-safe.
• Provide recovery options: guarded restart, safe shutdown, or halt with watchdog to reset.

Concrete techniques & example

• In assembly or C, read stacked registers saved by hardware on exception entry. Store to a fixed RAM struct (no malloc). Example extracting stacked PC/LR:

// Called from a naked handler that provides stack pointer
void HardFault_Handler_C(uint32_t *stacked_regs) {
    // stacked_regs: r0, r1, r2, r3, r12, lr, pc, xPSR
    crash_buffer.pc = stacked_regs[6];
    crash_buffer.lr = stacked_regs[5];
    crash_buffer.xpsr = stacked_regs[7];
    // capture other registers via manual reads if needed
    safe_mark_crash(); // set flag in RAM / checksum
    system_safe_halt();
}

• Use a small CRC/checksum to validate buffer on next boot and store a timestamp or reset reason.
• For BusFault/MemManage read CFSR, HFSR, BFAR/MMFAR to identify faulting address.

Safe logging & footprint

• Pre-allocate a fixed-size crash buffer in no-init RAM, keep writes atomic (disable interrupts briefly).
• Avoid syscalls, dynamic allocations, or peripheral drivers that may be in bad state.
• Optionally use DMA to persist log to flash or external store during reboot sequence—but use conservative retries.

Recovery & fail-safe

• If fault is transient and stack/registers sane, attempt controlled restart of offending task or peripheral.
• Else: disable outputs, put actuators to safe state, feed watchdog to force a reset or enter infinite loop to allow external monitor to act.
• Ensure bootloader reads crash buffer and reports via serial/telemetry, and include a mechanism to prevent boot loops (failcount threshold).

Trade-offs / best practices

• Maximize post-mortem data while keeping handler tiny.
• Prefer RAM buffer + boot-time reporter over complex in-handler I/O.
• Test handlers by fault injection and validate buffer integrity across resets.

Public API (header)

// temp_sensor.h
typedef struct temp_sensor_t temp_sensor_t;
typedef void (*temp_cb_t)(temp_sensor_t *dev, float temp, void *ctx);

// Initialize returns opaque handle or NULL on failure
temp_sensor_t *temp_sensor_init(const char *model, void *hw_cfg);
// Synchronous blocking read (timeout in ms, -1 for forever)
int temp_sensor_read_sync(temp_sensor_t *dev, float *out_c, int timeout_ms);
// Asynchronous read: starts acquisition, returns 0 on accepted
int temp_sensor_read_async(temp_sensor_t *dev, temp_cb_t cb, void *ctx);
// Power management
int temp_sensor_power_down(temp_sensor_t *dev);
int temp_sensor_power_up(temp_sensor_t *dev);
// Cleanup
void temp_sensor_deinit(temp_sensor_t *dev);

Design & separation

• Opaque handle (temp_sensor_t) hides internals. Public API only operates on pointer.
• Hardware/transport abstraction: a HAL ops struct provided per platform:

struct hw_ops {
  int (*write)(void *h, const uint8_t *buf, size_t n);
  int (*read)(void *h, uint8_t *buf, size_t n);
  int (*delay_ms)(unsigned);
  void *hw_ctx;
};

• Sensor protocol logic implemented in a model-agnostic core that uses hw_ops to talk to bus. Model-specific pieces (register maps, conversion formulas, init sequences) are small modules implementing a model_ops:

struct model_ops {
  int (*init)(temp_sensor_t *d);
  int (*start_measure)(temp_sensor_t *d);
  int (*fetch)(temp_sensor_t *d, float *out_c);
  int (*power_down)(temp_sensor_t *d);
};

• temp_sensor_init selects model_ops by model string or registration table, allocates temp_sensor_t containing hw_ops*, model_ops*, state and optional RTOS timer/worker for async.

Asynchronous reads

• Implemented at core level using either:
  • Interrupt-driven model: model_ops->start_measure triggers sensor, ISR calls core_notify() which schedules callback.
  • Polling/timer: core starts a non-blocking timer, on expiry calls model_ops->fetch and invokes user callback.
• Callbacks invoked from safe context (deferred work or thread) to avoid long work in ISR.

Multiple models, minimal duplication

• Factor common state machine (start, wait, fetch, convert) into core; only model_ops provide differences: init sequences, command bytes, raw->C conversion.
• Provide helper utilities for common protocols (I2C/SPI) so model implementations only supply parameters (addresses, commands).
• Use registration table for models to avoid switch-case proliferation.

Trade-offs & notes

• Opaque handle + ops increases portability at small cost of indirection.
• Keep callbacks non-blocking and document threading context.
• Provide unit-test hooks (mock hw_ops) to test protocol logic independent of hardware.

Overview
I would implement a graded, telemetry-preserving reset/recovery design combining hardware watchdogs, staged resets, transactional non-volatile storage and a safe-mode to guarantee rapid recovery without re-entering fault loops.

Watchdog placement

• Hardware independent watchdog (IWDG) enabled early in boot with short timeout to guarantee recovery from hangs.
• Windowed or dual-stage watchdog: short HW watchdog for hangs; supervised SW watchdog (task heartbeat) to detect livelock.
• Keep a watchdog unlock/lock policy: HW WDT enabled before peripheral init; only clear after a verified-progress checkpoint.

Staged reset levels

• Soft reset: OS-level task restart or process respawn; preserves RAM telemetry and NVM state.
• Hard reset: full MCU reset via watchdog or system reset line when soft fails.
• Bootloader reset: enter bootloader (via magic flag in battery-backed register or staged boot counter) for firmware rollback or recovery image.

Transactional non-volatile storage

• Two-slot journaling (A/B) or append-only ring with CRC + sequence number + signature.
• Write telemetry atomically: write record -> write commit marker (or update monotonic sequence in RTC-backed register). Use page-erase-aware patterns to avoid partial writes.
• Keep a golden-copy bootstrap area for critical counters/flags in battery-backed SRAM or EEPROM.

Crash counters & retry policy

• Maintain monotonic crash counter incremented on each boot before normal startup completes; persist ticket in NVM atomically.
• Threshold logic: if crashes > N within T minutes, escalate to next reset level or safe-mode; use exponential backoff on auto-retry to avoid rapid loop.

Safe-mode fallback

• Minimal runtime: disable nonessential subsystems, mount read-only config, open telemetry-only comms, accept remote commands/firmware update, expose diagnostics.
• Enter safe-mode when boot attempts exceed threshold or integrity checks fail.

Verification & loop prevention

• On every boot, verify last shutdown reason and crash-counter. Validate NVM CRC/signature and firmware image signature.
• Only clear the crash counter and commit “healthy” state after a sustained uptime window (e.g., 5 minutes or successful heartbeat cycles).
• Record reset reason (WDT, POR, software) and telemetry snapshot to NVM before rearming WDT. Use these logs to decide rollback vs retry.
• Limit automatic retries; require external intervention or OTA rollback after repeated failures.

This design preserves telemetry via transactional NVM, ensures rapid restart via graded resets and watchdogs, and prevents fault loops by using persisted crash counters, verification steps, and a guarded clear-of-error only after verified uptime.