Hardware Simulation, Emulation, and Debugging Questions

Approach (one-sentence)
Use the MCU’s on‑chip trace (ITM/SWO for low-bandwidth single-pin, ETM + TPIU for full instruction/data trace) to packetize events and stream them off-chip; configure trace clocks, ports and host receiver, encode events compactly, and fall back to aggregation/filters when bandwidth limits are hit.

MCU configuration (practical steps)

• Enable trace block: set DEMCR.TRCENA = 1 and enable DWT if timestamps needed.
• ITM/SWO:
  • Enable ITM and stimulus ports (ITM->TER bitmask).
  • Configure TPIU/SSP: set TPIU->ACPR prescaler so SWO baud ≈ trace_clk / (prescaler+1). Choose NRZ mode on the host if available.
  • Typical practical SWO bandwidth is a few Mbps (depends on MCU trace clock and TPIU settings).
• ETM + TPIU:
  • Configure ETM to emit selected events (context ID, PC samples, or user events).
  • Route ETM -> TPIU, choose parallel (ETM port) or serial (SWO/TPIU) formatter mode.
  • Ensure trace clock (TRACECLKIN or core clock) is set and host expects that sample rate.

Host settings

• Set the host trace tool (e.g., OpenOCD, Trace32, Keil ULINK) to the same SWO/TPIU baud and protocol (NRZ vs Manchester).
• If using ETM over parallel, configure host to accept the ETM width and clock. Enable formatter decoding.

Event encoding & efficiency

• Use minimal event IDs (8–16 bit) rather than strings; map IDs to messages on host.
• Pack event + delta-time: send event ID + small varint delta (use DWT cycle counter for base).
• Use ITM stimulus ports: e.g., Port 0 for logs, Port 1 for events, ensure each write is 8/16/32-bit aligned.
• Avoid sending full payloads; send index references to pre-shared tables.
• Example compact packet: [event_id:8][delta_us:variable][arg:optional 16/32].

Bandwidth & limitations

• SWO single-pin is limited (practical ~1–8 Mbps depending MCU and mode) — fine for sparse events but not for high-rate sampling.
• ETM can sustain much higher rates (tens to hundreds of MB/s) if the TPIU/trace interface and host support it; limited by trace clock, interface width, and transport (e.g., SWO formatter reduces bandwidth).
• Enabling trace might change timing slightly; keep critical timing checks.

If trace link saturates

• Reduce event rate: sample, throttle, or only emit state transitions.
• Aggregate on-chip: buffer events in RAM ring buffer and stream in bursts or post-mortem.
• Hardware filters: use DWT/ETM comparators to only trace_selected_modules or event masks.
• Backpressure strategy: count dropped events and expose a drop-counter event so host knows data loss occurred.
• Move to higher-bandwidth transport: switch from SWO to ETM+TPIU over parallel trace or use ETB + DMA to offload to host later.
• Compression: delta-encode timestamps and use variable-length encoding for IDs/args.

Practical example (sequence)

• Enable TRCENA, configure DWT cycle counter.
• ITM->TER |= (1 << 0); ITM->TCR = ITM_ENABLE | SYNC_ENABLE;
• Configure TPIU->ACPR to get desired SWO baud.
• On event: uint32_t ts = DWT->CYCCNT; write compact packet to ITM->PORT[0].u8.

This approach gives low-intrusion event capture; pick ITM/SWO for sparse, low-bandwidth telemetry and ETM/TPIU for high-rate, timing-accurate tracing.

Approach (brief)

Use a power-of-two sized ring buffer with head (read) and tail (write) indexes as unsigned integers. For SPSC between ISR (producer) and main-loop (consumer) on a single CPU, updates of head/tail must be atomic (naturally if same machine word) and declared volatile to prevent compiler reordering. No locks required; producer updates tail only, consumer updates head only.

Core code

#include <stdint.h>
#include <stdbool.h>
#include <stddef.h>

typedef struct {
    volatile uint32_t head;   // consumer-owned (read index)
    volatile uint32_t tail;   // producer-owned (write index)
    uint32_t size_mask;       // size - 1 (power of two)
    uint8_t buf[];            // flexible array
} spsc_ring_t;

/* allocate: size must be power of two */
static inline spsc_ring_t *spsc_create(void *mem, uint32_t size_pow2) {
    spsc_ring_t *r = (spsc_ring_t*)mem;
    r->head = r->tail = 0;
    r->size_mask = size_pow2 - 1;
    return r;
}

/* push single byte from ISR; returns false on full */
static inline bool spsc_push_irq(spsc_ring_t *r, uint8_t v) {
    uint32_t t = r->tail;
    uint32_t next = (t + 1) & r->size_mask;
    if (next == r->head) return false; // full
    r->buf[t & r->size_mask] = v;
    /* ensure byte written before tail update (compiler/CPU ordering) */
    r->tail = next;
    return true;
}

/* pop single byte from main loop; returns false if empty */
static inline bool spsc_pop(spsc_ring_t *r, uint8_t *out) {
    uint32_t h = r->head;
    if (h == r->tail) return false; // empty
    *out = r->buf[h & r->size_mask];
    r->head = (h + 1) & r->size_mask;
    return true;
}

Atomicity & volatile

• head and tail are volatile to avoid compiler optimizing away reads/writes.
• On common 32-bit MCUs, aligned 32-bit reads/writes are atomic; confirm for your target.
• Producer only writes tail and buffer[tail]; consumer only writes head — this separation avoids races.
• If memory ordering/CPU reordering is a concern, insert memory barrier (compiler intrinsics or asm) between data write and tail update in push, and between tail read and data read in pop.

Notes

• Buffer size must be power-of-two; use size_mask for fast modulo.
• For multi-byte transfers or different CPU architectures, add explicit barriers or disable interrupts briefly in consumer when needed.

Situation & goal
Explain how I'd find a slow-growing heap problem on an embedded device and prevent recurrence: detect leaks/fragmentation at runtime, reproduce with accelerated tests, and apply design/allocator changes.

Runtime instrumentation

• Add lightweight allocation tracking: wrap malloc/free/realloc to log size, caller (via __builtin_return_address), and allocation count; sample instead of logging every call to limit overhead.
• Export periodic heap stats (free/used/peak, largest free block, fragmentation metric) via telemetry or RTOS shell.
• Example malloc wrapper:

// simple example; keep it minimal for production
void *my_malloc(size_t s, void *caller){
  void *p = malloc(s);
  if(p) record_alloc(p, s, caller);
  return p;
}
#define malloc(s) my_malloc(s, __builtin_return_address(0))

Leak detection strategies

• Track outstanding allocations by size and stack trace; alert if growth trend > threshold.
• Use deterministic object ownership models: require owner ID on allocs, assert on shutdown.
• Periodic heap walk: free-list traversal (if allocator exposes) to compute largest contiguous block.

Simulated long-run tests

• Accelerated workload harness that runs realistic sequences with faster cycles, fuzz inputs, and heavy edge cases.
• Use synthetic fragmentation tests: allocate many varied sizes, free in patterns to reproduce fragmentation.
• Run on host emulator and on-device with telemetry capture; compare memory curve slopes.

Design changes to mitigate

• Prefer static allocation or object pools for frequently used short-lived objects to avoid churn.
• Use segregated fixed-size pools (slab allocator) for common sizes to eliminate fragmentation.
• For dynamic needs, adopt an allocator with coalescing and defragmentation or use region-based alloc/free phases.
• Add watchdog/graded restart for unrecoverable leaks and assert-on-threshold to catch issues early.

Why these help

• Instrumentation pinpoints sources with low overhead; simulated tests reproduce months-long behavior quickly; pools/static allocation remove root causes (fragmentation and leak surface) for reliable long-running embedded systems.

Approach summary
Use LEDs/GPIOs as low-cost telemetry: blink patterns encode events/errors; short GPIO pulses let an oscilloscope or logic analyzer measure precise timings; watchdog (WDT) enforces resets on hangs and signals crash reason via boot-blink.

Encoding diagnostics

• Use a small fixed frame: [preamble] + [ID nibble] + [error code byte] + [checksum] separated by long gap.
• Example blink: 3 fast flashes (preamble), 4-bit MSB-to-LSB as 4 pulses (1 = long high, 0 = short high), pause, next byte.
• Reserve codes: 0x01 boot OK, 0x10 sensor init fail, 0x20 comm timeout, 0xFF assert.

Precise timing measurement

• Toggle a dedicated GPIO at key points (enter/exit ISR, before/after sensor read). Use oscilloscope or external jig to capture edges; measure delta-time directly.
• For sub-microsecond needs, use assembler tight-timed toggles around critical code.

Watchdog use

• On watchdog reset, store a small persistent reason in battery-backed RAM or flash flag before triggering WDT during controlled failure; on next boot blink that code repeatedly.
• If no nonvolatile, use boot-count increases in retention RAM to signal repeated WDTs.

Minimize power/timing impact

• Drive LEDs/GPIOs only when in diagnostic mode; otherwise keep pins low-power.
• Use brief pulses (ms-level) and low duty-cycle; use MOSFET or current-limited drive to reduce LED current.
• Sampling: enable diagnostic toggles via a runtime flag or compile-time debug build. For production, enable only on field-service mode (button, external jumper, or over-the-air command).
• Avoid heavy blocking delays: implement non-blocking blink state machine so main loop timing remains intact.

Practical tips

• Document codes and create a small decoding script.
• Use oscilloscope persistence and single-shot capture for intermittent bugs.
• Correlate GPIO toggles with power supply current probe for combined timing/power analysis.

Goal & approach
Start with reproducible test case, then iterate: verify power rails & reset, check boundary interconnects, isolate subsystems, collect rich telemetry (timestamps, watchdogs, voltage/current). Work top-down: power → clocks → JTAG/boundary-scan → boot firmware → peripheral isolation.

Test points to add

• All rail voltages (pre/post regulator) and enable pins (TP_3V3_IN, TP_CPU_CORE, TP_FPGA_IO, TP_WIFI_VBAT)
• Key reset nets (CPU_NRST, FPGA_NCONFIG, PMIC_PWRGOOD)
• Clocks (main oscillator, FPGA fabric, WiFi ref clk)
• I2C/SPI/UART lines with series resistors for probes
• JTAG/TCK/TDI/TDO/TMS on CPU and FPGA
• GPIO lines bridging SoC↔FPGA and SoC↔WiFi
• Current sense shunt on major rails

JTAG & boundary-scan

• Run IEEE 1149.1 on CPU and FPGA IOCells to detect stuck/high-Z pins and shorts before powering peripherals.
• Use chain scan to confirm pin mapping and cable integrity.
• Use boundary-scan to drive/observe interconnects (toggle SoC pins, observe FPGA inputs) to validate signal levels and timing without full firmware.

Power-sequencing checks

• Log PWRGOOD transitions with logic analyzer; verify each regulator meets rise-time and ordered enables per datasheets.
• Inject controlled delays between rails using bench PSU or PMIC sequencing to test sensitivity.
• Stress-test undervoltage and brownout behavior; confirm resets align with spec.

Isolation experiments

• Reproduce issue with minimal configuration: CPU + RAM only; then add FPGA; then WiFi.
• Replace FPGA with golden bitstream that tri-states non-essential pins.
• Use software to disable WiFi driver and isolate via EMI/shielding to test interference.
• Swap modules (known-good WiFi) and use coaxial feed for antenna tests.

Logs & telemetry to collect

• Kernel logs with timestamped printk, earlycon, persistent console over UART, and circular buffer retention across resets.
• Firmware watchdog and panic hooks that dump CPU registers, stack pointer, last ISR, and mmio snapshots.
• FPGA: event counters, sync-loss, soft-reset traces, and AXI bus error registers exposed via memory-mapped debug region.
• PMIC telemetry: sequencing events, fault flags, thermal, and current measurements.
• High-resolution timestamps from logic analyzer/trace for latency spikes; correlate logs with power/clock events.

Why this works
Combines electrical validation (rails, clocks), non-invasive scan (JTAG), progressive isolation to narrow root cause, and correlated telemetry to link symptoms (latency spikes / resets) to electrical, firmware, or RF sources.