Performance Optimization for Constrained Devices Questions

Definition

volatile tells the compiler "this variable's value can change outside the normal flow of this code, so don't optimize away or reorder accesses to it." Without it, the compiler assumes a variable only changes through code it can see, and is free to cache its value in a register, skip what looks like a redundant read, or reorder accesses relative to other code.

Interaction with compiler optimizations

With volatile, every read and write in source order becomes a real memory access: none can be hoisted out of a loop, cached in a register across iterations, or eliminated as "dead" even if nothing in the visible code appears to use the result.

Memory-mapped registers

A hardware register, such as a UART status register, can change value because of hardware activity, not because of anything your code did. Without volatile, the compiler might read it once and treat it as unchanged forever:

/* Peripheral struct as the vendor header would declare it */
typedef struct {
    volatile uint32_t STATUS;   /* FIX: volatile so every read re-checks hardware */
    volatile uint32_t DATA;
} UART_TypeDef;

#define UART ((UART_TypeDef *)UART_BASE_ADDRESS)

/* BUG if STATUS were declared as a plain (non-volatile) uint32_t above:
 * the compiler could read it once, decide the loop condition never
 * changes, and either spin forever or optimize the loop away entirely.
 * With STATUS declared volatile as above, this loop is correct: */
while (!(UART->STATUS & TX_READY)) {}

ISRs and DMA

The same pattern applies to any variable shared between the main loop and code that runs asynchronously to it, an ISR (interrupt service routine, the function the CPU jumps to when a hardware interrupt fires) or a DMA (Direct Memory Access) controller writing to memory on its own. Main-loop code polling a flag the ISR sets must see the ISR's write, so the flag needs volatile:

volatile int ready = 0;
void wait_for_isr(void) { while (!ready) {} }
void ISR_Handler(void) { ready = 1; }

Without volatile here, the compiler could cache ready in a register inside wait_for_isr and never notice the ISR's write, spinning forever even though the flag really did change.

Where volatile is insufficient

volatile guarantees accesses aren't optimized away or reordered relative to each other; it does not guarantee atomicity, cross-variable memory ordering, or mutual exclusion.

• Atomicity: a volatile uint64_t counter on a 32-bit MCU (microcontroller unit, the chip running this code) is read or written as two separate 32-bit accesses; an ISR firing between them can see a torn value. Fix: a hardware atomic operation, or a critical section (disable interrupts around the access).
• Memory ordering: volatile doesn't stop out-of-order execution or insert a hardware memory barrier for DMA coherency. Fix: explicit barriers (__DMB, __DSB on ARM) and cache clean/invalidate operations around a DMA buffer handoff.
• Compound read-modify-write: flags |= BIT on a volatile variable is still a non-atomic read-then-write at the machine-code level, an interrupt landing between the two steps loses whichever side updated last. Fix: an atomic bit-set instruction where available, or a critical section around the whole read-modify-write.

Framing the conversation

I'd start from the product outcome, not the hardware detail: "we can't hit the 5ms requirement on the microcontroller unit (MCU, the single chip running this firmware) we're currently using, here's what it would take to get there, and here are some middle options." Leading with "we need a different chip" without that framing sounds like an unexplained cost increase; leading with the requirement and the current gap makes the ask legible.

Metrics I'd present

• Current measured worst-case latency versus the 5ms target, in plain terms, for example "the current chip's worst-case response is 12ms, more than double what's needed," so the gap is concrete, not abstract.
• Why the current chip can't close that gap through software alone: clock speed sets how many instructions fit in a given time budget, and available RAM determines whether techniques like double-buffering (which would help hide some of the latency) even fit in memory; if I've already tried the cheap software levers and hit a wall, I'd say so explicitly, since that's what makes the hardware ask credible rather than a first resort.
• Cost delta: the per-unit cost difference of the faster/bigger part, multiplied by expected production volume, is the number that actually matters to a PM's decision, not just "it's a more expensive chip."
• Schedule impact: if this requires a hardware respin (new board layout, re-certification), that's a real timeline cost that needs to be on the table alongside the dollar cost.

Trade-offs to lay out

A straight cost-versus-spec trade-off: hit 5ms exactly and pay the part cost and schedule cost, or accept a compromise. I'd present it as a real decision with the engineering trade-offs visible, not as a foregone conclusion that we must upgrade.

Graceful degradation and incremental options

• Ship at a partial spec with a roadmap: for example, hit 8ms now on the current hardware with the cheap software optimizations already applied, and commit to 5ms in the next hardware revision, if the product can tolerate that gap in the near term.
• Selective degradation under load: identify the specific worst-case path that drives the 12ms number, and see if that exact scenario can shed an optional step (skip a non-critical filtering pass, say) only when it's at risk of missing the deadline, rather than paying the cost on every request.
• Tiered hardware: if this is a product line rather than a single SKU, consider whether only the higher-tier product needs the faster part, keeping the cheaper chip where the latency requirement doesn't actually apply.

How I'd close it

I'd end with a clear recommendation, not just a menu, based on what the data supports, while being explicit about what's still uncertain (for example, if the cost delta hasn't been fully quoted yet), so the PM has both a decision and a sense of how firm the numbers are.

Framing

A 20 percent regression in a tight interrupt service routine (ISR, the function that runs when a hardware interrupt fires) after a toolchain upgrade, with reverting off the table, means the investigation has to isolate exactly what the new compiler or libc changed about that specific code path, then work around it without reverting the whole toolchain.

Investigation steps

1. Bisect what actually changed: if there were multiple intermediate toolchain versions between old and new, rebuild at each step to narrow down which specific version introduced the regression, rather than assuming it's purely the newest release.
2. Diff the generated assembly for the ISR, old toolchain versus new, using the disassembler (objdump -d or equivalent). This is the most direct way to see what actually changed: different instruction selection, different inlining decisions, or new instructions the old build didn't emit (like added bounds-checking or stack-protection code).
3. Check for new default compiler flags: toolchain upgrades sometimes change defaults, for example enabling a stack-protector (-fstack-protector-strong, which inserts a runtime canary check around function returns to catch stack buffer overflows) or a different default CPU tuning target, silently adding overhead to every function including the ISR, unless the build explicitly pins the old flags.
4. Check whether a call that used to be inlined now isn't, or vice versa: a new libc version might implement a function (like a small memcpy or a math helper) differently, or a changed inlining heuristic in the compiler might turn what used to be inline code into an actual function call inside the ISR, adding call/return overhead exactly where it hurts most.
5. Check for new libc safety features: some libc upgrades add runtime checks (like fortified string functions) that add overhead in exchange for safety, and that overhead might now be running inside a hot path that never needed it.
6. Measure interrupt latency directly on real hardware, using a cycle counter (for example the Cortex-M's DWT->CYCCNT register) or toggling a GPIO pin at ISR entry/exit and observing it on an oscilloscope, to separate "prologue/epilogue overhead grew" from "the ISR body itself got slower," since those point at very different fixes.

Getting performance back without reverting

Once the specific cause is isolated (say, a new default stack-protector flag on this one file, or a libc call that stopped inlining), the fix is usually targeted: pin the specific compiler flag or function attribute that regressed for just this translation unit (a single source file after preprocessing, i.e. one compiled .c/.cpp file) or function, rather than reverting the whole toolchain; or replace the regressed libc call in this one hot path with a small hand-written equivalent tuned for this specific ISR, while leaving everything else on the new toolchain. This keeps the rest of the codebase on the supported, upgraded toolchain and limits the workaround's blast radius to exactly the function that regressed.

What I'd check afterward

I'd re-measure the same latency metric after the fix to confirm the regression is actually closed, not just "probably better," and I'd document the root cause and the targeted flag/attribute so the next toolchain upgrade doesn't quietly reintroduce the same issue.

Definition

Worst-case execution time (WCET) is the longest amount of time a specific piece of code could possibly take to run, across every input and every hardware state it could encounter, not the average or typical time. For a safety-critical task with a hard deadline, what matters is this worst case, not the common case.

Approaches to estimating it

• Measurement-based: run the task repeatedly under many representative (and deliberately adversarial) inputs, recording execution time via a cycle counter, and take the observed maximum. This is straightforward to do, but it only ever reports the worst case actually observed, it can't prove nothing worse is possible, unless the test set is proven to exercise every relevant code path and timing scenario, which is hard to guarantee in practice.
• Static analysis: a tool models the CPU's pipeline, any cache behavior, and the program's control-flow graph to mathematically derive a provably safe upper bound, without running the code at all. This gives an actual guarantee (assuming the hardware model is accurate), but static bounds are often quite pessimistic (the tool has to assume the worst plausible combination of pipeline stalls, cache misses, and branch outcomes, which may never actually occur together), and accurate tooling for a specific chip isn't always available or cheap.
• Hybrid (measurement-based analysis): measure the execution time of individual basic blocks in isolation, then combine those measurements analytically along every possible path through the control-flow graph. This tends to land between the two: tighter than pure static analysis, more rigorous than a plain "run it a lot and take the max."

Accounting for interference

Whatever the method, the real worst case has to include more than the task's own code: interrupts preempting the task mid-execution, other bus masters (like a DMA channel) contending for memory bandwidth, and worst-case interrupt nesting all add real, sometimes large, time on top of the task's standalone execution time. A WCET estimate that only looks at the task in isolation, with no other interrupts firing and a cold or ignored cache/pipeline state, is not the number that matters for the real system.

How confident I could be before shipping

Honestly, less than "certain" unless I've used a rigorous static-analysis tool with an accurate hardware model, and even then only as confident as that hardware model is accurate. My practical approach:

• Build in real margin between the estimated/measured WCET and the actual deadline (a commonly used rule of thumb is something like 20 to 30 percent headroom), treated as a safety margin against an imperfect estimation method, not as proof the estimate is exact.
• Stress test specifically under worst-case system conditions: all relevant interrupts firing at their worst-case rate, other DMA/bus activity active, cache and pipeline in a cold or adversarial state, not just the task running alone on an idle system.
• Where feasible, add runtime monitoring in the field (a deadline-miss counter, or a watchdog tuned tighter than the hard deadline) so a near-miss or actual miss is caught and visible rather than silent.
• Match the rigor of the method to the actual criticality tier: for genuinely safety-critical code operating under a formal standard (for example DO-178C in aviation or IEC 61508 in industrial contexts), measurement-based estimation alone typically isn't considered sufficient, and certified static-analysis tooling becomes a real requirement, not just a nice-to-have.

Confirm it's really a leak

On bare metal, a "leak" usually means allocations that never get freed, but rising RAM usage can also come from things that only look like one: stack usage growing from deep or unbounded recursion or worst-case interrupt nesting, or a buffer or queue that's genuinely supposed to grow under load and just hasn't drained yet. Before chasing a leak, add periodic instrumentation logging the real metrics separately, heap high-water mark, stack high-water mark, and any application-level buffer or queue depths, and watch which one actually trends upward over the multi-day window rather than assuming it's the heap.

Localize where it's coming from

• If using a general allocator, wrap malloc/free/realloc in a thin shim that increments or decrements a counter and records the call site (via __builtin_return_address or a passed-in file and line macro), so you can log which call site's outstanding allocation count keeps climbing without a matching free.
• If using a custom fixed-size pool allocator, common in firmware specifically to avoid the general heap's non-determinism, tag each allocated block with its requesting subsystem (a debug field, compiled out in release builds) and periodically dump which subsystem holds how many blocks, watching for one whose count only ever goes up.
• If instrumentation can't be added quickly, bisect by disabling major subsystems, the radio stack, logging, a specific sensor driver, one at a time across separate long-running tests, and see which absence stops the growth. Cruder, but works with zero new code in the suspect path.

Reproduce reliably enough to fix with confidence

A multi-day real-time leak is painfully slow to iterate on directly, so the goal is compressing the timeline. Once the localization step points at an actual triggering event, an error-handling path, a retry loop, a specific message type, build a stress harness that fires that trigger at high frequency, simulating a network-drop-and-retry every second instead of waiting for it to happen naturally every few hours, so what took days now shows measurable growth in minutes, letting you bisect fixes quickly instead of waiting days per attempt.

Fixing with confidence

With a stress harness for fast reproduction and per-call-site or per-subsystem accounting for localization, the fix loop becomes: apply a candidate fix, run the harness for a fixed duration, confirm the outstanding-allocation count for that call site is now flat instead of climbing. Keep the harness, and ideally the accounting hooks behind a debug build flag, as a regression test going forward, this class of bug tends to come back silently when someone adds a new error path months later.