Performance Optimization for Constrained Devices Questions

Direct answer

Keep the real-time audio path out of the shared L2 (level-2) cache entirely, in fast core-local memory, so it never contends for L2 capacity with the other two tasks; hand audio samples from the interrupt to core A through a lock-free single-producer, single-consumer queue; put logging and networking on ordinary L2-backed shared memory with fine-grained per-queue locks; and treat every DMA (direct memory access) buffer with explicit cache clean and invalidate operations, since nothing guarantees the caches are automatically coherent with the DMA engines.

Structured elaboration

Buffer placement.

• Real-time audio (core A): allocate from on-chip TCRAM (tightly-coupled RAM, a small SRAM block wired directly to the core's local bus rather than through the shared L2) or a per-core private memory if the SoC provides one. Use small circular ring buffers, sized to comfortably fit that local memory, for double or triple buffering so DMA transfers can be pipelined.
• Network stack (core C): page-aligned buffers in the shared L2-backed DRAM, managed by a slab allocator (an allocator that pre-carves memory into fixed-size chunks to avoid fragmentation) sized to typical packet MTUs (maximum transmission unit, the largest packet size the network link allows), to keep fragmentation low.
• Logging and telemetry (core B): shared L2, but in a distinct memory region or cache color from the network buffers, so a burst of logging traffic does not evict the audio or network working sets.

Cache management and DMA coherency. Unless the SoC has a hardware-coherent DMA path, the CPU caches and the DMA engines see different views of memory and you must maintain that manually:

• CPU-to-device (e.g. audio samples handed to an output DMA): clean the D-cache (data cache) lines covering the buffer, pushing dirty CPU-cached data out to memory, before starting the DMA transfer.
• Device-to-CPU (e.g. network packets DMA'd in from the MAC): invalidate the CPU's D-cache lines covering the buffer after the DMA completes, so the CPU's next read fetches the fresh data from memory instead of a stale cache line.
• Align every DMA-visible buffer to the cache line size and round clean/invalidate operations up to full cache lines; operating on a partial line risks leaving stale bytes at the edges.
• Wrap this in driver helper functions (the same idea as Linux's dma_map_single/dma_unmap_single) so every call site gets the maintenance right instead of hand-rolling it per buffer.

Synchronization and lock granularity.

• Audio path: a lock-free single-producer, single-consumer ring buffer between the DMA-completion interrupt and core A, using atomic pointer updates with acquire/release memory ordering (a guarantee that a reader performing the acquire sees every write the producer made before its matching release). No mutex on this path; a blocked audio core is an audible glitch.
• Logging and network paths: per-queue spinlocks or lock-free multi-producer queues are fine here, since neither has a hard real-time deadline; protect shared allocators with per-core free lists to cut contention further.
• Give the audio core's ISR (interrupt service routine) the highest interrupt priority so it always preempts logging and network work.

L2 thrash mitigation.

• Keep the audio working set small enough to live entirely in TCRAM or L1, so it never touches L2 in the first place; that is the strongest form of thrash avoidance because there is nothing left to evict.
• Use cache coloring (assigning buffers to specific address ranges so two tasks' data is guaranteed to land in different cache sets and can't evict each other) or allocator-aware placement to keep the network and logging working sets from overlapping in the same L2 sets, if the SoC exposes that control.
• Where available, use hardware memory-QoS (quality of service: prioritized access to a shared resource) or priority settings to bias L2 arbitration toward the audio core's accesses (relevant mainly for any audio metadata that does live in shared memory).

DMA buffer policies. Make buffer ownership explicit and single-directional at any moment: the DMA completion callback marks a buffer safe for the CPU to touch (after invalidate), and the CPU marks it safe for the next DMA transfer (after clean); never let both sides assume they own a buffer at the same time.

The topology looks like this:

flowchart LR
    subgraph CoreA["Core A: audio ISR, real-time"]
        TCRAM["Tightly-coupled RAM: audio ring buffers"]
    end
    subgraph CoreB["Core B: logging / telemetry"]
        LOGQ["Per-core log queue"]
    end
    subgraph CoreC["Core C: network stack"]
        NETQ["Slab-allocated packet pool"]
    end
    L2["Shared L2 cache + DRAM"]
    DMA["DMA engines"]
    CODEC["Audio codec"] --> DMA
    NIC["Network MAC"] --> DMA
    DMA -- clean and invalidate --> L2
    TCRAM -. isolated from .- L2
    LOGQ --> L2
    NETQ --> L2

Worked example

Size the audio ring buffer from a latency budget instead of guessing. Say the audio path is 48kHz, 16-bit, stereo, and the budget is 4ms of buffering:

• Samples per channel in 4ms: 48,000 Hz * 0.004 s = 192 samples.
• Bytes per stereo sample pair: 2 bytes/sample * 2 channels = 4 bytes.
• One buffer: 192 * 4 = 768 bytes.
• Double-buffered (so DMA can fill the next buffer while core A drains the current one): 768 * 2 = 1,536 bytes.

That easily fits inside a typical tens-of-kilobytes TCRAM block with a wide margin left for the rest of the audio pipeline, which is the point of pulling this path out of shared L2 in the first place: the real-time working set is small and bounded, so it should live somewhere no other task can evict it from.

Trade-offs & pitfalls

• Isolating audio into core-local memory only helps if that memory actually exists and is large enough; on an SoC without private per-core SRAM, you are forced to fall back to careful L2 cache coloring instead, which is weaker.
• Skipping a cache clean or invalidate does not corrupt memory content, it makes the CPU's view diverge from the DMA engine's view intermittently, which looks exactly like a rare race condition and is correspondingly painful to debug.
• Fine-grained locks on the network and logging paths reduce contention but add per-lock overhead; if profiling shows contention is actually low, a single coarser lock with less bookkeeping may be simpler and just as fast.
• Cache coloring support, and how coherent the DMA engines are with the caches, both vary a lot by SoC vendor; verify the specific chip's TRM (technical reference manual) rather than assuming either capability exists.

Definitions

The stack is a last-in-first-out region for function call frames: local variables, return addresses, saved registers. It grows and shrinks automatically as functions call and return, allocation and deallocation are an O(1) pointer move, and a variable's lifetime is exactly its enclosing function's scope. The heap is a region for allocations whose size or lifetime isn't known at compile time (malloc/free, new/delete), managed by an allocator that has to search for space matching each request.

Typical sizes on microcontrollers

Total RAM on a small MCU might be 32 to 256 KB. The stack is usually a fixed region reserved by the linker script, commonly a few KB (2-8 KB is typical on small parts). Heap, if used at all, gets whatever's left, and a lot of bare-metal firmware avoids the general heap entirely.

Deterministic behavior

Stack allocation is deterministic: fixed cost, fixed location, automatically freed in reverse order at function return. Heap allocation is generally non-deterministic in embedded contexts, a general-purpose allocator has to walk free lists or find a fitting block, and that search time varies with the current fragmentation state, a real problem for hard real-time deadlines, not just a style preference.

Risks of dynamic allocation

• Fragmentation: repeated allocation and freeing of different sizes leaves the heap full of small gaps that in total have enough space but no single block big enough for the next request, and this gets worse the longer firmware runs without a reboot.
• Non-determinism: an unbounded allocation time is incompatible with hard real-time deadlines.
• No OOM (out-of-memory) recovery story: many small MCUs have no OS-level mechanism to reclaim memory when it runs low, and malloc returning NULL might not even be checked everywhere.

Interrupts and recursion

Every function call, including one an interrupt causes, pushes a new frame onto the stack. If an interrupt fires while deep in a recursive call chain, its frame stacks on top of whatever was already there, all on the same stack unless the RTOS (real-time operating system, a scheduler built to guarantee deadlines) gives each task or interrupt level its own. Recursion is especially dangerous because its depth is often data-dependent (a recursive parser's depth depends on the input), so worst-case stack usage isn't statically boundable the way straight-line code is, an untested deep call can silently overrun the stack region into adjacent memory.

Strategies to avoid heap allocation in firmware

• Prefer static or global allocation sized at compile time.
• Where dynamic-feeling behavior is genuinely needed, use a fixed-size memory pool allocator instead of general malloc/free, bounded and deterministic.
• Avoid recursion, or bound its depth explicitly and test the worst case.
• Check actual stack usage against the reserved size using toolchain support (e.g. GCC's -fstack-usage) rather than assuming it fits.

Definition

Fixed-point representation stores a real number as an integer scaled by an implicit, fixed factor, multiply the real value by 2^n and store the integer result, instead of floating point, which stores a separate exponent so the decimal point can "float."

Why embedded developers use it

Most small MCUs (microcontroller units, the chip running this firmware), especially Cortex-M0/M0+ class and many low-cost 8/16-bit parts, have no hardware FPU (floating-point unit). Floating-point operations there are emulated in software, costing many times more CPU cycles and code (flash) size than an equivalent integer operation. Even on MCUs with an FPU, fixed-point can still win where bit-exact, deterministic results matter, since float rounding behavior can subtly differ across compilers and optimization levels.

Representation, scaling, rounding, overflow

• Representation: a Qm.n format packs m integer bits and n fractional bits into a normal integer type. Q8.8 stores a 16-bit signed value where the low 8 bits represent the fraction, so real value = raw integer / 2^n.
• Scaling: converting to fixed-point is raw = round(value * 2^n); converting back is value = raw / 2^n.
• Multiplication needs rescaling: multiplying two Qn numbers produces a raw product with 2n fractional bits, so you must shift right by n afterward to get back to the original format, and the intermediate product typically needs a wider type (int32 for two int16 Q8.8 operands) to avoid overflow before the shift.
• Rounding: truncating by shifting always rounds toward zero or negative infinity rather than to nearest, a bias that accumulates over many operations. Adding a rounding constant before the shift, (product + (1 << (n-1))) >> n, rounds to nearest instead.
• Overflow: the representable magnitude and precision are both hard limits fixed at compile time, unlike float, which trades precision for range dynamically. An intermediate product that overflows wraps around silently, with no exception, unlike a float overflowing to infinity, which is at least detectable.

Worked example

$\text{raw\_product} = \left\lfloor \frac{a_{raw} \cdot b_{raw}}{2^{n}} \right\rfloor$

Intuition: multiplying two scaled integers doubles the scale factor, so you divide back out by 2^n to land back in the original Qn format. In C this division by a power of two is written as a right shift, (a_raw * b_raw) >> n, one instruction instead of a real divide.

Using Q8.8 (scale factor 256): 3.5 encodes as round(3.5256) = 896, and 2.25 encodes as round(2.25256) = 576. Their raw product is 896576 = 516096 (needs a 32-bit intermediate, it doesn't fit in 16 bits). Rescaling: 516096 >> 8 = 2016. Decoding: 2016/256 = 7.875, matching the true product 3.52.25 = 7.875.

Where it clearly wins

That multiply, on an FPU-less MCU, is one 16x16-to-32 integer multiply and one shift, a handful of cycles. The floating-point equivalent on that same core is typically a software-emulated library call costing many times more cycles and considerably more flash space. For a control loop running at a fixed tick rate, say a PID controller (a common feedback-control loop that continuously corrects toward a target value) ticking every 1 ms, that emulation cost can be the difference between meeting the deadline and missing it.

Framing

This bundles two separate problems: an unbounded RAM ceiling (32 KB won't hold the whole growing dataset indefinitely) and an unbounded time cost (O(n^2) work growing with n). They need different fixes, but on a constrained device they interact, a fix for one alone can still fail on the other.

If the O(n^2) shape comes from all-pairs comparison

Naive duplicate detection or a naive nearest-neighbor scan comparing every element to every other element is the classic source of O(n^2). The standard time fix is a better algorithm for the same problem: sort once (O(n log n)) and compare only adjacent elements, or use a hash-based approach for exact matches (O(n) average), whichever matches what's actually being computed.

If holding the full dataset is what's actually blowing the 32 KB budget

Stop assuming full materialization is possible at all. Process in a streaming or windowed fashion, keeping only a bounded summary or a bounded window of recent elements (a fixed-size ring buffer) rather than the whole growing set. Some algorithms fundamentally need to see all pairs across the entire unbounded history, exact duplicate detection forever, for instance, and if that genuinely can't fit in 32 KB, the honest move is redefining the problem in terms of a bounded window ("duplicates within the last N items"), a real product trade-off to surface explicitly, not something to silently approximate.

Approximate structures as a middle ground

If exact correctness over the full growing set isn't strictly required, a fixed-memory probabilistic structure, a Bloom filter for approximate membership or duplicate checks, gives bounded RAM regardless of how large the dataset grows, at the cost of a known false-positive rate. This directly targets the "dataset keeps growing but RAM doesn't" mismatch in a way plain algorithm-swapping doesn't.

External storage as a last resort

If the device has flash beyond the 32 KB of RAM, spill the working set there and read back only what's needed for each step, a merge-sort-style external algorithm, trading I/O time (flash latency and write endurance) for RAM. Whether this is viable depends on flash wear characteristics and whether the access pattern is sequential enough to suit flash's block-oriented nature.

What guides the choice

The actual required correctness semantics, does the answer really need the full unbounded history, or is a recent window acceptable, how tolerant the application is of approximation (a monitoring or dedup use case may tolerate a Bloom filter's false positives, a billing or safety-critical one may not), and whether flash or external storage is even available to trade against RAM, on the smallest MCUs it may not be.

Framing

With feature cuts off the table, every lever has to come from how the same source code gets compiled and linked, not from removing behavior. I'd work through this roughly in order of size payoff versus risk, checking the binary size after each change so I know what actually moved.

Compiler-level levers

• Optimize for size instead of speed (-Os rather than -O2/-O3): this tells the compiler to prefer smaller code sequences even when a larger one would run a few cycles faster, for example choosing a library call over inlining a loop. This is usually the single biggest lever and often the first thing to try, though the exact savings depend heavily on the codebase.
• Link-time optimization (LTO): normally the compiler only sees one source file at a time, so it can't tell that a small helper function is only ever called from one place, or that two translation units both define near-identical code. LTO defers codegen until link time, when the whole program is visible, enabling cross-file inlining and dead-code elimination that per-file compilation can't do.
• Disable C++ features you aren't using: -fno-exceptions and -fno-rtti remove exception unwind tables and runtime type info, which cost real flash even in code paths that never throw or use dynamic_cast.
• Choose a smaller C library: swapping a full-featured libc for a size-optimized one (for example a "nano" variant), and picking non-float-formatting versions of printf/scanf if the firmware doesn't format floats, both cut fixed library overhead that's otherwise linked in regardless of how little of it you use.

Linker-level levers

• Section-per-function/data plus garbage collection: compiling with -ffunction-sections -fdata-sections puts each function and each global in its own linker section instead of one big blob per file, so the linker's garbage-collection pass (--gc-sections) can drop any section nothing references, function by function rather than all-or-nothing per file.
• Identical code folding: some linkers can detect functions that compile to byte-identical machine code (common with template instantiations or thin wrappers) and merge them into one copy.
• Strip debug info and unwind metadata for the release build: symbol tables and debug sections don't cost flash on target (they're usually excluded from the flashed image already), but stack-unwind tables sometimes do if exceptions were ever enabled anywhere in the link, so confirm that's actually gone once exceptions are disabled.

What I'd check after each change

I'd track the .text/.rodata/.data sizes from the linker map file after each individual change, not just the final total, so I can attribute the savings to a specific lever and catch anything that helped less than expected (or, occasionally, made things worse, aggressive -Os inlining decisions and LTO can sometimes shift code around in ways that are not universally smaller for every function). I'd also re-run whatever timing-sensitive tests exist, since -Os can occasionally hurt performance on a hot path enough to matter, even though it isn't the primary risk here.