Senior Embedded Developer Interview Preparation Guide - Airbnb

Approach summary
I would create controlled microbenchmarks that intentionally exercise the suspected sharing pattern, measure relevant PMU events while toggling sharing, and then apply code/build fixes to remove contention. Below are concrete steps, examples, detection signals and remediation techniques.

Microbenchmark design

• Ping-pong writer test: two threads on separate cores repeatedly write to two variables at high frequency (tight loop, volatile or atomic stores). Measure when vars are placed:
  • In same cache line (intentional false sharing)
  • On separate cache lines (padded)
• Read-modify test: threads repeatedly read-modify-write a struct field each (increment loop).
• Batch test: same as above but with batching (do N ops on local buffer then publish).
• Example (C):

// simple ping-pong writer
volatile uint64_t a, b;
void thread0(){ for(;;) a++; }
void thread1(){ for(;;) b++; }

Build two variants: variables adjacent vs padded with CACHELINE_SIZE.

Hardware counters to collect
Use perf/ARM PMU or platform perf tools. Key events:

• CPU cycles, instructions
• L1/L2/LLC loads & misses (e.g., llc-loads, llc-load-misses)
• Cache references / cache misses (cache-references, cache-misses)
• Coherence/invalidation events (platform-specific: SNOOP/HITM, BUS_ACCESS, COHERENT_READS, REMOTE_INV)
• Store-forwarding or writebacks if available
  Collect while running:
• Variant A (shared cache line)
• Variant B (padded)

Expected signal of false sharing:

• Much higher cache-misses or coherence invalidations in Variant A
• Higher cycles per operation and lower throughput
• Large number of remote invalidations / snoop hits (HITM)

How to interpret

• If operations/sec drops massively and invalidations increase when variables share a line → false sharing confirmed.
• If misses don't change but latency increases, check memory subsystem or locking instead.

Code-level remediation

• Align/pad hot fields to cache line size:
  • struct { uint64_t x; char pad[64 - sizeof(uint64_t)]; };
  • Or use alignas(64) / attribute((aligned(64))).
• Reorganize data: put per-thread state in per-core arrays (one slot per core).
• Reduce write frequency: aggregate updates in thread-local buffer and flush infrequently.
• Use atomics with appropriate memory order (relaxed for non-synchronizing counters) to avoid unnecessary synchronization.
• Replace shared writes with message passing (lock-free queue) or use read-mostly patterns.
• Avoid false-sharing from compiler/linker by declaring variables static and forcing alignment.

Build-level / toolchain strategies

• Compiler attributes: align variables/structs (aligned), or use section placement to separate objects into different cache-line-aligned sections.
• Linker scripts: place hot variables into dedicated sections with proper alignment.
• Optimization flags: ensure inlining or optimization doesn't move hot fields into shared cache lines unexpectedly; use -fno-common if globals are combined.
• Use static analysis or tools (e.g., perf, valgrind/cachegrind, hwloc) to inspect placement.

Validation

• Re-run microbenchmarks and PMU counters after fixes: expect invalidations and cache-misses to drop and throughput/latency to improve.
• Use core pinning (taskset) to eliminate scheduling noise.

This approach gives repeatable detection and a set of practical, low-level fixes suitable for embedded SoCs.

Direct answer

I don't trade the whole release for the new approach on the spot: I separate the release commitment from the capability investment, run a small timeboxed spike to see how much of the uncertainty a limited amount of time can actually remove, and only then decide what, if anything, changes about the release.

Structured elaboration

Running a timeboxed spike rather than deciding from a hunch: a fixed, short window, often a day or less, to find out whether the new approach genuinely holds up on the specific problem, not to fully learn it.

Adopting on a narrow slice first: if the spike looks promising, I'd rather try it on one non-critical path than swap the whole system over mid-sprint, so a wrong bet stays cheap.

Who needs to be in the decision: this isn't a call to make alone once a committed date is at stake; whoever owns that commitment needs to be part of deciding whether to absorb any risk to it.

What's said to stakeholders, and when: early and specific, not after the fact. I'd rather say "here's a real trade-off, here are the two options and what each costs" than let the date slip quietly and explain it only once it's already happened.

Deferring with a concrete follow-up: if the answer is to ship on the existing approach, I don't leave the new one as a vague "later." I make sure there's already a concrete starting point, a branch, a short design note, prepared for the next cycle.

Spreading the exploration so it doesn't depend on one person: where possible, I involve at least one other person in the timeboxed spike itself, not because I'm training them afterward, but so the team's read on whether this is worth pursuing doesn't rest on my judgment alone.

Worked example

Partway through a sprint with a committed date, I found an approach that looked like it would meaningfully help on a specific hot path, but nobody on the team had used it. I ran a half-day timeboxed spike with one other engineer, and it confirmed the approach looked genuinely better there, but doing it properly would take real time we didn't have before the date. I went to the person who owned the release commitment early, laid out the honest trade-off, squeeze it in and risk the date, or ship on the existing approach and take a real run at the new one next cycle, and let them weigh in rather than deciding unilaterally. We shipped on time on the existing approach, and the next cycle started from a design note we'd already written during the spike, not from zero.

Trade-offs and pitfalls

The common failure here is quietly absorbing the new approach into the current sprint and letting the date slip without surfacing the trade-off explicitly to the people depending on it. The opposite failure is a spike too short to be genuinely informative, so the eventual decision ends up driven by excitement about the new approach rather than by evidence from the spike itself.

Direct answer

A crash only after hours, never in a short session, points at accumulation rather than a one-shot bug. On a fixed-budget console with a custom allocator, the leading suspect is fragmentation: total free memory looks fine, but no single block is big enough.

Structured elaboration

• Log the allocator's largest-contiguous-free-block size, not just total free bytes, at intervals during a soak run.
• Fragmentation comes from interleaving long-lived allocations (geometry, persistent state) with many short-lived, variable-sized ones (particles, audio buffers).
• Mitigate with dedicated pools sized per allocation lifetime, or periodic defragmentation at a safe pause point like a level load.
• Validate by rerunning the soak test and confirming the largest-block metric stays stable over hours.

Worked example

$40 \text{ MiB total free} > 10 \text{ MiB requested, yet } 3 \text{ MiB largest block} < 10 \text{ MiB requested}$

Total free memory alone would have looked fine; the largest-block metric is what actually explains the failed allocation.

Trade-offs and pitfalls

Dedicated pools trade flexibility for predictability. Defragmentation by moving live memory requires every pointer to be relocatable via handles, a bigger architectural commitment if not designed in from the start.

What the interviewer probes next

How you'd size pools up front, and what it takes to support safe relocation for defragmentation.

Strategy summary

• Use TCM (64KB) for time-critical code and lookup tables that require lowest latency and deterministic access.
• Use main SRAM (256KB) for stacks, heap, and most DMA buffers.
• Use backup SRAM (16KB, battery-backed) for persistent scratch/config that must survive resets/power loss.

Placement decisions & rationale

• TCM: place interrupt hot-paths, fixed L1 lookup tables, and real-time stacks for hard-RT tasks — lowest latency, no cache misses.
• SRAM: global/static data, main heap, RTOS thread stacks (unless a thread needs worst-case latency -> TCM), and DMA buffers that must be cache-coherent or accessible by DMA controller.
• Backup SRAM: store last-known state, crash counters, calibration data; small size limits use to critical persistent items.

Enforce via linker/attributes

• Linker script sections:

/* Example sections */
.tcm_fast  : { *(.tcm_fast .tcm_fast.*) } > TCM
.bkp_data  : { *(.bkp_data .bkp_data.*) } > BKP_SRAM
.dma_bufs  : { *(.dma_bufs .dma_bufs.*) } > SRAM

• In C:

/* place in TCM */
__attribute__((section(".tcm_fast"))) const uint32_t lut[] = { ... };

/* place in backup SRAM */
__attribute__((section(".bkp_data"))) volatile struct cfg cfg_saved;

/* DMA buffer aligned and non-cacheable region */
__attribute__((section(".dma_bufs"), aligned(32))) uint8_t dma_buf[2048];

Trade-offs

• Latency vs capacity: TCM is fastest but small — reserve only truly timing-critical data. Large tables must remain in SRAM.
• DMA accessibility: DMA typically cannot access cached SRAM safely; either use non-cacheable region, disable cache for DMA ranges, or use cache maintenance (clean/invalidate). Placing DMA buffers in SRAM simplifies DMA but may require cache-management.
• Survivability: Backup SRAM preserves across resets/power with battery, but is tiny; prefer checkpointing critical state only.
• Complexity: More regions increase linker/attribute complexity and testing burden.

Operational tips

• Keep deterministic stacks in TCM for hard real-time ISRs; set stack size conservatively and test worst-case.
• Use compile-time asserts for section sizes; add linker map checks to CI.
• For cache-based MCUs, implement wrappers that perform cache clean/invalidate before/after DMA transfers.

High-level difference

• ISR: hardware-triggered, executes at interrupt context with interrupts (maybe) masked, must be very short and deterministic. No scheduler context switch unless explicitly requested.
• Thread/Task: scheduled by RTOS, runs in thread context with full OS services (blocking, heap, syscalls), preemptible according to priority.

Constraints inside an ISR

• Blocking: never block (no waiting on mutexes, semaphores, normal RTOS waits). Use non-blocking or ISR-safe wake mechanisms.
• RTOS APIs: only call APIs explicitly documented "FromISR" (e.g., FreeRTOS xQueueSendFromISR, xTaskNotifyFromISR). These provide safe context handling and optional context switch flags.
• Heap/alloc: avoid malloc/free unless the allocator is ISR-safe (most are not). Prefer statically allocated buffers or lockless ring buffers.
• Reentrancy: code must be reentrant or protected (ISRs can preempt tasks and lower-priority ISRs). Avoid non-reentrant library calls (e.g., non-thread-safe stdio). Use volatile for shared flags, and memory barriers if required.
• Time/stack: keep ISR short to limit stack usage; avoid large local arrays. Be mindful of IRQ nesting and stack usage per context.

Deferred-work mechanisms

• Task notifications (RTOS-specific, e.g., FreeRTOS task notifications)
  • Very low-overhead, lightweight 32-bit value per task; good for signalling a single consumer or simple event counts. Use when single task handles the work and latency must be low.
• Message queues
  • Safe via FromISR APIs; carry payloads or pointers to buffers. Use when multiple messages/events, buffering, or producer-consumer pattern needed.
• Bottom-halves / Work queues / Deferred procedure calls (DPCs, softirqs, workqueues)
  • Schedule a deferred handler in thread context (worker thread). Good for heavier processing that cannot run in ISR (protocol parsing, disk I/O).
• Semaphores / Binary semaphores from ISR
  • Use to signal a waiting task to run work; suitable when simple wake-up semantics are enough.

When to choose

• If tiny amount of work and single consumer → task notification.
• If you need to queue multiple messages or variable-sized work → message queue.
• If work is expensive or uses blocking APIs (network, filesystem) → bottom-half/worker task.
• If multiple tasks/processors consume events → queue or publish-subscribe pattern.

Example pattern: ISR reads minimal status, copies small descriptor pointer to a preallocated ring buffer or xQueueSendFromISR, then signals a worker task with xTaskNotifyFromISR to process full work.

This approach keeps ISRs deterministic while leveraging RTOS features safely.

Approach (brief)

Declare a volatile pointer to the 32-bit control register at offset 0x10 from PERIPH_BASE, then provide three static inline functions to set, clear, and test a single bit by bit mask.

Register pointer declaration

/* PERIPH_BASE defined elsewhere, e.g. (uintptr_t)0x40000000 */
#define REG_CTRL_OFFSET 0x10
static volatile uint32_t * const REG_CTRL = (volatile uint32_t * const)(PERIPH_BASE + REG_CTRL_OFFSET);

Functions

/* set bit 'bit' (0..31) */
static inline void reg_set_bit(unsigned bit) {
    REG_CTRL[0] |= (1u << bit);
}

/* clear bit 'bit' (0..31) */
static inline void reg_clear_bit(unsigned bit) {
    REG_CTRL[0] &= ~(1u << bit);
}

/* return non-zero if bit is set */
static inline int reg_test_bit(unsigned bit) {
    return (REG_CTRL[0] & (1u << bit)) != 0;
}

Why volatile and barriers

• volatile prevents compiler from caching reads/writes; each access must touch the memory-mapped register.
• On strongly ordered CPUs this may suffice; on weaker-memory architectures or when ordering with DMA/other masters matters, insert explicit memory barriers (e.g., compiler barrier asm volatile("" ::: "memory") and CPU-specific dmb/isb) to ensure ordering and visibility.
• Also consider atomic read-modify-write requirements: use interrupt disable or spinlock if multiple agents can access the register concurrently.

Direct answer

Reverse the whole array, then reverse the first k elements and the remaining n-k elements separately; three linear passes compose into the fully rotated result with no auxiliary array. The same reversal trick generalizes directly: reversing a string in place is the identical two-pointer, swap-from-both-ends routine, and rotating a square matrix 90 degrees in place is a transpose followed by reversing each row, both built on the same in-place-swap primitive as the array rotation.

Structured elaboration

Why three reversals produce a rotation. Reversing the entire array puts every element in fully reversed order. Reversing the first k elements of that reversed array un-reverses exactly the block that should now sit at the front, restoring its original relative order; reversing the remaining n-k elements does the same for the remainder. Normalizing with k %= n handles k values larger than the array's length or equal to zero.

Generalizing to a string. The same in-place two-pointer swap from both ends is exactly what reverses a string, provided the string is held in a mutable container (a list of characters, for example, since Python's own string type is immutable and cannot be reversed truly in place without first converting it).

Generalizing to a square matrix. Transposing swaps matrix[i][j] with matrix[j][i] for every i < j, turning rows into columns. Reversing each row afterward flips left to right. Combined, what was the first column read top to bottom becomes the first row read left to right, which is exactly a 90-degree clockwise turn.

Related in-place-preprocessing techniques (with an honest space caveat). Prefix-sum preprocessing builds an auxiliary array once, in $O(n)$ time, so that any later range-sum query answers in $O(1)$; this trades $O(n)$ extra space for fast queries, so it is not itself an O(1)-extra-space technique, even though it shares this family's "one linear pass, reuse the result" character. Product-except-self, by contrast, genuinely can be done with O(1) extra space beyond the required output array: a first pass fills the output with the running product of everything to each index's left, and a second pass multiplies in the running product of everything to that index's right, needing no separate auxiliary array at all.

Worked example

def rotate_array(nums: list[int], k: int) -> None:
    n = len(nums)
    if n <= 1:
        return
    k %= n
    if k == 0:
        return

    def reverse(i, j):
        while i < j:
            nums[i], nums[j] = nums[j], nums[i]
            i += 1
            j -= 1

    reverse(0, n - 1)
    reverse(0, k - 1)
    reverse(k, n - 1)


def reverse_string_inplace(chars: list[str]) -> None:
    i, j = 0, len(chars) - 1
    while i < j:
        chars[i], chars[j] = chars[j], chars[i]
        i += 1
        j -= 1


def rotate_matrix_90_cw_inplace(matrix: list[list[int]]) -> None:
    n = len(matrix)
    for i in range(n):
        for j in range(i + 1, n):
            matrix[i][j], matrix[j][i] = matrix[j][i], matrix[i][j]
    for row in matrix:
        row.reverse()


if __name__ == "__main__":
    arr = [1, 2, 3, 4, 5, 6, 7]
    rotate_array(arr, 3)
    print(arr)

    chars = list("hello")
    reverse_string_inplace(chars)
    print("".join(chars))

    m = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
    rotate_matrix_90_cw_inplace(m)
    print(m)

Running this prints [5, 6, 7, 1, 2, 3, 4], then olleh, then [[7, 4, 1], [8, 5, 2], [9, 6, 3]].

Complexity

rotate_array: time $O(n)$ for the three reversal passes, since they compose additively into a single linear scan rather than multiplying; space $O(1)$ extra, using only the two index pointers inside each reversal call.
reverse_string_inplace: time $O(n)$, one pass with two pointers closing in from both ends; space $O(1)$ extra beyond the mutable character list itself.
rotate_matrix_90_cw_inplace: time $O(n^2)$ for an n-by-n matrix, since the transpose visits each of the $n^2$ cells once; space $O(1)$ extra, since both the transpose and the row reversals swap in place with no auxiliary matrix.

Edge cases

• k = 0, or k a multiple of the array's length once normalized via k %= n: rotate_array detects this and returns immediately without performing any reversals, since the array is already in its correct rotated position.
• Empty or single-element array or string: both rotate_array (via its n <= 1 guard) and reverse_string_inplace (via while i < j never firing) return immediately with nothing to do.
• A non-square matrix passed to rotate_matrix_90_cw_inplace: this implementation assumes a square matrix, and a non-square transpose changes the matrix's dimensions, so it cannot be rotated true in place this way.

Trade-offs & pitfalls

Forgetting k %= n for a k larger than the array's length either wastes work or, in a careless implementation, indexes out of range. The transpose-then-reverse-rows trick only works for a square matrix: transposing a non-square matrix changes its dimensions, so a genuinely non-square rotation needs a separate output buffer rather than a true in-place transform. Python's string immutability means a real in-place string reversal needs a mutable container (a list of characters, or a bytearray) first; there is no way to mutate a str object's characters directly.

Design summary
I propose a multi-level priority deferred-processing system: per-priority lock-free queues + a global scheduler that enforces preemption of lower-priority deferred work while using quota + aging to prevent starvation. Works with or without RTOS.

Data structures

• Fixed-size circular buffers per priority (0 = highest ... N-1 = lowest), implemented lock-free for ISR enqueue (atomic tail index).
• Per-priority counters: queued_count, processed_count, age_tick.
• Global bitmap of non-empty priorities for fast find-first-set.

Scheduling policy

• Worker runs a priority-select loop:
  • Find highest non-empty priority P.
  • If P higher than current running priority -> preempt (save context) and start P.
  • Execute up to quota[P] tasks (configurable, e.g., more quota for high priorities).
  • After quota exhausted, decrement a global fairness token and service next lower priority.
• Aging: each time a lower priority is skipped, increment its age_tick; when age_tick > AGE_THRESHOLD it temporarily boosts effective priority (or increases quota) to ensure progress.
• Quotas + aging guarantee upper bound on latency for high-priority tasks and bounded wait for low-priority.

Integration

• Without RTOS: run a background worker in main loop or a dedicated softirq thread in interrupt context:
  • ISR only enqueues and triggers worker via software interrupt or wake flag.
  • Worker executes in non-interrupt context; preemption simulated by nesting software interrupts or by breaking processing into small units checked between tasks.
• With RTOS: implement each priority as a task or use a single scheduler task with priority inheritance:
  • High-priority deferred tasks mapped to higher RTOS task priorities; use semaphore/event to wake worker.
  • Use RTOS preemption for true preemptive behavior; still enforce quotas in task code and aging to avoid starvation.

Implementation notes

• Keep ISR work minimal: no malloc, only atomic enqueue and event set.
• Use fixed quotas tuned by latency measurements.
• Protect shared indices with atomic ops or disable interrupts briefly if lock-free not feasible.
• Monitor metrics (max wait, processing time) and adjust quotas/AGE_THRESHOLD.

This design balances deterministic responsiveness for high-priority deferred work and fairness for low-priority tasks while remaining implementable on bare-metal or RTOS platforms.

Purpose of volatile (brief)
volatile tells the compiler that a variable's value can change outside normal program flow so it must not cache in registers or elide accesses. It's essential in embedded firmware to ensure every read/write actually hits memory.

How volatile interacts with optimizations

• Prevents the compiler from optimizing away loads/stores or reordering accesses to that object.
• Does NOT disable other optimizations: it does not make accesses atomic, nor does it enforce CPU-level memory ordering across multiple variables.

Common embedded uses

• Memory-mapped registers:

volatile uint32_t * const UART_STATUS = (uint32_t*)0x40001000;
if (*UART_STATUS & (1<<0)) { /* data ready */ }

• Shared variables modified in ISRs:

volatile bool data_ready = false;
void ISR() { data_ready = true; }

• Buffers written by DMA should be volatile if CPU polls the buffer pointer/flags.

When volatile is insufficient

• Atomicity: volatile doesn't make read-modify-write atomic. Example: incrementing a shared counter in main while ISR also writes -> race. Use atomic primitives or disable interrupts around the update:

__disable_irq();
shared++;
__enable_irq();

or use stdatomic.h / CMSIS atomic helpers.

• Memory ordering on multicore or with DMA: volatile won't issue hardware memory barriers. Use explicit memory barriers (e.g., DMB/DSB on ARM) or C11 atomic operations with memory_order, and ensure cache coherency for DMA (invalidate/clean caches).

Recommendation
Use volatile for visibility of hardware/ISR-driven variables, but combine with: critical sections or atomic APIs for correctness, and platform-specific memory/cpu barriers and cache management for ordering and DMA coherence.

Approach

• Use a small ring buffer protected by a mutex for reader/writer.
• ISR signals a worker via a semaphore/queue to do non-ISR work.
• Expose open/close/read/write with proper locking and sleep/wakeup when empty/full.

Pseudo-C skeleton

// pseudo-C for RTOS (posix-like primitives)
#include <rtos.h>

#define BUF_SIZE 256

static uint8_t buf[BUF_SIZE];
static size_t head=0, tail=0;
static mutex_t buf_lock;
static sem_t data_sem;      // signalled by ISR; worker waits
static cond_t read_wait;    // readers wait when buffer empty
static bool open_count=false;
static thread_t worker_thread;

static inline size_t buf_used() { return (head - tail) % BUF_SIZE; }
static inline size_t buf_free() { return BUF_SIZE - buf_used() - 1; }

void isr_data_ready(void)
{
    // minimal ISR: acknowledge hardware
    hw_ack_interrupt();
    // notify worker (use ISR-safe API)
    sem_give_from_isr(&data_sem);
}

static void worker(void *arg)
{
    while (1) {
        sem_take(&data_sem, WAIT_FOREVER);
        // do deferred processing
        mutex_lock(&buf_lock);
        while (hw_has_data() && buf_free()) {
            buf[head] = hw_read_byte();
            head = (head + 1) % BUF_SIZE;
        }
        mutex_unlock(&buf_lock);
        cond_broadcast(&read_wait); // wake readers
    }
}

int dev_open(void)
{
    if (!open_count) {
        open_count = true;
        mutex_init(&buf_lock);
        sem_init(&data_sem, 0);
        cond_init(&read_wait);
        worker_thread = thread_create(worker, NULL);
    }
    return 0;
}

int dev_close(void) { open_count = false; return 0; }

ssize_t dev_read(uint8_t *dst, size_t len)
{
    size_t copied=0;
    mutex_lock(&buf_lock);
    while (buf_used()==0) {
        mutex_unlock(&buf_lock);
        cond_wait(&read_wait, &buf_lock); // atomically unlock and wait
        mutex_lock(&buf_lock);
    }
    while (copied < len && buf_used()) {
        dst[copied++] = buf[tail];
        tail = (tail + 1) % BUF_SIZE;
    }
    mutex_unlock(&buf_lock);
    return copied;
}

ssize_t dev_write(const uint8_t *src, size_t len)
{
    size_t written=0;
    mutex_lock(&buf_lock);
    while (written < len && buf_free()) {
        buf[head] = src[written++];
        head = (head + 1) % BUF_SIZE;
    }
    mutex_unlock(&buf_lock);
    // optionally notify hardware to send data
    return written;
}

Key points

• ISR does minimal work and uses ISR-safe sem give.
• Worker performs slow operations and moves data to buffer.
• Mutex + condition variable protect and coordinate access.
• Ring buffer avoids copies and supports concurrent readers/writers safely.