Systems Programming & Low-Level Concepts Questions
Systems programming concepts including memory management, pointers, memory layout, CPU architecture considerations, concurrency primitives, OS interactions, and performance optimization in low-level languages (C, C++). Covers how languages expose low-level resources, toolchains, and platform-specific behaviors; excludes high-level application development.
MediumTechnical
51 practiced
A request-handling thread needs to read and update a small shared state object on every request. A mutex keeps the code simple, but latency is too high under load. How would you decide whether to keep the lock, change the data layout, or use a different synchronization strategy?
Sample Answer
**I would choose the synchronization strategy based on correctness first, then latency.** A mutex is still the right answer when several fields must change together and the object has an invariant that must never be observed half-updated. If the critical section is tiny but contention is high, the problem may be the lock, or it may be the layout around the lock.**Decision guide:**- Keep the mutex if updates must be atomic across multiple fields.- Change the data layout if the lock is not the bottleneck but cache contention is.- Use atomics if the state is just one or a few independent values, like counters or a version number.**Concrete example:** suppose the shared object holds `requests`, `errors`, and `last_seen_ms`. I might make `requests` and `errors` atomic counters, but keep a mutex for updating `last_seen_ms` together with other correlated state. That reduces lock traffic without weakening correctness.**What I would check in practice:** contention rate, p95 or p99 latency under load, and whether threads are bouncing the same cache line. If the mutex is simple and the real issue is false sharing, padding or splitting the object can help more than replacing the lock. If the state machine is complex, I would prefer the clear mutex design over a fragile lock-free rewrite.
EasyTechnical
96 practiced
Explain the difference between blocking and non-blocking I/O and list standard Linux APIs for evented IO (select, poll, epoll). For a single-threaded, high-concurrency network server, discuss when SREs should prefer event-driven (epoll) designs over thread-per-connection models.
Sample Answer
Blocking I/O vs Non-blocking I/O- Blocking I/O: a read/write call blocks the calling thread until data is available or the operation completes. Simple semantics but ties up a thread per operation.- Non-blocking I/O: calls return immediately; if no data is available they return an error (EAGAIN/EWOULDBLOCK). Requires readiness notification or polling to know when to retry.Standard Linux APIs for evented I/O- select: oldest, fixed FD limits (FD_SETSIZE), O(n) scan cost.- poll: removes FD_SETSIZE limit, still O(n) per call.- epoll: scalable, O(1) for large numbers of fds, supports edge-triggered or level-triggered modes and efficient kernel-user communication.When SREs should prefer event-driven (epoll) over thread-per-connection- High concurrency (thousands–tens of thousands of concurrent idle/waiting connections): epoll reduces memory and context-switch overhead versus one-thread-per-connection.- I/O-bound workloads with many idle sockets (websockets, long-polling, proxies): event loop maximizes CPU efficiency and predictability for latency SLOs.- Resource-constrained hosts: avoid hitting thread limits, mmap stacks, scheduler overhead.- Need for fine-grained backpressure and multiplexing (e.g., proxying many upstreams).When thread-per-connection may be fine- Low connection counts, primarily CPU-bound per-request work, simpler programming model, or when blocking libraries are unavoidable.Trade-offs and operational considerations- Epoll designs add complexity: careful handling of partial reads/writes, state machines, and edge-triggered pitfalls. Must use non-blocking sockets and implement timeouts/idle eviction.- Debugging and observability: instrument event loop latencies, stack usage, and epoll_wait wakeups.- Hybrid approach: use a small pool of worker threads each running an event loop (sharding) to combine simplicity and scalability.Recommendation: for an SRE operating services with high concurrent idle connections or strict latency/scale targets, default to epoll-based evented architectures; prefer thread-per-connection only when connection counts are small or developer ergonomics and blocking dependencies dominate.
HardTechnical
64 practiced
Design and implement (describe algorithms and memory-ordering) a lock-free multiple-producer multiple-consumer queue in C++11 suitable for passing pointers between threads. Discuss ABA problems, memory reclamation strategies (hazard pointers, epoch reclamation), and required atomic semantics.
Sample Answer
Approach: use the Michael & Scott (1996) lock-free MPMC queue: a singly-linked list with a dummy node, atomic head and tail pointers, and atomic next pointers. Producers push by atomically swinging tail->next then tail; consumers pop by advancing head and reading head->next. Use C++11 atomics with acquire/release semantics to ensure proper visibility. To avoid use-after-free and ABA, pair this with safe memory reclamation — either hazard pointers (per-thread published pointers preventing reclamation) or epoch-based reclamation (grace periods).Code (simplified; assumes a separate HazardPointer or Epoch system for safe reclamation):Key atomic semantics:- Use acquire on loads that synchronize-with subsequent reads; release on stores that publish state. The head CAS uses acq_rel to both read prior state and publish new head safely.- tail advancement is a performance optimization; correctness doesn't require successful CAS.ABA and memory reclamation:- ABA arises when a pointer seen by CAS is removed and a new node is allocated at same address; CAS succeeds incorrectly. Prevent by: - Hazard pointers: each thread publishes the nodes it accesses (head/next). Reclamation defers deletion until no hazard pointer references the node. Pros: deterministic safe reclamation; per-node overhead and scanning retired-list for safe frees. - Epoch (quiescent state) reclamation: threads enter/leave epochs; a node can be freed only after all threads have advanced past the epoch during which node was retired. Pros: low per-access overhead; requires threads to cooperate and periodically advance epochs.- Choose hazard pointers when threads are long-lived and need bounded reclamation; choose epoch for high-throughput low-latency systems where reclamation batching is acceptable.Edge cases and notes:- pop must protect nodes with hazard pointers before dereferencing; otherwise a concurrent free can cause UB.- Destructor here is unsafe if producers/consumers still run.- Memory/order tuning: relaxed loads are acceptable for some reads but must pair with acquire/release where publishing/consuming happens.- Complexity: push/pop are amortized O(1) with lock-free progress (wait-free for single-producer/consumer variants; MPMC is lock-free).- Testing: stress under contention, check for ABA via synthetic reuse, validate under TSAN and valgrind (with appropriate suppression for atomics), and verify reclamation correctness.
cpp
#include <atomic>
#include <cassert>
template<typename T>
struct Node {
std::atomic<Node*> next;
T* value;
Node(T* v=nullptr): next(nullptr), value(v) {}
};
template<typename T>
class MPMCQueue {
std::atomic<Node<T>*> head;
std::atomic<Node<T>*> tail;
public:
MPMCQueue() {
Node<T>* dummy = new Node<T>();
head.store(dummy, std::memory_order_relaxed);
tail.store(dummy, std::memory_order_relaxed);
}
~MPMCQueue() {
while (pop()); // drain (unsafe if threads still exist)
delete head.load();
}
void push(T* item) {
Node<T>* node = new Node<T>(item);
node->next.store(nullptr, std::memory_order_relaxed);
while (true) {
Node<T>* last = tail.load(std::memory_order_acquire);
Node<T>* next = last->next.load(std::memory_order_acquire);
if (last == tail.load(std::memory_order_acquire)) {
if (next == nullptr) {
if (last->next.compare_exchange_weak(next, node,
std::memory_order_release, std::memory_order_relaxed)) {
// try to advance tail (best-effort)
tail.compare_exchange_strong(last, node,
std::memory_order_release, std::memory_order_relaxed);
return;
}
} else {
// tail is behind; advance it
tail.compare_exchange_strong(last, next,
std::memory_order_release, std::memory_order_relaxed);
}
}
}
}
T* pop() {
while (true) {
Node<T>* first = head.load(std::memory_order_acquire);
// protect first with hazard pointer or epoch reservation here
Node<T>* last = tail.load(std::memory_order_acquire);
Node<T>* next = first->next.load(std::memory_order_acquire);
if (first == head.load(std::memory_order_acquire)) {
if (first == last) {
if (next == nullptr) return nullptr; // empty
// tail is behind; try to advance
tail.compare_exchange_strong(last, next,
std::memory_order_release, std::memory_order_relaxed);
} else {
// read value before CAS on head
T* value = next->value;
if (head.compare_exchange_strong(first, next,
std::memory_order_acq_rel, std::memory_order_relaxed)) {
// now safe to reclaim 'first' after ensuring no other thread holds hazard pointer to it
// reclamation: retire_node(first) => hazard-pointer or epoch-managed reclamation
return value;
}
}
}
}
}
};EasyTechnical
47 practiced
You're designing an in-memory index in C++ that stores millions of small objects. Some objects are short-lived, others stay alive for the entire process. How would you decide where to allocate them, and what trade-offs would matter most for latency, fragmentation, and cache locality?
Sample Answer
**I would separate objects by lifetime, not just by type.** Short-lived objects should go into an arena or pool that can be reset in bulk, while long-lived objects should live in a separate pool or the normal heap. That prevents short-lived churn from fragmenting memory used by long-lived data.**Trade-offs:**- Arena or pool allocation is fast and very cache-friendly, because objects are close together in memory.- Bulk release is cheap, but you usually cannot free one object at a time.- General heap allocation is flexible, but it can fragment more and adds allocator overhead.**Concrete example:** if I have `10,000,000` objects at `32` bytes each, that is about `320 MB`. If most of those objects live for only one request, allocating them from a per-request arena keeps the hot data contiguous and avoids millions of individual frees.**What matters most:** latency, fragmentation, and cache locality. For hot lookup paths, contiguous memory usually wins because fewer cache misses happen. For long-lived objects that are accessed often, I would still prefer a stable pool so their addresses stay predictable. If objects are shared across threads, I would also consider thread-local pools to avoid allocator contention. The key is matching the allocator to the lifetime pattern instead of using one strategy for everything.
MediumTechnical
67 practiced
A multi-threaded C service uses pthread_mutexes and intermittently experiences long stalls under load. Describe how you would profile to find mutex contention hotspots and propose practical fixes (code-level and architectural) to reduce contention and CPU waits.
Sample Answer
First, reproduce and measure: run load test that triggers stalls (wrk, ab, or synthetic). Capture system-wide and per-thread data.Profiling steps and tools- perf top / perf record -g to find hot functions and call stacks: perf record -g -F 99 -p <pid> && perf report- perf lock to show mutex hold times and contention: perf lock record -a -p <pid> && perf lock report- perf sched / perf trace for wakeups. Use eBPF tools (bcc/offline) like runqlat, funclatency to see latency sources.- gdb/stack traces: kill -SIGUSR1 to dump pthread stacks or use gdb --batch to sample threads.- mutrace or LockStat (if available) to see per-mutex contention counts.How to analyze- Identify mutexes with high average hold time and many waiters.- Correlate to code paths from perf callstacks to find the exact critical sections.Code-level fixes (examples)- Reduce lock scope: move non-critical work outside mutex.- Replace coarse mutex with finer-grained locks or per-shard locks.- Use atomics for simple counters (libatomic/__sync_bool_compare_and_swap).- Use pthread_rwlock_t for read-heavy cases.- Use trylock to avoid blocking in background-friendly code.- Batch operations or use lock-free queues (e.g., MPMC ring buffer).Example: reduce scope and use atomic incrementOr atomics for simple counter:Architectural fixes- Shard state by consistent hashing so locks are per-shard.- Convert synchronous operations to worker queues: producers push to lock-free queue; worker threads own state and update without global locks.- Introduce batching or coalescing to reduce update frequency.- Add backpressure/rate limiting to prevent overload.- Horizontal scaling: run multiple service instances behind a load balancer so per-instance contention reduces.Validation- Re-run perf and latency tests; monitor mutex hold/wait metrics and tail latency (p99/p999).- Add monitoring: export mutex metrics (contention counts, avg hold time) via custom counters and alert when above thresholds.Trade-offs- Finer-grained locks increase complexity and potential for deadlocks.- Atomics reduce overhead but not suitable for complex invariants.- Sharding and horizontal scaling require state partitioning and possible added operational complexity.This approach finds hotspots with perf, verifies fixes with measurements, and combines code and architecture changes to reduce CPU waits and improve tail latency.
c
// Bad: holds lock while doing logging
pthread_mutex_lock(&m);
shared_counter++;
log_slow_operation(); // slow I/O under lock
pthread_mutex_unlock(&m);
// Better: do minimal protected update, then non-critical work
pthread_mutex_lock(&m);
shared_counter++;
pthread_mutex_unlock(&m);
log_slow_operation(); // no lock heldc
#include <stdatomic.h>
atomic_fetch_add_explicit(&counter, 1, memory_order_relaxed);Unlock Full Question Bank
Get access to hundreds of Systems Programming & Low-Level Concepts interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.