Performance Profiling & Bottleneck Analysis Questions

The four pieces of a reproducible measure-first test

• Baseline: measure the CURRENT state, before any change, using the exact method you'll use afterward, so the before and after numbers are directly comparable.
• Controlled inputs: use a fixed, representative dataset or synthetic workload instead of live, variable production traffic, so that any difference you see afterward is caused by your change, not by different input.
• Isolation: run on a dedicated environment or resource that isn't contending with other jobs, so measurement noise from a neighbor isn't mistaken for a real effect.
• Replayability: the exact same test can be re-run later, by you or anyone else, and produce a comparable result, turning a one-off number into a checkable claim.

Steps to build the test

1. Define the target metric precisely, e.g. "p95 latency (95th percentile response time) of endpoint X under Y requests/sec of synthetic load," not just "make it faster."
2. Capture a baseline run before touching any code, under the same load generator, dataset, and hardware you'll use for every later run.
3. Fix and record everything that could otherwise vary between runs: random seeds, the input data snapshot, machine/instance type, concurrency level, and warm-up duration.
4. Run multiple repetitions, not one, and record the spread (not just a single number), since a single run can easily be noise that a one-off measurement can't distinguish from a real change.
5. Save the harness itself, the script or config that generates the load and measures the result, not just the resulting number, so the exact same test can be replayed later.

Why replayability matters

It's what lets you, a reviewer, or someone investigating a regression months later re-run the identical test and get a directly comparable number, turning "trust me, it got faster" into something checkable. It's also what catches regressions after the fact: the same harness can be dropped into CI or run before each release to confirm a past win hasn't quietly eroded.

Approach

Time and memory instrumentation are cheap individually, but wrapping EVERY call in a hot function with full memory tracing adds real overhead. The production-safe version samples: measure only a random fraction of calls, and only log when something is actually worth seeing (e.g. slower than a threshold), so the fast, common case pays almost nothing.

import time
import tracemalloc
import random
import functools
import threading
import logging

logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("perf")

# tracemalloc's tracing state is PROCESS-GLOBAL, not per-thread: start()/stop()
# affect every thread at once. An RLock (reentrant, so a recursive call from the
# SAME thread doesn't deadlock on itself) serializes only the sampled calls.
_trace_lock = threading.RLock()


def profile(sample_rate=0.1, log_threshold_ms=0.0):
    """Times a function and measures its memory delta, but only on a
    random sample of calls, so overhead stays safe for production."""
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            if random.random() > sample_rate:
                return func(*args, **kwargs)

            with _trace_lock:
                tracemalloc.start()
                start = time.perf_counter()
                try:
                    result = func(*args, **kwargs)
                finally:
                    elapsed_ms = (time.perf_counter() - start) * 1000
                    _, peak = tracemalloc.get_traced_memory()
                    tracemalloc.stop()
                    if elapsed_ms >= log_threshold_ms:
                        logger.info(
                            "%s took %.3fms, peak_traced_memory=%.1fKB",
                            func.__name__, elapsed_ms, peak / 1024,
                        )
            return result
        return wrapper
    return decorator


@profile(sample_rate=1.0)  # 1.0 here only so this demo always logs
def build_lookup(n):
    return {i: str(i) * 4 for i in range(n)}


if __name__ == "__main__":
    random.seed(7)
    for size in (1_000, 50_000):
        build_lookup(size)

Sample output (exact numbers vary by machine, the shape doesn't):

build_lookup took 0.500ms, peak_traced_memory=107.5KB
build_lookup took 26.345ms, peak_traced_memory=7753.5KB

Key points

• time.perf_counter() is used instead of time.time() because it's monotonic (never goes backward, e.g. from an NTP clock adjustment) and has the highest available resolution, exactly what a duration measurement needs.
• tracemalloc gives an accurate peak-memory delta for the wrapped call, but it has to trace every allocation while active, which is not free; that cost is why it's gated behind sample_rate instead of running on every call.
• Sampling (random.random() > sample_rate) means only a configurable fraction of calls pay the tracing cost at all, and log_threshold_ms further avoids flooding logs with entries for calls that were fast anyway.

Making it safe for production

• Keep sample_rate low (1 to 10 percent) on hot paths; it's a statistical view, not a complete record, the same trade-off sampling profilers make generally.
• Prefer functools.wraps so the decorator doesn't silently break introspection (__name__, docstrings) that other tooling or frameworks may depend on.
• tracemalloc.start()/tracemalloc.stop() are PROCESS-GLOBAL, not per-call or per-thread: if two sampled calls run concurrently on different threads, one thread's stop() turns off tracing for the other mid-measurement. Executed against a real multi-threaded workload, the unguarded version doesn't raise an error, it silently reports peak_traced_memory=0.0KB for every thread except whichever one happened to call stop() last, plausible-looking output that is simply wrong. Wrapping the whole start/measure/stop block in the module-level _trace_lock (an RLock, not a plain Lock, so a sampled call that recurses into itself on the same thread doesn't deadlock waiting on a lock it already holds) fixes this: sampled calls across threads are serialized instead of corrupting each other's readings, at the cost of a small amount of added latency on the (already rare) sampled path only.
• If even sampled tracemalloc overhead is too much for a very hot path, fall back to timing only (drop the memory block) and get memory signal from a separate, coarser process-level metric instead (e.g. periodic RSS, Resident Set Size, sampling).

Edge cases

• An exception inside func: the finally block still records and logs the partial timing before the exception propagates, so a slow-then-failing call still shows up in the data instead of disappearing silently.
• Reentrant/recursive calls to the decorated function, on the SAME thread, are handled by using RLock specifically: a plain Lock would deadlock the moment a sampled call recursed into itself, since the thread would block waiting to re-acquire a lock it's still holding. With RLock, the reported peak for an outer call can still be diluted by an inner call resetting the trace when both are sampled; for recursive functions, tracing at the outermost call only would need extra bookkeeping (e.g. a thread-local recursion-depth guard that only starts/stops tracing at depth 0).
• Concurrent calls on DIFFERENT threads: this is the case that matters most for a decorator marketed as "safe in production," since a live service under load runs the decorated function on many threads at once. See the lock above; without it, this class of bug produces confidently-wrong numbers rather than a crash, exactly the kind of silent failure a profiling tool must not have.

1. Capture low-overhead signal continuously, not a full heap dump on a timer

Full heap dumps pause the application and are too heavy to take routinely. Instead, capture cheap periodic snapshots: a class histogram, for example jcmd <pid> GC.class_histogram on the JVM or the equivalent allocation-profiler summary for other runtimes, every few minutes, plus GC (garbage collection, the runtime automatically reclaiming memory that is no longer referenced) log metrics (pause duration, heap size after each collection) continuously. This builds a timeline without the cost of a full dump.

2. Correlate the histogram trend with the pause spikes

Line up the class-histogram snapshots against the GC pause timeline. A class, or a small set of classes, whose instance count and retained size climb steadily across snapshots, right up to when pauses get worse, is your growth suspect.

3. Trigger a full heap dump automatically, only when it matters

Rather than dumping on a schedule, add a watchdog that polls heap usage after each full GC and triggers a dump, for example jcmd <pid> GC.heap_dump, only when post-GC heap usage crosses a threshold, say 80% of max heap, that it does not normally cross. A dump right after a full GC captures what is genuinely still retained, not short-lived garbage that just has not been collected yet.

4. Compare two dumps over time

Take a baseline dump early in the growth window and a second dump once usage has climbed further. Compare their dominator trees, a heap-analysis view showing what object is keeping a chunk of memory alive, to see what is retaining the growth: an unbounded cache, a queue that is not being drained, or listener objects never unregistered are the classic shapes.

5. Reproduce in staging by matching what is actually different about production

Since it does not reproduce under normal synthetic load, the growth almost certainly depends on something production has that staging's load generator does not: a specific request shape, a rare code path, session or connection longevity, or key cardinality, meaning many distinct cache keys instead of a few repeated ones. Build a staging workload generator that specifically replays or mimics that production characteristic, running for hours rather than minutes since production's onset is slow, and watch the same class-histogram timeline in staging.

6. Add allocation guards for next time

Once the retaining structure is identified, for example an unbounded cache, add an explicit size cap or a monitored gauge on it so growth is visible and bounded going forward, rather than discovered again the same way.

Framing

At 10k requests per second, storing a full trace for every request is not feasible for long-term retention, both for storage cost and the write throughput to whatever backend holds them, so the strategy has to separate two things: getting an accurate latency distribution, which can be done cheaply and in-process for every request, from keeping full trace detail, which has to be sampled.

Sampling rate and overhead limits

• For distributed tracing, use head-based sampling, deciding at request start with a documented rate, for example roughly 1 in 100 to 1000 requests, for the bulk of normal traffic, combined with tail-based sampling that always keeps a trace once it is known to be slow or erroring, since head-based sampling alone can miss exactly the requests you care about most.
• Budget tracing and profiling overhead against a hard ceiling agreed up front, for example a low single-digit percent added CPU or latency, and validate that ceiling under load in staging before enabling anything in production; a sampler whose overhead is proportional to the decision, not to full span collection, keeps this cheap even at 10k RPS.

Where to collect

Instrument at service boundaries, the ingress or gateway and each significant downstream call, using an OpenTelemetry-style SDK (a vendor-neutral standard for emitting traces and metrics), so each request produces a small set of spans rather than deep per-line instrumentation.

Aggregating and visualizing hotspots without storing every trace

Maintain a streaming percentile estimator, such as a t-digest or HDR histogram (data structures that approximate percentiles from a stream without keeping every raw value), per endpoint, updated on every request regardless of whether that request's full trace was sampled. This gives an accurate p50/p95/p99 view continuously and cheaply. For "where," periodically aggregate the sampled traces you did keep into a differential or merged flame graph, so hotspots show up as visual width across many requests rather than needing to inspect traces one at a time.

Why this is risky by default

Raw production traffic contains real personal data (PII, personally identifiable information) and, if replayed against systems wired up the normal way, can trigger real side effects: an actual payment charge, an actual email sent to a real customer, an actual downstream webhook fired. A benchmarking setup has to make both of those structurally impossible, not just "be careful."

Capture and scrub

Capture request shape, method, path, headers minus secrets, and body, via a lightweight tap or mirror at the edge, not by copying raw production logs wholesale. Scrub PII at capture time, against a reviewed, explicit field allowlist of which fields are PII, not a best-effort regex sweep, since regex-only scrubbing reliably misses unusual or new fields. Replace sensitive values with format-preserving fakes, for example a fake email that still passes an email-format validator, so downstream code paths still exercise the same validation logic.

Isolate downstream side effects

Point every third-party integration in staging at its sandbox or test mode: a payment gateway's test API keys, which accept requests but never move real money, and an email provider redirected to a local mail-catcher instead of real SMTP delivery. Back this up with an automated network policy or egress rule in staging that blocks any outbound call to a real, non-sandboxed endpoint, so a misconfiguration fails loudly instead of silently charging a card or emailing someone.

Sampling

Replay a stratified sample, by endpoint and response status rather than a uniform random slice, sized to reproduce the real concurrency and request mix, instead of trying to capture and store 100% of traffic.

Synthetic augmentation

For request shapes that matter but are rare in the sample, like large payloads or specific error paths, generate synthetic requests matching production's statistical distribution, payload size and key cardinality, instead of relying only on what happened to be captured.

Validate before trusting the benchmark

Run an automated PII scanner over the final scrubbed and synthetic corpus, and confirm via the egress policy's logs that zero calls reached a real external endpoint during the replay, before treating the benchmark's numbers as trustworthy.