Language-Level Memory Management (C/C++/Rust) Questions

Direct answer. An array is a block of N contiguous elements; its name stands for the whole block, so sizeof gives N times the element size. A pointer is one variable holding an address; sizeof gives the pointer width. In most expressions an array name is converted ("decays") to a pointer to its first element, and in a function parameter int a[10] is not an array at all: it is adjusted to int *a. So inside the callee the length is gone and sizeof a is the pointer size. Bugs follow whenever code assumes the callee still knows the length.

#include <stdio.h>
#include <string.h>

static void takes_array_syntax(int a[10]) {
    printf("inside callee: sizeof(a) = %zu (a is really int *)\n", sizeof a);
}
static size_t sum(const int *a, size_t n) {
    size_t s = 0;
    for (size_t i = 0; i < n; i++) s += (size_t)a[i];
    return s;
}
int main(void) {
    int arr[10] = {1,2,3,4,5,6,7,8,9,10};
    int *p = arr;                      /* array decays to &arr[0] */
    printf("in caller: sizeof(arr) = %zu, sizeof(p) = %zu\n", sizeof arr, sizeof p);
    printf("element count = sizeof arr / sizeof arr[0] = %zu\n", sizeof arr / sizeof arr[0]);
    takes_array_syntax(arr);
    printf("p + 1 moves by %td bytes\n", (char *)(p + 1) - (char *)p);
    printf("&arr has type int(*)[10]; &arr + 1 moves by %td bytes\n",
           (char *)(&arr + 1) - (char *)&arr);
    printf("sum = %zu\n", sum(arr, sizeof arr / sizeof arr[0]));
    /* off-by-one overrun through a decayed pointer: the callee cannot know the length */
    printf("reading p[10] (one past the end) is undefined behaviour\n");
    fflush(stdout);
    volatile int bad = p[10];
    (void)bad;
    return 0;
}

Built with gcc -g -O1 -fsanitize=address,undefined s4.c -o s4 (GCC also warns: 'sizeof' on array function parameter 'a' will return size of 'int *' [-Wsizeof-array-argument]) and run on a 64-bit Linux container:

in caller: sizeof(arr) = 40, sizeof(p) = 8
element count = sizeof arr / sizeof arr[0] = 10
inside callee: sizeof(a) = 8 (a is really int *)
p + 1 moves by 4 bytes
&arr has type int(*)[10]; &arr + 1 moves by 40 bytes
sum = 55
reading p[10] (one past the end) is undefined behaviour

then AddressSanitizer stops the program with ERROR: AddressSanitizer: stack-buffer-overflow ... READ of size 4, and its report says the access is at offset 88 of a frame object arr that spans [48, 88): that 88 is the array's own starting offset (48, where ASan placed it in the stack frame, which is bookkeeping, not something you compute) plus its full 40-byte size (10 ints x 4 bytes = 40), so 48 + 40 = 88 is exactly one int past the array's last byte, the first byte p[10] touches.

What differs

Decay and length bugs

Once decayed (the array name is replaced by a plain pointer to its first element), only the address travels. The callee cannot compute the length, so:

• sizeof a / sizeof a[0] inside the callee gives 8/4 = 2 for int *, not 10. This is the classic bug; the compiler warning above flags it.
• Callers must pass the length (size_t n) alongside the pointer, and the callee must check indexes against it. Nothing in C checks bounds for you: p[10] is UB, and usually just reads or overwrites the neighbouring object, which is how overruns become security bugs.
• Use sizeof arr / sizeof arr[0] only in the scope where arr is a real array. Do not apply it to a function parameter, a pointer from malloc, or a string literal's decayed address.

Pointer arithmetic basics

p + 1 advances by sizeof(*p) bytes, not by 1. The run shows 4 bytes for int * and, for &arr + 1, 40 bytes because the pointed-to type is the whole array. a[i] is defined as *(a + i). Pointer arithmetic is defined only within an array object or one element past its end (you may form that one-past address, but not dereference it).

Where this bites in network frame parsing

When a receive buffer is cast to a header struct or walked with p += n, the length that arrived on the wire is attacker-controlled. Treat (pointer, length) as a pair, check remaining >= needed before every field read, and never trust a length field inside the frame to stay within the buffer without comparing it to the bytes actually received. Subtracting or adding to lengths in unsigned types also wraps silently, so check before subtracting.

Pitfall

Passing arr and writing int a[10] in the signature is misleading documentation: the 10 is ignored. Prefer const int *a, size_t n.

Direct answer

Undefined behavior (UB) means the language standard places no requirements on what the program does once an execution hits certain constructs. The program may crash, may appear to work, or may do something that changes from one compiler version or flag to the next. It is especially dangerous with untrusted input because the attacker chooses the values, and the compiler is allowed to optimize on the assumption that UB never happens, so the checks you wrote can disappear and the memory errors that remain can be steered.

What UB is, precisely

The standard does not say "this crashes". It says there are no restrictions on behavior, and compilers are not required to diagnose it (cppreference, "Undefined behavior": https://en.cppreference.com/w/c/language/behavior). Two consequences matter in practice:

1. The compiler reasons backwards from the assumption that UB cannot occur. If a branch can only be reached by overflowing a signed integer, the compiler may treat that branch as dead and delete it.
2. Behavior is not stable. A bug that "works" at one optimization level, on one compiler version, or with one stack layout can break after an unrelated change.

Common UB, why it looks harmless in review, and a mitigation for each

Strict aliasing in one example: the table's char * to uint32_t * cast reads bytes through the wrong type. The defined way is to copy the bytes into an object of the right type. Built as a complete program with gcc -Wall -Wextra (no warnings) in an arm64 Linux container (GCC 14.4.0; little-endian, so the first byte is the least significant), it prints v = 1:

#include <stdint.h>
#include <stdio.h>
#include <string.h>

int main(void) {
    unsigned char bytes[4] = { 1, 0, 0, 0 };
    uint32_t v;
    memcpy(&v, bytes, sizeof v);          /* defined: copy the bytes into a real uint32_t */
    printf("v = %u\n", v);               /* prints: v = 1 */
    return 0;
}

Unsigned arithmetic is the contrast case: it is defined to wrap modulo 2^n, so UINT_MAX + 1 is 0, not UB. Signed overflow is UB and may wrap, trap, or be optimized out (https://en.cppreference.com/w/c/language/operator_arithmetic). That asymmetry is why size and offset math belongs in size_t.

Worked example 1: a guard the compiler removes

#include <limits.h>
#include <stdio.h>

/* Looks like an overflow guard. Signed overflow is UB, so the compiler may fold it (replace it by the constant) to 1. */
int will_not_wrap(int x) { return x + 1 > x; }

int main(void) {
    printf("INT_MAX + 1 > INT_MAX -> %d\n", will_not_wrap(INT_MAX));
    return 0;
}

Compiled and run in a Linux arm64 container with GCC 14.4.0:

gcc -O0 ub.c   -> INT_MAX + 1 > INT_MAX -> 1
gcc -O2 ub.c   -> INT_MAX + 1 > INT_MAX -> 1
gcc -O2 -fwrapv ub.c -> INT_MAX + 1 > INT_MAX -> 0

At -O2 the whole function compiled to mov w0, 1; ret. On arm64, w0 is the 32-bit register that carries a function's return value (the Arm AAPCS64 calling standard uses r0 to r7 for arguments and results), so this reads "put 1 in the return register, then return": it never even looks at x, and it returns true for every input, so the "overflow guard" guards nothing. With -fwrapv (tell GCC signed overflow wraps) the result is the 2's-complement answer, 0. (2's-complement is the usual way machines store signed integers: adding 1 to the largest value gives the most negative one, so INT_MAX + 1 > INT_MAX is false.) The point is not that wrapping is "right", it is that the program's meaning depended on a compiler assumption nobody wrote down. Notice also that on this GCC the check was folded away even at -O0: do not assume "debug builds are safe".

Worked example 2: a null check deleted

#include <stdio.h>

struct pkt { int len; };

/* Dereference first, check later: the check can be deleted. */
int get_len(struct pkt *p) {
    int n = p->len;
    if (p == NULL) return -1;
    return n;
}

int main(void) {
    struct pkt a = { 42 };
    printf("%d\n", get_len(&a));
    return 0;
}

Built with gcc -O2 -S for arm64, get_len came out as just ldr w0, [x0] then ret. x0 holds the first argument, which is p, so ldr w0, [x0] means "load the 4 bytes at the address in p into the return register", that is p->len, and ret returns it. The p == NULL test is gone, because after p->len the compiler may assume p is not null. In a real parser the same shape appears when a length field is read from a header before the header pointer is validated.

Worked example 3: catching it at runtime

#include <limits.h>
#include <stdio.h>

int main(int argc, char **argv) {
    (void)argv;
    int x = INT_MAX - 1 + argc;      /* argc is 1 when run with no arguments: INT_MAX */
    int y = x + argc;                /* INT_MAX + 1: signed overflow, undefined */
    printf("y = %d\n", y);
    return 0;
}

gcc -O1 -fsanitize=undefined ub3.c && ./ub3 prints ub3.c:7:9: runtime error: signed integer overflow: 1 + 2147483647 cannot be represented in type 'int' and then y = -2147483648. With -fno-sanitize-recover=undefined it stops with exit status 1 instead of continuing. -fsanitize=undefined instruments various computations to detect UB at runtime, and -fsanitize=signed-integer-overflow is the specific overflow check (GCC manual, Instrumentation Options: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html).

Detection and prevention toolbox

• Compile-time: -Wall -Wextra (and the compiler's static analyzer, or clang-tidy / Coverity-class tools). They catch only some patterns, so a clean compile is not evidence that the code is free of UB.
• Test-time: build unit tests and fuzz targets (fuzzing feeds the code huge numbers of generated inputs; a fuzz target is the small function that receives each one) with -fsanitize=address,undefined. AddressSanitizer finds out-of-bounds and use-after-free; UBSan finds overflow, bad shifts, misaligned access. Fuzzing with attacker-shaped input is what makes the sanitizers find input-dependent bugs.
• Production hardening (reduces impact, does not remove the bug). Treat the layers above as first priority and these as the last line:
  • -fstack-protector-strong: GCC places a guard value (a canary) next to the return address in functions with local arrays or addresses of locals, and checks it before returning; a stack overflow that overwrites it aborts the program.
  • -D_FORTIFY_SOURCE=2: with optimization on, glibc adds lightweight checks to string and memory functions such as memcpy and strcpy to detect some buffer overflows (per the glibc feature-test-macros man page).
  • A non-executable stack: the memory holding locals cannot be run as code, so injected bytes cannot simply be executed.
  • Address randomization (ASLR): the kernel places the stack, heap and libraries at unpredictable addresses (randomize_va_space in the Linux kernel documentation), so an attacker cannot hard-code them. These turn many overflows into a crash, which is a denial of service rather than code execution, but the underlying UB is still there.
• Language choice: for new parsers of untrusted data, a memory-safe language removes whole classes of the table above.

Constrained environments (embedded, no OS)

Sanitizers need runtime support and memory, which a small device usually lacks. The usual pattern is: keep protocol parsing and buffer logic in plain portable C, compile it on a host with the sanitizers and a fuzzer, then cross-compile the same source for the target with warnings as errors and static analysis. Where you cannot avoid hardware-specific behavior, isolate it in one small file so the rest stays testable.

Trade-offs and pitfalls

• -fwrapv and -fno-strict-aliasing (the second tells the compiler not to assume that pointers of unrelated types point to different memory) are safety nets that make some UB behave predictably; they do not make the code correct, and they cost some optimization. Use them to buy time, then fix the code.
• "It passes the test suite" proves nothing about UB: the optimizer decides, not the test. The same source can behave differently on a new compiler release.
• Do not write "defensive" checks that themselves rely on UB (if (x + 1 < x)). Check the operands before the operation.

Direct answer. A pointer returned from a C function has three different ownership stories depending on where it points: a pointer to a local automatic variable (one declared inside the function with the default, non-static storage, which lives only inside that one call's stack frame: the block of memory set aside for that call's local variables) is always wrong (the storage is gone the instant the function returns), a pointer to a static local variable is valid forever but shared by every caller (not reentrant, meaning a second call before the first caller is done with the result corrupts it; not thread-safe, meaning two threads calling concurrently race on it), and a pointer from malloc is valid until freed and ownership transfers to the caller, who must free it exactly once. Document and enforce which one you have with the function's naming convention, its return-value contract, and, for the heap case, an explicit paired free function.

Case 1: pointer to a local automatic variable. The variable's storage is the function's stack frame, which has automatic storage duration (allocated when the function is entered, deallocated the moment it returns, with no action from the programmer either way). That deallocation happens before the caller even gets the returned pointer back. Any use of that pointer afterward reads memory that may be overwritten by the very next function call. This is never valid and has no legitimate use; the fix is always one of the other two options.

Case 2: pointer to a static local variable. static inside a function gives the variable the same storage duration as a global: it is allocated once and lives for the entire program, so the returned pointer stays valid forever. The catch is there is exactly one instance shared by every call: a second call before you are done with the first call's result overwrites the same memory, and two threads calling concurrently race on it. This pattern is common in legacy APIs (classic ctime, asctime, localtime, which return a pointer into one static buffer; strtok keeps its position in static state) specifically because it needs no caller-side cleanup, but it is not reentrant (safe to call again, from the same or another thread, before a previous call's result is done being used) and not thread-safe, which is exactly why POSIX added _r (reentrant) variants such as ctime_r and strtok_r that keep the state or the result in caller-supplied storage instead.

Case 3: pointer from malloc. The block is valid until explicitly freed, and by convention the function that returns it transfers ownership to the caller: the caller, and only the caller, is responsible for calling free() on it exactly once. This is the most flexible case (each call gets its own independent, arbitrarily-lived block) but also the one with the most ways to get wrong: forgetting to free (leak), freeing twice (double free, undefined behavior), or the caller assuming it does NOT own the pointer when the function's contract actually says it does.

How to document and enforce ownership. Three mechanisms, from weakest to strongest:

• Naming convention, visible at every call site: a function that returns a heap pointer the caller must free is named so that is obvious (*_new, *_create, *_dup), paired with an equally obvious *_free/*_destroy. A function returning a static-local or borrowed pointer avoids an allocation-sounding name so callers do not reflexively free it.
• Comment/doc contract at the declaration, stating explicitly who owns the return value and its lifetime ("caller must free", "valid until the next call to this function", "valid for the lifetime of ctx").
• A paired API plus tooling: C's type system cannot express ownership (a char * looks the same whether or not the caller must free it), so the strongest options are a _create/_destroy pair on an opaque handle type (the caller never sees inside it, so the only way to release it is the matching _destroy), together with tools that catch violations (ASan/LeakSanitizer catch leaked or double-freed blocks at test time; -fanalyzer, a GCC static analysis pass, and Clang's static analyzer can flag some missing-free paths at compile time without even running the code).

Worked example: all four return shapes side by side.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static const char *label_local(int n) {        /* case 1: local automatic array */
    char buf[16];
    snprintf(buf, sizeof buf, "item-%d", n);
    return buf;                                  /* returns address of a dead stack frame */
}
static const char *label_static(int n) {         /* case 2: static local */
    static char buf[16];
    snprintf(buf, sizeof buf, "item-%d", n);
    return buf;
}
static char *label_new(int n) {                   /* case 3: heap, caller frees */
    char *p = malloc(16);
    if (!p) return NULL;
    snprintf(p, 16, "item-%d", n);
    return p;
}
static void label_into(char *out, size_t cap, int n) { /* case 4: caller-provided buffer */
    snprintf(out, cap, "item-%d", n);
}
int main(void) {
    const char *l = label_local(1);
    printf("local ptr=%p\n", (const void *)l);
    const char *a = label_static(1), *b = label_static(2);
    printf("a=%s b=%s same pointer: %s\n", a, b, a == b ? "yes" : "no");
    char *c = label_new(1), *d = label_new(2);
    printf("c=%s d=%s same pointer: %s\n", c, d, c == d ? "yes" : "no");
    free(c); free(d);
    char buf4[16]; label_into(buf4, sizeof buf4, 7);
    printf("buf4=%s\n", buf4);
    return 0;
}

Compiled with GCC 14 (gcc -Wall -Wextra at -O0 and at -O2, and -O1 -fsanitize=address,undefined, Linux arm64, in a container), the compiler says warning: function returns address of local variable [-Wreturn-local-addr] on label_local before the program ever runs, which is case 1 caught by the compiler itself. At run time label_local returned NULL at every optimization level (local ptr=(nil)): GCC itself substitutes a null pointer for the returned address of a local, because returning it is undefined behavior and the compiler may do anything; it is not a stack slot that happened to be reused. ASan reported nothing for the same reason, since no dead-stack address is ever dereferenced. Do not rely on that: a different compiler or version may return the dangling address and let you read garbage, which is the worse failure because it looks like it works. The remaining lines of the same run:

local ptr=(nil)
a=item-2 b=item-2 same pointer: yes
c=item-1 d=item-2 same pointer: no
buf4=item-7

label_static(1) then label_static(2) return the same address, and by the time both results are printed the first call's content has been overwritten (a=item-2 b=item-2 same pointer: yes), demonstrating the shared-buffer hazard directly, not just asserting it. label_new(1) and label_new(2) each get a distinct heap block (c=item-1 d=item-2 same pointer: no), which is correct only because each result is freed exactly once by the caller afterward. label_into (case 4, the caller-provided-buffer style, which sidesteps the whole ownership question because the caller already owns the only buffer involved) fills the caller's own buf4 directly (buf4=item-7), with no allocation and no lifetime question at all, the style you reach for when you want to avoid every pitfall above by construction.

Trade-offs and pitfalls. The static-local pattern's hazard is easy to miss in code review because it looks identical to the heap case at the call site (const char *x = some_func(...)); the only way to know which contract you are under is documentation or the naming convention, which is exactly why that discipline matters more here than in the other two cases. Never let a function sometimes return a heap pointer and sometimes a static or literal pointer depending on a branch: that forces every caller to guess per-call whether to free, which is unfixable by convention alone and should be refactored into two differently-named functions. When in doubt about an unfamiliar API's return value, check the documentation before freeing it speculatively: freeing a pointer that was not heap-allocated (a string literal, a static buffer, a pointer into a caller-owned buffer) is undefined behavior, not a safe no-op.

Direct answer. I would make the boundary a plain C ABI (application binary interface: the binary-level rules for calling and data layout) with a small, versioned table of function pointers, and enforce one rule everywhere: whoever allocates a block releases it, through the same module's code, and every pointer that crosses the boundary has a stated lifetime. Buffers handed in for a call are borrowed for that call only, buffers handed out are either filled into caller-provided memory or allocated through a host-provided allocator, an instance has a creation and a single destruction call, callbacks carry a context pointer and are only valid while the instance lives, and the host never unloads the library before the plugin has been destroyed. Plugins are shared libraries (.so files on Linux): code the host loads at run time with dlopen, finds functions in by name with dlsym, and unloads with dlclose. Nothing C++-specific (exceptions, std::string, STL containers, virtual calls) crosses the boundary, because the plugin may be built with a different compiler, standard library or allocator.

Why the contract has to be this strict

A host and a third-party plugin are separately compiled and share one process and its address space (the memory the process can see), so they may use different C runtimes and different malloc implementations. Freeing a plugin's block with the host's free (or the reverse) can corrupt the heap. A pointer to a stack buffer saved by the plugin for later is a dangling pointer (one that still holds an address after the memory behind it has been released or reused) once the call returns. A callback pointing into a library that has been unloaded jumps into unmapped memory (address space the operating system has taken back, so any access faults). These are the failures the rules prevent.

The interface

#ifndef PLUGIN_API_H
#define PLUGIN_API_H
#include <stddef.h>
#include <stdint.h>

#ifdef __cplusplus
extern "C" {
#endif

#define PLUGIN_ABI_VERSION 1u

enum plugin_status { PLUGIN_OK = 0, PLUGIN_ERR = -1, PLUGIN_TOO_SMALL = -2 };

/* Services the host lends to a plugin. Valid from create() until destroy() returns. */
typedef struct host_api {
    uint32_t size;                                    /* sizeof(host_api) as the host built it */
    void *(*alloc)(void *host_ctx, size_t n);         /* memory the HOST will free, or the plugin frees via free_mem */
    void  (*free_mem)(void *host_ctx, void *p);
    void  (*log)(void *host_ctx, const char *msg);    /* msg borrowed for the call only */
    void *host_ctx;                                   /* passed back on every callback */
} host_api;

/* What the plugin exports. The plugin owns this table; it lives as long as the library stays loaded. */
typedef struct plugin_api {
    uint32_t size;
    uint32_t abi_version;
    void *(*create)(const host_api *host, const char *config);   /* config borrowed; plugin copies what it keeps */
    int   (*process)(void *inst, const uint8_t *in, size_t in_len,
                     uint8_t *out, size_t out_cap, size_t *out_len);
    void  (*destroy)(void *inst);                                 /* exactly once, then inst is dead */
} plugin_api;

/* The only symbol the host looks up by name. Returns NULL if abi_version is unsupported. */
const plugin_api *plugin_get_api(uint32_t host_abi_version);

#ifdef __cplusplus
}
#endif
#endif

The rules, stated as a contract

Host and plugin, built and run

The plugin keeps its state in struct instance. Two lines are easy to miss: in->host = *host copies the host's callback table into the instance, because the host's own table is only borrowed for the create call; and host_api h = in->host in p_destroy copies it back out to a local before freeing, because in is about to be freed and the final free_mem call still needs the callback and host_ctx.

#include "plugin_api.h"
#include <ctype.h>
#include <string.h>

struct instance {
    host_api host;      /* copy: host's table is borrowed, ours is not */
    char *prefix;       /* copy of config, allocated through the host */
};

static void *p_create(const host_api *host, const char *config) {
    if (!host || host->size < sizeof(host_api) || !config) return NULL;
    struct instance *in = host->alloc(host->host_ctx, sizeof *in);
    if (!in) return NULL;
    in->host = *host;
    size_t n = strlen(config) + 1;
    in->prefix = host->alloc(host->host_ctx, n);
    if (!in->prefix) { host->free_mem(host->host_ctx, in); return NULL; }   /* undo partial init */
    memcpy(in->prefix, config, n);
    host->log(host->host_ctx, "plugin: created");
    return in;
}

static int p_process(void *inst, const uint8_t *in_buf, size_t in_len,
                     uint8_t *out, size_t out_cap, size_t *out_len) {
    struct instance *in = inst;
    size_t plen = strlen(in->prefix);
    size_t need = plen + in_len;
    *out_len = need;                                  /* always report the size needed */
    if (out_cap < need) return PLUGIN_TOO_SMALL;      /* nothing written */
    memcpy(out, in->prefix, plen);
    for (size_t i = 0; i < in_len; ++i) out[plen + i] = (uint8_t)toupper(in_buf[i]);
    return PLUGIN_OK;
}

static void p_destroy(void *inst) {
    struct instance *in = inst;
    if (!in) return;
    host_api h = in->host;                            /* copy before freeing the owner of the pointers */
    h.log(h.host_ctx, "plugin: destroyed");
    h.free_mem(h.host_ctx, in->prefix);
    h.free_mem(h.host_ctx, in);
}

static const plugin_api api = { sizeof(plugin_api), PLUGIN_ABI_VERSION, p_create, p_process, p_destroy };

const plugin_api *plugin_get_api(uint32_t v) { return v == PLUGIN_ABI_VERSION ? &api : NULL; }

#include "plugin_api.h"
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static size_t live_allocs;
static void *h_alloc(void *ctx, size_t n) { (void)ctx; live_allocs++; return malloc(n); }
static void h_free(void *ctx, void *p) { (void)ctx; if (p) live_allocs--; free(p); }
static void h_log(void *ctx, const char *m) { printf("[host log:%s] %s\n", (const char *)ctx, m); }

int main(int argc, char **argv) {
    const char *path = argc > 1 ? argv[1] : "./plugin.so";
    void *lib = dlopen(path, RTLD_NOW | RTLD_LOCAL);
    if (!lib) { fprintf(stderr, "dlopen: %s\n", dlerror()); return 1; }
    const plugin_api *(*get)(uint32_t) = (const plugin_api *(*)(uint32_t))dlsym(lib, "plugin_get_api");
    if (!get) return 1;

    printf("ABI 99 request -> %s\n", get(99) ? "accepted" : "refused (NULL)");
    const plugin_api *api = get(PLUGIN_ABI_VERSION);
    if (!api || api->size < sizeof(plugin_api)) return 1;

    host_api host = { sizeof host, h_alloc, h_free, h_log, "main" };
    void *inst = api->create(&host, ">> ");
    if (!inst) return 1;

    uint8_t small[4], big[64];
    size_t need = 0;
    int rc = api->process(inst, (const uint8_t *)"hello", 5, small, sizeof small, &need);
    printf("small buffer: rc=%d, need=%zu\n", rc, need);
    rc = api->process(inst, (const uint8_t *)"hello", 5, big, sizeof big, &need);
    printf("big buffer: rc=%d, out=\"%.*s\"\n", rc, (int)need, big);

    api->destroy(inst);                    /* instance first ... */
    inst = NULL;
    dlclose(lib);                          /* ... library last: api and callbacks' code are gone after this */
    printf("allocations still live after destroy: %zu\n", live_allocs);
    return 0;
}

Built and run in a Linux container (GCC 14.4, aarch64) with the plugin as a shared library and both sides under ASan (AddressSanitizer, which traps invalid memory accesses and frees) and UBSan (UndefinedBehaviorSanitizer, which traps operations the C standard leaves undefined): gcc -O1 -g -Wall -Wextra -fPIC -shared -fsanitize=address,undefined plugin.c -o plugin.so && gcc -O1 -g -Wall -Wextra -fsanitize=address,undefined host.c -o host -ldl && ./host ./plugin.so:

ABI 99 request -> refused (NULL)
[host log:main] plugin: created
small buffer: rc=-2, need=8
big buffer: rc=0, out=">> HELLO"
[host log:main] plugin: destroyed
allocations still live after destroy: 0

Reading the output line by line: the ABI 99 line shows that asking for an unsupported version yields NULL instead of a misread table. The created and destroyed log lines show the host's callback working in both directions across the boundary. small buffer: rc=-2, need=8 shows that a 4-byte output buffer was refused with nothing written, and that the plugin reported the 8 bytes it needs (the 3-byte prefix >> plus 5 input bytes). big buffer shows the retry succeeding with >> HELLO. The host's allocation counter returning to zero shows every block the plugin obtained through the host was returned through the host, and ASan and LeakSanitizer (the leak checker that runs with ASan) stayed silent. To confirm the "unload last" rule is real, swap the order in a copy of the host (dlclose before destroy) and run it: the process dies with a segmentation fault (exit status 139 without sanitizers; under ASan the report is SEGV and the exit status is 1), because destroy points into the unmapped library.

Trade-offs and what flips the design

Isolation decides whether this contract is enough at all; the other points refine it.

• Version evolution. Appending fields to plugin_api and host_api keeps old plugins working, because each side checks size before touching a new field. If you need to break the layout, bump PLUGIN_ABI_VERSION.
• Isolation. In-process plugins share the host's address space, so a buggy or hostile plugin can corrupt the host or leak secrets; the contract only prevents honest mistakes. If plugins are not trusted, put them in a separate process with a message protocol (and shared memory only for bulk data), accepting the copy and latency cost.
• Zero-copy. If large buffers make the copy in process too expensive, add a borrowed-view variant with an explicit lifetime rule ("valid until the next call on this instance") instead of transferring ownership.
• Thread safety and re-entrancy. Callbacks invoked from plugin-created threads need a documented threading rule, and the host must not destroy an instance while a callback is running.
• Testing the contract. Run a conformance harness against every plugin under ASan and LeakSanitizer with the counting allocator above; fail the plugin if the live count is not zero or if it calls anything outside the table.

Direct answer. Confirm it is a true leak (not legitimate cache growth or fragmentation) by checking whether in-use bytes grow without bound under steady traffic while live object counts should be flat; isolate the leaking call site with sampling-based profiling that has low enough overhead to run in production without a full heap dump (a snapshot of every currently-live allocation and its call stack, expensive to capture and ship); roll the fix out through the same canary path (a rollout to a small slice of production traffic first, watched before going wider) and the same rollback path you would use for any production change, verified by watching the same metric you used to confirm the leak.

Confirming it is a leak, not something else. "Memory growth under traffic" has at least three distinct causes that need different fixes, so the first step is telling them apart: (1) a true leak, where allocated-but-unreachable memory accumulates and would never stop growing; (2) a legitimate but unbounded cache or connection pool that the code intends to keep growing (not a bug, a sizing problem); (3) heap fragmentation, where freed memory exists but is too scattered to satisfy new requests, so RSS (resident set size: the physical RAM the process is actually occupying right now, as opposed to memory it has reserved but not touched) grows even though logical in-use bytes do not. mallinfo2() (glibc 2.33+) separates uordblks (bytes actually in use) from fordblks (bytes free but held by the allocator); if uordblks itself climbs without bound under steady-state traffic while your own object counts (sessions, connections, whatever the service tracks) stay flat, that is case (1), a real leak, not (2) or (3).

Isolating leaking allocation sites without full heap dumps. Two lighter options, listed in the order to try first for a production service you cannot easily recompile or restart with debug flags:

• eBPF-based tracing (memleak from the BCC toolkit) attaches to a running PID (process ID) with uprobes (a kernel tracing mechanism that intercepts a function call, here malloc/free, without modifying or recompiling the binary) with no recompilation or restart required, matches allocations against frees, and reports call stacks with memory still outstanding past an age threshold (-o, default 500 ms). Try this first precisely because it needs no rebuild: it supports a sampling rate flag specifically because, per its own example documentation, "memleak may introduce considerable overhead if your application or kernel is allocating and freeing memory at a very high rate," the same overhead concern driving you away from a full heap dump in the first place.
• Sampling allocation profilers (jemalloc's built-in profiler via MALLOC_CONF=prof:true,lg_prof_sample:<N>, or gperftools' heap profiler via the HEAPPROFILE environment variable) record a call stack roughly once per 2^N bytes allocated on average (jemalloc's lg_prof_sample is a base-2 logarithm; its default of 19 is 512 KiB, and jemalloc must be built with --enable-prof) rather than for every allocation, which bounds overhead to a small, tunable fraction while still surfacing the hot allocation sites statistically. Reach for these if the service already links one of those allocators, or if eBPF tracing is not available on the host (older kernel, no BCC installed). Take two sampled profiles minutes apart under live traffic and diff them (pprof --base=<earlier> <binary> <later>, or jemalloc's jeprof equivalent): call stacks whose sampled allocation volume only grows between the two snapshots, never shrinking back, are your leak candidates. A single snapshot cannot show this; the diff is what isolates "grows" from "churns."
  Either way, run it against the live-traffic process you cannot reproduce locally, since the leak is traffic-shape-dependent and a synthetic local load may simply not trigger it.

Worked demonstration. A toy service leaks a response context object on roughly one error path in fifty, storing the pointer in a volatile global on both paths so the compiler cannot prove the allocation unused and delete the malloc/free pair (on the error path the global is then cleared and the block is dropped, so it is genuinely unreachable, which is what makes it a leak):

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <malloc.h>

struct ctx { char buf[1000]; };
static struct ctx * volatile last;

static struct ctx *handle_request(int i) {
    struct ctx *c = malloc(sizeof *c);
    if (i % 50 == 0) {               /* error path, 1 in 50 requests */
        last = c;
        last = NULL;
        return NULL;                  /* BUG: returns without freeing c */
    }
    last = c;
    free(c);                          /* normal path: frees correctly */
    return NULL;
}

int main(void) {
    for (int i = 1; i <= 1000000; i++) {
        handle_request(i);
        if (i % 250000 == 0) {
            struct mallinfo2 mi = mallinfo2();
            printf("after %7d requests: uordblks = %zu KiB\n", i, mi.uordblks / 1024);
        }
    }
    return 0;
}

Running 1,000,000 requests through this and sampling mallinfo2().uordblks every 250,000 requests (GCC 14.4, glibc 2.41, aarch64 Linux container) shows steady, unbounded growth: 1 -> 5 -> 5 -> 5 KiB for a build with the leak branch disabled (change i % 50 == 0 to 0; flat, as expected) versus the leaking build climbing 4922 -> 9848 -> 14770 -> 19692 KiB (roughly 4.8 MiB to roughly 19.2 MiB) in equal 250,000-request steps, confirming linear growth proportional to request count, the signature of a true per-request leak rather than a one-time cache fill: each step adds roughly another 4.8 MiB. Derivation: 250,000 requests per step at 1-in-50 leaking is 5,000 leaked objects per step; each malloc(1000) occupies a 1,008-byte chunk (1,000 bytes plus 8 bytes of allocator header, rounded up to a multiple of 16), so 5,000 x 1,008 = 5,040,000 bytes = 4,922 KiB, exactly the first measured step (the 5,000,000 bytes of payload alone would be 4,883 KiB, slightly under what uordblks reports because it counts the headers). Running the same binary under LeakSanitizer (-fsanitize=leak) reports the exact culprit at program exit (built with gcc -g -fsanitize=leak leak.c, with the program above saved as leak.c). The report, trimmed here to its useful lines (the malloc interceptor frame #0, the frames below main and the addresses are omitted), reads Direct leak of 20000000 byte(s) in 20000 object(s) allocated from: followed by #1 ... in handle_request leak.c:10 (LeakSanitizer reports the 1,000 requested bytes per object, not the 1,008-byte chunk), which is 20,000 leaked objects for 1,000,000 requests at a 1-in-50 leak rate (1,000,000 / 50 = 20,000, exactly matching the injected bug), and gives the precise call site a sampling profiler would only narrow down statistically.

Rolling out the fix safely. Treat this as any production change: fix the specific missing free() on the error path (not a blanket "free more often" change, which risks a double free on a different path), ship it to a canary or small traffic percentage first, and watch the same uordblks-over-time metric (or RSS, with the caveat that RSS also reflects fragmentation and allocator retained pages, which can mask or exaggerate the real signal) stay flat under sustained traffic for at least as long as it previously took to observe growth. Keep the sampling profiler or eBPF trace running through the rollout rather than only before it, since a partial fix that reduces but does not eliminate the leak looks identical to a full fix on a short observation window.

Trade-offs and pitfalls. uordblks does not include blocks served by mmap (glibc counts those in hblkhd), so for a leak of large buffers watch hblkhd as well. A short check:

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <malloc.h>
int main(void) {
    void *p = malloc(1 << 20);
    struct mallinfo2 m = mallinfo2();
    printf("uordblks=%zu hblkhd=%zu\n", m.uordblks, m.hblkhd);
    free(p);
    return 0;
}

Run in the same GCC 14.4 / glibc 2.41 aarch64 container, it prints uordblks=656 hblkhd=1052672 (the 1 MiB request plus its chunk header, rounded up to a whole 4,096-byte page, is 1,052,672 bytes; the small uordblks is whatever the C library itself had allocated at startup and varies by libc version).

LeakSanitizer and a full debug ASan build are not something you run in production under real traffic; they are for reproducing the confirmed leak in a staging environment once you have a lead from the lighter production-safe tools. Do not conflate fordblks (memory the allocator is holding but could give back) with a leak: a process that allocates and frees heavily but never calls something like malloc_trim can show high RSS from retained-but-free pages, which looks alarming but is not the same bug and has a different fix (trim or arena-count tuning, not a code change). Finally, a leak that only manifests under "certain traffic" you cannot reproduce locally is a strong hint that the leaking path is itself an uncommon branch (an error path, a rare request shape, a timeout path): once sampling narrows you to a call site, re-read that specific function for every early return before the matching free, since that is the single most common shape of this bug.