Requirements & approach:- Pack variable-length fields tightly with a 64-bit-aligned wire format for CPU efficiency.- Use a small fixed primary header that contains: magic(8), version(4), presence_bitmap_len(4), presence_bitmap (variable, up to 56 bits in first 64-bit word), then consecutive 64-bit words containing field payloads packed bitwise. Optional fields indicated by bitmap. Backwards-compatible extension via version + TLV extension area after core fields.- Endian on-wire: network byte order (big-endian). Processors use ntoh64 helper to convert.Encoding rules:1. Primary 64-bit word layout (big-endian): - bytes[0]: magic (8) - bits 8-11: version (4) - bits 12-15: bitmap_len_words (4) — how many 64-bit words follow that contain presence bitmap (0..7) - bits 16-63: first presence bitmap bits (48)2. Presence bitmap bit i => field i present. Field definitions have fixed bit-lengths or variable-length encoded with a small length prefix (e.g., 6-bit length) stored adjacent.3. Following bitmap words (if any) complete presence map up to N fields.4. After bitmap, fields are packed sequentially bitwise into 64-bit words (MSB-first). For variable-length fields, store length (6 bits) + payload bits.5. After core fields, an optional TLV extension area: repeated {type(8), length(16), value(variable, byte-aligned)}. Parsers that don't understand a type skip by length.Parsing algorithm (C, low-branch, uses masks & shifts; minimal conditionals — use population counts and pointer arithmetic). Assumes buf points to network-order bytes and buf_len >= 8.c
#include <stdint.h>
#include <string.h>
#include <arpa/inet.h> // for ntohl, ntohs
static inline uint64_t ntoh64(const void *p){
uint32_t hi = ntohl(*(uint32_t*)p);
uint32_t lo = ntohl(*((uint32_t*)p + 1));
return ((uint64_t)hi << 32) | lo;
}
/* Read next n bits (n<=64) from stream (big-endian bit stream) */
typedef struct { const uint8_t *b; size_t len; size_t bit; } bitstream;
static inline uint64_t bs_read(bitstream *s, unsigned n){
uint64_t out = 0;
unsigned need = n;
while(need){
size_t byte_idx = s->bit >> 3;
unsigned bit_off = s->bit & 7;
uint8_t cur = (byte_idx < s->len) ? s->b[byte_idx] : 0;
unsigned take = (unsigned) ( (8 - bit_off) < need ? (8 - bit_off) : need);
uint8_t chunk = (cur << bit_off) >> (8 - take);
out = (out << take) | chunk;
s->bit += take;
need -= take;
}
return out;
}
int parse_packet(const uint8_t *buf, size_t buf_len){
if(buf_len < 8) return -1;
uint64_t w0 = ntoh64(buf);
bitstream s = { buf, buf_len, 0 };
uint8_t magic = (uint8_t)bs_read(&s,8);
uint8_t version = (uint8_t)bs_read(&s,4);
uint8_t bitmap_words = (uint8_t)bs_read(&s,4);
/* read remaining 48 bits of first bitmap word */
uint64_t bitmap = bs_read(&s,48);
for(unsigned i=0;i<bitmap_words;i++){
uint64_t extra = bs_read(&s,64);
bitmap = (bitmap << 64) | extra; /* for up to wide bitmaps; careful with sizes */
}
/* iterate fields: for each set bit, read according to schema; use popcount / scanning */
for(unsigned fid=0; fid < 128; ++fid){
int present = (bitmap >> (127 - fid)) & 1; /* MSB-first mapping */
/* low-branch: multiply mask to zero-out absent reads */
uint64_t len_prefix = 0;
uint64_t value = 0;
/* Example: fields 0..31 fixed 12 bits, fields 32..63 variable with 6-bit len */
if(fid < 32){
/* attempt to read 12 bits; if absent, shift bitstream by 0 (no-op) */
uint64_t v = bs_read(&s, present ? 12 : 0);
value = v;
} else if (fid < 64){
uint64_t l = present ? bs_read(&s,6) : 0;
uint64_t v = present ? bs_read(&s,(unsigned)l) : 0;
(void)l; value = v;
} else break;
/* handle value... */
}
/* After core fields, align to next 64-bit boundary: advance s->bit to ((s->bit+63)/64)*64 */
s.bit = ((s.bit + 63) / 64) * 64;
/* parse TLV extension area byte-aligned */
size_t byte_pos = s.bit >> 3;
while(byte_pos + 3 < buf_len){
uint8_t t = buf[byte_pos];
uint16_t l = (buf[byte_pos+1] << 8) | buf[byte_pos+2];
/* safe skip if beyond buf */
if(byte_pos + 3 + l > buf_len) break;
const uint8_t *v = buf + byte_pos + 3;
/* If unknown type, skip. If known, process. */
byte_pos += 3 + l;
}
return 0;
}
Endian handling:- Wire big-endian. Use ntoh helpers for word-level conversion. Bitstream reads interpret MSB-first within bytes.Versioning & backward compatibility:- version field (4 bits) allows schema changes. Parsers: if version > supported, they parse known subset (presence bitmap): unknown fields are skipped using length rules or TLV skip. Additions should prefer: - New optional fields assigned new bitmap indices. - If new parsing semantics needed, increment version; keep prior format unchanged. - Use TLV extension area for arbitrary future data; older parsers skip unknown TLVs cleanly.Extensions & evolution:- Reserve a range of bitmap indices for experimental fields; if semantics change, deprecate bits and introduce new version.- Keep core mandatory fields stable. Prefer adding optional fields or TLVs rather than changing existing field sizes/positions.- If needing more fields than bitmap size, set bitmap_words >0 to extend presence map without breaking older parsers that interpret extra bitmap bits as zeros.Trade-offs:- Very compact, fast 64-bit-aligned processing at cost of complex bit packing and a slightly larger parser. Low-branch bitstream reduces branching but requires careful bounds checking and testing.Testing & safety:- Provide exhaustive fuzz tests, property tests for endian/bit-order, and backward-compat test vectors for each version.