Protocol Vulnerability Analysis Questions

Systematic methods for finding and characterizing vulnerabilities in communication and cryptographic protocols. Topics include identifying authentication and authorization failures, key recovery and key reuse issues, replay and downgrade attacks, man in the middle scenarios, side channel leakage, incorrect use of cryptographic modes and primitives, poor randomness and parameter choices, protocol sequencing and state machine logic errors, and the distinction between theoretical attacks and practical exploitability. Candidates should be able to perform threat modeling for protocols, trace protocol execution to discover logical flaws, analyze published protocol vulnerabilities as case studies, and describe mitigation strategies and secure patching approaches.

MediumTechnical

63 practiced

A JSON Web Token (JWT) based API accepts tokens and uses the 'alg' field in the header to select verification: if alg == 'HS256' use HMAC with a symmetric key; if alg == 'RS256' use RSA public key. Describe how an algorithm confusion vulnerability can arise in this design, how you'd detect it during protocol analysis, and how to fix it at the server implementation level.

Sample Answer

**Brief description of the vulnerability**An algorithm-confusion flaw occurs when the server trusts the JWT header's "alg" field to choose verification logic. If the server supports both HMAC (HS256, symmetric) and RSA (RS256, asymmetric) and uses the same key storage, an attacker can switch algorithms (e.g., set alg=HS256) and have the server verify an attacker-controlled HMAC using the server's RSA public key as the HMAC secret — effectively forging tokens.

**How an attacker exploits it (concrete steps)**- Obtain the server's RSA public key (often public by design).- Create a JWT with the desired claims.- Set header "alg":"HS256".- Sign the token using HMAC-SHA256 with the RSA public key as the secret.- Present token to the server; if the server uses the 'alg' value to select HMAC and uses the public key as the HMAC secret, verification passes.

**How to detect during protocol analysis**- Threat-model endpoints supporting both symmetric and asymmetric algs.- Test cases: send tokens where alg differs from key type (e.g., sign with HMAC using public key, sign with RSA but claim HS algorithm).- Static review: search code paths that switch verification by header without validating key type/usage.- Fuzz inputs: malformed/contradictory headers, "none" algorithm, key-id (kid) mismatches.- Confirm whether public RSA keys are ever used as HMAC secrets in code or config.

**Fixes at server implementation level**- Never trust the JWT "alg" field to choose verification. Instead: - Maintain a mapping of accepted algorithm → key type and key entry (e.g., kid→(algorithm, key material, usage)). - Look up the key (by configured policy or kid) and enforce that the key's algorithm/type matches the token's expected algorithm. Reject tokens where they mismatch. - Prefer explicit server-side policy: only accept RS256 for tokens issued by asymmetric issuers and only HS256 for symmetric issuers; do not allow either on the same endpoint. - Reject "none" algorithm. - Use well-maintained crypto libraries and call the library verification function that takes the expected alg as a parameter (do not allow the library to auto-select alg based on header). - Separate key stores: never store symmetric secrets and asymmetric public keys in the same key slot or reuse public keys as HMAC secrets.

**Additional best practices**- Require and validate "kid", map it to a key with enforced usage flags (sign/verify, algorithm).- Perform automated regression tests that attempt algorithm-swapping attacks.- Log and alert on algorithm mismatch attempts.

HardTechnical

68 practiced

Evaluate the security and operational trade-offs between choosing a deterministic AEAD primitive (e.g., AES-SIV) and a nonce-based AEAD (e.g., AES-GCM) for a messaging protocol that requires forward secrecy, server-side deduplication, and occasional offline message replay. Which would you select and why? Describe migration and interoperability considerations.

Sample Answer

**Short answer / recommendation**Choose a hybrid: preserve forward secrecy with a nonce-based AEAD (e.g., AES-GCM or ChaCha20-Poly1305) for the end-to-end payload using ephemeral keys, and use deterministic primitives (AES-SIV or HMAC-based tokens) only for server-side deduplication metadata or an outer envelope. Pure AES-SIV everywhere sacrifices forward secrecy or leaks message-equality; pure AES-GCM risks nonce reuse catastrophically and makes deduplication harder.

**Trade-offs**- Deterministic AEAD (AES-SIV) - Pros: misuse-resistant to nonce errors; ciphertext equality allows server dedup without storing plaintext or per-message metadata; safe if same key used. - Cons: determinism leaks whether two messages are identical (linkability); incompatible with forward secrecy unless you derive short-lived keys per message (negating dedup benefits); slower (SIV mode double encryption).- Nonce-based AEAD (AES-GCM) - Pros: efficient, widely supported, fits naturally with ephemeral keys from DH for forward secrecy; less linkability when random nonces used. - Cons: catastrophic on nonce reuse (for same key); server cannot deduplicate without exposing deterministic tag; requires careful nonce management and replay protection.

**Design choices / rationale**- Forward secrecy requires ephemeral session keys (e.g., X25519 per session) and AEAD with unique nonces per message. That pushes toward nonce-based AEAD for payload encryption.- Server-side deduplication can be achieved without giving server plaintext by deriving a deterministic, unlinkable token: - Client computes dedup token = HMAC_Kd(message_id || H(message_payload)) where Kd is a long‑lived key the client shares with server for dedup only. Token space can be salted per recipient to limit cross-user linking. - Alternatively, encrypt payload under ephemeral keys, but compute AES-SIV over metadata or a truncated fingerprint for dedup.- Replay: include monotonic counters, timestamps, and short-lived nonces; server maintains replay window. For offline replay, include metadata that enables server to detect replays deterministically (token + time) while not breaking forward secrecy of payload.

**Migration & interoperability**- Versioned protocol header indicating AEAD mode and key derivation method. Allow negotiated fallback.- Rolling migration: deploy server support for both modes. New clients use hybrid scheme; old clients continue until phased out.- Key separation: maintain distinct keys for (a) E2E encryption (ephemeral, forward-secret), (b) dedup tokens (long-lived, deterministic), and (c) server envelopes (optional AES-SIV). Rotate dedup keys regularly; support key IDs in tokens.- Backwards compatibility: server should accept and process both token types; ensure libraries support AES-SIV and AEAD-Nonce modes; provide test vectors.- Security ops: monitor nonce use, audit token exposures, and design revocation/rotation for dedup keys so dedupability can be revoked without re-encrypting payloads (e.g., server can stop honoring tokens).

**Operational notes**- Performance: keep dedup token small (e.g., 32 bytes) and compute server-side indices.- Privacy: deterministic tokens leak equality; reduce linkability via per-recipient salts or blind tokens (client-side OPRF to server).- Failure modes: if dedup key compromised, attacker can link messages — plan incident response and key rotation.

This approach preserves forward secrecy for content, enables safe and private server-side deduplication, and limits the contexts where deterministic encryption is used to minimize linkability and key-management risks.

MediumTechnical

77 practiced

Write, in Python (pseudo-code acceptable), a command-line tool that parses a directory of PCAPs or live traffic to flag TLS ClientHello messages that advertise weak key-exchange algorithms (e.g., static RSA, export-DH, small DH groups). Describe the modules you would use, expected inputs/outputs, and performance considerations for running this at scale.

Sample Answer

**Approach (brief)**- Read PCAP files or live interface, locate TLS ClientHello records, parse CipherSuites and extensions (supported_groups, key_share). Flag ClientHellos that: advertise static-RSA-only key exchange, export ciphers, or small DH groups/weak curves.

**Modules**- scapy (fast packet parsing) or pyshark/tshark for robust TLS decode- dpkt or custom TLS ClientHello parser for fields- multiprocessing, asyncio for scale- sqlite or parquet for results

**Sample (pseudo-)code**

python

# python
from scapy.all import PcapReader
WEAK_CIPHERS = {'TLS_RSA_WITH_*', 'TLS_RSA_EXPORT_*'}
SMALL_DH_BITS = 2048

def is_weak_cipher(cs):
    return cs.startswith('TLS_RSA') or 'EXPORT' in cs

def parse_clienthello(pkt):
    # extract TLS handshake, cipher suites, extensions (pseudo)
    return {'suites': [...], 'groups': [...], 'key_share': [...]}

for p in PcapReader('file.pcap'):
    hello = parse_clienthello(p)
    if not hello: continue
    if any(is_weak_cipher(s) for s in hello['suites']): flag(p)
    if any(group.bits < SMALL_DH_BITS for group in hello['groups']): flag(p)

**Explanation & reasoning**- CipherSuites reveal key-exchange type (RSA static if only RSA key-exchange suites present). Extensions give DH parameters/curves; small/modest primes or legacy curves are weak.- Use pyshark for live capture with BPF filters "tcp port 443" to reduce load.

**Performance & scaling**- I/O bound: use parallel per-pcap file workers, mmap/fast readers.- Apply BPF/tshark display filters to reduce parsing.- For high-volume links use streaming (Kafka) + consumer pool; store results in indexed DB.- Complexity: O(P) packets; parsing cost per TLS handshake is small but CPU-heavy if full crypto inspected.

**Edge cases**- TLS 1.3: key exchange different (no RSA key-exchange); check supported_groups/key_share.- Encrypted ClientHello (ECH): cannot inspect — mark as unknown.

HardTechnical

74 practiced

A fleet of smart meters was shipped with RNG seeded from system time and a process id, generating device keys with low entropy. Describe a step-by-step attack approach to recover private keys at scale given a sample of captured public keys, and then design a remediation and key-rotation strategy that accounts for devices that cannot be physically accessed easily.

Sample Answer

**Attack — step-by-step**

1. Threat model & data: assume attacker has many captured public keys (e.g., device ECDSA secp256r1). Likely seed = f(time, pid). Estimate shipment window and PID range from firmware build info or factory logs.

2. Build entropy model: enumerate plausible time range (± margin), PID space, and any fixed salts. This reduces seed space to feasible size (e.g., 2^30 → 2^20).

3. Reimplement device RNG/keystream exactly (reverse-engineer firmware or infer from sample keys). Validate by reproducing any known ephemeral outputs.

4. Key derivation & verification loop: for each candidate seed, run RNG → derive private key → compute public key → compare to captured set. Stop when match.

Code sketch (Python + ec lib):

python

# pseudocode
for t in time_range:
  for pid in pid_range:
    seed = combine(t, pid)
    priv = derive_key_from_seed(seed)   # mimic device
    pub = point_mul(G, priv)
    if pub in captured_public_set:
      record(seed, priv, device_id_for_pub)

5. Scale: parallelize across GPU/FPGA or cloud workers; shard by time/PID. Use bloom filter for public-set to speed lookup. Precompute tables for common PIDs.

6. Post-recovery exploitation: impersonation, firmware signing bypass if same key used, decrypt past communications.

**Remediation & key-rotation strategy**

- Immediate engineering fixes: - Replace PRNG with a CSPRNG (use hardware TRNG where available, or NIST SP 800-90A DRBG seeded from hardware entropy plus device-unique secret). - Implement deterministic key derivation from an HSM-backed root (HKDF with device unique secret), avoid low-entropy seeds.

- Rotation plan for inaccessible devices: - Signed OTA update mechanism: create new firmware that performs a secure rekey on first boot after update. - Use authenticated delivery: update signed by manufacturer key; device verifies via existing public key (risk: if key compromised, use chained backup) - If existing device keys are compromised, include in OTA a vendor-rotated “bootstrap key” encrypted to a device-specific recovery token derived from immutable hardware ID + manufacturing secret (stored server-side). This prevents attacker replay. - Ephemeral session keys: move long-term keys offline; use long-term identity only for secure enrollment; issue short-lived certificates from manufacturer PKI. - Staged rollout & monitoring: rotate in waves; revoke old keys via CRL/OCSP-like mechanism on backend; block suspicious devices.

- Operational controls: - Revoke and blacklist recovered keys; issue new certificates and require mutual TLS to back-end. - Add logging, anomaly detection for mass rekey/failures. - For future devices: hardware root-of-trust, unique per-device secrets injected in secure environment, supply-chain audits.

Why this works: the attack exploits low entropy and deterministic RNG; reducing seed entropy makes brute-force feasible. The remediation replaces weak randomness with cryptographically sound seeds, provides a secure rekey channel for remote devices, and limits blast radius via short-lived credentials and attestation.

EasyTechnical

57 practiced

In the context of cryptographic protocols, explain the difference between authentication and authorization. Give protocol-level examples where authentication succeeds but authorization fails, and describe how you would detect and mitigate such failures in a system design and during testing.

Unlock Full Question Bank

Get access to hundreds of Protocol Vulnerability Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.