Networking Fundamentals and Protocols Questions

Direct answer
The TCP header carries, at minimum, a sequence number and acknowledgment number (for tracking and confirming data), the SYN/ACK/FIN/RST control flags (for connection setup and teardown), a window size (for flow control), and a set of options including MSS (Maximum Segment Size), window scale, SACK-permitted (Selective Acknowledgment), and timestamps (all negotiated at the handshake). If you could only check a few during a performance incident, window size and the options negotiated at the handshake (MSS, window scale, SACK) are the highest-value first checks, since they directly bound how efficiently the connection CAN perform, before even looking at anything dynamic.

Structured elaboration

• Sequence number: identifies the position, in bytes, of this segment's data within the overall byte stream; every byte sent gets a sequence number.
• Acknowledgment number: when the ACK flag is set, indicates the NEXT byte the receiver expects, effectively confirming everything before that point has arrived.
• Flags (SYN/ACK/FIN/RST): SYN initiates a connection, ACK confirms received data (present on nearly every segment after the handshake), FIN requests a graceful close, RST aborts the connection immediately.
• Window size: the receiver's advertised available buffer space (subject to the negotiated window SCALE factor from the handshake), the mechanism behind flow control.
• Options (MSS, window scale, SACK-permitted, timestamps): negotiated ONLY in the SYN/SYN-ACK exchange and fixed for the connection's lifetime; MSS caps the largest single segment, window scale extends the effective window size beyond the raw 16-bit field, SACK-permitted enables selective (rather than only cumulative) acknowledgment, and timestamps support accurate RTT measurement and protect against stale, wrapped sequence numbers.

Worked example
Triaging a performance incident with limited time, check the negotiated OPTIONS first: if window scale never negotiated successfully (visible by comparing the SYN and SYN-ACK), the connection is capped at a 64KB window for its ENTIRE lifetime regardless of anything else, a hard, structural ceiling worth ruling out before looking at anything dynamic. Then check the CURRENT window size value on live segments (has it collapsed to something small, suggesting a flow-control-limited receiver) alongside the flags (any unexpected RSTs indicating the connection is being torn down and re-established repeatedly, itself a red flag). Sequence and acknowledgment numbers matter most for confirming specific loss/retransmission behavior (comparing them across segments), a more detailed, second-pass check once the higher-level structural questions (options, window, flags) have been ruled out.

Trade-offs & pitfalls
It's easy to over-focus on sequence and acknowledgment numbers first because they feel like "the real data" of TCP's bookkeeping, but for a FIRST-PASS performance triage, the options negotiated once at the handshake (which structurally CAP what the connection can ever achieve) and the live window size (which shows whether that cap is even being approached) are higher-leverage checks, they answer "is there a hard ceiling here" before you spend time analyzing moment-to-moment sequence-level behavior.

Direct answer
An ICMP Type 3 Code 4 message ("Fragmentation Needed and Don't Fragment was Set") is a router along the path telling the sender its packet was too large for the next hop's MTU (Maximum Transmission Unit, here 1400 bytes) and, because the Don't Fragment (DF) bit was set, the router dropped it rather than fragmenting it, expecting the sender to resend at a smaller size. Path MTU Discovery commonly fails in production because something along the path (often a firewall with an overly broad "block all ICMP" rule) discards that very ICMP message before it reaches the sender, so the sender never learns to shrink its packets and its large packets just keep silently disappearing.

Structured elaboration
The DF bit tells every router along the path "do not fragment this packet under any circumstances, if it doesn't fit, drop it and tell me why." Path MTU Discovery relies entirely on that "tell me why" part actually reaching the sender: the sender starts by assuming its LOCAL interface's MTU is usable end-to-end, sends with DF set, and if a router along the path can't forward it at that size, the router sends back exactly this ICMP message, reporting the smaller MTU it needs (1400 bytes here). The sender is then supposed to shrink its packet size to that reported value and retry.

The reason PMTUD commonly fails in production: many firewalls and security appliances, misconfigured to block ALL ICMP as a blanket "security" measure, silently discard the "Fragmentation Needed" message on its way back to the sender. The sender then never learns it needs to shrink its packets, keeps sending at the original (too-large) size with DF still set, and those packets keep getting silently dropped at the same router, forever, with no error ever surfacing to the sender, the classic "large transfers hang, small transfers succeed" symptom (small packets happen to fit under the constrained MTU and sail through fine, while anything larger vanishes without explanation).

Worked example
To reconstruct what happened from the capture: the fragmented IPv4 packets observed likely represent an EARLIER part of the same flow that happened to still get through (perhaps fragmented by an intermediate device before DF took full effect, or from a portion of traffic that didn't have DF set), while the ICMP message with next-hop MTU 1400 is the router's report on a LATER, DF-set packet it could not forward. To fix this for BOTH TCP and UDP traffic: for TCP, the most common resilient fix is MSS clamping on a network device at the edge (rewriting the TCP Maximum Segment Size option in SYN packets passing through, so TCP negotiates a small-enough segment size up front and the oversized-packet problem never occurs at all, PMTUD independent); for UDP, since there's no equivalent MSS negotiation, the application itself must either send appropriately small datagrams from the start or correctly handle PMTUD feedback (which requires NOT blocking the relevant ICMP messages on the path, the actual root-cause fix). In both cases, the truly correct long-term fix is ensuring ICMP "Fragmentation Needed"/"Packet Too Big" messages are explicitly PERMITTED through every firewall along the path, rather than working around their absence.

Trade-offs & pitfalls
MSS clamping is a pragmatic, widely-used workaround specifically because it doesn't depend on ICMP getting through at all, but it only helps TCP; it does nothing for UDP traffic hitting the exact same oversized-packet problem, which is why "block all ICMP" as a firewall policy is a genuinely bad default rather than a harmless-looking hardening step, it breaks a real, load-bearing part of how IP networking is supposed to self-correct.

Direct answer
The bandwidth-delay product (BDP) is the amount of data that can be "in flight" on a link at any instant, bandwidth multiplied by round-trip time, and it's the minimum window size TCP needs to keep the link fully utilized. On a 10 Gbps link with 150ms round-trip time, the BDP is large enough that the original 16-bit TCP window field (max 65,535 bytes) can represent only a tiny fraction of it, which is exactly why window scaling exists.

Structured elaboration
The BDP calculation is:
$\text{BDP (bits)} = \text{bandwidth (bits/s)} \times \text{RTT (s)}$
For a 10 Gbps link ($10 \times 10^9$ bits/s) with a 150ms ($0.150$ s) round-trip time:
$\text{BDP} = 10 \times 10^9 \times 0.150 = 1.5 \times 10^9 \text{ bits} = 187{,}500{,}000 \text{ bytes} \approx 187.5\text{ MB}$
The plain TCP window field is 16 bits, so the largest window it can express without scaling is $2^{16}-1 = 65{,}535$ bytes, about 0.035% of the 187.5 MB needed. Without more window than that, the sender would have to stop and wait for an ACK every 64KB, and at this RTT that caps throughput far below the link's actual 10 Gbps capacity, no matter how fast the link itself is.

The window scale option (negotiated ONLY during the handshake, in the SYN and SYN-ACK) adds a scale factor $S$ (0 to 14) that the receiver applies to its advertised window: the real window becomes $\text{advertised value} \times 2^S$. To cover a 187.5 MB requirement, we need the smallest $S$ such that $65{,}535 \times 2^S \geq 187{,}500{,}000$, which comes out to $S=12$ (giving a maximum representable window of $65{,}535 \times 4096 = 268{,}431{,}360$ bytes, comfortably above the 187.5 MB requirement).

Worked example
For a smaller, more common case, a 100 Mbps link with 100ms RTT, the BDP is:
$100 \times 10^6 \times 0.100 = 1 \times 10^7 \text{ bits} = 1{,}250{,}000 \text{ bytes} \approx 1.25\text{ MB}$
Here the minimum scale factor needed is $S=5$ (giving a max window of $65{,}535 \times 32 = 2{,}097{,}120$ bytes), a much smaller ask than the 10 Gbps case, illustrating that window scaling matters more the higher the bandwidth-delay product climbs, not RTT or bandwidth alone. Beyond window sizing, actually reaching close to line rate on a link like this also typically needs the OS socket buffers (net.ipv4.tcp_rmem/tcp_wmem on Linux) raised to match the negotiated window (a window the OS hasn't allocated buffer space for is wasted), and Selective Acknowledgment (SACK) enabled so a single lost segment somewhere in a large in-flight window doesn't force retransmission of everything after it.

Trade-offs & pitfalls
Window scaling is negotiated ONLY at connection setup; if either endpoint doesn't advertise the option in its SYN, the connection falls back to the un-scaled 64KB ceiling for its entire lifetime, a frequent, hard-to-spot cause of "high-bandwidth link, mysteriously capped throughput" that shows up when an old middlebox strips the option or a misconfigured host has scaling disabled.

Direct answer
When packets are genuinely just arriving out of order (not lost), TCP has built-in tolerance thresholds so it doesn't mistake brief reordering for loss and trigger an unnecessary fast retransmit; PAWS (Protect Against Wrapped Sequence numbers) is a related but distinct mechanism that uses timestamps to detect and discard old, delayed duplicate segments from earlier in a long-running high-speed connection.

Structured elaboration
Modern TCP stacks track how far "out of order" a segment can arrive (measured in either packet count or time) before treating it as evidence of real loss rather than harmless reordering; Linux, for instance, exposes tunables around reordering tolerance that adapt based on the connection's OWN observed reordering history, rather than using one fixed global threshold for every connection. A connection on a path known to reorder packets occasionally (some multi-path or load-balanced routes do this by design) can raise its own tolerance so ordinary reordering doesn't repeatedly trigger unnecessary fast retransmits.

PAWS solves a different, narrower problem specific to very fast connections: on a high-bandwidth link, the 32-bit TCP sequence number space can actually WRAP AROUND within the lifetime of a single connection (send enough gigabytes fast enough and you'll reuse sequence numbers that were valid earlier in the SAME connection). PAWS uses the TCP timestamp option, present on nearly every modern connection, to disambiguate: if a segment arrives whose sequence number would otherwise look valid but whose timestamp is clearly older than the most recent one seen, it's recognized as a stale duplicate from earlier in the connection (possibly delayed en route) and discarded, rather than being misinterpreted as new data that happens to reuse an old sequence number.

Worked example
On a path prone to reordering (say, traffic load-balanced per-packet, not per-flow, across multiple physical links with slightly different latencies), segments can regularly arrive a few positions out of order without any data actually being lost. If TCP's reordering tolerance is too strict, this triggers spurious fast retransmits and duplicate-ACK-driven congestion-window reductions on a connection that never actually experienced loss, hurting throughput for no real reason. An application-level (rather than pure TCP-level) fallback in genuinely reordering-prone environments is to add its OWN sequencing at the application layer, buffering and reassembling out-of-order application messages itself, so the transport layer's retransmission machinery isn't repeatedly triggered by ordinary path-level reordering it can't avoid.

Trade-offs & pitfalls
Raising reordering tolerance too aggressively to avoid spurious retransmits comes at a real cost: on a connection that IS actually experiencing loss (not reordering), a higher tolerance delays how quickly that genuine loss gets detected and retransmitted, directly trading recovery latency for false-positive avoidance. There's no free lunch here; the right tolerance setting depends on knowing, for a specific path, whether reordering or loss is the dominant failure mode.

Direct answer
The TCP three-way handshake establishes a reliable connection before any data flows: the client sends a SYN, the server replies with a combined SYN-ACK, and the client finishes with an ACK. Its job is to let both sides agree on starting sequence numbers and confirm that both directions of the path actually work before committing application data to the wire.

Structured elaboration

1. SYN: the client picks an initial sequence number (ISN, essentially a large pseudo-random 32-bit number) and sends a segment with the SYN flag set and that sequence number. The client moves to the SYN-SENT state.
2. SYN-ACK: the server, if it's listening on that port, picks its OWN initial sequence number, and replies with a segment that has both the SYN flag set (announcing the server's own sequence number) AND the ACK flag set (acknowledging the client's sequence number + 1). The server moves to the SYN-RECEIVED state.
3. ACK: the client acknowledges the server's sequence number + 1 with a plain ACK segment. Both sides now move to ESTABLISHED, and either side may now send data.

Why three steps rather than two: TCP needs BOTH sides' sequence numbers acknowledged, since TCP is full-duplex (both directions need independent sequence tracking). A two-way handshake could confirm only one direction; the third message is what confirms the client's original SYN actually arrived, closing the loop for the client's own sequence space.

Worked example
Suppose a client opens a TCP connection to a web server on port 443. The client sends SYN, seq=1000. The server responds SYN, ACK, seq=5000, ack=1001 (acknowledging the client's SYN by number+1). The client responds ACK, seq=1001, ack=5001. From this point, the client's next data byte will carry sequence number 1001, and the server's next data byte will carry sequence number 5001; each side tracks the OTHER side's sequence space independently via the ACK field of every following segment.

Trade-offs & pitfalls
A frequent mistake is describing the handshake as three round trips; it's actually one and a half round trips of latency, because the SYN-ACK piggybacks the server's SYN onto its ACK of the client's SYN. This is also exactly why TCP always incurs at least one round trip of setup latency before any data can flow, which is the whole motivation behind newer mechanisms like TCP Fast Open that try to send data alongside the very first SYN.