Networking Fundamentals and Protocols Questions

Direct answer
For large object-storage transfers, the levers worth investigating, roughly in order of impact, are: making sure window scaling and socket buffers are large enough for the path's bandwidth-delay product, enabling NIC-level offloads so the CPU isn't the bottleneck at high throughput, and deciding whether to use several parallel connections or one well-tuned connection based on whether the limiting factor is per-connection window size or something else entirely (like a single flow being unfairly rate-limited by a middlebox).

Structured elaboration
First, confirm the connection's window (after scaling) and OS socket buffers are large enough to cover the path's bandwidth-delay product, undersized buffers here silently cap throughput regardless of how good everything else is (this is the same BDP-sizing exercise as tuning any other high-bandwidth, high-latency transfer). Second, check NIC-level segmentation offloads: TSO/GSO let the OS hand large chunks of data to the NIC and have the NIC itself split them into wire-sized frames, and LRO does the reverse on receive, coalescing many small incoming frames before handing them to the OS; without these, the CPU has to do that segmentation/coalescing work itself, and at multi-gigabit throughput that CPU cost can become the actual bottleneck well before the network link itself is saturated. Third, decide on Nagle's algorithm (which delays sending small writes to coalesce them into fewer, larger segments): for large sequential transfers, Nagle is rarely the bottleneck since writes are already large, but for anything issuing many small writes interleaved with reads, disabling it (TCP_NODELAY) avoids needless latency.

Worked example
The parallel-versus-single-connection decision comes down to WHAT is actually being limited. If a single connection's throughput is capped by its own maximum achievable window (even after correct BDP-based tuning, some paths or middleboxes limit an individual flow's effective window more aggressively than the path's own capacity), opening several parallel connections lets the AGGREGATE throughput exceed what one connection alone could reach, effectively working around a per-flow limit by using multiple flows. But parallel connections add real complexity: more connection-management overhead, more complexity assembling the transferred object back together correctly if it's split across connections, and, on a link SHARED with other traffic, an unfair grab of a disproportionate share of available bandwidth compared to a well-behaved single flow. If the true bottleneck is the raw link capacity itself (not a per-flow cap), splitting into parallel connections doesn't help, and one well-tuned single connection is simpler and just as fast.

Trade-offs & pitfalls
It's tempting to reach for "just add more parallel connections" as a default fix for slow transfers, but that only helps when the limiting factor is genuinely per-connection (a window/RTT/middlebox constraint on a single flow), and on a genuinely bandwidth-saturated shared link, more parallel connections from one client mostly just take a larger, less fair share of that link away from other traffic rather than achieving any real net throughput gain.

Direct answer
For a WAN replication job stuck at 20% of theoretical throughput, walk the transport-layer stack in this order: congestion-control behavior first (is loss even happening, and if so is the algorithm reacting sensibly), then window sizing relative to the path's bandwidth-delay product, then socket buffers, then NIC-level settings, since each of these can independently cap throughput and the cheapest checks come first.

Structured elaboration

1. Congestion-control algorithm and loss: check whether the connection is experiencing any packet loss at all (via retransmit counters, ss -i's retrans field). If there's meaningful loss and the algorithm is a loss-based one like Cubic, and the path has ANY non-congestive baseline loss (common on long WAN paths), the algorithm may be needlessly throttling itself; switching to BBR is a plausible fix here specifically because it doesn't over-react to non-congestive loss the way Cubic does.
2. Window size versus bandwidth-delay product: compute the path's bandwidth-delay product (bandwidth times round-trip time) and compare it to the connection's actual window (ss -i's cwnd and the negotiated window scale). If the window can never grow large enough to cover the bandwidth-delay product, no amount of congestion-control tuning will help, the connection is fundamentally window-limited, not congestion-limited.
3. Socket buffer sizing: even with window scaling negotiated, if the OS's actual send/receive socket buffers (net.ipv4.tcp_wmem/tcp_rmem on Linux) are capped below what the window scale option would otherwise allow, the effective window is capped at the smaller of the two, check both.
4. NIC-level settings: segmentation/offload settings (TSO/GSO/LRO) and interface MTU (Maximum Transmission Unit) affect how efficiently the CPU can push bytes onto the wire; a misconfigured or disabled offload setting can bottleneck a fast link at the CPU rather than the network itself, worth ruling out especially if CPU utilization on the sending host is unexpectedly high relative to the achieved throughput.

Worked example
Suppose the link is rated at 1 Gbps with a 200ms round-trip time. The bandwidth-delay product is 1e9 bits/s times 0.2s = 2e8 bits = 25,000,000 bytes (25 MB). If the connection's actual window, even after scaling, tops out at 5 MB (one-fifth of the required 25 MB), the connection can never exceed roughly one-fifth of the link's rated throughput, which lines up suspiciously well with an observed 20%. That's the single most likely explanation to check FIRST, since it directly predicts the exact ratio being observed, before assuming something more exotic like a congestion-control mismatch.

Trade-offs & pitfalls
It's tempting to jump straight to "switch congestion-control algorithms" as the fix, but that's the wrong first move if the real limiter is window size: no algorithm change fixes a window that's structurally too small for the path's bandwidth-delay product. Always compute the bandwidth-delay product FIRST and check whether the achieved throughput lines up with a window-limited explanation before reaching for an algorithm change.

Direct answer
Encapsulation is the process of wrapping data in a new header (and sometimes trailer) as it moves down the protocol stack, and stripping those headers back off as it moves up the stack on the receiving side. Each layer only understands its own header; it treats everything handed to it by the layer above as an opaque payload.

Structured elaboration
Follow a browser's HTTP request to a web server, header by header, going down the sender's stack:

1. Application layer: the browser produces an HTTP request (method, path, headers, body).
2. Transport layer: TCP wraps that request in a TCP segment, adding a header with source port, destination port, sequence number, acknowledgment number, and flags. The whole HTTP request becomes the segment's payload.
3. Network layer: IP wraps the TCP segment in an IP packet, adding a header with the source IP address and destination IP address. The TCP segment becomes the packet's payload.
4. Data link layer: Ethernet wraps the IP packet in a frame, adding a header with the source MAC address and destination MAC address (and a trailer with a frame check sequence for error detection). The IP packet becomes the frame's payload.
5. Physical layer: the frame is converted to bits and transmitted as electrical, optical, or radio signals.

On the receiving side, decapsulation runs in exactly the reverse order: the physical layer recovers bits into a frame, the data link layer strips the Ethernet header/trailer and hands the IP packet up, the network layer strips the IP header and hands the TCP segment up, the transport layer strips the TCP header and hands the HTTP request up, and the application layer finally sees the original request.

Worked example
Concretely, by the time the original HTTP request bytes reach the wire, they are surrounded by four layers of header (ignoring the physical layer, which isn't a header at all): [Ethernet header][IP header][TCP header][HTTP request bytes][Ethernet trailer]. Each intermediate device on the path (a switch, a router) only needs to look at the headers relevant to its own layer: a switch reads the Ethernet header to decide which port to forward the frame out of; a router strips the Ethernet framing entirely, reads the IP header to decide the next hop, and re-wraps the same IP packet in a NEW Ethernet frame addressed to the next hop's MAC address. The TCP header and the HTTP payload never change as the packet crosses the network; only the Layer 2 framing gets rewritten at each hop.

Trade-offs & pitfalls
The most common confusion is expecting the MAC addresses in the Ethernet header to stay constant end-to-end, they don't. Only the IP addresses (Network layer) stay constant from source to destination; the MAC addresses (Data Link layer) change at every hop, because Ethernet framing is only meaningful on a single link, not across the whole path.

Direct answer
The TCP/IP (Internet) model has four layers: Link, Internet, Transport, and Application. It maps onto the seven-layer OSI model by collapsing OSI's bottom two layers into "Link" and its top three into "Application."

Structured elaboration

Both models describe the same reality; they just carve it up at different granularity. OSI was designed as a general, vendor-neutral reference standard, drawn up before the protocols that would actually win were finalized, so it separates concerns the real internet protocol suite never bothered to split (nothing on the modern internet implements a distinct Session or Presentation layer as its own protocol). TCP/IP is the model that grew directly out of the protocols that were actually built and deployed, so it only has as many layers as the real stack needs to describe.

Worked example
Take a single HTTPS request. In TCP/IP terms: Application layer is HTTPS/TLS+HTTP, Transport is TCP, Internet is IP, Link is Ethernet or Wi-Fi. In OSI terms, the same request spans Layer 7 (HTTP semantics), arguably Layer 6 (TLS's encryption/format role, though in practice TLS libraries sit logically between transport and application), Layer 4 (TCP), Layer 3 (IP), and Layers 2/1 (the local network technology). Neither model is "more correct"; TCP/IP is more useful for describing what's actually running, OSI is more useful for a shared vocabulary when discussing where a NEW piece of functionality should live.

Trade-offs & pitfalls
A common mistake is treating the OSI-to-TCP/IP mapping as one-to-one instead of many-to-one. Session and Presentation don't disappear in TCP/IP, they're just not modeled as distinct layers because no widely deployed protocol maps cleanly onto only one of them. When someone asks "what OSI layer does TLS operate at," the honest answer is "it doesn't map cleanly, and that's expected," not a forced single number.

Direct answer
A port is a 16-bit number identifying an endpoint on a host; a socket is the (IP address, port, protocol) combination that uniquely names one endpoint; and a connection (the 4-tuple, or 5-tuple if you count the protocol) is the full pairing of BOTH endpoints' sockets, source IP, source port, destination IP, destination port. Many clients can share the same server port because what actually distinguishes their traffic is the FULL 4-tuple, not the destination port alone.

Structured elaboration
Port ranges are conventionally split three ways: well-known ports (0-1023, traditionally requiring elevated privilege to bind on Unix-like systems, and reserved by convention for standard services like 443 for HTTPS), registered ports (1024-49151, registered with IANA for specific applications but not privileged), and ephemeral (or dynamic/private) ports (49152-65535 by IANA convention, though many OSes use a wider practical range), which the OS assigns automatically to the CLIENT side of an outgoing connection.

A server listening on port 443 can serve thousands of simultaneous clients because the SERVER's port (443) is only ONE piece of what identifies each connection; every incoming connection also carries a distinct client IP and, typically, a distinct client-side ephemeral port, so the full 4-tuple (client IP, client port, server IP, server port) is what the OS actually uses to demultiplex incoming packets to the right connection, even though the server-side half of that tuple (server IP and port) is identical across all of them.

Worked example
Two different clients, 203.0.113.5 and 203.0.113.9, can both have simultaneous connections to a web server at 198.51.100.1:443. Client A's connection might use ephemeral port 51000 and client B's might use 51000 too (a coincidence, since each client picks its own ephemeral ports independently), yet the server has no trouble distinguishing them, because the full 4-tuples are different: (203.0.113.5, 51000, 198.51.100.1, 443) versus (203.0.113.9, 51000, 198.51.100.1, 443). If client A instead opens a SECOND connection to the same server, its OS will typically pick a DIFFERENT ephemeral port for that second connection (since a client can't have two identical 4-tuples open at once to the same destination), giving something like (203.0.113.5, 51001, 198.51.100.1, 443).

Trade-offs & pitfalls
NAT changes what a receiver actually observes on the wire: a device behind a NAT gateway sharing one public IP will have its ephemeral port REWRITTEN by the NAT device (Port Address Translation) so that multiple internal hosts sharing the same public IP can still be distinguished by the server, meaning the source port a server sees is often not the port the ORIGINAL client actually used, a common source of confusion when correlating server-side logs against client-side application behavior.