Language-Level Concurrency and Multithreading Questions

Direct answer

For a script that polls thousands of sockets (waiting on the network, almost no CPU per socket), use asyncio: one thread runs an event loop (a loop that waits for any of many sockets to become ready, then runs the code waiting on that socket) and handles all of them with little memory and no locking. The code that waits is written as a coroutine: a function declared with async def that can pause at each await (for example await reader.readline()) and hand control back to the loop, which resumes it when the awaited thing is ready. Use threads when the code you must call is blocking and has no async version. Use multiprocessing only when the work is CPU-bound Python and you need several cores.

What the GIL changes

The GIL (Global Interpreter Lock) is a lock in the standard CPython interpreter that lets only one thread execute Python bytecode (the low-level instructions the interpreter runs) at a time. Consequences:

• Threads running pure Python do not run in parallel: two CPU-bound threads take turns.
• A thread waiting on I/O (socket.recv, time.sleep, file reads) releases the GIL, so other threads run while it waits. Threads therefore do help I/O-bound work.
• Some C extensions (NumPy, hashing, compression) release the GIL during heavy work, so threads can speed those up too.
• Processes each have their own interpreter and their own GIL, so they run in parallel across cores, at the cost of process start-up, memory per process, and pickling (serializing an object to bytes with Python's pickle module so it can cross a process boundary) every argument and result.
• asyncio is single-threaded, so the GIL is not even a factor; the cost is that any blocking call inside a coroutine freezes the entire loop. A tiny demonstration: a ticker prints every 0.2 s while another coroutine does time.sleep(1.0) (a blocking call) after 0.3 s:

import asyncio, time

async def ticker(t0):
    for _ in range(5):
        print(f"tick at {time.perf_counter() - t0:.1f} s")
        await asyncio.sleep(0.2)

async def blocker(t0):
    await asyncio.sleep(0.3)
    time.sleep(1.0)            # blocking call: freezes the whole loop

async def main():
    t0 = time.perf_counter()
    await asyncio.gather(ticker(t0), blocker(t0))

asyncio.run(main())

A sample run in a python:3.12-slim container printed:

tick at 0.0 s
tick at 0.2 s
tick at 1.3 s
tick at 1.5 s
tick at 1.7 s

The ticks at 0.4 s to 1.2 s are missing: the whole loop stood still for the second the blocking call held it. Replacing time.sleep(1.0) with await asyncio.sleep(1.0) in blocker gave ticks at 0.0, 0.2, 0.4, 0.6 and 0.8 s with no gap. (Exact tick times shift by a few hundredths of a second between runs; the missing stretch is what matters.)

Status of the GIL (this is background depth; a PEP is a Python Enhancement Proposal, the design document for a language change): PEP 703 added an optional free-threaded build (no GIL) of CPython, experimental in 3.13. PEP 779 (status Final) made that build officially supported but still optional from Python 3.14; the default build still has the GIL, and extension compatibility and single-thread speed are the trade-offs (the 3.14 release notes put the single-thread penalty at roughly 5 to 10 percent, depending on platform and compiler). So do not design around a GIL-free interpreter being present: on Python 3.13 and later, sys._is_gil_enabled() tells you whether the interpreter you will deploy to is running with the GIL.

Measured comparison

import asyncio, time, threading
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def cpu_work(n):
    s = 0
    for i in range(n):
        s += i * i
    return s

def io_work(_):
    time.sleep(0.5)          # stands in for a blocking network call; releases the GIL

async def aio_work(_):
    await asyncio.sleep(0.5)

def timed(label, fn):
    t = time.perf_counter(); fn(); print(f"{label:<28}{time.perf_counter() - t:6.2f} s")

def main():
    N, W = 4_000_000, 4
    print("CPU-bound, 4 jobs of 4M iterations")
    timed("  sequential", lambda: [cpu_work(N) for _ in range(W)])
    with ThreadPoolExecutor(W) as ex:
        timed("  4 threads", lambda: list(ex.map(cpu_work, [N] * W)))
    with ProcessPoolExecutor(W) as ex:
        list(ex.map(cpu_work, [1] * W))    # warm up the worker processes
        timed("  4 processes", lambda: list(ex.map(cpu_work, [N] * W)))
    print("I/O-bound, 40 waits of 0.5 s")
    timed("  sequential", lambda: [io_work(0) for _ in range(40)])
    with ThreadPoolExecutor(40) as ex:
        timed("  40 threads", lambda: list(ex.map(io_work, range(40))))
    async def many(): await asyncio.gather(*(aio_work(i) for i in range(40)))
    timed("  asyncio, 40 tasks", lambda: asyncio.run(many()))

if __name__ == "__main__":
    main()

One sample run, Python 3.12.15 in a python:3.12-slim container on a 14-core aarch64 host:

CPU-bound, 4 jobs of 4M iterations
  sequential                  0.42 s
  4 threads                   0.41 s
  4 processes                 0.15 s
I/O-bound, 40 waits of 0.5 s
  sequential                 20.04 s
  40 threads                  0.51 s
  asyncio, 40 tasks           0.51 s

Reading it: ex.map(f, items) runs f on every item in the pool and returns results in order. The warm-up line sends tiny jobs first so process start-up is not counted in the timing, and asyncio.gather(*coros) starts all the coroutines at once and waits for all of them. For CPU-bound work, threads gave no speed-up (0.41 s against 0.42 s sequential) because of the GIL, and processes were about 2.8 times faster (0.15 s; with 4 workers the ideal would be 4 times, and the gap is process overhead on a short job). That speed-up needs at least 4 free cores: the same script in a container limited to 1 CPU showed the 4 processes no faster than sequential. For waiting, threads and asyncio both collapse 40 waits of 0.5 s to the length of one wait, 0.51 s. Timings vary run to run; the pattern is the finding.

Polling thousands of sockets

import asyncio, resource, time

N = 3000   # concurrent connections; 3000 client + 3000 server sockets = 6000 descriptors

# Many shells start with a soft limit of 1,024: raise it as far as the hard limit allows.
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
if soft < 8192:
    resource.setrlimit(resource.RLIMIT_NOFILE, (8192 if hard == resource.RLIM_INFINITY else min(8192, hard), hard))

async def handle(reader, writer):
    data = await reader.readline()
    writer.write(data.upper()); await writer.drain()
    writer.close()

async def client(port, i, ok):
    r, w = await asyncio.open_connection("127.0.0.1", port)
    w.write(f"ping {i}\n".encode()); await w.drain()
    reply = await r.readline()
    ok.append(reply.startswith(b"PING"))
    w.close()

async def main():
    server = await asyncio.start_server(handle, "127.0.0.1", 0, backlog=N)
    port = server.sockets[0].getsockname()[1]
    ok = []
    t = time.perf_counter()
    await asyncio.gather(*(client(port, i, ok) for i in range(N)))
    print(f"{len(ok)} of {N} sockets answered, all correct: {all(ok)}, {time.perf_counter() - t:.2f} s, one thread")
    server.close()

print("fd limit:", resource.getrlimit(resource.RLIMIT_NOFILE)[0])
asyncio.run(main())

In the same container (soft limit already 20,480, so no raise was needed) it printed:

fd limit: 20480
3000 of 3000 sockets answered, all correct: True, 0.26 s, one thread

Walk through the program. asyncio.start_server(handle, host, 0, backlog=N) listens on a free port (0 lets the OS choose) and runs handle as a new coroutine for each accepted connection; backlog is how many not-yet-accepted connections the OS may queue, raised so a burst of 3,000 is not refused. reader.readline() waits for one line without blocking the loop. writer.write(...) queues bytes and await writer.drain() waits until the send buffer has room. asyncio.open_connection is the client side and returns a reader and writer pair. The 3,000 client coroutines run concurrently under gather. That is 3,000 client connections and 3,000 matching server connections (6,000 file descriptors) served from one thread on loopback. The file-descriptor limit printed first matters: each socket uses one file descriptor (the small integer the OS uses to name an open file or socket), and many systems default to 1,024, so a script that polls thousands of sockets must raise ulimit -n (the shell command that sets the per-process descriptor limit) or call resource.setrlimit, as the program above does, regardless of the concurrency model. Without that, this program fails with OSError: [Errno 24] Too many open files when the soft limit is 1,024. Loopback with an instant echo is the easy case; against real remote servers the elapsed time is dominated by network latency, which is exactly what concurrency hides.

Choosing, with the numbers

For the socket-polling script, recommend asyncio with a cap on concurrent connections (an asyncio.Semaphore) and timeouts (asyncio.timeout, available from Python 3.11, or asyncio.wait_for, which also works on older versions) so one dead peer cannot hang the run. On Linux, asyncio uses epoll (a kernel facility that reports which of many sockets are ready without scanning them all) through the selectors module (Python's wrapper over such facilities), whereas a hand-written select() loop cannot watch descriptors numbered 1,024 or higher (the FD_SETSIZE limit, a fixed size of the descriptor set that select() accepts). You do not need these names to use asyncio; they explain why it scales. If part of the job is blocking or CPU-heavy (parsing large replies), hand that part to loop.run_in_executor or asyncio.to_thread, so the loop keeps answering sockets. What would flip the choice: an existing blocking client library with no async version (threads), or heavy per-reply computation (add a process pool behind the loop).

The models side by side

The same CPU-heavy stage in each, run once per runtime

The stage is a stand-in for a CPU-bound transform: 16 records, each run through 10 million iterations of a 32-bit hash loop (xor then multiply, FNV style). The kernel of the loop is a hash in the style of FNV: start h from a fixed seed XORed with the record id; then, 10 million times, XOR h with i & 255 (the low 8 bits of the loop counter) and multiply by the prime 16777619, keeping only the low 32 bits so the value wraps around. Each step needs the previous result, so the work cannot be shortened or skipped: it stands in for per-record CPU work. Go and Java 32-bit integers wrap automatically; Python integers never overflow, so the loop masks with & 0xFFFFFFFF; JavaScript uses Math.imul (a 32-bit integer multiply) and >>> 0 (reinterpret as unsigned). Every program XORs the 16 results into one checksum, and all printed the same value, 2613273088, so they computed the same thing. Run with docker run --cpus=4, and every parallel version uses 4 workers. Go in a golang:1.23 container, Java in eclipse-temurin:21 (JDK 21), Python 3.12.15 in python:3.12-slim, Node 22.23.3 in node:22-slim; all on one arm64 host. Every timing below depends on the host, the CPU count and the runtime version, so read the ratios as the shape of the result, not as benchmark rankings: with the same containers limited to 1 CPU (--cpus=1), the Go goroutines, the Java pool and the Node workers lose their advantage (Go takes several times longer than at 4 CPUs, and the Node workers are no faster than the single thread), because four workers cannot run in parallel on one core.

Go (go build then run):

package main

import (
	"fmt"
	"sync"
	"time"
)

const records, iters, workers = 16, 10_000_000, 4

func transform(id uint32) uint32 {
	h := uint32(2166136261) ^ id
	for i := uint32(0); i < iters; i++ {
		h = (h ^ (i & 255)) * 16777619
	}
	return h
}

func main() {
	start := time.Now()
	jobs := make(chan uint32)
	results := make([]uint32, records)
	var wg sync.WaitGroup
	for w := 0; w < workers; w++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for id := range jobs {
				results[id] = transform(id)
			}
		}()
	}
	for id := uint32(0); id < records; id++ {
		jobs <- id
	}
	close(jobs)
	wg.Wait()
	var x uint32
	for _, r := range results {
		x ^= r
	}
	fmt.Printf("go goroutines x%d: %v checksum %d\n", workers, time.Since(start).Round(time.Millisecond), x)
}

Java (java Pipeline.java; round 1 includes JIT warm-up (the JVM compiles frequently run code to machine code while the program runs, so the first round is slower), round 2 is warm):

import java.util.*;
import java.util.concurrent.*;

public class Pipeline {
    static final int RECORDS = 16, ITERS = 10_000_000, WORKERS = 4;

    static int transform(int id) {
        int h = (int) 2166136261L ^ id;
        for (int i = 0; i < ITERS; i++) h = (h ^ (i & 255)) * 16777619;
        return h;
    }

    static long run(ExecutorService pool) throws Exception {
        List<Future<Integer>> fs = new ArrayList<>();
        for (int id = 0; id < RECORDS; id++) { int k = id; fs.add(pool.submit(() -> transform(k))); }
        int x = 0;
        for (Future<Integer> f : fs) x ^= f.get();
        return x & 0xFFFFFFFFL;
    }

    public static void main(String[] a) throws Exception {
        for (int round = 1; round <= 2; round++) { // round 1 includes JIT warm-up
            ExecutorService pool = Executors.newFixedThreadPool(WORKERS);
            long t0 = System.nanoTime();
            long sum = run(pool);
            System.out.printf("java pool x%d round %d: %d ms checksum %d%n", WORKERS, round, (System.nanoTime() - t0) / 1_000_000, sum);
            pool.shutdown();
        }
        long t0 = System.nanoTime();
        int x = 0;
        for (int id = 0; id < RECORDS; id++) x ^= transform(id);
        System.out.printf("java single thread: %d ms checksum %d%n", (System.nanoTime() - t0) / 1_000_000, x & 0xFFFFFFFFL);
    }
}

Python (python pipeline.py):

import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

RECORDS, ITERS, WORKERS = 16, 10_000_000, 4

def transform(rec_id):
    h = 2166136261 ^ rec_id
    for i in range(ITERS):
        h = ((h ^ (i & 255)) * 16777619) & 0xFFFFFFFF
    return h

def checksum(results):
    x = 0
    for r in results:
        x ^= r
    return x

if __name__ == "__main__":
    t0 = time.perf_counter()
    r = checksum(map(transform, range(RECORDS)))
    print(f"python serial:            {time.perf_counter()-t0:6.2f} s checksum {r}")

    t0 = time.perf_counter()
    with ThreadPoolExecutor(WORKERS) as ex:
        r = checksum(ex.map(transform, range(RECORDS)))
    print(f"python threads x{WORKERS}:       {time.perf_counter()-t0:6.2f} s checksum {r}")

    t0 = time.perf_counter()
    with ProcessPoolExecutor(WORKERS) as ex:
        r = checksum(ex.map(transform, range(RECORDS)))
    print(f"python processes x{WORKERS}:     {time.perf_counter()-t0:6.2f} s checksum {r}")

Node.js (node pipeline.js):

const { Worker, isMainThread, parentPort, workerData } = require('node:worker_threads');
const RECORDS = 16, ITERS = 10_000_000, WORKERS = 4;

function transform(id) {
  let h = (2166136261 ^ id) >>> 0;
  for (let i = 0; i < ITERS; i++) h = Math.imul(h ^ (i & 255), 16777619) >>> 0;
  return h;
}

if (isMainThread) {
  let t0 = process.hrtime.bigint();
  let x = 0;
  for (let id = 0; id < RECORDS; id++) x = (x ^ transform(id)) >>> 0;
  console.log(`node single thread:   ${Number(process.hrtime.bigint() - t0) / 1e6 | 0} ms checksum ${x}`);

  t0 = process.hrtime.bigint();
  let next = 0, done = 0, sum = 0;
  const startWorker = () => {
    const w = new Worker(__filename, { workerData: null });
    w.on('message', (m) => {
      if (m.result !== undefined) { sum = (sum ^ m.result) >>> 0; done++; }
      if (next < RECORDS) w.postMessage({ id: next++ });
      else { w.terminate(); }
      if (done === RECORDS) {
        console.log(`node worker_threads x${WORKERS}: ${Number(process.hrtime.bigint() - t0) / 1e6 | 0} ms checksum ${sum}`);
      }
    });
    w.postMessage({ id: next++ });
  };
  for (let i = 0; i < WORKERS; i++) startWorker();
} else {
  parentPort.on('message', ({ id }) => parentPort.postMessage({ result: transform(id) }));
}

Reading the programs. Go starts 4 goroutines that pull record ids from the jobs channel and write each result into its own slot of results; wg.Wait() joins them. Java submits 16 tasks to a fixed pool of 4 threads and XORs the Future results; the single-thread line is the baseline. Python runs the same function serially, in a thread pool and in a process pool. In Node, the main thread starts 4 workers and hands each one a record id with postMessage; whenever a worker replies with a result, the main thread XORs it in, then either sends that worker the next id or terminates it, and prints the timing when all 16 results are in. Each worker just waits for a message, computes transform(id) and posts the result back.

One run of each printed (timings vary from run to run):

go goroutines x4: 53ms checksum 2613273088
java pool x4 round 1: 60 ms checksum 2613273088
java pool x4 round 2: 43 ms checksum 2613273088
java single thread: 183 ms checksum 2613273088
python serial:              9.03 s checksum 2613273088
python threads x4:         9.91 s checksum 2613273088
python processes x4:       2.65 s checksum 2613273088
node single thread:   202 ms checksum 2613273088
node worker_threads x4: 92 ms checksum 2613273088

Repeat runs on the same host keep the same ordering and the same overall shape, but the figures move: Go and Java stay close to the printed times, Node's single-thread and worker times can shift by tens of milliseconds, Python's by a second or so, and a different machine will differ more.

What the numbers say (ratios computed from the sample run above):

• Java and Go scale with cores. Java's warm pool is roughly 3 to 4x faster than its single thread on 4 CPUs: in the sample run 183 ms / 43 ms = 4.3, and other runs on the same host gave a somewhat smaller ratio. Go with 4 goroutines finished in about the same time as Java's pool (53 ms in the sample run).
• Python threads give no speed-up. 9.91 s with 4 threads against 9.03 s serial in the sample run, and about the same as serial in every run: the GIL lets one thread run bytecode at a time. Processes got about 3.4x (9.03 s / 2.65 s = 3.4 in the sample run) at the price of pickling inputs and results between processes.
• Node's workers gave roughly 2 to 3x on 4 workers, below 4x (202 ms / 92 ms = 2.2 in the sample run; other runs on the same host gave a somewhat larger ratio). This run does not isolate why; worker start-up (each loads its own V8 isolate) and message passing are the obvious candidates, so treat the cause as unmeasured.
• The per-core gap is large for this loop. Java's single thread (183 ms in the sample run) against Python serial (9.03 s) is about 49x (9.03 s / 0.183 s = 49). That is a pure-Python inner loop; real data work in NumPy, pandas or Polars runs the loop in native code, so this ratio does not transfer to such a stage.

Which I would pick, and how I would structure it

For a CPU-heavy stage written from scratch I would pick Go (or Java if the pipeline already lives on the JVM, for example next to Spark or Flink, JVM-based frameworks for large-scale data processing). Both give shared-memory parallelism over all cores with simple code, no per-record serialization, and one deployable artifact. What would flip the choice: if the transformation is already a NumPy, pandas or Polars operation, keep Python, because the heavy loop runs in native code (NumPy, pandas and Polars are libraries whose inner loops are compiled code, as are Cython-compiled extensions) and the stage is not GIL-bound; if the surrounding team and libraries are Node, use worker_threads rather than rewriting.

Structure in each:

• Go: a fixed pool of runtime.NumCPU() goroutines reading from a channel of records, writing results into a pre-sized slice by index or onto an output channel; a context.Context for cancellation; errgroup or sync.WaitGroup to join. The channel gives back-pressure when the producer is faster than the workers.
• Java: Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()) (or ForkJoinPool, a pool built for splitting a task into subtasks, or parallel streams), CompletableFuture (a handle to a result that arrives later, with methods to chain follow-up steps) to compose stages. Use platform threads for CPU-bound work: the JEP 444 text (the JDK Enhancement Proposal that defines virtual threads) says virtual threads provide scale (throughput), not speed, and that more threads than cores would not help a CPU-bound calculation, whether virtual or platform. A bounded queue gives back-pressure.
• Python: ProcessPoolExecutor sized to the CPU count, batching records into chunks (e.g. 1,000 per task) so pickling overhead (converting Python objects to bytes to send them to another process) is amortized, or push the loop into NumPy/Polars/Cython. The newer options: Python 3.14 officially supports a free-threaded build (PEP 779, the proposal that moved it out of experimental status; a CPython build without the GIL), not the default, with a documented single-thread penalty of roughly 5 to 10 percent, and adds InterpreterPoolExecutor (a pool that runs tasks in separate sub-interpreters inside one process); a library that has not declared free-threading support can re-enable the GIL, so test your dependencies before relying on it.
• Node.js: a worker_threads pool sized to the cores, keeping the main event loop free for I/O; transfer ArrayBuffers instead of copying objects; hand out records through a queue, as the program above does.

Trade-offs to state

Throughput versus memory: processes and workers each carry their own heap, so four workers cost roughly four runtimes' baseline memory (not measured here). Latency: a chunk dispatched to a pool pays a hand-off cost, which is why tiny records should be batched. Operability: Go and Java give one process to profile and one heap to size; the Python and Node pools add inter-process or inter-isolate messaging to monitor.