Language-Level Concurrency and Multithreading Questions

Concurrency: structuring a program so multiple tasks make progress independently (interleaved execution). Parallelism: executing multiple tasks at the same time on multiple CPU cores (simultaneous execution).

Examples — concurrent but not parallel

• Python (threads, GIL prevents CPU parallelism):

import threading, time

def io_task(n):
    time.sleep(1)
    print(n)

for i in range(5):
    threading.Thread(target=io_task, args=(i,)).start()

This is concurrent (threads interleave) but CPU-bound work won't run in parallel due to the GIL.

• Go (goroutines on single OS thread):

go func(i int){ /* network IO */ }(i)

If runtime is constrained to one OS thread (GOMAXPROCS=1) you get concurrency without parallel CPU execution.

• Java (single-core or thread contention):

new Thread(() -> { /* blocking IO */ }).start();

If run on one core or tasks block, it's concurrent but not parallel.

Examples — parallel

• Python multiprocessing (true parallel processes):

from multiprocessing import Pool
def cpu(n): return sum(i*i for i in range(10_000_00))
with Pool(4) as p: print(p.map(cpu, range(4)))

• Go with GOMAXPROCS>1: goroutines scheduled on multiple OS threads run in parallel.
• Java threads on multi-core JVM: ThreadPoolExecutor on multi-core runs in parallel.

Implications for AI Engineer

• Model training: needs parallelism (multi-GPU / multi-process) for throughput; concurrency alone (async) won’t speed GPU compute.
• CPU-bound preprocessing: use multiprocessing, vectorized NumPy, or thread pools tied to cores — avoid Python threads because of GIL.
• Serving architectures: combine concurrency (async I/O, event loops) to handle many requests with low latency and parallelism (multiple worker processes / replicas, GPU pooling) for heavy inference. Design: async front-end + worker pool + batching on GPU for throughput.

Approach: start with thread-safety inside one process, then extend to multiple processes and machines. Key problems are race conditions when two threads read-modify-write the same dict entry concurrently, losing increments.

Single-process, multi-thread (Python example):

• Naive dict increment is unsafe.
• Option A: coarse-grained Lock around the whole dict (simple, correct, but high contention).
• Option B: per-key locks or sharded locks to reduce contention.
• Option C: use collections.Counter with locks (Counter isn't atomic; still needs locking).
• Option D: atomic increments via thread-safe primitives (not built-in in CPython for ints), or use multiprocessing.Value/AtomicInteger-like wrappers.

Example — sharded locks to reduce contention:

from collections import defaultdict
from threading import Lock

NUM_SHARDS = 16
shards = [defaultdict(int) for _ in range(NUM_SHARDS)]
locks = [Lock() for _ in range(NUM_SHARDS)]

def _shard(user_id):
    return hash(user_id) % NUM_SHARDS

def increment(user_id, delta=1):
    i = _shard(user_id)
    with locks[i]:
        shards[i][user_id] += delta

Key points:

• Coarse lock: easy, but serializes all operations.
• Sharded locks: trade memory/complexity for parallelism.
• Per-key locks: finer but can be costly to manage.

Scaling to multiple processes on one host:

• Use multiprocessing.Manager dict (has locking) or shared memory with atomic counters.
• Or run a local in-memory service (e.g., Redis) and use INCR (atomic) commands for low-latency increments.

Scaling across machines:

• Single source of truth: external datastore (Redis, Cassandra) with atomic increment operations.
• Partitioning: shard by user_id across Redis clusters or Kafka partitions; use consistent hashing.
• Eventual consistency: if accepting eventual counts, use app-side batching and CRDTs (PN-Counters) to merge counts without coordination.
• High-throughput design: publish events to a durable log (Kafka), then have scalable consumers aggregate into stores; this separates ingestion from counting.

Trade-offs:

• Strong consistency vs throughput/latency.
• External services add network latency but provide atomic primitives and persistence.
• For AI workloads, choose Kafka+stream processors or Redis for real-time counters; use CRDTs when offline merging and availability are priorities.

Monitoring & correctness:

• Instrument lost/in-flight events, use idempotency keys if events can be retried.
• Add metrics for lock contention and latency to guide sharding choices.