Plan: instrument first to diagnose, then remediate incrementally (tuning → code → architecture). Collect data for several SLA-breaching and normal runs to compare.
Instrumentation / logs to collect
- JVM metrics (JMX/prometheus): heap usage (Eden/Survivor/Old), young/old gen sizes, metaspace, GC pause times, GC count/type, safepoint time, allocation rate, promotion rate, tenuring threshold, suspended threads.
- GC logs (G1/Parallel/ CMS) with timestamps and -Xlog:gc*, including GC cause, pause time, heap before/after, Humongous allocation info.
- Thread dumps at pause boundaries (jstack) and safepoint logs to see blocked threads.
- Application-level metrics: per-job processing time, record throughput, batch size, input size distribution, object allocation hotspots (async profilers, Flight Recorder).
- OS metrics: CPU, load, swap, page faults, I/O latency, network.
- Container/host limits (cgroups memory limits, JVM inside container).
Root causes to investigate
- Long full GCs due to fragmentation or promotion failures (old gen fills up).
- High allocation rate causing frequent young-gen collections and promotions.
- Humongous objects triggering G1 mixed/full collections.
- Incorrect heap sizing relative to working set or container limits causing OOM/GC storms.
- GC algorithm mismatch (e.g., CMS pauses with fragmentation; G1 un-tuned).
- Safepoint stalls from biased locks or native I/O.
- OS swapping due to oversubscription.
Remediation steps (ordered)
- Quick fixes:
- Increase heap modestly if near-full and host has headroom; ensure no swap and JVM has full physical memory (avoid cgroup surprise).
- Tune GC logging and enable -XX:FlightRecorder for deeper traces.
- GC tuning:
- For low-latency ETL prefer G1 or ZGC (Java 11+ / 17+): set target pause (e.g., -XX:MaxGCPauseMillis=100-200), tune region size and Humongous threshold, increase young gen fraction if allocation-heavy.
- If on JDK 17+, evaluate ZGC or Shenandoah for sub-10ms pauses if memory allows.
- For G1: tune -XX:InitiatingHeapOccupancyPercent, -XX:ConcGCThreads, -XX:ParallelGCThreads and -XX:MaxGCPauseMillis; monitor humongous object behavior.
- Allocation and code changes:
- Reduce temporary object allocation: reuse buffers, use primitive arrays, stream vs collect trade-offs.
- Batch processing: increase batch sizes to amortize per-record overhead but watch memory footprint.
- Avoid creating many short-lived large arrays; use object pools for heavy objects.
- Profile to find allocation hotspots and fix them.
- Off-heap and native:
- Move large buffers / caches off-heap (ByteBuffer, Netty, or native memory) to reduce GC pressure.
- Use memory-mapped files for large inputs or caches.
- Architectural:
- Horizontal scale workers with smaller heaps to reduce full-GC risk; stateless workers with partitioned input.
- Use backpressure and rate limiting upstream to avoid allocation bursts.
- Isolate heavy tasks into separate JVMs (worker for parsing vs. aggregation).
- Operational:
- Configure JVM and container memory consistently (Xmx < container limit).
- Implement alerting on GC pause thresholds and allocation rate.
- Run load tests with realistic data shapes including humongous objects.
Validation
- After each change, compare GC pause histograms, tail latency (P95/P99), throughput, and SLA compliance.
- Use A/B or canary rollouts to validate before full deployment.
This approach finds root causes with measurements, applies low-risk tuning first, then code/off-heap and architecture changes if necessary, always validating impact on tail latency and SLA.
