Java Programming Questions

Key components

• Pool manager maintaining idle and in-use sets.
• Acquisition algorithm: try take from idle queue; if empty and pool < max, open new connection; else wait with timeout and backpressure.
• Max pool size and wait policy: configurable max, exponential backoff or immediate failure (SQLTransientConnectionException) after timeout.

Validation & eviction

• Health-check on borrow (light validation query) or async validation for idle.
• Eviction thread scans idle connections, closes stale/broken ones after configured idleTimeout.

Leak detection

• Track borrow stack trace and timestamp; if held beyond leakThreshold, log warning and optionally close. Implement weak-reference based leak cleanup: schedule forced reclaim after grace period.

Thundering herd avoidance

• Use fairness queue with limited waiters; on exhaustion, requesters wait on Condition and wake a bounded number at a time. Use exponential backoff and circuit-breaker to reject fast failures.

Metrics

• Expose active_count, idle_count, wait_count, wait_time_ms, connection_create_rate, validation_failures, leak_warnings.

Implementation notes

• Use ConcurrentLinkedQueue for idle, AtomicInteger for size, ReentrantLock+Condition for waiters. Keep connection creation bounded to avoid resource storms.

Immediate mitigations

• Return to safe state: deploy previous release or disable offending feature via feature flag.
• Short-term traffic controls: scale replicas, apply rate-limits, route traffic to healthy zones.
• DB-level throttling: increase connection pool timeouts, reduce query concurrency, or apply statement_timeout.

Triage & root-cause steps

• Verify traces and identify slow SQL text and plan (capture full SQL with bind params).
• Run EXPLAIN (ANALYZE, BUFFERS) on production-like data to confirm sequential scan.
• Check recent changes: code, query text, ORMs, new indexes dropped, schema changes, pg_stat_statements.
• Inspect planner inputs: table statistics, recent VACUUM/ANALYZE, parameter changes (work_mem) or config flags.
• Reproduce with captured params on staging.

Safe rollback / hotfix

• If query change is in code: rollback deploy atomically.
• If stats or plan issue: run ANALYZE or CREATE necessary index; use CONCURRENTLY to avoid locks.
• Use partial rollout (canary) after fix.
• Ensure rollback plan tested and DB migrations reversible.

Postmortem outline

• Timeline, impact, root cause, detection method.
• Actionable items: automated detection alerts for query plan regressions, add tests for critical queries, require plan checks in deployment, schedule periodic ANALYZE, CI that runs EXPLAIN on sample datasets, runbooks for rollback.
• Owners, deadlines, verification steps and metrics to measure recurrence (99th latency, query duration percentiles).

Public API (example)

• V get(K key)
• void put(K key, V value, Duration ttl, Consumer<V> onEvict)
• void remove(K key)

Implementation outline

• Use ConcurrentHashMap<K, Node> for O(1) lookup.
• Maintain LRU order using a concurrent-friendly doubly-linked list or a striped LinkedHashMap; or better, use Caffeine which implements windowed LRU and is highly concurrent.
• Each Node holds value, expireAt, and references for eviction list. Use a background cleaner to remove expired entries and call eviction callback asynchronously.
• For thread-safety update operations: use CAS on Node references and small synchronized blocks only for list pointer updates or use lock-striping to reduce contention.

TTL & eviction callback

• On get, check expiry; if expired, remove and call callback.
• On put, if over capacity, evict least-recently-used node and trigger callback.

Correctness & tests

• Concurrency tests: many-thread get/put/remove verifying size bounds, no lost entries.
• Eviction tests: validate LRU order under access patterns.
• TTL tests: ensure entries expire roughly after TTL and callback invoked.

Recommendation

• Prefer Caffeine for production: offers O(1), TTL, eviction listeners, and proven concurrency and performance.

Problem with SELECT * in high-traffic services

• Network I/O: SELECT * transfers all columns; wide rows increase bytes over wire and slow responses and throughput.
• Preventing index-only scans: SELECT * forces DB to fetch full table rows (heap/TOAST) even when the index contains required columns; this breaks index-only scan optimizations.
• Coupling to schema changes: Adding/removing columns can change payload shape unexpectedly, causing client bugs and increasing memory usage.

Diagnose & fix I/O-bound wide-row queries

• Diagnose: use EXPLAIN/EXPLAIN ANALYZE to see seq scans vs index-only, track bytes_sent metrics, APM/span sizes, and DB pg_stat_statements to find heavy queries.
• Fix: explicitly select only required columns, add covering indexes (include/INCLUDE columns), normalize or move large columns to separate table, paginate large payloads or stream blobs.

Summary: prefer explicit column lists, add covering indexes, and measure with EXPLAIN and production metrics to reduce network and I/O pressure.

Diagnosis steps

• Enable GC logging (-Xlog:gc*), capture heap/GC metrics (G1 pause times, frequency) and thread stacks during pause.
• Correlate latency spikes with GC logs and CPU/memory/alloc rate metrics.
• Use jcmd/jmap/jfr to inspect heap usage, object allocation rates, and live set.

Tuning & config

• Right-size heap: avoid too small (frequent GCs) or too large (long pauses). Tune -Xms/-Xmx to match observed live set + headroom.
• G1 tuning: adjust MaxGCPauseMillis, initiateHeapOccupancyPercent to trigger concurrent cycles earlier, increase G1NewSizePercent for throughput.
• Consider ZGC or Shenandoah for lower pause goals if JVM/OS supported and production-tested; they have concurrent compaction with sub-ms pauses but require Java builds supporting them.

Validation

• Create staging workload mirroring production allocation patterns; measure latency, GC pause percentiles, and throughput before/after changes using JFR and GC logs.
• Gradual rollout in prod with canary instances, monitor p99 latency and GC pause stats.

Best practices

• Reduce allocation churn: reuse buffers, prefer primitive arrays, off-heap caches if needed. Monitor metaspace and GC ergonomics. Document changes and keep rollback plan.