Java Programming Questions

Zero/minimal downtime migration strategy

1. Plan additive-only changes first (new columns, tables, indexes) so old and new code can run concurrently.
2. Use migration tools (Flyway/Liquibase) for versioned schema changes. For large ALTERs (drop column, change type), use online tools (gh-ost or pt-online-schema-change) to avoid locks.
3. Backfilling computed fields: add nullable column, deploy app code that writes to both old and new fields (dual-write), run an online backfill job that updates existing rows in small batches during low traffic.
4. Online backfills: implement id-range or time-window batched updates with throttling and monitoring; consider using distributed job workers.
5. Rolling out application changes:
   • Phase 1: deploy code that reads from old column and writes to both (backward-compatible).
   • Phase 2: run backfill until zero rows left needing migration.
   • Phase 3: deploy code that reads from new column (feature flag), then remove dual-write.
6. Safe rollback:
   • Keep old code path until new code verified.
   • For destructive operations, schedule maintenance window or use re-creation strategies.

Tools

• Flyway/Liquibase for schema versioning; gh-ost or pt-online-schema-change for online ALTER TABLE on MySQL. For Postgres, use pg_repack or native online-friendly ALTERs and logical replication for complex migrations.

Best practices

• Test migrations in staging with production-like data size.
• Monitor performance, locks, replication lag during migrations.
• Use feature flags to minimize blast radius and allow gradual rollout.

Instrumenting endpoint with Micrometer

• Use Spring Boot Actuator with Micrometer. Create a Timer or use @Timed on controller/service.
• Metric naming: http_server_requests_latency_seconds (or app_http_request_latency_seconds)
• Tags: method, uri (use templated path), status, service_version, region.

Example code snippet:

@Bean
public Timer requestTimer(MeterRegistry registry){
  return Timer.builder("app.http.server.requests")
    .description("HTTP request latency")
    .publishPercentileHistogram()
    .tags("service","orders")
    .register(registry);
}
// Or use @Timed("app.http.server.requests") on controller

Prometheus query for 95th percentile over last 5m (histogram):
Use histogram_quantile on histogram buckets:

histogram_quantile(0.95, sum(rate(app_http_server_requests_seconds_bucket[5m])) by (le, uri))

If using summaries, you can query quantiles but they are client-side and not aggregatable well across instances.

Pitfalls

• Label cardinality: avoid high-cardinality tags (user_id, request_id). Use templated uri, not raw path parameters.
• Histogram vs Summary: histograms are recommended for Prometheus because they aggregate across instances; summaries compute quantiles per-instance and don't aggregate well. Choose buckets thoughtfully to cover expected latencies.

Compare OFFSET/LIMIT vs Keyset

• OFFSET/LIMIT: easy but becomes slow for large offsets because DB skips rows. Results unstable with concurrent inserts/deletes.
• Keyset (cursor) pagination: uses where clauses on indexed columns (e.g., id, created_at) to seek from last seen values; efficient and stable.

Example keyset SQL (posts sorted by created_at desc, id desc as tiebreaker)

SELECT id, created_at, title
FROM posts
WHERE (created_at, id) < ( :last_created_at, :last_id )
ORDER BY created_at DESC, id DESC
LIMIT 20;

Intuition: fetch posts older than last cursor.

Implementing in Java

• Spring Data: write @Query with the above SQL and pass last cursor params; return Page-like results.
• Manual JDBC/JPA: maintain lastCreatedAt and lastId in client; build query dynamically.

Trade-offs when sorting by multiple columns

• Need composite index on (created_at DESC, id DESC) to be efficient.
• Keyset cannot easily jump to arbitrary page number; suitable for 'Next'/'Prev' navigation.
• For complex sort (non-unique combos), include deterministic tiebreaker (PK) to maintain stable ordering.

Structured diagnostic approach

1. Observe metrics: latency, CPU, I/O, concurrent connections, locks (pg_stat_activity, pg_stat_io). Identify whether CPU, IO, or locks dominate.
2. Capture query plan: run EXPLAIN (ANALYZE, BUFFERS) for the slow JOIN to see join type, row estimates vs actual, and hotspots (e.g., nested loop reading many rows).
3. Check indexes & stats: ensure join keys are indexed (products.id, inventory.product_id); run ANALYZE if stats stale.
4. Inspect locks and waits: pg_locks, pg_stat_activity to see blocking queries causing serialization.
5. Check I/O: whether sequential scans read lots of pages; consider vmstat/iostat to see disk saturation.

Interpreting plans

• Nested Loop with large outer rows and expensive inner scans suggests missing indexes or bad join order.
• Hash join spilling to disk indicates memory insufficient.

Concrete fixes

• Add indexes on join/filter columns (inventory.product_id, products.id); consider covering indexes.
• Update statistics and consider increasing work_mem to avoid disk-spill for hash joins.
• Denormalize frequently-joined data into a single table or create a materialized view refreshed periodically for read-heavy APIs.
• For sub-ms response, move to read-optimized store (Elasticsearch) for search-heavy queries.
• If writes must be current, use incremental materialized view updates or maintain a cache.

Measure after each change; prefer non-disruptive index adds and config tuning before schema changes.

Step-by-step plan to find CPU hotspots with minimal impact

1. Monitor metrics: confirm high CPU via system and JVM metrics (process CPU, GC, threads).
2. Lightweight sampling: collect periodic jstack samples (thread dumps) to see hot threads; take several spaced dumps.
3. Use Java Flight Recorder (JFR) in production for low-overhead profiling (enable continuous or for short window). Analyze hotspots.
4. Use async-profiler (native) if allowed: capture flame graph (CPU) for a short period; it's very low overhead.
5. Interpret flame graphs: wide boxes = heavy methods; stack shows callers → follow hottest paths.
6. Correlate with application traces/logs to map endpoint/transactions to hotspots.

Reduction strategies

• Algorithmic: replace O(n^2) with O(n log n) algorithms, avoid repeated work.
• Caching: memoize expensive computations or DB results off hot path.
• Batch work / move off critical path: push heavy tasks to background jobs or worker queues.
• Reduce object allocation: reuse buffers, avoid boxing, reduce GC pressure.
• Database/IO: optimize queries, add indexes, or cache results.

Tools summary

• jstack for quick dumps, JFR for low-overhead continuous profiling, async-profiler for detailed flamegraphs, profilers like YourKit in staging.