Approach:- Use a bounded BlockingQueue<String> as the handoff.- Producers (one per input file or pooled) read lines and put into the queue.- Consumers poll/take and write to sink.- Use a poison-pill pattern: after all producers finish, insert one special POISON per consumer so each consumer can exit cleanly.- Use an AtomicReference<Exception> to capture first error and request shutdown; use ExecutorService to manage threads and ensure orderly termination.- Handle InterruptedException by re-setting the interrupt flag, recording the error and exiting.Java implementation (reads lines, consumers print processed lines to an OutputStream sink):java
import java.io.*;
import java.nio.file.*;
import java.util.List;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicReference;
public class FileProducerConsumer {
private static final String POISON = new String("__POISON__");
private final BlockingQueue<String> queue;
private final ExecutorService prodPool;
private final ExecutorService consPool;
private final AtomicReference<Exception> error = new AtomicReference<>();
public FileProducerConsumer(int queueCapacity, int producerThreads, int consumerThreads) {
this.queue = new ArrayBlockingQueue<>(queueCapacity);
this.prodPool = Executors.newFixedThreadPool(producerThreads);
this.consPool = Executors.newFixedThreadPool(consumerThreads);
}
public void run(List<Path> inputFiles, OutputStream sink, int consumerCount) throws Exception {
CountDownLatch producersDone = new CountDownLatch(inputFiles.size());
// Start consumers
for (int i = 0; i < consumerCount; i++) {
consPool.submit(() -> {
try (BufferedWriter out = new BufferedWriter(new OutputStreamWriter(sink))) {
while (true) {
String line = queue.take();
if (line == POISON) {
break;
}
// Process line (example: uppercase) and write to sink
String processed = line.toUpperCase();
synchronized (out) { // synchronize writes to shared sink
out.write(processed);
out.newLine();
}
}
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
error.compareAndSet(null, ie);
} catch (Exception e) {
error.compareAndSet(null, e);
}
});
}
// Start producers
for (Path p : inputFiles) {
prodPool.submit(() -> {
try (BufferedReader r = Files.newBufferedReader(p)) {
String line;
while ((line = r.readLine()) != null) {
// If an error was recorded, stop producing
if (error.get() != null) break;
// Use offer with timeout to be responsive to interrupts
boolean offered = false;
while (!offered) {
try {
offered = queue.offer(line, 500, TimeUnit.MILLISECONDS);
if (Thread.currentThread().isInterrupted()) throw new InterruptedException();
if (error.get() != null) break;
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw ie;
}
}
if (error.get() != null) break;
}
} catch (Exception e) {
error.compareAndSet(null, e);
} finally {
producersDone.countDown();
}
});
}
// Wait for producers, then insert poison pills
producersDone.await();
// Insert one poison per consumer to ensure each can exit
for (int i = 0; i < consumerCount; i++) {
// best-effort insertion; if interrupted, set error and break
try {
queue.put(POISON);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
error.compareAndSet(null, ie);
}
}
// shutdown pools and wait
prodPool.shutdownNow();
consPool.shutdown();
if (!consPool.awaitTermination(30, TimeUnit.SECONDS)) {
consPool.shutdownNow();
}
Exception ex = error.get();
if (ex != null) throw ex;
}
}
Key points and reasoning:- Poison-pill per consumer ensures deterministic shutdown without relying on queue drain ordering.- Bounded queue provides backpressure: producers block when consumers are slow.- CountDownLatch coordinates when to emit poison pills (only after all producers finished).- AtomicReference captures first error; other threads check it and stop producing/processing early.- Interruptions: threads catch InterruptedException, set interrupt flag and record error; executor shutdownNow interrupts workers.- Synchronize writes to a shared sink; for high-throughput, better use a dedicated writer thread or partitioned sinks.Complexity:- Time: O(total lines) processing cost.- Space: O(queueCapacity) plus per-line buffers.Edge cases:- Very large files: use streaming (BufferedReader) as shown.- Partial failures: recorded and propagated so caller can decide retry or alert.- Sink blocking or slow consumers: backpressure prevents OOM.- Ensure POISON is a unique object (using new String ensures identity check); if using String.equals for content, choose a unique sentinel object instead.