System Resource & I/O Optimization Questions

Tuning how a system uses CPU, memory, disk, and network at the OS and I/O layer. Covers I/O throughput and blocking, buffering and batching, filesystem and kernel-level performance settings, and resource contention between processes. Includes OS-level performance tuning and diagnosing resource saturation on the host.

MediumTechnical

26 practiced

Describe how to implement streaming file uploads for large model artifacts or datasets without loading the entire file into memory. Discuss chunked uploads, streaming directly to disk or object storage, resumable uploads, integrity verification (checksums), and how to validate file schemas while streaming.

Sample Answer

Start by treating uploads as a stream-of-bytes problem rather than an in-memory file. Key design goals: constant memory usage, resumability, integrity, and the ability to validate content incrementally.

High-level approach- Client splits the file into chunks (fixed size, e.g., 8–64 MB) or uses a streaming HTTP POST. Each chunk is uploaded independently and can be retried.- Server or object store accepts chunks and appends them to a persistent store (disk or object storage) without buffering the whole file.

Chunked uploads and storage- Use cloud-native multipart APIs (S3 Multipart Upload, GCS Resumable Upload) so chunks are uploaded directly to object storage via signed URLs. This offloads IO and avoids server memory pressure.- If proxying through your backend, stream each request to a temporary file on disk with backpressure (non-blocking IO) and fsync/flush to minimize data loss.

Resumable uploads- Give each upload a session ID and track chunk indices + status in a small metadata store. Clients can query which parts are present and resume.- Support range-based uploads and partial MD5/SHA metadata per chunk to permit out-of-order assembly.- For client interruptions, store committed parts and expose an API to complete the upload (assemble parts into final object).

Integrity verification- Compute per-chunk checksums (MD5/SHA256) client-side and send with each chunk. Verify server-side before accepting the chunk.- On finalization, compute/verify the full-file checksum (or combine per-chunk checksums) and store it as artifact metadata. For S3, use ETag or multipart etag checksumming patterns but prefer explicit SHA256 for security.- Use TLS in transit and server-side encryption at rest.

Validating file schemas while streaming- Parse incrementally: use streaming parsers (SAX-like JSON parser, Apache Commons CSV, or Protobuf streaming) to validate schema record-by-record without keeping everything in memory.- Early rejection: run lightweight schema checks (field presence, types, basic ranges) as you stream; reject the upload early if critical invariants fail.- Maintain counters/metrics and sample rows for deeper validation asynchronously (e.g., data types, embeddings shape). For ML artifacts, validate model metadata headers (format, version) before accepting body.- For complex validations (e.g., cross-record checks), persist streamed data to disk/object store first and run a background validation job that can mark the upload as accepted/failed and emit diagnostics.

Operational considerations- Use idempotent chunk APIs and strong guarantees for exactly-once assembly (manifest + finalize step).- Enforce rate limits, per-user quotas, and authentication (signed URLs, short-lived tokens).- Log per-chunk audit trails and provide client retry guidance and exponential backoff.- Test with network interruptions and simulate partial uploads to ensure resume correctness.

This design keeps memory usage constant, enables reliable retries/resumption, ensures integrity via checksums, and allows immediate or staged schema validation tailored to performance and correctness needs for large ML artifacts.

Unlock Full Question Bank

Get access to hundreds of System Resource & I/O Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.