Linux Filesystems and Storage Questions

Specialized knowledge of Linux storage subsystems, filesystem architectures, and maintenance. Covers filesystem internals and selection considerations for ext4, XFS, and btrfs; partitioning tools and workflows using fdisk and parted; logical volume management including physical volumes, volume groups, logical volumes, snapshots and resizing; adding and expanding disks, resizing filesystems, mount options and performance tradeoffs; filesystem integrity tools such as fsck, recovery techniques, inode and metadata concepts, capacity planning, and interaction with software and hardware RAID and storage arrays.

MediumTechnical

62 practiced

XFS metadata appears corrupted and the filesystem refuses to mount on a production host. Explain safe preparatory steps before running xfs_repair, including capturing evidence, creating block-level backups or snapshots, dry-run repair steps, and techniques to minimize data loss while recovering an XFS filesystem in production.

Sample Answer

Situation: Production host shows XFS metadata corruption and refuses to mount. Before running xfs_repair, take safe, repeatable steps to preserve evidence and minimize data loss.

1) Isolate and preserve- Stop services that touch the device, unmount if partially mounted, and ensure no automatic recovery/cron jobs run.- Record state: hostname, date, kernel version, dmesg output, /var/log/messages, output of lsblk, blkid, and xfs_info (if possible).Example commands:

bash

date > /root/recovery-note.txt
dmesg | tail -n 200 > /root/dmesg.log
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT > /root/lsblk.log

2) Create immutable evidence and checksums- Create a small evidence bundle and compute checksums of the device to prove bit-for-bit copies later:

bash

sha256sum /dev/sdX > /root/sdX.sha256

3) Make block-level backups / snapshots (do not run repair on original)- Preferred: take a hardware/storage snapshot (SAN/NAS snapshot) or an LVM snapshot if the filesystem sits on LVM:

bash

lvcreate --size 10G --snapshot --name snap_datavg /dev/vg/datavg
dd if=/dev/vg/snap_datavg of=/backup/images/datavg-snap.img bs=4M status=progress

- If snapshots not available, create a full disk image with dd or using dc3dd, piping through compression and storing off-host:

bash

dd if=/dev/sdX bs=4M | gzip -c > /backup/images/sdX-$(date +%F).img.gz

- Verify image integrity with sha256sum.

4) Work on copies only- Mount or attach the snapshot/image to an isolated recovery host (do not mount read-write). Use losetup for images:

bash

losetup -Pf /backup/images/sdX.img

5) Inspect with read-only tools and dry-run- Use xfs_repair -n (no-modify) to see what would change:

bash

xfs_repair -n /dev/loop0p1

- Use xfs_db and xfs_metadump to collect metadata safely:

bash

xfs_metadump /dev/loop0p1 metadump.out
xfs_db -r -c "sb 0" /dev/loop0p1

6) Minimize data loss when repairing- Never run xfs_repair -L on the production device without testing: -L zeroes the log and can lose recent transactions.- If xfs_repair on copy reports a safe sequence, run repair on the copy until you can mount it and validate data, then plan a controlled restore to production.- If partial file recovery needed, mount repaired copy read-only and rsync critical data out:

bash

mount -o ro /dev/loop0p1 /mnt/recover
rsync -aHAX --progress /mnt/recover/important /backup/recovered/

7) Rollback and validation- Run filesystem checks on the repaired copy, run application-level tests, verify checksums and important application data.- Once confident, schedule a maintenance window to restore from the repaired image/snapshot to production device or replace device, documenting all steps.

Key principles: never run destructive repair on the original without a block-level backup; collect forensic evidence; always test repairs on copies; prefer storage/LVM snapshots for speed; avoid -L unless you accept potential data loss and have no other option.

HardTechnical

79 practiced

Compare btrfs and ZFS for use in a cloud SRE environment. Discuss checksumming and self-healing, snapshot and replication capabilities, compression and deduplication trade-offs, metadata overhead and scalability, licensing/kernel integration issues, and which workloads or operational constraints would favor one over the other.

Sample Answer

Situation: As an SRE evaluating filesystem choices for cloud infrastructure, you need to weigh data integrity, operational complexity, performance, and licensing/compatibility.

Checksumming & self-healing- ZFS: End-to-end checksums on blocks with copy-on-write (COW) semantics and integrated redundancy (RAID-Z / mirrored vdevs). When a checksum mismatch occurs and a redundant copy exists, ZFS can auto-heal reads by replacing the bad block from a good copy. Strong for single-host pools and reliable cluster storage when paired with redundant devices.- btrfs: Also COW with checksums for data and metadata and supports self-repair when using RAID1/10/raid1c1/raid1c2 setups. Historically less mature than ZFS; repairs work but have had more bugs and edge cases. More conservative for production-scale healing.

Snapshots and replication- ZFS: Lightweight, consistent snapshots; efficient send/receive with incremental streaming and resume support. Widely used for backups and replication across hosts. Integration with zvols for block-level use.- btrfs: Also supports cheap snapshots and send/receive, but historically slower and less feature-complete (e.g., no native dedupe-aware send). Works well for single-node snapshots and local rollbacks.

Compression and deduplication trade-offs- ZFS: Inline compression (LZ4 recommended) with minimal CPU overhead and significant I/O reduction. Deduplication exists but is RAM-hungry (dedup table in memory) and can drastically increase metadata; generally discouraged except for narrow use cases with high redundancy and ample RAM.- btrfs: Inline compression available (zstd/lzo) with good performance. Deduplication is not inline (mostly post-process userland tools) — less RAM pressure but adds complexity and delayed benefits. Deduplication on btrfs is more operationally manual.

Metadata overhead & scalability- ZFS: Designed for very large pools; metadata structures scale well but expect high RAM requirements (rule of thumb: ~1–2 GB RAM per TB for certain workloads, though this varies). ZFS excels for multi-disk pools and large datasets.- btrfs: Metadata can become a scaling bottleneck for massive datasets; RAID implementations historically have caveats (e.g., RAID5/6 stability issues). Better for moderate scale and many small hosts but be cautious above several PB or very large metadata counts.

Licensing and kernel integration- ZFS: CDDL license incompatible with GPL, so ZFS is not mainline in Linux kernel; commonly used via out-of-tree modules (ZFS on Linux / OpenZFS). Operationally implies package management, kernel-module compatibility, and delayed updates; on platforms like Solaris/Illumos or native OpenZFS ecosystems this is smooth.- btrfs: In-kernel, GPL-compatible, part of mainline Linux — simpler deployment, fewer compatibility headaches, and works well with distributions and kernel upgrades.

Which workloads/constraints favor each- Favor ZFS when: you need enterprise-grade integrity and self-healing across large pools, heavy snapshot/replication workflows (backups, VM images), predictable performance at scale, and you can manage kernel module lifecycle or use OpenZFS-friendly OS (FreeBSD, illumos). Good for storage-heavy backend services, block-storage for VMs, and datasets where dedupe might be valuable and RAM is available.- Favor btrfs when: you prioritize tight kernel integration, simpler deployment on many Linux hosts, lightweight snapshots for containers/hosts, and lower memory footprint for metadata (relative to ZFS dedupe needs). Suitable for smaller clusters, developer CI hosts, per-node container storage, or when using distribution-supported features (e.g., Fedora/SUSE tooling).

Operational recommendations- Test at scale: run failure-injection, scrub/repair scenarios, and snapshot/replication tests with realistic workloads.- Monitor: ZFS-specific metrics (ARC size, scrub status, pool health) or btrfs checks (balance, device state). Automate scrubs and alert on checksum failures.- Avoid inline dedupe unless you’ve modeled memory and metadata growth; prefer compression (LZ4/LZ) first.- For cloud offerings, consider managed storage (block/object) vs. running ZFS/btrfs on VMs—managed services can offload complexity.

Summary: ZFS provides stronger, battle-tested integrity, replication, and scalability at the cost of kernel integration and memory requirements. btrfs offers smoother Linux integration and simpler ops for moderate scale but is less mature for large-scale, high-reliability storage needs. Choose based on scale, operational tolerance for out-of-tree modules, and workload characteristics.

EasyTechnical

78 practiced

Explain the inode concept on Linux filesystems: what metadata an inode holds (permissions, timestamps, pointers), how inodes relate to directory entries and pathnames, what inode exhaustion means in practice, how to inspect inode usage on a running system, and practical strategies SREs use to mitigate inode exhaustion in production.

Sample Answer

An inode is the filesystem data structure that stores metadata about a file (not its name or contents). Key metadata in an inode:- Permissions, owner (UID/GID), file type- Timestamps: atime, mtime, ctime, (and sometimes birth/creation)- Link count (number of directory entries pointing to it)- File size, block count- Pointers to data blocks (direct/indirect/double/treble indirect)- Extended attributes and flags

Relationship to directory entries and pathnames:- A directory maps names to inode numbers. Pathnames are resolved by walking directory entries to find the inode number, then reading the inode to access content/metadata. Multiple names (hard links) can reference the same inode; deleting a name decrements link count, and the inode is freed only when link count hits zero and no processes have it open.

Inode exhaustion:- Means the filesystem has run out of available inodes even if there is free disk space. Practically this prevents creating new files/directories; services may fail (mail queues, logs, databases).

Inspect inode usage:

bash

# per-filesystem summary
df -i
# find top directories by inode count
for d in /*; do echo $d; find "$d" -xdev -type f | wc -l; done
# find many small files
find /path -xdev -type f -printf '%h\n' | sort | uniq -c | sort -nr | head

SRE mitigation strategies:- Prevention: set appropriate inode ratio at mkfs (tune bytes-per-inode), choose filesystems with dynamic inode allocation (e.g. XFS ext4 with mkfs options).- Monitoring/alerts: alert on high inode utilization (e.g., >80-90%) per filesystem, not just disk usage.- Cleanups/retention: rotate and expire small temporary files, log rotation, TTL for cache directories, garbage-collect orphaned files.- Architectural: avoid storing millions of tiny files on a single filesystem — use object stores (S3), databases, sharding directories, or pack small objects into archives/hashed directory trees.- Automation: run periodic jobs to identify and remove stale files, throttled cleanup during operations, and preflight checks before deploys that create many files.- Emergency actions: move large sets of files to another FS, compress or archive old files, increase inode pool by reformatting with different mkfs parameters (requires downtime).

I routinely combine df -i alerts with automated sweeps and architecture changes (object storage or DB) to prevent inode exhaustion from affecting availability.

HardSystem Design

84 practiced

Design a highly-available, multi-AZ storage architecture for a cloud-hosted web service with a 99.99% availability SLO, 500k reads/sec, 100k writes/sec, and 50TB of hot data. Specify layers (block storage, replication, caching tier, filesystem choice on app servers), backup and restore approach, monitoring/alerting strategy, and trade-offs between cost, consistency and performance.

Sample Answer

Requirements & constraints:- SLO: 99.99% (≈52 min downtime/year)- Load: 500k reads/s, 100k writes/s- Hot data: 50 TB- Multi-AZ, cloud-hosted, strong durability, reasonable cost

High-level architecture:- API / app tier in multiple AZs behind global LB (route traffic by latency + health).- Storage tier: distributed primary storage (sharded) with synchronous replication within-region across AZs for HA, asynchronous cross-region replication for DR.- Caching tier: multi-layer — global CDN for static/edge-cachable content; in-region distributed cache (clustered Redis/Memcached) per AZ with consistent hashing and cross-AZ replication/failover.- Persistent storage for blocks: use managed NVMe-backed block volumes (or local NVMe + replication) attached to stateful nodes only where latency matters.- Database: use a horizontally sharded, write-optimised distributed DB (e.g., CockroachDB/Spanner/Aurora Global DB with writer nodes sharded) to handle 100k writes/s and 500k reads/s with read replicas per AZ.- Object store for large binary content: S3 (or equivalent) with lifecycle for warm/hot tiers.

Replication & consistency:- Intra-region: synchronous replication across 3 AZs for strong consistency and failover (< single-AZ RTO).- Cross-region: async replication with configurable RPO (minutes) for DR and geo-read replicas.- Leader-based writes per shard to keep strong consistency; allow read replicas that are slightly stale for high-throughput reads (tunable staleness).

Caching design:- Hot-path: in-region Redis cluster with active-active clients and replica read-scaling. Use cache-aside pattern and write-through for critical small items.- CDN for public static assets, short TTLs for dynamic cacheable pages, purge hooks on writes.

Filesystem on app servers:- Ephemeral containers/pods use overlay FS; shared file needs use NFSv4 or managed file service (EFS) with multi-AZ mount targets and performance mode tuned (or use object store + prefetching).- Prefer stateless apps storing state in DB/object store; local disk only for temp/scratch on fast ephemeral NVMe.

Backup & restore:- Continuous backups for DB (snapshot + WAL archiving). Daily full snapshots, incremental hourly, retained according to RTO/RPO. Test restores weekly in staging.- Object store: versioning + lifecycle + cross-region replication.- Disaster recovery runbooks: automated failover for AZ (quick), manual/automated cross-region promotion for region failure. Maintain infrastructure as code and warm standby region for RTO targets.

Monitoring & alerting:- Metrics: provisioned + actual throughput, latency (p99/p999), error rates, replication lag, cache hit ratio, disk IO, queue lengths, CPU/memory, free disk.- Traces: distributed tracing for tail latency sources.- Logging: centralized ELK/Fluent; audit logs.- Alerts: multi-tier—page on SLO breach trajectories, urgent on replication lag > threshold, node/volume failures, degraded cache hit ratio, sustained p99 latency above threshold. Use alerting runbooks and automated remediation (auto-replace nodes, scale-out).- SLO-based alerts: alert on burn-rate > threshold to protect error budget.

Capacity & scaling:- Shard DB to keep per-node QPS within limits (e.g., target 10k writes/node -> 10 shards initially).- Cache capacity sized to hold working set; monitor eviction rates.- Autoscaling policies for app and stateless layers; controlled scale for DB with automated shard rebalancing.

Trade-offs:- Cost vs consistency: synchronous cross-AZ replication increases cost and write latency; consider semi-sync or leader locality if lower latency required.- Performance vs durability: local NVMe gives best latency but needs replication; managed volumes are simpler but costlier.- Complexity vs availability: multi-shard distributed DB and active-active caches increase operational complexity but required to meet 99.99% at this scale.- Restore time vs cost: warm-standby cross-region increases cost but reduces RTO; cold DR cheaper but longer RTO.

Why this fits SRE goals:- Designed for high availability via multi-AZ sync replication, caches to absorb read load, autoscaling to handle traffic spikes, and robust monitoring/alerting tied to SLOs with automated remediation and tested backups to meet RTO/RPO targets.

HardSystem Design

71 practiced

Design a backup and disaster recovery system for 200 TB of production block storage spread across multiple data centers. Requirements: daily incremental backups, weekly full backups, RTO < 4 hours for critical datasets, RPO < 1 hour for highest-priority data, and retention/compliance policies. Detail architecture (snapshot vs block-level copy vs agent), cataloging, verification, restore runbooks, and how SREs should operate and test the system.

Sample Answer

Requirements clarification:- 200 TB block storage across multiple DCs. Backups: daily incrementals, weekly fulls. RTO <4h for critical datasets, RPO <1h for highest-priority. Retention & compliance (e.g., 1 year, WORM for some).

High-level architecture:- Primary on-prem/data-center clusters -> Backup gateway layer -> Global backup target (on-prem object store replicated across DCs + cloud immutable vault).- Use combination: crash-consistent snapshots for fast point-in-time, block-level incremental transfer for bandwidth/storage efficiency, and lightweight agents for application-consistent quiescing (databases, VMs).

Components and choices:- Snapshot engine: leverage underlying storage volume snapshots (hardware or hypervisor) for instant RTO-friendly checkpoints.- Block-level deduplicating transfer: use CBT/changed-block-tracking to send only deltas; apply inline/global dedupe and compression on backup gateway to reduce storage and network.- Agents: for databases/transactional apps, run pre-freeze/post-thaw hooks to create consistent checkpoints and log-shipping awareness.- Catalog/metadata store: central immutable catalog (DB + append-only log) storing backup manifests, checksums, encryption metadata, retention tags, legal hold flags; expose API and UI for search and audit.- Replication: weekly fulls and critical incrementals replicated to secondary DC and to cloud cold vault (immutable, versioned).

Verification and integrity:- Every backup: manifest with block checksums and encryption verification; hash-verify on write and after replicate.- Automated restore verification: daily automated test restores (target sandbox) for a sample set (smoke) and full weekly synthetic restore for highest-priority volumes.- Periodic CRC/bit-rot scanning on stored backups.

Restore/runbooks:- Pre-built runbooks per dataset tier: - Tier 1 (highest-priority, RPO<1h, RTO<4h): Failover steps to standby cluster: 1) Identify last consistent snapshot + replay WAL/logs; 2) Mount snapshot read-write on standby, remap network, update DNS/load balancer; 3) Validate app health checks; 4) Promote to primary. Expected time <4h. - Tier 2: Orchestrated restore of LVM/volume from deduped chunkstore; use fast-path snapshot restore where possible. - Tier 3 (archive): Restore to staging bucket and then to volumes (longer RTO).- Include rollback steps, data-consistency checks, and business contact list. Keep runbooks in version-controlled repo, with automated checklists.

SRE operations and testing:- SRE responsibilities: maintain backup gateways, monitor jobs, run restores, manage retention holds, and triage failures.- Monitoring/Alerting: job success/failure, throughput, backlog, retention window breaches, verification failure, replication lag, catalog drift. Integrate with pager and incident runbooks.- Chaos & testing: monthly “fire-drill” disaster recovery exercises for critical datasets: full failover simulation to secondary DC with RTO measurement. Quarterly restore drills for random volumes and annual compliance restore demonstrating retention/WORM.- Automation: use IaC and playbooks (Ansible/Terraform/Runbooks as code) to automatically perform restores into ephemeral test clusters; collect metrics (time-to-restore, data-verified).- Capacity & cost management: track growth, dedupe rates, and network windows; throttle background backups to avoid production impact.

Trade-offs & rationale:- Snapshots provide fastest RTO; block-level incremental reduces network and storage. Agents ensure app-consistency where snapshot alone isn't enough. Replication to secondary DC + immutable cloud meets compliance and improves durability.- Accept additional complexity (catalog, verification) to ensure RPO/RTO targets and auditability.

Key KPIs to track:- Backup success rate, average backup window, mean time to restore (for each tier), verification pass rate, storage growth, replication lag, and compliance audit pass rate.

This design balances speed, efficiency, and compliance while giving SREs clear procedures and automated tests to prove recoverability.

Unlock Full Question Bank

Get access to hundreds of Linux Filesystems and Storage interview questions and detailed answers.

Join thousands of developers preparing for their dream job.