Version Control and Developer Tooling Questions

Direct answer

Confirm it's actually git-level corruption, not a flaky CI runner, by reproducing the failure against a fresh mirror clone and running git's integrity checker. Then work from a known-good copy, a healthy mirror, a backup, or even a healthy developer clone, to restore service without losing history, and only attempt in-place repair on the primary repository if you fully understand what that repair would discard.

Structured elaboration and worked example

1. Detect. Reproduce the failure and get a clean picture of what git is actually complaining about (which object, which pack file):

git clone --mirror https://git.example.com/project.git /tmp/project-check.git
cd /tmp/project-check.git
git fsck --full --no-reflogs

git fsck is git's built-in integrity checker: it walks every object the repository knows about and reports anything missing, unreadable, or inconsistent. Running it against a fresh mirror clone, which clones every ref rather than just the default branch, tells you whether the problem is in the objects themselves or something environmental, like a bad network transfer during a previous clone.

2. Isolate. Note which object IDs and refs fsck flags. If only certain branches or a specific pack file (the compressed archive git bundles many objects into for storage efficiency) are affected, you can often scope the blast radius instead of assuming the whole repository is unusable. Make the remote read-only in the meantime, so nothing else writes on top of a damaged state while you investigate.

3. Restore service, preferring a known-good source over in-place repair:

# from a healthy mirror or off-site backup
git clone --mirror /backup/project.git /tmp/restore.git
cd /tmp/restore.git
git fsck --full          # confirm the source is actually clean before trusting it
git push --mirror https://git.example.com/project.git

If no backup exists but a developer's local clone is healthy, that clone can serve as the restore source instead, using the same clone, verify, and push --mirror sequence.

4. Only if no clean source exists, attempt in-place repair, understanding it can permanently discard unreachable data:

git reflog expire --all --expire=now
git gc --prune=now
git fsck --full

This is a last resort, not a first move, because pruning removes exactly the dangling objects you might otherwise have recovered data from.

5. Prevent recurrence. Automate regular mirror backups and periodic fsck runs, a nightly job that clones a mirror and checks it catches corruption early instead of it surfacing as a broken CI run days later. Restrict who has raw filesystem access to the git server, since corruption is more often caused by a bad disk, an interrupted gc, or direct filesystem tampering than by normal client usage.

Trade-offs and pitfalls

Repairing in place without a verified clean source risks silently losing history a stakeholder assumed was safe. Treating every fsck warning as an emergency is also wrong; some, like a dangling commit (one no branch or tag points to anymore, so nothing reaches it through normal history) from an abandoned branch, are completely benign, so isolate severity before reacting. Keeping only one backup mirror is a single point of failure, just like keeping only one copy of the primary repository; an off-site or separately-hosted mirror is what actually protects your recovery time.

Direct answer

df reports space at the filesystem level, asking the kernel directly how full each mounted volume is, which is near-instant. du walks a directory tree and sums up the actual size of the files it finds, which can be slow on a large tree. They can legitimately disagree: a large file that was deleted while a process still has it open inflates df's used-space number, but du never sees it since it is no longer in any directory listing.

Structured elaboration

# (a) overall filesystem free space, human-readable
df -h

# (b) top 5 largest directories immediately under /var, staying on one filesystem
du -h -d 1 -x /var 2>/dev/null | sort -rh | head -n 5

Key points:

• df -h: -h is human-readable (K/M/G). Typical GNU/Linux columns are Filesystem, Size, Used, Avail, Use%, Mounted on (macOS/BSD df adds inode columns but reports the same underlying data).
• du -d 1: limits depth to 1, immediate subdirectories only, not a full recursive per-file listing. -d N is the portable spelling across both GNU coreutils and BSD du (GNU also accepts --max-depth=N, but plain BSD du only understands -d).
• -x: stay on one filesystem, do not cross into a separately mounted filesystem. This is exactly how you exclude a mounted virtual filesystem, such as a tmpfs mounted at /var/run, without hardcoding its path.
• 2>/dev/null: suppress "Permission denied" noise on directories you cannot read.

Worked example

$ du -h -d 1 mockvar 2>/dev/null | sort -rh | head -5
5.4M	mockvar
1.9M	mockvar/lib
1.8M	mockvar/weird dir
880K	mockvar/run
492K	mockvar/log

This was run against a small mock directory tree, not the real /var (whose sizes are host-specific). Note the root of the tree you point du at (here mockvar itself, in production /var itself) shows up as its own total in the output; if you only want the children, pipe through tail -n +2 after sorting, or filter it out explicitly.

Trade-offs & pitfalls

• df answers "how full is the disk," du answers "how big is this specific tree." They measure different things, and disagreeing by design is normal, not a bug to chase before checking the classic cause: a held-open, deleted file. lsof +L1 finds those.
• du stats every file it counts. On a directory with millions of small files (a real risk under something like a build cache or node_modules), that can be slow and I/O heavy; -d 1 bounds the scan to one level to keep a quick "what's big" survey fast.
• Both tools report allocated (block-rounded) usage by default, not the logical byte count of the file's content. A sparse file can show a smaller size via du than its logical length would suggest; that is expected behavior, not a bug.

Direct answer

Keep the default CI clone shallow for speed, and only fetch the additional history a bisect actually needs, on demand, in the specific job that's running it, rather than deepening every routine build.

Structured elaboration and worked example

1. Routine builds stay untouched, shallow as today:

git clone --depth 1 https://example.com/repo.git

2. A dedicated bisect job, triggered manually or by a "regression, needs history" label, fetches just enough additional history before starting, rather than the full history by default:

# add enough recent commits to cover the suspected regression window
git fetch --deepen=200 origin

# or, if you know roughly when the regression window started
git fetch --shallow-since="2026-01-01T00:00:00Z" origin

# only if targeted deepening isn't enough
git fetch --unshallow origin

--deepen extends a shallow clone's history by a fixed number of additional commits, rather than replacing it; --shallow-since fetches all commits back to a given date, useful when you have a rough idea of when the regression window starts. Both are cheaper than a full --unshallow, so try them first and only fall back to --unshallow if the regression turns out to be older than expected.

3. Run the bisect with an automated test script, so it doesn't need a human to judge good or bad at every step:

git bisect start HEAD <last-known-good-commit>
git bisect run ./ci/regression-check.sh
git bisect reset

git bisect run automates the whole search: git repeatedly checks out a candidate commit and runs your script, treating a zero exit code as "good" and a nonzero exit code as "bad", narrowing the range by binary search each time. The script needs to reliably distinguish good from bad on its own; an inconsistent test makes bisect converge on the wrong commit.

Policy for when to fetch more history

Never deepen for routine pull request or merge builds, since that reintroduces exactly the cost the shallow clone was meant to avoid. Only deepen in a job explicitly requesting a bisect or regression search, or one that failed a first attempt at a shallow bisect and needs to widen its window. Log the depth fetched, before and after, so a slow or unexpectedly large deepen is visible and auditable rather than a silent cost.

Trade-offs and pitfalls

--deepen and --shallow-since are not equivalent: one is commit-count based, the other is date based, and picking the wrong one for a squashed or infrequently-committed history can still leave you short of the actual regression window on the first attempt, forcing a second, larger fetch. Caching git objects between bisect runs on the same runner helps, but only if the cache correctly distinguishes a shallow state from a deepened one; a naive cache that assumes everything is shallow can serve stale, incomplete history to a job that actually needed more.

Direct answer

The staging area, also called the index, is git's holding area for exactly the changes you intend to put into your next commit. git add copies changes from your working directory into that staging area; git commit takes whatever is currently staged and permanently records it as a new commit in history. They're separate steps specifically so you can build a commit out of a chosen subset of your changes, rather than being forced to commit everything you've touched.

Structured elaboration

# stage one specific file
git add config/nginx.conf

# create a commit from whatever is currently staged
git commit -m "Update nginx healthcheck path"

git status shows the distinction directly: files listed under "Changes to be committed" are staged; files under "Changes not staged for commit" are modified in your working directory but not yet added.

Worked example

Say you've edited one file, infra/nginx.conf, and it now contains both a real configuration change and a stray debug line you added while troubleshooting. Staging the whole file would commit the debug line too.

git add -p infra/nginx.conf (patch mode: it walks through the file's changes hunk by hunk and asks yes or no for each one) lets you stage only the real configuration change and leave the debug line unstaged, so it never enters history:

git add -p infra/nginx.conf
# git shows each hunk and asks: Stage this hunk [y,n,q,a,d,...]?

git add -A (stages every change across the whole repository: tracked file edits, brand-new untracked files, and deletions) is the opposite instinct: reach for it when you deliberately want one commit to capture everything that changed, for example right before cutting a release where several files were intentionally updated together:

git add -A
git commit -m "Release 2.4.0: bump image tags, remove deprecated manifests"

Trade-offs and pitfalls

git add -A is easy to reach for out of habit, and that habit is exactly how unrelated or debug changes end up in a commit unnoticed. git add -p takes more time per commit, but produces commits that are easier to review and, if one specific change turns out to be wrong, easier to revert individually rather than untangling it from everything else that landed alongside it.

Direct answer

I would put safeguards in place before the change ever lands in the main branch, not rely on catching it in review by eye: an explicit "do not change the signature" constraint in the prompt, a search across the codebase for every caller of the function being touched, and a CI (continuous integration) check that actually exercises those callers. Catching a broken signature by carefully reading a diff does not scale, since the break is often invisible in the diff of the function itself, it only shows up as a failure somewhere else entirely.

Structured elaboration

Preventing it at the prompt level

• Constrain the request explicitly: "modify the internal logic but do not change the function's name, parameter order, parameter names, or return type," rather than a general "improve this function" that leaves the interface up for grabs.
• Provide the function's current callers as context, so the assistant can see what would break, rather than reasoning about the function in isolation.

Catching it before merge

• Search the codebase for every call site of the changed function before approving the change (a simple text or symbol search, or an IDE's find-references), and confirm each one still matches the new signature.
• Run the full test suite, including integration tests that exercise the function through its real callers, not only unit tests that call the function directly with hand-picked arguments a test author already expects to work.
• Add a lightweight contract check where the interface is easy to characterize: a type-checked function signature, or a small test that specifically asserts the function's signature has not changed, so a future AI-assisted change trips this test immediately rather than being caught downstream.

Preserving backward compatibility going forward

• Default to additive changes: add a new optional parameter with a default rather than changing an existing required one, so old call sites keep working unmodified.
• When a breaking change is genuinely necessary, use a wrapper or adapter function that preserves the old signature and forwards to the new one, and version the interface explicitly if it is public.
• Require an explicit compatibility note in any pull request that touches a function with more than a couple of callers: what changed, who is affected, and how it was validated, so a reviewer does not have to reconstruct that from the diff alone.

Preserving behavior after a pure refactor

A pure refactor (same signature, intended to be behavior-preserving, done purely for readability) has its own distinct checklist beyond signature preservation: confirm the output is identical on a representative set of real inputs, not just that the code compiles; check latency has not regressed, since a "cleaner" rewrite can accidentally introduce an expensive operation inside a loop that used to be outside it; and check error handling specifically, since a refactor is a common place for an exception type or an edge-case branch to silently disappear during a rewrite that otherwise looks like a pure simplification.

Worked example

An AI assistant is asked to "clean up" a function parse_config(path) used in six places across the codebase. It returns a cleaner implementation, but the new version returns None on a missing file instead of raising a FileNotFoundError the way the original did, a behavior change with no signature change at all, so a caller relying on the exception (a try/except FileNotFoundError block elsewhere) now gets a silent None instead, and fails later in a way that is much harder to trace back to this change. A signature-only check would miss this entirely, which is exactly why behavior and error-handling checks matter as a separate step from an interface check, and why "did the tests pass" is not sufficient if none of the six callers had a test covering the missing-file case.

Trade-offs and pitfalls

Contract tests and full-codebase caller searches add real time to every review, which can feel excessive for a function with one caller. The safeguards are worth scaling to the function's actual blast radius: a private helper used in one place needs a lighter check than a function called from a dozen places across the codebase, and treating every change with maximum ceremony trains people to skip the ceremony rather than actually protecting the changes that matter most.