Senior Systems Administrator Interview Preparation Guide - Doordash

Direct answer
Code review becomes a coaching tool the moment you separate what has to change before this merges from what's worth teaching, and handle each differently, since blocking mixes poorly with explaining. What counts as the important risk to teach toward also shifts by what's being reviewed: correctness and style for typical application code, reproducibility and data leakage for ML work, and blast radius for infrastructure changes.

Separate blocking feedback from teaching feedback

• Mark comments explicitly as blocking versus non-blocking (or use a similar convention), so the author isn't left guessing what actually has to change before merge. Teaching comments that aren't required for merge belong in the non-blocking bucket, otherwise you either water down real teaching moments to keep the change unblocked, or block a mergeable change to make a point.
• Ask before you tell: a comment phrased as a question ("what happens if this list is empty?") invites the author to find the issue themselves, which teaches the underlying reasoning; a comment phrased as an instruction just transmits the fix.

What "the important risk" means shifts by artifact type

• Typical application code: the coaching focus is usually correctness, readability, and test coverage; the failure mode being taught against is a defect shipping or the next person not being able to follow the change.
• ML notebooks and experiment configs: the review risk is different in kind, not just degree. The critical things to check and teach toward are reproducibility (is the seed pinned, is the environment specified, can someone else get the same result) and data leakage (does the training data have any path back to the evaluation set, directly or through a shared preprocessing step). A notebook can be clean, readable code and still be dangerously wrong for reasons that have nothing to do with code style.
• Terraform and other infrastructure-as-code changes: the review risk is blast radius, not defects in the traditional sense. A small, correct-looking diff can still be catastrophic if it touches a shared resource or removes a safeguard. Coaching here means teaching someone to ask what does this affect beyond what's in the diff before asking is this line correct.

Making it a genuine teaching moment, not just a gate

• When there's something worth teaching, don't just fix it in the comment; explain the why, and where useful, point to a real example elsewhere in the codebase rather than a generic principle.
• For anything too deep to unpack asynchronously in a comment thread, offer a short pairing session instead of a long comment chain; some things teach faster live than in writing.
• Close the loop: after a pattern comes up more than once for the same person, raise it directly in a 1:1 rather than only ever surfacing it inside individual review threads, so it becomes a recognized growth area instead of a recurring surprise.

Worked example
Reviewing a teammate's change that added a new model training script, the code itself was clean and well-tested in the conventional sense. The actual coaching moment was elsewhere: the evaluation split was built after a preprocessing step that had already seen the full dataset, which meant the reported accuracy was optimistic in a way unit tests would never catch. Rather than just fixing the split order and moving on, the comment walked through why that ordering matters (what leakage actually does to the reported number) and pointed to another script in the repo where the split happened correctly, before the shared preprocessing step. That change did get blocked, since the leakage was a real correctness issue, but the teaching part was the explanation of why, not the fact that it was blocked.

Trade-offs and pitfalls

• Making every comment a teaching moment, including on merge-blocking issues, slows delivery and can read as review turning into a lecture; save the deeper explanations for the genuinely worthwhile ones and keep routine fixes routine.
• Applying the same review lens (say, defect-finding) to every artifact type misses the risks that matter most for that artifact; a Terraform change reviewed like application code will pass style and correctness checks while missing blast radius entirely.
• If teaching moments only ever show up as isolated review comments and never get named directly to the person as a pattern, growth stays implicit and slower than it needs to be.

Approach

• Read /proc/stat twice a short interval apart to compute instant CPU busy/total delta.
• Read /proc/meminfo once to compute used = total - available.
• Print Prometheus-style metric lines with label host set to the local hostname.
• Use only Python stdlib.

Code

#!/usr/bin/env python3
import time, socket

def read_cpu():
    with open('/proc/stat','r') as f:
        line = f.readline()
    parts = line.split()
    # fields: cpu user nice system idle iowait irq softirq steal guest guest_nice
    vals = list(map(int, parts[1:]))
    idle = vals[3] + vals[4]  # idle + iowait
    total = sum(vals)
    return idle, total

def read_mem():
    mem = {}
    with open('/proc/meminfo','r') as f:
        for l in f:
            k,v = l.split(':',1)
            mem[k.strip()] = int(v.split()[0])  # kB
    total = mem.get('MemTotal',0)
    available = mem.get('MemAvailable', mem.get('MemFree',0) + mem.get('Buffers',0) + mem.get('Cached',0))
    used_pct = 0.0 if total==0 else (total - available) / total * 100.0
    return used_pct

def main():
    host = socket.gethostname()
    idle1, total1 = read_cpu()
    time.sleep(0.1)
    idle2, total2 = read_cpu()
    idle_delta = idle2 - idle1
    total_delta = total2 - total1
    cpu_pct = 0.0
    if total_delta > 0:
        cpu_pct = (1.0 - idle_delta / total_delta) * 100.0
    mem_pct = read_mem()
    print(f'node_cpu_percent{{host="{host}"}} {cpu_pct:.2f}')
    print(f'node_memory_percent{{host="{host}"}} {mem_pct:.2f}')

if __name__ == '__main__':
    main()

Notes

• Short sleep (0.1s) gives near-instant utilization; increase for smoother average.
• Memory values are in kB; percentage calculation is unit-neutral.
• Edge cases: missing /proc entries (containers), zero total delta checked.

Runbooks rot because nothing forces them to change when the system they describe changes. Keeping them accurate means giving every runbook a named owner, tying updates to the events that actually invalidate a runbook (a relevant code or infra change, or an incident where it was used), and having a lightweight, recurring check that catches staleness before an incident does, rather than relying on someone remembering to update it.

Ownership

Every runbook has one named primary owner, the team or person who owns the service it covers, recorded in the runbook itself, not a separate spreadsheet that gets forgotten. Ownership isn't honorary: the owner is the one who signs off that the runbook is still accurate at each review point, and the one paged if a stale runbook causes a bad outcome during an incident.

What triggers a review, not just a calendar date

Calendar-only review cadences ("review every quarter") catch some staleness but miss the more common case: a runbook goes stale the day the system changes underneath it, not on a schedule. Two triggers matter more than the calendar:

• Change-linked: any deploy that touches the commands, infra, or thresholds a runbook references should require a runbook update as part of that same change, enforced by a checklist item on the pull request, not a follow-up ticket that competes with the next sprint.
• Incident-linked: every time a runbook is actually used during an incident, the post-incident review includes a specific question, did the runbook match reality, and any gap becomes a tracked follow-up before the incident is closed.

Lifecycle stages

Catching a stale runbook before someone relies on it

A stale runbook is genuinely dangerous exactly because it looks trustworthy right up until the moment it's wrong. The cheapest catch is a light automated check, does every command in the runbook reference a tool, dashboard, or endpoint that still exists, run periodically and flagging anything broken for owner review. The more valuable catch is a periodic game-day: pick a runbook, have someone unfamiliar with the system try to follow it against a staging environment, and see where it breaks. Anything that trips someone up in a drill would have tripped up the on-call engineer at 3am.

Worked example: what enforcement actually looks like

A pull request changes the retry/backoff config for a service. The PR template includes a checkbox, "does this change any runbook referenced by this service's on-call docs?" Because the change alters a value a runbook's remediation step depends on, the author checks yes and links the runbook update in the same PR. Two months later, that runbook gets used during an incident; the post-incident review confirms the values matched, so the runbook stays Active with its last-reviewed date updated, no separate ticket needed because the change-linked trigger already did the work.

Trade-offs and pitfalls

Tying every infra change to a mandatory runbook update adds friction to routine PRs, so the check needs to be scoped narrowly (does this specific change affect a specific runbook) rather than a blanket "update all docs" gate that people learn to click through without reading. The most common failure mode is having a lifecycle model on paper but no one actually enforcing the Needs review to Active transition, so runbooks accumulate silently in Needs review and the label stops meaning anything; the fix is making that queue visible (a dashboard, not a buried label) and reviewing it in the same recurring meeting as on-call handoffs.

Direct answer
Culture fit asks whether you already share a team's existing norms and behaviors; culture add asks what you would bring that the team does not already have. I would describe myself mostly as a culture add: I share the fundamentals a team needs to trust me (reliability, candor, respect for other people's time), but the useful thing I offer beyond that is a genuinely different working background rather than a mirror of the team that is already there.

Structured elaboration

1. Define both terms precisely before answering for yourself. Culture fit is about alignment on shared behaviors and values: does this person operate the way we already operate. Culture add is about complementary difference: does this person's background, working style, or perspective fill a gap the team doesn't currently have.
2. Explain why the distinction matters, not just define it. A team optimized purely for fit tends toward groupthink: everyone reasons the same way, so blind spots go unchallenged and the same kinds of mistakes recur. A team that only adds without any shared fit becomes uncoordinated: people can't predict each other's reasoning enough to move fast together. The healthy target is fit on a small number of load-bearing behaviors (honesty, follow-through, respect) plus deliberate add on everything else.
3. Give a genuine, specific example of your own add, not a generic trait. Vague claims ("I bring diverse perspectives") are the single most common failure mode here; a strong answer names the concrete gap and the concrete evidence.
4. Anticipate the natural follow-up: how do you know your difference is actually useful, versus just different for its own sake. The answer is to point at a specific decision, disagreement, or piece of feedback that changed because of the difference you brought, not just a credential or background fact.

Worked example
Suppose your last two teams were both product engineering teams building consumer-facing features, and the team you're interviewing for is mostly staffed by engineers with that same background. Your own prior role was on a data-platform team, closer to the systems that feed those consumer features than to the features themselves. A concrete add-story: in a past project, a product team wanted to ship a new recommendation feature quickly; because of your platform background, you asked a question the rest of the team hadn't raised (whether the upstream data pipeline's freshness guarantees actually matched what the feature's UI implied to users), which surfaced a real gap between a 24-hour batch refresh and a UI copy that said "updated just for you." The team fixed the copy and adjusted the refresh cadence before launch rather than after a user complaint. That is a genuine add: a different background produced a question the existing team composition was less likely to ask on its own, and it changed a real outcome.

Trade-offs & pitfalls
The common failure is answering only the definitional half (correctly explaining fit versus add) and then, when asked for a personal example, retreating to generic self-description ("I'm a good communicator", "I care about quality") that any candidate could say and that does not actually demonstrate difference. A second pitfall is overcorrecting into implying you don't fit at all; the strongest answers are explicit that you also share the small set of behaviors every functioning team needs, and that add is about everything on top of that baseline, not a replacement for it.

I would split the solution by responsibility, not by environment. A module should do one job well.

Module layout

• network: VPC, subnets, routes, NAT, and network tags
• compute: app instances, ECS or ASG, load balancer, security groups
• database: managed DB, subnet group, parameter group, DB security group
• root stack: wires the outputs together

Why this works
The network changes slowly, compute changes often, and the database has its own lifecycle and risk. Keeping them separate reduces blast radius and makes reviews easier.

Safe interface
I would expose only what callers need:

• From network: vpc_id, private_subnet_ids, public_subnet_ids
• From compute: alb_dns_name, app_sg_id
• From database: endpoint, port, and maybe a secret reference, not a password

Example
The root module can pass private_subnet_ids = ["subnet-101", "subnet-202"] into compute and database, while dev and prod use different sizes through variables. That keeps composition flexible without letting one team edit module internals.

Direct answer

Start with the component diagram. It establishes what pieces exist and who owns each one, before anything about behavior or payloads makes sense; architects can't reason about "what happens when" until they know "what's here."

Structured elaboration

1. Component diagram (what exists). Purpose: boundaries and ownership. Show: application services, cache cluster nodes, the source-of-truth database, an invalidation service, and a message broker. Leave off: exact protocol, message schema, and timing, those belong later.

2. Sequence diagram (what happens, in order). Purpose: the actual interaction for one invalidation event. Show: a write to the database, the database acknowledging it, an event published to the invalidation service, that service publishing an evict message on the broker, the broker fanning out to cache nodes, and one failure path (broker unavailable: what serves stale data, and for how long). Leave off: byte-level payload detail and retention settings, that's the next diagram's job.

3. Data-flow diagram (what exactly, and how stale). Purpose: payloads and guarantees. Show: the invalidation message's schema (key, version, timestamp), time-to-live, message size, and the one metric architects will actually watch, invalidation latency or staleness window. Leave off: anything already covered by the component-level framing.

Why this order helps comprehension: each diagram answers the question the previous one raised. Component diagram: "what is the invalidation service." Sequence diagram: "how does it know to fire." Data-flow diagram: "how stale can a read get before this evicts it." Reversing the order, starting with the sequence diagram, forces you to define every box mid-sentence instead of pointing at one the audience has already seen.

Worked example

The component diagram you'd draw first:

flowchart LR
    App[Application] -->|write| DB[(Database)]
    App -->|read| Cache[(Cache Cluster)]
    DB -->|change event| Invalidator[Invalidation Service]
    Invalidator -->|publish evict msg| Broker[[Message Broker]]
    Broker -->|fan out| Cache
    Cache -->|miss, reload| DB

Narrated: "The application writes to the database. That write triggers a change event to the invalidation service, which publishes an evict message on the broker. The broker fans that message out to every cache node, and the next read that misses reloads from the database."

Translating the core idea for the architects: the jargon term is "cache coherence." Plain version: "keeping the cache from serving an answer that's gone stale since the database changed." Analogy: it's like a library's card catalog. When a book gets re-shelved, someone has to walk over and update the card, or the next person who checks the card gets sent to the wrong shelf. Where the analogy breaks: no single librarian updates every card at once across a building, the fan-out to many cache nodes in parallel, possibly across regions, is exactly what makes this hard in practice, and that's the detail worth naming once the audience has the basic picture.

Trade-offs & pitfalls

The common wrong turn is leading with the sequence diagram because it feels more "technical," which forces you to define the invalidation service, the broker, and the cache cluster mid-sentence instead of pointing at boxes the audience already recognizes. A second pitfall: putting the failure path (broker down) in the component diagram instead of the sequence diagram, error paths are behavior over time and belong where the audience is already reasoning about timing. A third: overloading the data-flow diagram with architectural detail that duplicates the first diagram instead of adding new information (payload size, TTL, staleness), which makes the customer conversation feel repetitive rather than cumulative.

Direct answer

Rank by a deliberate criterion, not gut feel. Name the criterion you're using, which skills unlock the most future scope, which have the highest impact weighed against feasibility, or which close the gap between your current level and the next one, and include both technical and non-technical or soft skills rather than defaulting to an all-technical list, since most next-level gaps involve at least one of each.

Structured elaboration

Pick and state your ranking criterion explicitly before naming the skills, since the same three skills can be justified very differently depending on whether you're optimizing for near-term impact, long-term career growth potential, or impact weighed against feasibility. Naming the criterion is itself part of a strong answer.

Include at least one non-technical or soft skill alongside technical ones. An all-technical list usually signals either an early-career stage where that's genuinely the right focus, or a blind spot at a more senior stage, where communication, prioritization, or influence often matter more than additional technical depth.

Give each skill a brief, honest reason it made the cut over an alternative you considered and rejected. A ranked list without visible trade-offs reads as a wish list, naming what you left off, and why, shows the ranking was real.

Tie each skill back to a concrete situation where its absence cost you something or its presence would have helped, rather than justifying it in the abstract.

Worked example

"Using what most limits my scope right now as my ranking criterion, my list was: first, a specific domain depth I'm missing that currently forces me to hand off certain problems to someone else, second, clearer stakeholder communication, because I've noticed my updates sometimes need a follow-up conversation to clarify what I actually meant, third, prioritization under competing demands, since I've occasionally said yes to too much and delivered several things late rather than a few things well. I considered adding a fourth, negotiation, but ranked it below the other three because I have fewer real situations right now where it's the binding constraint, so investing there first would be lower leverage. If I ranked by long-term career growth potential instead of near-term scope, prioritization and communication would likely move above the technical depth item, since those compound more as scope grows."

Trade-offs & pitfalls

• A list with no explicit ranking criterion invites the interviewer to wonder whether it was thought through or assembled on the spot.
• An all-technical list at a more senior stage often signals a blind spot, since interpersonal and organizational skills tend to become the actual constraint past a certain level.
• An all-soft-skills list with no technical or domain component can read as avoiding the harder, more measurable half of growth.
• Naming skills with no honest reason for the ranking, or no example of where the gap actually showed up, turns a specific answer into a generic one that could apply to almost anyone.

Immediate goals: preserve evidence, gather diagnostic data, minimize downtime (failover/rollback), avoid further writes to damaged system.

Initial triage (both OSes)

• Verify reachability and alerts, note time of crash and affected services.
• Put host into maintenance mode (disable monitoring/automated remediation) and if possible shift traffic to standby.

If system still reachable (Linux)

• Capture uptime/last boot and kernel messages:

who -b
journalctl -k -b -1    # logs from previous boot
dmesg -T | tail -n 200

• Preserve crash dump / vmcore:

# check kdump
systemctl status kdump
ls -l /var/crash /var/lib/kdump
# copy vmcore off-box
scp /var/crash/<vmcore> analyst@collector:/data/

• Trigger safe sysrq (if hung and acceptable):

echo s > /proc/sysrq-trigger   # sync
echo u > /proc/sysrq-trigger   # remount ro
echo b > /proc/sysrq-trigger   # reboot (only if necessary)

• Collect config and state:

tar czf /tmp/sysinfo.tgz /etc /var/log
ss -tunap > /tmp/conns.txt
ps aux > /tmp/ps.txt
scp /tmp/sysinfo.tgz analyst@collector:/data/

• If memory dump missing, use makedumpfile/crash utilities to extract vmcore.

If system still reachable (Windows BSOD remote collection)

• Retrieve minidump and MEMORY.DMP:
  • Copy from \host\C$\Windows\Minidump and C:\Windows\MEMORY.DMP to safe storage.
• Query Windows event logs remotely:

Get-WinEvent -ComputerName host -FilterHashtable @{LogName='System';ID=1001} | Format-List
wevtutil qe System /q:"*[System[(EventID=1001)]]" /f:text > system_bsod.txt

• Check crash settings:

Get-WmiObject -Class Win32_ComputerSystem -ComputerName host

• If machine down, collect SAN snapshots or disk images rather than booting to avoid altering evidence.

Forensics & analysis

• Preserve timestamps and chain-of-custody (who copied which files and when).
• Analyze vmcore/minidump with crash, makedumpfile, WinDbg:
  • Linux: crash /usr/lib/debug/lib/modules/... /path/to/vmcore
  • Windows: WinDbg -z MEMORY.DMP !analyze -v

Restore service safely

• If quick rollback possible, failover to replica or boot last known-good image.
• If repair required, boot into rescue mode, apply kernel rollback or driver updates, test in staging, then bring back gradually.
• Postmortem: root cause, mitigations (kdump enabled, kernel/drivers pinned, monitoring alerts), and update runbooks.

Key priorities: copy crash dumps off-box, avoid writing to evidence disks, restore service via failover, and perform controlled analysis offline.

Direct answer

Effective NTFS (Windows' native filesystem, which stores its access control list directly on each file and folder) permission for a user comes down to three steps: expand the user's full group membership including nested groups, gather every access control entry (ACE, one "allow X to do Y" or "deny X from doing Y" rule) that applies to the path, then evaluate those ACEs in Windows' canonical precedence order: explicit Deny beats explicit Allow, and any explicit ACE on the object beats any inherited ACE regardless of how the inherited one arrived. A bit that no ACE ever mentions is denied by default.

Structured elaboration

Step 1: expand nested group membership. A naive script that only checks (Get-ADUser $user -Properties memberOf).memberOf against Active Directory (AD, Windows' enterprise directory service that stores users, groups, and their memberships) misses indirect membership (user in Group A, Group A nested inside Group B). Get-ADPrincipalGroupMembership does not fix this either: it returns only direct membership, with no built-in recursive expansion. Two options actually give you the full transitive closure, the same set of groups Windows itself burns into the user's access token at logon:

• The tokenGroups constructed attribute: a computed (not stored) attribute you can query in a single request, e.g. (Get-ADUser $user -Properties tokenGroups).tokenGroups, which returns every security-group SID (security identifier, the unique ID Windows uses instead of a name) the user carries, nesting already resolved.
• [System.DirectoryServices.AccountManagement.UserPrincipal]::FindByIdentity($ctx, $user).GetAuthorizationGroups(), a .NET method with the same fully-expanded semantics, usable directly from PowerShell.

Either is a single, efficient call. Avoid hand-rolled recursive Get-ADGroupMember -Recursive walks over every group in the domain; they are what makes a naive version of this script slow.

Step 2: gather the ACE list. Call Get-Acl $Path (or (Get-Item $Path).GetAccessControl()) exactly once. NTFS does not compute inheritance dynamically at access-check time; when inheritance is enabled, Windows copies each ancestor's inheritable ACEs down onto every descendant's own security descriptor at the time the ACL is set or propagated. That means the single Access collection returned by Get-Acl on the target file already contains both kinds of entry, distinguished by the .IsInherited boolean on each FileSystemAccessRule: explicit ACEs (IsInherited = $false) defined directly on that file, and inherited ACEs (IsInherited = $true) copied down from ancestors. You do not need to manually walk up the parent chain in the common case; you only need the ordering already reflected in that one collection, which Windows keeps canonical (explicit block first, inherited block after, nearer ancestors before farther ones within the inherited block).

Step 3: evaluate in precedence order. For each requested right, walk the ACEs in canonical order and let the first ACE that mentions the intersection of "this right" and "one of the user's resolved principals" decide it. Once a bit is decided it cannot be overridden by a later ACE. This is exactly why explicit Deny is the most dangerous kind of entry: it wins over every inherited Allow no matter how deep the inheritance chain, and it is invisible unless you inspect the target object directly.

Putting it together: the function signature. In production this is a single entry point taking exactly the two inputs the question asks for:

# Production shape (illustrative, not executed here: no live NTFS/AD environment
# is available in this authoring sandbox; the engine it calls IS executed below).
function Get-EffectiveNtfsPermission {
    param([string]$UserName, [string]$Path)
    # Step 2: real ACE list, both explicit and (already-propagated) inherited entries.
    $aces = (Get-Acl -Path $Path).Access | ForEach-Object {
        [Ace]::new($_.IdentityReference.Value, [string[]]$_.FileSystemRights.ToString().Split(','),
                    $_.AccessControlType.ToString(), (-not $_.IsInherited), 0)
    }
    # Step 1: real nested-membership expansion, e.g. via the tokenGroups attribute.
    $membership = (Get-ADUser -Identity $UserName -Properties tokenGroups).tokenGroups
    # Step 3: the precedence engine, unchanged from what is executed below.
    Get-EffectivePermission -UserName $UserName -AllAces $aces -Membership $membership
}

Get-Acl -Path supplies the real ACE list; tokenGroups supplies the real, transitively-resolved principal set; Get-EffectivePermission is the precedence engine, executed below against pinned, hand-built data standing in for those two real calls. The algorithm is identical either way; only where the inputs come from changes.

Worked example

The script below models this precisely (a self-contained scenario, not a live NTFS volume, so the algorithm is checkable anywhere PowerShell runs): jdoe is nested three levels deep (Interns -> Contractors -> AllStaff). An explicit Deny on the target file blocks Contractors from writing, while an inherited Allow from the parent folder grants AllStaff both Read and Write, and a farther-ancestor inherited Allow grants Everyone Read.

class Ace {
    [string]$Principal; [string[]]$Rights; [string]$AceType; [bool]$Explicit; [int]$Level
    Ace([string]$p, [string[]]$r, [string]$t, [bool]$e, [int]$l) {
        $this.Principal = $p; $this.Rights = $r; $this.AceType = $t; $this.Explicit = $e; $this.Level = $l
    }
}

function Expand-GroupMembership {
    # Stand-in for tokenGroups / GetAuthorizationGroups(): resolves nested membership.
    param([string]$UserName, [hashtable]$Membership)
    $principals = New-Object System.Collections.Generic.HashSet[string]
    [void]$principals.Add($UserName); [void]$principals.Add('Everyone')
    $queue = New-Object System.Collections.Generic.Queue[string]
    $queue.Enqueue($UserName)
    $visited = New-Object System.Collections.Generic.HashSet[string]
    while ($queue.Count -gt 0) {
        $current = $queue.Dequeue()
        if (-not $visited.Add($current)) { continue }   # cycle guard
        foreach ($groupName in $Membership.Keys) {
            if ($Membership[$groupName] -contains $current) {
                if ($principals.Add($groupName)) { $queue.Enqueue($groupName) }
            }
        }
    }
    return $principals
}

function Get-EffectivePermission {
    param([string]$UserName, [Ace[]]$AllAces, [hashtable]$Membership)
    $principals = Expand-GroupMembership -UserName $UserName -Membership $Membership
    # Canonical order: explicit Deny, explicit Allow, then inherited Deny/Allow nearest-level-first.
    $ordered = @()
    $ordered += $AllAces | Where-Object { $_.Explicit -and $_.AceType -eq 'Deny' }
    $ordered += $AllAces | Where-Object { $_.Explicit -and $_.AceType -eq 'Allow' }
    $levels = ($AllAces | Where-Object { -not $_.Explicit } | ForEach-Object { $_.Level } | Sort-Object -Unique)
    foreach ($lvl in $levels) {
        $ordered += $AllAces | Where-Object { -not $_.Explicit -and $_.Level -eq $lvl -and $_.AceType -eq 'Deny' }
        $ordered += $AllAces | Where-Object { -not $_.Explicit -and $_.Level -eq $lvl -and $_.AceType -eq 'Allow' }
    }
    $decided = @{}; $trace = @()
    foreach ($ace in $ordered) {
        if (-not $principals.Contains($ace.Principal)) { continue }
        foreach ($right in $ace.Rights) {
            if (-not $decided.ContainsKey($right)) {
                $decided[$right] = $ace.AceType
                $scope = if ($ace.Explicit) { 'explicit' } else { "inherited(level $($ace.Level))" }
                $trace += "  $right -> $($ace.AceType) via $scope ACE on principal '$($ace.Principal)'"
            }
        }
    }
    $effective = @('Read','Write','Execute','Delete','ChangePermissions','TakeOwnership') |
        Where-Object { $decided.ContainsKey($_) -and $decided[$_] -eq 'Allow' }
    [pscustomobject]@{ User=$UserName; Principals=($principals -join ', '); Effective=($effective -join ', '); Trace=$trace }
}

$Membership = @{ 'Interns' = @('jdoe'); 'Contractors' = @('Interns'); 'AllStaff' = @('Contractors') }
$Aces = @(
    [Ace]::new('Contractors', @('Write'), 'Deny', $true, 0)               # explicit, on the file itself
    [Ace]::new('AllStaff', @('Read','Write'), 'Allow', $false, 1)         # inherited from parent folder
    [Ace]::new('Everyone', @('Read'), 'Allow', $false, 2)                 # inherited from grandparent
)

$result = Get-EffectivePermission -UserName 'jdoe' -AllAces $Aces -Membership $Membership
Write-Output "User: $($result.User)"
Write-Output "Resolved principal set (direct + nested groups): $($result.Principals)"
Write-Output "Effective rights: $($result.Effective)"
Write-Output "Decision trace:"
$result.Trace | ForEach-Object { Write-Output $_ }

Output (actually run with pwsh, unmodified):

User: jdoe
Resolved principal set (direct + nested groups): jdoe, Everyone, Interns, Contractors, AllStaff
Effective rights: Read
Decision trace:
  Write -> Deny via explicit ACE on principal 'Contractors'
  Read -> Allow via inherited(level 1) ACE on principal 'AllStaff'

jdoe ends up with Read only. Despite AllStaff (which jdoe reaches through two levels of nesting) being granted Write by inheritance, the explicit Deny on Contractors (also reached through nesting) wins, because explicit always outranks inherited regardless of level. This is the exact trap a naive "just check if any Allow ACE matches" script misses: it would wrongly report Write access.

Complexity. With $g$ total groups in the directory and $d$ the nesting depth, tokenGroups/GetAuthorizationGroups() resolve membership in one call rather than the $O(g \cdot d)$ a hand-rolled recursive walk would cost. ACE evaluation itself is $O(a \cdot p)$ for $a$ ACEs on the path and $p$ resolved principals (a simple membership test per ACE), which is small in practice (single-digit to low-double-digit ACE counts per object). Edge cases: a broken inheritance chain (-Protected set somewhere in the ancestry) truncates the inherited block below that point, an ACE that names a right the algorithm never evaluates should be ignored, and a user removed from AD mid-session still holds their old token's groups until re-authentication, so a live effective-permission check and a currently-open session's actual access can legitimately disagree.

Trade-offs and pitfalls

The single biggest pitfall is treating "is there an Allow ACE that matches" as sufficient; every explicit Deny anywhere on the object must be checked first, and it is easy to miss because it does not show up when you only look at inherited permissions in the folder's parent. The second is under-resolving group membership: memberOf and Get-ADPrincipalGroupMembership both under-report nested access, which silently produces false "access denied" conclusions.

Policy-simplification recommendation. The scenario above is also a demonstration of why relying on scattered explicit Deny ACEs is bad policy design, not just hard to audit: this one Deny only exists because someone needed a narrow, ad hoc exception to an otherwise-correct inherited Allow. The structural fix is to remove the need for the Deny at all, by restructuring group membership (pull the users who need the exception out of the inheriting chain into a separate security group scoped to a sub-folder with its own, narrower inherited Allow) rather than layering a Deny on top. Microsoft's own long-standing guidance is the same: prefer restructuring group nesting or removing a principal from the granting group over adding a Deny ACE, because every explicit Deny added is one more rule a future admin (or this exact algorithm) has to evaluate first, forever, to get the right answer.

Direct answer

Regulatory and compliance regimes almost never dictate a specific recovery architecture; they dictate what your continuity program has to contain, document, and be able to prove on demand. In practice that means: a written plan with defined scope, a documented business impact and criticality analysis, defined and justified recovery objectives, evidence that the plan has actually been tested (not just written), and, for most regimes that matter to a mid-size company, explicit coverage of the third parties your critical functions depend on. Different regimes add different specifics on top of that shared skeleton, and the program has to satisfy all of the ones that apply at once, not run a separate plan per regulation.

Structured elaboration

What auditors and examiners actually check. Across the regimes I've worked with, the common ask is: a documented plan naming its scope, a documented impact/criticality analysis, recovery objectives with a rationale for how they were set, dated records of exercises and their findings, evidence that findings got remediated, and a defined review and maintenance cadence. The plan is the artifact; a well-run internal system with no paper trail behind it often fails an audit anyway, because there's nothing to sample.

How named regimes each shape the program:

What actually changes in the program because of these obligations. Scope has to be explicit (what's covered, what's deliberately out, and why), evidence has to be retained (dated exercise records, findings, remediation sign-off), third parties have to be inventoried and included proportional to criticality, and there has to be a named owner and review cadence for the plan itself, not just for the systems it covers.

Worked example

A mid-size regional hospital network under HIPAA needs a written contingency plan covering every system that touches electronic protected health information. Concretely: a data backup plan (required), a disaster recovery plan to restore lost data (required), and an emergency-mode operations plan describing how critical clinical functions, like verifying a patient's active medication list before a new prescription is administered, keep running safely if the electronic health record system is down (required). Testing and a criticality analysis are "addressable": the network can satisfy this by running an annual tabletop exercise and documenting the results, or by documenting a specific reason that approach isn't reasonable for a given system and describing the equivalent measure it uses instead. If the Office for Civil Rights investigates after an incident, what gets requested first is the plan itself, the criticality analysis, and the exercise records, not a description of the network's storage architecture.

A second, shorter example: a mid-size payments company subject to bank-examiner-style BCM oversight maintains a register of its critical ICT third-party providers (its card processor, its cloud host) that states each provider's own tested recovery capability, the contractual audit and notification rights the company holds, and how that provider is included in the company's own exercise program. That register, not a technical diagram, is what an examiner reviews.

Trade-offs and pitfalls

The most common and costly failure is treating compliance as a paperwork exercise: a plan that exists, was reviewed once at signing, and has never been tested satisfies no one, regulator or business, when a real disruption hits. A second failure is misreading HIPAA's "addressable" as "optional": it still requires either the specified control or a documented, reasoned equivalent, and "we didn't get to it" is not that. A third is scoping the plan too narrowly to reduce testing burden, which creates a gap exactly where the regulator (and the business) cares most. Finally, when an organization sits under more than one regime at once (a healthcare company that also processes card payments, for example), the efficient answer is one program whose documentation maps to each regime's specific asks, not parallel plans that drift out of sync with each other.