DoorDash Systems Administrator (Entry Level) - Comprehensive Interview Preparation Guide

Direct answer

The system has one authoritative trigger source (the HR platform's joiner/mover/leaver events), one authoritative identity store (Active Directory or LDAP, AD/LDAP), a lifecycle engine that translates HR events into group-membership changes, SSSD (System Security Services Daemon, the Linux client that resolves and caches AD/LDAP identity locally) on all 1,000 hosts, a Privileged Access Management (PAM) platform that brokers time-boxed break-glass access for contractors, a configuration-management reconciliation loop that catches hosts back up after disconnection, and a central, append-only audit log that every other component writes to. Disconnected hosts are handled by treating propagation as eventually consistent rather than instantaneous, and every temporary grant is enforced with an expiry the system checks itself, not one a human has to remember.

Structured elaboration

Components.

• HR system (source of truth). Emits joiner, mover, and leaver events, including a contractor's contract end date, as the single primary trigger; manual tickets remain an exception path, not the normal mechanism, so the system's behavior does not depend on someone remembering to file a request.
• Identity lifecycle engine. Consumes HR events and translates a business event ("Alice moved from Sales to Finance," "Bob's contract ends on this date") into concrete access actions (add and remove specific role groups, schedule an auto-disable). This is the single place role-to-group mapping logic lives, rather than being duplicated per downstream system.
• Central identity store (AD/LDAP). The authoritative account and group-membership database that every Linux host defers to instead of maintaining local accounts.
• SSSD on each of the 1,000 Linux hosts. Resolves AD/LDAP-defined users and groups into local Linux identity and authenticates against the central store, caching recently resolved identity data locally, which matters directly for the disconnected-host case below.
• Privileged Access Management (PAM) platform. Note the acronym collision worth flagging for clarity: this is a distinct thing from Linux's own Pluggable Authentication Modules, also abbreviated PAM, which is the local authentication framework SSSD plugs into on each host. The Privileged Access Management platform here brokers break-glass and other temporary elevated access for contractors: vaulting credentials, granting time-boxed access, recording sessions, and enforcing expiry, rather than the lifecycle engine building bespoke privileged-access logic of its own.
• Configuration-management / reconciliation layer. Applies host-local policy (sudoers scoping derived from group membership, for example) on every host and, critically, re-pulls current authoritative state on every scheduled run, acting as the retry mechanism for any host that missed a live push while offline.
• Central audit log aggregation. Every other component ships its events here: HR event ingestion, the lifecycle engine's decisions, native AD/LDAP change auditing, the PAM platform's grant/use/expiry and session-recording events, and each host's own local authentication logs.

flowchart TD
    A[HR system: joiner, mover, leaver events] --> B[Identity lifecycle engine]
    B --> C[Central identity store: AD/LDAP]
    C --> D[SSSD on each of 1000 Linux hosts]
    B --> E[PAM platform: break-glass and temporary elevation]
    E --> D
    F[Configuration management reconciliation loop] --> D
    D --> G[Central audit log aggregation]
    B --> G
    C --> G
    E --> G

Data flow per event type. Joiner: the HR event drives the lifecycle engine to create the AD/LDAP account and assign baseline and role-derived group memberships; SSSD on any host the new hire needs picks up the identity on its next lookup, and the configuration-management layer converges any host-local artifacts (home directory, derived sudoers entries) on its next run. Mover: the lifecycle engine computes the difference between the old role's groups and the new role's groups and applies both the removals and the additions in the same operation, since removing only the additions and forgetting the removals is the single most common gap in otherwise well-designed lifecycle systems. Leaver: the lifecycle engine disables (never immediately deletes) the account to preserve forensic history, revokes any PAM-platform-vaulted grants tied to that identity immediately, and schedules deletion or archival for later under a retention policy rather than instantly. Contractor break-glass: a request triggers a PAM-platform-brokered, time-boxed grant (temporary sudoers-mapped group membership or a vaulted credential checkout), which the platform expires automatically, with the full session recorded and logged.

Contractor auto-disable and re-enable on approval. Because a contractor's engagement is date-bound rather than open-ended, the lifecycle engine tracks the contract end date from the same HR/contract feed and disables the account automatically on that date without waiting for a separate leaver event to be filed. If the engagement is extended, the extension goes through an explicit approval step (the engagement owner or manager approves it), and the same account is re-enabled and its end-date attribute updated, rather than a new account being created; reusing the same identity keeps its entire prior audit trail, including any earlier break-glass activity, attached to one continuous record instead of fragmenting it across two identities for the same person.

Handling disconnected hosts. SSSD's local cache is the primary mechanism that lets a network-partitioned host keep authenticating previously seen users for a bounded offline window, but that same cache is also the risk: a host that is disconnected when a leaver event fires may keep honoring a now-terminated user's credentials until it reconnects. Three things bound that risk: the cache's offline validity window is kept deliberately short rather than indefinite, so a prolonged disconnection expires the cached credential rather than trusting it forever; the configuration-management reconciliation loop re-pulls current authoritative state and forces a cache refresh on every scheduled run, so a host that missed a live push still catches up on its next run rather than staying silently stale; and for genuinely high-risk terminations (involuntary, security-related), an explicit fast-path revocation targets disconnected hosts specifically once they become reachable again, rather than relying purely on the standard cache-expiry timeline. The reconciliation system itself also tracks and alerts on hosts that have not successfully checked in within an expected window, so a silently stale host is visible to operations instead of assumed compliant.

Temporary access expiry. Every temporary grant, contractor break-glass elevation or an emergency admin session, is created with an explicit, system-enforced expiry from the start, never a "remember to remove this" convention. The PAM platform removes the grant automatically at the scheduled time, independent of any human follow-through. Because expiry enforcement is itself a push to the affected host, it inherits the same disconnected-host problem described above: if the host is offline at the scheduled expiry moment, the same reconciliation-on-reconnect mechanism must re-check and enforce the expiry once the host is reachable again, rather than assuming the original expiry action succeeded. Both the scheduled removal and the confirmation that it actually took effect on the host are logged as two separate events, closing the loop between intending to revoke access and verifying it was revoked.

Auditable trail of changes. Every layer, the HR feed ingestion, the lifecycle engine's decision, the native AD/LDAP directory change, the PAM platform's grant/use/expiry and session recordings, and each host's own local authentication log, ships to the same central, append-only log store, correlated by a consistent event or request identifier that threads from the original HR trigger through the directory change to the host-level effect. That correlation is what makes a single audit query able to answer "why did this account have this access, from which triggering event, approved by whom, and when was it revoked," rather than requiring a manual cross-reference across four separate systems' logs by guessing at timestamps. The audit store itself is kept append-only and access-controlled separately from the systems that generate its events, since an attacker who compromised the identity system itself would otherwise be able to erase their own tracks from a log store that same system controls.

Worked example

A contractor, csmith-ext, is onboarded on 2026-01-06 with an initial contract end date of 2026-03-31, sourced from the HR/contract system. The lifecycle engine creates the AD/LDAP account and assigns baseline contractor group membership. On 2026-02-10, csmith-ext requests break-glass elevated access during a production incident; the PAM platform grants a 4-hour window, 14:00 to 18:00 UTC, auto-expiring at 18:00 regardless of whether the session is still active, with the full session recorded and logged. On 2026-03-25, the engagement is extended to 2026-06-30; the engagement owner approves the extension, and the lifecycle engine updates the same account's contract-end-date attribute rather than creating a new account, so the February break-glass event remains attached to the same continuous identity. When 2026-03-31 arrives, the original end date, the scheduled auto-disable check reads the account's current end-date attribute, which by then already reflects 2026-06-30, so the auto-disable does not fire; had the extension approval not landed in time, the account would have auto-disabled on 2026-03-31 regardless, and a late-arriving approval would then go through the explicit re-enable path rather than silently reactivating the account on its own.

Trade-offs and pitfalls

Bounding the SSSD cache's offline validity window trades some availability (a disconnected host cannot authenticate a user whose cached credential has expired, even if that user is still legitimately employed) for security (a stale cache cannot indefinitely honor a terminated user's access), and that trade-off should be made explicitly and tuned, not left as an unexamined default in either direction. Relying on the PAM platform as the only path to emergency access is itself a single point of failure hiding inside the system meant to handle emergencies; a genuinely offline, sealed break-glass credential kept as a last resort, separate from the platform's own break-glass feature, is what actually protects against the platform itself being unavailable during an incident. Auto-disabling contractor accounts strictly by date is only as reliable as the HR/contract system's own data timeliness; a verbally agreed extension that has not yet been entered into that system will still result in the account auto-disabling on schedule, a legitimate but disruptive false positive, and the correct response is a fast, clearly documented re-enable-on-approval path, not disabling the auto-disable behavior itself, which would reintroduce exactly the forgotten-account risk it exists to close. Treating HR as the sole, always-timely trigger source is itself a risk, since HR systems occasionally lag or contain errors; an independent periodic reconciliation between the directory's account state and HR's current roster, flagging mismatches for review, catches HR-side data problems that a purely event-driven design would otherwise miss entirely. Finally, at 1,000 hosts, propagation is inherently eventually consistent, not atomic; the audit trail has to capture "revoked centrally at this time" and "confirmed enforced on this specific host at this later time" as two distinct events, not one, or the audit record will silently overstate how quickly a revocation actually took effect across the fleet.

Direct answer

Tell a story with a clear arc: the risk that motivated the change, how you tested it on a small set before production, how you watched it afterwards, and a rollback that was written and rehearsed before the change. The sample below uses a real setting so you can see the shape. Use your own incident in its place.

Situation

Our file servers allowed unsigned SMB (Server Message Block, the Windows file-sharing protocol) traffic. Microsoft documents that unsigned SMB packets can be intercepted and modified in transit, and that on member servers the policy "Microsoft network server: Digitally sign communications (always)" is disabled by default, while it is enabled on domain controllers. Those are the long-standing defaults. Microsoft's SMB documentation says Windows 11 24H2 and Windows Server 2025 changed them (Windows 11 24H2 requires signing both outbound and inbound; Windows Server 2025 requires outbound signing, and Microsoft's pages differ on whether it also requires inbound), so on newer builds test what is actually enforced instead of assuming unsigned traffic is allowed. This story assumes older server builds that still accepted unsigned inbound SMB. Security asked us to require signing on the file servers.

What I did

1. Wrote the change: set the policy to Enabled on a pilot OU (organizational unit, a folder of computers in Active Directory) containing two low-risk file servers.
2. Identified who could break: older clients and appliances that cannot sign SMB. Microsoft's SMB signing page describes a connection to a server that does not support signing failing (for example with STATUS_INVALID_SIGNATURE), so by the same logic unsigned-only devices would lose access once signing is required. I inventoried clients before the change by turning on SMB signing auditing on the pilot servers for two weeks and listing each client that could not sign.
   How the audit works, from Microsoft's SMB signing documentation: on a server build that has the setting (Microsoft documents these audit settings from Windows 11 24H2 and Windows Server 2025), run this in an elevated PowerShell session (or set the Group Policy Computer Configuration\Administrative Templates\Network\Lanman Server\Audit client does not support signing):

Set-SmbServerConfiguration -AuditClientDoesNotSupportSigning $true

The server then writes an event for each client that connects without signing support to Event Viewer under Applications and Services Logs, Microsoft, Windows, SMBServer, Audit (the documentation lists event IDs 3021 and 3022 for that log). Reading the log after two weeks gives a client list; in this story it named two devices (illustrative names: nas-old-01 and copier-scan-02). If a server build lacks the setting, the fallback is the inventory of client operating systems and appliances plus a test connection from each type.
3. Validated: after enabling on the pilot, I tested from one current Windows 11 laptop, one older Windows 10 laptop, one Linux client mounting the share, and the two flagged devices: from each I opened the share by name, saved a file into it and read it back, so 'shares still opened' meant those three actions succeeded. I also watched file-copy performance. Microsoft's documentation says signing has limited impact on 1 Gb Ethernet but can matter more on faster networks, so I measured large copies on the pilot rather than assuming.
4. Wrote the rollback in advance: set the policy back to Disabled in the pilot GPO (Group Policy Object). Microsoft documents no restart requirement for this policy, so rollback is a policy change, not a reboot, and I confirmed that on the pilot before relying on it.
5. Rolled out in waves, with a communications note and a service-desk script for "cannot open share".

Result and lesson

The pilot surfaced two legacy devices that could not sign. We upgraded one and isolated the other on a separate share with a documented exception and an end date. Looking back, I would have run the audit first on every server group instead of just the pilot, and would have scheduled the first production wave away from month-end close.

Pitfalls

Do not say "no issues." A hardening story without a surprise sounds untested. Keep claims about numbers to what you measured. And say what you would do differently.

Direct answer

A nice value (-20 to +19, default 0) is a weight for tasks in the normal time-sharing class: each step changes a task's share of CPU by a factor of about 1.25 when tasks compete. It has no effect when the CPU is idle and never makes a task run ahead of a real-time task. The real-time policies (SCHED_FIFO and SCHED_RR, with static priorities 1 to 99) are a different kind of rule: strict priority. Any runnable real-time task beats every normal task, however niced, and a SCHED_FIFO task keeps the CPU until it blocks, yields or is preempted by a higher real-time priority. On a shared host the main risk is that one runaway real-time thread starves everything else on its CPU, including the tools you would use to fix it. My recommendation: use nice and cgroup weights (a cgroup is a kernel mechanism that groups processes so CPU, memory and other resources can be limited or divided per group) for background work, reserve real-time for one small, bounded, latency-critical thread, and never give it priority 99.

What nice changes

• Range -20 (highest priority) to +19 (lowest); the Linux sched(7) page states each unit of difference gives a factor of 1.25.
• Two CPU-bound tasks at nice 0 and nice 5 split one CPU about 75.3 percent and 24.7 percent (the weight ratio is 1.25 to the 5th power, about 3.05, and 3.05 / 4.05 = 75.3 percent). At nice 0 against nice 19 the lower task gets about 1.4 percent (1.25 to the 19th is about 69).
• Unprivileged users can only raise a nice value (lower their priority) unless they hold CAP_SYS_NICE (a Linux capability: one slice of root's powers, here the right to change scheduling priorities) or have a resource-limit allowance (RLIMIT_NICE, a per-user cap on how low a nice value may be set; RLIMIT_RTPRIO is the matching cap for real-time priority).
• With the autogroup feature (a kernel option that automatically places each terminal session's tasks in their own scheduling group, common on desktops) tasks are grouped by session and nice applies mainly within a group.
• Nice does not give latency guarantees; it only skews a share. For hard CPU limits use cgroup cpu.max; for proportional shares use cpu.weight.

How the scheduling policies differ

The deadline class sits above the real-time class in Linux's scheduler ordering, and the kernel's admission control refuses a deadline task whose reservation would overcommit the CPUs.

Check the ranges on your system with chrt -m, and each task's class with ps -eo pid,cls,rtprio,ni,comm (class TS for normal, FF for FIFO, RR, B for batch, IDL, DLN).

Risks of moving a process to real-time on a shared host

1. Starvation of everything else on that CPU. A SCHED_FIFO busy loop never blocks, so normal tasks on that CPU (ssh, monitoring, the shell you would use to kill it) get no CPU. The kernel's safety net is RT throttling: by default sched_rt_runtime_us is 950000 out of a sched_rt_period_us of 1,000,000, so real-time tasks may use at most 95 percent of each second, leaving 50 ms in each second for the rest. That keeps the machine reachable, but a runaway task still takes almost all of a core, and anyone with the privilege can loosen the setting.
2. Priority inversion. If a high-priority real-time task waits on a lock held by a low-priority task that a medium-priority real-time task keeps preempting, the high one waits for the medium one. Use priority-inheritance mutexes (PTHREAD_PRIO_INHERIT) for locks shared across priorities.
3. Competing with the kernel's own helpers. Some kernel threads (migration, watchdogs) are real-time too. Setting a user task above them, such as priority 99, can delay work the kernel needs.
4. cgroup and container interaction. Placing real-time tasks in cgroups needs explicit real-time bandwidth with the group real-time scheduler (cpu.rt_runtime_us, a cgroup v1 file, in kernels built with the CONFIG_RT_GROUP_SCHED option, the build setting that enables real-time scheduling per cgroup); the kernel documentation says a group must be allocated real bandwidth before it can run real-time tasks. On cgroup v2 the documentation warns that with that option the cpu controller can only be enabled when all real-time processes are in the root cgroup, so on a v2 host real-time threads usually have to stay outside the container's cgroup tree or the host needs the option left off. Containers also need CAP_SYS_NICE.
5. Diagnosing starvation you did not cause. A normal task that seems frozen may be starved by a real-time task on its CPU (look at the class column above) or throttled by a cgroup quota (nr_throttled in the group's cpu.stat rising). Both look the same from the starved task's side: runnable, not running.

Worked example (executed)

In a default container (no CAP_SYS_NICE) the commands below were run as written. They show the policy ranges, a niced run, and the permission wall that protects a shared host.

chrt -m
nice -n 5 nice
chrt -f 50 true; echo rc=$?
cat /proc/sys/kernel/sched_rt_runtime_us /proc/sys/kernel/sched_rt_period_us

Output:

SCHED_OTHER min/max priority	: 0/0
SCHED_FIFO min/max priority	: 1/99
SCHED_RR min/max priority	: 1/99
SCHED_BATCH min/max priority	: 0/0
SCHED_IDLE min/max priority	: 0/0
SCHED_DEADLINE min/max priority	: 0/0
5
chrt: failed to set pid 0's policy: Operation not permitted
rc=1
950000
1000000

nice -n 5 nice prints 5: the child ran with the niced value. chrt -f 50 true fails because an unprivileged process cannot enter a real-time policy; on a real host you would hold CAP_SYS_NICE or have an RLIMIT_RTPRIO allowance. The last two lines are the default 95 percent throttle.

Recommendation, and what would change it

For a shared host: leave services at SCHED_OTHER, lower background work with nice or cpu.weight, and bound CPU hogs with cpu.max. If one thread truly needs bounded wake-up latency (for example an audio or control loop), give it SCHED_FIFO at a modest priority (not 99), keep the throttle at its default, pin it to a dedicated core with a cpuset (a cgroup setting that restricts a group to chosen CPUs) so starvation cannot reach other services, and put a watchdog on it (a separate monitor that notices the thread has stopped making progress and restarts or demotes it). Consider SCHED_DEADLINE if you can state runtime and period honestly. What would flip this: a dedicated machine with isolated cores and no co-tenants, where the throttle can be relaxed and real-time is the whole point.

Trade-offs and pitfalls

• Raising priority fixes latency only when the delay is queueing for CPU; if the thread waits on I/O or a lock, real-time makes no difference.
• Giving many tasks the same real-time priority invites the starvation it was meant to avoid.
• taskset, chrt and cgroup writes change the live system; record the original values so the change can be reversed.

Direct answer

Detecting a SYN-flood attack means recognizing a specific signature: a very high rate of SYN packets with no matching completed handshakes, often from addresses that are spoofed or do not respond to the SYN-ACK at all. The immediate priority is distinguishing this from a legitimate traffic spike and applying mitigations that block the attack without also blocking real users.

Structured elaboration

1. Confirm the signature by counting half-open connections directly: ss -tan state syn-recv | wc -l (or the older netstat -n | grep SYN_RECV | wc -l) counts sockets stuck in SYN-RECV on the target host. Note that the aggregate ss -s summary line on current iproute2 versions (TCP: N (estab X, closed X, orphaned X, timewait X)) does not break its count down by state, so it will not show a SYN-RECV figure; use the explicit state filter (ss -tan state syn-recv), not ss -s, to get a state-specific count. A number that is unusually high and growing, combined with a packet capture showing many distinct source IPs each sending a single SYN with no follow-up ACK after the server's SYN-ACK, is the core signature of a SYN flood.
2. Distinguish spoofed from real sources: legitimate clients complete the handshake (their OS replies with an ACK) even under load; a flood using spoofed source addresses will show SYN-ACKs going out with no corresponding ACK ever arriving, because the spoofed address either does not exist or never sent the original SYN. A quick way to sanity-check this is whether the apparent source IPs in the capture cluster in ways real client populations would not (sequential or clearly fabricated ranges).
3. Apply immediate, reversible mitigations: enable SYN cookies at the OS/kernel level (sysctl -w net.ipv4.tcp_syncookies=1, after confirming the current value with sysctl net.ipv4.tcp_syncookies), which lets the server avoid committing state for half-open connections until the handshake completes; apply rate-limiting on SYN packets per source at the firewall with iptables, for example iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 3 -j ACCEPT followed by a default DROP for the remainder, which caps the rate of new SYNs accepted per interval; and, only as a last resort for the worst offenders, blackhole-filter specific source ranges that are unambiguously not legitimate customers.
4. Preserve legitimate traffic: before broadly rate-limiting, sample the traffic to estimate what fraction of current SYN volume is legitimate versus attack, so the rate limit is set above real demand rather than accidentally throttling real users during the incident.

Worked example

ss -tan state syn-recv | wc -l reports 40,000 sockets in SYN-RECV, versus a normal baseline of under 100 (the aggregate ss -s line would not have surfaced this number at all, since it does not report per-state counts). A capture shows SYN packets arriving at roughly 8,000 per second from source addresses that never appear again after the initial SYN (no ACK, no further traffic), consistent with spoofed sources. Enabling SYN cookies immediately stops the server from exhausting its connection-table memory on these half-open entries, since it no longer allocates state until the three-way handshake actually completes; concurrently, an iptables SYN rate limit (iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 3 -j ACCEPT, with a trailing DROP for anything over the limit) at the upstream firewall reduces the volume reaching the server without needing to identify or block specific IPs, which would be futile against spoofed sources anyway.

Trade-offs & pitfalls

SYN cookies have a real cost: they disable some TCP options (like window scaling, in implementations that cannot preserve them without the TCP timestamp option) for the connections they protect, which can silently degrade throughput for high-bandwidth-delay-product connections during the mitigation window; know this trade-off before enabling them by default rather than only during an active attack. Blackhole-filtering by source IP is close to useless against spoofed traffic (you would be blocking addresses that were never real senders) and should be reserved for cases where you have confirmed the sources are not spoofed. Also do not rely on the ss -s summary line to detect this: on current Linux tooling it reports only aggregate estab/closed/orphaned/timewait counts, not a SYN-RECV breakdown, so use the explicit state filter or netstat instead.

Direct answer

Move the data with Storage Migration Service (SMS), the Windows Server tool that inventories the old servers, copies files together with share settings and security, and can hand each old server's name and address to the new one at cutover. Target a clustered File Server for general use role (not a Scale-Out File Server), turn on SMB encryption on the shares, seed the data while the old servers stay live, run a short final transfer inside the cutover window, then prove correctness by comparing stored security descriptors and prove resilience by moving the role between nodes and failing a node on purpose. Robocopy is the fallback when SMS cannot do the job, for example NTFS to ReFS.

Choose the target before copying anything

Two kinds of clustered file server exist, and Microsoft's comparison table decides which one fits user shares:

So: File Server for general use on top of a failover cluster. If the cluster uses Storage Spaces Direct (software-defined storage that pools the local disks of the nodes into shared volumes), the shares still live on a general-use role, not on a scale-out share. Build the role in Failover Cluster Manager with its own client access point (the network name and IP address that clients type to reach the clustered file server, which moves with the role from node to node) before migrating.

File system choice has one trap. Microsoft's comparison table recommends the Resilient File System (ReFS) with Storage Spaces Direct, but SMS supports NTFS (the New Technology File System) to NTFS and ReFS to ReFS only, never NTFS to ReFS. If the old shares are NTFS and you want SMS cutover, create NTFS volumes for these shares. If you insist on ReFS, plan the copy with robocopy.

"Encryption on" means two separate things:

• In transit: Server Message Block (SMB) encryption, enabled per share with Set-SmbShare -Name Finance -EncryptData $true, or for the whole server with Set-SmbServerConfiguration -EncryptData $true. It requires SMB 3.0 or later on both sides. SMB 3.1.1 negotiates the AES-128-GCM cipher (a modern authenticated encryption mode) by default, and AES-256 suites exist from Windows Server 2022. A client that only speaks SMB 2.x or 1.0 gets Access denied and the server logs event ID 1003 in Microsoft-Windows-SmbServer/Operational, so inventory old clients first. SMS carries each share's Encrypt Data flag across, but verify it on the new shares.
• At rest: SMB encryption does not cover it; Microsoft's SMB documentation points to BitLocker for data at rest. Decide that separately for the new volumes.

Copy so permissions and ownership survive

With SMS (three stages: inventory, transfer, cutover): the transfer moves files, shares and security configuration. It migrates every share flag and its security descriptor (the access control list, ACL, and related data), including access-based enumeration (users see only the files and folders they have permission to open), continuous availability (a clustered-share setting that lets open file handles survive a node failover, at the cost of writing through to disk) and encryption. A cluster can be the destination: SMS migrates a standalone source into a file server resource on the cluster (it migrates the file server resource, not the cluster itself). Behaviours to plan around:

• The destination must hold no production data. The first transfer backs up what is there; later transfers mirror the source, overwriting and deleting anything not on the source.
• Files that applications hold exclusively locked are retried (three tries, sixty seconds apart by default) and skipped files are picked up by re-running the transfer.
• Previous Versions (shadow copies) are not migrated, domain controllers are not supported as sources for cutover, and consolidating several servers onto one server with colliding names is not supported. The supported way to consolidate is one file server resource per source on the cluster.

With robocopy, the flags that keep security (all verified against the command reference):

robocopy \\old-fs1\Finance E:\Shares\Finance /E /COPYALL /DCOPY:DAT /ZB /R:2 /W:5 /MT:16 /NP /LOG+:C:\Logs\finance-seed.log
robocopy \\old-fs1\Finance E:\Shares\Finance /MIR /COPYALL /DCOPY:DAT /SECFIX /ZB /R:2 /W:5 /MT:16 /NP /LOG+:C:\Logs\finance-final.log

Reading the two commands in plain words: the first is the seed copy. It copies the whole tree including empty folders (/E) with data, attributes, timestamps, ACLs, owner and audit settings (/COPYALL), keeps folder timestamps (/DCOPY:DAT), retries a locked file twice, five seconds apart (/R:2 /W:5), uses 16 copy threads (/MT:16), hides per-file percentages (/NP) and appends to a log (/LOG+). It never deletes anything on the destination. The second is the final pass inside the cutover window. /MIR makes the destination an exact mirror, so it also removes files that no longer exist on the source, and /SECFIX re-applies security to files that were skipped as unchanged. Run the seed command first, repeat it while users work, and run the mirror command only once users are stopped.

/COPYALL equals /COPY:DATSOU (data, attributes, timestamps, ACLs, owner, auditing). /DCOPY:DAT adds directory timestamps to the default directory copy. /ZB is restartable mode that falls back to backup mode when access is denied. /MT:16 uses 16 threads (default 8, range 1 to 128). /MIR is /E plus /PURGE, so it deletes destination files missing from the source; point it the right way round. /SECFIX repairs security on files that were skipped as unchanged. Exit codes of 8 or higher mean at least one failure, so a wrapper script must test for that, not for non-zero.

Choosing between the tools. Each row is a different job, so the right one depends on whether you are moving a server's shares, copying files, keeping servers in sync or protecting storage:

Cut over with minimal disruption

Worked timing, with stated assumptions: 6 TB in total, a 1 Gbit/s path that sustains 70% of line rate, and 2% of the data changing per day.

Step by step:

1. Effective rate: 1 Gbit/s x 70% = 0.70 x 10^9 = 700,000,000 bits per second.
2. Data in bits: 6 TB = 6 x 10^12 bytes, and each byte is 8 bits, so 6 x 10^12 x 8 = 48 x 10^12 = 4.8 x 10^13 bits.
3. Seed time: 4.8 x 10^13 / 7 x 10^8 = 68,571 seconds, which is 68,571 / 3,600 = 19.05 hours.
4. Daily change: 2% of 6,000 GB = 120 GB = 1.2 x 10^11 bytes = 9.6 x 10^11 bits.
5. Time for one nightly pass: 9.6 x 10^11 / 7 x 10^8 = 1,371 seconds = 22.9 minutes.

So: run the seed over a weekend while users keep working (19 hours, with spare), repeat the transfer nightly (each pass moves about 23 minutes of changes), and make the cutover window about 45 minutes as a floor: final transfer (23 minutes), cutover and checks. SMS restarts the servers several times during cutover and waits on Active Directory and DNS replication, so rehearse the cutover and extend the window if the restarts take longer than the remaining 22 minutes.

1. Lower risk first: re-run transfers until a pass is short, and fix the skipped-file list.
2. In the window, stop users (set the old shares read-only or disconnect sessions), run the final transfer, run the checks below.
3. Cut over. In SMS the destination takes the source server's name and IP address, and the old server is renamed and given a new IP address so users and applications can no longer reach it, while its contents are left untouched, so rollback is possible. SMS does not update certificates that name the old server, so reissue them.
4. If users reach data through a Distributed File System (DFS) namespace path, move them with namespace targets instead of renaming servers. A DFS namespace is a single virtual folder path such as \\corp.contoso.com\dfs\Finance; each folder in it points to one or more namespace targets (the real shares). When a client opens the path, a namespace server hands back a referral, an ordered list of targets to try. Add the new target and take the old one offline; clients cache referrals, so shorten the referral cache time beforehand:

New-DfsnFolderTarget -Path '\\corp.contoso.com\dfs\Finance' -TargetPath '\\fs-finance\Finance'
Set-DfsnFolderTarget -Path '\\corp.contoso.com\dfs\Finance' -TargetPath '\\old-fs1\Finance' -State Offline

Clients receive no referrals for an offline folder target. For availability, Microsoft's guidance is to put a stand-alone namespace on a failover cluster, or to host a domain-based namespace on multiple namespace servers, so decide where the namespace lives before cutover and do not assume it moves with the file cluster role.

Prove it is correct

1. Data: a list-only robocopy pass (/L /E) after the final transfer should report nothing to copy.
2. Security: build a map of path to SDDL (Security Descriptor Definition Language, the text form of an ACL) on each side and compare. To read an SDDL string such as O:BAG:DUD:PAI(A;OICI;0x1301bf;;;DU): O:BA is the owner (Builtin Administrators), G:DU is the primary group (Domain Users), D: starts the DACL (the list of allow and deny entries), P means inheritance from the parent is blocked and AI means the list was set up by automatic inheritance. Each parenthesised entry reads type;flags;rights;;;who: A is Allow, OICI means object inherit plus container inherit (it applies to files and subfolders), the rights are a number, and DU is the trustee, Domain Users. In an entry, ID in the flags field marks an entry inherited from a parent folder, FA is full control and SY is the Local System account. For the rights number, 0x1301bf is the NTFS Modify permission (read, write, execute and delete) and 0x1200a9 is Read and execute, so the \Finance\Managers mismatch below means Domain Users has Modify on the source and only Read on the target. The function below is the comparison; the sample data shows each kind of mismatch it reports. I ran it under PowerShell 7; the Get-Acl calls that fill the maps need Windows and were not run.

function Compare-SddlMap {
    param(
        [Parameter(Mandatory)][hashtable]$Source,
        [Parameter(Mandatory)][hashtable]$Target
    )
    foreach ($path in $Source.Keys | Sort-Object) {
        if (-not $Target.ContainsKey($path)) {
            [pscustomobject]@{ Path = $path; Problem = 'missing on target' }
        } elseif ($Source[$path] -ne $Target[$path]) {
            [pscustomobject]@{ Path = $path; Problem = 'ACL differs' }
        }
    }
    foreach ($path in $Target.Keys | Sort-Object) {
        if (-not $Source.ContainsKey($path)) {
            [pscustomobject]@{ Path = $path; Problem = 'extra on target' }
        }
    }
}

# In production each map is built with:
#   Get-ChildItem -Recurse | ForEach-Object { $map[$_.FullName.Substring($root.Length)] = (Get-Acl -LiteralPath $_.FullName).Sddl }
$source = @{
    '\Finance'                = 'O:BAG:DUD:PAI(A;;FA;;;SY)'
    '\Finance\Managers'       = 'O:BAG:DUD:PAI(A;OICI;0x1301bf;;;DU)'
    '\Finance\Working\a.xlsx' = 'O:BAG:DUD:AI(A;ID;0x1301bf;;;DU)'
}
$target = @{
    '\Finance'                = 'O:BAG:DUD:PAI(A;;FA;;;SY)'
    '\Finance\Managers'       = 'O:BAG:DUD:PAI(A;OICI;0x1200a9;;;DU)'
    '\Finance\Working\b.xlsx' = 'O:BAG:DUD:AI(A;ID;0x1301bf;;;DU)'
}
Compare-SddlMap -Source $source -Target $target | Format-Table -AutoSize | Out-String -Width 120

Path                    Problem
----                    -------
\Finance\Managers       ACL differs
\Finance\Working\a.xlsx missing on target
\Finance\Working\b.xlsx extra on target

3. Shares: compare Get-SmbShare | Select-Object Name, EncryptData, FolderEnumerationMode, ContinuouslyAvailable between the old and new server. EncryptData is the per-share SMB encryption flag and FolderEnumerationMode shows whether access-based enumeration is on (AccessBased) or off (Unrestricted).
4. People: sign in as a member of each access group and confirm open, write and denied cases on a sample folder.

Prove it is resilient

• Run Test-Cluster -Node 'node-a','node-b' and read the report before moving data. Check quorum: with one static vote per node and no witness, two nodes have 2 votes and need 2 for a majority, so losing either node loses quorum. Adding a witness (a small disk or file share that casts a tie-breaking vote, so the cluster never splits evenly) gives 3 votes and a majority of 2, so one node can fail.
• Planned failover: Move-ClusterGroup -Name 'FS-FINANCE' -Node 'node-b', then back to node-a, while a client copies a large file. Record how long shares are unreachable and whether the copy resumes. Microsoft describes this cmdlet as a way to simulate failover and to prepare for node maintenance.
• Unplanned failover: power off the node that owns the role, in a test window, and time how long until \\fs-finance\Finance answers again. Measure it; do not quote a figure from documentation.
• Performance: compare a many-small-files copy and a large-file copy against the old servers. Continuously available shares must write through to disk, so large copies can be slower than on a plain share; decide whether you need that setting from the measurements.
• Redundancy of the paths: confirm the clients use more than one network path to the cluster (SMB Multichannel, the SMB 3 feature that spreads one client's traffic across several network connections for speed and for surviving a failed one) and that the storage layer reports healthy before you decommission anything.

Pitfalls

• Pre-staged files on the destination are overwritten or deleted by SMS mirroring.
• Local users and groups used in ACLs must be migrated too (SMS has a transfer setting for local users and groups; migrated accounts arrive disabled with a random password) or replaced by domain groups, otherwise their security identifiers (SIDs) will not resolve on the new server.
• Never run /MIR with source and destination reversed.
• Keep the old servers, untouched, until a full business cycle has passed.
• Folder Redirection (a Group Policy setting that stores users' Documents or Desktop folders on a share) points at a share path, so confirm each redirected path still resolves after cutover.

Direct answer

Encryption in transit protects data while it is moving between two points, for example your laptop and a website. Encryption at rest protects data while it is sitting in storage, for example on a server's hard drive. Both work the same basic way: the data is scrambled using a digital key, and only someone with the matching key can unscramble it back to something readable. HTTPS (the lock icon in a browser) is the everyday example of encryption in transit; a company laptop or database with disk encryption turned on is the everyday example of encryption at rest.

Structured elaboration

When I explain this to a stakeholder who is not technical, I make three deliberate choices:

1. Pick an analogy that survives the obvious follow-up question. "Scrambling data" invites "how do you unscramble it back?" so I go straight to a locked box with a key: the box (the data) can sit on a shelf (at rest) or travel in a delivery truck (in transit), and either way, only someone holding the matching key can open it. This analogy already answers the natural next question ("who has the key?") instead of dodging it.
2. Decide what to omit, not just simplify. I leave out algorithm names, protocol versions, and how the encryption keys themselves are generated and stored. Those details do not change the stakeholder's decision (do we need this, is it enough for compliance). What I keep is the one thing that matters to them: even if someone steals the disk or intercepts the network traffic, they get scrambled data they cannot use without the key.
3. Check understanding without quizzing them. Instead of asking "does that make sense?" (which invites a reflexive yes), I ask them to restate it in their own words, or pose a concrete scenario: "if someone stole this laptop from a parked car, what would they actually get?" Their answer tells me whether the concept landed.

Worked example

Here is close to what I would actually say:

"Think of our data like paperwork in a locked filing cabinet. 'Encryption at rest' means the paperwork sitting in that cabinet, meaning on our servers or backups, is written in a code that only a specific key can decode. If someone breaks into the building and steals the cabinet, they walk away with pages of gibberish. 'Encryption in transit' is the same idea for paperwork that is being carried from one office to another, meaning data moving between your browser and our servers. The little padlock icon you see next to a website address means that trip is protected the same way: even if someone intercepts the envelope mid-delivery, they cannot read what is inside. In both cases the 'key' is just a digital code that locks and unlocks the data. We keep that key separate from the data itself and tightly restrict who can use it, the same way you would not tape the safe combination to the safe."

If they ask a follow-up like "so is our data safe no matter what," that is the moment to add the caveat below rather than let the analogy imply more than it should.

Trade-offs and pitfalls

• The locked-box analogy breaks down around key management: a real safe has one physical key, but a digital key can be copied, and who is allowed to use it (and how that is audited) matters as much as the encryption itself. If the stakeholder is making a security or compliance decision, that caveat has to come back in, even though it complicates the clean story.
• Oversimplifying to "it's encrypted so it's safe" can create false assurance. Encryption at rest does not protect data while an application has it decrypted in memory for processing, and encryption in transit does not protect against someone who is logged in as an authorized user misusing their access. Naming that boundary once, briefly, is worth the extra sentence.
• Dropping all technical vocabulary can cost credibility with a stakeholder who has picked up some of it (for example, someone who has heard the term TLS from a vendor). It is fine to mention the term once, defined in one plain clause ("TLS, the technology behind that browser padlock"), so the explanation still connects to language they may encounter elsewhere.

Direct answer
A SYN flood works by sending a stream of SYN segments (often with spoofed source addresses) so the server allocates per-connection state for each one and replies with a SYN-ACK, but the final ACK never arrives, exhausting the server's backlog of half-open connections. SYN cookies let the server defer allocating any state at all until the final ACK actually shows up, so a flood of SYNs that never complete costs the server almost nothing.

Structured elaboration
Without SYN cookies, a normal server implementation stores an entry in its SYN backlog queue for every SYN it receives, holding onto that entry until the handshake completes or times out. A large flood of SYNs fills the backlog with half-open connections, so genuine clients' SYNs get silently dropped once the queue is full, that's the denial of service.

With SYN cookies enabled, once the backlog nears capacity, the server stops storing per-connection state up front. Instead, when it receives a SYN, it encodes the essential information it would normally have stored (effectively a hash of the source/destination IP, ports, and a secret, folded into the initial sequence number it sends back in the SYN-ACK) directly into its own sequence number field, and discards the SYN backlog entry immediately. If the final ACK ever arrives, the server checks that its acknowledgment number is the cookie value plus one, reconstructs the connection state on the spot, and only THEN commits real resources, at the exact moment a real, completing client shows up. If the SYN was part of the flood and no valid ACK ever comes, the server never allocated anything for it in the first place.

Worked example
From the legitimate client's point of view, the handshake looks completely normal: SYN, SYN-ACK, ACK, exactly as usual, it never knows cookies were involved. The difference is entirely server-side bookkeeping. The cost of the cookie scheme is that the server must give up storing some TCP options across the handshake (since there's no room left in a single 32-bit sequence number to also encode arbitrary options like the negotiated MSS (Maximum Segment Size) or window scale precisely), which is why some cookie implementations round MSS to one of a small handful of common values instead of preserving whatever exact value the client requested.

Trade-offs & pitfalls
SYN cookies alone do not stop a volumetric flood from consuming link bandwidth or CPU cycles processing the flood of SYNs in the first place, they only protect the connection-state table from being exhausted. A senior answer distinguishes "the handshake structure survives an attack that tries to exhaust connection state" (what SYN cookies solve) from "the network survives an attack that tries to exhaust bandwidth or CPU" (a separate problem requiring rate limiting, upstream filtering, or scrubbing, outside the scope of the handshake mechanism itself).

Direct answer

A Business Impact Analysis (BIA) is the exercise that translates "this business function is down" into a number the organization can act on: how much it costs per hour, in revenue, penalties, regulatory exposure, and customer harm, and how long the business can tolerate the outage before that harm becomes unacceptable. A continuity or risk manager typically commissions it, but the answers come from the business-function owners themselves. Its output (criticality tiers, tolerance windows, and recovery targets) becomes the backbone of everything downstream: which functions get recovered first, how much continuity budget each one justifies, and what the crisis team and exercise program actually rehearse.

Structured elaboration

What it measures. For each business function (not each application or server) the BIA asks: what breaks if this stops, who is affected, and how does the harm grow over time. That last part matters: the harm from a one-hour outage of payroll processing is trivial, the harm from a five-day outage is a legal problem, and the BIA is what turns that curve into a decision.

Who is involved.

Core outputs, defined at first use.

• Maximum Tolerable Period of Disruption (MTPD), sometimes called Maximum Allowable Outage (MAO): the longest the function can be down before the damage is unrecoverable for the business, not a technical estimate.
• Recovery Time Objective (RTO), from the business side: the target time by which the function must be working again, set below the MTPD with margin for the recovery process itself.
• Recovery Point Objective (RPO), from the business side: how much data loss (measured in time, e.g. "up to the last hour of transactions") the function can absorb.
• A criticality tier per function, usually 3 to 5 levels, used to rank recovery priority when resources are limited.

The BIA states these as business requirements. How a technical team meets a given RTO or RPO (through backup cadence, replication, or standby capacity) is a separate, downstream engineering decision, not something the BIA itself prescribes.

How the output is used downstream. The tiered list drives recovery sequencing (who gets resources first when multiple functions are affected at once), justifies continuity and resilience budget to leadership, defines the scope of the exercise program (a Tier 1 function is drilled more often than a Tier 4 one), and often becomes the evidentiary artifact regulators or auditors ask for first.

Worked example

A mid-size payments company runs its BIA on the "supplier payment processing" function. The finance lead reports the function processes about $2M/day in scheduled supplier payments, and that missed payments trigger a contractual late-payment interest charge of 1.5% per day on the unpaid balance. If a full day's batch is delayed, the direct cost is:

That number alone would suggest a loose tolerance, but the finance lead also flags that two of the company's largest suppliers have a contractual right to suspend shipment after 3 consecutive missed payment days, which would stop production, a much larger and harder-to-quantify impact. Leadership sets the MTPD at 2 business days on the strength of that qualitative flag, not the dollar figure alone, and the function is tiered as Tier 1 with an RTO of 4 hours. That tiering, RTO, and the underlying reasoning are the artifact that gets handed to the technical owners; how they achieve a 4-hour RTO is out of scope for the BIA itself.

Trade-offs and pitfalls

A BIA is frequently confused with a risk assessment: a risk assessment asks what could go wrong and how likely it is, while a BIA assumes the disruption already happened and asks how much it costs. Conflating the two produces a document that's neither useful for prioritization nor for threat planning. A second common failure is treating the BIA as a one-time deliverable rather than a living input that gets revisited when the business changes (new product lines, new regulatory exposure, an M&A integration); a BIA that's three years old is usually wrong. Finally, because the audience for this answer includes engineering roles, it's worth naming the trap directly: the BIA states what the business needs (a tolerance window and a target), not how to build it. An answer that jumps straight to backup and replication design has answered a different, narrower question than the one being asked here.

Direct answer

Build the policy as three cooperating Pluggable Authentication Module (PAM) stacks, quality enforcement, account lockout, and password history, defined once in the distribution's shared password stack rather than duplicated per service, so SSH inherits the same policy as console and su logins automatically. Use pam_faillock, the current, actively maintained lockout module, rather than pam_tally2, which it has replaced on modern Red Hat-family distributions. Test with a disposable account and a second, already-open administrator session before ever applying the policy broadly.

Structured elaboration

What each module does.

• pam_pwquality enforces quality rules on a new password at set time: minimum length, character-class variety, repeated-character limits, and similarity to the previous password, configured in /etc/security/pwquality.conf.
• pam_unix is the traditional Unix module: it verifies the password hash at login, and, in the password-change stack, computes and stores the new hash. Its remember=N option also enforces password history by keeping the last N hashes in /etc/security/opasswd, so a user cannot immediately reuse a recent password.
• pam_faillock tracks consecutive failed login attempts and enforces a temporary lockout after a configured threshold. pam_tally2 did the same job on older systems but is deprecated in favor of pam_faillock; naming the current module matters here since the question names both.

Configuration, in the shared stack file most Red Hat-family systems already use (password-auth, included by service-specific files like sshd, login, and su rather than each defining its own copy):

# /etc/security/pwquality.conf
minlen = 14
minclass = 3
maxrepeat = 3
maxsequence = 3
dcredit = -1
ucredit = -1
lcredit = -1
ocredit = -1
difok = 8
retry = 3
enforce_for_root

# /etc/pam.d/password-auth (relevant lines)
auth        required                                     pam_faillock.so preauth silent deny=5 unlock_time=900
auth        [success=1 default=ignore]                   pam_unix.so try_first_pass
auth        [default=die]                                pam_faillock.so authfail deny=5 unlock_time=900
auth        sufficient                                   pam_faillock.so authsucc deny=5 unlock_time=900
auth        required                                     pam_deny.so

account     required                                     pam_faillock.so
account     required                                     pam_unix.so

password    requisite                                    pam_pwquality.so retry=3
password    sufficient                                   pam_unix.so sha512 shadow use_authtok remember=10
password    required                                     pam_deny.so

session     required                                     pam_limits.so
session     required                                     pam_unix.so

# /etc/pam.d/sshd (relevant lines)
auth       substack     password-auth
account    required     pam_nologin.so
account    include      password-auth
password   include      password-auth
session    required     pam_selinux.so close
session    required     pam_loginuid.so
session    include      password-auth
session    required     pam_selinux.so open env_params

Password expiration (the one setting neither pam_pwquality nor pam_faillock enforces). Complexity, length, and history govern what a password can be; expiration governs how long any password, however strong, stays valid before it must change, and that comes from the aging fields in /etc/shadow that pam_unix's account-management phase checks on every login, not from either module configured above. Set the fleet-wide default in /etc/login.defs so every newly created account inherits it, then apply the same values to an existing account with chage, since login.defs only sets the default for accounts created after the change:

# /etc/login.defs (relevant lines)
PASS_MAX_DAYS   90
PASS_MIN_DAYS   1
PASS_WARN_AGE   14

# Apply the same policy directly to an existing account's shadow entry
chage -M 90 -m 1 -W 14 alice

PASS_MAX_DAYS 90 forces a change at least quarterly. PASS_MIN_DAYS 1 closes a gap the history setting alone leaves open: without a minimum age, a user could change their password twice back to back to cycle straight back to the one before, defeating remember=10 from the other direction. PASS_WARN_AGE 14 gives two weeks' notice before expiration forces a change, instead of a login that suddenly demands a new password with no warning.

Recommended settings and why each one exists (complexity, length, and history): minlen=14 and minclass=3 (requiring variety across at least three of the four character classes) drive real entropy rather than complexity theater. maxrepeat=3 blocks strings like aaaa1111. difok=8 requires enough of the new password to differ from the old one that a trivial increment does not pass. deny=5 unlock_time=900 locks an account for 15 minutes after five failed attempts, balancing brute-force resistance against a legitimate user's ability to eventually get back in without a help-desk call. remember=10 blocks the "change it, then change it right back" workaround by keeping the last ten hashes.

Integration with SSH. sshd's own PAM file does not define password or account logic from scratch. It includes the shared password-auth stack (auth substack password-auth, account include password-auth, password include password-auth). A policy change made once in password-auth therefore applies automatically to SSH logins, console logins, and su, without needing to be copied into every service file separately, which is also what keeps the policy from silently drifting out of sync between services over time.

Testing without locking out administrators. By default, pam_faillock's lockout does not apply to the root account unless the even_deny_root option is explicitly added, so the worst-case scenario is already guarded against, but this should be verified rather than assumed. Before rolling out further: keep a second, already-authenticated administrative session open on a separate connection, so a mistake in the new policy does not strand you without a way back in. Use faillock --user <test-account> to inspect a specific account's current failure count without needing a real failed login, deliberately fail logins against a disposable test account (never an admin account) to confirm the lockout actually triggers and the unlock_time countdown behaves as configured, then clear it with faillock --user <test-account> --reset. Roll out to one non-critical host first and confirm both lockout and recovery work end to end before applying it fleet-wide.

Worked example

The configuration above was checked with a complete, self-contained static syntax and ordering linter (copy-pasteable and runnable as shown, given the three files above saved alongside it as password-auth, sshd, and pwquality.conf). This does not exercise a live PAM authentication, there is no real login attempt against a real account here, it parses the exact lines shown above and verifies (a) every line is syntactically legal and (b) the pam_faillock/pam_unix/pam_pwquality lines are stacked in the order that actually enforces lockout-then-quality-then-history, which is the part hand-written PAM configs most often get wrong:

VALID_TYPES = {"auth", "account", "password", "session"}
SIMPLE_CONTROLS = {"required", "requisite", "sufficient", "optional", "include", "substack"}


def parse_line(line):
    line = line.split("#", 1)[0].strip()
    if not line:
        return None
    tokens = line.split()
    if len(tokens) < 2:
        raise ValueError(f"line has too few fields: {line!r}")
    ptype = tokens[0]
    if ptype not in VALID_TYPES:
        raise ValueError(f"unknown PAM type {ptype!r} in line: {line!r}")
    rest = tokens[1:]
    if rest[0].startswith("["):
        # bracketed control, e.g. [success=1 default=ignore]
        joined = " ".join(rest)
        end = joined.find("]")
        if end == -1:
            raise ValueError(f"unclosed bracketed control in line: {line!r}")
        control = joined[: end + 1]
        remainder = joined[end + 1 :].split()
    else:
        control = rest[0]
        if control not in SIMPLE_CONTROLS:
            raise ValueError(f"unknown control {control!r} in line: {line!r}")
        remainder = rest[1:]
    if control in ("include", "substack"):
        if len(remainder) != 1:
            raise ValueError(f"{control} must name exactly one stack file: {line!r}")
        return {"type": ptype, "control": control, "target": remainder[0], "args": []}
    if not remainder:
        raise ValueError(f"line has a control but no module path: {line!r}")
    module = remainder[0]
    args = remainder[1:]
    return {"type": ptype, "control": control, "module": module, "args": args}


def parse_stack(text):
    parsed = []
    for raw in text.splitlines():
        entry = parse_line(raw)
        if entry:
            parsed.append(entry)
    return parsed


def check_password_auth_ordering(parsed):
    auth_lines = [p for p in parsed if p["type"] == "auth"]
    modules_in_order = [p.get("module") for p in auth_lines]

    def first_index(pred):
        for i, p in enumerate(auth_lines):
            if pred(p):
                return i
        return None

    preauth_i = first_index(lambda p: p.get("module") == "pam_faillock.so" and "preauth" in p["args"])
    unix_i = first_index(lambda p: p.get("module") == "pam_unix.so")
    authfail_i = first_index(lambda p: p.get("module") == "pam_faillock.so" and "authfail" in p["args"])
    authsucc_i = first_index(lambda p: p.get("module") == "pam_faillock.so" and "authsucc" in p["args"])
    deny_i = first_index(lambda p: p.get("module") == "pam_deny.so")

    assert None not in (preauth_i, unix_i, authfail_i, authsucc_i, deny_i), (
        f"missing an expected auth-stack line: {modules_in_order}"
    )
    assert preauth_i < unix_i < authfail_i < authsucc_i < deny_i, (
        "faillock/pam_unix stack is out of order: preauth must run before pam_unix, "
        "authfail/authsucc must run after it, pam_deny must be last. Got order: "
        f"{modules_in_order}"
    )

    pwq_lines = [p for p in parsed if p["type"] == "password"]
    pwq = next(p for p in pwq_lines if p.get("module") == "pam_pwquality.so")
    assert pwq["control"] == "requisite", (
        "pam_pwquality.so must use control 'requisite' (immediate stop on a weak "
        "password), not 'required' (which would still run pam_unix.so and hash a "
        "password it just rejected)"
    )

    unix_pw = next(p for p in pwq_lines if p.get("module") == "pam_unix.so")
    assert "remember=10" in unix_pw["args"], "password-history line must set remember=N"


def main():
    with open("password-auth") as f:
        parsed = parse_stack(f.read())
    with open("sshd") as f:
        sshd_parsed = parse_stack(f.read())
    with open("pwquality.conf") as f:
        pwq_conf_lines = [l.split("#", 1)[0].strip() for l in f if l.split("#", 1)[0].strip()]

    print(f"password-auth: {len(parsed)} lines parsed, all syntactically valid PAM lines")
    print(f"sshd: {len(sshd_parsed)} lines parsed, all syntactically valid PAM lines")
    print(f"pwquality.conf: {len(pwq_conf_lines)} directives parsed")

    check_password_auth_ordering(parsed)
    print("ORDERING CHECK: preauth -> pam_unix -> authfail -> authsucc -> pam_deny : PASS")
    print("PWQUALITY CONTROL CHECK: pam_pwquality.so uses 'requisite' : PASS")
    print("PASSWORD HISTORY CHECK: pam_unix.so sets remember=10 : PASS")

    sshd_targets = {p["target"] for p in sshd_parsed if p["control"] in ("include", "substack")}
    assert "password-auth" in sshd_targets, "sshd stack must reference password-auth"
    print("SSHD INTEGRATION CHECK: sshd pam stack includes/substacks password-auth : PASS")

    print("\nALL_CHECKS=PASS")


if __name__ == "__main__":
    main()

Actual output from running this against the three files shown above:

password-auth: 12 lines parsed, all syntactically valid PAM lines
sshd: 8 lines parsed, all syntactically valid PAM lines
pwquality.conf: 11 directives parsed
ORDERING CHECK: preauth -> pam_unix -> authfail -> authsucc -> pam_deny : PASS
PWQUALITY CONTROL CHECK: pam_pwquality.so uses 'requisite' : PASS
PASSWORD HISTORY CHECK: pam_unix.so sets remember=10 : PASS
SSHD INTEGRATION CHECK: sshd pam stack includes/substacks password-auth : PASS

ALL_CHECKS=PASS

Two of these checks catch real, common hand-written mistakes: the ordering check would fail if pam_unix.so were placed before the preauth pam_faillock.so line (which breaks lockout counting), and the control check would fail if pam_pwquality.so used required instead of requisite (which would let a rejected weak password still reach pam_unix.so and get hashed anyway).

Trade-offs and pitfalls

• Using required instead of requisite for pam_pwquality.so is a subtle, common mistake. required still runs the rest of the stack even after a failure is recorded, so a rejected weak password could still reach pam_unix.so in some configurations. requisite stops immediately, which is what the policy actually needs.
• A lockout policy tested only against admin accounts risks looking safe when it is not, since even_deny_root defaults to off. Test against a disposable account specifically so the test reflects what an ordinary user will experience.
• Configuring the policy directly in each service's own PAM file, instead of the shared stack, invites drift, one service gets updated and another is quietly left on the old policy. The shared-stack approach exists specifically to prevent that.
• unlock_time set too aggressively (very long) turns a security control into a denial-of-service surface, an attacker can lock a real user out for the full window with a handful of deliberately failed attempts. 900 seconds balances brute-force resistance against that risk; a much longer value should be a deliberate choice, not a default left unexamined.

Direct answer

I treat the first hour of a server's life as the cheapest time to harden it: it carries no traffic, no one depends on it, and a mistake costs a rebuild rather than an outage. In order, I patch it, remove or disable everything the service does not need (accounts, packages, listening ports), lock down remote access (key-only SSH, scoped sudo, default-deny firewall), set safe kernel and file permission defaults, turn on logging that leaves the host, and then prove the result with commands and a scanner. To make it repeatable I never do any of that by hand on the real server: the steps live in an image build and in configuration management (a tool such as Ansible that describes the desired state and applies it idempotently, meaning a second run changes nothing), and a scheduled run reports drift (a host that no longer matches the desired state).

The point of every step is to shrink the attack surface: the set of things an attacker can reach or abuse. An assessor looking at the host should see few open ports, few accounts that can log in, and no weak defaults left to find.

The eight steps, why each matters, and how I check it

Repeatable: the same decisions as code

sshd reads the files in /etc/ssh/sshd_config.d/ in alphabetical order and, for each setting, the first value it finds wins. That is why the file is named 00-hardening.conf: it sorts before vendor or cloud-image drop-ins such as 50-cloud-init.conf, so its values cannot be overridden by them. The play's become: false means it does not escalate to root with sudo, because in the test container it already runs as root; on real hosts the play would use become: true.

This playbook was executed with ansible-core 2.16.3 inside an ubuntu:24.04 container (root, ansible-playbook baseline.yml). It writes the SSH drop-in, a validated sudoers rule and the sysctl drop-in, then fails the run if the effective SSH configuration is not what was intended.

- hosts: localhost
  connection: local
  become: false
  tasks:
    - name: sshd hardening drop-in (read first because of the 00 prefix)
      ansible.builtin.copy:
        dest: /etc/ssh/sshd_config.d/00-hardening.conf
        mode: "0644"
        content: |
          PasswordAuthentication no
          KbdInteractiveAuthentication no
          PermitRootLogin no

    - name: sudoers rule for the deploy user, syntax-checked before install
      ansible.builtin.copy:
        dest: /etc/sudoers.d/deploy-webapp
        mode: "0440"
        validate: /usr/sbin/visudo -cf %s
        content: |
          deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart webapp

    - name: kernel network hardening drop-in
      ansible.builtin.copy:
        dest: /etc/sysctl.d/90-hardening.conf
        mode: "0644"
        content: |
          net.ipv4.conf.all.accept_redirects = 0
          net.ipv4.conf.all.send_redirects = 0
          net.ipv4.conf.all.accept_source_route = 0
          net.ipv4.conf.all.log_martians = 1

    - name: refuse to continue if the effective sshd config is not what we intend
      ansible.builtin.command: /usr/sbin/sshd -T
      register: eff
      changed_when: false
      failed_when: "'passwordauthentication no' not in eff.stdout"

Run output (first run, then second run on the same host):

TASK [sshd hardening drop-in (read first because of the 00 prefix)] ************
changed: [localhost]
TASK [sudoers rule for the deploy user, syntax-checked before install] *********
changed: [localhost]
TASK [kernel network hardening drop-in] ****************************************
changed: [localhost]
TASK [refuse to continue if the effective sshd config is not what we intend] ***
ok: [localhost]
localhost                  : ok=5    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
--- second run
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

The check task runs sshd -T, which refuses to start with "Missing privilege separation directory: /run/sshd" on a host where the SSH service has never run; on a real server ssh.service creates that directory, and in the bare test container it was created first with mkdir -p /run/sshd. The recap line says ok=5 although the play lists four tasks: Ansible runs an automatic Gathering Facts step first (it collects details about the host), and that counts as the fifth. In the first run three tasks changed something and the fourth only read the result, which is why it shows ok and changed=3. changed=0 on the second run is the idempotence proof. The sysctl file only takes effect once the settings are loaded (sysctl --system reads every system directory, -p reads one file), which I did not run because a container does not let you change host kernel settings. The kernel's own documentation (ip-sysctl) gives tcp_syncookies a default of 1 and rp_filter a default of 0; on the test kernel the defaults read back as rp_filter = 0, tcp_syncookies = 1, accept_redirects = 0, send_redirects = 1 and accept_source_route = 0, which is why the drop-in sets send_redirects explicitly rather than trusting the default.

Reverse-path filtering (rp_filter, which drops packets whose source address would not be routed back out the same interface) is deliberately not in the drop-in. Strict mode is correct for a single-homed web server but breaks hosts with asymmetric routing, and the kernel uses the larger of the all and per-interface values (the kernel documentation: "The max value from conf/{all,interface}/rp_filter is used"; the values are 0 off, 1 strict, 2 loose), so setting all to strict cannot be undone for one interface, and I decide it per host class. Asymmetric routing means replies leave by a different network interface than the one the request arrived on. Illustrative case: a server has a public interface and a management interface, and a request from a monitoring system arrives on the management interface while the server's default route sends replies via the public one. Strict mode checks "would I send a reply to this source out of the interface it came in on?", answers no, and silently drops the request.

File permissions on the application tree, as run in the same kind of container (users webapp and deploy created first):

$ chmod 750 /srv/app; chmod 2770 /srv/app/shared
$ setfacl -m u:deploy:rx /srv/app
$ getfacl -p /srv/app
# file: /srv/app
# owner: webapp
# group: webapp
user::rwx
user:deploy:r-x
group::r-x
mask::r-x
other::---
$ stat -c '%A %U:%G %n' /srv/app /srv/app/shared
drwxr-x--- webapp:webapp /srv/app
drwxrws--- webapp:webapp /srv/app/shared
$ touch /srv/app/oops; chmod 666 /srv/app/oops
$ find /srv -xdev -type f -perm -0002 -print
/srv/app/oops

The s in drwxrws--- is the setgid bit on a directory, so files created inside inherit the directory's group. The ACL (access control list, a per-user permission beyond owner, group and other) lets deploy read the tree without joining the webapp group, and other has no access at all. The find line is the audit: any file under the application tree that anyone can write is a finding. find /usr/bin -xdev -type f -perm -4000 lists setuid files; on the test image it printed chfn, chsh, gpasswd, mount and newgrp among others, and the review question is whether each is expected for this server's role.

Steps 4, 7 and 8 as commands

These three steps are not part of the playbook above; each was run in an ubuntu:24.04 container. The firewall needs --cap-add NET_ADMIN to run there.

Step 4, default-deny inbound with ufw (Ubuntu's firewall front end), allowing SSH only from an illustrative admin range (203.0.113.0/24 is a documentation address block) and 443 from anywhere:

ufw default deny incoming
ufw default allow outgoing
ufw allow from 203.0.113.0/24 to any port 22 proto tcp
ufw allow 443/tcp
ufw enable
ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    203.0.113.0/24
443/tcp                    ALLOW IN    Anywhere
443/tcp (v6)               ALLOW IN    Anywhere (v6)

Run over an SSH session, ufw enable first asks "Command may disrupt existing ssh connections. Proceed with operation (y|n)?"; the rule for port 22 above is added before enabling so that your own session survives the prompt. Read the table as: anything inbound that has no row is dropped; port 22 answers only the admin range; 443 answers everyone, on IPv4 and IPv6.

Step 7, audit rules. Create /etc/audit/rules.d/50-hardening.rules (the augenrules tool merges every file in that directory into /etc/audit/audit.rules, which auditd, the Linux audit daemon, loads):

-a always,exit -F arch=b64 -F dir=/etc/ssh -F perm=wa -F key=ssh_config
-a always,exit -F arch=b64 -F dir=/etc/sudoers.d -F perm=wa -F key=sudoers
-a always,exit -F arch=b64 -F path=/etc/passwd -F perm=wa -F key=accounts
-a always,exit -F arch=b64 -F path=/etc/shadow -F perm=wa -F key=accounts

Each line says: always log, on exit from a system call, when a file in this directory (dir=) or this exact file (path=) is written to or has its attributes changed (perm=wa: w is write, a is attribute change; r and x exist too), and tag the record with a key so ausearch -k ssh_config finds it later. Running augenrules in the container merged the file into /etc/audit/audit.rules; loading rules into a kernel was not possible there, so the test edit that produces an audit record needs a real host. Shipping logs off the host (the other half of step 7) is done with your log forwarder, such as rsyslog or a vendor agent; logger "test" then proves the path.

Step 8, file integrity with AIDE (Advanced Intrusion Detection Environment). Illustrative minimal setup, using a throwaway configuration file that watches one directory and records permissions, owner, group, size and SHA-256 hash:

# /tmp/aide.conf
database_in=file:/tmp/aide.db
database_out=file:/tmp/aide.db.new
database_new=file:/tmp/aide.db.new
report_url=stdout
Rules = p+u+g+s+sha256
/srv/demo Rules

aide -c /tmp/aide.conf --init builds the baseline database as aide.db.new; copy it to aide.db (the file named by database_in) to make it the baseline, then aide -c /tmp/aide.conf --check compares the disk with it and a clean run exits 0. After echo "more text" >> /srv/demo/app.conf (the file held hello plus a newline, 6 bytes), the check printed the following, abridged: the real report also has dashed section headings, shows the new SHA-256 value beside the old one, and ends with a block of database checksums.

AIDE found differences between database and filesystem!!

Summary:
  Total number of entries:	2
  Added entries:		0
  Removed entries:		0
  Changed entries:		1

Changed entries:
f > ...     H     : /srv/demo/app.conf

Detailed information about changes:
File: /srv/demo/app.conf
 Size      : 6                                | 16

The summary counts entries added, removed and changed; the detail block shows the stored value on the left of | and the current value on the right (the file grew from 6 to 16 bytes, and its SHA-256 differs as well). The exit status is a sum: 1 if files were added, 2 if removed, 4 if changed, so this run exited 4 and a clean run exits 0, which makes it easy to alert on from a schedule.

Worked example: attack surface before and after

Before step 4 a default image running only SSH listens on one port. On the test container, with sshd started on port 2222, ss -tln printed:

State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      128          0.0.0.0:2222      0.0.0.0:*          
LISTEN 0      128             [::]:2222         [::]:*          

On a real web host the target list is explicit: SSH (restricted by the firewall to the bastion or admin range) and 443. Anything else that appears in ss -tln after provisioning (a database listening on all interfaces, a metrics agent, a leftover test service) is either bound to localhost, firewalled, or removed. That before and after listing is also what I hand a security assessor.

Trade-offs and pitfalls

• Developer workflow. Developers lose "just sudo to root and fix it". The fix is to give them what they actually do as named sudo rules (restart the service, read its logs), to deploy through the pipeline instead of by hand, and to give them a read-only way to see logs. Pushing back on blanket root is easier when the three commands they need work on day one.
• Emergency access. Key-only SSH plus a firewall can lock out the people who must fix an incident. Keep an out-of-band path (cloud serial console or the server's management controller) and a documented break-glass credential that is sealed, audited and rotated after use. Test it before you need it.
• Hardening that breaks the app. Apply the baseline to a staging host built from the same image, run the application's smoke tests, and only then promote the image. A baseline that is only ever tested on production eventually causes an outage.
• One-off hand edits. A hardened server that someone then tweaks by hand drifts. Either rebuild instead of editing, or have configuration management revert and report the change.
• File integrity is noisy until it is told about patch windows: baseline after patching, and alert on changes outside them.