Identity, Authentication, and Access Management Questions

Direct answer

Treat a suspected privileged Active Directory (AD, Microsoft's enterprise directory service that stores accounts, groups, and policy) account compromise as an active incident, not a password-reset ticket. Contain the account's ability to act while evidence is still fresh, collect the specific Windows and AD logs that reconstruct what it touched, and only rotate or disable the credential once you have captured what an immediate reset would destroy.

Structured elaboration

1. Containment actions. Disable the account rather than delete it, deleting destroys the security identifier (SID) history you need for forensic correlation later. Force logoff any active sessions on domain controllers (DCs) and member servers the account is logged into. Purge or invalidate its Kerberos tickets (Windows' default authentication protocol, which issues time-limited tickets instead of repeatedly sending a password): a full ticket invalidation ultimately requires a password reset, but that reset should happen after evidence capture, not before, since it tears down the very session artifacts you are trying to preserve. If the account was actively logged into a specific endpoint, isolate that endpoint from the network (not power it off) so you preserve volatile memory for later analysis while stopping further lateral movement.

2. Forensic evidence collection: which logs and artifacts. This is the part that determines whether the rest of the investigation is even possible, so specific event IDs matter:

• Domain controller Security event log: Event ID 4624 (successful logon), checking the Logon Type (3 = network, 10 = RDP, 2 = interactive) to tell remote exploitation from console access; 4625 (failed logon) to look for a preceding brute-force attempt; 4672 (special privileges assigned to new logon, confirming admin-equivalent rights were actually exercised, not just held); 4768/4769 (Kerberos ticket-granting-ticket and service-ticket requests, which trace which resources were requested even without a fresh interactive logon); 4720/4722/4738 (account created, enabled, or changed, in case the attacker used the account to plant persistence).
• File and share access: Event ID 4663 (an attempt was made to access an object) only fires if a System Access Control List (SACL) was configured on the sensitive share ahead of time. If it was not, that absence is itself a finding, no reconstruction of exactly which files were read is possible, and the team falls back to file-system access timestamps (often disabled for performance) or a file-integrity monitoring tool if one exists.
• AD change auditing: Event ID 5136 (a directory service object was modified) to catch unexpected changes to Group Policy Objects (GPOs) or security group membership, for example an unexpected addition to Domain Admins.
• Endpoint artifacts on any host the account authenticated to: the PowerShell operational log's Event ID 4104 (script block logging, which catches obfuscated or encoded commands), Sysmon process-creation and network-connection events if deployed, and any Endpoint Detection and Response (EDR) telemetry from that host.

3. Account remediation. Rotate the credential immediately after evidence capture, disable the account if it is not required for ongoing operations, and review every group membership and delegated permission it held, a reused or synced credential elsewhere can otherwise re-grant the same access even after this specific account is fixed.

4. Determining scope of access. Correlate every log source above by timestamp and source host to build a session timeline from first suspicious logon to containment. Enumerate every share, host, and object touched during that window using the file-access and Kerberos service-ticket logs. The strongest signal is deviation from baseline: a privileged account that normally touches the same three file servers suddenly touching a dozen new ones in one session is the thing to chase, not the raw fact that a privileged account has broad access (it is supposed to).

Worked example

A realistic timeline for a service account, svc-backup-admin, holding domain-admin-equivalent rights, flagged because a security information and event management (SIEM) system alerted on an unusual logon source:

Reading this timeline: the anomaly (4624 Logon Type 3 from an unfamiliar workstation, paired with 4672) is detectable by 02:10 in an instrumented environment, well before the 03:10 exfiltration. Scope is the Payroll share, for that time window, originating from WKSTN-114. Remediation is not just disabling svc-backup-admin, it is also treating WKSTN-114 as the likely actual entry point: the account's credential had to come from somewhere, and a workstation with a cached privileged credential (harvestable by a credential-dumping tool if that account had ever logged into it interactively) is a common root cause that would keep producing new compromised accounts if left unaddressed.

Trade-offs and pitfalls

• Resetting the password before capturing ticket and session artifacts destroys evidence. The default order should be capture first, then remediate, but under confirmed active exfiltration that order can reasonably flip. Whichever way it goes, the decision should be a documented judgment call, not a reflex.
• Object-access auditing has to be turned on before the incident, not during it. Event ID 4663 requires a SACL configured on the object ahead of time. Teams that skipped this cannot reconstruct which files were actually read after the fact, only that a logon happened, and this gap is usually discovered mid-incident, when it is too late to fix for this event.
• Treating the compromised account as the root cause instead of a symptom is the most common wrong turn. A privileged domain account rarely "goes rogue" on its own; the actual initial-access vector (phishing, a cached credential, an unpatched service on the endpoint it logged in from) is what needs fixing, or the same technique compromises the next privileged account.
• Disabling immediately versus monitoring briefly is a real trade-off, not a checklist item. Immediate disable stops the bleeding but ends your visibility into what the attacker is actually doing. A short monitored window can reveal additional footholds, but only makes sense once you are confident you are not watching active, ongoing data exfiltration.

Direct answer

Privileged Access Management (PAM) is the set of tools and processes specifically for securing, controlling, and monitoring access to accounts that carry outsized power, such as root, domain administrator, database administrator, or cloud organization-administrator accounts, because a compromise of one of these does far more damage than a compromise of an ordinary user account. Its four common controls are vaulted credentials (nobody memorizes or directly holds the actual password), just-in-time (JIT) elevation (privileged access exists only for the duration it is actually needed, not standing indefinitely), session recording (every privileged session is logged and auditable), and bastion hosts (a single hardened, monitored gateway that all privileged access must pass through, rather than direct network access to sensitive systems).

Structured elaboration

Vaulted credentials. Passwords or keys for privileged accounts are stored in a secure vault rather than known to, or memorized by, the administrator who needs them. When access is required, the vault checks out the credential (or injects it directly into a session without ever displaying it to the user) and can automatically rotate it after each use, so a privileged credential is never a static, reusable secret sitting in someone's memory or a password manager.

Just-in-time elevation. Rather than a standing administrator account that exists at full privilege all the time, a user requests elevated access for a specific task, often through an approval workflow, and receives it for a bounded time window that expires automatically afterward. This shrinks the amount of time a privileged account's risk actually exists in the environment, since most of the time, nobody holds standing elevated access at all.

Session recording. A PAM system brokers the actual connection to the target system rather than letting the user connect directly with their own credential, and it records what happens during that session, keystrokes, commands, or a full screen recording depending on the tool, producing a forensic record of exactly what a privileged session did. This matters both for after-the-fact investigation and as a deterrent: a session known to be recorded is less likely to be misused.

Bastion hosts. A single hardened, closely monitored gateway machine is the only path through which privileged access can reach sensitive internal systems, rather than administrators connecting directly over the network. Concentrating access at one narrow chokepoint is what makes the vaulting and session-recording controls above practical to enforce consistently: there is one place to apply them, rather than dozens of direct paths to secure individually.

Integration into the identity lifecycle. PAM enrollment is not a separate, disconnected system; it is provisioned and deprovisioned alongside a person's broader identity lifecycle. When someone is granted a privileged role (promoted to an on-call role, hired as a database administrator), their PAM eligibility, vault access, and just-in-time elevation permissions are provisioned as part of that same onboarding step. The more consequential direction is offboarding: when someone leaves a privileged role or the organization, their PAM access has to be revoked immediately, because standing privileged access left behind after someone's role changes is exactly the highest-risk kind of orphaned access to leave lingering. Because of that risk, privileged entitlements are also typically reviewed and re-certified more frequently than ordinary access, as part of the organization's periodic access-review process.

Emergency break-glass workflows. Break-glass access exists for the specific scenario where the normal control plane itself is unavailable or too slow, the identity provider or multi-factor authentication (MFA) system is down, or an incident is severe enough that waiting for a standard just-in-time approval would cause unacceptable harm. A break-glass credential is vaulted under tighter controls than routine privileged access, often requiring two people to jointly retrieve it or an out-of-band physical retrieval step, and its use triggers immediate, high-visibility alerting to the security team. Critically, break-glass deliberately bypasses the normal approval workflow, which is exactly why its use has to be rare, loudly alerted on the moment it happens, and always followed by a mandatory post-use review and credential rotation, regardless of whether the underlying emergency turns out to have been genuine.

Worked example

A database administrator (DBA) needs emergency read access to a production database during an incident. Walking through all four controls together: the DBA requests just-in-time elevation for the specific production system, naming the incident as justification; an on-call approver grants it for a two-hour window. The DBA connects through the organization's bastion host, the only network path into the production environment, rather than directly to the database server. The PAM system checks a vaulted database credential out on the DBA's behalf and injects it into the session without ever displaying the actual password, and the entire session is recorded. Two hours later, the elevation expires automatically and the checked-out credential is rotated, closing the window. If instead the identity provider itself were down during a severe outage and no one could complete the normal just-in-time approval flow, the on-call lead would use the break-glass procedure instead: retrieving a specially vaulted emergency credential (requiring a second approver to also confirm the retrieval), which immediately pages the security team, and after the incident, a mandatory review confirms the break-glass use was legitimate before the emergency credential is rotated and re-vaulted.

Trade-offs and pitfalls

• A PAM program that only covers well-known accounts (root, a handful of domain admins) but misses newer privileged surfaces, such as a cloud console's organization-admin role or a Kubernetes cluster-admin binding, leaves real gaps. Privileged access keeps expanding into new platforms faster than a PAM rollout typically covers them, so the inventory of "what counts as privileged" needs to be revisited regularly, not defined once.
• Just-in-time elevation only reduces risk if the approval step is genuinely meaningful. An approval workflow that rubber-stamps every request without real scrutiny provides the audit trail of a JIT system without the actual risk reduction, which is a common way JIT programs quietly become theater.
• Break-glass access that is never tested is a real operational risk, not just a security one. If the emergency procedure is only ever exercised during an actual crisis, the first real use may reveal that the vaulted credential has expired, the retrieval process is broken, or the people who know the procedure have since left, exactly when there is no time to fix it.
• Session recording without a review process only helps after an incident is already known to have happened. Recordings that are stored but never sampled or reviewed do not provide the early-warning value they could; some organizations pair session recording with automated anomaly detection over the recorded activity, not just passive storage.

Direct answer

NTFS (New Technology File System) permissions and share permissions are two independent access-control layers on a Windows file server: NTFS permissions live on the file or folder itself and apply to every access path, local or over the network, while share permissions apply only to the network share definition and are irrelevant to someone logged on locally. For a network request, the share permission is evaluated first as a gate at the SMB (Server Message Block, the file-sharing network protocol) layer, then NTFS permission is evaluated on the underlying object, and the effective result is whichever of the two is more restrictive, never the more permissive.

Structured elaboration

Which is evaluated first, and how they combine. For a request over \\server\share\path, the server checks the share's access control list first; if that already denies the requested access, NTFS is never even consulted. If the share allows it, NTFS permission on the actual file or folder is evaluated next. The final effective access for that network request is the intersection, the more restrictive of the two results, not an average and not "whichever came first wins." A local logon directly on the server skips the share layer entirely, since there is no SMB negotiation involved; only NTFS applies.

flowchart TD
    A[Client requests network access to a UNC path] --> B{Access path}
    B -->|Network SMB| C[Evaluate share ACL first]
    B -->|Local logon on the server| D[Share layer skipped entirely]
    C --> E[Evaluate NTFS ACL on the underlying file or folder]
    D --> E
    E --> F[Effective access: more restrictive of share and NTFS, or NTFS alone if local]

How permissions combine within NTFS itself (a distinct evaluation from the share/NTFS intersection above). Within the NTFS access control list, Windows evaluates access control entries (ACEs) in a fixed canonical order: explicit Deny, then explicit Allow, then inherited Deny (nearest parent first), then inherited Allow. The access check walks this order and stops as soon as every requested access bit has been resolved. This produces a genuinely counter-intuitive but real behavior: an explicit Allow set directly on a folder is evaluated, and can fully satisfy the requested access, before the check ever reaches an inherited Deny coming down from a parent folder. "Deny always wins" is therefore only a half-truth: it is true when comparing two entries at the same explicit-or-inherited level, but an explicit Allow beats an inherited Deny, because explicit entries are evaluated first.

How this behaves through a clustered file server. On a Windows failover cluster hosting a file-server role (including a Scale-Out File Server serving SMB from multiple nodes at once), both the share definition and the NTFS access control list live on the shared storage itself, not on any individual node, so a failover moving the resource to another node does not change which permissions apply. What can change during a failover is client-visible continuity: SMB 3.0's transparent failover lets an open file handle survive the transition without an application-visible error, but only if the share is configured for continuous availability and both client and server support it; without that, in-flight connections drop and reconnect, which simply forces a fresh permission check on reconnect rather than changing the permission model itself. The practical implication is to avoid making permission or ACL changes in the middle of a planned failover, since a change that has not fully replicated to the shared storage before the transition can be applied inconsistently.

Diagnosing a real access-denied ticket with logs, rather than hand-tracing ACEs. Rather than guessing which of the two layers is at fault, enable object-access auditing (a System Access Control List, or SACL, on the folder, and the "File System" advanced-audit subcategory) and check the Security event log for the layer-specific events: event ID 4656 for a handle request at the NTFS layer (which, since Windows Server 2012, includes an access-reasons expansion naming exactly which ACE produced a denial) and event ID 5145 for the share-layer access check over SMB. Where auditing was not already enabled before the incident, or as a faster first pass, the Advanced Security Settings "Effective Access" tab computes the same combined-precedence result described above for a chosen user or group without requiring anyone to hand-trace ACE order from memory, which is exactly where mistakes happen under ticket pressure.

Typical administrator mistakes. Assuming a permission granted at one layer ("they have Full Control on the share") automatically means access at the other layer, when the two are independent and only their intersection matters. Leaving the share layer wide open and relying entirely on NTFS, which is the standard, recommended pattern (below), but only works as intended if NTFS is actually kept correct, since the share layer then provides no backstop at all if an NTFS entry is ever wrong. The reverse misconfiguration, NTFS wide open and access controlled only at the share, is equally common and worse: it looks correctly restricted over the network but is fully open to any local logon path on the server, such as an administrator's RDP session, since NTFS is the only thing that would apply there. And adding a broad explicit Deny near the top of a folder tree to "lock everyone out by default," then being confused when a more specific group further down still cannot get access, without realizing an inherited Deny from above only loses to a more specific EXPLICIT Allow, not to another inherited entry.

Guidance for designing a permission strategy. Grant Everyone (or Authenticated Users) a permissive share permission (Change or Full Control) and do all real, fine-grained access control at the NTFS layer; this avoids maintaining two separately drifting permission models and concentrates auditing, effective-access tooling, and troubleshooting on the one layer that actually has rich tooling for it. Assign NTFS permissions to security groups, never individual accounts, consistent with a group-based access model. Avoid explicit Deny ACEs except for genuinely exceptional cases; an explicit Deny is a blunt override that is easy to forget about later and is a common source of the confusing tickets described above, so structuring group membership so someone simply is not in an Allow-granting group is usually the safer way to exclude them. Leave inheritance enabled by default and only break it deliberately, with the exception documented, since disabled inheritance on one folder is itself a frequent source of "why does this one folder behave differently from everything around it" surprises.

Worked example

Consider D:\Data, shared as \\FS01\Data with the recommended wide-open share permission (Everyone: Full Control at the share layer). On D:\Data itself, an administrator previously added an inherited-down NTFS access control entry, "Domain Users: Deny Full Control," intending to lock the whole tree down by default. Later, a subfolder D:\Data\ProjectX had an explicit (not inherited) NTFS entry added directly on it: "ProjectX-Team: Allow Modify."

Can a member of ProjectX-Team reach \\FS01\Data\ProjectX over the network?

1. Share layer: Everyone has Full Control at the share, so the share check allows the request.
2. NTFS layer on D:\Data\ProjectX: the relevant entries for this user are an explicit Allow (ProjectX-Team: Modify, set directly on this folder) and an inherited Deny (Domain Users: Deny Full Control, inherited from the parent). Per the canonical evaluation order, explicit Allow is checked before inherited Deny, and the requested Modify-level access bits are fully satisfied by the explicit Allow before the check ever reaches the inherited Deny entry. NTFS effective result: Allow, at the Modify level.
3. Combine the two layers: share allows Full Control, NTFS allows Modify; the effective access is the intersection, the more restrictive of the two, which is Modify.

So the computed answer is yes: a ProjectX-Team member gets Modify access over the network, despite the broad Domain Users Deny sitting on the parent folder, precisely because an explicit Allow beats an inherited Deny. An administrator who assumes "Deny always wins, full stop" would expect this user to be locked out, and the mismatch between that expectation and the actual computed result is exactly the kind of confusing ticket the Effective Access tab or the 4656/5145 event pairing above is meant to resolve without guesswork.

Trade-offs and pitfalls

The wide-open-share, NTFS-does-the-work pattern is the industry-standard approach and the one this answer recommends, but it is a genuine trade-off, not a free win: it centralizes enforcement in one place, which is simpler to audit and reason about, at the cost of having zero backstop at the share layer if an NTFS entry is ever misconfigured, so it depends entirely on NTFS being kept correct rather than providing defense in depth through two independent layers. Explicit Deny ACEs are powerful exactly because they override inheritance in one direction but not the other, which is precisely what makes them easy to misuse and hard to debug months later; treat them as a last resort, not a default tool for restricting a folder. Finally, log-based diagnosis only works retroactively if the relevant SACL auditing was already enabled before the incident happened; a common and costly pitfall is only turning on object-access auditing after a confusing access-denied ticket arrives, by which point the historical evidence for that specific event is already gone, so enabling baseline auditing on sensitive shares proactively, before there is a ticket to investigate, is what actually makes the log-based method useful when it is needed.

Direct answer

There are four practical ways to give a non-root container least-privilege read/write access to an NFS volume owned by specific numeric user IDs (UIDs): Kubernetes fsGroup, supplemental groups, Linux user namespaces with subordinate UID/GID (subuid/subgid) mapping, and host-level UID mapping done outside Kubernetes entirely. They differ mainly in WHERE the UID translation happens and how much isolation it buys you: fsGroup and supplemental groups are the simplest and rely on the container's process UID genuinely matching (or group-matching) what the NFS server expects, while user namespaces and host-level mapping let the container run as a UID that does not exist on the host at all, translating it only at the kernel boundary.

Structured elaboration

Kubernetes fsGroup
Set in the pod's securityContext.fsGroup. On volume mount, Kubernetes (via the kubelet) recursively changes group ownership of the volume's files to the specified GID and sets the setgid bit on directories, so new files created inside inherit that group. The container process itself can still run as a non-root UID; it gets access because it is a member of the fsGroup, not because it owns the files.

• Security: works well when you can standardize on one GID for a workload; the recursive chown on mount can be slow on large volumes and is skipped by some CSI drivers for NFS specifically (NFS ownership is often controlled by the export itself, not by what Kubernetes chowns), so verify your specific NFS CSI driver actually honors fsGroup before relying on it.
• Backups: backup tooling that walks the filesystem sees ordinary POSIX group ownership, so this is the most backup-tool-friendly option; nothing exotic to reason about.
• Cross-environment consistency: fragile across clusters if the underlying NFS export enforces its own UID/GID mapping (for example root-squash or a fixed export UID), since fsGroup does not override what the NFS server itself decides to expose.

Supplemental groups
Set via securityContext.supplementalGroups (pod-level) or runAsGroup plus explicit group membership baked into the container image. The container process runs with an additional GID attached directly to its process credentials, rather than relying on Kubernetes to rewrite ownership on the volume.

• Security: more predictable than fsGroup for NFS specifically, since it does not depend on a chown-on-mount step; the container simply presents the right GID when it makes NFS calls, and the NFS server's own export permissions decide access.
• Backups: same POSIX-standard picture as fsGroup; no special handling required.
• Cross-environment consistency: requires the SAME GID to mean the same thing on every cluster and on the NFS server's export map, which is a coordination burden if the platform later runs on a different NFS server or a different UID/GID numbering scheme.

User namespaces with subuid/subgid mapping
Each container gets its own private UID/GID range (backed by /etc/subuid and /etc/subgid on the host), so a process that appears to be UID 0 or UID 1000 inside the container is actually mapped to a high, host-unique UID like 100000 or 165536 outside it. Kubernetes added stable user namespace support for pods (hostUsers: false) with per-node kubelet support reaching general availability in the 1.36 release line, after being available in earlier releases behind a feature gate.

• Security: this is the strongest isolation of the four, because a container escape no longer lands the attacker on a host UID that matches anything meaningful; the container's "root" is an unprivileged, unmapped UID on the host.
• Backups: this is also the sharpest edge case. Standard NFS (NFSv3, and NFSv4 without Kerberos) has no concept of a mapped identity; it authenticates purely by the numeric UID presented on the wire. A user-namespaced container's REMAPPED UID is what actually appears in the NFS RPC calls, so the NFS server has to be configured to recognize that mapped UID, not the UID the application believes it is running as. This is a common integration gap: teams enable user namespaces for isolation and then discover their NFS exports were configured for the pre-mapping UID.
• Cross-environment consistency: the subuid/subgid ranges are typically allocated per node by the container runtime, so the SAME logical container can get a DIFFERENT mapped UID on a different node or cluster unless you deliberately pin and coordinate the ranges, which undermines portability unless it is planned for up front.

Host-level UID mapping
Done outside Kubernetes: either by using the NFS server's own ID-mapping feature (NFSv4 idmapping, or all_squash/anonuid/anongid export options that force every request to a fixed UID/GID regardless of what the client presents), or by running an in-cluster sidecar/proxy that terminates the true NFS protocol and re-authenticates internally.

• Security: forcing a fixed anonymous UID via export options is simple and works regardless of what the container's internal UID is, but it also means every workload mounting that export shares the same identity at the NFS layer, so you lose per-workload accountability at the storage layer even if Kubernetes-level RBAC still distinguishes workloads.
• Backups: predictable, since the effective UID at the storage layer never changes; backup and restore tooling only needs to know one fixed identity.
• Cross-environment consistency: this is the most portable of the four for the container itself (the container's declared UID becomes irrelevant to the NFS server), at the direct cost of losing fine-grained, per-workload storage-level identity.

Trade-offs and pitfalls

The pitfall that catches teams most often is treating this as a purely Kubernetes-side decision and forgetting that NFS, especially NFSv3 and non-Kerberized NFSv4, authenticates by bare numeric UID with no cryptographic identity behind it at all: whichever of these four approaches you pick, the UID that ends up on the wire to the NFS server is what actually determines access, not what the Kubernetes API server thinks the pod's securityContext says. In practice, fsGroup or supplemental groups are the right default for a single cluster with a stable UID/GID scheme; user namespaces are the right choice when container-escape isolation matters more than storage-layer simplicity, but only after confirming the NFS export is configured for the REMAPPED UID range, not the application's apparent UID; and host-level mapping (fixed anonymous UID) is the fallback when you need the storage layer to stay simple regardless of how many different UID schemes the workloads above it use, at the cost of losing per-workload accountability at that layer.

Direct answer

That specific message means the SSH daemon's (sshd's) strict permission checks rejected bob's home directory or .ssh directory before it ever evaluated the key itself. By default, sshd refuses to trust a public key if the account's home directory, its .ssh directory, or the key file is writable by anyone other than its owner, because a group- or world-writable path there would let another local user plant their own key and log in as bob. The fix is tightening ownership and permissions on the exact chain of directories sshd checks, then verifying the fix actually took effect.

Structured elaboration

Root causes. Any of these produce the same "bad ownership or modes" message:

1. /home/bob is not owned by bob (or root), often from files restored by a backup tool or copied in by another administrator's account.
2. /home/bob or /home/bob/.ssh has a group- or world-writable bit set (sshd's StrictModes setting, on by default, rejects this).
3. /home/bob/.ssh/authorized_keys itself is writable by group or other, even if the directories above it are fine.

Commands to fix it, applied in the order that matches how sshd walks the path:

# 1. Home directory must be owned by bob, and not writable by group or other.
sudo chown bob:bob /home/bob
sudo chmod go-w /home/bob

# 2. .ssh directory: owner-only access.
sudo mkdir -p /home/bob/.ssh
sudo chown bob:bob /home/bob/.ssh
sudo chmod 700 /home/bob/.ssh

# 3. authorized_keys file: owner-only read/write, no execute needed.
sudo chown bob:bob /home/bob/.ssh/authorized_keys
sudo chmod 600 /home/bob/.ssh/authorized_keys

Verifying the fix. Confirm the actual mode bits, not just that the commands ran without error:

stat -c "%a %U:%G %n" /home/bob /home/bob/.ssh /home/bob/.ssh/authorized_keys

Expect something like 755 bob:bob /home/bob, 700 bob:bob /home/bob/.ssh, and 600 bob:bob /home/bob/.ssh/authorized_keys. Then attempt the actual login with verbose logging so sshd's own reasoning is visible: ssh -vvv bob@host, and separately watch the server side with sudo tail -f /var/log/auth.log (or /var/log/secure on Red Hat-family distributions) for the corresponding "Accepted publickey" line, rather than assuming success from the client side alone.

If permissions look correct and it still fails, check SELinux (Security-Enhanced Linux, a mandatory access control system available on many Linux distributions). Standard Unix permission bits and SELinux are two independent checks; a directory can have perfectly correct ownership and mode and still be denied if its SELinux file context (label) is wrong, most commonly after files were restored from a backup or copied from another location without preserving context:

ls -Z /home/bob/.ssh /home/bob/.ssh/authorized_keys

.ssh and its contents should carry the ssh_home_t context. If it shows something else (commonly a generic context like user_home_t after a restore), restore the correct label rather than trying to set it by hand:

sudo restorecon -Rv /home/bob/.ssh

If that does not resolve it, check for an actual denial being logged rather than guessing further:

sudo ausearch -m avc -ts recent

An access vector cache (AVC) denial entry naming sshd and the .ssh path confirms SELinux, not a stale permission bit, is the actual blocker.

Worked example

A concrete before-and-after for this exact scenario. Suppose /home/bob/.ssh was left group-writable (mode 775) after a provisioning script ran as root and did not lock it down, a common way this exact error gets introduced:

Before: drwxrwxr-x  bob bob  /home/bob/.ssh   (group-writable, sshd rejects this)
        -rw-r--r--  bob bob  /home/bob/.ssh/authorized_keys

After applying the fix commands above:
        drwx------  bob bob  /home/bob/.ssh
        -rw-------  bob bob  /home/bob/.ssh/authorized_keys

Running the fix commands against exactly this before-state (verified directly: chmod go-w on the home directory, chmod 700 on .ssh, chmod 600 on authorized_keys) produces exactly the after-state shown, the group-write bit is gone from .ssh and authorized_keys no longer has any bits set beyond owner read/write. That is the state sshd's strict-mode check requires before it will even look at the key's contents.

Trade-offs and pitfalls

• Fixing only authorized_keys and skipping the parent directories is a common half-fix. sshd checks the whole path, home directory, .ssh, and the key file itself, so a perfectly permissioned key file inside a group-writable .ssh directory still fails.
• Assuming a fix worked because the commands returned no error is a common mistake. Always verify the actual mode bits with stat, and confirm the login itself succeeds with verbose logging, rather than trusting that chmod silently doing nothing wrong means the state is now correct.
• Jumping straight to SELinux before confirming ownership and mode bits wastes time. SELinux denials produce a similar practical symptom (login refused) but a different log signature; check the simpler, more common cause first, and only chase an SELinux label mismatch once the standard permission chain is confirmed correct.
• Manually setting an SELinux context instead of using restorecon risks getting the label subtly wrong. restorecon restores the context the system's own policy defines for that path, which is more reliable than guessing a context string by hand.