Data Protection and Encryption in Practice Questions

Direct answer

Data protection has to cover three different moments in a value's life: while it sits on a disk (at rest), while it moves across a network (in transit), and while a program is actively working with it in memory (in use). Each state has a different attacker in mind, and being strong in one gives you no protection in the others.

Structured elaboration

The same logic scales across an enterprise's whole storage surface, not just one database: a relational database's TDE, an object store's SSE, a message queue's on-disk encryption (for example Kafka's disk-level encryption), and encrypted backups are all just different instances of "at rest," judged by the same threat model. Who actually holds the key matters as much as whether encryption exists at all: a secrets manager might use a fully provider-managed key inside a cloud KMS (Key Management Service, the service that generates and guards encryption keys), or you might bring your own key (BYOK), which changes whether the provider itself could ever access your data even under compulsion.

Worked example

A payment record moves through three states in one request: it is written to a database with TDE enabled (at rest), read back by an API service over mTLS (in transit), then held in that service's memory while an interest calculation runs (in use). If the at-rest and in-transit controls are both configured correctly, a SQL injection vulnerability in the application layer can still read the row in plaintext, because the app is trusted to decrypt it as part of normal operation. Encryption at rest defends against someone bypassing the app to read raw storage, not against someone abusing the app itself.

Trade-offs and pitfalls

Encryption at rest and in transit are inexpensive, mature, and should be the default everywhere. Encryption in use is a much heavier tool: it requires specialized hardware, has real performance and compatibility costs, and should be reserved for cases where you specifically distrust the infrastructure operator (your own cloud provider, or a shared host) rather than applied by default. None of the three states protect against an authorization bug, an insider with legitimate key access, or a compromised credential; they are complementary controls, not substitutes for access control.

Direct answer

Encrypt PII (personally identifiable information) fields like a social security number or email address at the service boundary, wrapping a per-field data key with a key from a central KMS (Key Management Service), and choose deterministic encryption only for the specific fields that must remain exactly searchable, randomized encryption for everything else, since randomized ciphertext reveals nothing about whether two values match.

Structured elaboration

Deterministic versus randomized: Deterministic encryption produces identical ciphertext for identical plaintext every time, which enables equality lookups and database joins, but leaks pattern information: anyone who can see the ciphertext column can tell which rows share a value, and frequent values become visible through simple frequency analysis. Randomized encryption (for example AES-GCM with a fresh random nonce each time) produces different ciphertext every time for the same plaintext, so no lookup is possible without decrypting, and no pattern leaks. Default to randomized; use deterministic only where a genuine business need for exact-match search exists on that specific field.

Key-per-field versus key-per-tenant: Key-per-field uses a separate data key for each field type (one for SSNs, another for emails) across all tenants, limiting the blast radius of a key compromise to one field type. Key-per-tenant uses one key hierarchy per tenant covering all of that tenant's fields, limiting blast radius to a single tenant, and it enables crypto-shredding: deleting a tenant's key instantly and irreversibly makes all of that tenant's encrypted data unreadable, which is a fast, reliable way to satisfy a tenant-offboarding data-deletion requirement without a slow row-by-row delete job.

Where decryption happens: Decrypting inside the calling application, using a shared internal library, keeps the database and any database proxy from ever seeing plaintext, and centralizes the crypto logic so individual teams don't reimplement it insecurely. Decrypting at a database-proxy layer (a sidecar sitting between the application and the database) centralizes crypto operations without requiring every service to integrate the library, but turns that proxy into a single high-value target that needs its own hardening. For most PII, application-layer decryption is the safer default; a proxy is a reasonable compromise for a large number of legacy services that can't easily be touched.

Library selection for a polyglot stack: The same field must decrypt correctly whether it was written by a Java service or read by a Python one, so pick a single, well-vetted, cross-language cryptographic library or specification, such as Google's Tink, rather than letting each team choose its own primitives independently, and verify that test vectors produce byte-identical ciphertext behavior across every language in use before relying on it.

Schema versioning: Attach a small header to every ciphertext recording the key ID and algorithm version used. When a field's encryption scheme changes, for example moving email from randomized to deterministic because a new search requirement appeared, run a background job that decrypts with the old key and version, re-encrypts with the new one, updates the version tag, and only then removes the old ciphertext. This allows field-by-field migration without a risky, all-at-once cutover.

Concrete field examples. Email is a strong candidate for deterministic encryption, since "does this email already exist" is a common exact-match check during signup. A credit card number generally should not be handled by this kind of general field-level encryption at all; it belongs in a dedicated tokenization flow with its own PCI-scoped vault, not the shared field-encryption path used for identifiers like email or SSN.

Worked example

A signup service receives { email, ssn }. The email field is encrypted deterministically with the tenant's per-tenant data key, so a later WHERE email = ? lookup at signup time works without decrypting every row. The SSN field is encrypted with a randomized scheme using the same tenant key, since nothing in the product needs to search on it, and its ciphertext changes every time even for the same value. Both data keys are themselves wrapped by a tenant-scoped key held in the central KMS, so revoking that tenant's access is a single key operation, not a per-field cleanup.

Trade-offs and pitfalls

Deterministic encryption on a low-cardinality field, a boolean flag or a two-digit country code, leaks almost the entire value through frequency analysis, since there are only a handful of possible ciphertexts to distinguish; never apply it there. Combining key-per-tenant encryption with a shared, cross-tenant search index also breaks isolation unless the index itself is scoped per tenant.

Direct answer: TLS (Transport Layer Security, the protocol that encrypts and authenticates traffic between a client and a server) protects data in transit by having the client and server agree on a shared secret key over a public network without ever sending that key in the clear, then encrypting everything that follows with it. Reviewing a server's configuration means checking four things: which cipher suites (the specific combination of algorithms used for key exchange, encryption, and integrity checking) are enabled, which protocol versions are allowed, whether certificate validation is strict, and whether renegotiation is handled safely.

Structured elaboration:

The handshake, at a high level. First, the client sends a ClientHello proposing a TLS version and a list of cipher suites it supports. The server replies with a ServerHello picking a cipher suite from that list, plus its certificate, which proves its identity by being signed by a certificate authority the client already trusts. Both sides then derive a shared symmetric session key through a key exchange, most modern TLS uses a Diffie-Hellman-based exchange, which lets both sides compute the same secret from public values even though an eavesdropper sees those same public values. Finally, both sides confirm the handshake wasn't tampered with, and every message after that point is encrypted with the shared session key. TLS 1.3 compresses this into fewer round trips than TLS 1.2, but the underlying shape, agree on parameters, exchange keys, confirm, then encrypt, is the same.

What to check when reviewing a server's configuration. Cipher suites: disable legacy or weak suites, anything using RC4, 3DES, or export-grade cryptography, or that lacks forward secrecy (a property where each session uses a fresh key, so stealing the server's long-term key later cannot decrypt past recorded traffic), and prefer AEAD (authenticated encryption with associated data, a cipher mode that encrypts and checks integrity in one step) ciphers such as AES-GCM. Protocol versions: disable SSLv3, TLS 1.0, and TLS 1.1; require TLS 1.2 as a floor and prefer TLS 1.3 wherever every supported client can use it. Certificate validation: confirm the certificate chains to a trusted root, hasn't expired, and its subject or SAN (Subject Alternative Name, the field listing which hostnames a certificate is valid for) actually matches the hostname being served, and confirm revocation checking via OCSP or a certificate revocation list is configured, since an expired-but-unchecked certificate is a common gap. Renegotiation: TLS renegotiation is a client or server re-running the handshake on an already-open connection to refresh keys or parameters; disable client-initiated renegotiation, or at minimum ensure secure renegotiation is enforced, since insecure renegotiation was the root of a well-known man-in-the-middle class of attack against older TLS deployments.

Certificate validation in practice means checking the whole trust chain, not just the leaf certificate: the chain runs from the server's own certificate up through one or more intermediate certificate authorities to a root the client already trusts, and a server that omits an intermediate certificate will fail validation for clients that don't already cache it, even though the leaf certificate itself is fine. Mutual TLS, or mTLS, where both sides present a certificate rather than just the server, is called for when you need cryptographic proof of the client's identity too, which is the normal case for service-to-service traffic inside a zero-trust network, or for a business-to-business API where you want to authenticate the calling organization by certificate rather than, or in addition to, an API key.

Worked example: A server offering TLS 1.0 alongside TLS 1.3, with a cipher list that still includes a 3DES suite, and no OCSP stapling configured, fails this review on three of the four checks: drop TLS 1.0 and the 3DES suite, and add OCSP stapling, where the server proactively attaches revocation-status proof to its own handshake instead of making every client query the certificate authority separately.

Trade-offs and pitfalls: Disabling old protocol versions and weak ciphers can break traffic from clients that genuinely can't be upgraded, some older devices only speak TLS 1.0. That's usually still the right trade to make, but it needs to be a deliberate, communicated decision rather than a surprise outage.

Direct answer

OIDC (OpenID Connect, an identity layer on top of OAuth 2.0) lets the CI provider act as an identity provider: it issues a short-lived, signed JSON Web Token (JWT) for each workflow run, containing claims like the repository, branch, and environment. The cloud account is configured to trust that specific OIDC issuer and to map particular claim values to an IAM (Identity and Access Management) role. The CI job presents the token to the cloud's token service, which verifies the signature and claims against the trust policy and returns temporary, expiring credentials. No long-lived cloud secret is ever stored in CI.

Structured elaboration

The trust relationship, step by step:

1. The cloud account registers the CI provider (for example GitHub Actions) as a trusted OIDC identity provider, pinning its issuer URL and public signing keys.
2. An IAM role is created with a trust policy whose condition checks specific token claims, most importantly the repository and the branch or environment (for example repo:my-org/my-repo:environment:production), not a wildcard.
3. At run time, the CI job requests an OIDC token scoped to the intended cloud audience.
4. The job calls the cloud's token-exchange API (AWS's AssumeRoleWithWebIdentity via STS, AWS's Security Token Service, or the equivalent Workload Identity Federation call on GCP) presenting that token.
5. The cloud validates the signature against the registered issuer's public keys, checks the trust-policy condition against the token's claims, and if it matches, returns short-lived credentials, typically expiring within an hour.

Worked example

sequenceDiagram
    participant Runner as CI Runner
    participant OIDC as CI OIDC Provider
    participant STS as Cloud Token Service
    Runner->>OIDC: request short-lived signed token (claims: repo, branch, env)
    OIDC-->>Runner: signed JWT
    Runner->>STS: AssumeRoleWithWebIdentity(JWT)
    STS->>STS: verify signature + match trust policy condition
    STS-->>Runner: temporary credentials (expire in ~1 hour)

Mitigating replay against the OIDC endpoint: each token carries a short exp (expiry) claim and an aud (audience) claim pinned to the specific cloud account it's meant for, so a token intercepted after issuance is only useful for a narrow window and only against the one consumer it was minted for. The trust policy should also match exact claim values rather than wildcards (an exact branch name, not refs/heads/*), since a wildcard match lets anyone who can open a branch or fork mint a token that satisfies the condition. Monitoring AssumeRoleWithWebIdentity calls for unexpected source repositories or branches adds a detective control on top of the preventive narrowing.

Trade-offs and pitfalls

Two additional edge cases worth naming explicitly. First, multi-environment scoping: separate trust-policy conditions and separate IAM roles per environment claim (environment:staging versus environment:production) so a staging pipeline's token can never assume the production role, even if both use the same OIDC issuer. Second, build-time secrets ending up in the resulting container image: if the OIDC-derived temporary credentials are used mid-build (to pull a private dependency, for example), they must be passed as build-step environment variables, never as a Dockerfile ARG or ENV, since both persist in the image's layer history; use BuildKit secret mounts (--secret) instead, which are not written into any layer.

Direct answer

Native Kubernetes Secrets are only base64-encoded, not encrypted, and by default persist in plaintext-equivalent form in etcd (the cluster's key-value store); anyone who can read etcd or take a backup of it can recover them unless etcd encryption at rest is explicitly enabled. External secret store integration keeps the actual secret material in Vault or a cloud secrets manager and either mounts it directly into the pod's filesystem or syncs it into a native Secret object, with pods authenticating to the store using their own Kubernetes service account identity.

Structured elaboration

Two common integration shapes:

• Secrets Store CSI Driver: a Container Storage Interface driver mounts the secret directly from Vault, AWS Secrets Manager, or Azure Key Vault into the pod as a volume. The secret material can be kept off etcd entirely if you skip the driver's optional "sync as native Kubernetes Secret" feature.
• External Secrets Operator: watches an external secret and syncs its value into a native Kubernetes Secret object, which is more convenient for apps that already read Secrets the standard way, at the cost of the value now also existing in etcd.

Pod-to-store authentication, concretely with Vault's Kubernetes auth method: the pod presents its own projected service account token (a signed JWT (JSON Web Token) that Kubernetes automatically mounts into the pod) to Vault; Vault validates that token against the Kubernetes API's TokenReview endpoint, confirms which service account and namespace it belongs to, and maps that identity to a Vault policy scoped to that namespace, for example granting read only on secret/data/<namespace>/*. This means two pods in different namespaces authenticate with the same mechanism but land on completely different, non-overlapping policies.

Worked example

A pod in the orders namespace mounts its projected service account token automatically. A Vault Agent sidecar in that pod uses the token to authenticate via Vault's Kubernetes auth method, and Vault's role binding maps system:serviceaccount:orders:orders-app to a policy that only allows reading secret/data/orders/*. If an attacker compromises a pod in a different namespace, its service account token authenticates successfully to Vault but is bound to a different policy, so it cannot read the orders namespace's secrets even though authentication itself succeeded.

Trade-offs and pitfalls

Protecting material in etcd and node memory takes more than picking an integration pattern:

• Enable etcd encryption at rest (an EncryptionConfiguration backed by a KMS (Key Management Service) provider) so even a stolen etcd backup is ciphertext.
• Restrict etcd network access to control-plane nodes only; it should never be reachable from a worker node running arbitrary workloads.
• Where possible, avoid the CSI driver's "sync as native Secret" mode, keeping the value only in the pod's memory-backed (tmpfs) mount, so it never touches etcd at all.
• Use tmpfs, not a regular disk-backed volume, for any file-based secret mount, so the value doesn't persist to node disk either.

The most common mistake is treating "we use the CSI driver" as sufficient by itself while leaving the sync-to-native-Secret option enabled, which quietly recreates the exact etcd-plaintext exposure the external store was meant to avoid.