Compliance Automation and Tooling Questions

Direct answer

Policy as code means writing a rule such as "no storage bucket may be publicly readable" as a small program kept in version control, instead of a wiki page. A machine evaluates every proposed infrastructure change against the rule and returns allow or deny with a reason. For "can never reach production", one check is not enough: you stack a plan-time check in CI (continuous integration, the automated build and test pipeline), an account-level guardrail that cannot be bypassed, and a runtime scan that catches drift (a live setting changing away from what the code says, for example someone editing a bucket by hand in the console).

Why it helps compliance

• Rules are versioned, so the history shows who changed which rule and when.
• Changes go through pull-request review, like application code.
• Rules are unit-testable, so you can prove the rule denies what it should.
• The same rule produces the same answer every run, which makes it usable as audit evidence.

Three layers, and which one gives the "never"

The account-level guardrail is what actually makes "never" true. The CI rule is the fast, friendly feedback loop on top of it.

A plan-time rule (Rego, the language used by Open Policy Agent)

package terraform.s3public

import rego.v1

public_acls := {"public-read", "public-read-write", "authenticated-read"}

block_flags := ["block_public_acls", "block_public_policy", "ignore_public_acls", "restrict_public_buckets"]

changed(rc) if {
	some action in rc.change.actions
	action in {"create", "update"}
}

deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_s3_bucket_acl"
	changed(rc)
	rc.change.after.acl in public_acls
	msg := sprintf("%s: ACL '%s' makes the bucket publicly readable", [rc.address, rc.change.after.acl])
}

deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_s3_bucket_public_access_block"
	changed(rc)
	some flag in block_flags
	rc.change.after[flag] != true
	msg := sprintf("%s: %s must be true", [rc.address, flag])
}

Given a plan excerpt with one public ACL and one half-open public-access block:

opa eval -f pretty -d s3_public.rego -i input.json 'data.terraform.s3public.deny'

with this input.json:

{"resource_changes":[{"address":"aws_s3_bucket_acl.site","type":"aws_s3_bucket_acl","change":{"actions":["create"],"after":{"acl":"public-read"}}},{"address":"aws_s3_bucket_public_access_block.reports","type":"aws_s3_bucket_public_access_block","change":{"actions":["create"],"after":{"block_public_acls":true,"block_public_policy":false,"ignore_public_acls":true,"restrict_public_buckets":true}}}]}

the real output is:

[
  "aws_s3_bucket_acl.site: ACL 'public-read' makes the bucket publicly readable",
  "aws_s3_bucket_public_access_block.reports: block_public_policy must be true"
]

Each message names the resource and the setting to fix, which is what keeps developers from filing a ticket.

Reading the rule in plain words

The package line names the rule set, and the two collections at the top are data: the ACL values that count as public (including authenticated-read, which grants read access to any AWS account holder and which S3 Block Public Access also treats as public), and the four switches of S3 Block Public Access (an AWS setting that stops a bucket from being made public) that must all be on. changed(rc) is a small test that is true when a resource change creates or updates something. Each deny contains msg if { ... } block reads as "add a message to the deny list if every line inside the braces is true". Inside, some rc in input.resource_changes means "look at each change in the plan in turn", and the lines after it filter that change down. The first block keeps only ACL resources being created or updated whose acl value is in the public set. The second keeps only public-access-block resources and, for each of the four switches, checks whether it is anything other than true (rc.change.after.flag is the value the setting will have once the plan is applied). msg := sprintf(...) builds the text. In the example input, site trips the first block because its ACL is public-read, and reports trips the second because block_public_policy is false, while its other three switches are on and produce nothing. An empty deny list means the plan passes.

Not a bottleneck

• The rule runs on the plan JSON inside the existing CI job, as a local function over a file, with no network service in the path.
• Start in warn mode, fix false positives, then enforce, and announce the enforce date.
• Offer a compliant-by-default Terraform module so most teams never trigger the rule.
• Exceptions are a reviewed file in the repo with an owner and an expiry date.

What it does not cover

Policy as code on the plan only sees what Terraform manages. A bucket created by hand never appears in a plan, which is why the account guardrail and the runtime scan exist.

In CI versus at runtime

CI classes: public exposure, missing encryption, mandatory tags, banned regions. Runtime or admission classes (Kubernetes admission is the moment the cluster accepts or rejects a new object): privileged containers, images from unapproved registries, and console-made changes. Tools to know: Open Policy Agent with Rego (used above and usable across Terraform, Kubernetes and APIs), Checkov (a scanner that ships with a large library of ready-made rules for Terraform and other infrastructure files), HashiCorp Sentinel (policy as code built into HashiCorp's commercial products), and, for Kubernetes clusters only, Kyverno and OPA Gatekeeper.

Direct answer

Treat the approved baseline as code in version control, compare reality to it on a schedule and on every change, and split findings by blast radius (how much could break, and for how many users, if the automatic fix is wrong): auto-revert the ones that are reversible and clearly wrong, ticket the ones that could break something or need a decision. Every detection, action and exception is logged, since that log is the audit evidence.

1. Establish the baseline

Define approved settings per system class (hardened server image, database, storage, network rules), based on an accepted benchmark such as a CIS Benchmark (a hardening checklist published by the Center for Internet Security, with recommended settings for operating systems, clouds and databases) where one exists, and keep them in git. Baseline changes go through review, so the baseline itself has a history.

2. Detect drift in three places

• Before it ships (pre-commit and CI): check infrastructure-as-code against the same baseline. Policy checks run on every pull request.
• Against reality: terraform plan -detailed-exitcode exits 0 for no changes, 1 for error and 2 when the plan differs from real state, which makes a scheduled drift job simple. terraform plan compares three things: the code, Terraform's state (its record of the resources it created) and what actually exists in the cloud, then lists what it would change to make reality match the code. Run on a schedule against code that has not changed, any difference it reports is drift rather than a pending change. Cloud inventory and configuration services cover resources not under code. On-premises hosts need an agent or configuration-management tool run in check mode (a dry run that reports what it would change without changing anything).
• Across many repos and teams: publish the baseline as a shared rule library and make the pipeline template mandatory through organisation-level required checks (a setting on the code-hosting platform that makes chosen checks mandatory before a merge in every repository), so a new team inherits enforcement, not a to-do list.

3. Decide: auto-remediate or ticket

Auto-remediate only if the change is reversible, the correct state is unambiguous, the blast radius is small and the fix has been tested. Ticket if it touches stateful or application-specific settings, could cause an outage, or may be an approved exception. Always fix the code too (the code is the source of truth, the one place the approved state is defined), or the next apply reintroduces the drift.

Worked example (code run, output pasted)

import json

baseline = {
    "bucket-logs":  {"encryption": "aws:kms", "public_access_block": True, "versioning": True},
    "sg-web":       {"ingress_ports": [443], "ssh_open_to_world": False},
    "db-prod":      {"storage_encrypted": True, "backup_retention_days": 7},
}
observed = {
    "bucket-logs":  {"encryption": "aws:kms", "public_access_block": False, "versioning": True},
    "sg-web":       {"ingress_ports": [443, 22], "ssh_open_to_world": True},
    "db-prod":      {"storage_encrypted": True, "backup_retention_days": 3},
}
AUTO_FIX = {("bucket-logs", "public_access_block"), ("sg-web", "ssh_open_to_world")}

def diff(base, obs):
    for res, want in base.items():
        for key, expected in want.items():
            actual = obs[res][key]
            if actual != expected:
                yield res, key, expected, actual

for res, key, expected, actual in diff(baseline, observed):
    action = "AUTO-REMEDIATE" if (res, key) in AUTO_FIX else "TICKET"
    print(json.dumps({"resource": res, "setting": key, "expected": expected,
                      "actual": actual, "action": action}))

Output:

{"resource": "bucket-logs", "setting": "public_access_block", "expected": true, "actual": false, "action": "AUTO-REMEDIATE"}
{"resource": "sg-web", "setting": "ingress_ports", "expected": [443], "actual": [443, 22], "action": "TICKET"}
{"resource": "sg-web", "setting": "ssh_open_to_world", "expected": false, "actual": true, "action": "AUTO-REMEDIATE"}
{"resource": "db-prod", "setting": "backup_retention_days", "expected": 7, "actual": 3, "action": "TICKET"}

Reading the code: baseline is the approved state and observed is what a scan found, each a set of settings per resource. AUTO_FIX is the short allow-list of (resource, setting) pairs that are safe to revert without a person. diff walks every approved setting, compares it with the observed value, and yields a record only where they differ. The loop at the bottom labels each difference AUTO-REMEDIATE if it is on the allow-list and TICKET otherwise, so anything not explicitly judged safe goes to a human by default.

Four drifts, two auto-fixed, two ticketed. Closing SSH to the world is revert-safe, but an extra open port may be something an application needs, so a person decides. Shortened backup retention is a data-protection decision, not a safe revert.

4. Evidence for the auditor

Keep the baseline version, each scan result with timestamp, each remediation (who or what, when, before and after values), exceptions with approver and expiry, and time to remediate by severity. Show the drift rate trending down, not a flat zero.

Pitfalls

• Auto-remediation fighting the deployment tool in a loop. Fix the source of truth first.
• Remediating with no notification: owners should learn their change was reverted.
• A baseline nobody updates trains people to ignore drift reports.

Direct answer

I would treat the policy layer as a production dependency with its own reliability target, not as a script. Two terms matter throughout. When a policy check is broken or unreachable it must still return something, and the two choices have names: fail closed means deny the change when the checker cannot answer (safe but it blocks work), and fail open means allow it (work continues but an unchecked change gets through). CI (continuous integration, the automated build and test step) does the full evaluation in-process against a versioned policy bundle, so there is no live policy service to take builds down. Admission (the checkpoint inside a Kubernetes cluster, the system that schedules containers across servers, where every request to create or change something is accepted or rejected; it is what blocks pods, the running units of containers) re-checks only the small set of rules that matter if CI is bypassed, using a cached last-known-good bundle (the versioned package of rules and reference data). When the policy layer fails, a short list of severe rules fails closed and everything else fails open with an alert and a follow-up scan. Exceptions are data with expiry dates, not code changes.

Where policies are evaluated

To avoid evaluating the same thing twice, CI can sign an attestation (a signed statement that this artifact passed these policies) and admission verifies the signature for the common path. Kubernetes ValidatingAdmissionPolicy, a Kubernetes feature that runs rule expressions inside the API server (the cluster's front door) with no external webhook and is stable since Kubernetes 1.30, covers rules that fit its expression language and removes a network dependency for them.

What happens when the policy layer fails

Last month's outage was a single central service that every deploy depended on, all rules failing closed. The fix is to separate rules by consequence:

• Tier 1, fail closed: internet-exposed resources, privileged containers, public data. Evaluated from the cached bundle in-process, so a policy-service outage does not trigger the failure case at all. If even that fails, deny.
• Tier 2, fail open with a compensating control: everything else. Allow, alert, and run a detective scan over everything admitted during the outage, with a stated window to remediate.
• In Kubernetes, failurePolicy is the per-webhook setting that says what the cluster does when the webhook cannot answer: Fail rejects the request (fail closed) and Ignore lets it through (fail open). It defaults to Fail, so use two webhook configurations, one per tier.
• Exclude the policy engine's own namespace (a namespace is a named partition of a cluster) and system namespaces so the engine cannot block its own recovery.
• Set a short explicit timeout so a slow engine cannot stall every deploy.
• Break glass (an emergency bypass kept for incidents): a named role can bypass, which opens an incident ticket automatically, expires on its own and is logged as evidence.

A worked example, with illustrative numbers. The central policy service goes down for 20 minutes. Developer A deploys a service whose pod asks for privileged mode (root-level access to the host server). That rule is Tier 1, evaluated in-process from the cached bundle, so no call to the dead service is needed: the pod is denied, correctly. Developer B deploys a service that is missing its owner label. That rule is Tier 2, checked through the webhook set to Ignore. The call times out after the short timeout, the cluster allows the pod, and an alert fires saying "policy service unreachable, rules skipped". Say 14 deploys were admitted unchecked during the window. After recovery, a scan over those 14 finds one missing owner label, a ticket opens with a 48-hour window to fix it, and nothing severe went through unchecked. Under last month's design, A and B would both have been blocked, along with every other deploy and every pod reschedule.

An admission webhook that fails closed also blocks pods being rescheduled or scaled, so the policy layer can hurt running production, not just deploys. That is the reason for tiers and a cache.

Exceptions

An exception is a data record keyed by resource and rule, with owner, reason, approver and expiry, shipped inside the policy bundle. The policy evaluates it, so the expiry is enforced by the same mechanism and no code redeploy is needed.

Reliability targets for the layer itself

Because the layer gates deploys and, at admission, pod scheduling, its availability must exceed that of what it protects. Arithmetic: 99.9% availability allows 43.2 minutes down in 30 days. Two 99.9% components in series (a request needs both to be up, so the availabilities multiply) give 0.999 x 0.999 = 99.8001%, about 86.4 minutes. A policy layer at 99.95% allows 21.6 minutes, and in series with a 99.9% platform gives 99.85%. These targets are illustrative. Set yours from your own deploy and scaling volume. Also measure the following. Evaluation errors and bundle freshness show the layer is unhealthy before developers notice, so alert on those first; the others tune quality over time.

• Policy-change failure rate: roll out new rules in audit mode first and by staged namespaces.
• Evaluation errors counted separately from denials.
• Bundle freshness, meaning the age of the policy version in use.
• Decision-log completeness (the decision log is the record of every allow and deny, so a gap in it is a gap in evidence).
• Developer overrides per 1,000 evaluations, as a false-positive signal.

Slow checks

Evaluate only changed resources, push reference data into the bundle instead of calling out during evaluation, run policy in parallel with tests, and report per-rule cost so the expensive rules get fixed or removed.

Remediation and evidence

Auto-remediate only reversible, low-risk findings (for example adding a missing tag), preferably by opening a pull request. Destructive fixes stay with a human. Every decision is logged with the policy version and an input hash, which gives auditors an evidence feed with no extra work.

Direct answer

A green dashboard usually proves a setting exists, not that the control works. I would add automated, scheduled probes that try to do the thing the control is supposed to stop (or notice), in staging and in production-like environments, measure whether the control reacted, and treat a failed probe as a control failure with an owner, a deadline and a retest. Dashboards would then show two things per control: configured, and last proven effective on a date.

Config present versus control effective

A security group rule, an endpoint agent installed, a log forwarder enabled: these are configuration. Effectiveness is the outcome. Example: the EDR (endpoint detection and response, software on laptops and servers that watches for malicious activity and raises alerts) agent is installed on every host, yet a policy exclusion means it never alerts on a harmless test file.

What to test (one probe per control family)

The table labels each control by type. A preventive control stops the bad thing from happening, a detective control notices it after it happens and raises an alert, and a corrective control repairs the damage or restores service.

Detection checks matter most for SIEM and EDR: measure the detection rate (alerts raised divided by probes sent, so 97 alerts from 100 probes is 97%) and the time from probe to alert.

A concrete logging probe, with illustrative timings. At 10:00:00 a scheduled job writes one event containing a unique ID, for example probe-7f3a, on a canary host, and records the send time. It then searches the SIEM for that ID every 30 seconds. If a poll finds the event at 10:03:30 the measured delay is 210 seconds (the polls land on 30-second marks, so the delay is accurate to within 30 seconds); against an agreed limit of 300 seconds that is a pass, and the delay is stored as a data point. If the ID is still absent at 10:05:00 the probe fails, a ticket opens, and the same job reruns after the fix to prove it. The same pattern works for the EDR row: the job drops the test file, then looks for the alert.

Scale and production safety

• Very large fleets: do not probe every host. Hosts built from the same image and policy share the same control behaviour, so stratify (split the fleet into groups that behave alike and sample from each group): a handful of canary hosts per image and configuration group, where a canary host is a representative machine you deliberately probe. For 100,000 hosts built from 40 image and configuration combinations, 10 canaries each is 400 probes per cycle, not 100,000.
• Avoid production impact: use harmless artifacts, dedicated test tenants (separate customer or organisation spaces used only for testing) and accounts, a rate limit, an announced window, and a kill switch (one button or command that stops all probing immediately). Tag probe traffic so the SOC (security operations centre) can recognise it and not escalate, while a separate check proves the SOC would have seen it.
• Deeper security testing (exploitation, red-team technique) is a separate discipline; this pipeline only checks that the specific controls respond.

Acting on a failed validation

1. Open a ticket automatically with the failing probe, environment and evidence.
2. Classify severity by the control's importance, not by how annoying the fix is.
3. Assign the control owner, set a deadline, and apply a compensating control (an alternative safeguard in the meantime, such as extra monitoring or a manual check) if the fix takes long.
4. Fix, rerun the exact probe, and store both the failure and the pass as evidence.
5. Look for the pattern: 3 misses from 100 EDR probes (97% detection) matter little in isolation, but if all three share one image, the image is the finding.

Trade-offs and pitfalls

• A probe that always passes is as useless as a dashboard that is always green. Run a deliberate failing case now and then to prove the probe can fail.
• Staging may not match production. Run a thin, safe subset in production-like or production environments.

Direct answer

Build the obligation as data and the checks as code: counsel and the privacy or compliance team translate each local legal requirement into a plain-language control, engineers turn the objectively testable ones into pipeline checks tagged by market (consumer-protection law covers things like fair advertising, pricing display and refunds), and the judgement-heavy ones become a documented human approval. Adding a market then means adding a configuration file, not designing a new pipeline.

Step 1: A single obligation register

For each requirement keep: jurisdiction, source citation and effective date, the internal control statement in plain English, an owner, how it is tested (automated, manual or both) and the status. Lawyers interpret the law; engineers do not. You are translating counsel's interpretation into tests, and recording who decided.

Step 2: Sort every requirement into three tiers

Step 3: Structure for scale

• A core rule library (one definition per rule) plus a per-market manifest listing which rules apply and with which parameters (allowed regions, retention limits, notice locale).
• The pipeline reads the manifest of every market a release targets and runs only those rules. New market means a new manifest, plus new rules only where law differs.
• Every rule has fixture tests (small saved examples, one that must pass and one that must fail, which the rule is run against whenever it changes), so changing a rule is safe.
• A central platform team owns the engine; product teams self-serve through manifests. A shared template means 50 teams do not each build their own checks.

What the register and manifest look like

One register row, with illustrative values:

One market manifest, a small configuration file the pipeline reads (values illustrative):

market: DE
rules:
  - id: consent-before-tracking
    params: { max_tracker_requests_before_consent: 0, banner_locale: de }
  - id: data-region
    params: { allowed_regions: [eu-central-1, eu-west-1] }
  - id: privacy-notice-version
    params: { locale: de, min_version: 4 }
  - id: dpia-linked
    params: { required_for: high_risk_features }

The rules themselves live once in the core library; the manifest only chooses them and sets their parameters. A fixture test for consent-before-tracking would hold two saved page loads: one that fires 0 tracker requests before consent (must pass) and one that fires 2 (must fail). Adding another market means copying a manifest and changing its values, and writing a new rule only where that market's law genuinely differs.

Step 4: Change in law

Counsel or a regulatory-monitoring service raises changes; each becomes a register entry with an effective date and an owner. Updated rules are versioned and switched on at the effective date, with a warning period where the check reports but does not block. Release gates evaluate rules in force on the release date.

Where automation stops

Interpretation, balancing tests (weighing the company's interest in using data against the individual's rights) and legal risk acceptance stay with counsel. The pipeline proves that required artifacts and technical conditions exist; it does not prove the law is satisfied. Keep human approval narrow: ask only for the tier-three items, so the gate is not a queue for everything.

Worked example

A team launches in a new EU market: the manifest enables four checks (consent-before-tracking, data-region, privacy-notice-version and dpia-linked). The first three pass. The dpia-linked check passes on presence, because a DPIA exists and is linked, but the feature uses a new data category the DPIA did not cover, so the register flags legal approval. The release proceeds when legal signs. Nothing was hand-built for this market.

Pitfalls

• Treating a green pipeline as legal clearance.
• Rules that only counsel can read; keep the plain-language statement next to each test.
• Stale rules after a law changes, which is why each rule carries a review date.