Compliance Automation and Tooling Questions

Direct answer

Treat the approved baseline as code in version control, compare reality to it on a schedule and on every change, and split findings by blast radius (how much could break, and for how many users, if the automatic fix is wrong): auto-revert the ones that are reversible and clearly wrong, ticket the ones that could break something or need a decision. Every detection, action and exception is logged, since that log is the audit evidence.

1. Establish the baseline

Define approved settings per system class (hardened server image, database, storage, network rules), based on an accepted benchmark such as a CIS Benchmark (a hardening checklist published by the Center for Internet Security, with recommended settings for operating systems, clouds and databases) where one exists, and keep them in git. Baseline changes go through review, so the baseline itself has a history.

2. Detect drift in three places

• Before it ships (pre-commit and CI): check infrastructure-as-code against the same baseline. Policy checks run on every pull request.
• Against reality: terraform plan -detailed-exitcode exits 0 for no changes, 1 for error and 2 when the plan differs from real state, which makes a scheduled drift job simple. terraform plan compares three things: the code, Terraform's state (its record of the resources it created) and what actually exists in the cloud, then lists what it would change to make reality match the code. Run on a schedule against code that has not changed, any difference it reports is drift rather than a pending change. Cloud inventory and configuration services cover resources not under code. On-premises hosts need an agent or configuration-management tool run in check mode (a dry run that reports what it would change without changing anything).
• Across many repos and teams: publish the baseline as a shared rule library and make the pipeline template mandatory through organisation-level required checks (a setting on the code-hosting platform that makes chosen checks mandatory before a merge in every repository), so a new team inherits enforcement, not a to-do list.

3. Decide: auto-remediate or ticket

Auto-remediate only if the change is reversible, the correct state is unambiguous, the blast radius is small and the fix has been tested. Ticket if it touches stateful or application-specific settings, could cause an outage, or may be an approved exception. Always fix the code too (the code is the source of truth, the one place the approved state is defined), or the next apply reintroduces the drift.

Worked example (code run, output pasted)

import json

baseline = {
    "bucket-logs":  {"encryption": "aws:kms", "public_access_block": True, "versioning": True},
    "sg-web":       {"ingress_ports": [443], "ssh_open_to_world": False},
    "db-prod":      {"storage_encrypted": True, "backup_retention_days": 7},
}
observed = {
    "bucket-logs":  {"encryption": "aws:kms", "public_access_block": False, "versioning": True},
    "sg-web":       {"ingress_ports": [443, 22], "ssh_open_to_world": True},
    "db-prod":      {"storage_encrypted": True, "backup_retention_days": 3},
}
AUTO_FIX = {("bucket-logs", "public_access_block"), ("sg-web", "ssh_open_to_world")}

def diff(base, obs):
    for res, want in base.items():
        for key, expected in want.items():
            actual = obs[res][key]
            if actual != expected:
                yield res, key, expected, actual

for res, key, expected, actual in diff(baseline, observed):
    action = "AUTO-REMEDIATE" if (res, key) in AUTO_FIX else "TICKET"
    print(json.dumps({"resource": res, "setting": key, "expected": expected,
                      "actual": actual, "action": action}))

Output:

{"resource": "bucket-logs", "setting": "public_access_block", "expected": true, "actual": false, "action": "AUTO-REMEDIATE"}
{"resource": "sg-web", "setting": "ingress_ports", "expected": [443], "actual": [443, 22], "action": "TICKET"}
{"resource": "sg-web", "setting": "ssh_open_to_world", "expected": false, "actual": true, "action": "AUTO-REMEDIATE"}
{"resource": "db-prod", "setting": "backup_retention_days", "expected": 7, "actual": 3, "action": "TICKET"}

Reading the code: baseline is the approved state and observed is what a scan found, each a set of settings per resource. AUTO_FIX is the short allow-list of (resource, setting) pairs that are safe to revert without a person. diff walks every approved setting, compares it with the observed value, and yields a record only where they differ. The loop at the bottom labels each difference AUTO-REMEDIATE if it is on the allow-list and TICKET otherwise, so anything not explicitly judged safe goes to a human by default.

Four drifts, two auto-fixed, two ticketed. Closing SSH to the world is revert-safe, but an extra open port may be something an application needs, so a person decides. Shortened backup retention is a data-protection decision, not a safe revert.

4. Evidence for the auditor

Keep the baseline version, each scan result with timestamp, each remediation (who or what, when, before and after values), exceptions with approver and expiry, and time to remediate by severity. Show the drift rate trending down, not a flat zero.

Pitfalls

• Auto-remediation fighting the deployment tool in a loop. Fix the source of truth first.
• Remediating with no notification: owners should learn their change was reverted.
• A baseline nobody updates trains people to ignore drift reports.

Direct answer

The four frameworks are GDPR (the EU General Data Protection Regulation), HIPAA (the US health-data law), PCI DSS (the Payment Card Industry Data Security Standard) and ISO 27001 (the international information security management standard). I would not build four sets of checks. I would build one library of machine-checkable tests that measure facts (for example, "which human users lack multi-factor authentication"), plus a versioned mapping file that links each test to the controls it supports in each framework. Continuous control monitoring (CCM) means running those tests repeatedly and alerting on change, not once a year. The mapping, not the test, carries framework differences, so one test can serve several frameworks while each framework's owner still decides what "satisfied" means.

One test, several frameworks (illustrative mapping for "MFA enforced for all human users")

The table has a row for each of the four frameworks in the question, plus a fifth row for SOC 2 (the AICPA report on a company's security controls) to show that a new framework is one more row in the mapping, not one more test.

The test outputs a measured value (such as "0 of 412 human users lack MFA") and a scope tag. The mapping says which framework control it supports. I would never state that a passing test equals framework compliance. The citations show how one result attaches to each framework; the pattern to carry is a measured fact, a scope, and a caveat, not recall of every clause number.

A traced example (illustrative numbers). The test mfa-enforced reads the normalized identities from all three clouds: 412 human users, of which 3 have no MFA, 2 of those 3 tagged as CDE accounts. It emits {"test": "mfa-enforced", "evaluated": 412, "failed": 3, "failed_in_cde": 2}. The mapping file lists mfa-enforced under ISO A.8.5, PCI DSS 8.4.2 (counting only failed_in_cde), HIPAA 164.312(d) as supporting evidence, SOC 2 CC6.1 and GDPR Article 32 as supporting evidence. So the one result fails PCI 8.4.2 for 2 accounts and shows 3 exceptions for the ISO and SOC 2 controls, with no second test needed.

Architecture

1. Collectors per cloud and per region read from AWS, Azure and GCP, using their native config and policy services as inputs.
2. A normalizer (a translation layer) turns each cloud's different resource shapes into one model (an identity, a storage resource, a network rule). For example, an AWS IAM user, an Azure Entra user and a GCP account all become {kind: identity, cloud, region, human: true, mfa: true or false, scope_tags: [...]}.
3. The test library evaluates the model. Each result is tri-state (three possible outcomes): pass, fail, or error/unknown.
4. The mapping file connects tests to controls many-to-many.
5. A results store feeds a GRC (governance, risk and compliance) platform and the auditor view. Control owners see their own failures.

Keeping false passes down

• Never treat error or unknown as a pass.
• Measure coverage: resources evaluated divided by resources found by an independent inventory. A test that passes on 80% of the estate is not a pass.
• Write a negative test for every check: a seeded violation that the check must fail.
• Keep a canary in each cloud that must be flagged daily.
• Reconcile periodically with a human spot check.

Keeping false alarms down

Route to the owning team, deduplicate by resource, allow eventual-consistency grace periods (cloud APIs can take minutes to report a new resource's settings, so a brand-new resource can look noncompliant briefly) before alerting on a just-created resource, give suppressions an expiry, and measure alert precision from triage labels.

Data residency for the evidence

Evidence can contain personal data (user emails in an access list). Collect and store it in the region where it originates, and send only hashes, counts and pass/fail results to the central index. A hash of an email address is pseudonymised, not anonymous, so whether even hashes may leave the region is part of counsel's call; counts and pass/fail results carry no personal data. GDPR Chapter V (Articles 44 to 50) restricts transfers of personal data to third countries (countries outside the EU and EEA), so whether a transfer is lawful is counsel's call, but data minimization (collecting and keeping only what is needed) in the design means most of the evidence never needs to leave. Store configuration values, never PHI or card numbers.

Divergent regional interpretations

The test stays the same. A regional overlay changes the parameters and the scope, such as a longer retention threshold in one jurisdiction or evidence kept in-region, and the divergence is recorded in the mapping. Illustratively, a test log-retention-months >= threshold uses a threshold of 12 in the baseline mapping, and an overlay for one jurisdiction sets 24. The same test then passes a log store set to 12 months under the baseline mapping and fails that same 12-month store under the overlay, with no second test written.

Scoring and integration

Report coverage (illustratively 70 of 120 in-scope controls automated, 58.3%) separately from pass rate (66 of those 70 passing, 94.3%). A single blended score hides which one moved.

Direct answer

Policy as code means writing a rule such as "no storage bucket may be publicly readable" as a small program kept in version control, instead of a wiki page. A machine evaluates every proposed infrastructure change against the rule and returns allow or deny with a reason. For "can never reach production", one check is not enough: you stack a plan-time check in CI (continuous integration, the automated build and test pipeline), an account-level guardrail that cannot be bypassed, and a runtime scan that catches drift (a live setting changing away from what the code says, for example someone editing a bucket by hand in the console).

Why it helps compliance

• Rules are versioned, so the history shows who changed which rule and when.
• Changes go through pull-request review, like application code.
• Rules are unit-testable, so you can prove the rule denies what it should.
• The same rule produces the same answer every run, which makes it usable as audit evidence.

Three layers, and which one gives the "never"

The account-level guardrail is what actually makes "never" true. The CI rule is the fast, friendly feedback loop on top of it.

A plan-time rule (Rego, the language used by Open Policy Agent)

package terraform.s3public

import rego.v1

public_acls := {"public-read", "public-read-write", "authenticated-read"}

block_flags := ["block_public_acls", "block_public_policy", "ignore_public_acls", "restrict_public_buckets"]

changed(rc) if {
	some action in rc.change.actions
	action in {"create", "update"}
}

deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_s3_bucket_acl"
	changed(rc)
	rc.change.after.acl in public_acls
	msg := sprintf("%s: ACL '%s' makes the bucket publicly readable", [rc.address, rc.change.after.acl])
}

deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_s3_bucket_public_access_block"
	changed(rc)
	some flag in block_flags
	rc.change.after[flag] != true
	msg := sprintf("%s: %s must be true", [rc.address, flag])
}

Given a plan excerpt with one public ACL and one half-open public-access block:

opa eval -f pretty -d s3_public.rego -i input.json 'data.terraform.s3public.deny'

with this input.json:

{"resource_changes":[{"address":"aws_s3_bucket_acl.site","type":"aws_s3_bucket_acl","change":{"actions":["create"],"after":{"acl":"public-read"}}},{"address":"aws_s3_bucket_public_access_block.reports","type":"aws_s3_bucket_public_access_block","change":{"actions":["create"],"after":{"block_public_acls":true,"block_public_policy":false,"ignore_public_acls":true,"restrict_public_buckets":true}}}]}

the real output is:

[
  "aws_s3_bucket_acl.site: ACL 'public-read' makes the bucket publicly readable",
  "aws_s3_bucket_public_access_block.reports: block_public_policy must be true"
]

Each message names the resource and the setting to fix, which is what keeps developers from filing a ticket.

Reading the rule in plain words

The package line names the rule set, and the two collections at the top are data: the ACL values that count as public (including authenticated-read, which grants read access to any AWS account holder and which S3 Block Public Access also treats as public), and the four switches of S3 Block Public Access (an AWS setting that stops a bucket from being made public) that must all be on. changed(rc) is a small test that is true when a resource change creates or updates something. Each deny contains msg if { ... } block reads as "add a message to the deny list if every line inside the braces is true". Inside, some rc in input.resource_changes means "look at each change in the plan in turn", and the lines after it filter that change down. The first block keeps only ACL resources being created or updated whose acl value is in the public set. The second keeps only public-access-block resources and, for each of the four switches, checks whether it is anything other than true (rc.change.after.flag is the value the setting will have once the plan is applied). msg := sprintf(...) builds the text. In the example input, site trips the first block because its ACL is public-read, and reports trips the second because block_public_policy is false, while its other three switches are on and produce nothing. An empty deny list means the plan passes.

Not a bottleneck

• The rule runs on the plan JSON inside the existing CI job, as a local function over a file, with no network service in the path.
• Start in warn mode, fix false positives, then enforce, and announce the enforce date.
• Offer a compliant-by-default Terraform module so most teams never trigger the rule.
• Exceptions are a reviewed file in the repo with an owner and an expiry date.

What it does not cover

Policy as code on the plan only sees what Terraform manages. A bucket created by hand never appears in a plan, which is why the account guardrail and the runtime scan exist.

In CI versus at runtime

CI classes: public exposure, missing encryption, mandatory tags, banned regions. Runtime or admission classes (Kubernetes admission is the moment the cluster accepts or rejects a new object): privileged containers, images from unapproved registries, and console-made changes. Tools to know: Open Policy Agent with Rego (used above and usable across Terraform, Kubernetes and APIs), Checkov (a scanner that ships with a large library of ready-made rules for Terraform and other infrastructure files), HashiCorp Sentinel (policy as code built into HashiCorp's commercial products), and, for Kubernetes clusters only, Kyverno and OPA Gatekeeper.

Direct answer

An annual audit is a snapshot: it checks a sample of evidence from a window and passes you if what it looked at was in order. It says little about the days it did not look at. Continuous compliance replaces the occasional look with automated checks that run all the time, so a problem is found in hours or days, not at the next audit.

An analogy for a newer engineer

A yearly car inspection can pass a car whose brake warning light came on in March. Continuous compliance is the warning light: it tells you when the problem starts. The inspection still matters, because it checks that the light itself works.

Why point-in-time fails

• Sampling: auditors test a sample, meaning a small subset of records picked from the period, not every day of it.
• Preparation effect: teams tidy up before the audit, so the audited state is not the typical state.
• Gap: if a misconfiguration is live for 90 days, that is 90 of 365, about 24.7% of the year, with nobody watching.
• Cost of lateness: a problem found months later has had months to be exploited and is harder to trace.

What I would automate first

A baseline is the approved list of settings a system should have, so most of these checks are comparisons against it. Pick checks that are objective, frequent and costly when wrong:

1. Configuration baselining: encryption on, public exposure off, admin MFA on, logging on. The check compares live state to that baseline daily.
2. Change management evidence: every production change links to a reviewed, approved pull request, and the pipeline flags changes with no approval.
3. Immutable artifacts: build once, sign, and deploy that exact artifact; servers are replaced, not hand-edited. An artifact is the packaged build output (a container image or a program file). Signing means the build system attaches a cryptographic stamp made from the artifact's contents and a secret key, and the deploy step refuses anything whose stamp does not verify. If someone edits the file by hand, its contents change, the stamp no longer matches, and the edit becomes a detectable difference. The artifact's hash (a short fingerprint of its bytes) recorded at deploy time shows exactly what was deployed.
4. Alert health: check that logging and alerting still work, since silent failure here undermines all the others.

For example, a change record might read: change 4812, reviewed and approved by a named second person, merged at 14:02, artifact hash recorded, deployed at 14:09 (illustrative). For alert health, a daily test event must appear in the alerting system within a set time, and the record shows the last pass. These signals double as audit evidence: baseline scan results, change records with approvals and artifact hashes are exactly what an auditor asks for, already timestamped.

Worked example

The annual audit passed in January. A storage bucket became public in March and stayed so until July. A daily baseline check would have failed on day one in March, opened a ticket, and the fix would be visible as evidence. The July discovery would not exist.

Caveats

Automation covers what can be measured; policies, training and judgement calls still need people. Start with a handful of checks that matter, not a hundred that nobody reads.