Compliance Automation and Tooling Questions

Direct answer

If the budget allows one placement, I would start with checks against the live cloud configuration, because it shows what actually exists and is the strongest evidence of continuous compliance. I would add pre-deploy checks next, driven by the same rules, because they stop problems cheaply, and use runtime and log signals only for the controls the other two cannot see. The three placements are not alternatives to rank; they answer different questions.

The three placements

Read the first row of the table first: each placement answers a different question, and the rows below it say where that placement is blind. Infrastructure-as-code means describing cloud resources in files (Terraform is a common tool) that a pipeline applies, instead of clicking in a console.

Deciding where to put the first check

1. What share of infrastructure is created through code? Suppose 400 of 500 buckets come from code (80%). Pre-deploy checks can then see 400 of them and miss the other 100, which only a live scan reaches; the higher the share, the more of the problem pre-deploy catches.
2. Is there drift or manual change today? If yes, live configuration is mandatory, since pre-deploy cannot see it.
3. Is the control about configuration or behaviour? Encryption and public exposure are configuration; "admins only use MFA" and "detections fire" are behavioural.
4. What does the auditor accept as evidence? Live state with timestamps is the easiest to defend.

Combining them with one rule set

Write each rule once, tag it with the control it supports, and run it at two points. Disagreement is signal: a pass at deploy and a fail live means drift or an out-of-band change, so reconcile results by control and resource. Reserve runtime signals for a short list (privileged access use, detection of the control being disabled).

Worked example: a public storage bucket

• Pre-deploy: the plan shows aws_s3_bucket_acl.site with acl = "public-read". A rule like ACL 'public-read' makes the bucket publicly readable fails the pipeline step, so the merge is blocked (or passes it with a note if the account-level block applies).
• Live: a console user flips another bucket public at 3pm. The AWS Config rule S3_BUCKET_PUBLIC_READ_PROHIBITED (it checks Block Public Access, the bucket policy and the ACL) re-evaluates on the configuration change and marks that bucket NON_COMPLIANT within minutes; a periodic scan would catch it at the next run.
• Runtime: the bucket's access logs show GET requests where the requester field is - (an unauthenticated request) with operation REST.GET.OBJECT, which turns a misconfiguration into a possible incident.

What would change the call

Mostly code-driven shop with strong review: pre-deploy first. Heavy regulatory detection requirement: raise runtime priority. Very small team: a managed live-posture service (a cloud or vendor product, such as AWS Security Hub or Microsoft Defender for Cloud, that runs configuration checks against your live accounts for you) with a short rule list beats a custom stack.

Direct answer

An annual audit is a snapshot: it checks a sample of evidence from a window and passes you if what it looked at was in order. It says little about the days it did not look at. Continuous compliance replaces the occasional look with automated checks that run all the time, so a problem is found in hours or days, not at the next audit.

An analogy for a newer engineer

A yearly car inspection can pass a car whose brake warning light came on in March. Continuous compliance is the warning light: it tells you when the problem starts. The inspection still matters, because it checks that the light itself works.

Why point-in-time fails

• Sampling: auditors test a sample, meaning a small subset of records picked from the period, not every day of it.
• Preparation effect: teams tidy up before the audit, so the audited state is not the typical state.
• Gap: if a misconfiguration is live for 90 days, that is 90 of 365, about 24.7% of the year, with nobody watching.
• Cost of lateness: a problem found months later has had months to be exploited and is harder to trace.

What I would automate first

A baseline is the approved list of settings a system should have, so most of these checks are comparisons against it. Pick checks that are objective, frequent and costly when wrong:

1. Configuration baselining: encryption on, public exposure off, admin MFA on, logging on. The check compares live state to that baseline daily.
2. Change management evidence: every production change links to a reviewed, approved pull request, and the pipeline flags changes with no approval.
3. Immutable artifacts: build once, sign, and deploy that exact artifact; servers are replaced, not hand-edited. An artifact is the packaged build output (a container image or a program file). Signing means the build system attaches a cryptographic stamp made from the artifact's contents and a secret key, and the deploy step refuses anything whose stamp does not verify. If someone edits the file by hand, its contents change, the stamp no longer matches, and the edit becomes a detectable difference. The artifact's hash (a short fingerprint of its bytes) recorded at deploy time shows exactly what was deployed.
4. Alert health: check that logging and alerting still work, since silent failure here undermines all the others.

For example, a change record might read: change 4812, reviewed and approved by a named second person, merged at 14:02, artifact hash recorded, deployed at 14:09 (illustrative). For alert health, a daily test event must appear in the alerting system within a set time, and the record shows the last pass. These signals double as audit evidence: baseline scan results, change records with approvals and artifact hashes are exactly what an auditor asks for, already timestamped.

Worked example

The annual audit passed in January. A storage bucket became public in March and stayed so until July. A daily baseline check would have failed on day one in March, opened a ticket, and the fix would be visible as evidence. The July discovery would not exist.

Caveats

Automation covers what can be measured; policies, training and judgement calls still need people. Start with a handful of checks that matter, not a hundred that nobody reads.

Direct answer

Policy as code means writing a rule such as "no storage bucket may be publicly readable" as a small program kept in version control, instead of a wiki page. A machine evaluates every proposed infrastructure change against the rule and returns allow or deny with a reason. For "can never reach production", one check is not enough: you stack a plan-time check in CI (continuous integration, the automated build and test pipeline), an account-level guardrail that cannot be bypassed, and a runtime scan that catches drift (a live setting changing away from what the code says, for example someone editing a bucket by hand in the console).

Why it helps compliance

• Rules are versioned, so the history shows who changed which rule and when.
• Changes go through pull-request review, like application code.
• Rules are unit-testable, so you can prove the rule denies what it should.
• The same rule produces the same answer every run, which makes it usable as audit evidence.

Three layers, and which one gives the "never"

The account-level guardrail is what actually makes "never" true. The CI rule is the fast, friendly feedback loop on top of it.

A plan-time rule (Rego, the language used by Open Policy Agent)

package terraform.s3public

import rego.v1

public_acls := {"public-read", "public-read-write", "authenticated-read"}

block_flags := ["block_public_acls", "block_public_policy", "ignore_public_acls", "restrict_public_buckets"]

changed(rc) if {
	some action in rc.change.actions
	action in {"create", "update"}
}

deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_s3_bucket_acl"
	changed(rc)
	rc.change.after.acl in public_acls
	msg := sprintf("%s: ACL '%s' makes the bucket publicly readable", [rc.address, rc.change.after.acl])
}

deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_s3_bucket_public_access_block"
	changed(rc)
	some flag in block_flags
	rc.change.after[flag] != true
	msg := sprintf("%s: %s must be true", [rc.address, flag])
}

Given a plan excerpt with one public ACL and one half-open public-access block:

opa eval -f pretty -d s3_public.rego -i input.json 'data.terraform.s3public.deny'

with this input.json:

{"resource_changes":[{"address":"aws_s3_bucket_acl.site","type":"aws_s3_bucket_acl","change":{"actions":["create"],"after":{"acl":"public-read"}}},{"address":"aws_s3_bucket_public_access_block.reports","type":"aws_s3_bucket_public_access_block","change":{"actions":["create"],"after":{"block_public_acls":true,"block_public_policy":false,"ignore_public_acls":true,"restrict_public_buckets":true}}}]}

the real output is:

[
  "aws_s3_bucket_acl.site: ACL 'public-read' makes the bucket publicly readable",
  "aws_s3_bucket_public_access_block.reports: block_public_policy must be true"
]

Each message names the resource and the setting to fix, which is what keeps developers from filing a ticket.

Reading the rule in plain words

The package line names the rule set, and the two collections at the top are data: the ACL values that count as public (including authenticated-read, which grants read access to any AWS account holder and which S3 Block Public Access also treats as public), and the four switches of S3 Block Public Access (an AWS setting that stops a bucket from being made public) that must all be on. changed(rc) is a small test that is true when a resource change creates or updates something. Each deny contains msg if { ... } block reads as "add a message to the deny list if every line inside the braces is true". Inside, some rc in input.resource_changes means "look at each change in the plan in turn", and the lines after it filter that change down. The first block keeps only ACL resources being created or updated whose acl value is in the public set. The second keeps only public-access-block resources and, for each of the four switches, checks whether it is anything other than true (rc.change.after.flag is the value the setting will have once the plan is applied). msg := sprintf(...) builds the text. In the example input, site trips the first block because its ACL is public-read, and reports trips the second because block_public_policy is false, while its other three switches are on and produce nothing. An empty deny list means the plan passes.

Not a bottleneck

• The rule runs on the plan JSON inside the existing CI job, as a local function over a file, with no network service in the path.
• Start in warn mode, fix false positives, then enforce, and announce the enforce date.
• Offer a compliant-by-default Terraform module so most teams never trigger the rule.
• Exceptions are a reviewed file in the repo with an owner and an expiry date.

What it does not cover

Policy as code on the plan only sees what Terraform manages. A bucket created by hand never appears in a plan, which is why the account guardrail and the runtime scan exist.

In CI versus at runtime

CI classes: public exposure, missing encryption, mandatory tags, banned regions. Runtime or admission classes (Kubernetes admission is the moment the cluster accepts or rejects a new object): privileged containers, images from unapproved registries, and console-made changes. Tools to know: Open Policy Agent with Rego (used above and usable across Terraform, Kubernetes and APIs), Checkov (a scanner that ships with a large library of ready-made rules for Terraform and other infrastructure files), HashiCorp Sentinel (policy as code built into HashiCorp's commercial products), and, for Kubernetes clusters only, Kyverno and OPA Gatekeeper.

Direct answer

Treat the approved baseline as code in version control, compare reality to it on a schedule and on every change, and split findings by blast radius (how much could break, and for how many users, if the automatic fix is wrong): auto-revert the ones that are reversible and clearly wrong, ticket the ones that could break something or need a decision. Every detection, action and exception is logged, since that log is the audit evidence.

1. Establish the baseline

Define approved settings per system class (hardened server image, database, storage, network rules), based on an accepted benchmark such as a CIS Benchmark (a hardening checklist published by the Center for Internet Security, with recommended settings for operating systems, clouds and databases) where one exists, and keep them in git. Baseline changes go through review, so the baseline itself has a history.

2. Detect drift in three places

• Before it ships (pre-commit and CI): check infrastructure-as-code against the same baseline. Policy checks run on every pull request.
• Against reality: terraform plan -detailed-exitcode exits 0 for no changes, 1 for error and 2 when the plan differs from real state, which makes a scheduled drift job simple. terraform plan compares three things: the code, Terraform's state (its record of the resources it created) and what actually exists in the cloud, then lists what it would change to make reality match the code. Run on a schedule against code that has not changed, any difference it reports is drift rather than a pending change. Cloud inventory and configuration services cover resources not under code. On-premises hosts need an agent or configuration-management tool run in check mode (a dry run that reports what it would change without changing anything).
• Across many repos and teams: publish the baseline as a shared rule library and make the pipeline template mandatory through organisation-level required checks (a setting on the code-hosting platform that makes chosen checks mandatory before a merge in every repository), so a new team inherits enforcement, not a to-do list.

3. Decide: auto-remediate or ticket

Auto-remediate only if the change is reversible, the correct state is unambiguous, the blast radius is small and the fix has been tested. Ticket if it touches stateful or application-specific settings, could cause an outage, or may be an approved exception. Always fix the code too (the code is the source of truth, the one place the approved state is defined), or the next apply reintroduces the drift.

Worked example (code run, output pasted)

import json

baseline = {
    "bucket-logs":  {"encryption": "aws:kms", "public_access_block": True, "versioning": True},
    "sg-web":       {"ingress_ports": [443], "ssh_open_to_world": False},
    "db-prod":      {"storage_encrypted": True, "backup_retention_days": 7},
}
observed = {
    "bucket-logs":  {"encryption": "aws:kms", "public_access_block": False, "versioning": True},
    "sg-web":       {"ingress_ports": [443, 22], "ssh_open_to_world": True},
    "db-prod":      {"storage_encrypted": True, "backup_retention_days": 3},
}
AUTO_FIX = {("bucket-logs", "public_access_block"), ("sg-web", "ssh_open_to_world")}

def diff(base, obs):
    for res, want in base.items():
        for key, expected in want.items():
            actual = obs[res][key]
            if actual != expected:
                yield res, key, expected, actual

for res, key, expected, actual in diff(baseline, observed):
    action = "AUTO-REMEDIATE" if (res, key) in AUTO_FIX else "TICKET"
    print(json.dumps({"resource": res, "setting": key, "expected": expected,
                      "actual": actual, "action": action}))

Output:

{"resource": "bucket-logs", "setting": "public_access_block", "expected": true, "actual": false, "action": "AUTO-REMEDIATE"}
{"resource": "sg-web", "setting": "ingress_ports", "expected": [443], "actual": [443, 22], "action": "TICKET"}
{"resource": "sg-web", "setting": "ssh_open_to_world", "expected": false, "actual": true, "action": "AUTO-REMEDIATE"}
{"resource": "db-prod", "setting": "backup_retention_days", "expected": 7, "actual": 3, "action": "TICKET"}

Reading the code: baseline is the approved state and observed is what a scan found, each a set of settings per resource. AUTO_FIX is the short allow-list of (resource, setting) pairs that are safe to revert without a person. diff walks every approved setting, compares it with the observed value, and yields a record only where they differ. The loop at the bottom labels each difference AUTO-REMEDIATE if it is on the allow-list and TICKET otherwise, so anything not explicitly judged safe goes to a human by default.

Four drifts, two auto-fixed, two ticketed. Closing SSH to the world is revert-safe, but an extra open port may be something an application needs, so a person decides. Shortened backup retention is a data-protection decision, not a safe revert.

4. Evidence for the auditor

Keep the baseline version, each scan result with timestamp, each remediation (who or what, when, before and after values), exceptions with approver and expiry, and time to remediate by severity. Show the drift rate trending down, not a flat zero.

Pitfalls

• Auto-remediation fighting the deployment tool in a loop. Fix the source of truth first.
• Remediating with no notification: owners should learn their change was reverted.
• A baseline nobody updates trains people to ignore drift reports.

Direct answer

Build a daily loop: collect configuration state from every cloud account, evaluate it against a small set of rules, store every result as timestamped evidence, route failures to owners with deadlines, and report pass rate and age of open failures to leadership. This is continuous control monitoring (CCM): automated, frequent testing of whether a control is in place, instead of one annual check.

Pipeline

1. Collect. Use the cloud provider's configuration service in every account and aggregate centrally. On AWS, AWS Config is the service that records the settings of resources in an account and checks them against rules, and its managed rules (rules AWS ships ready-made) fit the three asks: S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED (storage buckets encrypted by default), ROOT_ACCOUNT_MFA_ENABLED (root user requires multi-factor authentication) and CLOUD_TRAIL_ENABLED (an audit-log trail is enabled). Admin MFA for day-to-day administrators needs a check on the identity provider (IdP, the system that signs people in, such as Okta or Microsoft Entra ID) or IAM users too, since the root rule covers only the root user.
2. Evaluate. Each rule returns compliant, non-compliant or not applicable per resource. Treat not applicable carefully: ROOT_ACCOUNT_MFA_ENABLED returns it when no root credentials exist, which is fine, but a rule that is not applicable because the feed is broken is not.
3. Store as evidence. Write each result (account, resource, rule and version, result, timestamp) to write-once storage (storage where saved records cannot be edited or deleted for a set period) with a hash, so the history cannot be quietly rewritten.
4. Report. Executives see pass rate by control and open-failure age; engineers see the failing resource.

When a check fails

Making results usable for auditors

• Keep the rule definition, its version and the framework control it supports (for example NIST SP 800-53, a US government catalogue of security controls, where IA-2(1) is the entry in the identification and authentication family that requires multi-factor authentication for privileged accounts) next to the results.
• Keep both failures and fixes; auditors trust a record that shows problems being found and closed more than one that shows no failures at all.
• Record gaps: if the collector was down on Tuesday, say so rather than leaving a hole.
• Sample the automation, because auditors still test whether the rule means what you say it means. A typical sample: the auditor picks a few accounts and dates, checks that the live setting matches the stored result for that date, and picks a few failed checks to confirm a ticket was opened and closed within the promised deadline.

Worked example

40 accounts times 3 checks is 120 evaluations per day. Two fail (one unencrypted bucket, one account with logging off): 118 of 120 pass, 98.3%. Over a year that is 43,800 data points, against one sample on one day in a point-in-time audit.

Limits

A passing rule shows the setting, not that it is the right setting. For example, a trail being enabled does not prove it covers every region or that anyone reviews it. Pair CCM with periodic effectiveness testing.