Internal Controls Design and Effectiveness Testing Questions

Direct answer. I would tier the controls by risk, give each tier a test frequency, and fill a year calendar that avoids owners' peak periods and batches requests per owner. Fixed controls are re-tested only after they have operated long enough to sample. The rules, not my memory, drive the schedule, so it can be rebuilt each year.

Frequency by risk. Score each control on inherent risk of what it protects, importance to key requirements (a key control is one that, if it failed, would not be caught by others), whether it is manual or automated, recent changes, and past results. Inherent risk means the risk before any control is applied. A worked score (illustrative, 1 to 3 on each of four factors, total 4 to 12; recent changes and past results are folded into one history factor, so the five factors above become four scored columns): a quarterly privileged-access review has inherent risk 3, key control 3, manual 3, history of problems 2, total 3+3+3+2 = 11. An automated disk-encryption configuration check has inherent risk 2, not a key control 1, automated 1, clean history 1, total 2+1+1+1 = 5. Cut-offs of 10 or more for Tier 1, 7 to 9 for Tier 2 and 6 or less for Tier 3 put the first control in Tier 1 and the second in Tier 3; applying such cut-offs across all 300 controls is how a split like the one below arises. Illustrative tiers for 300 controls:

tiers = {"Tier 1 (quarterly)": (60, 4), "Tier 2 (twice a year)": (120, 2), "Tier 3 (annual)": (120, 1)}
total = sum(n * freq for n, freq in tiers.values())
print("test executions per year", total)                 # 600
print("per month at even load", total / 12)              # 50.0
normal, peak = 60, 30   # peak months at half capacity
print(8 * normal + 4 * peak)                             # 600

Printed: 600, 50.0 and 600. Capacity is 8 normal months at 60 tests plus 4 peak months at 30 tests (illustrative quarter-end months), which fits all 600 with no spare capacity, so in practice I would hold some back for re-tests of fixed controls and for failed tests. With 40 owners (illustrative), that is about 1.5 tests per owner in a normal month (60 tests across 40 owners), which batching turns into a single request per owner per cycle. Automated controls with stable configuration and effective general controls over them (the baseline access and change controls on the system the automated control runs on) can sit in a lower tier, subject to the assessor's rules.

Timing. Test earlier in the period for controls with a history of problems, so there is time to fix and re-test. Avoid quarter-end close, major releases and holidays for the owners involved, using blackout dates (dates on which no test requests go out) for each owner. Batch all of an owner's controls into one request per cycle, and cap the monthly load per owner.

Re-testing fixed controls. A fixed control is re-tested after it has operated for enough occurrences to sample, for example one full cycle for a quarterly control or a few weeks for a daily one (illustrative; follow the assessor's guidance). Move it up one tier (for example from annual to twice a year) until it passes two tests in a row, because one pass can be luck while two show the fix holds; then return it to its normal tier.

Owner testing and independent testing. Owners self-test lower tiers and send evidence (first-line assurance). An independent tester re-performs a sample from every tier and tests Tier 1 directly.

Global scale and tests-as-code. For a global company, use one calendar with regional blackout dates (local holidays, regional closes), time-zone-aware evidence requests, and local-language templates. For automated controls, write the tests themselves as scripts (tests-as-code) that run daily against configuration, store results with timestamps and open a ticket on failure. Treat the test code as a controlled asset: version control, review and change approval, since a flawed check gives false comfort.

Pitfall. Equal testing for all controls: it overloads owners with low-value requests and leaves the risky ones under-tested.

Direct answer. A compensating control is a substitute control used when a real constraint stops you meeting a requirement as written; it must give equivalent protection against the same risk. Here the legacy app cannot support MFA (multi-factor authentication, a second proof of identity beyond the password), so I would put the strong authentication in front of the legacy app instead of inside it: isolate the app on its own network segment (a separate part of the network behind firewall rules), allow access only through a privileged-access gateway or jump host (a hardened intermediate server that administrators must log into first, which enforces MFA), and add session recording, just-in-time (JIT) privileged access and frequent access reviews. Then I would show these controls meet the intent of the MFA requirement, record the residual risk and test them regularly with an expiry date.

Controls I would propose
Items 1 and 2 close the attack path the MFA requirement guards; items 3 to 7 limit and monitor what is left.

1. Network isolation: the app sits in its own segment; firewall rules permit only the jump host.
2. Jump host with MFA: users authenticate with MFA to the gateway, then reach the app. The protocol limitation is bypassed rather than argued with.
3. JIT privilege: admin rights granted for a short window with approval, not standing.
4. Credential vaulting and rotation (a password vault stores the app's credentials and changes them regularly) for app passwords, so shared or static credentials are not memorised.
5. Session monitoring: record privileged sessions and alert on unusual use (outside hours, new source).
6. Access review cadence: quarterly review of who can reach the app (monthly for admins).
7. Residual risk: document what remains, meaning the exposure left after the controls (for example an attacker already on the jump host after MFA, weak in-app audit logs), and have the risk owner accept it in writing.

Showing sufficiency. State the objective of the requirement: a stolen or guessed password alone must not be enough. Map the attack path: attacker needs the network path (blocked) and the MFA factor (at gateway). The controls must go beyond what is already required elsewhere; counting existing controls is not compensation. For PCI DSS (the Payment Card Industry Data Security Standard, the rules for organisations handling card payments) this follows its formal compensating-control rules: a legitimate documented constraint, equivalent defence, above and beyond other requirements, and a worksheet (the compensating controls worksheet in PCI DSS Appendix C), which in outline asks for the constraint, the original requirement's objective, the added risk, how the substitute meets the objective, how it was validated and how it is maintained. Whether an assessor accepts it is the assessor's decision.

Document and test over time

• Design test now (is the control, as designed, capable of meeting the objective): walk through the firewall rules and gateway config.
• Operating tests (does it actually run as designed over time): quarterly connectivity test from a non-jump subnet (must fail); review a sample of recorded sessions; check the access review happened.
• Set an expiry or annual review date and a retirement plan (replace or upgrade the app).

Pitfall. Treating a compensating control as permanent. It is a time-limited bridge.

Direct answer. I would test the patching control in four steps: confirm the population is complete, test the whole population by analytics, reperform a statistical sample against independent evidence, and set the pass rule in advance. Illustrative policy being tested: critical patches installed within 14 days of release, or an approved, unexpired exception.

Terms used below. The population is the full list of 2,000 servers. A deviation is a sampled server that breaks the rule (a critical patch not installed within 14 days and no approved, unexpired exception). Reperformance means redoing the check yourself from raw evidence instead of trusting a report. Crown jewels are the systems whose compromise would hurt the business most.

1. Population completeness. Reconcile the 2,000 against the CMDB (configuration management database, the official list of servers), cloud account inventory and a network scan. A server missing from inventory is never patched, so unknown servers are a design finding.

2. Whole-population analytics. Use scanner data on all 2,000 servers to measure how many have critical patches past due without an exception. Add 100% review of the highest-risk servers (for example internet-facing or crown jewels, say 100 of the 2,000).

3. Sample for reperformance. Scanner reports can be wrong (agents offline, stale data), so I test a sample directly. I choose two risk thresholds in advance: the highest deviation rate I will tolerate (10%, a judgment about how much non-compliance the business can live with) and the confidence I want (95%). The question the sample answers is: if the control were really that bad, would a clean sample be likely? Suppose exactly 10% of the 2,000 (200 servers) deviate. Drawing servers one at a time without replacement (the hypergeometric distribution, the exact method for a finite list), the chance the first server is clean is 1800/2000, the second 1799/1999, and so on. Multiplying 29 of these fractions, (1800/2000) x (1799/1999) x ... x (1772/1972), gives 4.60% (computed). Since 4.6% is under 5% (1 minus 95%), seeing zero deviations in 29 would be unlikely if the true rate were 10% or worse, so I can say with at least 95% confidence the rate is below 10%. With 28 servers the chance is 5.12%, just above 5%, so 29 is the smallest sample that works. For each server: check package history or OS patch logs, not just the scan, and confirm date versus the 14-day rule.

4. Pass and fail rules

A "pass" at 29 means "likely under 10% deviation", not "no deviations".

Continuous monitoring and retest. After a failure, retest on a fresh sample once a full patch cycle has passed. Between tests, track weekly the percent past SLA from the scanner and alert when it rises above the threshold.

Evidence trail. Save scanner exports, queries and sample evidence in a write-once store (storage that cannot be edited or deleted after saving) with timestamps, the tester's name and a file hash (a fingerprint such as SHA-256 that changes if the file is altered) so results can be reproduced for an auditor or regulator.

Pitfall. Testing only the scanner's report: that proves the scanner, not the patching.

Direct answer

A control is a safeguard that reduces a risk. A preventive control stops the bad event from happening (a blocked action), a detective control notices that it happened or is happening (an alert or a review), and a corrective control limits the damage and restores the good state afterwards (a rollback or a restore). I choose the mix by how bad and how reversible the harm is, and I distrust any "preventive" label until someone has tried and failed to get around the control.

What each type looks like on a cloud customer-facing app

The same split exists outside IT. In payments, requiring a second approver before money is released is preventive and manual; a monthly reconciliation review (comparing two records and investigating differences) is detective. For site reliability engineering (SRE), a deploy gate that blocks a release failing health checks is preventive, an alert on error-rate burn (errors using up the service's allowed failure budget faster than planned) is detective, and an automatic rollback is corrective. They complement each other: prevention lowers how often you need detection, detection triggers correction, and correction shortens how long a miss hurts.

How I decide the mix

1. Rank what could go wrong by impact and reversibility. Data exposure cannot be undone, so it gets strong preventive controls. A slow page can be fixed in minutes, so detection plus fast rollback is cheaper than blocking every change.
2. Ask how quickly I would find out. If detection is slow or blind, I push harder on prevention.
3. Weigh cost: preventive controls add friction for engineers, so I place them where the harm justifies it.
4. Decide with partial information: assume some failures will get through and some detections will be missed, so keep at least two layers for anything high impact, and name the response step (the specific person who acts in the first hour, for example who isolates the system, who rolls back, who informs customers). That step is what turns a detective alert into a corrective action.
5. Prefer automated over manual where possible: a manual control depends on a person remembering, so I check how often it runs and what happens on holiday weeks.

A compensating control is a substitute used when the intended control cannot be applied; its effectiveness is shown by testing it over time, not by describing it. Example: a legacy system cannot support multi-factor sign-in, so admins may only reach it through a hardened jump host (a locked-down gateway machine) with session recording and a weekly review of the recordings.

Why I would distrust a "preventive" label

• It can be bypassed: an admin override, a break-glass path (a pre-approved emergency admin route that skips the normal checks) nobody reviews, or a "warn only" mode left on.
• It covers new resources but not the old ones, or only some accounts.
• It is manual and relies on a person, so it is really a hope.
• It has never been tested with a negative test (deliberately try the forbidden action and confirm it is refused).
• The people it constrains can edit it.

Example: a rule says "no public storage buckets". I try to create one in a test account. If it succeeds, or if existing buckets were never checked, the label was wrong and the control is at best detective.

Direct answer. In a move to SaaS (software the vendor runs for you), the controls over the underlying infrastructure move to the vendor and you lose direct visibility; the controls over your users, your configuration and your data stay with you; and some controls are new (vendor oversight, integrations, exit). Assurance gathered on the old system does not carry over: it described a different system. I would plan the transition in three phases (before cutover, at cutover, after cutover) and build fresh assurance on the new system.

What changes.

Relying on the vendor. Request the vendor's SOC 2 report (an independent auditor's report on controls against the Trust Services Criteria) or SOC 1 where the system affects financial reporting (SOC 2 covers security-type criteria; SOC 1 covers controls relevant to the customer's financial statements). Check the audit period, the scope, exceptions found, and the complementary user entity controls (CUECs): controls the report assumes you perform yourself. Ask for a bridge letter (a vendor statement that its controls have not materially changed since the report's end date) covering the gap between that date and today.

Old-system assurance. Past test results stay valid for the past period on the old system. They are not evidence the new system works. For a period containing the cutover, an auditor tests the old controls until the cutover date and the new ones after it. Keep the old evidence for the retention period (set by your records policy and applicable law; illustrative: seven years for financial-system evidence) and archive the old system's logs and records, even after decommission.

Control transition plan.

1. Before cutover: map each old control to keep, replaced by vendor, redesigned, or retire. Review the vendor reports and CUECs. Design the new controls and walk through them with owners. Define success criteria and a rollback decision date.
2. At cutover: reconcile data (record counts and control totals (sums or counts such as total invoice value, computed on both systems to prove nothing was lost) agree between old and new, with exceptions explained), freeze changes, review the migrated access list line by line (migrated access should be re-approved rather than copied blindly), and keep an interim manual or detective control where a new one is not ready.
3. After cutover: test the new controls for operating effectiveness after at least one full operating cycle (long enough for each control to have run enough times to test; illustrative: a monthly reconciliation needs at least two month-ends on the new system, and the assessor sets the real minimum), close CUEC gaps, and decommission the old system through a change record.

Pitfall. Assuming the vendor's report means you have no controls left. The CUECs are your obligations.