Security Architect (Entry Level) - FAANG-Standard Interview Preparation Guide

Direct answer: use each cloud's own native workload identity for calls that stay inside that cloud, since that is the boundary each provider's identity system is actually built for, and for the genuinely cross-cloud calls, do not try to make the two clouds trust each other directly, route through a common trust anchor, either a shared external identity provider both clouds federate with, or a small broker service, so neither side ever needs a long-lived static credential for the other.

Within AWS: use identity and access management (IAM) roles as the workload's identity, on Kubernetes specifically via IAM Roles for Service Accounts (IRSA), letting a pod assume a role with no stored credential, based on the platform attesting which service account the pod runs as. For cross-account calls, use role assumption with a scoped trust policy naming exactly the source account and role permitted, rather than sharing static keys across accounts.

Within Azure: use Managed Identities, system-assigned, tied to one resource, or user-assigned, attachable to several, so compute authenticates to other Azure services with no credential stored in code or config. Azure Kubernetes Service supports workload identity federation, the same "the platform attests which pod this is" pattern IRSA uses on AWS. For cross-tenant access, use Azure Active Directory app registrations with explicitly scoped cross-tenant access settings.

The genuinely cross-cloud case: neither AWS IAM nor Azure Managed Identity federates directly with the other cloud out of the box, so a workload in one cloud calling a service in the other needs an intermediate trust anchor. Two common patterns: (1) both clouds trust a THIRD identity provider, an external OpenID Connect (OIDC) issuer both clouds' federation settings recognize, so a workload gets a token from that shared issuer and each cloud independently validates it and hands back its own short-lived credential, or (2) a purpose-built credential-broker service authenticates the calling workload using its OWN cloud's native identity, then, using its own pre-established identity in the OTHER cloud, mints or forwards a short-lived credential for that other cloud.

Avoiding static long-lived credentials throughout: every hop above uses a credential that is short-lived and automatically issued based on platform-level attestation, not a key generated once and stored in a secret. That is the real "least privilege" story beyond narrow scoping alone, a narrowly scoped key that never expires is still a standing risk a short-lived, automatically-rotated credential does not carry.

Worked example: a data-processing job on Amazon Elastic Kubernetes Service (EKS) needs to write to an Azure Blob Storage container. Using the broker pattern, the job's pod assumes an AWS IAM role via IRSA as normal, calls a small broker service running with its own least-privilege identity, the broker verifies the caller's AWS identity, for example checking the assumed-role identifier against an allow-list of jobs permitted to write to that container, and the broker, separately holding an Azure Managed Identity scoped to only that one container, performs the write on the job's behalf or issues it a short-lived, container-scoped Azure access token. At no point does the AWS job hold a standing Azure credential, and the broker's own Azure identity is scoped to exactly one resource.

Trade-offs & pitfalls: the broker pattern concentrates cross-cloud trust into one service, which needs the tightest scoping and monitoring of anything in this design, since it is the one component that legitimately spans both clouds' trust boundaries. Relying on a shared external identity provider instead avoids that single broker, but requires both clouds' federation configuration to stay correctly scoped independently, a mistake in EITHER cloud's trust configuration undermines the whole design.

Direct answer

A strong answer proposes two or three initiatives that are genuinely cross-team, not just a bigger version of your current work, and names, for each, the specific thing it would prove about your scope: usually that you can align people who don't report to you, or that something you build outlives your own use of it. The initiatives can take different shapes, a shared platform or tooling effort, a bounded stretch project outside your current lane, or leaning further into an existing cross-functional partnership, but each one needs a clear owner-question attached, not just a description of the work.

Structured elaboration

1. Screen each candidate initiative against a bar: does it require influence without formal authority, getting other teams to adopt or align, not just executing your own team's roadmap? If not, it isn't actually cross-team scope-scaling.
2. Pick a vehicle deliberately. Three common shapes: a shared platform or tooling initiative that multiple teams adopt, a bounded stretch project in an adjacent area, or leaning into an existing cross-functional partnership and deepening it into something you lead. Different vehicles prove different things: tooling proves you can build something others depend on, a stretch project proves range, a partnership proves you can operate at the seams between teams.
3. Name explicitly what each initiative would prove, for example "that people outside my team will adopt something I build without me pushing it," or "that I can represent a decision to a group that doesn't report to me and get buy-in."
4. Sanity-check the scope. Too small and it won't register as scale-worthy, too large and it becomes unrealistic for a six-to-twelve month window. A good initiative is genuinely finishable in that window with a checkable outcome.

Worked example

"In my own case I proposed a shared internal tool that a couple of adjacent teams had each separately half-built versions of. The value wasn't the tool itself, it was proving I could get two teams with slightly different priorities to agree on one version and actually switch to it. Alongside that, I proposed picking up a partnership that already existed informally between my team and a downstream one, and turning it into something with a real cadence and shared goals, meant to prove I could operate at the seam between two teams rather than just inside my own."

Trade-offs & pitfalls

• Proposing something that's really just a bigger project inside your own team, dressed up as cross-team, doesn't prove the thing this question is testing for.
• Failing to name what each initiative proves turns the answer into a project list rather than a scope-growth case.
• Picking an initiative so large it can't plausibly land in six to twelve months undermines credibility more than picking a smaller, real one.
• Proposing initiatives that all use the same vehicle, three tooling projects, say, misses the chance to show range across different kinds of influence.

Direct answer

Separate the authorization decision from the authorization data: build a dedicated policy-evaluation layer that answers "can this principal take this action on this resource" against a cached, denormalized view of the permission graph, rather than computing that answer with a live join across primary relational tables on every request. At millions of principals and resources, the live-join approach is both too slow and too tightly coupled to core application schema.

Structured elaboration

Modeling resource-level and action-level permissions
Model permissions as tuples of principal, action, and resource, or principal, action, and resource pattern, rather than a single flat role per user. Real APIs need both "can Alice read document 42," resource-level and specific, and "can any Editor create documents in project X," action-level and pattern-based. A model that can only express one of these two shapes forces awkward workarounds for the other.

Hierarchical roles
Define roles that inherit from other roles, Viewer as a subset of Editor as a subset of Admin, so permission grants are authored once per role rather than once per user-permission pair. Combine this with resource hierarchies, a permission granted at a folder or project level implicitly applying to everything nested underneath, so you are not writing millions of individual grants for millions of individual resources.

Attribute-based access control (ABAC)
Layer attribute-based rules on top of the role hierarchy for decisions that depend on context a static role cannot express: time of day, a resource's sensitivity tag, whether a principal's department matches the resource's owning department, or the request's originating IP range. The practical pattern at scale is "roles for the coarse default, ABAC for the exceptions," not replacing roles entirely; a pure-ABAC system where every decision is a fresh rule evaluation becomes far harder to reason about and audit as the rule set grows.

Evaluation latency
The authorization check sits directly on the request hot path, so it has to be fast. Achieve that by pre-computing or denormalizing the parts of the decision that do not change often, role-to-permission mappings and resource hierarchy, into a form the evaluation layer can check in memory or via a single cache lookup, reserving genuinely dynamic per-request evaluation for the smaller set of cases that actually need ABAC attribute matching.

Caching of permissions
Cache the evaluated decision, or a principal's effective permission set, close to the service making the check, with a short time-to-live (TTL). Cache the underlying role and hierarchy graph, which changes far less often than individual grants, more aggressively. These two caches need different invalidation strategies, since they change at very different rates.

Keeping revocation latency low
This is the direct tension with caching: a cached "yes" that should now be "no" is a live authorization bug, not merely staleness, so revocation needs an active invalidation path, pushing a targeted cache-bust for the specific principal, resource, or role that changed, rather than relying on TTL expiry alone to eventually catch up. A common pattern combines a short default TTL, seconds, not minutes, with an active invalidation event fired on any grant or role change, so the cache stays fresh in the common case and the invalidation event closes the remaining gap immediately rather than waiting out the TTL.

Worked example

A document-sharing API has a folder hierarchy. Granting "Editor" on a top-level folder to a principal applies to every document created under it going forward, with no new grant row written per document. The evaluation layer first checks the principal's cached effective-permission set, the fast path, doing no real work at all on a cache hit. On a miss, it walks the resource's hierarchy chain upward to the nearest ancestor with an explicit grant, then caches that result with a short TTL. Revoking the folder-level grant fires an invalidation event that busts the cached decision for every principal-resource pair touched by that grant, rather than waiting for each individual cache entry's TTL to expire on its own.

Trade-offs & pitfalls

Hierarchical inheritance is powerful but makes "why does this principal have this permission" hard to answer without good tooling; a permission-explain or trace capability is not optional at this scale, it is how the system actually gets debugged and audited. ABAC's flexibility is also its risk: rules that reference many attributes become hard to reason about and can interact in unintended ways, so keep the ABAC rule set small and reviewed rather than letting it grow into an ever-larger, ever-harder-to-audit pile. Treating revocation latency as "eventually consistent is fine" is the single most common mistake at this scale; a stale cache that grants access after it should have been revoked is a security incident, not a minor user-experience issue, so the invalidation path deserves as much engineering attention as the fast-path cache itself.

Direct answer

For a mid-sized company facing both PCI (Payment Card Industry Data Security Standard) and HIPAA (Health Insurance Portability and Accountability Act) obligations on a limited budget, the design that satisfies both compliance regimes without paying enterprise-scale storage costs is a three-tier structure, hot (fully indexed, fast search) for the shortest window regulation actually requires immediate availability for, warm (still indexed but on cheaper storage) to complete the shorter regulation's full retention window, and cold/compressed archive for the remainder needed to satisfy the longer-horizon obligation, since paying hot-tier prices for years of rarely-touched data is the single biggest avoidable cost in a retention design like this.

Structured elaboration

What each regulation actually requires, stated carefully since the two are different kinds of obligation. PCI DSS gives an explicit, numeric baseline that has stayed stable across recent versions of the standard (Requirement 10.5.1 in PCI DSS v4.0, previously 10.7): retain audit trail history for at least 12 months, with a minimum of 3 months immediately available for analysis (online, or otherwise readily restorable). HIPAA's Security Rule requires reasonable and appropriate audit controls (45 CFR 164.312(b)) but does not itself state a specific numeric retention period for system-generated audit logs the way PCI does; HIPAA's explicit 6-year retention figure (45 CFR 164.316(b)(2)(i)) applies to REQUIRED DOCUMENTATION (policies, procedures, and Security-Rule-mandated records), not a blanket "keep every log for 6 years" rule. In practice, many healthcare organizations conservatively align their audit-log retention to that same 6-year documentation window as a defensible compliance posture, but that is an organizational choice built on top of the documentation-retention rule, not a literal quote of a HIPAA log-retention mandate. Getting this distinction right matters, because designing to a fabricated or misremembered "HIPAA requires 6 years of logs" rule versus a deliberately-chosen 6-year alignment changes what an auditor will actually ask to see justified.

Tiering, mapped to the above:

• Hot (0-3 months): fully indexed, fast search, satisfies PCI's "3 months immediately available" requirement directly. This is the tier query-heavy investigative and compliance-reporting work actually needs fast.
• Warm (4-12 months): still indexed and searchable, but on cheaper storage with reduced replication, completing PCI's 12-month minimum retention window at a materially lower cost than keeping the same 9 months at hot-tier density.
• Cold/archive (13 months-6 years): compressed, not directly indexed for ad-hoc search (restorable/rehydratable on demand for the rare case an investigation or audit needs data this old), covering the organization's own chosen alignment with HIPAA's documentation-retention posture.

Compression/archival strategy: raw, uncompressed retention in the cold tier is the single most avoidable cost at this scale; columnar or standard compression in the 6-10x range is realistic for structured security log data and should be applied at the transition into cold storage, not before, since the hot/warm tiers need fast, uncompressed (or lightly compressed) access.

Access controls: role-based access scoped by tier makes sense here specifically, most analysts need routine access to hot/warm for day-to-day triage, while cold-tier access (and any retrieval/rehydration action) should require a documented business or investigative justification and be itself logged, since cold-tier data is disproportionately likely to be the subject of a compliance audit or legal request, and an access trail on the retention system itself is part of demonstrating the program's integrity.

Worked example

Assume this mid-sized company ingests 500 gigabytes (GB) per day of security-relevant log volume. Applying 1.3x index overhead and 2x replication to the hot tier, 1.3x overhead and 1x replication to warm, and 8:1 compression to cold:

• Hot (90 days): $500 \times 90 = 45{,}000$ GB = 45 TB raw; indexed footprint $45 \times 1.3 \times 2 = 117$ TB.
• Warm (270 days, months 4-12): $500 \times 270 = 135{,}000$ GB = 135 TB raw; indexed footprint $135 \times 1.3 \times 1 = 175.5$ TB.
• Cold (5 more years, 1,825 days, to reach the chosen 6-year total): $500 \times 1{,}825 = 912{,}500$ GB = 912.5 TB raw; compressed at 8:1, $912.5 / 8 = 114.1$ TB.

Total tiered footprint: roughly $117 + 175.5 + 114.1 = 406.6$ TB. Compare this against the counterfactual of keeping all 6 years at hot-tier density and replication: $500 \times 365 \times 6 / 1000 \times 1.3 \times 2 = 2{,}847$ TB. The tiered design uses about $2{,}847 / 406.6 \approx 7.0$ times less storage than a flat hot-tier-everywhere design for the identical 6-year retention obligation, which is the concrete, quantified argument for why tiering (not a smaller retention window) is the right lever to pull under a limited budget, rather than trying to shrink the retention period itself below what compliance requires.

Trade-offs and pitfalls

• Legal hold overrides normal tiering and deletion: any data subject to a legal hold (active litigation, a regulatory investigation) must be preserved regardless of what the standard retention schedule would otherwise do to it, including scheduled deletion past the 6-year window; the retention system needs an explicit hold mechanism that can pin specific data indefinitely, separate from and overriding the normal tier-transition and deletion automation.
• Summarization/sampling as a further budget lever, used carefully: for the longest-horizon cold tier specifically, some organizations further reduce cost by summarizing very old, low-value telemetry (rolling up routine, already-reviewed events into aggregate statistics) rather than retaining every raw record; this is a more aggressive trade-off than compression alone and should be scoped narrowly (never applied to data still within a regulation's explicit numeric retention window, like PCI's 12 months, only to data being kept past that point purely for the organization's own longer-horizon posture).
• Common mistake: treating "12 months" (PCI's explicit number) and "keep everything forever to be safe" as the only two options; an under-scoped retention window risks a compliance finding, but an unbounded one accumulates unnecessary cost AND unnecessary legal exposure (data an organization did not need to keep is data that can be subpoenaed or breached), so the retention period itself should be a deliberate, justified decision tied to actual regulatory and business need, not a default.
• Common mistake: applying compression before the data's active investigative window has passed; compressing the hot tier to save cost defeats its purpose (query latency on compressed, non-indexed data is far higher), which is exactly why the tiering boundary should track WHEN data stops needing fast search, not just its age in the abstract.

The right DevSecOps KPIs measure whether the program is actually reducing risk and improving over time, not just whether tools are technically running; a metric that only tracks activity (scans run, findings generated) can look healthy while the actual security posture stays flat or worsens.

The KPIs, and why each matters

Mean time to remediate (MTTR) a finding, split by severity. This measures whether the organization is actually FIXING what its tools find, not just detecting it; a low detection rate paired with a high MTTR means the program is generating findings nobody acts on, which is arguably worse than not scanning at all, since it creates a false sense of coverage.

Percentage of pull requests with a passing security scan. This measures adoption and coverage: are teams actually running through the gates, or finding ways around them; a declining percentage over time is an early warning that developer trust or tooling reliability is degrading before it shows up as an actual incident.

False-positive rate of tools. This measures whether the tuning discipline discussed throughout this topic is actually holding; a rising false-positive rate predicts developer trust erosion before that erosion becomes visible in adoption metrics, making it a leading indicator rather than a lagging one.

Why each matters, and potential pitfalls

MTTR alone can be gamed by closing tickets without genuinely fixing the underlying issue, so it should be paired with a re-scan confirming the finding is actually gone, not just that the ticket was closed. Percentage-of-PRs-scanned can look artificially high if teams route around the gate for a subset of changes (a hotfix branch that skips the normal PR flow, for instance), so it needs to be measured against ALL changes reaching production, not just changes that went through the standard PR path. False-positive rate needs a consistent definition (confirmed-by-a-human as a false positive, not just 'dismissed', since a dismissed finding might be a true positive someone dismissed without proper review) or it becomes trivially gameable by encouraging developers to dismiss anything inconvenient.

Collecting and presenting these metrics

Collect MTTR and scan-coverage data directly from the CI/CD platform and ticketing system's own event logs, rather than a manual survey, so the numbers are objective and can't be softened in reporting. Present engineering teams with their own team-level breakdown (so they can act on their specific gaps) and present executive leadership with an aggregate trend over time (is the program improving, plateauing, or regressing) rather than a team-by-team scorecard that reads as ranking teams against each other, which tends to produce defensive gaming of the metrics rather than genuine improvement.

Trade-offs

Tracking MTTR by severity and re-scan-confirmed-fixed rather than a simple ticket-closure count is more work to instrument, but it's the difference between a metric that reflects reality and one a team can quietly game by closing tickets faster without actually fixing anything, which would make the whole KPI program actively counterproductive rather than merely imperfect.

Framing

SHA-1 (Secure Hash Algorithm 1) has a practically demonstrated collision break: attackers have published real pairs of distinct inputs producing the same hash, which matters enormously for certificate signatures (a certificate authority's signature over a certificate is only trustworthy if nobody could have engineered a colliding, malicious certificate with the same signature) and matters somewhat differently for a version-control system, where hashes are primarily content-addressing identifiers, not signatures over untrusted third-party input, but a maliciously crafted collision could still let an attacker substitute code silently. The migration has to run on two independent tracks with different urgency, plus a compatibility bridge for anything that can't move immediately.

Phased plan

1. Inventory and freeze new usage. Find every place SHA-1 is used: certificate signing requests still being issued with SHA-1, internal certificate authorities configured to sign with it, git repositories, and any code computing SHA-1 for reasons other than pure legacy compatibility. Immediately stop issuing anything new with SHA-1, every new certificate gets a SHA-256 signature, this alone is low-risk and shrinks the problem going forward.
2. Certificate replacement, done by criticality. Re-issue certificates in order of exposure: externally-facing services first (least time for an attacker to have obtained a colliding forgery in the wild, since publicly trusted certificate authorities stopped issuing SHA-1 certificates industry-wide years ago), then internal services with their own private certificate authority. For each certificate, reissue with a SHA-256 (or stronger) signature, deploy to the service, and confirm it serves the new chain before revoking the old one, not the other way around, to avoid an outage window with no valid certificate at all.
3. Interoperability with older clients. Some very old clients may not support SHA-256-signed certificate chains. Handle this the same way any protocol version deprecation is handled: identify which specific clients are actually still connecting (from real traffic logs, not assumption), give them a defined, communicated end-of-support date, and only keep a SHA-1 fallback path alive for that dwindling population, with monitoring on its usage so you know when it's safe to remove.
4. Repository migration. For internal git repositories still keyed by SHA-1 object identifiers, the practical mitigation already in wide use is a hardened SHA-1 implementation that detects known collision-attack patterns (this is what major hosting platforms adopted after the first public collision was demonstrated) rather than an immediate wholesale rehash, since git's own transition to a stronger hash is a larger, longer-running effort tracked by the git project itself; the near-term action for an internal deployment is ensuring the collision-detecting variant is actually in use, and tracking the upstream migration path for the longer term.
5. Verification. Confirm success with evidence, not assumption: scan all active certificates for signature algorithm, monitor certificate authority issuance logs to confirm zero new SHA-1 signatures, and run a hash-algorithm audit against the code and configuration inventory from step 1 to catch anything missed.

Trade-offs and pitfalls

The riskiest failure mode in this kind of migration isn't SHA-1 itself, it's a broken rollout: revoking an old certificate before confirming the new one is deployed and trusted causes a real outage, and rushing repository rehashing without coordinating with every consumer of those commit hashes (build systems, deployment pipelines, external mirrors) can silently break more than it fixes. Sequence certificate work by exposure and treat the legacy-client interoperability period as a tracked, time-boxed exception with an actual removal date, not a permanent fallback that quietly never gets revisited.

Approach (brief)
Group hosts by their tags from CSV, produce JSON policy with groups as source/destination, and allow rules for predefined service ports. Ensure idempotency by using deterministic IDs (hashes) and merging existing policy file if present.

Sample Python (idempotent)

#!/usr/bin/env python3
import csv, json, hashlib, os

PORTS = [{"name":"ssh","port":22},{"name":"http","port":80},{"name":"https","port":443}]
POLICY_FILE = "microseg_policy.json"

def id_for(name):
    return hashlib.sha1(name.encode()).hexdigest()[:8]

def load_inventory(path):
    tags = {}
    with open(path) as f:
        reader = csv.DictReader(f)
        for r in reader:
            for t in r['tags'].split(';'):
                tags.setdefault(t.strip(), []).append({"hostname":r['hostname'],"ip":r['ip']})
    return tags

def build_policy(groups):
    policy = {"groups":[], "rules":[]}
    for gname, hosts in sorted(groups.items()):
        policy['groups'].append({"id": id_for(gname), "name": gname, "members": [h['ip'] for h in hosts]})
    for src in policy['groups']:
        for dst in policy['groups']:
            if src['id']==dst['id']: continue
            for p in PORTS:
                rid = id_for(src['id']+dst['id']+p['name'])
                policy['rules'].append({"id":rid,"src":src['id'],"dst":dst['id'],"port":p['port'],"protocol":"tcp","action":"allow"})
    return policy

inv = load_inventory("assets.csv")
new = build_policy(inv)

# idempotent write/merge: if file exists, only replace rules/groups when changed
if os.path.exists(POLICY_FILE):
    with open(POLICY_FILE) as f: old = json.load(f)
else:
    old = {}
if old != new:
    with open(POLICY_FILE,'w') as f: json.dump(new,f,indent=2)
    print("Policy updated")
else:
    print("No changes")

Idempotency reasoning
Deterministic IDs and sorted iteration guarantee same output for same input; merge/compare avoids unnecessary writes.

Staging tests before enforcement

• Run script in staging with realistic CSV; verify JSON diff against expected (git, unit tests).
• Deploy read-only into policy engine (simulation mode) to validate no unintended denies.
• Run connectivity tests (nmap, service checks) from representative VMs to ensure allowed flows work and others remain blocked in a non-production sandbox.
• Peer review and run automated CI checks that validate ID stability and schema.

Direct answer

A squad-autonomy model, small, cross-functional, semi-independent teams, as popularized by Spotify's own engineering culture writeups (though by Spotify's own later account, only partially realized in practice), shifts goal-setting from top-down assigned targets toward squads proposing their own goals against a shared mission, which means tracking progress and escalating risk both depend much more on voluntary, structured transparency than on a manager simply reviewing a task list, so you'd adopt lightweight rituals that make that transparency happen by default rather than by exception.

Structured elaboration

• Goal-setting shift: instead of "here is your quarterly plan," squads typically align to a broader shared mission and then define their own concrete goals within it, which requires real upfront investment in making sure that mission is genuinely well understood, or squads quietly drift apart.
• Progress-tracking shift: without a single central plan to check against, you rely on regular, structured squad-level updates visible across the wider group rather than a manager independently knowing status, turning tracking into a broadcast problem rather than a manager-checks-in problem.
• Risk-escalation shift: since squads are meant to self-manage, honest early escalation only works if raising a risk isn't punished, otherwise autonomy quietly turns into siloed teams hiding problems until they become unavoidable.
• Cross-squad coordination: initiatives spanning multiple squads need an explicit, lightweight coordination mechanism, since no single manager owns all the relevant squads, commonly a cross-squad sync for a shared concern, or a designated point person per squad for a specific initiative.

Two concrete rituals worth adopting or adjusting:

1. A regular, short cross-squad demo where each squad shows real, working progress to the wider group rather than a status slide, making progress visible without a central tracker and surfacing overlapping or conflicting work early.
2. A lightweight risk check built into each squad's own retro or planning ritual, where the squad explicitly states anything at risk of missing its own goal as a short written note shared upward, so risk surfaces without someone above the squad having to go dig for it.

Worked example

Coordinating a cross-squad initiative like a new checkout flow that touches a Payments squad and a Fulfillment squad. Instead of a shared manager assigning tasks to both, you'd set up a short biweekly cross-squad sync focused only on the shared interface between the two squads' work, and each squad would flag in its own retro anything at risk of slipping that would affect the other squad, rather than waiting for a status meeting to surface it.

Trade-offs and pitfalls

The most common misreading of this model is that it means no accountability. In a well-functioning version, accountability shifts from "did you follow the plan I gave you" to "did you deliver on the goal you committed to, and did you raise it honestly if you couldn't," which is a real, sometimes harder form of accountability, not an absence of one. It's also worth being honest that Spotify itself has said in later retrospectives that the model as commonly described was more aspirational than a rigid blueprint fully implemented at any single point in time, so lean on the underlying principles, autonomy paired with alignment and transparency, rather than treating any one named ritual as a rulebook to copy exactly.

Direct answer

Compare the two paths in money and in tail risk, not in gut feel. Accept the risk only if (1) the expected cost of accepting plus the contingency is lower than the cost of the delay, (2) the worst case is survivable and reversible, and (3) someone with the authority to own that loss signs for it. In the numbers below accepting is cheaper over the first year, but only until the probability estimate moves or the risk stays open for a second year, so the estimate and the contingency's proven readiness are what I would pressure-test.

Worked example (all figures illustrative)

Expected loss means probability times impact, per year. Tail risk means a rare but very severe outcome, the kind an average hides. The three lines model splits risk work into a first line (the teams that build and run the product and own its risks), a second line (security, risk and compliance specialists who advise and challenge the first line) and a third line (internal audit).

• Impact if the risk occurs: $2,000k. Chance in the first year: 20%. Expected loss = 0.20 x 2,000 = $400k.
• Contingency plan (tested rollback, kill switch, customer communication) cuts the impact by 60%, so residual expected loss = 400 x 0.40 = $160k, plus $40k to build and maintain the plan: $200k to accept.
• Preventive control: removes the risk, but the six-week delay costs about $300k in deferred revenue and team time: $300k to eliminate. The $300k delay is paid once, while the $160k expected residual loss recurs each year the risk stays open. So $200k against $300k is a first-year view: over two years accepting costs 40 + 2 x 160 = $360k, more than $300k, unless a later fix closes the risk (which is why the acceptance gets an expiry date).
• Break-even probability p: p x 2,000 x 0.40 + 40 = 300, so p = 32.5%. At 20% accepting wins by $100k. At 40% accepting costs 0.40 x 2,000 x 0.40 + 40 = $360k and delaying wins.
• The break-even depends on how long the risk stays open, because the delay is paid once and the residual loss recurs. Over two years: p x 800 x 2 + 40 = 300, so p = 260 / 1,600 = 16.25%. Turned around, at the 20% estimate (assumed to repeat each year) accepting costs 40 + 160n for n years open, which equals $300k at n = 260 / 160 = 1.625 years. So the 20% estimate favours accepting only if the risk is closed (by a later fix) within about a year and a half, which is exactly what the expiry date and the funded follow-up enforce.

Who has to agree

• The risk owner: the executive who owns the business outcome (product or general manager), because they absorb the loss. Engineering recommends but does not accept on its own.
• Security or risk (second line, the specialists who challenge the first line's assessment) confirms the assessment is honest and the residual sits inside risk appetite (how much risk the company is willing to pursue) and the delegated authority of the signer. Above that limit it escalates.
• Legal or compliance if a contract or regulation is involved. A signature cannot waive a legal obligation; counsel decides that.
• Write it down: risk, options, numbers, signer, expiry date, review trigger.

What would change my answer

• The probability estimate and the time the risk stays open (the 32.5% first-year break-even is the line to watch, and it falls to 16.25% if the risk stays open two years).
• Whether the contingency has actually been exercised. A plan that exists on paper has only been designed (design effectiveness: it should work if followed); one that has been drilled has shown it works in practice (operating effectiveness). Count the 60% reduction only for the drilled kind.
• The tail: if the loss is irreversible (data loss, safety, regulatory breach), a contingency does not make it acceptable.
• A middle path often beats both: ship behind a feature flag to a small share of users, cut scope, or implement a smaller preventive control in two weeks.
• A hard external deadline raises the cost of delay and shifts the answer toward accept.

Pitfalls: letting the loudest stakeholder set the probability, treating "strong contingency" as proven, and accepting with no expiry so the exception becomes permanent.

Direct answer

I would frame it as a decision about where the data and the model should live, not a slogan. My default recommendation is a split: run a smaller personalization model on the device so raw behavioural data never leaves it, and keep heavy training and any non-personal model work in the cloud. I would put the heavy model fully in the cloud only if the device genuinely cannot run it at acceptable quality, and then minimize what is sent.

How I frame the recommendation

1. What data does the model need (clicks, location, health, messages), and how sensitive is it?
2. Who benefits from keeping it local, and who bears the cost (battery, storage, app size, device fragmentation)?
3. What quality does the product need, and does a compressed model deliver it?
4. Who owns the decision: product and the ML lead decide quality and cost, security and privacy sign off on data flows.

Privacy and security benefits of on-device

• Raw personal data stays on the phone, so there is less to breach, subpoena (a legal order forcing a company to hand over data it holds), over-retain or misuse for a new purpose later (data minimization).
• Smaller attack surface (fewer places an attacker can reach) on the server side, no per-user profile store to protect.
• Works offline and with lower latency.
• Simpler deletion: clearing app data clears the personalization state.

Benefits of cloud

• Larger, better models and quick updates, with central monitoring.
• Easier evaluation and abuse detection.
• Model and weights are protected from extraction by users, and keys are handled centrally.

What you give up with on-device

Worked example

A shopping app wants personalized rankings from browsing history. Option A sends the full browse history to the server each session. Option B keeps history on the device, scores 200 candidate items locally using a small model, and sends only the IDs of the items the user tapped, not the browsing history (the server can then count taps per item across users, but each session still uploads a short list). With illustrative numbers: Option A uploads about 300 browse events of roughly 100 bytes each, 30 KB per session, and ranks at a 6.0% click-through rate. Option B uploads 5 tapped item IDs, well under 1 KB, and a smaller model might reach 5.4%, a 10% relative drop (0.6 / 6.0), while scoring 200 items at about 5 ms each costs roughly 1 second of processor time per session (200 x 5 ms = 1,000 ms), which is the battery cost to measure. Option B loses some ranking quality and some insight into long-tail behaviour, but the server never holds the browse history. I would recommend B, with a cloud fallback for devices below a hardware threshold that sends only the minimum features and is opt-in.

Decision flips

If the model needs data from many users to work (collaborative signals) or needs frequent updates, I would move that part to the cloud (collaborative signals, such as "people who bought this also bought that", only exist when many users' data is combined) with pseudonymous, minimized inputs. If the data is highly sensitive (health), I lean on-device. If model theft is the bigger business risk, cloud wins.