Microsoft Staff DevOps Engineer Interview Preparation Guide

Whether to fully automate, keep human-in-the-loop, or leave manual comes down to four questions applied to each specific action: how often does it happen, how bad is it if it goes wrong, can it be safely retried, and can success be verified automatically. High frequency, low blast radius, idempotent, and observable pushes toward full automation; anything destructive or hard to verify stays manual or gated behind a human, no matter how routine it feels.

The criteria

Applying it to the three actions

Restarting a cached worker instance: high frequency, low blast radius (stateless, replaceable), fully idempotent, and easily verified with a health check. This is a strong automate candidate: drain connections, spin up a replacement, run a health check, cut traffic over, roll back automatically if the health check fails.

Reattaching a detached volume: lower frequency, meaningfully higher blast radius (attaching to the wrong instance or double-attaching can corrupt data), and only moderately idempotent, reattaching twice isn't necessarily safe. This sits in the middle: automate the pre-checks and the mechanical steps (verify volume ID, verify target instance, snapshot before attaching), but require a human to confirm before the final attach executes.

Running a database schema migration: low frequency, high blast radius (can be destructive and hard to reverse), low idempotency for anything involving DDL (data definition language: schema-altering SQL statements like ALTER TABLE), and success often isn't verifiable by a simple automated check, it needs someone to look at whether the data actually came out right. This stays manual, or more precisely, human-gated: automation handles the mechanical parts (schema diff, pre-migration validation, backup, staged rollout to a canary), but a person approves the production apply.

The underlying argument for phasing automation in gradually

Automating a step doesn't just remove toil, it also removes the moment a human would have caught something unusual about this particular instance of the problem. That's fine for the worker-restart case, where "unusual" mostly doesn't exist, but risky for the migration case, where every migration is a little different. The practical path is phasing: run a new automation in shadow mode first (it proposes the action but a human executes), then human-in-the-loop (it executes after one-click approval), and only promote to full automation once it has a track record across enough real incidents that its false-positive and false-negative rate are actually known, not assumed.

Guarding automated actions with least privilege

Whatever is automated should run with only the permissions that specific action needs, a worker-restart automation shouldn't hold credentials that could also run a schema migration, and every automated action should be logged with who (or what) triggered it and why. For the human-in-the-loop tier, the approval step itself should require a specific person's action (not a shared bot token anyone can trigger), so there's a real approval trail, not a rubber stamp.

Trade-offs and pitfalls

The common wrong turn is automating based on how annoying a task feels rather than how safe it is, restarting workers manually is annoying but safe to automate; migrations are also annoying, but the annoyance is not the variable that should decide it. The other pitfall is leaving a human-in-the-loop step gated behind an approval that nobody actually reads before clicking, if the approval doesn't include enough context (what will run, what's the blast radius, what's the rollback) to make a real judgment, it's automation with an extra click, not a genuine safety gate.

Direct answer

All three mechanisms trade a usage commitment for a lower price, but they commit to different things: AWS Reserved Instances (RIs) commit to a specific instance configuration, AWS Savings Plans commit to a dollar-per-hour spend level that flexes across instance types, Google Cloud committed-use discounts (CUDs) commit to either a resource quantity or a dollar-per-hour spend depending on which CUD type you buy, and Azure Reservations commit to a specific VM configuration similar to AWS RIs. The general pattern across every provider is the same trade-off: the more precisely you commit to a specific instance shape, the bigger the discount; the more flexibility you keep, the smaller the discount but the lower your risk if the workload changes shape.

Structured elaboration

Mechanism comparison

All four providers offer some form of upfront, partial-upfront, or no-upfront (pay monthly) payment on these commitments at the same total cost, so the payment option is a cash-flow decision, not a discount-size decision on most of these products.

Why the scope difference matters in practice
A resource-level commitment (Standard RI, resource-based CUD, Azure Reservation) only pays off if the workload keeps needing that exact shape for the whole term; if the team migrates to a different instance family six months in, the commitment sits partially wasted (though AWS and Azure both allow some exchange or resale mechanisms to recover part of that). A spend-based commitment (Compute Savings Plan, flexible CUD) survives an instance-family change automatically, because the discount is applied to dollars spent on eligible usage, not to a specific SKU, at the cost of a somewhat smaller discount than the narrowest resource-level option.

Deciding what percentage of steady-state capacity to commit
Start from the floor, not the average: pull 3 to 6 months of utilization history for the workload, and find the usage level that held true on the worst week, not the typical week. That floor, not the mean, is the safe commitment baseline, because a commitment above the actual steady floor pays for idle capacity on every low-usage day. From there, the commitment size is a risk trade-off, not a fixed rule: a stable, mature workload with a long recent history of holding above that floor supports committing close to the full floor, while a workload still changing shape (recent re-architecture, aggressive growth, planned migration) justifies leaving more of the floor on-demand or covering it with a flexible, spend-based commitment instead of a rigid resource-level one, specifically because the risk being managed is "commitment outlives the workload's actual shape," not "commitment size in the abstract."

Worked example

A team's steady-state EC2 fleet held at a minimum of 40 instances of a given family over the last 4 months, with normal weekday peaks around 55 and occasional bursts to 70. The 40-instance floor is the commitment candidate, not the 55-instance average and not the 70-instance peak: committing at 55 would mean paying the commitment rate for capacity that isn't reliably used on quieter days, and any spike above 40 (up to and including the 70-instance bursts) is served by on-demand or spot capacity regardless of the commitment size. If this workload is expected to stay on the same instance family for the full term, an EC2 Instance Savings Plan or Standard RI sized to 40 instances captures the largest discount available on that stable floor; if a re-platforming project is likely to change instance family within the year, a Compute Savings Plan sized to the equivalent dollar-per-hour spend protects the same floor's discount while surviving the family change.

Trade-offs and pitfalls

The most common mistake is committing to the peak or the average instead of the floor, which either overpays for capacity that isn't reliably used or, worse, sizes a "safe" commitment so conservatively it captures almost none of the available discount. The second is choosing the narrowest, highest-discount resource-level commitment for a workload that's still changing shape, and then discovering the commitment doesn't match the new instance family, wasting real money for the rest of the term. The third, specific to the flexible/spend-based products, is assuming "flexible" means "no attention needed": a spend-based commitment still needs the underlying usage to stay above the committed dollar level, or the unused portion is still paid for and simply not applied to any usage.

Picking SLO targets

Start from what actually affects the business and users, not an arbitrary round number: for a billing API, a slow or failed checkout call directly risks lost revenue and support load, so the SLO (Service Level Objective: the internal target you set for a metric like latency or availability) should track what customers actually feel. Anchor the target against your own baseline distribution first (don't set a target far tighter than what the system currently, realistically achieves without major investment), and set it off percentiles, not the average, since the average can look healthy while a real slice of checkout attempts are painfully slow.

Metrics to use

• Latency: p95 and p99 response time for the billing endpoints, being explicit about whether failed or timed-out requests are included in the latency population or tracked separately (excluding them silently is a common way an SLI (Service Level Indicator: the actual measured metric, e.g. p99 latency, that gets compared against the SLO) quietly stops meaning what people think it means).
• Availability: the percentage of requests that complete successfully (not 5xx, not timed out), often tracked as its own SLO alongside latency rather than folded into it.

Translating business impact into thresholds

Look at your own funnel or conversion data for where checkout abandonment starts climbing as latency increases, and set the latency ceiling with real headroom below that point, not right at the edge of where users start leaving. The same logic applies to availability: estimate what a percentage point of failed billing requests costs in lost transactions and support tickets, and use that to justify how tight (and how expensive) the target should be, since tighter targets generally cost more engineering effort and infrastructure to sustain.

Error budgets

The error budget is simply what's left over: 1 minus the SLO. A 99.9% availability SLO over a 30-day month leaves a 0.1% error budget:

30 days = 43,200 minutes
0.1% of 43,200 minutes = 43.2 minutes

That's roughly 43 minutes of full-equivalent downtime allowed per month, whether spent as one incident or many smaller partial-degradation periods that add up.

How the error budget should drive release cadence and incident response

• Release cadence: while budget remains healthy, ship at normal (or even accelerated) velocity. As the budget gets consumed, slow down: require canary rollouts (releasing a change to a small slice of traffic or users first, before rolling out to everyone), extra review, or freeze risky changes entirely until the budget recovers. This turns "reliability" from a vague value into a concrete, negotiable resource that product and engineering both watch.
• Incident response: alert on burn RATE (how fast the budget is being consumed) rather than only on absolute SLO breach, so a fast, severe burn pages immediately while a slow, minor burn becomes a lower-urgency ticket. Postmortems and remediation priority should also weigh directly against how much budget an incident consumed, giving the team a shared, quantitative way to decide whether to prioritize a reliability fix over the next feature.

Direct answer

Build one VPC, then iterate AZs with for_each (not count) so each subnet is keyed by AZ name instead of a list index. That matters because with count, removing an AZ from the middle of the list shifts every subsequent index and Terraform proposes destroying and recreating subnets that didn't actually change. Keying the resource by AZ name only fixes half of that: the CIDR block assigned to each subnet also has to be tied to the AZ itself, not to that AZ's position in a list, so each AZ gets a stable, explicitly-assigned subnet index (a small az => netnum map) instead of index(var.azs, each.value). NAT gateway creation is gated behind a boolean input and created per AZ only when enabled. Outputs expose subnet IDs and route table IDs as maps keyed by AZ, so a caller can look up exactly the subnet it needs by name instead of guessing a list position.

Approach

• One aws_vpc, sized by var.vpc_cidr.
• One aws_subnet per AZ via for_each = var.az_netnums (a map of AZ name to a stable subnet index), with each subnet's CIDR carved out via cidrsubnet(var.vpc_cidr, var.subnet_newbits, each.value), where each.value is that AZ's fixed index, not its position in a list.
• One public route table (shared, routes to the internet gateway) and one private route table per AZ (routes to that AZ's NAT gateway only if NAT is enabled; otherwise no default route at all).
• NAT gateways and their EIPs are for_each-keyed the same way as the subnets, and only exist when var.enable_nat_gateway = true.

The code below is illustrative HCL meant to be read, not executed output from a real terraform apply (no CLI available here to run it against real state).

variables.tf

variable "vpc_cidr" {
  type        = string
  description = "CIDR block for the VPC, e.g. 10.0.0.0/16."
}

variable "az_netnums" {
  type        = map(number)
  description = "Map of AZ name to a stable subnet index used by cidrsubnet, e.g. { \"us-east-1a\" = 0, \"us-east-1b\" = 1 }. Assign each AZ its number once, in the order it was first added, and treat this map as append-only: never renumber or reuse an existing AZ's index, even after that AZ is removed."
}

variable "subnet_newbits" {
  type        = number
  description = "Additional bits for cidrsubnet when carving subnets out of vpc_cidr (e.g. 8 turns a /16 into /24s)."
}

variable "enable_nat_gateway" {
  type        = bool
  default     = false
  description = "Whether to create one NAT gateway per AZ for private-subnet egress."
}

main.tf

resource "aws_vpc" "this" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags = { Name = "vpc-module" }
}

resource "aws_subnet" "this" {
  for_each          = var.az_netnums
  vpc_id            = aws_vpc.this.id
  availability_zone = each.key
  cidr_block        = cidrsubnet(var.vpc_cidr, var.subnet_newbits, each.value)
  tags = { Name = "subnet-${each.key}" }
}

resource "aws_internet_gateway" "this" {
  vpc_id = aws_vpc.this.id
}

resource "aws_eip" "nat" {
  for_each = var.enable_nat_gateway ? var.az_netnums : {}
  domain   = "vpc"
}

resource "aws_nat_gateway" "this" {
  for_each      = var.enable_nat_gateway ? var.az_netnums : {}
  allocation_id = aws_eip.nat[each.key].id
  subnet_id     = aws_subnet.this[each.key].id
  tags = { Name = "nat-${each.key}" }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.this.id
}

resource "aws_route" "public_internet" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id              = aws_internet_gateway.this.id
}

resource "aws_route_table_association" "public" {
  for_each       = aws_subnet.this
  subnet_id      = each.value.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table" "private" {
  for_each = var.az_netnums
  vpc_id   = aws_vpc.this.id
  tags = { Name = "rt-private-${each.key}" }
}

resource "aws_route" "private_egress" {
  for_each               = var.enable_nat_gateway ? aws_route_table.private : {}
  route_table_id         = each.value.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.this[each.key].id
}

resource "aws_route_table_association" "private" {
  for_each       = aws_subnet.this
  subnet_id      = each.value.id
  route_table_id = aws_route_table.private[each.key].id
}

outputs.tf

output "subnet_ids" {
  value = { for k, s in aws_subnet.this : k => s.id }
}

output "route_table_ids" {
  value = {
    public  = aws_route_table.public.id
    private = { for k, rt in aws_route_table.private : k => rt.id }
  }
}

Key design decisions

• for_each keyed by AZ name, not count, and the CIDR index keyed the same way. The AZ string is a stable, meaningful for_each key, but that alone isn't enough: cidrsubnet's index argument also has to come from a fixed az => netnum map (var.az_netnums) instead of index(var.azs, each.value), or removing/reordering an AZ silently recomputes every other AZ's index() result and reshuffles their CIDR blocks, forcing a destroy/recreate on subnets that didn't actually change (cidr_block is a ForceNew attribute on aws_subnet). With both the resource key and the CIDR index pinned to the AZ itself, adding or removing an AZ only touches the resources tied to that specific key. az_netnums is append-only: never reuse or renumber an existing AZ's index, even after that AZ is removed.
• A "private" subnet with NAT disabled gets no default route at all, rather than falling back to routing straight to the internet gateway. Routing a "private" route table to the IGW when NAT is off would silently make those subnets not actually private, which is a common mistake worth naming explicitly if you see it in someone else's module.
• Outputs are maps keyed by AZ, not lists. A consuming module or root config wants module.vpc.subnet_ids["us-east-1a"], not module.vpc.subnet_ids[0] and a separate assumption about ordering.

Complexity

Resource count scales linearly with the number of AZs: for n AZs you get n subnets, n private route tables, and (if NAT is enabled) n EIPs and n NAT gateways, so the plan/apply graph is O(n) in both size and, roughly, cost. cidrsubnet with subnet_newbits = 8 splits a /16 into /24s, giving up to $2^{8} = 256$ possible subnet blocks, of which this module only allocates n of them (one per AZ), leaving room to grow the AZ list later without re-carving existing subnets, since each AZ's block is computed from its own fixed netnum in az_netnums, not from a list position.

Edge cases

• Fewer than 2 AZs: the module still works, but a single-AZ deployment has no real availability benefit; worth flagging back to whoever is calling it.
• The az_netnums map is reordered in the config file, or an AZ is removed: since both the for_each key and the cidrsubnet index come from az_netnums rather than a list position, every remaining AZ keeps its assigned netnum and therefore its exact cidr_block; HCL map literals have no meaningful order in the first place, so reordering entries in the file is purely cosmetic. Removing an AZ destroys only that AZ's own subnet, route table, and (if enabled) NAT gateway and EIP; its netnum must not be reassigned to a different AZ afterward, or that AZ's CIDR (and, since cidr_block is ForceNew, its subnet) would change too.
• enable_nat_gateway flips from true to false on an existing deployment: this destroys every NAT gateway and EIP and removes the private route tables' default routes, cutting off private-subnet egress. That is a real production-impacting change, not just a cosmetic one, and belongs behind the same "review the plan's destroy count" discipline as any other production change.
• subnet_newbits too small for the number of AZs: cidrsubnet errors out at plan time if you ask for more subnet bits than the parent CIDR has room for, so this fails fast rather than silently.

Trade-offs and pitfalls

Making this production-ready and reusable pulls in a few things beyond the base module:

• VPC Flow Logs: add an aws_flow_log resource pointed at CloudWatch Logs or an S3 bucket, gated behind its own optional variable, so network visibility is available to callers who want it without forcing it (and its cost) on every caller.
• An S3 gateway endpoint: add an aws_vpc_endpoint of type Gateway for S3, associated with the private route tables. This lets private-subnet workloads reach S3 without traversing the NAT gateway, which avoids NAT data-processing charges and keeps that traffic off the public path entirely.
• Versioning, testing, and documentation for reuse: tag the module with semantic versions in git (v1.2.0) and have consumers pin to a version (source = "git::...?ref=v1.2.0" or a registry version constraint) rather than tracking a branch. Write a README documenting every input and output with an example call. Test with terraform test (native since Terraform 1.6) or Terratest against a real or LocalStack-backed provider, asserting that toggling enable_nat_gateway actually changes the private route tables' routes and that the exposed output maps contain exactly the AZ keys passed in.
• One-shared-NAT-gateway alternative: creating one NAT gateway per AZ (as above) maximizes availability but multiplies NAT cost by the AZ count; a single shared NAT gateway in one AZ is cheaper but makes every other AZ's egress depend on that one AZ staying healthy, worth surfacing as an explicit cost/resilience trade-off rather than a silent default.

For a Kubernetes production deployment specifically, the pipeline's job is to make sure nothing gets scheduled onto the cluster unless it's been scanned, signed, and can be verified as such at the moment Kubernetes actually tries to run it, using named, concrete tooling rather than abstract components.

Recommended tools and the flow

flowchart LR
  Src[Source] --> Tekton[Tekton pipeline]
  Tekton --> Trivy[Trivy scan]
  Trivy --> Cosign[cosign/notation sign]
  Cosign --> Registry[Container registry]
  Registry --> ArgoCD[ArgoCD sync]
  ArgoCD --> Gatekeeper[OPA/Gatekeeper admission]
  Gatekeeper -->|signature + scan verified| Cluster[K8s cluster]

Tekton (or an equivalent Kubernetes-native pipeline engine) builds and orchestrates the pipeline itself, running natively alongside the cluster it deploys to. Trivy scans the built image for known vulnerabilities before it's allowed to be signed; a CRITICAL finding blocks the pipeline before signing ever happens, since there's no point cryptographically attesting to an image you already know is unsafe to ship. cosign (or notation, Microsoft's equivalent) signs the scanned image, again preferably keylessly via OIDC (OpenID Connect) rather than a long-lived key the pipeline has to protect. ArgoCD reconciles the desired state from Git to the cluster, and OPA/Gatekeeper enforces, at admission time, that any image being scheduled carries a valid signature from the expected signer and comes from an approved registry; an unsigned or unverifiable image is rejected at the Kubernetes API layer itself, not merely flagged.

Key and secret management

Signing keys (or, for keyless signing, the OIDC trust configuration binding which CI identity is allowed to produce a valid signature) are the highest-value asset in this whole chain, since anything with signing authority can make an attacker-controlled image look legitimate; keyless signing via Sigstore's Fulcio removes the long-lived-key management burden entirely, tying signing authority to short-lived, OIDC-verified certificates instead. Application secrets needed at runtime are handled separately from signing, typically via the Secrets Store CSI driver or a Vault Agent injecting them at pod startup, scoped per namespace.

Automated rollback and audit trails

ArgoCD's own sync history gives a built-in rollback mechanism (reverting to a prior, known-good commit's manifests triggers a resync to that state); pairing this with the admission controller's own decision log (every allow and deny decision, with the reason) gives a complete audit trail of both what was deployed and why any given deployment was allowed or blocked.

Trade-offs

Enforcing signature verification at the admission-controller layer, rather than earlier in the pipeline alone, is what actually prevents an unsigned or tampered image from running even if it somehow bypassed the CI pipeline entirely (a compromised registry push, for instance); the cost is that a misconfigured admission policy can block legitimate deploys just as effectively as it blocks malicious ones, so the policy itself needs its own careful testing and staged rollout before being made mandatory cluster-wide.

Direct answer

PaaS sits between IaaS and serverless on a single spectrum of "how much of the running instance do I still manage," rather than off to the side as an unrelated third category. IaaS gives you a virtual machine and you build everything above it. PaaS gives you a managed runtime and deployment target: you push code and the platform runs it continuously. Serverless goes one step further than PaaS by removing the concept of a continuously running instance altogether, scaling to zero and billing per invocation instead.

How PaaS differs from each neighbor

Versus IaaS. On IaaS you decide the OS, patch it, install a runtime, and configure a process manager, and an instance typically stays running whether or not it's serving traffic. On PaaS you supply application code or a container image and a deployment manifest; the platform handles OS patching, runtime installation, health checks and restarts, and autoscaling policy configuration replaces manual instance management.

Versus serverless. PaaS still asks you to think in terms of how many instances and what size, even if it autoscales them for you, and instances generally stay warm between requests. Serverless removes the instance concept entirely, scales per invocation including to zero, and bills per invocation and duration rather than per provisioned hour.

Examples for the two workload shapes the question asks about

Web applications: AWS Elastic Beanstalk, Azure App Service, and Google App Engine let you push source code and get a managed, autoscaled web service with a built-in load balancer.

Container-based workloads: managed container-runtime platforms such as Google Cloud Run or AWS App Runner take a container image, giving you the middleware and dependency control of a container, while the platform still handles the OS, orchestration, and autoscaling, including scaling to zero when idle. This is why these products sit right at the boundary between PaaS and serverless.

Operational trade-offs of choosing PaaS

Instead of IaaS: you give up fine-grained OS and kernel control and the ability to install arbitrary system-level software, and you gain freedom from patch management and instance-level monitoring and alerting.

Instead of serverless: you give up the cost benefit of scaling fully to zero for spiky or idle workloads and take on slightly more capacity-planning thinking, how many instances, what size, but you gain a simpler mental model for stateful or long-running processes and typically fewer cold-start surprises (the extra delay a request hits when it wakes an idle instance or function that has to initialize before it can respond).

Worked example

A small team is choosing between the three for a customer-facing web app with steady, moderate daily traffic that also needs a custom authentication middleware wrapping every request. Raw IaaS would mean owning OS patches for no real benefit, since nothing about the workload needs kernel-level control. Pure serverless functions would technically work but would mean re-architecting the middleware into a function-per-endpoint shape, adding real engineering cost for a workload that isn't particularly spiky. A container-based PaaS product lets the team package the existing middleware into the container image unchanged, deploy it once, and let the platform handle scaling and OS management, which is the best fit given steady traffic and existing middleware investment.

Trade-offs and pitfalls

The main pitfall is picking PaaS reflexively as "the safe middle choice" without checking whether the workload is actually spiky, where serverless's per-invocation billing would save real money, or genuinely needs OS-level control, where PaaS's restrictions become a real blocker, for example a workload needing raw socket access a managed platform won't expose. A second pitfall is assuming all PaaS products are equally restrictive: a container-based PaaS product gives you a much larger slice of control over the runtime and middleware than a source-code-only PaaS product, so the choice between PaaS flavors matters almost as much as the IaaS-versus-PaaS-versus-serverless choice itself.

Direct answer

Azure SQL Database (single database or elastic pool) is the right default for a new application that doesn't need instance-level SQL Server features: Microsoft owns patching, backups, and most high availability (HA) entirely. Azure SQL Managed Instance (MI) is the migration-friendly middle ground: near-complete SQL Server surface-area compatibility (SQL Server Agent, Common Language Runtime/CLR, cross-database queries and transactions, linked servers, Service Broker) inside your own Virtual Network (VNet), still as a managed service. SQL Server on an Azure virtual machine (VM) is the last resort for workloads that need something even MI doesn't support, or need an exact SQL Server version, edition, or operating-system-level control pinned, at the cost of owning patching and HA yourself.

Comparison

Evaluation checklist

When choosing among the three for a given workload, walk through, in order: (1) which SQL Server compatibility features the application actually calls, since that alone can rule out SQL Database; (2) the team's appetite for owning patching and HA operations; (3) the target Recovery Time Objective (RTO, how long the application can stay down before it must be back up) and Recovery Point Objective (RPO, how much recent data you can afford to lose); (4) the budget model, Database Transaction Unit (DTU) or virtual core (vCore) consumption pricing versus a VM plus a SQL Server license.

Worked example

A legacy on-premises application uses SQL Server Agent jobs, several cross-database transactions, and one linked server to a partner system. Azure SQL Database supports none of that by default (cross-database access there is limited to a workaround called elastic query, not true cross-database transactions), so a rewrite would be required to land it there. SQL Managed Instance supports SQL Agent, cross-database transactions, and linked servers directly, letting the migration proceed largely unchanged; the real migration effort shifts to the VNet integration itself, MI requires a dedicated subnet, correctly sized and with no other resource types placed in it, plus Domain Name System (DNS) and network security group planning before cutover. If the application also depended on something MI still doesn't support (an unusual Component Object Model/COM object hosted in-process, for example), SQL Server on a VM would be the fallback, accepting full patching and HA ownership in exchange for full compatibility.

Trade-offs and pitfalls

• Picking SQL Database first because it's the simplest to provision, without checking for SQL Agent or linked-server dependencies, is a common way to discover a rewrite is needed mid-migration.
• MI's VNet requirement is not optional and has real subnet-sizing and DNS-planning work attached; underestimating that work is the most common MI migration surprise.
• Running SQL Server on a VM to dodge MI's constraints reintroduces the exact patching and HA burden the other two options were built to remove; only choose it when a specific, named feature gap forces it.

A provenance attestation is fundamentally a signed statement, so the implementation has two halves that both have to be exactly right: building a deterministic payload to sign, and signing it in a way a verifier can independently reproduce and check.

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
	"fmt"
	"os"
	"time"
)

type FileDigest struct {
	Path   string `json:"path"`
	Sha256 string `json:"sha256"`
}

// Field order is fixed by the struct definition, which is what makes
// json.Marshal deterministic for this payload: the same logical content
// always produces the same bytes to sign and to re-verify against.
type Attestation struct {
	Inputs    []FileDigest `json:"inputs"`
	Outputs   []FileDigest `json:"outputs"`
	Timestamp string       `json:"timestamp"`
	BuilderID string       `json:"builderId"`
	Signature string       `json:"signature"`
}

func signPayload(payload []byte, priv *rsa.PrivateKey) (string, error) {
	digest := sha256.Sum256(payload)
	sig, err := rsa.SignPKCS1v15(rand.Reader, priv, crypto.SHA256, digest[:])
	if err != nil {
		return "", fmt.Errorf("signing attestation payload: %w", err)
	}
	return base64.StdEncoding.EncodeToString(sig), nil
}

func loadPrivateKey(pemPath string) (*rsa.PrivateKey, error) {
	data, err := os.ReadFile(pemPath)
	if err != nil {
		return nil, fmt.Errorf("reading key file: %w", err)
	}
	block, _ := pem.Decode(data)
	if block == nil {
		return nil, fmt.Errorf("no PEM block found in %s", pemPath)
	}
	key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
	if err != nil {
		return nil, fmt.Errorf("parsing private key: %w", err)
	}
	return key, nil
}

func buildAttestation(inputsPath, outputsPath, keyPath string) (*Attestation, error) {
	inputsRaw, err := os.ReadFile(inputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading inputs file: %w", err)
	}
	var inputs []FileDigest
	if err := json.Unmarshal(inputsRaw, &inputs); err != nil {
		return nil, fmt.Errorf("parsing inputs JSON: %w", err)
	}

	outputsRaw, err := os.ReadFile(outputsPath)
	if err != nil {
		return nil, fmt.Errorf("reading outputs file: %w", err)
	}
	var outputs []FileDigest
	if err := json.Unmarshal(outputsRaw, &outputs); err != nil {
		return nil, fmt.Errorf("parsing outputs JSON: %w", err)
	}

	builderID := os.Getenv("BUILDER_ID")
	if builderID == "" {
		return nil, fmt.Errorf("BUILDER_ID environment variable is not set")
	}

	priv, err := loadPrivateKey(keyPath)
	if err != nil {
		return nil, err
	}

	unsigned := Attestation{
		Inputs:    inputs,
		Outputs:   outputs,
		Timestamp: time.Now().UTC().Format(time.RFC3339),
		BuilderID: builderID,
	}

	payload, err := json.Marshal(unsigned)
	if err != nil {
		return nil, fmt.Errorf("canonicalizing payload: %w", err)
	}

	sig, err := signPayload(payload, priv)
	if err != nil {
		return nil, err
	}

	unsigned.Signature = sig
	return &unsigned, nil
}

func main() {
	if len(os.Args) != 4 {
		fmt.Fprintln(os.Stderr, "usage: provenance <inputs.json> <outputs.json> <private_key.pem>")
		os.Exit(2)
	}
	att, err := buildAttestation(os.Args[1], os.Args[2], os.Args[3])
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}
	out, _ := json.MarshalIndent(att, "", "  ")
	fmt.Println(string(out))
}

Deterministic JSON serialization

The struct's field order is fixed at compile time by its definition, so json.Marshal always emits fields in the same order for the same logical content; this is why the payload is built as a typed Attestation struct rather than a map[string]interface{}, since Go's JSON encoding of a map sorts keys alphabetically by default but a struct preserves declaration order, either of which is deterministic on its own, but mixing the two within one payload risks a verifier reconstructing a different byte layout than the signer used. The signature field itself is excluded from the signed payload (set to empty string, or omitted, at signing time) since the signature obviously can't be part of what it's signing over.

Verified

Compiled with go build. Generated a real RSA keypair with OpenSSL, ran the program to produce a signed attestation, then wrote an independent Go verifier that re-reads the attestation, strips the signature field, re-marshals the remaining struct exactly as the signer did, and calls rsa.VerifyPKCS1v15 against the public key: verification succeeded. A tamper test, flipping one character in an input's SHA-256 digest and re-running verification, correctly failed with a signature-mismatch error, confirming the scheme actually detects tampering rather than passing regardless of content.

Trade-offs

Using PKCS1v15 padding and a single RSA key here is simple and fully supported by the Go standard library alone (no external dependency), matching the question's constraint; a production system would more likely use Sigstore's keyless signing (short-lived, OIDC (OpenID Connect)-backed certificates) to avoid the operational burden of protecting a long-lived private key file: that file has to be generated once and then protected for its entire lifetime, encrypted at rest, access-restricted, rotated on a schedule, and revoked immediately if it is ever exposed, and if it does leak, whoever holds it can forge valid attestations indefinitely, or at least until the leak is discovered and the key revoked, with no built-in record of who actually produced a given signature. Keyless signing removes that persisted secret entirely: at sign time the CI runner exchanges a short-lived OIDC token (proof that this exact signing request came from this exact workflow run) for a certificate from a public certificate authority (Sigstore's Fulcio), signs with an ephemeral key that is discarded the moment signing completes, and records the signature alongside the certificate and a transparency-log entry (Rekor) that anyone can later audit. There is no long-lived key file to steal, back up, or rotate, at the cost of depending on the OIDC identity provider and Sigstore's infrastructure being reachable at sign time. Container image signing, discussed separately in this topic, makes the identical trade-off: a persisted key you must guard forever versus an ephemeral, OIDC-backed identity with nothing to steal.

Direct answer

Five metrics cover the core questions a responder needs answered first: is the database slow (a high-percentile query or write latency), is it actually serving load correctly (throughput and error rate), is it running out of room (connection and CPU or memory saturation), is its working set still fitting in memory (buffer or cache hit rate), and, if replicas exist, is it falling behind (replication lag). Pick alert thresholds relative to the workload's own historical baseline and its SLO (service-level objective, an internal target for how the system should perform), not a generic industry number, and expect an OLTP workload (online transaction processing, many small, fast operations) and an OLAP workload (online analytical processing, fewer, larger, longer-running queries) to need different baselines for the same metric.

Structured elaboration

On a managed database

A managed relational database service typically removes OS shell access, but the same five categories are still visible, just organized through the provider's own console or API instead of OS commands: host-level metrics (CPU, memory, disk, network, exposed as instance-level monitoring), database-level metrics (connections, buffer hit ratio, via the engine's own views proxied through the console), query-level metrics (top SQL by load, via a performance-insight feature layered on top rather than raw direct access to the engine's own statistics views), and replication metrics (lag, as a first-class console metric since the provider manages the replica topology). Losing shell access doesn't mean losing observability; it means the same five things live in a different namespace. Know where each one lives before an incident, not during one.

OLTP versus OLAP baselines

The same five metrics need structurally different thresholds depending on workload shape. An OLTP system, many small, fast, latency-sensitive operations, should alert tight on p99 latency, since milliseconds matter, and treat connection saturation as urgent, since a starved OLTP system fails user-facing requests immediately; its throughput baseline is usually fairly steady and predictable. An OLAP system, fewer, larger, longer-running analytical queries, should alert loose on p99 latency, since a single query legitimately taking minutes is normal, but tight on sustained resource saturation, CPU, memory, or I/O held near capacity for an extended period is the actual OLAP health signal, since a single slow query is expected but the whole system staying pinned means queuing is building, and on queue depth (how many queries are waiting to run) rather than on individual query latency. Applying an OLTP-shaped p99-latency alert to a warehouse would page constantly on completely normal behavior, and applying an OLAP-shaped hourly-average view to an OLTP system would miss a real user-facing latency spike entirely.

Worked example

Turning the p99-latency alert condition into concrete numbers, showing why a baseline-relative threshold rather than a fixed one is the right shape:

p99_baseline_ms = 42   # trailing 7-day baseline
multiple = 3
threshold_ms = p99_baseline_ms * multiple
print(threshold_ms)

126

At a 42ms trailing baseline, the alert fires above 126ms. If legitimate growth naturally moves the baseline to, say, 55ms next quarter, the threshold recalculates to 165ms automatically, whereas a fixed ">100ms" rule set today would already be uncomfortably close to normal, healthy behavior by then and would need someone to remember to revisit it.

Trade-offs and pitfalls

• A fixed threshold copied from a blog post or another team's dashboard, rather than derived from this workload's own baseline, is the most common reason alerts are either constantly noisy (too tight for this workload's normal variance) or dangerously quiet (too loose, so real incidents don't page). Baseline-relative thresholds avoid this, but need enough historical data to compute a meaningful baseline; don't set them from one week of a system still finding its steady state right after launch.
• Alerting only on averages instead of a high percentile hides exactly the tail-latency problems users actually feel; a p99 spike can coexist with a perfectly healthy-looking average if it affects a small fraction of requests, which is often the fraction most likely to matter most.
• Five metrics is a floor, not a ceiling. This is the minimum viable health picture, not a complete observability strategy; deeper investigation, specific slow-query identification, lock waits, log growth, still needs its own dedicated tooling once one of these five flags a problem.

Direct answer

A retro after a release with repeated design-engineering friction should produce two things: an honest, specific account of where the handoff actually broke down, not a vague 'communication issues,' and a small number of concrete process changes, each with an owner and a way to tell in a quarter whether it worked. Running it well means separating fact-finding from diagnosis, and diagnosis from blame.

Structured elaboration

Design principles for the session

• Facts before diagnosis: start from a timeline of what actually happened (spec dates, handoff dates, bug counts, points where implementation and design diverged), not from opinions about who was at fault.
• Root cause, not the nearest symptom: 'engineering didn't follow the spec' is a symptom; the root cause might be that the spec didn't capture edge-case states, or that both sides were working from different versions of a shared design system mid-migration.
• Few, high-leverage commitments: two or three process changes people will actually do beat ten action items that quietly get dropped.
• Everyone leaves with the same understanding of what changed, not just what went wrong.

A workable structure

One illustrative shape, adaptable to a team's own rhythm:

What 'actually changes how the two teams work' looks like

The output isn't a list of intentions, it's a specific artifact or habit that exists after the meeting and didn't before: a shared checklist embedded in the handoff process, an automated check that catches a class of mismatch before it ships, or a standing short sync during implementation windows. Whatever it is, it needs a way to tell if it worked, not just that it happened.

Worked example

One team's root cause turned out to be that design tokens (colors, spacing values) were maintained in the design tool but hand-copied into code, so drift was inevitable and nobody could tell which side was 'correct' when they disagreed. The concrete fix was an automated export from the design tool into the codebase, checked by both a design reviewer and a frontend reviewer before merge, plus a short recurring sync during active implementation. A quarter later, the team had a real signal that it worked: noticeably fewer visual-mismatch comments on pull requests and less late-stage rework than the release that triggered the retro. The same root-cause pattern shows up in other domains as a hand-copied data contract or config value instead of a design token, so the same fix shape (automate the handoff, add a lightweight check, add a short sync during the risky window) generalizes well beyond design and engineering specifically.

Trade-offs and pitfalls

• A retro that produces ten action items usually produces zero completed ones; prioritizing ruthlessly matters more than being thorough.
• If the room jumps straight to solutions or blame instead of facts first, the real root cause, often structural or tooling-related rather than a person's failure, never surfaces.
• A retro that isn't revisited becomes theater. Put the check-in on the calendar before the room disperses, not as a vague intention afterward.
• Watch for a fix that only addresses this specific release's symptom (a one-off manual double-check) rather than the structural cause; it holds for one cycle and then quietly stops happening.