Microsoft Azure Services and Architecture Questions

Requirements & constraints:

• Prevent vulnerable/unsigned images reaching AKS; enforce least privilege for registry access; generate SBOMs; integrate SAST and build-time scanning; support automated CI/CD with policy gates and manual exceptions; logging/auditability; low latency deploys.

High-level architecture:

• Dev repo (GitHub/Azure Repos) → CI (GitHub Actions/Azure Pipelines) → Build & Scan → Push to private ACR → Signing & promotion → AKS pull via RBAC + admission control.

Core components & flow:

1. Pre-commit / PR: SAST (e.g., Semgrep, SonarQube) runs in CI. PR fails on high/critical code findings.
2. Build stage:
   • Reproducible build in isolated runner; build image with buildkit.
   • Generate SBOM (Syft/OSV or in-toto) and attach as image artifact.
   • Run image-as-code checks (hadolint, dockerfile-linter).
3. Image scanning & vulnerability assessment:
   • Use Clair/Trivy/Anchore integrated in CI to scan built image layers and SBOM against CVE feeds and vendor advisories.
   • Automate policy decision: critical CVEs => block; high => block by default unless approved; medium/low => warn.
4. Image signing/content trust:
   • Use Sigstore/Notary v2 with short-lived keys or Azure Confidential Registry signing. Sign image and SBOM; store signatures in ACR content trust.
5. Registry security (ACR):
   • Private ACR with RBAC: least-privilege roles for CI, deployer, devs; disable anonymous pulls.
   • Network rules: firewall, service endpoints, private link to AKS & CI runners.
   • Anti-malware and retention policies; enforce immutable tags for promoted images.
6. Promotion & tagging:
   • Use immutable tags (build-<sha>), stage promotion (dev → staging → prod) only after scans & signatures present.
7. Runtime admission in AKS:
   • Use OPA/Gatekeeper or Kyverno webhook policies to:
     • Require presence of valid Sigstore/Notary signature.
     • Verify SBOM exists and check for CVE policy compliance at admission.
     • Block images not from allowed ACR or unsigned/vulnerable images.
   • Use Azure AD workload identities for node/pod pull credentials via ACR integration.
8. Monitoring & audit:
   • Centralized logs (Azure Monitor/Log Analytics), alerting on blocked deployments, new critical CVEs, or signature failures.

Handling discovered critical vulnerabilities in production images:

• Triage: automated alert to SRE/security on detection (from runtime scanner or external CVE feed). Correlate SBOM to impacted deployments.
• Immediate mitigation:
  • If vulnerability exploitable in current runtime, use AKS admission controller to quarantine by:
    • Patch by deploying a replacement image (built from patched code).
    • If urgent: use image policy to mark image as blocked and rollout emergency CVE-fix image; or cordon/evict workloads and run patched pods.
    • Apply network micro-segmentation or temporary pod-level AppArmor/SELinux restrictions and limit privileges.
• Root cause & fix:
  • Create an urgent ticket for dev team to patch dependency or rebuild base image. Use reproducible build with dependency upgrade.
  • Rebuild, re-scan, re-sign, and promote through pipeline.
• Post-incident:
  • Rotate keys if signature compromise suspected.
  • Run forensic audit from logs and SBOM history.
  • Update policies: lower threshold, add blocking for similar classes, improve dependency scanning (supply-chain tools).
    Trade-offs & considerations:
• Strict gating increases lead time; mitigate with fast scanners and caching, progressive rollouts, and exception workflows.
• Prefer automated signing (Sigstore) and short-lived keys to reduce key management overhead.
  This design enforces defense-in-depth across code, build, registry, and runtime with automation to minimize human error and time to remediate critical vulnerabilities.

Requirements & constraints:

• Functional: SaaS multi-tenant API + web UI, per-tenant data isolation, tenant admin controls.
• Non-functional: strong isolation options, Azure AD-based auth, encryption-at-rest/in-transit, auditability, compliance (GDPR, SOC2), scalability.

High-level architecture:

• Front door (Azure Front Door) → API Gateway (Azure API Management) → App Services / AKS (stateless) → Tenant data stores (Azure SQL / Cosmos DB / Blob) → Key Vault/Managed HSM, Azure AD for auth, Log Analytics & Sentinel for auditing.

Tenant isolation options (trade-offs & recommendation):

1. Shared schema with tenant_id:
   • Pros: lowest cost, simplest to scale, single deployment.
   • Cons: weakest isolation, harder per-tenant compliance.
   • Use when low security requirements and many small tenants.
2. Schema-per-tenant:
   • Pros: stronger logical isolation, easier per-tenant backups/restore.
   • Cons: management complexity at large scale.
3. DB-per-tenant:
   • Pros: strongest isolation, ability to place DBs in different regions/respect data residency, easier per-tenant SLAs.
   • Cons: operational overhead, connection limits, cost.
     Recommendation: support hybrid model—default shared-schema for small tenants; migrate high-risk/high-revenue tenants to schema-per-tenant or DB-per-tenant (automated provisioning, use Elastic Pools for cost control).

Authentication & Authorization (Azure AD):

• Use Azure AD (OIDC/OAuth2). Two app models:
  • Multi-tenant application (single app registration): convenient, single codebase, supports users from any Azure AD tenant after admin/user consent. Use when you need single app identity and central permission model.
  • Tenant-isolated apps (per-tenant app registrations): stronger per-tenant control, allows tenant admin-specific permissions and granular app roles, but operationally heavier.
    Recommendation: use a multi-tenant app registration for the SaaS product with tenant admin consent flow and provisioning via SCIM for user/group sync. Implement:
  • ID tokens for auth, access tokens for APIs.
  • AppRoles and dynamic role assignment or Azure AD groups; include tenant_id and roles in JWT claims.
  • Tenant-scoped admin role that can manage tenant settings.
  • Token validation with MSAL libraries and caching.

Authorization patterns:

• Centralized policy service (policy microservice) that enforces tenant-scoped RBAC. Use claims-based checks and ABAC for fine-grained controls (resource tags = tenant, owner).

Encryption & Key Management (Azure Key Vault / Managed HSM):

• Layered approach: platform-managed keys for convenience; customer-managed keys (CMK) for compliance.
• Envelope encryption: Data encrypted with data keys (DEKs), DEKs encrypted by tenant master keys (KEKs) stored in Key Vault/HSM.
• Per-tenant keys vs shared key:
  • Per-tenant keys: best for legal/data separation & key rotation per tenant; supports tenant-level key revocation and BYOK. Recommend for high-security tenants.
  • Shared key: simpler but single blast radius; acceptable for low-risk tenants.
• Use Azure Key Vault with RBAC and access policies; use Managed HSM for FIPS/HSM-backed keys when required. Enable soft-delete, purge protection, logging of key access, and key rotation policies (automate via Azure Functions/Automation).

Compliance, auditing & observability:

• Audit logging: enable Diagnostic Settings on App Services, SQL, Key Vault to stream to Log Analytics, Storage, and Event Hub; forward to Azure Sentinel or SIEM.
• Tenant-scoped logging: include tenant_id in all logs and telemetry; store indices/partitions by tenant to produce tenant-level audits and export capabilities.
• Tenant-scoped RBAC & admin audit: implement tenant admin role in app (backed by Azure AD role/group) and log all admin actions to immutable append-only storage (e.g., storage account with ACLs and immutability policy).
• Data residency: provision DBs per-region for tenants requiring locality; use tags & Azure Policy to enforce.
• Backup & restore: per-tenant backup strategy; for DB-per-tenant straightforward; for shared-schema take logical exports with tenant filter.
• Compliance controls: encryption at rest/in-transit (TLS 1.2+), vulnerability scanning, patching, pen testing, Data Processing Agreement, DSR support (right to erasure/export).

Operational concerns & scalability:

• Provisioning automation: ARM/Bicep or Terraform to spin DB/schema/Key Vault keys + Azure AD group or app role assignment.
• Connection pooling & limits: use pooling, gateway, and Elastic Pools for many small DBs.
• Monitoring & alerts: per-tenant SLA metrics, abnormal activity detection (anomalous access to keys/data).
• Cost vs isolation trade-off: automate migration between isolation tiers and track cost impact.

Key trade-offs summary:

• Security vs cost/complexity: per-tenant DBs and keys maximize isolation but add cost/ops complexity. Hybrid model gives flexibility.
• Multi-tenant Azure AD app simplifies auth but reduce per-tenant administrative separation; per-tenant app grants stricter control at operational expense.
• Use envelope encryption + per-tenant CMKs stored in Key Vault/Managed HSM for tenants with regulatory needs; fall back to shared CMK for low-risk.

Example flows:

• Tenant onboarding: tenant sign-up → automated provisioning pipeline creates schema/DB (if chosen), creates/deploys tenant key (Key Vault), sets up Azure AD group or provisions via SCIM, seeds roles, sends admin consent link for multi-tenant app.
• Data access: client obtains token from tenant Azure AD → API validates token (tenant_id claim) → policy service enforces RBAC → data layer fetches DEK, decrypts with KEK in Key Vault (HSM) → returns data.

This design balances security, compliance, and operational scalability while allowing per-tenant customization for isolation and key management.

Approach: build a parametric model (Excel/Google Sheets or script) that computes monthly cost = sum(compute + storage + networking + monitoring + ops). Make inputs configurable so you can run sensitivity scenarios.

Key assumptions (example defaults you should validate):

• 50 AKS clusters: 30 dev, 10 staging, 10 prod.
• Node sizes: dev/staging use Standard_D4s_v3 (4 vCPU, 16GB); prod uses Standard_D8s_v3.
• Nodes per cluster (avg): dev 3, staging 5, prod 10.
• Average utilization: dev/staging 30%, prod 60% (affects right-sizing/reserved decision).
• Reserved Instances: 70% of prod compute covered by 1‑yr reserved or savings plan.
• PVs: 100GB per cluster dev/staging, 500GB prod; managed Premium SSD pricing.
• ACR: 500GB shared across clusters.
• Backups: daily snapshots, 30-day retention, incremental.
• Networking egress: 200GB/month per prod cluster, 20GB dev/staging.
• Monitoring: Log Analytics ingestion 1GB/day per prod cluster, 0.1GB/day dev; retention 90 days primary, archive policy thereafter.
• Operational overhead: 0.1 FTE per 10 clusters for routine ops + 1 on-call engineer; use loaded fully-burdened salary to hourly rate.

Model compute:

• Compute_cost = (#nodes * price_per_node_hour * 24 * 30) * (1 - reserved_discount*reserved_coverage)
• If utilization << 100%, consider autoscaling and smaller SKUs or burstable instances; include cost for system nodes (kube-proxy, system pods).

Model storage:

• PV_cost = total_GB * price_per_GB_month
• Snapshot_cost = per-snapshot storage * snapshot_count
• ACR_cost = storage + network egress for pulls

Model networking:

• Egress_cost = total_egress_GB * price_per_GB
• Include cross-region, load balancer and NAT gateway hourly costs.

Model monitoring:

• Log_ingest_cost = ingest_GB * price_per_GB
• Retention_cost = stored_GB * price_per_GB-month
• Consider using sampling, Azure Monitor Diagnostic settings, and tiered retention to reduce costs.

Operational overhead:

• Salaries, CI/CD runners, image build cost, incident SLA on-call uplift, license costs.

Sensitivity analysis (run scenarios):

• Reserved coverage: 0%, 50%, 100% — shows largest delta for compute.
• Utilization change ±20% — impacts whether to downsize nodes or buy fewer reserved instances.
• Log retention: 30 vs 90 vs 365 days — major impact on monitoring cost.
• Egress variance ±50% — test traffic spikes or cross-region transfers.
• Number of clusters scaled ±20% — linear scaling check.

Outputs to produce:

• Per-cluster monthly cost broken into categories and per-environment aggregates.
• Annualized TCO with and without reserved commitments.
• Recommendations: apply reserved/savings plans for stable prod baseline, aggressive retention policies/sampling for logs, centralize ACR and shared PV where possible, use autoscaler + right-sized SKUs for dev, and run regular cost reviews.

Validation steps:

• Compare model to actual billing for one pilot region for 1 month, refine assumptions (actual CPU/memory utilization, log volumes, egress).

Differences and typical use-cases

• Azure Load Balancer (L4)

  • What: Network-layer TCP/UDP load balancer (no HTTP awareness).
  • Use-cases: High-throughput, low-latency load distribution for VMs, internal service-to-service traffic, and scenarios needing direct IP/protocol passthrough (e.g., databases, game servers).
  • Not for: HTTP routing, WAF, caching, or global routing.

• Application Gateway (L7 with WAF)

  • What: Regional, HTTP(S)-aware reverse proxy with path-based routing, session affinity, and built-in WAF (OWASP rules).
  • Use-cases: Web app routing within a region, URL-based routing to microservices, end-to-end TLS options, and defending against OWASP-class attacks at regional layer.
  • Not for: Global multi-region routing or CDN-level caching at edge.

• Azure Front Door

  • What: Global, edge-based HTTP(S) load balancer and CDN-like service providing global routing, anycast, SSL termination at edge, edge caching, and an optional WAF.
  • Use-cases: Global traffic distribution with low latency, caching static content at POPs, global failover, and TLS termination close to users.

Recommendation for a global web app serving static content and APIs

• Primary edge: Azure Front Door

  • Reasons: global routing + anycast reduces latency worldwide, edge TLS termination simplifies cert management, built-in caching for static assets reduces origin load, and WAF at edge blocks malicious traffic before it reaches origin.
  • Configure: caching rules for static content (long TTLs), Front Door WAF ruleset enabled, and health probes + priority-based routing for failover.

• Regional backend routing: Application Gateway (optional, per-region)

  • Reasons: if you need complex path-based routing, cookie affinity, or stronger per-region WAF policies and end-to-end TLS between edge and backend, put an App Gateway in front of your app servers/APIs in each region.
  • Use-case: APIs that require sticky sessions or header-based routing, or compliance requiring regional inspection.

• Internal L4 balancing: Azure Load Balancer

  • Reasons: use for internal VM/VMSS traffic, or to distribute non-HTTP services; also for scaling backend pools behind an App Gateway if you host on VMs.

TLS termination, WAF, caching, global routing summary

• Termination: Front Door at edge for performance; optionally re-encrypt to App Gateway/backends for end-to-end TLS.
• WAF: Front Door WAF for global protection; App Gateway WAF for region-specific policies or deeper inspection.
• Caching: Front Door for static asset caching at edge (primary).
• Global routing: Front Door handles global traffic, failover, and geo routing.

This combination gives low-latency global delivery (static content cached), centralized edge security, and flexible regional routing and deep inspection where needed, while Azure Load Balancer handles high-performance internal L4 scenarios.

Recommendation: use Private Link (Private Endpoints) for both Azure SQL and Key Vault so the resources have no public endpoints and traffic flows over the VNet privately. Service Endpoints can simplify VNet-to-service trust but they still allow a public service IP and require service-side firewalling; Private Endpoints give a private IP per resource and stronger isolation.

Architecture (high level):

• App servers in one or more subnets inside a VNet (app-subnet, mgmt-subnet).
• Private Endpoints for Azure SQL and Key Vault placed in a dedicated pe-subnet (or per resource).
• Optional Azure Firewall/NVA in a hub VNet for centralized outbound control and logging (hub-spoke model).
• Private DNS zones:
  • For SQL: link the zone privatelink.database.windows.net (or for managed instance appropriate zone) to the VNet; Private Endpoint creates an A-record pointing to the private IP.
  • For Key Vault: use privatelink.vaultcore.azure.net (or vault.azure.net depending on region); link the private DNS zone to the VNet so vaultname.vault.azure.net resolves to the private IP.
• If hub-spoke, link DNS zones in hub and enable conditional forwarding to the hub resolver.

DNS details:

• Create Azure Private DNS zones that Azure suggests for Private Link (e.g., privatelink.database.windows.net and privatelink.vaultcore.azure.net).
• Link the zones to every VNet that needs resolution (app VNet, hub VNet if using firewall).
• Ensure VM/containers use Azure-provided DNS or a custom DNS forwarder that forwards to Azure DNS IPs so private zone records resolve.
• Verify with nslookup that the resource FQNs return private IPs.

NSG considerations:

• NSGs apply to subnets/VM NICs. Allow outbound to the private endpoint IPs (TCP 1433 for SQL, TCP 443 for Key Vault).
• Block outbound 1433/443 to internet IP ranges if you want to prevent bypass to public endpoints.
• Be careful: Private Endpoint is an ENI in your VNet; lock down access to that ENI by subnet NSGs so only app-subnet(s) can reach it.
• Use service tags sparingly—Private Link uses private IPs, not service tags.

Prevent data exfiltration:

• Disable public network access on the resource (Azure SQL: set PublicNetworkAccess = Disabled; Key Vault: firewall to deny public).
• Use Private Endpoint exclusively; reject or remove any public firewall rules that permit 0.0.0.0/0.
• Implement Azure SQL server-level firewall rules to allow only private endpoint IP ranges / VNet service endpoints off.
• Use Azure Firewall (or NVA) with forced-tunnel/NAT gateway to control and log outbound traffic—allow only necessary egress (e.g., to Microsoft Update, ACR).
• Enable Microsoft Defender for Cloud / Advanced Threat Protection for additional exfil detection.
• For extra protection, use Azure Policy to deny creation of public endpoints or to require Private Link.

Securing Key Vault access from the VNet:

• Create a Private Endpoint for Key Vault and ensure DNS resolves vault FQDN to the private IP.
• Disable "Allow trusted Microsoft services" if not needed, and set firewall to "Selected networks" with private endpoints only.
• Use Key Vault access controls (Azure RBAC + Key Vault access policies) and assign managed identities to app servers (system-assigned or user-assigned) to authenticate—avoid client secrets in code.
• Enable purge protection and soft-delete.
• Require TLS and set minimum TLS version.
• Monitor logs: enable Key Vault diagnostic logs to Log Analytics/Storage/Event Hub; alert on unusual access patterns.

Operational / Validation steps:

1. Create Private Endpoints, link Private DNS zones to VNets.
2. Verify FQDN resolves to private IPs from app servers.
3. Disable public access on SQL and Key Vault.
4. Lock down NSGs to only permit app-subnet → private endpoint ports.
5. Route outbound via Azure Firewall/NAT gateway; apply egress allow-lists.
6. Test connectivity, RBAC, and rotation flows (managed identity access to Key Vault).
7. Apply Azure Policy and automation to enforce design.

Trade-offs:

• Private Link has management overhead (DNS, per-resource PE) and costs for Private Endpoint and hub resources, but gives stronger isolation and simpler exfiltration prevention than service endpoints.
• Service Endpoints simpler and cheaper but leave a public service surface to manage with firewall rules.

This approach ensures app servers access SQL and Key Vault without public endpoints, DNS resolves privately, NSGs and Firewall control who can talk to the private endpoints, and resource-level controls plus managed identities prevent credential-exfiltration.