Microsoft Azure Services and Architecture Questions

Failure modes (AZ & Region-level)

1. Zone power/network failure

• Detection: VM/VMSS health probes fail, Azure Service Health alerts, increased app latency, metrics: hostHeartbeat, Disk IO errors.
• Immediate mitigation: Shift traffic to zone-redundant instances (cross-zone LB), scale instance counts in healthy zones, enable Autoscale.
• Recovery: Redeploy workloads to restored zone, verify data consistency, drain and reprovision nodes.
• Runbook validation: Simulate node/zone drain in staging, confirm LB failover & session continuity.

2. Zone control-plane/API degraded

• Detection: ARM API errors, Terraform/CI failures, Azure Portal errors, elevated 429/500.
• Immediate mitigation: Use alternate region management plane via Azure Lighthouse or runbook in paired region, avoid destructive ops.
• Recovery: Coordinate with MS incident, reapply pending infrastructure changes once stable.
• Validation: Inject API throttling in test environment; validate rollback safe-guards.

3. Region-wide outage (network, datacenter fire)

• Detection: Azure Health incident, cross-service outages, DNS resolution failures, global telemetry drop.
• Immediate mitigation: Failover to secondary region (pre-replicated data), swap traffic via Azure Front Door/Traffic Manager with low TTL, activate runbooks.
• Recovery: Reprotect data back to primary, DNS cutback, reconcile state (conflict resolution for writes).
• Validation: Periodic DR DR drills (see test plan).

4. Data corruption/replication lag

• Detection: Data integrity checks, application-level checksum mismatches, replication lag metrics.
• Immediate mitigation: Promote consistent replica, pause writes, mount read-only copies.
• Recovery: Point-in-time restore, replay logs, rehydrate caches.
• Validation: Runbackups/restore exercises and DB consistency checks.

Validating runbooks

• Peer-reviewed, version-controlled runbooks in repo; automated tests using Azure DevTest Labs.
• Tabletop walkthroughs quarterly, then automated dry-runs in sandbox: runbooks executed by CI (GitHub Actions) with simulated failures, assert expected alerts, metrics, DNS changes.

DR rehearsal test plan (non-impactful)

• Maintain pre-warmed secondary region with replicated infra (IaC).
• Monthly automated failover test: use Traffic Manager weighted shift to 0/100 for a canary app; validate connectivity, SLOs.
• Quarterly blue/green rehearsal: route subset of traffic (5-10%) to green region using Front Door, monitor for 24–48 hrs, rollback on anomalies.
• Annual full-scale failover: orchestrated cutover using runbooks during maintenance window with rollback plan.
• Automation: Use runbooks + ARM/Bicep + Azure Site Recovery + feature flags; use synthetic transactions and chaos testing (Chaos Studio) in pre-prod.
• Success criteria: RTO/RPO met, automated checks pass, no data loss, runbooks updated with lessons learned.

High-level comparison
Azure Load Balancer (L4) operates at transport layer—routes TCP/UDP based on 5‑tuple, low latency, high throughput, and is used for north‑south or east‑west traffic where packet-level distribution is sufficient. Application Gateway (L7) understands HTTP/S, URL paths, headers and can make routing decisions per request.

Typical scenarios

• Load Balancer (L4)
  • Scale stateful/stateless compute (VMSS), non-HTTP protocols (SQL, Redis), internal service mesh east‑west traffic
  • High performance, simple health probes, no HTTP inspection
• Application Gateway (L7)
  • Web applications requiring URL‑based routing, host-based routing, cookie-based affinity, SSL offload, WAF protection, or WebSocket support

Capabilities

• Session affinity: Load Balancer supports source IP affinity (clientIP) but not cookie affinity; App Gateway supports cookie‑based affinity for HTTP.
• SSL termination: Load Balancer does not terminate SSL (pass‑through); App Gateway can terminate/re‑encrypt SSL at edge.
• WAF: Load Balancer has no WAF; Application Gateway integrates OWASP‑based WAF (managed/custom rules).

Concrete example where App Gateway is required
A multi‑tenant web app needing host/path routing, SSL offload with certificate management, and protection from OWASP top‑10 attacks—e.g., route api.company.com → backend pool A, www.company.com/shop → pool B, terminate TLS and apply WAF rules. Load Balancer cannot perform path/host routing or WAF, so Application Gateway is required.

Overview
As a Cloud Architect I choose between Availability Sets and Availability Zones to meet required uptime, fault tolerance, and latency constraints. Both increase VM resiliency but operate at different scopes.

Availability Set

• Scope: Single Azure region, spread VMs across fault domains (FDs) and update domains (UDs) within one datacenter cluster.
• FDs isolate hardware/network failures; UDs stagger platform updates.
• SLA: With two or more instances in an Availability Set, platform SLA for VMs increases (per Azure SLA docs).
• Supported services: Classic for IaaS VMs (VMs, scale sets can use availability sets).
• Use when: App requires low-cost intra-region redundancy, shared low-latency storage (no cross-AZ dependencies), or region lacks zones.

Availability Zone

• Scope: Physically separate datacenters within a region (AZs are independent FDs).
• Provides isolation from datacenter-level failures and independent power/network.
• SLA: Higher SLA when VMs are spread across zones (Azure offers specific zonal/zone-redundant SLAs).
• Supported services: Many PaaS and IaaS (zonal VM placement, managed disks, load balancers, SQL Managed Instance, Storage Account zone-redundant options).
• Use when: Need stronger fault isolation, regional resiliency, or to survive datacenter outages; preferred for mission-critical, multi-tier apps.

Decision guidance

• Prefer Zones for maximum resilience and when app tiers can tolerate cross-AZ network latency and use zone-redundant storage/load balancing.
• Use Availability Sets when cost, ultra-low latency within a single datacenter, or region limitations make zones impractical.
• Combine with autoscaling, zone-redundant services, and global DR (paired regions) for full HA strategy.

Clarify requirements

• 100k events/sec sustained, ordering per partition, at-least-once delivery, multi-tenant producers, low latency (ms–100s ms), durable storage for replay.

Service choice

• Primary: Azure Event Hubs — built for high-throughput ingress, partitioned ordering semantics, throughput units / Dedicated namespaces for 100k+/s.
• Not Service Bus: good for FIFO and transactions but not cost-effective at this scale.
• Event Grid: good for notifications, not for high-throughput bulk ingestion or strict ordering.

Partitioning strategy

• Choose partition key that preserves required ordering domain (e.g., customerId, sessionId).
• Estimate partitions = ceil(peak throughput / per-partition target). Target ~5–10k events/s per partition → ~10–20 partitions; add safety buffer and scale with throughput units or Dedicated capacity. Balance hot keys; use hashing or sharding layer to avoid hot partitions.

Downstream processing

• Use consumer groups with scalable consumers:
  • Stream Analytics for simple SQL-windowed aggregations (managed, limited throughput).
  • Azure Functions (Event Hub trigger) for light-weight processing; scale out with partition ownership but monitor cold-starts.
  • Custom Kafka-style consumers (Kafkify Event Hubs or Spark Structured Streaming / Flink on Databricks) for high-throughput, exactly-once semantics patterns.
• Prefer Stateful streaming (Spark/Flink) if complex joins, windowing, or heavy compute.

Checkpointing & delivery

• Use Event Hubs offsets with durable checkpointing to Blob Storage (or managed state in Stream Analytics / Spark). Checkpoint after successful processing to ensure at-least-once. Design idempotent downstream operations (idempotency keys, dedup store) to tolerate duplicates.

Backpressure & throttling

• Apply producer-side throttling and retry with exponential backoff and jitter. Use ingress rate-limiting gateway (API Management or custom ingress) and token bucket per-tenant to avoid hot keys. On consumer side, autoscale consumers based on partition load and use batching to increase throughput. Configure Event Hubs quotas and monitor ThrottledRequests metric; implement circuit-breakers and DLQ pattern (store to durable queue/blob for later replay) for sustained overload.

Observability & operational

• Monitor throughput, partition skew, consumer lag, throttling, errors. Use autoscale rules, capacity planning, and chaos tests. Document recovery playbooks (resend from offset, replay).

Trade-offs

• Event Hubs + Spark/Flink gives throughput and flexible semantics; Stream Analytics simplifies ops but may not scale for complex heavy workloads. Service Bus chosen only if strict single-entity FIFO + transactions trump scale.

Overview (role perspective)
As a Cloud Architect I choose between Kubenet and Azure CNI based on IP needs, scale, and operational constraints. Below are concise, decision-oriented differences and trade-offs.

IP management

• Kubenet: Nodes get Azure VNet IPs; pods receive RFC1918 IPs from node-local bridge (no Azure allocation). Fewer VNet IPs consumed — good when VNet CIDR is tight.
• Azure CNI: Each pod receives an IP from VNet/subnet — direct Azure-IPAM. Predictable addressing, simpler security (NSGs) but consumes many VNet addresses.

Pod-to-pod cross-node communication

• Kubenet: Uses Kubernetes routes; node-to-node encapsulation and Azure routing required. Cross-node pod traffic is routed via node IPs; you may need additional routing or Azure routes for on-prem connectivity.
• Azure CNI: Pod IPs are routable across VNet/subnet natively; simpler east-west connectivity and integration with other Azure resources.

NAT/outbound behavior

• Kubenet: Pod outbound uses node NAT (masquerade) — predictable source IP (node). Easier external firewall rules.
• Azure CNI: Pods have their own VNet IPs; outbound may require Azure NAT Gateway or Load Balancer SNAT to avoid SNAT exhaustion. Without NAT Gateway, SNAT limits can cause failures.

Scaling limitations

• Kubenet: Scales well re: VNet IP usage but has complexity for large cross-subnet routing and CNI plugins. Less per-pod IP pressure.
• Azure CNI: Limited by subnet IP capacity and Azure provisioning limits (IPs-per-subnet, route table sizes). For very large clusters use larger subnets, IP prefix planning, or Azure CNI IP-masq-agent alternatives (or use Azure CNI with IP prefixes/Azure Kubernetes Service cluster-autoscaler planning).

Recommendation

• Choose Kubenet when conserving VNet IPs and simple outbound NAT via nodes is acceptable.
• Choose Azure CNI when you need native pod routability, Azure-level security controls, and simpler integration with VNet services — plan subnet IP capacity and NAT Gateway for large clusters.