Microsoft Azure Services and Architecture Questions

Situation: You need low-latency, high-IOPS storage for a transactional DB. Choose disks and design for required IOPS/throughput.

Disk comparison (short):

• Standard HDD: low cost, high latency, low IOPS — only for archival/logging, not for transactional DBs.
• Standard SSD: better latency and IOPS than HDD, bursty workloads OK, but limited baseline performance — OK for dev/test or small DBs.
• Premium SSD: predictable IO, low latency, sizes with guaranteed IOPS and throughput (P10–P80+). Good for most OLTP workloads.
• Ultra SSD: highest IOPS/throughput, sub-ms latency, dynamically configurable IOPS/throughput/size per disk — suited for very high-performance OLTP.

Designing for required IOPS/throughput:

• Calculate baseline: transactions/sec × average IOs per txn × IO size. Add headroom (1.5–2×) for peaks.
• Choose disk SKU(s) whose provisioned IOPS/MBps match needs. If a single disk limit is insufficient, stripe multiple disks (RAID0) to aggregate IOPS and throughput.
• Example: required 200k IOPS and 4 GiB/s throughput → use multiple Ultra disks or many Premium P80s, or prefer Ultra for simpler design.

Striping disks:

• Use OS-level software RAID0 or Storage Spaces to stripe logical volumes across multiple disks to sum IOPS/throughput.
• Ensure alignment and similar-sized disks to avoid hotspots.
• Monitor per-disk latency; a slow disk degrades overall performance.
• Consider file placement (separate data, log, tempdb on different stripes) to avoid contention.

Caching settings:

• ReadOnly or ReadWrite caching on VM OS disk and data disks affects latency and durability.
• For write-heavy transactional DBs: disable caching for write-intensive log disks (or use none) to guarantee durability; use ReadOnly or None for data depending on DB caching behavior. Some DB engines perform their own caching (e.g., SQL Server buffer pool) so OS-level cache may be redundant.
• Premium and Ultra rely less on host caching; Ultra disks do not support host caching — you control throughput at the disk level.

Operational implications of Ultra Disks:

• Requires Ultra-capable VM sizes and region availability; attach limits per VM apply.
• Billing is based on provisioned IOPS/throughput and capacity — costs can be significant; right-size and autoscale strategy recommended.
• Dynamic reconfiguration is possible without recreation, but some changes may require disk detach/attach or VM restart depending on VM family.
• Snapshot support and backup constraints differ; ensure backup solution supports Ultra (or move snapshots to standard tiers).
• Monitoring: track provisioned vs. consumed IOPS/latency; alerts for throttling.
• Recovery/DR: ensure target DR environment supports same performance or accept reduced performance.

Recommendation: For typical high-throughput OLTP use Premium SSDs (sized to need) and use striping for scale; choose Ultra SSD when single-digit-ms or extreme IOPS/throughput and predictable billing are required. Always model workload, provision headroom, test with realistic load, and define monitoring/alerting and cost controls.

StorageV2 (general-purpose v2), Blob Storage, and ADLS Gen2 are related but have different feature sets and target use-cases.

• StorageV2 (GPv2)
  • General-purpose account supporting blobs, files, queues, tables.
  • Supports all modern features (tiering hot/cool/archive, lifecycle, encryption).
  • Flexible for mixed workloads (app data, backups, etc.).
• Blob Storage account
  • Blob-only account optimized for block blobs.
  • Slightly simpler surface and historically used for large object storage with tiering.
  • Lacks some GPv2 extras (but functionality often overlaps now).
• ADLS Gen2 (Azure Data Lake Storage Gen2)
  • Built on StorageV2 but with Hierarchical Namespace (HNS) enabled and filesystem semantics.
  • Adds directory-level operations, POSIX-like ACLs, and improvements for analytic workloads (rename/atomic directory operations).
  • Designed for big-data: Hadoop/Spark/Databricks/Synapse integration, efficient directory scans, and parallel access.

Key differences & considerations

• Hierarchical namespace: only ADLS Gen2 (HNS) provides true directories and efficient directory-level operations. Without HNS you have a flat blob namespace and directory semantics are simulated (prefix scans).
• Performance: For analytics (large files, many append/rename operations, directory listings), ADLS Gen2 with HNS is faster and more efficient — avoids costly list/rename patterns and reduces metadata overhead. Raw throughput scale is similar (depends on account limits), but HNS improves metadata-heavy workloads.
• Costs: Storage capacity and egress pricing are similar (since ADLS Gen2 sits on GPv2), but transaction costs differ. ADLS Gen2 can reduce transaction costs for workloads that would perform many list/rename operations in a flat namespace. Small-file workloads may still be expensive due to per-operation charges. Consider access tiering, lifecycle policies, and transactions when estimating cost.

Recommendation for a data lake (analytics)

• Choose StorageV2 with ADLS Gen2 (HNS) enabled. Reason: it provides filesystem semantics, POSIX-like ACLs for fine-grained security, and optimized metadata operations that big-data engines expect. It integrates natively with Spark/Databricks/Synapse and simplifies ETL (atomic renames, efficient directory listings). Also retains GPv2 features like tiering and lifecycle so you can optimize costs.
• Additional considerations: design for large files to reduce transaction overhead, apply lifecycle rules (move cold data to cool/archive), use RBAC + ACLs for security, monitor transaction/egress to manage costs, and consider storage account throughput limits—partition data across containers/accounts if you need extreme parallelism.

Requirements & constraints (clarifying assumptions):

• Global SaaS serving NA, EU, APAC with target <100ms read latency for user-facing reads and 99.99% availability.
• Read-heavy user profile access, occasional profile writes, per-request session/state needed.
• Prefer managed Azure PaaS, want automatic failover, low operational overhead.

High-level architecture:

• Global ingress: Azure Front Door (Premium) for global anycast, TLS termination, WAF, and FastPath routing to nearest region. Front Door health probes + priority/latency routing.
• Regions: Primary deployment in 3 Azure regions (e.g., West US, West Europe, Singapore) with identical app pools (AKS or App Service) behind regional Application Gateway + regional subnets.
• Global DB for user profiles: Azure Cosmos DB (Core SQL API) configured multi-region, multi-master, with regions matched to app regions. Use session or bounded staleness consistency for fast reads while keeping read-your-writes semantics as needed.
• Session-state: Azure Cache for Redis (Enterprise or Premium) deployed regionally with active geo-replication / Redis Enterprise active-active to keep session data local and replicated for fast reads and failover. Keep app servers stateless.
• Caching layer: Regional Azure Cache for Redis as L1 read cache for profiles and hot data; optionally Azure CDN for static assets.
• Secrets/config: Azure Key Vault globally accessible with regional redundancy.
• Observability: Azure Monitor, Application Insights, distributed tracing.

Data flow:

1. Client → Front Door routes to nearest healthy region.
2. App reads profile: first check regional Redis cache; if miss, read Cosmos DB (multi-region reads served from local replica).
3. Writes: write to Cosmos DB (multi-master handles concurrent writes; conflict resolution policy set per entity).
4. Session read/writes: Redis local cluster; asynchronous geo-replication to other regions.

Why Cosmos DB vs Azure SQL:

• Cosmos DB pros: turnkey multi-region reads with local replicas for <100ms global reads, multi-master for low-latency writes in any region, tunable consistency (session/bounded staleness), SLA-backed single-digit-ms reads, elastic RU/s scaling.
• Azure SQL pros: strong relational model and ACID; but multi-region read replicas (Geo-Replication) give read locality but cross-region write latency remains to primary or requires complex leader-follower and failover orchestration. Harder to achieve global write locality and sub-100ms reads in all regions.
• Recommendation: Cosmos DB for user profiles when global low-latency reads + occasional writes needed. If complex relational transactions required, consider hybrid: Cosmos for user-facing profile, Azure SQL for transactional back-office data.

Session-state strategy:

• Make apps stateless. Use regional Azure Cache for Redis as canonical session store for low-latency reads/writes.
• For HA: use Redis Enterprise active-active (CRDT) or Redis replication with Sentinel-like failover; configure TTLs and write-behind patterns to persist critical session snapshots to Cosmos DB if needed.

Failover approach:

• Front Door: health probes + automatic failover across regions; route traffic to secondary regions on outage.
• Cosmos DB: automatic regional failover configured; multi-master significantly reduces failover impact. Configure custom failover priority for planned maintenance.
• Redis: geo-replicated clusters; if entire region fails, app in target region will use its local Redis; for sticky session requirements, rely on Front Door and app to route users to other region and rehydrate session from replicated Redis or persisted snapshot.

Scalability & performance considerations:

• Read latency: local Redis hit ~<5ms; local Cosmos read ~<10–20ms depending on RU and item size; combined with application + network, 100ms realistic if app and TLS optimized and CDNs used for static assets.
• Consistency: choose session consistency for read-your-writes; bounded staleness acceptable where slight lag tolerable.

Trade-offs:

• Cost vs latency: multi-region Cosmos + Redis per-region + Front Door increases cost (see cost drivers) but required for latency/SLA.
• Consistency vs latency: stronger consistency increases cross-region coordination cost; session or bounded staleness offers best balance.
• Operational complexity: multi-master simplifies writes but requires conflict resolution logic and testing.
• SQL alternative reduces RU/e/cost for relational workloads but complicates global write locality and failover.

Estimated cost drivers (high-level):

• Cosmos DB RU/s provisioning or autoscale across regions (largest driver): per-region RUs + egress costs for multi-region writes/replication.
• Azure Cache for Redis Enterprise clusters in each region (size & active-active increases cost).
• Azure Front Door Premium (per routing + WAF + rules).
• Compute (AKS/App Service) across 3 regions scaled for peak load.
• Network egress (cross-region replication and client traffic) and data transfer charges.
• Monitoring/Logging retention (Application Insights, Log Analytics).

Availability & SLA alignment:

• Front Door + Cosmos multi-region + Redis geo-replication + multi-region app deploy can meet 99.99% if each component configured per SLA, distributed across fault domains, and automated runbooks for failover tested.

Operational recommendations:

• Run chaos tests and DR drills to validate failover paths.
• Use RU autoscale and Redis scaling policies; instrument read/write latencies.
• Implement conflict resolution for multi-master writes (last-writer-wins only for non-critical fields; use vector clocks or application merge for important fields).
• Use CDN for static content and compress responses to minimize latency.

Summary:

• Use Azure Front Door, regional AKS/AppService, Cosmos DB multi-region multi-master for profiles, regional Azure Cache for Redis (Enterprise active-active) for session + hot cache, and automated Front Door + Cosmos failover. This yields sub-100ms reads globally and >99.99% availability with higher cost driven by multi-region Cosmos RU provisioning, Redis clusters, Front Door, and multi-region compute.

Short answer: All three are viable for Azure IaC. Choice depends on team skills, multi-cloud needs, and governance. Below is a focused comparison and recommended patterns.

Comparison

• Authoring ergonomics
  • ARM templates: JSON -> verbose, repetitive, poor readability.
  • Bicep: Declarative DSL built for Azure -> concise, readable, native transpiles to ARM.
  • Terraform: HCL -> readable, rich provider ecosystem; slightly different model than ARM/Bicep.
• Modularity
  • ARM: supports nested/linked templates but awkward.
  • Bicep: modules first-class, easy reuse and parameterization.
  • Terraform: modules are mature and promote registry-driven reuse across clouds.
• Repeatability
  • All support idempotent deployments; Terraform and Bicep give clearer workflows for repeatable pipelines.
• State management
  • ARM/Bicep: no external state file; Azure Resource Manager is the source of truth.
  • Terraform: maintains state (remote backends like Azure Storage/Blob, Terraform Cloud/Enterprise) — powerful but must be managed & secured.
• Drift detection
  • ARM/Bicep: use Azure resource graph/ARM What-If and deployment validations; Azure Policy for continuous drift prevention.
  • Terraform: plan detects drift relative to state; require periodic plan/apply and integrations with cloud-native checks.
• Large-enterprise multi-subscription deployments
  • ARM/Bicep: map naturally to management groups/subscriptions; good for Azure-native governance at scale.
  • Terraform: excellent for multi-cloud / multi-subscription via providers; needs careful state partitioning (one state per scope or workspace) and collaboration controls.

Recommended patterns (Solutions Architect)

• Collaboration
  • Use Git-based workflows (feature branches, PRs, protected branches).
  • CI/CD pipelines to validate (lint, unit tests), run plan/what-if, and require approval for apply to production.
  • Enforce module registries (private Terraform module registry or Bicep modules in shared repo) for reuse.
• Governance
  • Use Azure Policy + Initiative to enforce guardrails; integrate policy checks in PR pipelines.
  • For Terraform, store state in secure remote backends with locking (Azure Storage + blob lease or Terraform Cloud) and enable encryption and RBAC.
  • Separate environments and subscriptions; adopt IaC scoping: infra modules for platform, environment overlays for config.
  • Centralize landing zones (CAF/Enterprise-Scale) implemented as modules/templates; automate subscription provisioning.
• Practical recommendation
  • If purely Azure and you want native support and simpler ops -> Bicep + Azure DevOps/GitHub Actions + Azure Policy.
  • If multi-cloud or need mature module ecosystem -> Terraform with remote state, locking, and strong CI controls.
  • Keep ARM only when you need lowest-level support or must interop with existing JSON assets.

This approach balances developer ergonomics, secure state management, drift detection, and enterprise governance.

Requirements & constraints

• Ingest 500k events/sec reliably, sub-second analytics (<=1s), durable downstream persistence, multi-tenant, bursty traffic, cost-sensitive, ops/monitoring.

High-level architecture

• Ingress: Azure Event Hubs (primary) → Stream processing tier (low-latency) → Serving & long-term storage.
• Optional: IoT Hub if you need device identity, provisioning, direct method/desired properties or per-device throttling. Use IoT Hub only for device management features (adds cost); IoT Hub routes into Event Hubs for analytics.
• Kafka on AKS: viable if you need Kafka API compatibility or advanced custom tooling, but operational burden and higher OPEX; Event Hubs for Kafka (managed) is a good compromise.

Ingress detail & sizing

• Choose Event Hubs Standard/Provisioned Throughput with dedicated Event Hubs Dedicated or Event Hubs Premium for guaranteed high throughput. 500k ev/sec => estimate message size (e.g., 1KB) => ~500 MB/s ~ ~1.8 TB/hr. Use Dedicated namespaces or throughput units (TUs) aggregated to match ingress — each TU = ~1 MB/s ingress for Basic? (note: verify SKU specifics in procurement). For large scale, use Event Hubs Dedicated (capacity units) or Azure Event Hubs clusters to avoid TU limits.
• Partitioning: create N partitions to enable parallel consumers. Aim for partitions >= consumers * concurrency and to satisfy partition-level throughput (~1MB/s or SKU limit). Example: for 500 MB/s, with per-partition sustainable throughput ~50 MB/s on premium/dedicated, create ~10–20 partitions; validate against SKU docs.

Stream processing options

• Azure Stream Analytics: simplest, low operational overhead, can achieve sub-second with low-latency queries for aggregations. Good for simple sliding-window real-time metrics.
• Apache Flink (Azure Stream Analytics Gen2 or Flink on Azure Kubernetes/Databricks Structured Streaming): best for complex event processing, exactly-once semantics, stateful joins, low-latency with tuned parallelism.
• Databricks Structured Streaming: excellent for micro-batch near-real-time (latency ~hundreds ms to seconds), good for ML enrichment and complex pipelines. Use Photon/auto-scaling clusters to meet latency.
  Recommendation: use Flink (managed via Azure Stream Analytics Gen2 or Flink on HDInsight/AKS) for strict sub-second SLAs and stateful processing; use Databricks for advanced analytics/ML in parallel.

Downstream storage

• Hot/serving: Azure Cosmos DB (low-latency single-digit ms reads), or Redis Cache for aggressive read throughput.
• Analytical/long-term: Azure Data Lake Gen2 (parquet) on Blob Storage for cost-effective analytics and archival.
• OLAP: Synapse Analytics or Databricks Delta Lake for batch/interactive analytics.

Scaling & reliability

• Autoscale Event Hubs (use dedicated clusters), scale partitions only by redeploy (plan for headroom). Scale processing with parallelism keyed to partitions (1:1 or >1 consumers per partition if supported). Use state backend with durable checkpointing (Azure Blob) for Flink/Databricks.
• Cross-region replication for DR: Geo-DR for Event Hubs + active-passive processing in secondary region; use Event Hubs Capture to archive raw events in ADLS for reprocessing.

Monitoring & observability

• Azure Monitor + Event Hubs metrics (incoming/outgoing bytes, throttled requests, consumer lag). Application Insights for processing latency and error rates. Prometheus/Grafana for Flink/Databricks metrics. Alerts on throughput, e2e latency, CPU/memory, throttling.

Cost implications & trade-offs

• Event Hubs managed reduces ops cost vs Kafka on AKS. Dedicated clusters increase fixed cost but needed at this scale. IoT Hub adds device mgmt value; otherwise route devices to Event Hubs.
• Flink has higher engineering/ops cost but best latency/semantics. Databricks higher compute cost but accelerates analytics and ML.
• Optimize by batching small messages at producers, compress payloads, use appropriate retention (Capture to ADLS to avoid long retention in Event Hubs).

Example flow
Producers -> Event Hubs (Dedicated, 20 partitions) -> Flink job (parallelism 20, checkpointing to Blob) -> outputs: Cosmos DB (serving), ADLS Gen2 parquet (archive) -> downstream Databricks for ad-hoc analytics.

Final note
Validate assumptions: average message size, acceptable latency percentiles, retention needs, and budget. Prototype with load tests (Azure Event Hubs perf tools) and tune partition counts, processing parallelism, and SKU selection before production.