Situation: Managing quotas across 100+ AWS accounts requires a combination of inventory, proactive automation, monitoring, governance and resilient architecture so production never fails silently when a limit is hit.Approach (high level):1. Inventory & baseline- Run an account-wide inventory (AWS Config + Service Quotas ListServiceQuotas) to record current quotas, usage, and owner tags into a centralized datastore (DynamoDB/Config aggregator).- Maintain a desired-quota catalog per service (per environment: prod/non-prod) in SSM Parameter Store or a Git-backed config repo.2. Automated quota-increase workflow- When usage trends/thresholds breach a policy (e.g., 70% of quota), trigger a Step Functions workflow: - Step 1: Check current quota via Service Quotas API (GetServiceQuota). - Step 2: Attempt programmatic increase using Service Quotas RequestServiceQuotaIncrease. - Step 3: If API returns manual-required, create a support case via AWS Support API and attach context (account, load history, expected growth). - Step 4: Post status to ticketing (Jira/ServiceNow) and notify owners (SNS/Slack).- Use cross-account IAM roles for the centralized automation account, least privilege for Service Quotas and Support APIs.Example automation snippet (Lambda pseudo):python
from boto3 import client
sq = client('service-quotas')
resp = sq.request_service_quota_increase(ServiceCode='ec2', QuotaCode='L-1216C47A', DesiredValue=200)
3. Monitoring & alerting- Emit quota usage metrics into CloudWatch (custom metrics) with tags for account, region, env. Use short-term and long-term trend detection (CloudWatch Anomaly Detection or Lookout).- Create alarms at multiple thresholds (70% warning, 85% action, 95% urgent) that trigger the Step Functions workflow and paging via PagerDuty.- Daily/weekly reports for capacity planning pushed to Slack/ops dashboard.4. Governance & preventative controls- Enforce tagging and quota request guardrails via Service Control Policies and IAM permissions.- Require pre-approved quota increases for prod through an approval pipeline (IAM + Step Functions manual-approval step).5. Architectural fallbacks when a quota is exhausted- Graceful degradation: feature flags to disable non-critical functionality.- Circuit breakers and throttling at the application layer (API Gateway usage plans, token bucket rate limiters) to protect downstream services.- Backpressure & retries: exponential backoff + jitter, idempotency.- Partitioning: distribute load across accounts/regions or use multi-tenant resource pools (e.g., multiple NAT Gateways, multiple S3 prefixes).- Capacity reservations and pre-warming: e.g., warm Lambda provisioned concurrency, pre-create ENIs/Elastic IP pools, reserved EC2 capacity if quota-bound.- Fallback service: degrade to cached data (DynamoDB DAX, CloudFront) or read-only mode.- Automated failover: if region quota exhausted, route traffic to another region/account with synced state.6. Continuous validation & runbooks- Scheduled chaos tests: simulate quota exhaustion in non-prod to validate fallbacks and runbooks.- Runbooks with playbooks for manual escalation and rollbacks.- Quarterly reviews with stakeholders, capacity forecasting with growth models.Trade-offs & reasoning:- Automated increases reduce manual toil but require governance to avoid runaway costs.- Centralized control simplifies visibility but must use cross-account roles to avoid privilege sprawl.- Architectural fallbacks add complexity but are critical for availability when AWS limits are reached unexpectedly.Result:This approach provides proactive detection, automated remediation where possible, clear human workflows when needed, and resilient architecture so production remains available or degrades safely if a quota is exhausted.