Google Cloud Platform Services and Architecture Questions

Requirements:

• Serve private video to authenticated users with low latency, global scale, and strong auditability.
• Controls: only authorized users can access content; minimize origin load; retain ability to revoke access and audit reads.

High-level architecture:

• Store videos in Cloud Storage (private buckets, uniform bucket-level access).
• Front with Cloud CDN in front of an external HTTPS Load Balancer (Cloud Load Balancing) -> backend bucket (Cloud Storage) or backend service.
• Enforce access via short-lived signed URLs generated by an authentication backend (e.g., Identity-Aware Proxy or custom auth service using service account key or Cloud KMS-signed tokens).
• Use Signed URLs on Cloud CDN so cached content is delivered without revalidating origin while preserving per-request auth.

Flow:

1. User authenticates (OAuth2 / OIDC). Client receives ID token.
2. Client requests access to a video from application backend.
3. Backend verifies identity/entitlement, creates a signed URL for the Cloud Storage object (or creates a signed CDN URL) with tight expiration (e.g., 5–15 minutes) and attaches response-cache-control headers.
4. Client uses signed URL to fetch video via CDN/Load Balancer. Cloud CDN caches content keyed by signed URL (if URL contains signature portion in path/query) or by removing signature from cache key and using signed cookies for caching (see below).

Cache-control and caching strategies:

• Prefer signed cookies for long-lived CDN caching where signature is separate from cache key: set Cloud CDN to cache based on path only and use signed cookies to authorize requests. That allows long TTLs (hours) for cached video while keeping authorization separate.
• If using signed URLs, include cache-control headers (Cache-Control: public, max-age=86400) when object is safe to cache and signatures are short; realize signed URL query params often create unique cache entries—short signed URLs reduce cache reuse.
• Use Cache-Mode: Use Origin Cache Control and set appropriate max-age depending on content volatility.

Token expiration & refresh:

• Use short-lived signed URLs (5–15 min) or signed cookies with TTL matching session risk.
• For long streams, implement URL refresh: client detects expiry (401/403 or progress near expiry) and requests a refreshed signed URL from backend via an authenticated channel. For players, support token-refresh callbacks (HLS token rotation by re-requesting playlist segments).
• To support offline or shareable links, issue revocable tokens via an authorization database mapping token IDs to status.

Revocation & compromise handling:

• Use short TTLs to limit exposure.
• Maintain a token blacklist in a fast store (Cloud Memorystore/Redis or Firestore) and check at backend issuance time; for signed cookies, rotate signing keys frequently (Cloud KMS) and perform key roll-over; to force immediate revoke, rotate keys and invalidate cache entries (purge CDN).

Auditing & compliance:

• Enable Cloud Storage access logs and Cloud CDN edge logs; stream logs to Cloud Logging and export to BigQuery for analysis/retention.
• Log: user identity (from backend), token ID, object path, source IP, timestamp, and HTTP response codes.
• Correlate CDN logs with application issuance events using token IDs or custom headers inserted by the backend when generating signed cookies/URLs.
• Retain logs per compliance retention policies, use IAM conditions and Cloud Audit Logs to record admin actions (key rotations, permission changes).
• Regularly run anomaly detection queries in BigQuery (e.g., unusual download volume, geographic anomalies).

Security best practices & trade-offs:

• Keep buckets private and limit service account scopes. Use Cloud KMS for signing keys; prefer IAM-based signing (signBlob) rather than embedding static keys.
• Signed URLs are simpler but reduce cache hit ratio (if signature in cache key). Signed cookies improve cache efficiency at cost of more complex client logic.
• CDN purge is expensive—prefer short TTLs and key rotation over frequent purges.
• Monitor costs: balance origin egress vs CDN cache TTLs.

This pattern delivers secure, auditable private content with low latency; choose signed cookies for heavy-read, cacheable content and short-lived signed URLs where per-request simplicity and limited sharing are priorities.

Cloud SQL (managed MySQL/Postgres) supports cross-region high-availability via regional primary + zonal failover (read replicas can be in other regions). Cross-region replicas are asynchronous (logical or physical, depending on engine) so they can lag; failover to a cross-region replica is manual/promoted and may result in some data loss. Automated HA (regional) protects against zone failures but not region failure. Read replicas enable offloading reads but consistency is eventual.

Cloud Spanner is a globally-distributed, strongly consistent relational DB with synchronous (or configurable) Paxos/RAFT replication across zones and regions. Multi-region instances provide transparent, strongly consistent reads/writes with automated leader placement and fast failover; you can configure read-only follower replicas in other regions for low-latency reads with no leader change and minimal staleness if you use read-only staleness options or use convergent reads (exactly-once reads are strongly consistent if served from leader).

Recommendation: choose Cloud Spanner for a transactional application needing low-latency reads across multiple regions and strong consistency at scale.

Suggested topology:

• Multi-region Spanner instance with a main compute region group covering your primary markets (e.g., europe-west1 + us-central1) configured as a multi-region instance (e.g., nam3 or eur3) to span those regions.
• Deploy compute frontends (App servers) in each region; route user traffic via global load balancer with proximity-based routing.
• Configure Spanner with read-write replicas distributed across regions; add read-only follower replicas in secondary regions to serve low-latency local reads without promoting leaders.
• Enable backup and automated failover (Spanner handles failover automatically); set SLOs and monitor replication lag and CPU/latency metrics.

Trade-offs:

• Spanner higher cost but gives global strong consistency and automated failover.
• Cloud SQL cheaper for single-region or read-mostly workloads; cross-region replicas risk data loss and manual promotion—acceptable if eventual consistency is fine.

This design meets low-latency, transactional multi-region reads with high availability and automated failover.

Requirements & SLOs:

• 99.99% global availability (<=52.6 min downtime/year), sub-second checkout latency for most geos.
• PCI-compliant payments, global catalog with correct inventory levels at checkout, quick reads, high throughput during peaks.

High-level architecture (GCP):

• Global External HTTP(S) Load Balancer + Cloud CDN for TLS termination, edge caching, and latency-based routing to nearest region.
• Regional compute in multiple GCP regions: Cloud Run / GKE for frontend + checkout microservice; Autoscale per region.
• Global data layer: Cloud Spanner configured as a multi-region instance (choose 3+ read replicas across major geos) for strong consistency for critical write paths (orders, inventory reservations).
• Regional caches: Memorystore (Redis) for session state & hot product caches; Cloud CDN + edge caching for catalog pages and static assets.
• Asynchronous backbone: Pub/Sub + Dataflow for events (analytics, email, eventual inventory reconciliation).
• Payments: Use PCI-ready external gateways (Stripe/Adyen) with tokenization; secrets in Secret Manager, HSM via Cloud KMS; minimal PCI touch in our infra.
• Observability & security: Cloud Monitoring, Cloud Trace, Cloud Armor, VPC Service Controls, IAM.

Key design choices

1. User sessions:

• Stateless JWT for auth + short-lived session tokens. For server-side session needs (cart-in-progress), keep session in regional Redis with periodic write-through to Spanner for persistence. This allows sub-second reads locally and global durability for cart checkout.

2. Inventory consistency:

• Strong consistency for inventory decrement at checkout — implement inventory reservation pattern in Spanner (single transaction: check available, reserve, create order). Use Spanner’s global ACID transactions to guarantee no double-sell.
• To reduce Spanner write contention, use per-SKU sharding keys (range/hashed keys) and optimistic concurrency / retry for hot SKUs.
• Eventual reconciliation: publish inventory events (Pub/Sub) to update regional caches and analytics asynchronously.

3. Payments:

• Checkout flow: reserve inventory (Spanner), tokenized payment auth with external gateway, then commit order transaction. If payment fails, rollback reservation.
• All payment requests go to gateway; only tokens stored. Use Cloud KMS for encryption keys and ensure PCI compliance by minimizing card data flow.

4. Caching:

• Cloud CDN for static and cacheable catalog responses with short TTLs on dynamic segments.
• Memorystore per-region for hot product details and session/cart caches; cache invalidation via Pub/Sub messages from Spanner change streams (use change data capture patterns).
• Application-level read-through cache patterns and circuit breakers.

5. Database patterns:

• Spanner for global canonical data (orders, inventory, user accounts).
• Bigtable / Firestore (regional) for high-volume analytics or user activity streams where eventual consistency is acceptable.
• Cloud Storage for assets.

6. Traffic routing & multi-region replication:

• Global LB routes traffic to nearest healthy region using latency and proximity; health checks trigger automatic backend failover.
• Spanner multi-region config (e.g., multi-region with leader located in primary geo or use Paxos/TrueTime config) provides synchronous replication; choose commit-witness/replica topology matching latency/availability trade-offs.
• Read-only replicas in other regions reduce read latency.

7. Failover processes:

• Automated: LB health checks + instance groups with regional autoscaling and automated instance replacement. Spanner automatic failover for regional failure with configured leader priorities.
• Playbooks: automated DNS TTLs, runbooks to shift traffic, automated replay of pending events via Pub/Sub dead-letter queues.
• DR drills: regular failover testing, chaos engineering, and runbook rehearsals.

Scalability & performance:

• Place checkout microservice in regions within 50–100ms RTT to the Spanner leader write region when possible; optimize to keep critical write path minimal.
• Use batching for asynchronous tasks; keep synchronous path minimal to meet sub-second latency.

Trade-offs: cost vs complexity

• Spanner: strong consistency, simple global transactional model, predictable SLAs — high cost. Reduces complexity of correctness across geos.
• Alternative lower-cost: multi-master, application-level sharding (e.g., regional databases + async reconciler) reduces Spanner spend but increases application complexity, risk of oversell, and longer reconciliation windows.
• Cache-heavy approach reduces Spanner read load and cost but increases invalidation complexity.
• Self-managed distributed DB (CockroachDB on GKE) may cut licensing costs but increases operational burden and risk.

Recommendation:

• For a client requiring 99.99% availability and global strong consistency with minimal business risk on checkout, use Cloud Spanner multi-region + global LB + regional Redis caches + external PCI payment gateway. Accept higher cost for lower operational complexity and stronger guarantees; mitigate cost by tuning Spanner instance topology (right-sized nodes, per-region replicas) and push reads to replicas and caches. Implement rigorous testing, monitoring, and runbooks for failover.

Pub/Sub, Cloud Tasks, and Cloud Scheduler solve different orchestration/messaging needs. Below is a concise comparison with guarantees, primary use cases, and examples.

Pub/Sub

• Purpose: Asynchronous, scalable messaging/pub‑sub for event-driven architectures.
• Use cases: Fan‑out event distribution, streaming telemetry, decoupling microservices, real‑time pipelines.
• Guarantees: At‑least‑once delivery (clients must be idempotent), ordered delivery optionally via ordering keys, durable message storage, automatic retry until ack or message retention expires.
• When preferred: Use Pub/Sub when you need broadcast to multiple independent subscribers or high throughput event ingestion.

Cloud Tasks

• Purpose: Managed distributed task queue for directed, ordered, retryable HTTP tasks.
• Use cases: Offloading work to background workers, rate‑limited APIs, guaranteed single‑consumer processing, delayed/retryable jobs with configurable backoff.
• Guarantees: Exactly‑once delivery semantics from client perspective if you design idempotent handlers and use task leasing; configurable retry schedules; tasks are targeted to a single worker (no fan‑out).
• When preferred: Use Cloud Tasks when you need strict control over retries, scheduling per task, ordered processing, and one‑time work dispatch to a specific endpoint.

Cloud Scheduler

• Purpose: Cron-like scheduling to trigger HTTP endpoints, Pub/Sub topics, or App Engine jobs on a time schedule.
• Use cases: Periodic batch jobs, maintenance tasks, triggering daily reports or scheduled pipelines.
• Guarantees: Executes scheduled triggers; retries configurable for transient failures but not intended as a durable queue.

Example scenarios:

• Pub/Sub over Cloud Tasks: A telemetry service publishing sensor events to multiple analytics, monitoring, and alerting consumers simultaneously — Pub/Sub supports high throughput, fan‑out, and independent subscribers.
• Cloud Tasks over Pub/Sub: Processing user-initiated image conversions where each image must be processed exactly once with controlled rate to avoid overwhelming a downstream API — Cloud Tasks provides single-consumer dispatch, per‑task retries, and rate controls.

Trade-off summary:

• Use Pub/Sub for scalable, decoupled pub/sub patterns and fan‑out.
• Use Cloud Tasks for targeted, reliable, rate‑controlled work dispatch.
• Use Cloud Scheduler to trigger either Pub/Sub or Cloud Tasks on a schedule.

Requirements:

• Ingest 1 TB/day (~11.6 MB/s average, peaks higher), low-latency analytics, durable storage, schema evolution, cost control.
  High-level architecture:
  Pub/Sub (ingest) → Dataflow (stream processing, enrichment, windowing, dedupe) → BigQuery (OLAP / analytics) and Bigtable (hot key-value / low-latency lookups) → GCS (cold raw storage, archives).

Partitioning & sharding:

• Pub/Sub: use multiple topics/subscriptions or partitioning keys (attribute-based) so publishers route by customer/region to distribute load.
• Dataflow: set keying to match Pub/Sub attributes to avoid hotspots. Use hash-based sharding for high-cardinality keys.
• BigQuery: write to partitioned (date/_PARTITIONTIME) and clustered tables (cluster on customer_id, event_type) to reduce scan cost.
• Bigtable: design rowkeys with reverse-timestamp + hashed prefix to avoid hotspotting.

Windowing & aggregation:

• Use Dataflow (Beam) with event-time windowing and allowed lateness (e.g., 5–15 min) and watermarks. For sessionization use session windows. Emit both incremental streaming inserts to BigQuery and periodic micro-batches for heavy aggregations.

Fault tolerance:

• Rely on Pub/Sub at-least-once delivery. Implement idempotent processing in Dataflow using deduplication with state (event IDs TTL) or exactly-once sinks where supported. Enable Dataflow’s checkpointing and autoscaling. Persist raw events to GCS for replay.

Autoscaling & backpressure:

• Dataflow FlexRS / Streaming Engine with autoscaling workers; configure min/max workers and CPU/memory thresholds. Use batching and flow-control: Pub/Sub subscriber batching, Dataflow’s maxOutstandingElement/Bytes settings. Apply backpressure by throttling upstream producers (token-bucket) or using Pub/Sub subscription flow-control (ack deadlines, pull rate). Implement circuit-breaker patterns when downstream (BigQuery) quota is reached.

Storage & cost controls:

• BigQuery: use partitioned + clustered tables, use streaming buffer for low latency but avoid long-term heavy streaming (costly). For large ingest, prefer micro-batched loads via GCS (parquet/avro) using load jobs to reduce streaming insert costs. Use partition expiration and table-level cost controls; use materialized views for repeated queries.
• Bigtable: use for high-throughput lookup/store; reserve appropriate node counts and use autoscaling. Compact cold data to GCS.
• Monitor with Stackdriver: set alerts for Pub/Sub backlog, Dataflow worker count, BigQuery streaming insertion errors, Bigtable latency.

Trade-offs:

• Streaming inserts to BigQuery = low-latency but higher cost & quota management; micro-batched loads reduce cost at slight latency increase.
• Bigtable adds operational cost but provides low-latency random access versus BigQuery’s analytical strengths.

Operational practices:

• Define SLOs for latency, duplication, and cost. Run chaos tests, maintain schema registries, backfill paths from GCS, and provide runbooks for quota incidents.