Google Cloud Platform Services and Architecture Questions

Approach

Build a small, reusable Terraform module around one google_storage_bucket resource: versioning enabled, a lifecycle rule that deletes noncurrent object versions after a configurable number of days (30 by default), and a customer-managed encryption key (CMEK, a Cloud KMS key you control the lifecycle of, instead of Google's default encryption key) applied via the bucket's encryption block.

Terraform (HCL)

terraform {
  required_version = ">= 1.5"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 8.0"
    }
  }
}

variable "project" {
  description = "GCP project ID that owns the bucket"
  type        = string
}

variable "name" {
  description = "Globally unique bucket name"
  type        = string
}

variable "location" {
  description = "Bucket location, e.g. US, EU, or a region like us-central1"
  type        = string
  default     = "US"
}

variable "kms_key_name" {
  description = "Fully qualified self link of the Cloud KMS CryptoKey used as the customer-managed encryption key (CMEK). The GCS service agent must already hold roles/cloudkms.cryptoKeyEncrypterDecrypter on this key."
  type        = string
}

variable "noncurrent_version_ttl_days" {
  description = "Days to retain a noncurrent object version before it is deleted"
  type        = number
  default     = 30
}

resource "google_storage_bucket" "this" {
  project                     = var.project
  name                        = var.name
  location                    = var.location
  uniform_bucket_level_access = true
  force_destroy               = false

  versioning {
    enabled = true
  }

  lifecycle_rule {
    condition {
      days_since_noncurrent_time = var.noncurrent_version_ttl_days
    }
    action {
      type = "Delete"
    }
  }

  encryption {
    default_kms_key_name = var.kms_key_name
  }
}

output "bucket_name" {
  description = "Name of the created bucket"
  value       = google_storage_bucket.this.name
}

output "bucket_url" {
  description = "gs:// URL of the bucket"
  value       = google_storage_bucket.this.url
}

output "bucket_self_link" {
  description = "Terraform/API self link of the bucket, for cross-module references"
  value       = google_storage_bucket.this.self_link
}

Verified with terraform init, terraform fmt -check, and terraform validate against the hashicorp/google provider; no cloud credentials are available in this environment, so this confirms the configuration is structurally valid, not that an apply would succeed against a real project.

Inputs and outputs

Inputs: project, name (must be globally unique across all of Cloud Storage, not just this project), location, kms_key_name, and noncurrent_version_ttl_days with a sensible default so most callers never need to set it. Outputs: the bucket's name, its gs:// url for referencing in application config, and its self_link so other modules (an IAM binding module, for instance) can depend on this bucket without re-deriving its identifier.

State management for multi-team reuse

Use a remote backend for the module's state, a GCS backend is the natural choice here, with versioning enabled on the state bucket itself and a distinct object prefix per calling team or bucket instance, for example prefix = "buckets/${var.name}". Never let multiple teams share one state file for this module: a single bad apply from one team locking or corrupting shared state blocks everyone else. Publish the module from a versioned source, a Git tag or a private Terraform Registry module version, rather than a mutable branch reference, so teams pin a version and upgrade deliberately instead of inheriting an unreviewed change on their next apply. One state file per bucket instance, one root module invocation per team, keeps the blast radius of any mistake limited to that one bucket.

Trade-offs and pitfalls

uniform_bucket_level_access = true disables legacy per-object ACLs in favor of IAM-only access control, the secure default, but it means any workflow relying on per-object ACLs has to migrate to IAM conditions instead. days_since_noncurrent_time counts from the moment a version becomes noncurrent, not from when the object was originally created; turning on versioning for the first time on an already-old bucket means existing objects have no noncurrent history yet, so the rule only starts working going forward, which is an easy thing to misread as "broken" when it is actually working exactly as designed. The single most common way this exact module fails on its first apply is forgetting the Cloud Storage service agent's own IAM grant on the KMS key (roles/cloudkms.cryptoKeyEncrypterDecrypter); without it, every write to the bucket fails with a permissions error that has nothing obviously to do with the bucket configuration itself.

Direct answer

VPC Service Controls (VPC-SC) draws a service perimeter around a set of projects and the Google-managed APIs they use (Cloud Storage, BigQuery, and others), and by default blocks any request that would move protected data across that boundary, even if the caller's IAM (Identity and Access Management) permissions would otherwise allow it. Recommend setting up a perimeter whenever the real threat is a legitimate, authenticated identity (an employee with valid credentials, or credentials that leak) exfiltrating data to a personal project or an external account, since that is exactly the class of attack IAM alone cannot stop: IAM answers "is this identity allowed to touch this data," not "is this data allowed to leave this boundary."

Structured elaboration

What it's actually doing

• IAM controls who can call an API; VPC-SC controls where the data such a call touches is allowed to flow, independent of who's asking.
• Concretely, it blocks things like copying a BigQuery table or a Cloud Storage object to a project outside the perimeter, or accessing protected data from an unauthorized network location, even from someone who has full IAM permissions on both sides.
• Every perimeter can run in dry-run mode first: it logs what would have been blocked without actually blocking it, which is how you find out what breaks before you find out the hard way.

When to recommend it

• Data with real exfiltration consequences (customer PII, financial records, anything under a compliance mandate) sitting in Cloud Storage or BigQuery, where the org already trusts IAM to gate access but wants a second, independent control against a valid credential being misused or a config mistake exposing a bucket.
• Not a reflexive recommendation for every project: it adds real setup and ongoing maintenance cost (below), so it earns its place where the data's sensitivity justifies it.

Limitations to set expectations on up front

• VPC-SC does not support every Google Cloud service; an unsupported service inside a protected project can behave unpredictably or simply not work once the perimeter is enforced, so the supported-products list has to be checked against the project's actual footprint before enabling enforcement, not after.
• It is not designed to comprehensively control metadata movement (things like resource names or IAM policy metadata, as opposed to the data itself); IAM and custom roles remain the tool for governing metadata access.
• It only governs traffic to Google-managed APIs; it does not replace VPC firewall rules or network-level controls for traffic between your own VMs.

Worked example

A company keeps customer financial statements in a BigQuery dataset. Today, any employee with BigQuery Data Viewer on that dataset can bq extract or run a query with a destination table in any project their account can reach, including a personal sandbox project, entirely within IAM's rules. Wrapping the dataset's project in a VPC-SC perimeter (started in dry-run mode, monitored for a few weeks against real traffic, then switched to enforced) makes that same export attempt fail with a perimeter violation, unless the destination project is inside the same perimeter or an explicit perimeter bridge has been configured to allow it.

Trade-offs & pitfalls

The most common practical failure is enabling enforcement before checking which services in the protected project are unsupported, which breaks something like a CI/CD pipeline or a third-party integration that reaches a Google API from outside the perimeter, and does so in a way that reads as a mysterious outage rather than an obvious permissions error. Skipping dry-run mode to save time is the single riskiest shortcut here; it trades a controlled, observable rollout for an unplanned one.

Direct answer

A global external Application Load Balancer with Cloud CDN (content delivery network) in front of a stateless compute tier, Memorystore for Redis handling both caching and session state rather than sticky sessions on the app tier (pinning a given user's repeat requests to one specific backend instance so it can keep their session in local memory), and a managed relational database with read replicas for anything needing strong consistency. Size the compute tier off the stated 10,000 requests per second (RPS) peak, not the 1 million monthly active users (MAU) headline figure, since those two numbers imply very different things about actual concurrent load.

Architecture

flowchart LR
    C[Client] --> CDN[Cloud CDN]
    CDN --> LB[Global external Application Load Balancer]
    LB --> MIG[Compute tier: Cloud Run or GKE]
    MIG --> CACHE[Memorystore for Redis: sessions and cache]
    MIG --> DB[Primary database: Cloud SQL or Spanner]
    DB --> REPLICA[Read replicas]
    MIG -.metrics.-> MON[Cloud Monitoring]

• Compute. Cloud Run as the default for a request-driven web workload at this scale: it scales to zero between peaks, which matters because 1 million MAU very likely means far less than 10,000 RPS most of the day, only at peak. GKE (Google Kubernetes Engine) is the flip if the team already runs Kubernetes elsewhere or needs long-lived connections and custom autoscaling signals Cloud Run does not expose.
• Load balancing and CDN. The global load balancer terminates TLS and fans requests out by region; Cloud CDN caches static assets and cacheable API responses at Google's edge, which directly reduces the RPS the origin actually has to serve, worth stating explicitly since the 10,000 figure should be read as origin load after CDN offload, not before it.
• Caching and session state. Memorystore for Redis holds both the application cache and session state in one place, so any replica can serve any user's request without sticky sessions at the load balancer, which keeps autoscaling and failover simple.
• Database. Cloud SQL (managed Postgres or MySQL) with read replicas covers the typical read-heavy access pattern of a web application; Cloud Spanner is the right flip only once write throughput or dataset size genuinely outgrows what a single Cloud SQL primary can sustain, not adopted preemptively "just in case."

Worked capacity example

As an explicit assumption to validate with a real load test before launch, not a measured benchmark: if a single e2-standard-4 (a 4 vCPU Compute Engine machine type) application replica sustainably serves roughly 500 RPS for a lightweight JSON API, then 10,000 RPS divided by 500 RPS per replica is 20 replicas needed at peak. Provision the autoscaler with headroom above that, 25 to 30 replicas as a ceiling rather than exactly 20, to absorb one zone briefly losing capacity during a rolling update or a zonal disruption. The 500 RPS-per-replica figure is the first number to re-measure once real traffic exists, since it depends entirely on what a request actually does: one that hits the database on every call behaves very differently from one served mostly out of cache.

Balancing availability against cost

The single biggest availability-versus-cost decision here is regional scope. A multi-zone deployment within one region (multiple zones behind the regional backend, a Cloud SQL primary with a same-region standby for automatic failover) survives a zone outage at a modest cost premium over a single-zone design, roughly the cost of running enough replicas to tolerate losing one zone's worth of capacity, and is the right default at this scale. A full multi-region active-active design survives an entire region going down, but it roughly doubles infrastructure cost and adds real complexity keeping the database consistent across regions (cross-region replication lag, or a much more expensive globally-consistent option like Spanner in multi-region mode). For 1 million MAU with a 10,000 RPS peak, I would default to multi-zone within one region and treat full multi-region as something to add only once a specific SLA (service-level agreement) or regulatory requirement demands surviving a whole-region outage, since that jump in cost is large relative to the incremental availability it buys at this traffic level.

Trade-offs and pitfalls

Cloud Run's scale-to-zero is a double-edged default at this scale: a genuine cold start on the first request after an idle period adds latency exactly when a user is least tolerant of it, mitigated by setting a minimum instance count sized off historical trough traffic, not zero. Putting session state in Memorystore means a Memorystore outage takes down active sessions fleet-wide, a single point of failure traded for architectural simplicity, worth pairing with a Memorystore high-availability replica. Read replicas only help a read-heavy pattern; if the 10,000 RPS peak is actually write-heavy, an ingestion API rather than a typical read-mostly web app, the compute sizing above is optimistic and Spanner or a sharded design becomes relevant much sooner than the numbers here suggest. The most common wrong turn is sizing compute off the 1 million MAU headline instead of the stated 10,000 RPS peak; those two figures imply very different average concurrency depending on session length and usage pattern, and sizing off the wrong one either wastes budget or under-provisions for the load that actually matters.

Direct answer

The hard part of this design isn't the Kubernetes layer, which the fleet model (GKE's mechanism for managing multiple clusters, on-prem and in GCP alike, as one logically grouped unit with shared configuration and policy) handles reasonably directly, it's that stateful workloads carry data that has to physically move or replicate between on-prem and GCP. Minimal disruption means the migration plan has to be built around that data's replication and cutover timeline, with multi-cluster service discovery and centralized config sync handling the "make it look like one fleet" part around the edges of that core data problem.

Designing the hybrid architecture

Multi-cluster service discovery

Fleet-level multi-cluster services let a service in the GCP cluster be discoverable and callable from the on-prem cluster, and vice versa, using the same service name and mesh-level routing regardless of which cluster it's actually running in. This is what lets a stateless caller of the stateful workload migrate independently of migrating the stateful workload itself, the caller doesn't need to know or care which cluster currently hosts the data tier it calls.

Centralized policy enforcement and config sync

Config Sync keeps both the on-prem and GCP clusters' RBAC (role-based access control), admission policy, and namespace conventions identical by continuously reconciling both clusters against the same Git-sourced configuration. A stateful workload migrating from one cluster to the other lands in an environment enforcing exactly the same policy it left, no surprise admission-control rejection or RBAC gap on the new side.

Storage persistence, two distinct problems

This is the crux of the question, and it splits into two related but distinct problems. First, persistence through a routine cluster upgrade, not a migration: stateful workloads rely on PersistentVolumes (the Kubernetes objects that give a pod durable disk storage that survives the pod being rescheduled or restarted), so the storage class (the setting that tells Kubernetes which underlying disk technology and behavior to provision a PersistentVolume from) and underlying backend need to support live migration or reattachment of volumes across nodes during a surge-style upgrade, the same surge-upgrade concept that keeps capacity from dipping during any routine GKE (Google Kubernetes Engine) cluster upgrade, extended to the on-prem cluster too, so a node being upgraded doesn't strand a volume. Second, persistence across a migration from on-prem to GCP, where the two sides genuinely use different storage backends: this needs its own continuous data-replication mechanism, database-level replication for a database workload, the same kind of continuous, monitored replication a cross-region database failover setup relies on, or a storage-level replication tool for other stateful workloads, run continuously ahead of cutover, with a validated cutover step rather than a single-shot copy that risks losing anything written during the copy window.

Sequencing for minimal disruption

Stand up the destination GCP cluster with Config Sync already applying the same policy baseline, establish continuous data replication for the stateful workload from on-prem to GCP, migrate stateless callers first using multi-cluster service discovery so they can reach the still-on-prem data tier transparently, then cut the data tier over last, once replication lag is at zero and validated, mirroring the same cutover discipline any careful database migration needs, but applied to whatever storage technology backs this specific workload.

Worked example

A stateful order-database workload running on-prem needs to move to GCP with minimal disruption. The team stands up the GCP-side GKE cluster under the same fleet, with Config Sync already enforcing identical namespace and admission policy. They configure continuous database-level replication from the on-prem primary to a GCP-hosted replica, following the same discipline any careful database migration needs, continuous replication, monitored lag, a defined point of no return. Meanwhile the stateless API layer reading and writing this database migrates to the GCP cluster first, using multi-cluster service discovery to keep calling the still-on-prem database transparently, no code change required to reach across clusters. Only once replication lag is confirmed at zero does the team promote the GCP-side database, cut the API layer's connection over, and decommission the on-prem database, keeping the real risk concentrated at the single validated moment rather than spread across the whole migration window.

Trade-offs and pitfalls

Treating this as a pure Kubernetes-manifest migration, and only discovering the storage persistence problem once the stateful workload actually needs to move, is the biggest pitfall, the multi-cluster service discovery and config sync pieces are the easy 80 percent, data replication and cutover discipline is the hard 20 percent that actually determines whether this is disruptive. Relying on config sync alone to make the two environments "the same," while skipping validation that the on-prem and GCP storage backends actually behave equivalently for PersistentVolumes, is another, a storage class that behaves differently on each side can silently break assumptions the workload depends on. A third pitfall, familiar from any database migration, is having no defined point of no return for the data cutover, once writes start landing on the GCP side, rolling back to on-prem is a second migration, not an undo.

Direct answer

With an 8-hour job and a 12-hour window, there are 4 hours of slack, a third of the window, which is exactly the kind of margin that makes discounted, preemptible capacity the right default: run on preemptible VMs with checkpointing, and treat stable custom machine types in an autoscaled managed instance group as a time-boxed fallback that kicks in only if the job falls behind schedule partway through, rather than choosing one mode for the whole run up front.

Decision framework

• What decides preemptible vs. stable custom machine types here is deadline slack, not just cost. A job with zero slack against its deadline can't absorb an unplanned interruption at all, so it has to run on guaranteed capacity regardless of price. A job with generous slack can absorb the occasional restart and still land inside the window. This job's slack, 4 of 12 hours, sits in the "worth it, but verify" zone rather than an obvious call either way.
• Orchestration matters as much as the pricing model. A hand-rolled fleet of preemptible VMs needs its own checkpoint and restart logic. If the workload already runs on Dataflow (for streaming or batch data processing) or Dataproc (managed Spark and Hadoop), those services already handle worker loss, and layering preemptible capacity on top is close to free from an engineering-effort standpoint. Building that resilience from scratch on a bespoke managed-instance-group pipeline is real work, and that effort belongs in the decision, not just the sticker price of the VM.
• A middle ground exists: run predominantly on preemptible capacity, but track elapsed time against expected progress, and if the job is meaningfully behind pace partway through, automatically fail the remainder over to a stable custom machine type sized to the workload, running in an autoscaled managed instance group, to guarantee the 12-hour deadline. This turns the choice from a single up-front bet into a policy that only pays the guaranteed-capacity premium on the minority of runs that actually need it.
• Justifying it to two audiences with different concerns: finance cares about the average and worst-case monthly bill; the honest framing is that most runs finish well within budget on discounted capacity, and the rare run needing the stable-capacity fallback costs more than average but still finishes on time, not that this is always cheap. The ML team cares about whether the pipeline reliably delivers a trained model inside the window; the honest framing is that the deadline is protected by a fallback policy, not by hoping interruptions don't happen, with the fallback trigger as a number they can see and push back on.

Worked example

Historical runs show the job completing its 8 hours of actual compute in a median of 8.5 wall-clock hours on preemptible capacity, some interruption-driven restarts included, comfortably inside the 12-hour window, but with a long tail where a bad run stretches past 10 hours. Set the fallback trigger at the 6-hour mark: if less than roughly 60% of total work is complete by then, switch the remaining partitions to a stable custom machine type in the autoscaled group. A job on pace would show close to 75% complete at 6 hours (6 of 8 expected hours), so falling well under 60% is a legitimate signal that this run's interruption luck is bad enough to put the 12-hour deadline at risk, and paying the guaranteed-capacity premium for the remainder is worth it. This threshold is a policy decision to state explicitly to the ML team, not a fixed law: tune it against real historical run data once it exists.

Trade-offs and pitfalls

• Committing entirely to preemptible capacity with no fallback is a real bet against the deadline: a bad interruption streak is rare but not impossible, and "fine on average" is a weak answer when a stakeholder asks about the worst case that breaks a 12-hour window.
• Committing entirely to stable custom machine types to eliminate all risk throws away the savings that justified considering preemptible capacity for a workload explicitly described as interruption-tolerant, the opposite failure.
• A fallback policy adds orchestration complexity, progress tracking, a decision point, a path to re-provision on stable capacity mid-run, that must itself be tested, or the safety net doesn't actually catch anything the one time it's needed.
• Discounted capacity availability varies by region and machine type; if the fallback pool draws from the same constrained configuration, "switch to stable" may not be instantly available either. Reserve headroom, or a committed baseline, for the fallback path specifically.