Google Cloud Platform Services and Architecture Questions

Direct answer

I would not grant root or SSH (Secure Shell) access to GKE (Google Kubernetes Engine) nodes, but I would not just say no either. The nodes are managed infrastructure, GCP controls their kernel, patch level, and lifecycle, and out-of-band SSH changes break the assumptions the whole managed offering rests on. What I would do instead is find out what the customer is actually trying to prove with "custom diagnostics" and offer a path that gets them that without handing over the node itself.

The alternatives, and when each fits

• Debug containers (kubectl debug, which attaches an ephemeral container to a running pod's namespaces, its network namespace in particular, without modifying the pod spec or touching the node) cover most application-level diagnostics: strace, tcpdump, or curl run from exactly the pod's point of view. This is the right answer for almost everything a customer means by "run custom diagnostics" against their own workload.
• An audited ephemeral bastion: a short-lived VM, or an Identity-Aware Proxy (IAP, GCP's context-aware access proxy) tunneled session, granted roles/container.clusterViewer plus a narrow, time-boxed elevated role for the one diagnostic in question, with the session's commands captured in Cloud Audit Logs at minimum, ideally session-recorded. This covers cluster-level diagnostics that genuinely need more than a pod-scoped debug container, without ever putting a shell on a production node.
• Private clusters with restricted access, framed the right way: if the underlying ask is really "prove you can support this in an incident," the honest answer is to walk them through what an actual support engineer sees during a real incident, aggregated logs and metrics in Cloud Monitoring and Cloud Logging, GKE control-plane diagnostics, not raw node access. That is usually the proof they are actually looking for.

Worked example

A concrete way to open this conversation: "I hear that you want confidence you can debug this in production. Root on the node would actually make that harder, because any change GKE's node auto-upgrade or auto-repair doesn't know about gets silently overwritten on the next maintenance cycle, so anything you fixed by hand there disappears without warning. Let me show you kubectl debug against a real pod instead, it gets you the same tcpdump output you're asking for, and it's still there next week." Then actually run it live against a sample workload rather than just describing it, since a live demonstration is what actually closes this objection.

Security and support implications

Root on a node bypasses every namespace and RBAC (role-based access control) boundary the cluster depends on: a single careless or compromised root session can read every Secret mounted on that node across every pod scheduled there, not just the one the customer cares about. Any manual, out-of-band change (an installed package, an edited config file) creates configuration drift that GKE's own node auto-upgrade or auto-repair will silently erase on the next cycle, so a customer who insists on this is implicitly signing up for their own fix disappearing without notice. On the support side, once a node has been modified outside of GKE's own lifecycle, "it's still a stock GKE node" is no longer a safe assumption; if something breaks afterward, it becomes an open-ended debugging exercise instead of a known, reproducible managed platform, which undercuts the reliability argument the demo itself is usually trying to make.

Trade-offs and pitfalls

Refusing outright, with no alternative offered, reads as unhelpful and can lose the deal on tone alone even when the underlying reasoning is correct; leading with an alternative and only naming the refusal in passing lands much better. The other common mistake runs the opposite direction: granting SSH "just for the demo, we'll lock it down afterward" is a slippery slope, that access is rarely fully revoked once granted, and it quietly becomes the customer's standing expectation for production too, not just the proof-of-concept.

Direct answer

The throughput requirement, 1 Gbps sustained, is comfortably within HA VPN's per-tunnel capability on paper, but the latency requirement, under 10ms round-trip time (RTT), is the one that actually decides this: HA VPN runs over the public internet, where RTT depends on internet peering paths outside GCP's control, while Dedicated Interconnect is a direct physical cross-connect that bypasses the public internet entirely, so it's the only one of the two that can reliably guarantee a low, stable RTT. The recommended architecture is Dedicated Interconnect as the primary path, with redundant circuits for availability, and HA VPN kept as a backup path that's explicitly allowed to miss the latency target during the rare window it's actually carrying traffic.

Decision framework

• Throughput math for HA VPN: each HA VPN tunnel is capped at 250,000 packets per second (pps) for combined ingress and egress traffic, a ceiling that translates into a bandwidth figure depending on average packet size.

At a representative 1,200-byte average packet size, a single tunnel can theoretically clear the stated 1 Gbps requirement, so throughput alone doesn't rule out HA VPN. A smaller average packet size lowers that ceiling, and HA VPN's standard two-tunnel active-active configuration, needed anyway for its 99.99% SLA (service-level agreement), provides real headroom above 1 Gbps regardless.

• Latency is the real constraint: achieving a reliable sub-10ms RTT over the public internet is possible only if the on-prem site happens to be very close, in network terms, to a well-peered Google entry point, not an assumption to design around. Dedicated Interconnect's physical cross-connect at a colocation facility avoids the public internet path entirely, giving a round-trip time that depends mainly on the physical distance between the on-prem site and the interconnect facility, the only one of the two architectures that can be engineered to reliably meet a hard sub-10ms target.
• Redundancy and failover: provision at least two Dedicated Interconnect connections in different edge availability domains (the 99.9% SLA tier) or, if the requirement justifies the added cost, across two metros for the 99.99% tier, with Cloud Router running BGP (Border Gateway Protocol, the routing protocol used to exchange reachable network ranges between the on-prem router and GCP) on each. Configure HA VPN as a genuinely separate backup path, with less-preferred BGP routing metrics so it only carries traffic when both Interconnect paths are down, and treat that failover window as an explicit, disclosed exception to the latency target rather than a silent gap: RTT will be worse than 10ms during that window, and that trade-off needs to be agreed with the business up front.

Worked example

flowchart LR
  ONPREM[On-prem router] --> IC1[Dedicated Interconnect edge domain A]
  ONPREM --> IC2[Dedicated Interconnect edge domain B]
  ONPREM --> VPN[HA VPN gateway as backup path]
  IC1 --> CR[Cloud Router running BGP]
  IC2 --> CR
  VPN --> CR
  CR --> VPCN[GCP VPC and application]

For the stated 1 Gbps and sub-10ms requirement, provision a 10-Gbps Dedicated Interconnect circuit, well above the 1 Gbps sustained need, in two edge availability domains within the same metro as the on-prem site, giving the 99.9% redundant SLA tier without requiring a second physical location. Cloud Router on each connection exchanges routes via BGP, and HA VPN is configured as a lower-priority backup route for the rare case both Interconnect paths fail simultaneously. As a related scenario worth flagging: if the actual sustained requirement were 10 Gbps instead of 1 Gbps, a single 10-Gbps circuit would be sized with exactly zero headroom against the requirement, a real risk during any burst above baseline or during a failover event that concentrates traffic onto one remaining circuit. At 10 Gbps sustained, the right move is either two bonded 10-Gbps circuits or a single 100-Gbps circuit, so the requirement itself doesn't consume the entire available margin.

Trade-offs and pitfalls

• Treating the throughput number as the binding constraint and stopping the analysis there is the most likely mistake here: the math shows HA VPN can plausibly carry 1 Gbps, but the latency requirement is what actually rules it out as the primary path, and a design that satisfies throughput while ignoring latency fails the stated requirement anyway.
• Sizing the Dedicated Interconnect circuit exactly to the 1 Gbps requirement, with no headroom, repeats the same margin problem the 10-Gbps case above illustrates at a different scale.
• Presenting HA VPN as a backup path without disclosing that failover means accepting worse latency for the failover window is a common way this design surprises the business later; state that trade-off explicitly in the proposal, not as a footnote discovered during an incident.
• The pps-based throughput ceiling depends on packet size, which the application's own traffic pattern controls; a workload sending mostly small packets gets meaningfully less throughput per tunnel than the 1,200-byte assumption used here, so this calculation should be revisited against the actual traffic, not assumed.

Direct answer

Workload Identity and Workload Identity Federation solve the same underlying problem, letting a workload authenticate to Google Cloud without a downloaded, long-lived service account key, for two different populations. Workload Identity is specifically for workloads running on GKE (Google Kubernetes Engine), binding a Kubernetes service account to a Google Cloud service account so pods get automatically rotated credentials. Workload Identity Federation is the more general mechanism for workloads running anywhere else, another cloud, a CI/CD system, on-prem, letting an external identity provider's token be exchanged for a short-lived Google Cloud credential. Setting up GitHub Actions to deploy without a stored key is a Workload Identity Federation configuration, since GitHub Actions runs entirely outside GCP.

How both mechanisms work

Why long-lived keys are the thing both replace

A downloaded service account key is a bearer credential, whoever holds the file can authenticate as that service account from anywhere, until someone notices and revokes it. It typically doesn't expire on any short timeline by default, and it lives wherever it was downloaded to, a laptop, a CI secret store, sometimes accidentally a public repository, all of which make it a disproportionately attractive and long-lived target if it leaks.

Workload Identity, GKE-specific

Binds a Kubernetes service account, a namespaced identity inside the cluster, to a Google Cloud service account via an IAM (Identity and Access Management) policy binding, so a pod running as that Kubernetes service account can request short-lived Google Cloud credentials through the metadata server, with no key file involved anywhere. Scope is per-namespace and per-Kubernetes-service-account, which is what makes a per-tier, per-workload service account design practical to enforce all the way down at the pod level.

Workload Identity Federation, general and external

A workload identity pool holds a trust relationship with an external identity provider, GitHub's OpenID Connect (OIDC, a standard way for an external system to prove an identity via a signed token) issuer, another cloud's identity mechanism, or a generic OIDC provider, and a pool provider within it maps claims from that external token, for GitHub Actions, things like the repository name and the workflow that's running, onto attributes GCP can make an authorization decision on. Google's Security Token Service exchanges a valid, unexpired external token for a short-lived Google Cloud access token, which the workload then uses to impersonate a specific service account, the more common pattern, keeping IAM permissions defined on the service account rather than scattered across the federation configuration itself.

Setting up GitHub Actions to deploy without a key

The conceptual steps, verify exact console or CLI syntax against current documentation before executing, since exact commands were not run in this write-up: create a workload identity pool for external CI/CD identities; add a provider inside it configured for GitHub's OIDC issuer, with an attribute condition restricting which specific repository and workflow is trusted, this condition is the actual security boundary, without it any GitHub Actions workflow anywhere could potentially claim to be your organization's; grant the target service account, the one with the actual deploy permissions, scoped as narrowly as the specific deploy task actually requires, the IAM permission allowing that specific pool, provider, and repository combination to impersonate it; and in the workflow itself, use Google's official authentication action configured to request a token from the workload identity provider and impersonate the target service account, with no key ever stored as a repository secret.

Worked example

Before this setup, a "deploy-to-prod" GitHub Actions workflow stores a downloaded service account key as a repository secret, a standing risk for as long as it exists, readable by anyone with the right repository permissions and usable from anywhere if it ever leaks out of the secret store. After configuring Workload Identity Federation scoped to that exact repository and workflow, the workflow instead presents GitHub's own short-lived OIDC token, issued fresh for that specific run and expiring shortly after, which Google's Security Token Service exchanges for a Google Cloud credential able to impersonate the deploy service account, but only because the pool provider's attribute condition specifically named this repository. The repository secret can be deleted entirely, there is nothing left to leak.

Trade-offs and pitfalls

Configuring the pool provider without a restrictive attribute condition is the single most consequential mistake here, an unrestricted provider trusting any GitHub Actions token rather than one scoped to your specific organization, repository, and branch is a much bigger blast radius (the scope of what an attacker can do with it) than a leaked key would have been, verify the condition is actually scoped before considering this done, never assume the default is safe. Conflating Workload Identity and Workload Identity Federation as the same feature because they solve the same underlying problem is another pitfall, they have different setup mechanics and apply to different workload locations, GKE's Workload Identity documentation will not configure a GitHub Actions pipeline, or the reverse. A third pitfall is granting the impersonated service account broad permissions "since it's not a key anymore anyway," removing the key doesn't remove the need for least privilege on what that identity is actually allowed to do once authenticated.

Direct answer

The four sit on two axes: relational versus non-relational, and single-region-scale versus horizontally-distributed-scale. Cloud SQL is a single-region relational database for standard transactional applications. Cloud Spanner is a globally-distributed relational database for applications that need both strong consistency and horizontal write scale beyond what one machine can do. Firestore is a document database for applications built around flexible, hierarchical JSON-like records with real-time sync to clients. Bigtable is a wide-column store for massive-scale, high-throughput workloads with simple, known access patterns (look up by key, or scan a key range).

Structured elaboration

Four application patterns mapped to a fit:

1. A subscription billing system needing foreign keys between customers, invoices, and payments, with standard SQL joins and moderate write volume: Cloud SQL. The relational model and ACID transactions match the domain directly, and the write volume doesn't need horizontal sharding.
2. A global multiplayer game's inventory and matchmaking ledger, needing strict consistency (an item can't be duplicated by a race condition) at a write volume no single machine could sustain, across players worldwide: Cloud Spanner. This is the case that specifically needs both properties Cloud SQL doesn't offer together: horizontal write scale and strong consistency.
3. A collaborative note-taking or task app with a mobile client that needs to work offline and sync automatically when reconnected, with a flexible per-document schema (some notes have attachments, some have checklists): Firestore. The document model and built-in real-time listener/offline-sync support solve exactly that client-facing problem.
4. An IoT platform ingesting sensor telemetry from millions of devices, where the access pattern is almost entirely "write a new reading" and "scan a device's recent readings by time range," with no need for joins or multi-row transactions: Bigtable. Its wide-column model and linear scaling handle this write-heavy, simple-access-pattern workload far more cheaply per unit of throughput than a relational engine would.

Worked example

Take the same underlying entity, a "user session," across two of these services to show the model actually drives the choice, not habit. In Cloud SQL, a sessions table with a foreign key to users, queried with a JOIN to fetch session plus user profile in one round trip, fits naturally because the access pattern is relational (session belongs to exactly one user, and you often want both together). In Bigtable, the same conceptual data modeled as row_key = user_id#reverse_timestamp, with columns for session metadata, fits a workload that instead needs "give me this user's most recent 50 sessions, fast, at massive scale, with no joins," because the reverse-timestamp suffix keeps a user's most recent sessions physically adjacent for fast range scans without needing a secondary index. The same "session" concept, modeled two structurally different ways, because the question being asked of the data differs.

Trade-offs & pitfalls

• Reaching for Spanner as a default "just in case we need to scale" choice when Cloud SQL would comfortably handle the actual write volume adds real operational and cost overhead (Spanner's minimum viable footprint costs more than a modest Cloud SQL instance) for a scale problem you don't have yet.
• Using Firestore for data that's genuinely relational (heavy cross-collection joins simulated via multiple reads and client-side merging) fights the model; Firestore has no server-side join, so relational-shaped data forces either denormalization (duplicating data across documents, with the consistency risk that implies) or multiple round trips.
• Using Bigtable for anything needing multi-row transactions or ad hoc query flexibility is a common mismatch; Bigtable's atomicity guarantee stops at a single row, and its query model is fundamentally "scan by key or key range," not "filter by an arbitrary non-key column" without a design decision baked into the row key up front.

Approach

Provision a private GKE cluster (control plane, the managed Kubernetes API server that runs the cluster, and nodes both without public IPs) with two node pools carrying distinct labels and taints so system and application workloads never land on the same nodes by accident, VPC-native IPs (pods and Services get real, routable addresses from the VPC's own address space via secondary ranges, rather than an overlay network), and a Cloud NAT gateway (a managed Network Address Translation service that lets private nodes reach the internet outbound, for example to pull images, without any node having a public IP).

Terraform (focused snippets)

terraform {
  required_version = ">= 1.5"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 8.0"
    }
  }
}

variable "project" {
  type = string
}

variable "region" {
  type    = string
  default = "us-central1"
}

variable "network_self_link" {
  description = "Self link of an existing VPC with a VPC-native subnet already sized for pods/services secondary ranges"
  type        = string
}

variable "subnetwork_self_link" {
  type = string
}

provider "google" {
  project = var.project
  region  = var.region
}

resource "google_compute_router" "nat_router" {
  name    = "gke-nat-router"
  network = var.network_self_link
  region  = var.region
}

resource "google_compute_router_nat" "nat" {
  name                               = "gke-nat"
  router                             = google_compute_router.nat_router.name
  region                             = var.region
  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
}

resource "google_container_cluster" "private" {
  name     = "private-gke"
  location = var.region

  # Manage node pools as separate resources below, not the cluster's default pool.
  remove_default_node_pool = true
  initial_node_count       = 1

  network    = var.network_self_link
  subnetwork = var.subnetwork_self_link

  networking_mode = "VPC_NATIVE"
  ip_allocation_policy {
    # Uses the subnet's configured secondary ranges for pod and service IPs.
    cluster_secondary_range_name  = "pods"
    services_secondary_range_name = "services"
  }

  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = false # set true to also remove the public control-plane endpoint
    master_ipv4_cidr_block  = "172.16.0.0/28"
  }

  master_authorized_networks_config {
    cidr_blocks {
      cidr_block   = "10.0.0.0/8"
      display_name = "corp-and-cloud-nat-ranges"
    }
  }

  # Security defaults
  workload_identity_config {
    workload_pool = "${var.project}.svc.id.goog"
  }

  release_channel {
    channel = "REGULAR"
  }

  network_policy {
    enabled  = true
    provider = "CALICO"
  }

  binary_authorization {
    evaluation_mode = "PROJECT_SINGLETON_POLICY_ENFORCE"
  }

  database_encryption {
    state = "DECRYPTED" # set ENCRYPTED with a kms_key_name for CMEK-encrypted etcd secrets
  }
}

resource "google_container_node_pool" "system" {
  name     = "system-pool"
  cluster  = google_container_cluster.private.name
  location = var.region

  node_count = 2

  node_config {
    machine_type    = "e2-standard-4"
    service_account = google_service_account.gke_nodes.email
    oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]

    labels = {
      workload-class = "system"
    }

    taint {
      key    = "workload-class"
      value  = "system"
      effect = "NO_SCHEDULE"
    }

    shielded_instance_config {
      enable_secure_boot          = true
      enable_integrity_monitoring = true
    }

    workload_metadata_config {
      mode = "GKE_METADATA"
    }
  }
}

resource "google_container_node_pool" "application" {
  name     = "app-pool"
  cluster  = google_container_cluster.private.name
  location = var.region

  autoscaling {
    min_node_count = 1
    max_node_count = 10
  }

  node_config {
    machine_type    = "e2-standard-4"
    service_account = google_service_account.gke_nodes.email
    oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]

    labels = {
      workload-class = "application"
    }

    taint {
      key    = "workload-class"
      value  = "application"
      effect = "NO_SCHEDULE"
    }

    shielded_instance_config {
      enable_secure_boot          = true
      enable_integrity_monitoring = true
    }

    workload_metadata_config {
      mode = "GKE_METADATA"
    }
  }
}

resource "google_service_account" "gke_nodes" {
  account_id   = "gke-node-sa"
  display_name = "Minimal-privilege GKE node service account"
}

output "cluster_endpoint" {
  value     = google_container_cluster.private.endpoint
  sensitive = true
}

Verified with terraform init, terraform fmt -check, and terraform validate against the hashicorp/google provider; there is no live GCP project behind this, so this confirms structural validity, not a successful apply.

Security defaults to enable

• Private nodes and, optionally, a private endpoint (private_cluster_config): nodes never get public IPs; enable_private_endpoint can additionally remove the control plane's public IP, at the cost of needing a bastion or a private connectivity path to reach it.
• Workload Identity Federation for GKE (workload_identity_config, commonly still called just Workload Identity): pods authenticate to Google Cloud APIs as a Google service account, not by reading a key off the node, closing off a classic path to credential theft.
• Shielded GKE nodes (shielded_instance_config): secure boot and integrity monitoring detect boot-level tampering on a node.
• Binary Authorization at the cluster level: only images carrying required attestations are admitted.
• A REGULAR release channel: nodes and the control plane receive tested upgrades automatically, instead of aging out of support on a static pinned version.
• Network policy enabled (Calico): pod-to-pod traffic can be default-deny, rather than every pod reaching every other pod by default.
• master_authorized_networks_config: even though the endpoint is already private or restricted, this further limits which CIDR (Classless Inter-Domain Routing, the notation for an IP address range) ranges can reach the control plane API at all.
• GKE_METADATA workload metadata mode: prevents a pod from reading the node's own metadata server directly, another common path to node-level credential theft.

Trade-offs and edge cases

Leaving enable_private_endpoint false, as this snippet does, keeps a public IP on the control plane reachable only from the CIDR ranges in master_authorized_networks_config, a pragmatic default that still lets kubectl work from a corporate network or Cloud Build without extra plumbing. Flipping it to true removes that public IP entirely, more secure, but then reaching the API needs a bastion, Cloud NAT-based egress (outbound-only internet access) from a jump host, or Connect Gateway, real operational overhead. Taints on the system and application node pools only isolate workloads that actually declare a matching toleration; a workload deployed without one silently schedules onto whichever pool has room, quietly defeating the separation this design is meant to enforce. Cloud NAT needs enough allocated addresses for the outbound connection volume the cluster will actually generate; undersizing it does not fail at apply time, it fails later as connection exhaustion under real load. One genuine edge case: remove_default_node_pool = true with initial_node_count = 1 still briefly creates and tears down a default pool node on the very first apply, expected behavior, not a bug, but it does make that first apply noticeably slower than every one after it.