Kubernetes Architecture, Operations, and Troubleshooting Questions

GPU scheduling in Kubernetes works through three cooperating pieces: the device plugin framework, a kubelet-side extension mechanism that advertises non-standard hardware as an allocatable resource, the resulting extended resource name a pod requests, and optional node labels used for affinity. None of it works out of the box; the cluster needs a working NVIDIA driver, a GPU-aware container runtime, and the NVIDIA device plugin running before any pod can actually get scheduled onto a GPU.

How the pieces fit together

• Node prerequisites. Every GPU node needs the NVIDIA driver installed at the kernel level, and the container runtime, containerd or CRI-O, since Docker itself was removed as a supported Kubernetes container runtime in version 1.24, configured with the NVIDIA Container Toolkit so a container can actually use the device, not just have it requested.
• Device plugin. The NVIDIA device plugin runs as a DaemonSet, one copy per GPU node, and registers with the kubelet over the device plugin API, advertising each GPU as a unit of the extended resource nvidia.com/gpu. Only after this registration does kubectl describe node show nvidia.com/gpu under that node's allocatable resources.
• Resource request. A pod requests a GPU the same way it requests CPU or memory, by naming the extended resource under resources.limits. GPUs are requested as a whole-number limit, not split fractionally the way CPU can be; a container either gets a whole GPU or none.
• Node labels. Labels such as nvidia.com/gpu.present or a GPU-model label are typically populated automatically by an NVIDIA feature-discovery component (part of the NVIDIA GPU Operator), not applied by hand; they exist so a pod can use nodeSelector or node affinity to target GPU-capable nodes, or a specific GPU generation, rather than relying on the extended-resource request alone to imply placement.

Minimal pod requesting one GPU

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: cuda-test
    image: nvidia/cuda:12.4.1-base-ubuntu22.04
    resources:
      limits:
        nvidia.com/gpu: 1

Worked example: why a GPU request can go Pending

Suppose kubectl describe node gpu-node-1 shows the following, an illustrative example rather than a live cluster:

Capacity:
  nvidia.com/gpu:     4
Allocatable:
  nvidia.com/gpu:     4
Allocated resources:
  Resource         Requests   Limits
  --------         --------   ------
  nvidia.com/gpu   3          3

Free capacity on this node is

$4 - 3 = 1$

GPU. A new pod requesting nvidia.com/gpu: 2 cannot fit here even though the node exists and is healthy; if it is the only GPU node, or the only one with free capacity, the pod stays Pending, and kubectl describe pod on it shows an Insufficient nvidia.com/gpu scheduling event, the extended-resource equivalent of the far more common insufficient-CPU or insufficient-memory Pending cause.

Trade-offs and pitfalls

• Whole-GPU requests only: unlike CPU, a container cannot request half a GPU through the base device plugin; workloads that only need a fraction of a GPU's capacity either waste the rest or need NVIDIA's Multi-Instance GPU (MIG) partitioning on hardware that supports it, current-generation data-center GPUs, which exposes each partition as its own schedulable unit.
• Forgetting a node label, when policy requires one, while the device plugin resource is present still lets the pod schedule anywhere with free nvidia.com/gpu capacity, which is not always the intended node if a cluster mixes GPU generations.
• This is a genuinely narrow, machine-learning-infrastructure-specific mechanism; a general Kubernetes operations interview is unlikely to probe device plugins in depth, but it is a real and common question in ML-platform-focused loops.

The Horizontal Pod Autoscaler (HPA) never talks to Prometheus, your app, or a cloud queue directly: it only ever queries one of three Kubernetes metrics APIs (metrics.k8s.io, custom.metrics.k8s.io, external.metrics.k8s.io), so scaling on queue length requires something that exposes the queue's depth through one of those APIs. The standard pipeline is: the application or a sidecar exposes the metric, a metrics system collects it, and a metrics adapter (most commonly prometheus-adapter, or a project like KEDA, Kubernetes Event-Driven Autoscaling, which ships its own adapter and is now the more common current choice specifically for external event sources like queues) is registered with the API server as an aggregated API and translates queries into that metric API's shape.

The three metrics APIs

A message queue's depth (say, a managed queue service or a Kafka consumer-group lag) is not a property of any Kubernetes object, so it belongs under external.metrics.k8s.io and the HPA metric type: External, not Pods or Object.

Required components and how the metric gets exposed

1. Instrumentation: the application (preferred) or a sidecar exporter publishes a metric, e.g. a Prometheus gauge myapp_queue_length{queue="orders"} on /metrics.
2. Prometheus scrapes it on a scrape job.
3. prometheus-adapter is deployed with rules mapping a PromQL query to an external metric name Kubernetes will expose, and it registers an APIService (apiregistration.k8s.io) so kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 returns real data. This registration needs its own RBAC: a ClusterRole granting the HPA controller's service account (system:kube-controller-manager reaching through the aggregation layer) permission to read the external metrics API, plus the adapter's own service account needing permission to read Prometheus.
4. The HPA references the metric by name:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric: {name: queue_length_orders}
        target: {type: AverageValue, averageValue: "100"}

(autoscaling/v2 has been the stable API since Kubernetes 1.23; the older v2beta2 was removed in 1.26, so any manifest or tooling still referencing it is out of date.)

Operational pitfalls

• A silently wrong zero is worse than a visible failure. If the adapter genuinely can't reach its data source, the HPA does not quietly treat that as "no load": it sets the ScalingActive condition to False with reason FailedGetExternalMetric and shows the metric's current value as <unknown> in kubectl describe hpa, holding replica count steady. The real danger is the opposite case: an adapter that, on a query error, returns a literal 0 instead of erroring. That looks healthy to the HPA and drives a real scale-down during an actual outage in the metrics path, which is why adapter error-handling is worth testing explicitly rather than assumed.
• Cardinality. High-cardinality labels on the underlying metric (one series per customer ID, for instance) can make Prometheus memory blow up long before the HPA ever sees a problem; keep the label set the adapter maps from small and stable.
• Flapping. A noisy queue-length signal causes replica oscillation; use behavior.scaleDown.stabilizationWindowSeconds (part of the autoscaling/v2 HPA behavior fields) or pre-aggregate with a Prometheus recording rule rather than reacting to raw noise.
• Latency in the chain. Scrape interval, adapter caching, and the HPA's own sync period all stack up between a real queue-depth change and a scaling action; a 15s scrape interval plus a slow adapter cache can easily add tens of seconds of lag, which matters for a bursty queue.
• Cost. Recomputing an expensive PromQL query on every HPA sync (default every 15 seconds) across many HPAs can meaningfully load a Prometheus instance; pre-aggregate with recording rules for anything non-trivial.

Validate end to end in staging with synthetic queue load before trusting this in production, watching the full chain (producer to queue to exporter to Prometheus to adapter to HPA) rather than any single hop in isolation.

Trade-off note

Hand-rolling prometheus-adapter rules gives full control over the PromQL mapping but is fiddly YAML to maintain; KEDA trades some of that flexibility for purpose-built scalers for dozens of common event sources (queues, streams, schedules) and is usually less operational overhead for exactly this "scale on queue depth" scenario, at the cost of being one more component to run alongside (or instead of) prometheus-adapter.