Container and Kubernetes Security Questions

Requirements:

• Scan 1M images/day (~11.6 images/sec sustained), low latency for CI, support incremental layer scans, cache results, distinguish OS-package CVEs vs language-dep vulnerabilities, integrate with registries and CI, provide triage workflows, respect registry rate limits, high availability and observability.

High-level architecture:

• Ingest/API layer: REST + webhooks to accept image events from CI/registry (push, tag, pull-request). Auth via token/OAuth.
• Orchestrator/queue: Kafka or Pub/Sub for durable work queue; partition by image repo to preserve ordering.
• Scanner workers: Stateless autoscaled containers (K8s) that:
  • Fetch image manifest from registry
  • Determine layers and content-addressed digests
  • Check cache store for per-layer scan results
  • For missing layers, pull layer blobs and run appropriate scanners:
    • OS/package scanner (e.g., dpkg/rpm parsing + vulnerability DB)
    • Language-dep scanner (language-specific SBOM/lockfile parsing or dependency graph + vulnerability feeds)
  • Merge results, dedupe CVEs, tag with provenance (layer digest, scanner version)
  • Store results and notify downstream
• Cache & metadata store:
  • Layer-scan cache: key = layer-digest + scanner-version -> scan-result (Redis for hot, backed by durable store like DynamoDB/Cassandra)
  • Image-assembly index: image-manifest -> list of layer digests and combined result
  • SBOM store: per-image SBOM (Rekor/CycloneDX)
• Vulnerability DB & enrichment: nightly sync from NVD, vendor feeds, language-specific feeds. Provide normalized schema and severity scoring.
• API + UI + triage:
  • Query API serving results with pagination, filters (OS vs language), severity, fixability
  • Triage UI: allow assigning, marking false positives, silencing, patch links, CVE grouping, audit log
  • Export to ticketing (Jira) and Slack alerts
• CI integration:
  • Pre-merge fast-path: lightweight SBOM-based scan using cached layer results; fail-fast policies with configurable thresholds
  • Post-push full scan: async thorough scan, results stored and surfaced
• Rate-limit & registry considerations:
  • Respect registry concurrency & per-IP limits: use tokenized pull proxies and per-repo rate limiter in orchestrator
  • Use conditional GET / If-None-Match for manifests and range requests for blobs to reduce bandwidth
  • Peer local caches (pull-through cache) close to workers (Harbor/Registry cache) and use blob TTLs
  • Throttle retries with exponential backoff and prioritized backoff for noisy repos
• Storage & retention:
  • Hot cache: Redis cluster for recent layer results (TTL e.g., 30 days)
  • Durable store: S3-compatible object store for full scan payloads, DB for metadata (Postgres for indexes, DynamoDB for scale)
  • Long-term archive of scan artifacts and SBOMs in cold storage
• Scalability & cost controls:
  • Autoscale workers based on queue depth, CPU, network
  • Batch layer fetches and shared downloads via shared blob cache
  • Deduplication via content-addressing reduces compute drastically
• Observability & SLOs:
  • Metrics: throughput (images/sec), latency (CI fast-path < 60s), cache hit rate, queue lag, error rates
  • Alerts: queue backlog, scanner version drift, registry 429 spikes
  • Tracing for per-image processing
• Triage workflow:
  • Automatic grouping: same CVE across images -> group, show common layer(s)
  • Prioritization: score = severity * exposure * pull-frequency (exposure from CI/production labels)
  • Human triage: assign, mark FP, escalate to security team; actions can create PR templates with suggested fixes
  • Feedback loop: triage decisions feed back to suppression rules and ML for future de-duplication
• Incremental scanning & correctness:
  • Because layers are content-addressed, re-scan only new/changed layers or when vulnerability DB or scanner version changes
  • Maintain scanner-versioned cache keys so re-scan happens when rules update
• Trade-offs:
  • Strong dedup via layer caching reduces cost but requires reliable content-addressing and cache invalidation on scanner updates
  • Fast-path SBOM checks are quicker but may miss runtime-generated deps; full layer introspection is slower
  • Aggressive TTLs lower cost but may miss newly disclosed CVEs unless vulnerability DB triggers re-scans
• Security & reliability:
  • Run scanners in isolated runtime with resource limits; sign SBOM results; immutable scanner images; vulnerability DB signed feeds
  • Disaster recovery: metadata DB replicas, cross-region storage for objects
    This design handles 1M/day by maximizing deduplication (layer-level caching), asynchronous orchestration, registry-friendly fetching, CI fast-paths, clear separation of OS vs language vulnerability analysis, and a prioritized triage workflow tied into CI and ticketing.

Approach: Use kubectl (or the Kubernetes Python client) to list Secrets across all namespaces, parse each secret's metadata.creationTimestamp, compare it to now minus 180 days, and print namespace/name and creationTimestamp. This is read-only and safe.

#!/usr/bin/env python3
"""
List Kubernetes Secrets not modified (created) in the last 180 days.
Outputs: namespace/name <tab> creationTimestamp
Requires: kubectl in PATH and current kubeconfig context, or run in-cluster with RBAC read access.
"""
import subprocess, json, datetime, sys

DAYS = 180
threshold = datetime.datetime.utcnow() - datetime.timedelta(days=DAYS)

# get all secrets as JSON
proc = subprocess.run(["kubectl", "get", "secrets", "--all-namespaces", "-o", "json"],
                      capture_output=True, text=True)
if proc.returncode != 0:
    print("kubectl error:", proc.stderr, file=sys.stderr)
    sys.exit(1)

data = json.loads(proc.stdout)
for item in data.get("items", []):
    ns = item["metadata"].get("namespace", "<no-namespace>")
    name = item["metadata"].get("name", "<no-name>")
    ts = item["metadata"].get("creationTimestamp")
    if not ts:
        continue
    # parse RFC3339 timestamp
    created = datetime.datetime.fromisoformat(ts.replace("Z", "+00:00")).replace(tzinfo=None)
    if created < threshold:
        print(f"{ns}/{name}\t{ts}")

Key points:

• Uses read-only kubectl; safe for production.
• Compares creationTimestamp (not last-modified annotation—Kubernetes doesn't store last-modified by default).
• If you need last update time, consider auditing or tracking an annotation on mutation.

Time/Space:

• Time: O(N) where N = number of secrets (single API call).
• Space: O(1) extra memory besides JSON from kubectl.

Edge cases:

• Large clusters: JSON might be large; consider streaming with kubectl -o jsonpath or pagination.
• Clusters without accurate timestamps or with timezone anomalies.
• Secrets created long ago but rotated (no native last-modified). Recommend enforcing an annotation (e.g., rotatedAt) via automation.

Safe testing in production-like environment:

1. Run in a staging cluster with similar number of namespaces and secrets.
2. Point kubeconfig to production read-only user (RBAC role with get/list) — never run with admin.
3. Dry-run in prod: run script but redirect output to a file and review; do not delete or modify resources.
4. Add unit tests mocking kubectl JSON responses; run on CI.
5. If rolling into automation (cleanup/rotation), add approvals, logging, and canary on a single namespace before wider rollout.

Framework: treat base-image vulnerabilities as product risk — prioritize by severity/exploitability, blast radius (how many services use the image), and mitigation cost. Then choose a strategy per-priority class and automate.

Options and trade-offs:

• Patching existing images: quick (apply distro security updates via package manager), preserves build semantics and image size, good for urgent CVEs. Downside: depends on upstream package availability, may leave legacy cruft, requires rebuilding and redeploying anyway to push fixes. Manual patching scales poorly across many services without automation.
• Rebuilding on updated base layers: recommended for most cases. Rebuilds ensure you inherit official base fixes, maintain reproducible images, and pick up non-package fixes (glibc, kernel patches in distroless parents). With automated CI that rebuilds images when base layers change and runs tests, you get continuous remediation with low developer effort. Requires robust CI, image signing, and can surface runtime incompatibilities—so include smoke tests and staged rollouts.
• Switching to minimal distros (distroless): reduces attack surface and CVE count long-term, and often produces smaller images. Good for greenfield services or where you control runtimes. Costs: migration effort, loss of shell/tools for debugging, possible incompatibilities with existing tooling, and some CVEs still possible in language runtimes. Also needs improved observability tooling (debug images, sidecar shells).

Preferred approach for an SRE:

• Short term: patch critical/high CVEs immediately (hotfix images), use emergency rollouts and canaries.
• Medium/long term: automate regular rebuilds on updated base layers in CI/CD, enforce image policies (SBOMs, vuln scans, signed images), and run integration tests + gradual deploys.
• Strategic: adopt minimal/base-hardened images (distroless or locked-down minimal distros) for new services and refactor high-risk legacy services incrementally.

Operational practices:

• Automate alerts when base image CVEs appear; trigger automated rebuild pipelines.
• Maintain SBOMs, vulnerability dashboards, and SLOs for remediation time.
• Provide debug images or ephemeral sidecar shells so distroless doesn’t block incident response.

Conclusion: Rebuilding on updated base layers + automation is the best default; patching is for urgent fixes; distroless is ideal where migration cost and operational tooling are acceptable.

High-level summary: each approach trades detection signal (precision/recall), maintenance/operational cost, and the kinds of attacks it finds. As an SRE you’d usually combine them to cover complementary gaps.

1. Signature-based detection

• Precision/Recall: Very high precision for known IOCs (low false positives), but recall is low for novel variants or polymorphic payloads.
• Operational costs: Low runtime cost; frequent updates needed to signature feeds; management overhead scales with number of signatures and CI/CD to distribute updates.
• Best for: Known malware binaries, exploit payload patterns, specific CVE-based indicators. Poor for lateral movement, 0-day or subtle behavioral misuse.

2. Rule-based behavioral detection (e.g., Falco)

• Precision/Recall: Balanced — well-crafted rules yield good precision and decent recall for defined suspicious behaviors. Recall depends on rule coverage; aggressive rules can raise false positives.
• Operational costs: Moderate — requires rules authoring, tuning per workload, versioning rules, and maintaining context (K8s labels, process paths). Integrates well with alerts, audit logs, and incident playbooks.
• Best for: Runtime suspicious system calls (unexpected execs, container escape attempts, abnormal network binds), drift from least-privilege baselines, compliance checks.

3. ML-based anomaly detection

• Precision/Recall: Can have high recall for novel/unknown anomalies but often lower precision (more false positives) until models are well-trained and context-aware. Concept drift and noisy baselines reduce reliability.
• Operational costs: High — data collection, feature engineering, model training, validation, retraining, explainability tooling, and pipelines. Requires labeled incidents for supervised models or strong ops for unsupervised tuning.
• Best for: Novel attacks, subtle deviations (slow data exfiltration, privilege creep), multi-signal correlations across metrics/logs. Less reliable for pinpointing specific attack vectors without explainability.

Trade-offs and SRE guidance:

• Start with rule-based (Falco) for immediate runtime coverage; use signatures as a low-noise supplement for known threats. Layer ML for high-signal environments where you can invest in model ops and handle false positives.
• Invest in feedback loops: incident labels from on-call teams should feed rule tuning and ML retraining to improve precision without sacrificing recall.
• Operationalize: alert routing, severity mapping, automated containment (e.g., network isolation), and runbooks to make any approach actionable and minimize toil.

Container image signing is the process of cryptographically attaching a verifiable signature to a container image (or its digest) so consumers can prove the image’s origin and integrity. In a secure supply chain this prevents running tampered or unauthenticated images, enabling policies that only allow images from trusted builders.

Why it matters:

• Confirms provenance (who built it) and integrity (unchanged since signing).
• Supports enforcement in CI/CD and runtime (admission controllers, image policies).
• Reduces risk of supply-chain attacks (malicious image replacement).

Two solutions:

1. Sigstore / cosign

• Sigstore is a modern, cloud-native stack providing transparent signing and short-lived keys, backed by Rekor (a public transparency log).
• cosign is the client: signs image digests, can store signatures in OCI registries or Rekor, supports keyless signing via OIDC (short-lived certs).
• Verification: at deploy time, automated tools (e.g., Kubernetes admission controllers, CI checks, cosign verify) fetch the signature and public key/cert (or look up Rekor entry) and validate the signature against the image digest and expected signer identity.

2. Notary (Docker Content Trust / Notary v1 and TUF-based)

• Notary implements The Update Framework (TUF) to provide delegation, rotation, and expiry of keys; signatures live in a Notary server.
• Verification: registries or clients query the Notary server to fetch trusted metadata and verify the signature chain before pulling/deploying.

How verification is enforced at deployment:

• CI gates that verify signatures before promoting images.
• Kubernetes admission controllers (e.g., OPA/Gatekeeper, Kyverno, or Sigstore’s k8s-webhook) that reject pods referencing unsigned or untrusted images.
• Runtime policies in registries or deployment pipelines ensure only signed digests are allowed.

Risks if signing keys are compromised:

• Attacker can sign malicious images as if they were trusted — undermines trust; rollback or revocation required.
• With long-lived keys (classic Notary setups), compromise window is large; attacker can persistently poison environments.
• Mitigations: use short-lived credentials (OIDC/keyless), hardware-backed keys (HSM/Cloud KMS), key rotation, revocation mechanisms and transparency logs (Rekor) to detect misuse quickly, and enforce least-privilege for signing processes.

Summary: Signing plus automated verification at build, registry, and runtime boundaries materially raises the bar for attackers; choosing solutions with key rotation, transparency, and automation (e.g., Sigstore/cosign) reduces impact if keys are lost.