Shell Scripting and Automation Questions

Situation: Multiple teams and CI pipelines rely on static secrets embedded in shell scripts — high risk for leakage, hard to rotate, and no centralized audit.

Task: As staff data engineer, I needed to design a secure, auditable secrets architecture with short‑lived credentials, policy enforcement, least privilege, and a low-friction migration path so pipelines keep running.

Action — Secure architecture (high level):

• Central secrets service: HashiCorp Vault (or cloud KMS/Secret Manager with IAM) as source of truth; enable dynamic secrets (DB/Cloud IAM) for short‑lived credentials.
• Authentication: Use platform identity (instance/profile, OIDC for CI, or mTLS) — no long‑lived static tokens.
• Access control: RBAC + policies per team/environment; use Vault policies mapping roles to secrets and TTLs.
• Injection patterns:
  • CI integration using native plugins (GitLab CI Vault, GitHub OIDC -> Vault) to fetch secrets at runtime.
  • For shell scripts, provide a lightweight local wrapper (vault-fetch) that caches secrets in memory for their TTL, not on disk.
  • For long‑running jobs, use sidecar agent (vault-agent) with selective templating and renewing tokens.
• Auditing: Enable Vault audit devices (file/remote) into centralized log platform (ELK/Splunk) with immutable storage and alerting for anomalous access.
• Minimal privileges: Issue least-privilege dynamic creds (short TTL, constrained DB roles, scoped cloud IAM).
• Backwards compatibility: Offer a compatibility shim that supports current env var usage but sources values from Vault; fall back only with admin override.

Action — Migration steps (practical phased plan):

1. Discovery & Inventory (1–2 weeks): scan repos/CI for secret patterns, classify by sensitivity and owner; create a migration backlog.
2. Pilot (2–4 weeks): pick a low-risk pipeline; deploy Vault, enable OIDC for CI, create policies, implement vault-fetch wrapper and CI plugin; validate rotation and audit trails.
3. Build developer tooling (1–2 weeks): publish vault-fetch CLI, shell examples, CI templates, terraform modules for policies, and runbooks.
4. Incremental rollout (4–8 weeks): migrate teams in waves by priority. For each team:
   • Create Vault role/policy and dynamic creds
   • Replace secret reads with vault-fetch or CI secret injector
   • Run in parallel: allow old static secret but set expiry and monitor usage
   • Rotate old secret immediately after successful run
5. Enforce and harden (2 weeks): once majority migrated, enable denylist for legacy static secrets in CI, enforce OIDC auth, and set policy TTLs.
6. Decommission (ongoing): revoke unmanaged credentials, remove old secrets, and certify compliance.

Result / Metrics & Governance:

• Success measured by % pipelines migrated, mean time to rotate credentials (target <1 hour), audit coverage, and reduction in static secrets (target 100% elimination within 3 months).
• Add SLA for secret rotation, periodic audits, training sessions, and a rollback plan per wave (re-enable temp token for 1 hour).

This approach balances security (short‑lived creds, audit, least privilege) with developer ergonomics through wrappers and phased migration to minimize disruption.

Approach: build a small CLI orchestration shell script that submits jobs, records application IDs, polls status, fetches logs on demand/failure, enforces concurrency limits (semaphore), retries with backoff, and consults cluster capacity (YARN queue metrics or Kubernetes node/pod counts) before submitting to avoid overload.

Example script skeleton (bash):

#!/bin/bash
CONCURRENCY=5
RETRIES=2
SLEEP=10
QUEUE="default"

sem_acquire(){ while [ "$(jobs | wc -l)" -ge $CONCURRENCY ]; do sleep 2; done; }

submit_and_watch(){
  local cmd="$1"; local attempt=0
  while [ $attempt -le $RETRIES ]; do
    attempt=$((attempt+1))
    # submit
    app=$(eval "$cmd" | awk '/application_/ {print $1; exit}')
    echo "$(date) SUBMIT $app attempt=$attempt" >> orchestrator.log
    # poll status
    while true; do
      # YARN check
      status=$(yarn application -status $app 2>/dev/null | awk -F: '/State/{gsub(/ /,"",$2);print $2}')
      # Kubernetes alternative: kubectl get sparkapplication/<name> -o jsonpath=...
      if [[ "$status" == "FINISHED" || "$status" == "SUCCEEDED" ]]; then
        yarn logs -applicationId $app > logs/$app.out 2>&1
        return 0
      elif [[ "$status" == "FAILED" || "$status" == "KILLED" ]]; then
        yarn logs -applicationId $app > logs/$app.err 2>&1
        break
      fi
      sleep $SLEEP
    done
    sleep $((SLEEP * attempt)) # exponential backoff
  done
  return 1
}

# Example usage: read commands file, enforce concurrency and cluster-aware checks
while read -r line; do
  # basic cluster check: avoid submitting if running apps > threshold
  running=$(yarn application -list | grep RUNNING | wc -l)
  if [ $running -ge 50 ]; then sleep 30; fi

  sem_acquire
  ( submit_and_watch "$line" ) &

done < jobs.txt
wait

Key points and reasoning:

• Capture application IDs from spark-submit stdout (YARN prints application_...; on K8s capture driver pod name or SparkApplication CR status).
• Poll driver state rather than relying on process exit; use yarn application -status or kubectl API for authoritative state.
• Fetch logs with yarn logs -applicationId <id> (YARN) or kubectl logs <driver-pod> --all-containers (K8s); store under logs/<app>.
• Concurrency via semaphore: limit background jobs to CONCURRENCY to control cluster pressure.
• Cluster-aware checks: query YARN queue metrics (capacity, running apps) or K8s node/pod resource usage (kubectl top nodes/pods) to pause submissions when load high.
• Retries with backoff and log capture help transient failures.
• Support kill: expose a function that calls yarn application -kill <id> or kubectl delete pod/<driver> and mark orchestration state.
• Robustness: write state to persistent file (CSV with app, job, status, attempts) to recover after script crash.
• Security: run spark-submit with appropriate queue/namespace and resource requests to help scheduler placement and prevent overload.

Edge cases:

• Duplicate app ID parsing; ensure unique naming via --conf spark.app.name or --name.
• Long-running submissions: increase poll interval adaptively.
• Multi-tenant clusters: coordinate with scheduler (fair/Capacity) and respect queue limits.

Environment variables in Unix shells are key-value pairs available to processes. Scope depends on how they’re set:

• Shell-local: visible only in current shell process.
• Exported: inherited by child processes.
• System-wide: available to all users/processes when set in global files or service configs.

To export a variable (make it inherited):

MY_VAR="value"         # shell-local
export MY_VAR          # now child processes see MY_VAR
# or combine:
export MY_VAR="value"

Set a variable for a single command (temporary, no export needed):

AWS_PROFILE=prod aws s3 ls s3://bucket   # only for this command

Set for a user's session (persist across login shells):

• Add to ~/.bashrc or ~/.bash_profile (interactive/login shells) and export:

# ~/.bashrc
export JAVA_HOME="/usr/lib/jvm/java-11-openjdk"
export PATH="$JAVA_HOME/bin:$PATH"
export AWS_PROFILE="dev"

• After editing, apply with source ~/.bashrc or log out/in.

Set system-wide (all users):

• /etc/environment (simple KEY=VALUE format; no shell expansion)
• /etc/profile or a file in /etc/profile.d/ for bash-specific exports
  Example /etc/environment:

JAVA_HOME="/usr/lib/jvm/java-11-openjdk"
AWS_PROFILE="system-default"

Or create /etc/profile.d/java.sh:

export JAVA_HOME="/usr/lib/jvm/java-11-openjdk"
export PATH="$JAVA_HOME/bin:$PATH"

Persistence considerations:

• ~/.bashrc affects interactive shells; ~/.profile or ~/.bash_profile affects login shells.
• /etc/environment is read by PAM at login (affects GUI and ssh sessions).
• Services (systemd) often need variables in service unit files or /etc/default/<service>.
• For cron or systemd jobs, set env in the job’s config: Environment=VAR=value (systemd) or VAR=value in crontab.

Examples relevant to data engineering:

• JAVA_HOME used by Spark/Hadoop; set and export in ~/.bashrc or /etc/profile.d so worker processes find Java.
• AWS_PROFILE used by aws-cli; for CI or ETL jobs set in the job's environment or export in service unit so jobs use the correct credentials.

Edge notes:

• Avoid storing secrets in plain env files; use secret managers for production.
• Remember to export if child processes (like Spark driver/executor) must see the variable.

Approach: use capped exponential backoff with full jitter (sleep = random between 0 and min(cap, base * 2^attempt)). Implement retry loop around curl, log each attempt with timestamp, and exit non-zero if all attempts fail.

#!/usr/bin/env bash
set -euo pipefail

# POST with retries using capped exponential backoff + full jitter
# Usage: post_with_retry URL DATA_JSON MAX_ATTEMPTS BASE_SECS CAP_SECS
post_with_retry() {
  local url="$1"
  local data="$2"
  local max_attempts="${3:-5}"
  local base="${4:-1}"   # base backoff in seconds
  local cap="${5:-30}"   # max backoff cap in seconds

  for attempt in $(seq 1 "$max_attempts"); do
    ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
    echo "[$ts] Attempt $attempt/$max_attempts: POST $url"

    # perform POST; capture exit code and response (optional)
    resp=$(curl -sS -w "%{http_code}" -o /tmp/curl_body.$$ -X POST -H "Content-Type: application/json" --data "$data" "$url") || curl_code=$?
    http_code="${resp:-000}"
    curl_code="${curl_code:-0}"

    if [[ "$curl_code" -eq 0 && "$http_code" =~ ^2[0-9][0-9]$ ]]; then
      echo "[$ts] Success (HTTP $http_code)"
      cat /tmp/curl_body.$$
      rm -f /tmp/curl_body.$$
      return 0
    fi

    echo "[$ts] Failure (curl_code=$curl_code http_code=$http_code)."

    if [[ "$attempt" -ge "$max_attempts" ]]; then
      echo "[$ts] Exhausted $max_attempts attempts. Exiting with failure."
      cat /tmp/curl_body.$$ 2>/dev/null || true
      rm -f /tmp/curl_body.$$ 2>/dev/null || true
      return 1
    fi

    # compute capped exponential backoff: cap_value = min(cap, base * 2^(attempt-1))
    # full jitter: sleep = random_uniform(0, cap_value)
    pow=$(( base * (2 ** (attempt - 1)) ))
    if (( pow > cap )); then
      cap_value=$cap
    else
      cap_value=$pow
    fi

    # generate fractional random sleep in [0, cap_value)
    # use /dev/urandom for better randomness
    rand_fraction=$(awk -v seed="$(od -An -N4 -tu4 < /dev/urandom | tr -d ' ')" 'BEGIN{srand(seed); print rand() }')
    sleep_secs=$(awk -v r="$rand_fraction" -v c="$cap_value" 'BEGIN{printf "%.3f", r*c}')

    echo "[$ts] Backing off for ${sleep_secs}s (cap_value=${cap_value}s, full jitter)."
    sleep "$sleep_secs"
  done
}

# Example call:
# post_with_retry "https://example.com/ingest" '{"event":"x"}' 6 1 20

Key points:

• Full jitter (random between 0 and cap_value) prevents thundering herd by spreading retries across a distribution rather than aligning all clients to identical backoff points. If many clients fail simultaneously, randomized waits avoid synchronized retry storms against the endpoint.
• Cap prevents exponential growth into impractical waits.
• Logging per-attempt includes timestamps, attempt number, HTTP/curl codes for observability.
• Edge cases: ensure safe temporary file handling, consider exponential overflow for very large attempt counts (we cap it), and tune base/cap/max_attempts to your pipeline SLA.
• For production: replace /tmp handling with streams, add retryable-status detection (e.g., only retry 5xx and network errors), and add circuit-breaker/metrics integration.

Requirements & constraints:

• Preserve schedule, SLAs, idempotency, secrets, and observable logs. Support safe rollouts, retries, and easy rollback.

High-level approach:

1. Containerize each ETL script into minimal images.
2. Replace system cron with Kubernetes CronJob resources.
3. Use CI to build/push images and apply manifests (Helm/Kustomize).
4. Gradual migration with canary/testing and rollback plan.

Toolchain (shell + CI jobs):

• build-and-push.sh: builds image, tags by git SHA, pushes to registry, runs image scan.
• render-cronjob.sh: generate CronJob YAML from template (env, schedule, resources).
• deploy-cronjob.sh: apply manifests to target namespace; supports --dry-run and --canary flags.
• promote.sh / rollback.sh: tag/promote image or revert k8s manifests via gitOps.

Containerization steps:

• Wrap script in a small base (python/alpine). Entrypoint runs script and exits with proper code.
• Emit all logs to stdout/stderr; avoid file-based logs.
• Include lightweight health check endpoint if long-running.

Example CronJob YAML (key parts):

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etl-job-foo
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid          # avoid overlap
  startingDeadlineSeconds: 3600
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 3               # retry attempts
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: etl
            image: registry/example/etl:gitsha
            env:
            - name: ENV
              valueFrom:
                configMapKeyRef: { name: etl-config, key: ENV }
            - name: DB_PASS
              valueFrom:
                secretKeyRef: { name: etl-secrets, key: DB_PASS }
            resources: { requests: { cpu: "500m", memory: "1Gi" }, limits: { cpu: "1", memory: "2Gi" } }

Environment & secrets:

• Use ConfigMaps for non-sensitive env. Use Kubernetes Secrets or HashiCorp Vault (with CSI driver) for credentials.
• Avoid baking secrets into images. Mount secrets as env vars or files.

Logging & observability:

• Logs -> stdout/stderr. Deploy cluster-level log aggregation (Fluentd/Fluent Bit -> ELK/Cloud Logging).
• Emit structured JSON for parsing. Expose metrics via Prometheus client or export job-level metrics (duration, status).
• Alerts for failures and increased latency.

Concurrency & retries:

• Use concurrencyPolicy: Forbid to prevent overlaps for non-idempotent jobs; Allow for idempotent.
• backoffLimit controls retries. For complex transient retries, let the container implement exponential backoff and exit non-zero when fatal.
• Use activeDeadlineSeconds to cap runaway jobs.

Testing strategy:

• Unit test scripts locally; integration test inside container.
• Create a "staging" namespace; run CronJob with a frequent schedule (*/5 * * * *) and short TTLs to validate behavior.
• Canary: deploy job in production namespace with a different schedule or single-run Job pointing at production resources but limited scope (sample partition).
• Use --dry-run=client during apply and validate manifests with kubeval/kustomize/helm lint.
• Validate logs, metrics, and downstream data for correctness.

Rollback & safe cutover:

• Keep old cron entries disabled but retained until validation passes.
• Migrate per-job: deploy CronJob with image:canary, run through 1-2 cycles, verify outcomes, then update schedule and promote image tag to stable.
• Rollback by re-applying previous manifest (store manifests in git). CI should support automatic rollback on critical alert thresholds.
• For quick revert, pause CronJob (kubectl patch cronjob -p '{"spec":{"suspend":true}}') and re-enable legacy cron where necessary.

Operational best practices:

• Use resource limits and requests; set PodDisruptionBudgets if needed.
• Tag images with immutable digests; use imagePullPolicy: IfNotPresent for dev, Always for nightly.
• Maintain job history limits to avoid API object bloat.
• Document idempotency expectations; add locking via external locks (DB row, Redis) for cross-instance safety if needed.
• Automate migration tracking (spreadsheet/grafana dashboard) showing migrated vs legacy jobs.

This plan provides a reproducible, testable migration path with observability, safe rollbacks, and clear ownership for each ETL job.