Proposed schema (fields = indexed metadata/tags for correlation; metrics = numeric/time-series values; message = human text):Indexed tags/fields:- run_id (uuid): unique training run- experiment_id (string): logical experiment grouping- job_id (string): orchestration task id (K8s pod/step)- commit_sha (string): code commit used- dataset_name (string)- dataset_version (semver or hash)- model_name / model_version- env (prod/staging/dev)- host / gpu_id- trace_id (for distributed traces)Metric fields (numeric/time-series):- epoch, step, lr- loss, val_loss, accuracy, val_accuracy- throughput_samples_per_sec- gpu_util_pct, gpu_mem_mb, cpu_pct- sample_count, batch_sizeError/debug fields:- level (INFO/WARN/ERROR)- error_code- stack_trace (if error)- batch_hash (to reproduce bad input)- hyperparams (json)Sample JSON log entry:json
{
"timestamp": "2025-11-22T14:03:17.123Z",
"level": "ERROR",
"run_id": "d7f3a9b2-5c1e-4a9b-8f2e-0b7f1c2a9e6d",
"experiment_id": "exp-recommender-2025-11",
"job_id": "train-rl-xyz-57",
"commit_sha": "a1b2c3d4",
"dataset_name": "user_events",
"dataset_version": "2025.11.20+sha256:9f8e7",
"model_name": "rec-net-v4",
"model_version": "v4.2.0",
"env": "staging",
"host": "gpu-node-12",
"gpu_id": "0",
"trace_id": "trace-8b3f",
"epoch": 3,
"step": 12450,
"loss": 2.431,
"val_loss": 2.98,
"throughput_samples_per_sec": 512,
"gpu_util_pct": 93.2,
"batch_size": 256,
"sample_count": 64000,
"error_code": "DATA_PARSE_ERR",
"message": "Failed to parse example: TensorProto shape mismatch",
"stack_trace": "ValueError: Tensor shape ... at data_loader.py:78",
"batch_hash": "sha1:3c4f...",
"hyperparams": {"optimizer":"adam","lr":0.0003,"weight_decay":1e-5}
}
Field explanations (brief):- timestamp: event time (UTC) for ordering and correlation.- level: log severity for filtering.- run_id: primary correlation key — join logs, metrics, traces.- experiment_id/job_id: group runs and map to orchestrator.- commit_sha: tie behavior to code state.- dataset_name/dataset_version: reproduce errors due to data drift.- model_name/model_version: link artifacts to runs.- trace_id: correlate distributed operations and traces.- epoch/step/loss/val_loss/accuracy: core training metrics for charting and alert rules.- throughput/gpu_util/gpu_mem: resource telemetry to detect bottlenecks.- batch_hash/sample_count: reproducibility — identify problematic batches.- error_code/stack_trace/message: machine-parseable error triage.- hyperparams: record config for debugging and hyper-search analysis.Why this helps:- Single run_id + trace_id links logs, time-series metrics, and traces.- Dataset version + batch_hash localizes data-caused failures.- Structured numeric fields enable dashboards/alerts; structured error fields enable automated grouping and reruns.