Safety and Responsible Development Questions
Addresses risk, safety, and governance challenges specific to generative systems, including hallucinations and factual errors, bias and unfair outputs, toxicity, adversarial vulnerabilities such as prompt injection, and privacy and data leakage concerns. Covers mitigation and safety practices such as retrieval augmentation and grounding, fact checking, ensemble methods, calibration and uncertainty estimation, prompt and instruction design, content filtering, monitoring and anomaly detection, red teaming and adversarial testing, documentation and model cards, and operational controls such as access restrictions and human review. For senior candidates, includes designing systems and development processes proactively for safety, accountability, and responsible deployment.
HardSystem Design
34 practiced
Hard: For a multimodal generative system (text + images) used in medical triage, propose a safety-layered architecture to prevent hallucinated diagnoses. Include dataset curation, clinical validation loops, human oversight, and model explainability elements necessary for regulatory approval.
Sample Answer
Requirements & constraints:- Functional: accept patient text (symptoms, history) + images (photos, scans) → triage recommendation (urgency level, next steps).- Non-functional: near-zero hallucination of diagnoses, traceability for regulators, clinician-in-the-loop, latency < few seconds, high availability, audit logs.High-level architecture (layered safety):1) Data curation & provenance layer- Source-restricted datasets: EHRs, annotated clinical image datasets (DICOM), curated case repositories with patient consent and IRB approval.- Metadata capture: source, device, signer, timestamp, demographic distribution, labeler IDs and confidence.- Data governance pipeline: automated de-duplication, bias audits, differential privacy controls, and simulated adversarial examples (out-of-distribution, corrupted images).- Human labeling protocol: multi-rater consensus with adjudication by board-certified clinicians; store disagreement logs.2) Model development & modularization- Modular multimodal stack: separate encoders (clinical text encoder, vision encoder) + calibrated diagnosis/triage head.- Use retrieval-augmented generation (RAG) for evidence grounding: retrieve similar documented cases, guidelines (e.g., triage protocols), cite source snippets.- Conservative output policy: model produces triage level + confidence interval + top-k evidence items; prohibits unconditional diagnostic claims (labels limited to "suggests", "cannot rule out", or "refer to clinician").3) Clinical validation & continuous loop- Offline: retrospective validation on hold-out hospital datasets, stratified by site/device; metrics: false negative rate for high-acuity, precision for urgent flags, calibration (ECE), AUC-ROC, and hallucination rate (manual audit).- Prospective shadow deployments: run model in parallel; flagged discrepancies trigger case review.- Continuous improvement: physician review queue for uncertain/novel cases; adjudicated cases feed back to training set with provenance tags.4) Human oversight & workflow integration- Human-in-the-loop gating for medium/high-risk outputs: require clinician sign-off before patient-facing recommendations.- Explainable UI: show evidence snippets, highlighted image regions (saliency / segmentation masks), attention maps, counterfactuals ("if symptom X absent, urgency decreases by Y%").- Escalation policy: automatic urgent routing if model confidence low or contradictory evidence present.5) Explainability, calibration & regulatory artifacts- Deterministic logging: inputs, model versions, random seeds, retrieved documents, confidence scores, and clinician actions — immutable audit trails.- Explainability methods: concept-based (TCAV), attribution maps for images (Grad-CAM + uncertainty-aware smoothing), natural-language rationales grounded to cited sources.- Calibration & uncertainty: temperature scaling, conformal prediction sets to provide prediction intervals; flag predictions outside calibration envelope.6) Safety monitoring & incident response- Real-time monitoring: drift detectors, out-of-distribution detectors, hallucination detectors (compare generated claims to retrieved evidence).- Post-deployment audits: periodic sample reviews, KPI dashboards (missed critical cases), M&M style morbidity reviews.- Rollback & kill-switch: automated rollback on threshold breaches; manual override for regulators/clinical leads.Trade-offs and justification:- Conservative policy reduces automation but minimizes clinical risk — acceptable in regulated healthcare.- RAG improves grounding but requires trusted, indexed knowledge sources and strong provenance to avoid propagating errors.Regulatory readiness:- Provide validation report (datasets, metrics, clinician review outcomes), risk assessment (FMEA), usability testing, and traceable change management. Apply Good Machine Learning Practices (FDA/EMA guidelines), Data Protection Impact Assessment (GDPR/HIPAA), and ensure biobanking consent records.This layered architecture combines rigorous dataset governance, evidence-grounded generation, clinician oversight, explainability, and traceable validation to minimize hallucinated diagnoses and meet regulatory expectations.
EasyTechnical
34 practiced
Explain confidence calibration for probabilistic model outputs. Why is calibration important for safety in generative systems, and name one simple technique (e.g., temperature scaling) used to calibrate model probabilities.
Sample Answer
Confidence calibration means that a model's predicted probabilities match true empirical frequencies: when the model outputs 0.7 confidence, about 70% of those predictions should be correct. Well-calibrated probabilities quantify aleatoric and epistemic uncertainty and let downstream components make safe decisions (e.g., defer to a human, trigger verification, or apply conservative actions) instead of blindly trusting overconfident outputs. In generative systems, miscalibration can cause plausible-sounding but incorrect or risky content to be presented with unjustified certainty, increasing safety hazards.A simple post-hoc technique is temperature scaling: divide the model logits z by a scalar temperature T > 0 before the softmax: softmax(z / T). T is optimized on a validation set (minimizing negative log-likelihood) so that predicted probabilities better match observed outcomes. T>1 softens/confidence-reduces overconfident models; T<1 sharpens underconfident ones. Temperature scaling is easy, light-weight, and preserves model accuracy, but it only adjusts confidence globally—one should consider class- or input-dependent calibration or more advanced methods (isotonic regression, Platt scaling, Bayesian approaches) for complex safety needs.
HardSystem Design
30 practiced
Hard: Architect a safety-focused generative AI platform for an enterprise knowledge assistant that must: handle 10k employees, integrate internal docs (searchable), prevent data leakage to external users, and provide audit trails for compliance. Describe high-level components, data flow, controls (RBAC, encryption), and red-team testing strategy.
Sample Answer
Requirements clarification:- 10k employees, internal doc search + generative answers, zero leakage to external users, full auditability/compliance, low-latency RAG where needed.High-level architecture (components):- Ingestion & ETL: connectors (SharePoint, Confluence, S3, databases) → normalizer, PII tagger, DLP scanner → metadata store.- Indexing & Vector DB: semantic embeddings, chunking, vector store (self-hosted like Milvus/FAISS or managed VPC), encrypted at rest.- Retrieval layer / RAG orchestrator: supervised retriever + policy-aware reranker that returns vetted context windows.- Model hosting: private LLMs (fine-tuned) in VPC/GPU cluster with an inference gateway; no calls to public APIs.- Policy Engine / Safety Filter: deterministic filters, prompt sanitizer, intent classifier, and DLP policy enforcer applied to input + model output.- Access & Gateway: API gateway with mutual TLS, tenant-aware routing, rate limits, and provenance headers.- Auth & RBAC: enterprise IAM integration (OIDC/SAML) + attribute-based access control (ABAC) for document-level permissions.- Audit & Compliance: immutable append-only audit logs (WORM), query/response hashing, UIs for auditors, integration with SIEM.- Secrets & Key Management: KMS/HSM for encryption keys; envelope encryption for models and vector DB.- Monitoring & Observability: telemetry, anomaly detection, usage metrics, drift detection.Data flow (condensed):1) Connector pulls documents → normalization → PII tagging & DLP → encrypted storage + metadata.2) Chunk & embed → store vectors with doc-level ACL metadata.3) User request → authenticate (SAML/OIDC) → ABAC checks → retriever fetches contextual chunks respecting ACLs.4) Policy engine sanitizes prompt and blocks disallowed intents; RAG orchestrator builds prompt with provenance.5) Inference in private LLM → post-output filters (DLP, hallucination detector, redaction) → human-in-loop escalation if high-risk.6) Response returned; full trace (user, query, retrieved doc ids, model version, filters applied) logged to immutable audit store.Controls & security:- Network: VPC-only deployment, no egress to public model APIs; allow-list outbound; private endpoints.- RBAC/ABAC: document-level ACLs stored with vectors; policies enforced at retriever and gateway. Admin roles limited; separation of duties.- Encryption: TLS in transit; AES-256 at rest; envelope keys in KMS/HSM; per-tenant keying optional.- Data minimization: store only embeddings + pointers; redact or tokenize sensitive fields before embedding; maintain retention policies.- DLP & Secrets: regex + ML-based PII detection; block or redact secrets; scanner on output and logs.- Model guardrails: fine-tune on company-safe data; apply output classifiers for sensitive content, hallucination scoring, and refusal templates.- Audit & Compliance: immutable logs, cryptographic hashes for tamper-evidence, retention and export for legal requests, periodic access reviews, automated attestation reports.- Operational: least privilege, key rotation, regular pentests, patching, SSO MFA enforced.Red-team testing strategy:- Threat modeling: identify adversaries (malicious insider, compromised account), attack surfaces (prompt injection, data exfil via responses, embedding inversion).- Test suites: - Prompt-injection and jailbreak corpuses (automated fuzzing + human adversaries). - Exfiltration scenarios: craft queries to reconstruct docs, chain-of-thought leakage, iterative probing. - Retrieval poisoning: inject malicious docs in connectors to test DLP and provenance checks. - Role abuse: escalate privileges, API abuse, rate-limit bypass tests. - Model-behavior stress tests: adversarial paraphrases, multi-turn extraction, instruction chaining.- Metrics & gates: measure leakage rate, false accept/refuse, time-to-detect, and impact classification. Define acceptable thresholds; block releases if exceeded.- Continuous evaluation: integrate tests into CI/CD for model and policy updates; maintain a "red-team findings" backlog and remediation SLAs. Periodic external audits and compliance attestation.Trade-offs & rationale:- Self-hosted/private models increase cost but are required to guarantee no external leakage.- Strong filters + human-in-loop reduce productivity slightly but are necessary for safety/compliance.- Storing pointers vs full text reduces exposure but requires reliable connectors for forensic retrieval.This design enforces defense-in-depth: prevent leakage at ingestion, retrieval, inference, and output stages; enforce strict RBAC and cryptographic controls; and maintain full, tamper-evident audit trails validated by continuous red-team testing.
MediumSystem Design
33 practiced
Medium: You need to implement rate-limited, audited access to an internal model endpoint for external contractors. Specify the OAuth flow, token scopes, per-call logging, and how you would detect suspicious use patterns that might indicate data exfiltration.
Sample Answer
Requirements & constraints:- Contractors need time-limited, least-privilege programmatic access to an internal model endpoint.- Enforce per-contractor rate limits, full per-call auditing, and automated detection of suspicious/exfiltration patterns.- Low latency for inference; logs must be tamper-evident and retained for forensics.High-level architecture:- Authorization Server (OAuth2 + OIDC) + Policy Engine (fine-grained scopes).- API Gateway (rate-limiting, authentication, logging proxy).- Model Serving Cluster (internal), Audit Log Store (immutable append-only), SIEM/Analytics engine.OAuth flow & token model:1. Contractor registers as a client (client_id, redirect_uri); registration requires org approval.2. Use Authorization Code Flow with PKCE for interactive human contractors; use Client Credentials for machine-to-machine where allowed, but require short TTL (e.g., 5–15 min) and rotation.3. Issue JWT access tokens with: - scoped claims: scope = ["model.infer.read-only", "model.infer.metadata"], resource_id, client_id, org_id, exp. - x-attrs: rate_limit_bucket_id, audit_level.4. Refresh tokens only for interactive flows; refresh tokens are one-time use and revocable.Token scopes and policy:- Minimal scopes by role: infer.read (calls model), infer.meta (model metadata), infer.batch (higher quota, requires justification).- Policy engine enforces scope-to-endpoint mapping; scopes cannot be widened after minting.Per-call logging (audit):- API Gateway authenticates JWT, enforces rate limit, then forwards to Model Serving with an immutable request-id header.- For each request log: - timestamp, request-id, client_id, org_id, token_jti, scope, endpoint, input hash (e.g., sha256), input size, response size, latency, model version, worker node id, sampled request content (if allowed).- Logs written to append-only storage (WORM) or cloud immutable bucket + signed digest chain (Merkle-tree-style) so tampering is detectable.- Store full request/response payloads encrypted with envelope keys; decryption requires audit access.Rate-limiting:- Per-client & per-scope token bucket in API Gateway (Redis or managed service).- Soft limits with 429 responses and hard block when thresholds exceeded; provide backoff headers and telemetry.- Burst allowances controlled separately from sustained throughput.Suspicious-use / exfil detection:- Real-time analytics pipeline (stream logs → feature extraction → ML-based detectors + rules engine).- Rules examples: - High entropy outputs: repeated requests yielding large unique outputs or large output size spikes. - Pattern-based: sequences of requests that stitch outputs together (many overlapping prompts with sliding windows). - Data-probing: many prompts that look like structured extraction (SSNs, emails, configuration files) — detect via regex features. - Unusual cadence/geography: tokens used from new IPs, unusual regions, or impossible concurrent sessions. - Model-specific anomalies: abnormally low prompt diversity but high token output, repeated access to training-data-sensitive endpoints.- Detection stack: - Feature extraction (per-client): avg output_size, stddev, n_requests/hour, distinct n-grams ratio, entropy score, repetition of extraction patterns. - Anomaly detector: unsupervised (isolation forest / streaming sketching) + supervised classifiers trained on labeled exfil attempts. - Rule engine for high confidence signatures (regex, blacklisted file patterns).- On alert: automated actions — throttle or suspend token, require re-authentication, create high-priority incident in SIEM, snapshot logs and request payloads, notify security and contractor admin.Operational controls & trade-offs:- Privacy: redact or sample PII in logs unless forensics authorized; use encryption and RBAC for audit logs.- Latency vs auditing: synchronous minimal metadata logging on request path; bulk payload persistence async to avoid added latency.- False positives: tune detectors, provide human-in-the-loop review and allow safelists for approved contractors with stronger agreements.- Governance: contract clauses, periodic attestation, key rotation, token revocation lists, audit retention policy.This design enforces least privilege via OAuth scopes, protects capacity via rate-limits, provides tamper-evident per-call auditing, and detects exfiltration with layered ML + rules, enabling automated containment and human investigation.
HardSystem Design
35 practiced
Hard: Architect a simulation environment to evaluate long-term safety risks (e.g., feedback loops) for a generative recommendation assistant that iteratively adapts to user behavior. Describe simulation components, metrics to capture runaway amplification, and intervention strategies to prevent drift.
Sample Answer
Requirements & constraints:- Functional: simulate multi-step interaction between users and a generative recommendation assistant that adapts online (personalization, content selection, reward updates).- Non-functional: simulate long horizons (months/years), diverse user types, stochasticity, scalability for many runs, ability to inject interventions and measure metrics.High-level architecture:User population generator → Environment & content model → Agent (recommendation + generative response model) → Interaction simulator → Logging & analyzer → Intervention controllerCore components:1. User population model: parameterized personas (preferences, susceptibility, novelty-seeking, conformity, retention), stateful (mood, fatigue, belief state). Can be a mixture of parametric agents, behavioral-cloned policies, or reinforcement-learning based proxies.2. Content world: catalog of items/memes with attributes, quality signals, temporal dynamics (novel items, trends), and external exogenous events.3. Assistant model: recommendation policy + generative response module with adaptation loop (online fine-tune, reward-shaping, implicit preference inference).4. Interaction simulator: models click/view/engagement feedback, content creation by users (amplification), social graph diffusion.5. Metrics & analyzer: computes amplification metrics, drift indicators, and safety signals.6. Intervention controller: implements throttles, exploration boosts, counterfactual injections, and audits.Metrics to detect runaway amplification:- Concentration: Gini/HHI of content exposure over time (rising → narrowing).- Feedback gain: ratio of probability of content being recommended at t+Δ given positive feedback at t.- Diversity decay: item-attribute entropy across users over time.- Belief polarization: divergence (Jensen-Shannon) between subpopulation preference distributions.- Amplification multiplier: R_eff-like metric: expected downstream impressions produced per initial impression.- Novelty starvation: rate of new item adoption.- Reward hacking score: surrogate objective vs. user utility mismatch.Simulation experiments:- Sensitivity sweeps over user susceptibility, learning rates, reward weightings.- Counterfactual runs with/without user-generated content and social sharing.- Long-horizon Monte Carlo to estimate tail risks.Intervention strategies:- Conservative model updates: restrict online fine-tuning bandwidth; use replay buffers with balanced content.- Exploration incentives: optimism/Thompson sampling, forced randomization, or entropy bonuses to prevent premature convergence.- Diversity constraints: optimization with explicit diversity regularizers or constrained bandit formulations.- Rate limits on virality: cap per-item amplification or decay kernels.- Adversarial testing & red-teaming: synthetic adversaries that try to push harmful drift.- Human-in-the-loop safety gates: periodic audits, explainability hooks, rollback triggers based on thresholds.- Reward shaping: incorporate long-term user welfare proxies (retention quality, satisfaction surveys) and penalize amplifying signals.- Continuous monitoring: real-time computation of amplification multipliers and automated rollback if exceed thresholds.Trade-offs & validation:- More conservative interventions reduce personalization utility; tune via multi-objective optimization.- Validate simulator fidelity: calibrate to real telemetry, use offline A/B to match short-term behavior before trusting long-term projections.- Use ensemble simulators (different user models) to bound epistemic uncertainty.This environment enables stress-testing policies, quantifying tail amplification risks, and iterating interventions before deployment.
Unlock Full Question Bank
Get access to hundreds of Safety and Responsible Development interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.