Model Interpretability and Explainability Questions
Discuss techniques for understanding what models learn: attention visualization, feature importance methods (SHAP, LIME), saliency maps, concept-based explanations. Understand the difference between post-hoc explainability and inherent interpretability. Discuss trade-offs with model complexity.
HardTechnical
84 practiced
Design an adversarial evaluation for explanation robustness: define threat models where an attacker attempts to manipulate inputs so that explanations change substantially while predictions remain similar. Propose attack algorithms, quantitative metrics for explanation change, and defenses to detect or mitigate such attacks.
Sample Answer
Goal: quantify how easily an attacker can change an explanation e(x) while keeping model output y = f(x) (or its top-k) nearly unchanged.Threat models- White-box targeted explanation attack: attacker knows model f and explainer E (e.g., GradCAM, Integrated Gradients) and crafts δ to maximize explanation change while constraining prediction change: argmax_δ D_exp(E(x), E(x+δ)) s.t. D_pred(f(x), f(x+δ)) ≤ τ and ||δ||_p ≤ ε.- Black-box query-limited attacker: only can query f and E; builds surrogate models or estimates gradients via finite differences.- Physical/realistic perturbations: δ constrained to perceptual transforms (lighting, occlusion), or semantic edits (text paraphrase) rather than arbitrary pixel noise.Attack algorithms1. Gradient-based optimization (white-box): - Solve using projected gradient ascent on objective L = D_exp(E(x), E(x+δ)) - λ D_pred(f(x), f(x+δ)).2. Targeted saliency flip: pick target explanation pattern s* (e.g., shift heatmap to background) and optimize cross-entropy between E(x+δ) and s* under prediction constraint.3. Surrogate/transfer attack (black-box): train surrogate f̂ and Ê then transfer perturbation; or use NES/Finite-diff gradient estimates.4. Constrained semantic attacks: for text, use paraphrase + token-level substitutions maximizing explanation change while preserving label.Example pseudocode (white-box PGD-style):Quantitative metrics- Prediction-preservation: D_pred = |f(x) - f(x+δ)| (probability L_inf), top-k label agreement.- Explanation change: - Attribution L1/L2: ||E(x) - E(x+δ)||_1 - Rank-based: Spearman's ρ between feature importance rankings. - IoU for salient regions: IoU(binary(E>t), binary(E(x+δ)>t)) - Correlation-weighted localization: weighted center shift of heatmaps. - Structural metrics: SSIM for image explanations, Earth Mover’s Distance for spatial shifts.- Composite score: Attack success if D_pred ≤ τ and D_exp ≥ η. Report ROC-like curves over (ε, τ).Evaluation protocol- Vary ε and τ; report median and worst-case explanation change; include clean baselines.- Transferability: test attacks crafted against surrogate/explainer variants.- Query-efficiency for black-box: success vs. query budget.Defenses- Detection: - Consistency checks: compare explanations from multiple explainers or randomized model ensembles; flag large disagreement. - Input invariants: test small random augmentations — if explanations vary unusually vs. clean distribution, raise alarm.- Mitigation: - Adversarial training on explanations: augment training with adversarial δ that maximizes explanation loss and minimize both prediction and explanation loss: min_θ E_x [ max_{δ∈S} ( L_task(f_θ(x+δ), y) + β D_exp(E_θ(x), E_θ(x+δ)) ) ]. - Robust explainers: use smoothed gradients (Randomized Smoothing) or integrated/averaged attributions across noise to reduce sensitivity. - Certifiable explanations: develop bounds on attribution change under allowed perturbation using Lipschitz constants or interval bound propagation. - Input transformations: denoising, JPEG, or learned purification before explanation/prediction.Trade-offs and validation- Defenses may reduce explanation fidelity or model accuracy; measure trade-offs (accuracy vs. explanation-robustness).- Use human evaluation: do changed explanations alter human trust? Combine automated metrics with user studies.This evaluation framework yields measurable attack success rates, defenses’ effectiveness, and practical guidance for deploying explanation-sensitive systems.
python
# python-like pseudocode
x_adv = x.copy()
for t in range(T):
grad = ∇_x [D_exp(E(x), E(x_adv)) - λ D_pred(f(x), f(x_adv))]
x_adv = clip(x_adv + α * sign(grad), x - ε, x + ε)
if D_pred(f(x), f(x_adv)) > τ: project_pred_constraint()HardSystem Design
126 practiced
Design an audit trail and compliance layer for explanation outputs. Specify what to log (input hash, model version, explainer version, background set, attribution vector, timestamp), storage schema, access controls, retention policy to satisfy regulations such as GDPR, and how to enable efficient retrieval for investigations.
Sample Answer
Requirements & constraints- Functional: record every explanation output with verifiable metadata so investigators can reconstruct inputs/outputs and model provenance.- Non-functional: low-latency writes at scale, efficient retrieval by time, input hash, user id, model version; encryption, tamper-evidence, GDPR right-to-be-forgotten compliance.High-level architecture- Inference service → Explanation layer → Audit API → Write-ahead queue (Kafka) → Immutable append store (WORM-enabled S3 or object store) + indexing DB (Postgres or DynamoDB) + cold archive (Glacier).- Integrity service computes and stores cryptographic merkle roots and signs batches with KMS-managed keys.- Access & audit layer provides RBAC/ABAC, audit logs for access, and SIEM integration.What to log (per explanation event)- event_id (UUID), timestamp (ISO8601 UTC)- input_hash (SHA-256 of canonicalized input; store hash only unless retention requires raw)- input_metadata pointer (reference to minimal metadata: user_id pseudonymized, consent_flag, request_id)- model_version (git sha / artifact id), model_checksum- explainer_version (package version + checksum), explainer_config (parameters)- background_set_id (reference), background_set_checksum- attribution_vector (serialized float array; consider storing compressed + quantized)- explanation_summary (human-readable), confidence scores- storage_location(s) for full artifacts (signed object URLs in object store)- signature (batch signature), provenance_chain (links to training dataset snapshot id)- retention_policy_id, legal_hold_flagStorage schema & indexing- Immutable object store: stores full payloads (input if required, raw attribution vectors) in gzipped JSON/AVRO, path by date/model/event_id. Enable object lock (WORM).- Index DB (Relational or NoSQL): - Primary key: event_id - Columns: input_hash, user_pseudonym, model_version, explainer_version, background_set_id, timestamp, storage_path, retention_policy_id, legal_hold_flag - Indexes: timestamp, input_hash, model_version, user_pseudonym- Secondary inverted index for attribution_vector metadata (e.g., top-5 features) to support semantic queries.- Integrity ledger: append-only table of batch merkle roots + KMS signatures.Access controls & security- RBAC enforced via IAM; least privilege for services vs. humans.- ABAC for investigators: require justification, purpose_of_access, and scoped time-limited tokens.- All data encrypted at rest (KMS) and in transit (TLS). Field-level encryption for PII.- Audit every access to index or object store; push to SIEM and create immutable access logs.- Separation of duties: deletion requests go through compliance workflow with approvals and are logged.GDPR & retention policy- Data minimization: prefer storing input_hash and metadata; store raw inputs only when legally required or with consent.- Retention tiers: - Active (0–90 days): indexed DB + object store for quick retrieval. - Nearline (90–365 days): object store + cheaper index; reduced retention of raw inputs (keep hashes + pointers). - Archive (>365 days): cold archive; only hashes, metadata, and signatures retained unless legal hold.- Right-to-be-forgotten: - If user requests deletion and no legal hold: remove PII (pseudonymize), redact raw input from object store (overwrite with placeholder, update checksum and sign operation), record deletion_event with signature. Keep cryptographic hashes and signed audit of deletion. - If deletion conflicts with legal/regulatory holds, mark legal_hold_flag and route to legal team.- Retention_policy_id links to automated TTL jobs; legal_hold overrides TTL.Efficient retrieval for investigations- Search API supports queries by: - input_hash, event_id, model_version, time range, user_pseudonym, background_set_id, top-feature hits.- Use precomputed materialized views for heavy queries (e.g., model_version + date) and partitioning by date.- Attribution vectors: store top-k features and quantized vector summary in index for fast narrowing; fetch full vector from object store only for selected events.- Batch retrieval: support bulk export (signed tarball of objects) with integrity signatures.- Forensics mode: freeze relevant objects (extend legal_hold) and snapshot merkle root for chain-of-custody.Integrity & verifiability- Batch-level merkle trees; sign roots with KMS; publish roots to internal ledger and optionally to external timestamping service.- Periodic verification jobs to detect tampering.Trade-offs & notes- Storing raw inputs increases compliance risk and storage cost; prefer hashes + reversible encryption when consent exists.- Index DB optimized for reads; object store for immutable heavy payloads.- Performance: synchronous write → enqueue to Kafka then ack to inference to keep latency low; ensure consumer writes to immutable store and index asynchronously.This design balances auditability, GDPR compliance, security, and retrieval efficiency while enabling provable integrity for explanations.
EasyTechnical
67 practiced
What is a saliency map for vision models? Describe gradient-based saliency, Grad-CAM, and class-activation maps. Explain when saliency maps can be misleading (e.g., gradient saturation, model focusing on textures) and how to validate that a saliency map is meaningful.
Sample Answer
A saliency map for vision models is a visualization that highlights image pixels or regions most influential to a model’s prediction—used to interpret what the model “looks at.”Gradient-based saliency:- Compute ∂score(class)/∂input_pixels (or absolute/positive gradients). The magnitude shows sensitivity: large gradient means small pixel change strongly affects output.- Simple, fast, works for any differentiable model, but noisy and suffers from gradient saturation or cancellation.Class Activation Maps (CAM):- Original CAM (Zhou et al.) requires a CNN with global average pooling before the final linear classifier. CAM = weighted sum of the final convolutional feature maps using classifier weights → class-specific spatial heatmap.Grad-CAM:- Generalizes CAM to any CNN. Compute gradients of class score wrt final conv feature maps, global-average-pool those gradients to get weights, then weight feature maps and apply ReLU. Produces coarse, class-discriminative localization maps that align with high-level semantics.When saliency maps can be misleading:- Gradient saturation or vanishing: gradients near zero despite important features.- High-frequency/noisy gradients lead to visually uninterpretable maps.- Models may rely on texture or background cues rather than object shape; saliency will reflect those spurious cues.- Attributions can be diffuse or concentrate on unrelated pixels (dataset bias).How to validate meaningfulness:- Input perturbation tests: systematically occlude or alter top-saliency regions and measure drop in class score (causal effect).- Randomization tests: randomize model weights or labels—meaningful maps should degrade.- Compare methods (gradients, Grad-CAM, integrated gradients); consistency increases confidence.- Use localization metrics if ground-truth masks exist (IoU, pointing game).- Check robustness across similar inputs and apply smoothing/aggregation (SmoothGrad) to reduce noise.Takeaway: use saliency maps as hypotheses, validate with interventions and multiple methods before trusting them in production.
MediumTechnical
62 practiced
You must choose between an inherently interpretable model (e.g., GAM, rule-list) and a black-box model that is 3% more accurate for a lending application. List the factors you would evaluate (regulatory risk, fairness, cost of errors, explainability needs, operational complexity) and describe how you would present the trade-offs to business stakeholders.
Sample Answer
I would evaluate the decision across five categories, quantify risks/benefits, and present a clear decision framework to stakeholders.Factors to evaluate- Regulatory risk: Determine if regulations require explainability (e.g., adverse-action notices, GDPR/CCPA). If so, quantify probability and cost of non-compliance (fines, remediation).- Fairness & bias: Compare disparate impact, equalized odds, subgroup error rates. A black box’s 3% gain might hide larger subgroup harms.- Cost of errors: Translate false positives (bad loans) and false negatives (lost revenue/consumer harm) into dollar impact and lifetime value. Use confusion-matrix-weighted cost.- Explainability needs: Identify audiences (underwriters, customers, regulators). For each, specify what explanations are required (feature importance, counterfactuals, rules).- Operational complexity: Estimate deployment, monitoring, and maintenance effort—e.g., need for model cards, explanation tooling (SHAP/LIME), latency and retraining cadence.How I’d present trade-offs- Start with a one-page executive summary: net expected profit lift vs. incremental risk/costs.- Show a table: model A (interpretable) vs. model B (black box) with metrics: overall AUC, lift, subgroup gaps, regulatory risk score, expected financial impact per 10k decisions, and ops cost.- Visuals: ROC/lift curves, subgroup error bar chart, and a stress-test scenario (shifts in input distribution) showing robustness.- Decision framework: If regulatory/fairness constraints are binding or explainability is required for >X% of decisions, favor interpretable. Otherwise, allow black box if mitigations exist (post-hoc explanations, human-in-loop, constrained complexity) and ROI after compliance costs exceeds threshold.- Proposed mitigations: deploy black box behind an interpretable gate for high-risk cases, implement monitoring, produce automated counterfactuals, and run a pilot/A-B test with close guardrails.- Recommendation: give a preferred option plus contingency plan and success metrics (precision/recall targets, monitor alerts, periodic fairness audits) and a go/no-go checklist for production.
HardTechnical
77 practiced
Formulate counterfactual explanations under causal constraints. Present an optimization objective that minimizes a perturbation norm (e.g., L0/L2) to flip model prediction while respecting causal graph constraints (immutable nodes and downstream dependencies). Explain solution approaches and relaxations for efficiency.
Sample Answer
Goal: find minimal perturbation δ to input x that flips model f’s prediction while respecting causal graph G and immutable nodes I (e.g., age). Use structural causal model (SCM) X = g(U) with adjacency from G to enforce downstream dependencies.Optimization objective (mixed form):minimize ||δ||p + λ·L_flip(f(x̃))subject to x̃ = x + δx̃_I = x_I (immutable nodes fixed)for each downstream node j: x̃_j = g_j(x̃_pa(j), u_j) (structural equations)f(x̃) = y_target (or soft constraint via loss)where L_flip = max(0, margin - s(f(x̃), y_target))Approaches:- Exact MIP/MIQP: encode discrete features and nonlinearities with integer vars — precise but scales poorly.- Continuous relaxation: replace L0 with L1 or use elastic net; relax structural equations with differentiable surrogates (learned ĝ_j via regressors or neural nets). Solve with gradient-based optimizers and projected gradients to enforce immutable nodes and bounds.- Greedy & graph-aware search: identify mutable ancestor set A of decision node, search sparse interventions on A using beam search or coordinate descent.- Latent-space/generative models: learn invertible mapping z↔x (flow/VAE); optimize in z (smooth), then map back, enforcing causal corrections by re-simulating downstream nodes via SCM.- L0 approximations: use hard concrete/Gumbel-softmax or iterative reweighted L1 to promote sparsity.Practical tips:- Precompute affected downstream set to reduce variables.- Penalize unrealistic suggestions using density or domain constraints.- Use counterfactual validity checks by simulating SCM with sampled exogenous noise.Complexity: MIP exponential; relaxed gradient methods polynomial per iteration but nonconvex — use multiple restarts and warm starts. Edge cases: unidentifiable interventions when SCM unknown; discrete-only actions require combinatorial solvers.
Unlock Full Question Bank
Get access to hundreds of Model Interpretability and Explainability interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.