Research Problem Formulation and Scoping Questions

Overview
When proposing a research project I document constraints across compute, data availability & quality, annotation cost, timeline, regulatory/ethical, and team expertise—each with mitigations and quick feasibility checks.

Compute (why)

• Why: Needed for model training, reproducibility, and experiments (GPU/TPU, storage).
• Mitigations: Use smaller-scale proofs, mixed precision, spot instances, curriculum of experiments from cheap → expensive. Containerize for portability.
• Quick checks: Benchmark a toy model (1–2 epochs) on target hardware; estimate cost with cloud pricing calculator.

Data availability & quality (why)

• Why: Research depends on enough representative data and labels; biases affect validity.
• Mitigations: Augment with synthetic data, transfer learning, self-supervised pretraining, domain adaptation.
• Quick checks: Sample statistics (size, class balance, missingness), run simple baseline model (logistic/linear) for signal-to-noise.

Annotation cost (why)

• Why: High-quality labels (expert annotation) can be expensive/time-consuming.
• Mitigations: Active learning, weak supervision, crowdsourcing with quality controls, label propagation.
• Quick checks: Label a small pilot (100–500 examples), measure inter-annotator agreement and per-label cost/time.

Timeline (why)

• Why: Paper deadlines, resource allocation, and milestone planning.
• Mitigations: Stage-gated milestones (proof-of-concept → scale → analysis), parallelize tasks, reserve buffer for unexpected findings.
• Quick checks: Gantt-like estimate for major milestones; identify critical path and earliest go/no-go decision.

Regulatory / Ethical constraints (why)

• Why: Data privacy, consent, export controls, dual-use risks can block research or publication.
• Mitigations: Privacy-preserving methods (DP, federated), IRB review, legal consultation, pre-registration of risk mitigations.
• Quick checks: Confirm data licensing/consent status; run a quick privacy risk checklist and flag need for IRB.

Team expertise (why)

• Why: Required skills (theory, systems, domain knowledge) determine feasibility and quality.
• Mitigations: Hire/contract experts, collaborate with domain partners, schedule mentoring, reduce scope to align with strengths.
• Quick checks: Map required skills vs. current team; identify 1–2 critical hires or collaborators and availability.

This structured feasibility snapshot enables an early go/no‑go and a clear mitigation plan for reviewers and stakeholders.

Approach (brief)
I treat gap analysis as a multi-source evidence synthesis that connects observed limitations to a novel, high-impact research direction.

Evidence sources (concrete)

• Systematic literature review: keyword + backward/forward citation chaining; extract tasks, methods, metrics, datasets, and explicit limitations.
• Benchmark performance: run or collate leaderboards, compute effect sizes and variance across datasets; identify hard instances where SOTA fails.
• Failure-mode cataloging: label model errors (e.g., calibration, spurious correlation, OOD collapse) with representative examples and root causes.
• Practitioner interviews: structured interviews or surveys with domain experts to surface unmet needs, constraints, and deployment pain points.

Triangulation method

• Cross-map: align literature-identified limitations with benchmark failure clusters and practitioner-reported pain points.
• Quantitative convergence: require at least two sources to corroborate a gap (e.g., literature notes robustness gap; benchmarks show >20% drop OOD; practitioners report deployment failures).
• Qualitative synthesis: use thematic coding from interviews to explain why benchmarks/literature gaps matter in practice.

Artifacts produced

• Capabilities-vs-gaps table (methods × capabilities, with evidence links and severity scores).
• Prioritized failure-case list (impact × feasibility matrix, representative examples).
• Short reproducible benchmark suite and sample failure instances.
• One-page evidence dossier linking each proposed research claim to specific artifact entries.

How artifacts support novelty & impact

• Novelty: show that proposed method targets high-severity gaps not addressed by existing methods in the capabilities table.
• Impact: use prioritized failure cases and practitioner testimony to argue real-world importance; bench results quantify expected improvement.
• Reproducibility and traceability: evidence links let reviewers verify claims and see why the work advances state of the art.

Overview — why literature review reframes an idea
A deep literature review converts a vague “novel” idea into a precise contribution by (1) revealing true gaps, (2) aligning claims to measurable baselines, and (3) avoiding reinvention. For top-tier venues novelty must be both technically distinct and well-positioned against closest prior work.

Discovering and citing closest prior work

• Systematic searches: query arXiv, ACL/NeurIPS/ICLR/ICML proceedings, Google Scholar, Semantic Scholar, and citation graphs (forward/backward).
• Keywords + method tags + year filters; track influential authors and labs.
• Identify the 3–5 closest works: summarize their problem statement, assumptions, metrics, and limitations. Cite them in-text with precise contrasts (e.g., “extends X by relaxing assumption Y”).

Framing and Related Work

• Start Related Work with a taxonomy: categories that map to your contribution. For each category, state what is solved, what remains, and where your paper sits.
• Use comparative statements and a small table (if space allows) listing tasks, assumptions, datasets, and performance. Conclude with one paragraph: “Our contribution differs because…”.

Choosing experiments: essential vs optional

• Essential: experiments that validate your core claims — ablations isolating proposed components, comparisons to the strongest baselines (including reproduced key baselines), and evaluations on standard datasets/metrics. Include statistical significance and runtime/memory where relevant.
• Optional: extended studies (additional domains, large-scale ablations, user studies) that strengthen generality but don’t affect the central claim. Run these if time/resources and reviewers’ likely weaknesses suggest them.

Reasoning
This workflow maximizes clarity (explicit contrasts, reproducible baselines) and novelty (shows incremental/orthogonal advances), which reviewers at top venues expect.

Principled checklist for selecting baselines (Research Scientist perspective)

1. Goal & constraints

• State the evaluation goal (accuracy, latency, robustness) and resource limits (GPU hours, memory). This narrows baseline choice.

2. Trivial / sanity baselines

• Simple heuristics and random predictors (e.g., majority class, TF-IDF+logistic for NLP, linear regression). Ensure your method improves beyond obvious, cheap solutions.

3. Strong SOTA baselines

• Reproduce or use published implementations of top-performing recent papers and well-maintained libraries. Match training protocol, data splits, augmentation, and hyperparameter search to ensure fair comparison.

4. Ablations & oracle baselines

• Remove or replace each novel component to show contribution.
• Include oracle or upper-bound baselines (e.g., access to future info) to contextualize gap to ideal.

5. Computational / engineering baselines

• Compare to cheaper variants: smaller models, distillation, pruning, and tuned hyperparameters of baselines. Report FLOPs, latency, memory, cost-to-train.

6. Robustness and transfer baselines

• Test domain-shifted data, adversarial/noisy inputs, and downstream-task performance against baselines.

7. Reproducibility checklist

• Fix seeds, report hyperparameters, hardware, evaluation scripts, and use shared datasets/splits.

Why this matters: combine trivial, SOTA, ablations and efficiency baselines to demonstrate statistical improvement, causal attribution, and practical usefulness.

Clarify goals & artifacts

• I will publish code, training/inference scripts, model checkpoints, evaluation scripts, environment specification, and a reproducibility README that documents exact commands to reproduce provable SOTA.

Code & checkpoints

• Release a GitHub repo with version tag matching paper (v1.0). Include training and eval entrypoints (train.py, eval.py).
• Upload checkpoints to a stable storage (Hugging Face Hub or institutional S3) with SHA256 checksums and model card describing architecture, hyperparameters, training steps, and provenance.

Environment & seeds

• Provide Dockerfile + conda/environment.yml and a small container image. Pin major dependency versions (PyTorch, CUDA).
• Record and expose all RNG seeds (Python, NumPy, PyTorch, CUDA) as CLI arguments and defaulted in config. Note any non-deterministic CUDA ops and how to avoid them (torch.use_deterministic_algorithms).

Dataset & licensing

• Document dataset source, exact split files (hashes), and license or access instructions. If redistribution is restricted, provide preprocessing scripts and a manifest to recreate the exact training set; include citation and any required acknowledgements.

Evaluation & metrics

• Publish eval scripts that load checkpoint by hash and run the same preprocessing and metric code used in paper. Include unit tests comparing small gold wafers (tiny inputs) to expected outputs.

CI / verification

• Set up GitHub Actions to:
  • Lint and run unit tests.
  • Smoke-train for 1–2 steps and run eval on a tiny subset to ensure end-to-end runs.
  • Periodically (or on release) run a full verification job on a held-out compute account that re-runs final eval and reports matching results within stated tolerance.

Reporting variability & compute costs

• Report mean ± std over N independent seeds (e.g., N=5) and include per-run scores in a reproducibility appendix. Provide bootstrap confidence intervals and the code to reproduce them.
• Report exact wall-clock hours, GPU type(s), number of GPUs, total GPU-hours, peak memory, and estimated FLOPs and carbon/energy if measured.

Practical notes

• Provide a reproducibility checklist in README with exact commands to (1) prepare data, (2) train model, (3) evaluate checkpoint. Encourage community verification via issues/PRs and include contact for help.