Research Problem Formulation and Scoping Questions

Problem statement (concise)
Design a supervised learning algorithm that improves few-shot image classification accuracy on domain-shifted target datasets given labeled source data and ≤5 labels/class on target.

Core assumptions

• Source and target share label space but differ in input distribution (covariate shift).
• A small set of representative target labels is available.
• Feature extractor pretrained on large corpus provides useful representations.

Measurable success metrics

• Primary: top-1 accuracy on target few-shot test set (averaged over N-way K-shot episodes).
• Secondary: calibration (ECE), robustness under simulated shift (accuracy drop), and compute cost (GPU-hours).

Constraints

• Data: ≤5 labeled examples/class on target; access to unlabeled target pool limited to 10k images.
• Compute: single 8-GPU node, training budget ≤200 GPU-hours.

Testable hypothesis & failure modes
Hypothesis: Fine-tuning only classifier head with a contrastive regularizer improves target few-shot accuracy by ≥5% vs. frozen backbone baseline.
Test: run matched experiments with seeds, report confidence intervals.
Failure modes: (1) regularizer overfits small target set and reduces generalization; (2) domain discrepancy is too large—backbone features not transferable; (3) hyperparameter sensitivity causes unstable gains.

Goal & Hypothesis
Debunk the assumption “scaling data always improves generalization.” Hypothesis: beyond task-, model-, and data-quality-specific regimes, additional data yields diminishing or negative returns due to label noise, covariate shift, and model mis-specification.

Literature review plan

• Scope: empirical scaling laws, double descent, dataset curation, label noise, domain shift, active learning, transfer learning.
• Sources: NeurIPS/ICLR/ICML papers, arXiv preprints, replication studies, benchmark datasheets (Datasheets for Datasets), and meta-analyses.
• Deliverable: annotated map linking claims to experimental settings (model size, optimizer, dataset provenance, label quality).

Experimental design

• Tasks: image classification (CIFAR-10/StreetView subset), NLP classification (IMDB / noisy Web-scraped), and synthetic controlled tasks.
• Factors (fully crossed): data size (log scale), label noise (0–40%), domain shift (in-distribution vs OOD), model capacity (under/overparameterized), training compute (epochs, early stopping).
• Metrics: test accuracy, calibration, worst-group accuracy, sample-wise loss dynamics, compute per-point efficiency.

Controlling confounders

• Use synthetic data to isolate noise/shift effects.
• Fixed training protocol per capacity tier; hyperparameter sweep held constant across sizes or tuned on a validation set independent of size.
• Repeat experiments (seeds) and use statistical tests (ANOVA, interaction terms).
• Propensity-weighting to correct sampling-induced covariate changes.

Narrative & anticipating objections

• Present stepwise evidence: synthetic shows failure modes; real-world benchmarks confirm; ablations rule out hyperparameter or capacity artifacts.
• Address selection bias (“we cherry-picked datasets”) by releasing dataset scripts and preregistering protocols.
• Address overfitting to evaluation by reporting multiple metrics and OOD tests.
• Discuss scope: claim is conditional, not universal—provide practical guidelines (when more data helps vs when curation/modeling is superior).

Repro & Impact

• Release code, seeds, datasets, and a reproducibility checklist; propose follow-up recommendations for dataset builders and practitioners.

Clarify goals & scope

• Objective: unify five papers’ tasks into a single reproducible benchmark that enables fair comparison across datasets, models and metrics. Target: research-grade reproducibility and extensible evaluation.

High-level design

• Single canonical dataset bundle + metadata layer + evaluation harness + provenance & compute reporting.

Dataset harmonization steps

1. Inventory: list data sources, formats, label schemas, licenses, sampling procedures.
2. Schema mapping: define canonical label space (merge/split labels; keep mapping table).
3. De-duplication & overlap detection across sources.
4. Normalization: unify file formats, tokenization (or raw text), image sizes, metadata fields.
5. Bias & split stratification: compute per-source distributions; create stratified train/val/test splits preserving original domain balance and OOD splits when relevant.
6. Provide dataset manifests with checksums and lineage.

Common preprocessing

• Provide reference pipeline modules (versioned): tokenizer settings, image transforms, normalization constants, missing-value handling. Export both “raw” and “preprocessed” artifacts and exact seed-controlled pipelines.

Choice of metrics & protocol

• Pick primary metric per task (e.g., F1-macro for class-imbalanced classification) and secondary ones (accuracy, AUROC, calibration, latency).
• Report confidence intervals (bootstrap) and per-slice metrics (by domain/label).
• Define single scoring script; open-source and deterministic.

Compute & hyperparameter reporting

• Mandate: hardware (GPU/CPU types), wall-clock, FLOPs estimate, peak memory, batch size, seed.
• Publish full hyperparameter sweep space, selection procedure (validation criteria), best config, and number of trials.
• Use standardized benchmarks for TPU/GPU FLOP measurement where possible.

Fairness & reproducibility

• Enforce fixed random seeds and deterministic kernels, provide Docker/Conda images, CI tests reproducing a published run.
• Baselines: include re-implemented baselines with identical preprocessing and hyperparameter-budget limits.
• Define budgets: equal compute budgets for comparative runs and a separate “unbounded” leader board with compute reported.

Trade-offs

• Harmonization can remove task-specific niceties; keep raw data and mapping to allow downstream research.
• Reproducibility vs. ease-of-use: provide both lightweight and heavyweight reproducibility artifacts.

Example: provide a canonical repo with dataset manifests, preprocessing scripts, evaluation harness, baseline models, and a results registry that stores config, seeds, hardware, and artifact checksums.

Decision framework (overview)

I use a weighted-prioritization framework combining five axes: Scientific Novelty, Publication Potential, Product Impact, Resource Requirements, and Risk. Each candidate project is scored 1–5 on each axis, with initial weights (example): Novelty 25%, Publication 25%, Product Impact 25%, Resources 15%, Risk 10%. Scores produce a ranked list; top candidates enter a staged-gating process (explore → prototype → scale).

Why this works

• Balances long-term knowledge creation (novelty/publication) with near-term value (product impact).
• Explicit resource and risk adjustments prevent overcommitment to high-risk/high-cost ideas.
• Staged gates allow failing fast while preserving runway for high-reward science.

Soliciting input

• Triage workshop with researchers (technical feasibility), product managers (impact), and infra/SRE (resources).
• Anonymous ranking survey to reduce bias.
• Quarterly reviews with external academic collaborators to validate novelty.

Metrics for success

• Scientific: number of submissions/acceptances at top conferences, citation velocity.
• Product: prototypes integrated, user/ML-metric improvements (e.g., accuracy, latency, retention uplift).
• Operational: time-to-prototype, cost per experiment, reproducibility score.
• Risk: fraction of projects terminated at explore gate vs. successful transitions.

Governance

• Re-evaluate weights biannually based on org goals; reserve 15% capacity for high-risk blue-sky work.

Overview — why literature review reframes an idea
A deep literature review converts a vague “novel” idea into a precise contribution by (1) revealing true gaps, (2) aligning claims to measurable baselines, and (3) avoiding reinvention. For top-tier venues novelty must be both technically distinct and well-positioned against closest prior work.

Discovering and citing closest prior work

• Systematic searches: query arXiv, ACL/NeurIPS/ICLR/ICML proceedings, Google Scholar, Semantic Scholar, and citation graphs (forward/backward).
• Keywords + method tags + year filters; track influential authors and labs.
• Identify the 3–5 closest works: summarize their problem statement, assumptions, metrics, and limitations. Cite them in-text with precise contrasts (e.g., “extends X by relaxing assumption Y”).

Framing and Related Work

• Start Related Work with a taxonomy: categories that map to your contribution. For each category, state what is solved, what remains, and where your paper sits.
• Use comparative statements and a small table (if space allows) listing tasks, assumptions, datasets, and performance. Conclude with one paragraph: “Our contribution differs because…”.

Choosing experiments: essential vs optional

• Essential: experiments that validate your core claims — ablations isolating proposed components, comparisons to the strongest baselines (including reproduced key baselines), and evaluations on standard datasets/metrics. Include statistical significance and runtime/memory where relevant.
• Optional: extended studies (additional domains, large-scale ablations, user studies) that strengthen generality but don’t affect the central claim. Run these if time/resources and reviewers’ likely weaknesses suggest them.

Reasoning
This workflow maximizes clarity (explicit contrasts, reproducible baselines) and novelty (shows incremental/orthogonal advances), which reviewers at top venues expect.