Research Tools and Platforms Proficiency Questions

Demonstrate proficiency with research tools and platforms commonly used at FAANG companies: user research platforms (Validately, UserTesting, Respondent), analytics tools (Google Analytics, Mixpanel, Amplitude), survey tools (Qualtrics, SurveyMonkey), and analysis software (NVivo, ATLAS.ti, or custom analysis approaches). Be prepared to discuss tool selection based on research needs.

MediumTechnical

24 practiced

Design a recruitment screener for Respondent to recruit 100 customers who used the new checkout in the past 30 days across three countries, with quotas for 18–25, 26–40, 41+. Include minimum necessary screening questions, sample size rationale, incentive considerations, and how you'll prevent repeat participants from prior studies.

Sample Answer

**Overview & goal**Recruit 100 customers (across 3 countries) who used the new checkout in the past 30 days, with age quotas 18–25, 26–40, 41+. Balanced sample for cross-country UX feedback.

**Sample-size & quota plan**- Total N = 100 (reason: sufficient for qualitative usability + basic quantitative subgroup comparisons; gives ~30–35 per country for cross-country patterns)- Country quotas: aim for roughly equal split (33/33/34) unless product priorities adjust.- Age quotas per country: 33% each age band where possible (e.g., per country target ≈ 11 per age group).- Allow oversample +15% to account for dropouts (invite ~115).

**Minimum screening questions (order & logic)**1. Country of residence (required)2. Age (enter DOB or select band) — force into 18–25 / 26–40 / 41+3. In the past 30 days, have you completed a purchase using our new checkout? (Yes / No) — if Yes, follow-ups: - Approx. date of last purchase (to verify within 30 days) - Platform used (mobile web / app / desktop) - Did you complete the purchase successfully? (Yes/No)4. Purchase frequency (first-time / occasional / frequent)5. Are you a current employee, contractor, or close relative of the company? (exclude)6. Contact method for incentive (email) and consent to be recontacted7. Short verification: order ID or last 4 digits of card + date OR screenshot upload (optional but preferred) — alternative: match via panel ID if using Respondent.

Include a brief attention check (e.g., “Select ‘Yes’ to confirm you read this”).

**Incentive considerations**- Target incentive: $40–$75 USD equivalent depending on task length (e.g., 45–60 minute session -> $60). Rationale: compensates for time and experience-specific recruitment difficulty (recent checkout users), increases completion and quality.- Tiered bonus: extra $10 for providing verifiable proof (screenshot/order ID) to encourage validation.

**Preventing repeat participants**- Use Respondent panel IDs and cross-check against prior-study lists (upload historical participant IDs/emails to blocklist).- Capture and dedupe on unique identifiers: email, phone, panel ID.- Server-side device fingerprinting + IP checks for suspicious repeats.- Require order-specific verification (order ID, date) not reusable across studies.- Maintain a rolling internal blocklist for participants who appeared in prior closely related checkout studies within a defined cool-down window (e.g., 90 days).

**Quality control & logistics**- Soft-launch with small batch to validate screener logic, verification flow, and quota fill.- Communicate privacy: explain how order info is used and stored securely.- Provide clear scheduling windows and confirmation reminders to reduce no-shows.

This screener balances recruitment speed, data quality, and participant privacy for actionable design research across countries and age groups.

HardTechnical

23 practiced

You have qualitative codes exported from NVivo (CSV with transcript_id, code, start_char, end_char) and Amplitude event data (CSV with user_id, event_name, event_time). Describe a reproducible data-processing pipeline (tools and steps) to link codes to users, compute per-user code frequencies, and run a logistic regression in Python to predict conversion based on the presence of certain codes. Provide pseudocode or skeleton Python for the join and model step.

Sample Answer

**Overview & tools**- Tools: Python (pandas, scikit-learn, statsmodels), SQLite or Postgres for joins, DVC/Git for versioning, Jupyter for exploration, pytest for small checks.- Inputs: nvivo.csv (transcript_id, code, start_char, end_char), amplitude.csv (user_id, event_name, event_time), transcript→user mapping file (transcript_id, user_id).

**Reproducible pipeline steps**1. Repo + environment: Git + requirements.txt or conda env + DVC to track CSVs and outputs.2. Ingest: load CSVs into a database or pandas DataFrames. Validate schema and dedupe.3. Map codes → users: join nvivo on transcript→user mapping to attach user_id to each coded segment.4. Aggregate per-user code frequencies: group by user_id and code to count occurrences; pivot to wide format; produce both counts and binary presence features.5. Merge with Amplitude conversion label: define conversion (e.g., presence of event_name == "Purchase" within window); create y (0/1) per user.6. Modeling: hold-out test split, standardize if needed, fit logistic regression with L1/L2 regularization, inspect coefficients and p-values (use statsmodels for inference).7. Reproducibility & checks: save intermediate datasets, unit tests for joins, record random seeds and hyperparameters.

**Skeleton Python for join + model**

python

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

# Load
nv = pd.read_csv("nvivo.csv")
map_df = pd.read_csv("transcript_user_map.csv")
amp = pd.read_csv("amplitude.csv")

# Attach user_id
nv = nv.merge(map_df, on="transcript_id", how="left")
nv = nv.dropna(subset=["user_id"])

# Per-user code counts and binary presence
code_counts = nv.groupby(["user_id","code"]).size().unstack(fill_value=0)
code_presence = (code_counts > 0).astype(int).add_prefix("code_")

# Build target: conversion if any purchase event
conv = amp.groupby("user_id")["event_name"].apply(lambda evs: int((evs=="Purchase").any())).rename("converted")
df = code_presence.merge(conv, left_index=True, right_index=True, how="left").fillna(0)

X = df.drop(columns="converted")
y = df["converted"]

# Train/test and model
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
Xtr = scaler.transform(X_train)
Xte = scaler.transform(X_test)

clf = LogisticRegression(penalty="l2", C=1.0, solver="saga", max_iter=1000)
clf.fit(Xtr, y_train)
pred = clf.predict_proba(Xte)[:,1]
print("AUC:", roc_auc_score(y_test, pred))
# Inspect top positive/negative codes
coef = pd.Series(clf.coef_[0], index=X.columns).sort_values(ascending=False)
print(coef.head(10))

**Notes, choices & pitfalls**- If transcripts map to multiple users or anonymized IDs, handle many-to-one carefully.- Consider time windows: only codes before conversion should be used for prediction.- Use regularization to avoid overfitting with many codes; consider dimensionality reduction (PCA) or feature selection.- Document pipeline steps, seed, and dataset versions for reproducibility.

MediumTechnical

31 practiced

Describe how you would build a repeatable dashboard in your BI tool (Looker, Mode, or internal dashboard) that surfaces the top 10 research-derived product insights each quarter by combining Amplitude event exports and survey results. Outline data transforms, join keys, and how you'd operationalize citations to original artifacts (video clips, transcripts).

Sample Answer

**Approach summary**Build a repeatable ETL + BI pipeline that merges Amplitude event exports with survey results, ranks research-derived insights, and exposes citation links (video clip, transcript) for each insight. I’d implement in Mode or Looker backed by a data warehouse (Snowflake/BigQuery).

**Data transforms & join keys**- Amplitude events -> canonical events table - Transform: normalize user_id, event_time, event_name, properties JSON -> flattened columns - Key: user_id, event_time- Surveys -> canonical responses table - Transform: normalize respondent_id, submit_time, question_id, answer_text, theme tags - Key: respondent_id- Mapping table: user_id <-> respondent_id (from consent or hashed email) or session_id - Key: user_id = respondent_id or session_id- Insight extraction table - Transform: researcher-tagged insight_id, theme, summary, confidence_score, supporting_evidence_ids (array), created_at

Example SQL (extract join)

sql

SELECT i.insight_id, i.theme, i.summary, i.confidence_score,
       COUNT(DISTINCT e.user_id) as user_count,
       AVG(e.session_duration) as avg_session
FROM insights i
LEFT JOIN events e
  ON e.user_id = i.user_id -- or join via mapping table
LEFT JOIN survey_responses s
  ON s.respondent_id = i.user_id
GROUP BY 1,2,3,4

**Ranking & business logic**- Score = weighted function: researcher confidence (0.5) + event signal strength (0.3 normalized) + survey frequency (0.2)- Materialize top 10 per quarter into a table with quarter_id, rank, and TTL refresh schedule.

**Operationalizing citations**- Store evidence artifacts in a media store (S3/Drive) with stable URLs and metadata table: - evidence_id, artifact_type (video, transcript), url, start_time, end_time, transcript_snippet- In dashboard, surface click-to-open links and embed video player where supported; show transcript snippet and "open full transcript" link.- In Looker: create a LookML view for insights with nested fields and url parameters; in Mode: use parameterized queries and visual card with rich links.

**Governance & repeatability**- Scheduled DAG (Airflow) to run ETL quarterly + incremental- Tests: key presence, mapping coverage, artifact accessibility- Documentation: README with tagging taxonomy and scoring weights, and an exportable CSV of top-10 with evidence links for stakeholders.

EasyTechnical

31 practiced

As a Design Researcher using NVivo for interview analysis, outline the basic workflow you'd follow from importing transcripts to producing a coded-theme report. Mention nodes (codes), memos, inter-coder reliability, and one NVivo feature that helps surface patterns.

EasyTechnical

47 practiced

Describe the practical differences between recruiting participants on Respondent.co vs using an external in-product recruitment intercept for an unmoderated usability study. Address speed to recruit, sample control, incentives, data quality, and tracking/repeat participants.

Unlock Full Question Bank

Get access to hundreds of Research Tools and Platforms Proficiency interview questions and detailed answers.

Join thousands of developers preparing for their dream job.