Role Responsibilities and Success Criteria Questions

In short: As a data science (DS) organization grows, it typically moves from reporting directly into a business leader with no dedicated management layer, to a dedicated DS manager balancing central and embedded priorities, to a full DS or Applied Science org with a formal technical ladder. Each stage trades speed and visibility for structure and mentorship.

Startup (2 to 5 data scientists)

• Reporting: Usually reports directly to the Head of Product, CEO, or an Engineering lead; no dedicated DS management layer yet.
• Pros: High visibility, freedom to define your own scope, a direct line into strategic decisions.
• Cons: Little structured mentorship, an invented-on-the-fly promotion path, and a real risk of getting pulled into ad hoc "just run this query" work with no protected time for real modeling.
• Effect on prioritization: driven almost entirely by whatever the founder or CEO's problem of the week is; there is no formal roadmap.
• Effect on career development: growth is self-directed, and what "senior" even means is negotiated per person rather than defined.

Mid-size (5 to 20 data scientists)

• Reporting: Usually a dedicated DS or Analytics manager, who reports to a VP of Product, Engineering, or a Head of Data.
• Pros: Enough peers for real code and design review, the first real career ladder starts to exist, and some specialization (for example, one DS embedded per product team) becomes possible.
• Cons: A "two-boss problem" can appear: reporting centrally to the DS manager while being embedded day to day with a product team whose priorities differ.
• Effect on prioritization: split between a central roadmap and each embedded team's asks, requiring explicit negotiation between the DS manager and product leads.
• Effect on career development: a junior and mid/senior leveling framework typically appears here, with a manager able to advocate for people in calibration (a formal cross-manager meeting where performance ratings and promotion decisions are compared and normalized, so one manager's "exceeds expectations" means the same thing as another's).

Large (more than 20 data scientists)

• Reporting: Usually organized under a Head of Data Science or Applied Science, sometimes with individual data scientists still dotted-lined (formally reporting to one manager on paper while also reporting, day to day, into a different team) into product verticals.
• Pros: A mature technical ladder up to Staff or Principal, dedicated platform or MLOps (machine learning operations, the tooling and practices for running ML systems reliably in production) support, and cross-team knowledge sharing.
• Cons: More process and approval overhead, a risk of being organizationally distant from the actual product decision, and political friction between centralized and embedded reporting models.
• Effect on prioritization: a formal quarterly planning process across a portfolio of embedded and platform teams, rather than informal negotiation.
• Effect on career development: the most mature ladder and calibration process, but promotion can be slower and more competitive since more data scientists compete for the same senior slots.

Trade-offs and pitfalls: there is no universally "right" structure; a candidate should be able to describe which band they've actually worked in and reason about how their approach would need to change in a different one, rather than assuming their own experience generalizes.

Direct answer

Data-science-owned deliverables need two different kinds of metrics: quality metrics that describe whether a model or pipeline is doing its job well, and reliability commitments, service-level objectives and agreements (SLOs and SLAs, an internal target versus a formal promise for how reliably a system performs), that describe whether it's available and fresh enough to be trusted at all.

Metrics and SLAs by deliverable

Models

• Accuracy or business-proxy metric (e.g. precision, recall, or conversion lift). Matters because a model can run perfectly and still be wrong, that's the core risk of a model as opposed to a pipeline. Measure via shadow evaluation (running the candidate model in parallel on real traffic without letting its output affect any decision, then comparing it after the fact) against ground truth (the actual correct outcome, confirmed after the fact) on a rolling window, or an online experiment. Escalate by alerting the model owner if the metric drifts past a set threshold for several consecutive days, for example if weekly precision sits more than three points below the trailing 30-day median on three consecutive days, so a single noisy day does not wake anyone.
• Serving latency SLA, e.g. the 95th percentile (p95, the value below which 95% of requests fall) response time under a fixed budget, for example p95 under 150 milliseconds for a scorer called inside a page render that has a 500-millisecond budget of its own. Derive that number by subtracting from the caller's budget rather than picking a round figure, since the only thing that makes the target defensible is what it leaves for everything else on the request path. Matters because a slow model breaks the calling application's own SLA. Measure by instrumenting the serving endpoint. Escalate to on-call if the p95 breaches the threshold for a sustained window.
• Feature or data drift. Matters because input distributions can silently shift and degrade a model with no code ever changing. Measure with a statistical distance (a single number summarizing how different two distributions are, so a large jump signals real drift) between current and training-time feature distributions, checked daily. Escalate to the model owner, not a general pager, unless drift is severe.

Pipelines

• Data freshness SLA, e.g. new data lands within a fixed window of the source event: perhaps 60 minutes for a table someone reads every morning, 24 hours for a metric reported monthly. Set the window from how soon a human actually acts on the number, not from how fast the pipeline happens to run today, otherwise the SLA just ratifies current behavior and can never be breached. Matters because stale input silently produces stale output. Measure via timestamp-lag monitoring. Escalate to data engineering and notify downstream consumers.
• Pipeline success rate, for example 99% of scheduled runs completing with output row counts inside an expected range, measured monthly. Matters because a silent partial failure (job "succeeds" but writes incomplete data) is worse than an obvious crash. Measure with row-count and null-rate checks on the output, not just the job's exit code. Escalate via one automatic retry, then page. Where several models read from one shared feature store (a central service that computes model input variables once and serves the same values to training and to live scoring), these pipeline commitments are shared rather than per-model: one freshness breach there degrades every consumer at the same time, so name its owner and its pager before an incident rather than during one.

Dashboards

• Uptime and load time, for example loading in under 3 seconds for a dashboard people open between meetings. Matters because an unusable dashboard gets abandoned even when the underlying data is correct. Measure with uptime and load-time monitoring. Escalate as a standard ticket, urgent only for executive-facing dashboards.
• Adoption. Matters because a technically healthy dashboard nobody opens isn't succeeding. Measure with weekly active viewers. Escalate is not a page, it's a quarterly review with the requesting stakeholder.

Trade-offs and pitfalls

Setting SLAs too tight burns the team on false alarms; too loose lets real degradation go unnoticed for weeks. The clearest pitfall is applying a model's SLA and a pipeline's SLA the same way: a slow-but-correct pipeline and a fast-but-wrong model are very different failures, and they need different owners and different urgency.

Five 90-day metrics should be concrete, cheap to measure with data you already have or can get quickly, and tied to something the team actually cares about, not vanity numbers. I use the SMART framework (Specific, Measurable, Achievable, Relevant, Time-bound) as the filter for whether a proposed metric is real or just an aspiration. Below is the set I would commit to, each with why it matters, the data source and method, and a target range. The ranges are opening positions stated on the assumption of an average existing baseline; I would confirm or renegotiate each one in week two or three against the team's real numbers, and I say that out loud rather than pretending day-one precision I do not have.

1. Shipped, reviewed deliverables tied to the team's top objective. Why it matters: it proves I can navigate the codebase, the review culture, and the release process, which is the precondition for everything else. Source and method: count merged changes in version control, filtered to the objective's tracking label, excluding trivial changes. Target: at least two substantive changes merged by day 45 and six to eight by day 90, and every one of them reviewed by someone other than my onboarding buddy.

2. One validated impact result, with its unit cost. Why it matters: output that nobody can attribute is indistinguishable from busywork. Source and method: an A/B test (a controlled comparison against a held-out group that does not get the change) read from the experimentation platform, paired with the serving or query cost per unit pulled from the cloud billing export. Target: one experiment designed, shipped and read out inside 90 days, with the result reported honestly whether it won or lost, and unit cost held at or below the agreed ceiling rather than traded away for the win.

3. A reliability improvement on something a downstream consumer depends on. Why it matters: the pipeline or service running is not the point, the consumer trusting it is. Source and method: the scheduler or monitoring tool's run history for failure and late-delivery rate over a rolling 30 days, confirmed with the two or three teams that consume the output. Target: cut the failure-plus-late rate on one named dependency by roughly half against its trailing 30-day baseline, and get that confirmed by a consumer in their own words, not just on the dashboard.

4. Finished work with a quality bar attached. Why it matters: counting work in progress is how a quarter disappears with nothing landed. Source and method: the team's tracker, counting only items that reached production and carry a documented accuracy, adoption, or acceptance check. Source of the quality number is whatever the team already records; if it records nothing, establishing that record is itself part of the metric. Target: two to three finished items by day 90, none of them still labelled in review.

5. Adoption of one thing I built, by a named group. Why it matters: a dashboard or tool nobody opens is an output metric pretending to be an impact metric. Source and method: the tool's own usage log for weekly active viewers in the target group, plus a short qualitative check with three of those users on whether it changed a decision. Target: a majority of the named target group using it weekly by day 90, and at least one concrete decision someone can point to. I would avoid leaning on a Net Promoter Score here: it is a loyalty measure built from a single likelihood-to-recommend question scored zero to ten, it needs far more respondents than a ten-person stakeholder group to be stable, and weekly active use plus three honest conversations tells you more at this size.

How the same five slots look in different roles

The slots are deliberately generic because they travel. Slot 1 is a Software Engineer's "did specific, verifiable work land." Slot 2 is the AI Engineer flavor: impact plus inference cost per prediction, not accuracy alone. Slot 3 is the Data Engineer's consumer-benefit framing. Slot 4 is the Data Scientist's way of counting finished models or analyses rather than work in progress. Slot 5 is the BI Analyst's dashboard-adoption framing. A Solutions Architect's version blends slots 3 and 5: pick a specific customer or internal team consuming your architecture recommendations, and track whether that team actually adopted the recommended design, not just whether you delivered a document, confirmed directly with them.

Trade-offs and pitfalls

Committing to a hard numeric target before you have seen the team's real baseline is a common mistake, which is why every range above is explicitly an opening position to be reset in week two or three. The opposite mistake is worse in an interview: refusing to name any number at all reads as hedging, so state the range and state what would move it. A subtler pitfall is picking five metrics that are all about your own output (things you did) with none about downstream impact (things that changed because of what you did); slots 1 and 4 are output, slots 2, 3 and 5 are impact, and a strong set always has at least one of each.

A Data Scientist on a cross-functional product team translates a business or product question into a measurable, testable analysis or model, and is responsible for both getting the right answer and making sure it actually gets used.

A typical day

Mornings usually start with checking any running experiments or deployed model's performance for regressions, then standup. Mid-day is heads-down: exploratory analysis, feature engineering, or model iteration. Afternoons are for stakeholder syncs and reviewing results with whoever will act on them.

Stakeholders and why

Product frames the question and prioritizes it against everything else on the roadmap. Design shapes how a model's output is presented to a user, for example how a recommendation's confidence is communicated. Engineering productionizes the model and owns the data pipeline dependencies it needs to keep running. Sales asks for signals like churn-risk accounts and needs the reasoning explained simply enough to act on in a client conversation. Support surfaces data quality issues found through customer escalations and sometimes needs a model's decision explained to a confused customer.

Deliverables

Exploratory analysis write-ups, A/B test designs and their readouts, a deployed model with monitored production performance, and a specific recommendation that's expected to move a KPI (key performance indicator).

Tools and timelines

Primarily SQL and Python, a notebook environment for exploration, an experimentation platform, and a feature pipeline maintained jointly with engineering. A pure exploratory question might take a few days; a model that reaches production typically takes several weeks to a couple of months once validation and integration are included.

Worked example

If trial-to-paid conversion softens, I'd propose an uplift model, meaning a model that predicts the incremental effect of an action on a specific individual (who converts BECAUSE of the nudge, not just who was already likely to convert anyway), identifying which trial users respond best to a specific nudge. Concretely: pull and clean the relevant event data, build and validate the model offline against a held-out set (data deliberately set aside from training and never shown to the model, used only to test how it performs on data it hasn't seen), agree with product on the target metric and a realistic timeline, and validate the real effect with an A/B test before rolling it out broadly.

Trade-offs and pitfalls

The most common failure mode is the "it works in the notebook" trap, an analysis or model that never gets shipped or trusted by stakeholders moves nothing, no matter how sound the methodology. There's also a real tension between chasing a short-term metric bump and protecting longer-term product health; a model that boosts one week's numbers by encouraging behavior that hurts retention later is a net loss.

Before I join a new team, I want answers in four areas: what I'll be measured on, what data and tooling I'll inherit, how the team actually operates day to day, and what's already on fire. Below are twelve questions: ten across those four areas, plus two on team structure, each with why it matters.

Metrics and KPIs (key performance indicators, the specific numbers used to judge whether the work is succeeding)

1. "What are the two or three metrics you'll actually judge my first six months against?" Aligns my effort with what's measured, rather than what merely looks good.
2. "Is there an existing dashboard or report showing the current baseline for those metrics?" Grounds any target I commit to in reality instead of a guess.
3. "Which of those metrics are actually inside my control, and which can I only influence?" A metric I am judged on but cannot move without three other teams agreeing is a target I should renegotiate now, not at the review.

Data and tooling

4. "What's the primary stack, and where does data quality or reliability break down today?" Surfaces known landmines before I inherit them silently.
5. "Which tools are the actual source of truth versus legacy systems people still half-use?" Avoids wasting early weeks mastering a tool the team is already abandoning.

Process and rhythms

6. "What's the meeting cadence, standups, planning, one-on-ones, and who actually attends?" Sets realistic expectations for calendar load and where decisions get made.
7. "How do decisions get made when the team disagrees, and who has the final call?" Clarifies decision rights (who is actually authorized to decide) before I need that answer under pressure.

Immediate problems

8. "What's the one thing that's been broken or blocked the longest?" Reveals the real priority behind the polished job description.
9. "What did the person previously in this role do well, and where did they struggle?" Sets a realistic bar and flags landmines without me having to rediscover them the hard way.
10. "What has been tried on that problem already, and why did it not stick?" Stops me from proposing in week two the exact fix that failed last year for reasons nobody wrote down.

Team and reporting

11. "Who do I report to day-to-day versus who formally evaluates my performance?" These can differ on matrixed teams (a structure where you have both a functional manager and a separate day-to-day project or product lead, each with real influence over your work), and the gap matters.
12. "How many people are on the team, and how is work currently split among them?" Shapes whether I'm stepping into an ownership role or a support role.

Trade-offs and a closing practice

Ask the hiring manager and a team lead separately where you can; their answers to "what's broken" often diverge, and the gap is informative. Don't ask questions the job posting already answered, and prioritize the ones most likely to change your decision or your first 90 days. A strong habit once you land the role: turn the answers into a short written role-expectations document you both sign off on, so priorities don't quietly drift a month later.