Consultative Discovery and Requirements Gathering Questions

Discovery objective
Identify which ETL components require refactor to meet real-time SLAs while minimizing risk.

1. Stakeholder interviews

• Ask consumers: required end-to-end latency, acceptable staleness, peak throughput, failure tolerances, and use-cases (alerts vs dashboards).
• Talk to engineers about current batch windows, dependencies, and pain points.

2. Telemetry to collect

• Ingest latency, per-job processing time, queue/backlog length, input arrival vs processing rate, downstream consumer wait times, error rates.

3. Risk assessment

• Components with tight coupling, high statefulness, or heavy joins are higher risk to refactor.
• Prioritize by business impact and technical complexity.

4. Prioritization criteria

• Latency gap (current vs required), consumer count, cost to refactor, data criticality.

5. Migration plan & pilot

• Start with low-risk, high-impact streams: implement event-driven ingestion (Kafka), change data capture (Debezium), lightweight stream processing (Flink/Spark Structured Streaming).
• Pilot: build an end-to-end micro-pipeline for one domain, run in shadow-mode parallel to batch, compare outputs.

6. Success metrics

• End-to-end 95th percentile latency, data completeness (% of events processed), error/retry rates, resource cost delta.

This discovery-driven, telemetry-led approach ensures we refactor the right parts, validate with pilots, and measure success before broad migration.

Quick rapport & extracting constraints with non-technical stakeholders

Techniques

• Start on their terms: ask about goals, metrics, and pain points
• Use analogies and avoid jargon; translate constraints into business impact
• Active listening: mirror their language and confirm understanding

Script & phrasing examples

• Opening: "Thanks — can you tell me the single most important decision this data will support?"
• Clarify constraints: "What timelines matter for that decision — minutes, hours, days?"
• Translate constraints: "If we had to choose between fresher data or richer detail, which is more valuable for your team?"

One clarifying question (short script)

• "Can you walk me through the last time this report missed the mark — what happened, what was the consequence, and how quickly did you need it fixed?" (pause, take notes, then recap: "So the key pain was X, and acceptable lag was Y — does that capture it?")

These approaches build trust quickly and surface concrete technical constraints framed in business outcomes.

Open-ended discovery questions (why they matter):

1. What business question are we trying to answer? — Clarifies scope and priority.
2. Who consumes this data and how often? — Drives freshness and access patterns affecting cost.
3. What decisions will change based on this pipeline? — Determines accuracy SLA and auditability.
4. What sources/systems hold the data today? — Reveals integration complexity.
5. Are there compliance or retention constraints? — Shapes storage and retention policy.

Probing technical follow-ups (why they matter):

1. Typical and peak data volume per day? — Impacts choice of processing framework and cost.
2. Schema stability: how often fields change? — Determines need for schema evolution handling.
3. Required SLA for data availability and latency? — Affects streaming vs batch design.
4. Retention period and archival needs? — Influences storage tiering and cost.
5. Are there PII fields needing masking or access controls? — Drives encryption, RBAC, and compliance work.

Each question maps to concrete design/cost decisions (compute sizing, storage tiers, operational complexity).

3-month analytics feature (short project)

• Weekly sprint check-ins (30–60m)
• Bi-weekly demos with stakeholders
• Artifacts: requirements doc, data contract, sample dataset, acceptance tests, deployment checklist
• Checkpoints: end of discovery (data fit confirmed), mid-sprint contract signed, pre-prod acceptance test (owner: data engineer + product), release sign-off (owner: PM)

Cross-quarter sources migration (program)

• Bi-weekly program sync, monthly steering committee
• Artifacts: source inventory, profiling reports, mapping docs, automated data contracts, migration runbook
• Checkpoints: data contract agreed per source (owner: source owner + data steward), pilot ingestion + acceptance tests (owner: infra eng), production cutover with rollback plan (owner: release manager), post-cutover validation & SLA review

Roles: Data engineer (ingest & tests), data steward (contracts), product/owner (acceptance), security/compliance (reviews). Use automated CI for contracts and nightly profiling.

Categories I use:

• Decision-makers: set goals/budget (e.g., PM, Head of Product)
• Approvers: finance/legal/compliance who sign off on spend or data access
• Consumers: analysts, data scientists, BI users who rely on outputs
• Maintainers: SRE/ETL engineers who operate pipelines
• Data owners: source system stewards who control schema and access

Prioritization criteria:

• Impact on KPI and budget control → engage decision-makers early
• Gatekeeping power (approvers/data owners) → must engage before design
• Frequency of consumption and dependency → high-use consumers prioritized for requirements
• Operational burden → involve maintainers to assess run-time costs

Example stakeholder map (simple matrix):

• High influence/high interest: PM (decision-maker), Finance (approver)
• High influence/low interest: Legal (approver)
• Low influence/high interest: Analysts (consumers), DS team
• Low influence/low interest: External vendors

Cadence: revisit after each major milestone or change (requirements finalized, pre-prod sign-off, post-launch), typically every 2–6 weeks depending on project risk. Revisit earlier if scope, compliance, or cost assumptions change.