Role Understanding and Success Criteria Questions

Situation: As a solutions architect supporting a large prospect, the sales lead handed me a proposal request with a single high-level success metric: “reduce infrastructure costs.” No baseline, timeframe, or constraints were provided, and the customer had mixed priorities (performance, uptime, compliance).

Task: I needed to translate that vague metric into measurable, agreed-upon success criteria so the solution would meet both the customer’s business goals and our delivery capabilities — without delaying the sales cycle.

Action:

• I scheduled a 45-minute scoping call with the sales lead, the customer’s IT manager, and a product owner to surface assumptions.
• I asked targeted questions to establish baseline (current monthly infra spend, peak load, SLAs, compliance needs) and timeframe (6 vs 12 months) and to prioritize trade-offs (cost vs latency vs resiliency).
• I proposed a small decision document (one page) that defined 3 measurable KPIs: % reduction in monthly infra cost (baseline), 99.95% availability, and <10% latency increase for peak transactions. I included acceptance criteria and a RACI for who validates metrics.
• I aligned with engineering on feasibility and created a lightweight monitoring plan (cloud cost dashboards + synthetic latency checks) to track progress post-deployment.
• I circulated the doc, requested sign-off from the customer and sales lead, and built the KPIs into the proposal and contract milestone payments.

Result: The stakeholders signed off within 72 hours, which kept the deal on schedule. By defining measurable KPIs and a monitoring plan we avoided scope creep during implementation. After deployment we hit a 22% monthly infrastructure cost reduction within three months while maintaining 99.97% availability — exceeding the agreed targets. The clarity also sped approval cycles for future engagements with the same customer.

High-level approach: build a reusable, event-driven measurement platform that links SA activities to downstream commercial and customer outcomes using a multi-touch, hybrid attribution model (weighted touch + causal inference for lift).

Attribution model:

• Multi-touch weighted model (assign weights by stage and impact): discovery/demo = 0.2, solution design/spec = 0.4, POC/proof = 0.3, architecture review/enablement = 0.1. Adjust weights per portfolio via uplift experiments.
• Complement with causal methods: difference-in-differences or propensity-score matched cohorts and A/B or holdout experiments on offers/POCs to measure incremental revenue and churn reduction.

Data sources:

• CRM (opportunities, stage timestamps, SA owner, deal value, close dates)
• Finance (bookings, billing, ARR, discount levels, payment terms)
• Telemetry & usage (product usage, feature adoption, latency/uptime) tied to account IDs
• Engagement logs (meeting invites, emails, artifact deliveries, POC start/stop)
• Customer success (NPS, support tickets, renewals)
• Identity mapping (account and contact resolution)

Transformation logic / pipeline:

• Ingest via CDC/streaming into a canonical event schema (account_id, contact_id, event_type, timestamp, sa_id, metadata).
• Enrich: join CRM opps to account events; attach finance outcomes; map telemetry sessions to accounts.
• Sessionize SA activities to opportunity lifecycle windows (lookback and lookforward rules).
• Compute attribution: allocate opportunity value across SA events using weight functions, then aggregate per SA and team.
• Calculate incremental metrics via treatment/control cohorts and time-series models; produce ROI = (attributed incremental revenue − SA cost) / SA cost.

Validation & credibility:

• Data quality checks: row counts, referential integrity, duplicate detection, outliers
• Reconciliation: weekly rollups compared to CRM and GL (finance) aggregates with tolerance thresholds
• Sensitivity analysis: vary weights, windows; publish ranges and confidence intervals
• Run periodic randomized pilots/holdouts for high-impact offers to estimate causal lift
• Auditability: store raw events + transformation deterministic SQL/ETL with versioning; expose lineage in a catalog.
• Governance: stakeholder reviews (finance, sales, CSM) and monthly publication with methodology notes and change logs.

Operationalization:

• Dashboards: per-SA, per-product, per-region ROI and lift metrics; drill-to-raw-events
• Alerts when attribution deviates from expected or data freshness fails
• Continuous improvement: use experimental results to update attribution weights and models.

Requirements & constraints:

• Executive reviews monthly (regional) and quarterly (global); audience: regional execs, global SA lead, Sales/VPOPs.
• Must be comparable across regions, allow region-specific context, scalable to 10+ regions, low ceremony.

Standardized agenda (60–75 min):

1. Opening (5 min) — objective, key decisions requested.
2. Metrics snapshot (10 min) — 3 core KPIs + trend.
3. Top deals pipeline review (15 min) — 3–5 deals: solution status, risks, action owner, ETA.
4. Operational risks & escalations (10 min) — blockers needing exec attention.
5. Capability & enablement (10 min) — skills gaps, key hires, tooling.
6. Decisions & actions (10 min) — assign owners, deadlines, follow-ups.
7. Parking-lot Q&A (5 min).

Core metrics (definitions & calculation):

• SA Win Rate = #Deals with SA-approved architecture accepted and closed-won / #Deals SA-supported in period. (Include only opportunities with SA engagement logged in CRM.)
• Time-to-Architecture (TTA) = median(hours) from SA engagement start to delivery of CTO-level architecture doc. (Measure per deal; exclude redesign rework >2 iterations flagged.)
• Escalation Resolution SLA = % escalations resolved within target window (P1=48h, P2=5 business days) = (#resolved within SLA / total escalations) *100.

Evidence required:

• CRM exports (opportunity ID, stage, SA owner, close date)
• Architecture docs (timestamped, stored in doc repo) and acceptance email or CR approval
• Timestamped tickets / Slack threads for escalations; audit trail showing action and resolution
• Pipeline slides linked to source records

Escalation rules:

• Triage: SA lead evaluates within 4 business hours.
• Auto-escalate to Regional Exec when P1 or if impacted revenue > $X or TTA > SLA by 50%.
• Global escalation to VP SAs if cross-region impact, >2 regions affected, or unresolved >7 days.
• Each escalation must include: impact statement, evidence links, proposed remediation, owner, due date.

Governance model:

• Global: Standard metric definitions, measurement scripts, data sources, cadence, and escalation thresholds maintained by Global SA Ops.
• Regional: Allowed contextual notes, up to 2 region-specific KPIs, and discretionary remediation tactics. Regions must map region KPIs to global schema and publish translation document.
• Compliance: Monthly automated data quality checks; quarterly audits of 10% of deals for evidence integrity.
• Tooling: Central dashboards (BI) with per-region filters; templates for slides and evidence links.
• Change control: Any change to metric definitions requires 30-day notice, pilot in 2 regions, and global approval.

Why this works:

• Consistent data & evidence preserve comparability; escalation tiers align attention to severity; regional flexibility preserves local context without breaking global aggregation.

Hypothesis:

• Primary: Providing standardized Solutions Architecture (SA) interventions (templates + reference architectures + prebuilt collateral) reduces average sales cycle time and increases win rate compared to current ad-hoc SA support.

Experiment design:

• Unit of randomization: sales opportunity (deal) stratified by segment (ARR band), product, and region.
• Experimental group: opportunities where AE + SA must use the standardized kit (template, reference architecture, checklist, 1-hour templated workshop).
• Control group: opportunites receive “business-as-usual” SA support.

Sample size considerations:

• Determine minimum detectable effect (MDE): e.g., 10% reduction in cycle time and 5 percentage-point lift in win rate. Use historical mean cycle time and win rate, desired power 80%, alpha 0.05. For win rate (binary) apply two-proportion sample size; for cycle time (continuous, often right-skewed) use log-transform — compute n per arm accordingly and inflate for expected loss-to-follow-up.

Key metrics:

• Primary: time from opportunity creation → closed-won/closed-lost (median & mean), win rate (closed-won / closed-decisions).
• Secondary: time to first technical deliverable, number of SA touchpoints, proposal rework rate, deal size (ACV), customer satisfaction (post-win NPS).
• Safety/quality: % of technical escalations post-close.

Analysis approach:

• Pre-specify primary endpoints and analysis window.
• For win rate: two-proportion z-test or logistic regression (adjusted for stratification covariates) to estimate odds ratio and adjusted risk difference.
• For cycle time: compare log-transformed times using t-test or linear regression; report geometric means and median differences via nonparametric tests (Wilcoxon) if skewed.
• Use survival analysis (Kaplan-Meier + Cox proportional hazards) to handle censored opportunities (still open at analysis cutoff).
• Adjust for multiplicity if testing both endpoints (e.g., hierarchical testing or Bonferroni).
• Report effect sizes, 95% CIs, and p-values. Conduct subgroup analyses (enterprise vs. SMB) pre-specified.

Potential confounders and mitigations:

• Deal complexity, salesperson skill, seasonality, marketing campaigns, pricing changes, product roadmap shifts. Mitigate via stratified randomization, regression adjustment, and blocking by time windows.
• Contamination: SAs learning templates and applying to control deals—mitigate by clear assignment rules, logging SA actions, and possibly cluster-randomizing by SA for a sensitivity analysis.
• Selection bias: ensure randomization occurs at opportunity creation and is enforced by CRM hooks.
• Operational constraints: run long enough to capture sales cycles across segments; monitor interim metrics but avoid peeking without pre-specified stopping rules.

Interpretation:

• Combine statistical and practical significance: even modest win-rate lifts at scale can justify adoption if quality metrics remain stable.
• If positive, plan rollout with training, feedback loop, and A/B follow-ups for iterative improvements.

Requirements (implicit): executive-summary view of Solutions Architect (SA) impact across portfolio—concise, trustable, aligned to revenue/velocity/quality/risks; supports quarterly reviews and sales enablement.

Key metrics (6–8) with visualizations and cadence:

1. Revenue Influenced (ARR/TCV) — KPI card + trend line (quarterly, daily roll-up)
2. Win Rate with SA Engagement (%) — stacked bar by deal size and funnel stage (weekly aggregate, refresh daily)
3. Deal Velocity (average days from SA engagement → contract) — box plot + median trend (weekly)
4. Architecture Reuse Rate (%) — donut + time series of reused patterns vs bespoke (monthly)
5. Technical Risk Score Distribution — heatmap by account (severity x probability) (weekly)
6. Post-deployment Defects attributable to design (%) — bar chart by release + cumulative trend (monthly)
7. Customer Satisfaction (technical NPS) — KPI + trend + top feedback themes (quarterly)
8. Resource Utilization / Capacity (SA hours per active deal) — stacked area by team (weekly)

Minimal data lineage to ensure credibility:

• Source systems mapped per metric (CRM: deals, ARR; Time tracking: SA hours; Ticketing: defects; Surveys: tNPS; Repo/artifact store: architecture templates)
• Transformation rules (e.g., how “SA engagement” is defined — first SA touchpoint in CRM; risk scoring algorithm with weights)
• Refresh schedule and latency per source
• Owner and SLAs for each data source
• Versioned metric definitions and last-change timestamp visible on dashboard

High-level architecture: ETL from sources → staging → metric layer (with documented transformations) → BI layer (dashboard with filters). Include audit links to original records for executive drill-downs.