Cloud Data Platforms and Managed Services Questions

Direct answer. At 10,000 events/second averaging 2KB each, you need 20 shards, and the binding constraint is throughput, not record count.

Structured elaboration. Each Kinesis shard has two independent write limits, and whichever one you hit first is the one that matters:

$\text{limit}_1 = 1{,}000 \text{ records/sec}$
$\text{limit}_2 = 1 \text{ MiB/sec}$

With a 2KB average record size, the throughput limit binds well before the record-count limit does. Convert the throughput limit into an equivalent records/sec figure at this record size:

$\text{records/sec before hitting the 1 MiB/sec cap} = \frac{1 \times 1024 \text{ KB}}{2 \text{ KB}} = 512 \text{ records/sec per shard}$

Since 512 records/sec is well below the 1,000 records/sec cap, throughput is the binding constraint per shard, not record count. Total required shard count:

$\text{shards} = \left\lceil \frac{10{,}000 \text{ events/sec}}{512 \text{ events/sec per shard}} \right\rceil = 20$

Cross-checking directly in raw bytes: total ingest is $10{,}000 \times 2\text{KB} = 20{,}000 \text{ KB/sec} \approx 19.53 \text{ MiB/sec}$, and $19.53 / 1 = 19.53$, which rounds up to 20 shards, matching the record-based calculation exactly (both approaches must agree, since they measure the same binding constraint two ways).

events_per_sec = 10_000
avg_kb = 2
bytes_per_sec = events_per_sec * avg_kb * 1024
mb_per_sec = bytes_per_sec / (1024 * 1024)
print(f"total throughput: {mb_per_sec:.2f} MiB/sec")  # 19.53 MiB/sec

shard_mb_limit = 1.0
shard_record_limit = 1000
records_per_shard_before_mb_cap = (shard_mb_limit * 1024) / avg_kb
effective_capacity_per_shard = min(records_per_shard_before_mb_cap, shard_record_limit)
print(f"binding per-shard capacity: {effective_capacity_per_shard:.0f} records/sec")  # 512

import math
shards_needed = math.ceil(events_per_sec / effective_capacity_per_shard)
print(f"shards needed: {shards_needed}")  # 20

Running this script prints total throughput: 19.53 MiB/sec, binding per-shard capacity: 512 records/sec, and shards needed: 20, confirming both derivations.

Worked example: scaling shard count without losing data. To scale up, split individual shards (a shard-split operation) rather than tearing down and rebuilding the stream. A split takes one shard and divides its hash-key range into two child shards, each inheriting half the original's capacity; Kinesis keeps the parent shard readable until all its existing records have been consumed, so no data is lost during the transition. Well-behaved consumers using the Kinesis Client Library detect the parent shard's CLOSED status, finish draining it, then automatically discover and start consuming the new child shards, so a properly implemented consumer experiences no data loss and only a brief period of reduced parallelism while it catches up on the closing parent shard. Plan splits ahead of an anticipated growth event (do not wait until you are already throttling), since a split takes a small amount of time to propagate and briefly affects write availability on the shard being split.

Trade-offs and pitfalls. This calculation assumes traffic is spread evenly across shards, but an uneven partition-key distribution can create a hot shard that throttles well before the stream's aggregate 20-shard capacity is reached; monitor per-shard WriteProvisionedThroughputExceeded metrics, not just aggregate throughput, to catch this. Also budget headroom above the bare-minimum 20 shards, since traffic bursts above the 2KB average or above 10,000 events/sec will throttle a stream sized to the exact average; many teams provision 20-30% above the calculated minimum specifically to absorb normal variance without constant resharding.

Direct answer. Migrating BI workloads from an on-premise warehouse to a managed cloud platform, whether the destination is Snowflake or Synapse, requires four coordinated workstreams: schema migration, performance validation, cost governance, and analyst training, sequenced so that regressions are caught before the business depends on the new platform.

Structured elaboration.

• Schema migration. Assess whether to lift-and-shift the existing schema as-is or take the opportunity to re-architect (for example, moving from a heavily normalized on-prem schema to a star schema better suited to the target platform's columnar engine). Lift-and-shift is faster and lower-risk short-term; re-architecture pays off longer-term but extends the migration timeline and adds validation surface area. The conversion surface differs meaningfully by target platform: Snowflake auto-manages physical layout through micro-partitions, so schema conversion is mostly SQL-dialect translation, while a Synapse dedicated SQL pool requires you to explicitly choose a distribution style (HASH, ROUND_ROBIN, or REPLICATE) and a clustered columnstore index per table during conversion, an extra design decision with no Snowflake equivalent that has real performance consequences if left as an afterthought.
• Query-performance regression risk. Query patterns tuned for the old platform's engine (index usage, join strategies, partitioning assumptions) do not automatically translate to the new platform's optimizer. Run the existing production query set against the new platform before cutover and flag any query whose latency regresses beyond an agreed threshold for manual tuning.
• Cost implications. Estimate the new platform's ongoing cost from a realistic sample of production query volume and data growth, not just current data size, since cloud warehouses bill differently (compute-and-storage-separated) than an on-prem warehouse's largely fixed hardware cost; build in a cost-monitoring dashboard from day one so an unexpected spike is caught early rather than discovered on the first invoice. The billing shape also differs by target: Snowflake's virtual warehouses auto-suspend and bill per-second while actually running a query, whereas a Synapse dedicated SQL pool reserves DWU-based capacity that keeps billing continuously until you explicitly pause it, so a Synapse migration needs an operational pause/resume discipline built into the cost plan that a Snowflake migration does not.
• Training. BI analysts accustomed to the on-prem tool's quirks (specific SQL dialect, specific performance characteristics, specific administrative workflows) need hands-on training before cutover, not just documentation; a short-format transition period running both platforms side by side lets analysts validate their own familiar reports against the new platform before it becomes the system of record. Budget training time differently by target: analysts coming from an on-prem SQL Server-flavored warehouse typically find Synapse's T-SQL dialect and SSMS-style tooling closer to their existing habits, while a move to Snowflake's SQL dialect and Snowsight interface is a bigger syntax and tooling jump, regardless of which platform is technically the better fit.
• Measuring success. Define success criteria before migration starts: query-latency parity (or improvement) across the core report set, cost within a defined budget band, and analyst-reported confidence in the new platform's numbers matching the old one during a parallel-run period.

Worked example. A BI team migrating from an on-prem warehouse ran the top 50 most-frequently-executed reports against the new platform during a two-week parallel-run period before cutover, comparing both results (to catch silent correctness regressions from a schema or join-logic difference) and latency (to catch performance regressions). Three reports showed silent numeric discrepancies traced to an implicit NULL-handling difference between the two platforms' aggregate functions, caught and fixed before any analyst relied on the new platform's numbers in a real decision; two reports were meaningfully slower on the new platform, both attributable to a missing distribution/clustering key equivalent to what the on-prem indexes had provided, and were fixed by adding the new platform's native performance features before cutover rather than after analysts started complaining.

Trade-offs and pitfalls. The most damaging failure mode in this kind of migration is a silent correctness regression, not a performance regression, since a slow report is annoying but a wrong number that nobody catches erodes trust in the whole platform once discovered. Budget the parallel-run comparison specifically to catch numeric discrepancies, not just latency, and do not declare the migration successful until analysts have validated their own familiar reports against the new platform themselves.

Direct answer. Compare Snowflake, BigQuery, and Redshift on five axes: architecture model (how compute and storage separate), concurrency handling, pricing model, storage format support, and operational overhead. There is no universal winner; the right choice depends on your team's existing cloud, your query concurrency profile, and how predictable your workload is.

Structured elaboration.

Three platform-specific units in that table are worth defining plainly, since the question is explicitly asking about concurrency handling and pricing: a Snowflake compute credit is its per-second billing unit for warehouse compute, so a bigger or longer-running warehouse simply burns credits faster. A BigQuery slot is the platform's unit of parallel query-processing capacity; the "shared slot pool" is the pot of these units Google draws from to run your query, and a slot reservation just reserves a guaranteed number of them for you instead of sharing the pool with every other BigQuery customer. A Redshift WLM (Workload Management) queue is a named lane that routes a query to a specific, bounded share of the cluster's memory and concurrency; hitting a concurrency limit means that particular queue's lane is full, not that the whole cluster is out of capacity.

Worked example. For a fast-growing team with roughly 500 analysts running around 10,000 BI queries a day against a 10TB active dataset, concurrency handling is the deciding factor more than raw performance: Snowflake's ability to spin up independent warehouses per team or workload avoids one group's heavy queries starving another's dashboard, and its per-second billing means idle warehouses cost nothing when auto-suspended. BigQuery is an equally strong fit if the team is already GCP-native and wants zero cluster management, especially if the query pattern is bursty rather than continuously heavy, since on-demand pricing avoids paying for idle capacity at all. Redshift becomes the stronger choice when the workload is large and steady enough that reserved/provisioned capacity is cheaper than pay-per-use, or when the team already has deep AWS-ecosystem integration (IAM, Glue, Lake Formation) that reduces the value of switching platforms. At petabyte scale with a high-concurrency BI user base, total cost of ownership becomes the deciding axis rather than raw price-per-query, since the storage-versus-compute separation and auto-scaling behavior of Snowflake or BigQuery tend to avoid the manual capacity-planning overhead that a large provisioned Redshift cluster requires, while a spiky, bursty query pattern specifically favors either platform's auto-scaling over a fixed-size cluster.

Trade-offs and pitfalls. Benchmarking these platforms fairly is hard: comparing default settings without tuning distribution/clustering keys, using a dataset too small to expose real concurrency behavior, or ignoring egress and data-transfer cost between your existing systems and the new platform will all produce misleading conclusions. Vendor lock-in is real in all three directions (proprietary SQL extensions, proprietary storage formats, ecosystem integrations), so weigh switching cost alongside today's price and performance, not just today's benchmark numbers.

Direct answer. A serverless query service (Athena, BigQuery on-demand) charges per byte scanned with no infrastructure to manage, which fits unpredictable, bursty workloads well; a provisioned warehouse (Redshift, a dedicated Synapse pool) reserves compute you pay for continuously, which fits steady, high-volume workloads better. For a startup with an unpredictable query pattern and a limited budget, the serverless model is usually the safer starting point.

Structured elaboration.

• Cost predictability. Serverless bills scale with usage, so a quiet month costs almost nothing, but an unexpectedly large or inefficient query can produce a cost spike with little warning. Provisioned capacity costs the same every month regardless of usage, which is predictable but wasteful if usage is low or spiky.
• Performance on large joins. A provisioned warehouse can be tuned (partitioning, sort/distribution keys, dedicated compute) to make large joins consistently fast. A serverless engine reading raw files typically re-scans the full dataset for every large join unless the data is well-partitioned, so performance is more variable and depends heavily on how the underlying files are laid out.
• Concurrency. Serverless engines generally scale to many simultaneous queries without you doing anything, since there is no shared cluster to contend for. A provisioned warehouse has a fixed pool of compute, so concurrent heavy queries can queue behind each other unless you have configured workload management.
• Operational burden. Serverless requires no cluster sizing, patching, or pause/resume decisions. A provisioned warehouse requires someone to right-size the cluster, monitor utilization, and decide when to scale up or down.

Worked example. A startup with three analysts running a handful of exploratory queries a day against a dataset that grows unpredictably should start serverless: at low query volume, the pay-per-byte-scanned cost is a fraction of what even the smallest provisioned cluster would cost sitting idle most of the day, and there is no capacity-planning burden for a two-person data team to carry. If that same startup grows to have dozens of analysts running the same set of dashboard queries hundreds of times a day against a stable, well-understood dataset, the calculus flips: a provisioned warehouse, with its data laid out and indexed specifically for those repeated queries, becomes cheaper per query and gives more predictable dashboard latency than continuing to pay per byte scanned on every refresh.

Trade-offs and pitfalls. The most common mistake is staying on the serverless model well past the point where usage has become steady and repetitive, since at high, predictable volume, provisioned capacity is almost always cheaper. The opposite mistake is over-provisioning a warehouse for a startup's earliest, lightest workload, which locks in cost the team does not yet need. Revisit the decision as usage grows rather than treating the initial choice as permanent; many teams end up running both, serverless for exploration and new datasets, provisioned for the small set of queries that run on a predictable, heavy schedule.

Direct answer. Amazon Kinesis Data Streams is a fully managed, AWS-native streaming service with no cluster to run; a self-managed Kafka cluster on EC2 (or the managed MSK alternative) gives you the real Kafka protocol and ecosystem at the cost of more operational responsibility (self-managed) or a closer-to-Kafka managed option (MSK). For a new product with no existing Kafka dependency, Kinesis is usually the simpler starting point; MSK becomes the better choice once Kafka-specific ecosystem tools are required.

Structured elaboration.

• Scalability. Kinesis scales by adding shards (manually or via On-Demand mode's automatic scaling); Kafka scales by adding brokers and rebalancing partitions, which is more powerful at extreme scale but requires more expertise to execute safely, whether self-managed or via MSK.
• Latency. Both offer low, typically sub-second, end-to-end latency for well-provisioned streams; Kafka's design historically achieves slightly lower tail latency at very high throughput, though the gap has narrowed as Kinesis has matured.
• Operational burden. Kinesis has zero cluster operations. Self-managed Kafka on EC2 means your team owns broker provisioning, patching, partition rebalancing, and ZooKeeper/KRaft cluster health. MSK removes the cluster-operations burden while keeping the Kafka protocol.
• Durability guarantees. Both replicate data across multiple availability zones by default; the specific replication factor and consistency settings differ and should be verified against your durability requirements rather than assumed identical.
• Ecosystem tooling. Kafka's ecosystem is extensive and mature: Kafka Streams is a library for processing data as it continuously flows through Kafka, Kafka Connect is a set of pre-built connectors for moving data in and out of Kafka without hand-written integration code, ksqlDB lets you write SQL directly against a stream instead of custom code, and Schema Registry is a central service that enforces and versions the record format producers and consumers agree on. Kinesis has its own, smaller and AWS-specific tooling: Kinesis Data Analytics runs SQL or Flink-based processing directly against a stream, and the Kinesis Client Library (KCL) is a library that helps a consumer application reliably read from every shard in a stream.
• Cost. Kinesis is billed per shard-hour and per data volume; self-managed Kafka's cost is dominated by EC2 instance and storage cost, which can be cheaper at very large, steady scale for a team with the expertise to run it efficiently; MSK sits between the two, priced for the managed convenience.
• Vendor lock-in. Kinesis is AWS-proprietary; both self-managed Kafka and MSK use the open Kafka protocol, so application code is portable to any Kafka-compatible service, including a different cloud, with far less rework than migrating off Kinesis would require.

Worked example. A new real-time analytics product built entirely on AWS with no existing Kafka Connect pipelines or Kafka Streams applications to preserve should default to Kinesis: the team gets AWS-native integration (IAM, CloudWatch, Lambda triggers) with zero cluster management, which matters most in a new product's early stage when engineering time is scarcer than infrastructure cost. If the same team later needs multi-cloud portability, or needs to adopt an existing Kafka Connect connector for a source system with no Kinesis equivalent, migrating to MSK preserves the Kafka ecosystem without taking on full self-management.

Trade-offs and pitfalls. The most consequential trade-off is lock-in: choosing Kinesis for its simplicity is a legitimate call for a new product, but revisit that decision explicitly if the product later needs to run on another cloud or needs Kafka-specific tooling, since Kinesis application code does not port to Kafka without a rewrite of the client integration layer.