Cloud Data Platforms and Managed Services Questions

Direct answer. Open-source distributed query engines (Spark, Presto/Trino) give you flexibility, no per-vendor licensing cost, and the ability to run anywhere; managed cloud data warehouses (Snowflake, BigQuery) give you a fully optimized, low-maintenance SQL experience at the cost of some flexibility and a vendor relationship. Choose based on workload shape: warehouses win for ad-hoc SQL and dashboards, open-source engines win when you need programmatic, mixed-language processing or must run identically across environments.

Structured elaboration.

Worked example. A team building nightly batch ETL jobs that primarily filter, join, and aggregate structured data can often express the whole pipeline as SQL running inside a managed warehouse (the ELT pattern), which avoids maintaining a separate Spark cluster entirely and keeps the transform logic close to where the data already lives. A team that needs to run custom Python machine-learning feature transforms, join against unstructured or semi-structured data at large scale, or run the exact same pipeline logic on-premises and in the cloud for portability reasons is better served by Spark or Presto/Trino, since a SQL-only warehouse cannot easily express arbitrary code and open-source engines run identically regardless of where the compute lives. Interactive BI dashboards belong on the managed warehouse in almost every case: Presto/Trino can serve interactive SQL too, but matching a managed warehouse's concurrency and caching behavior requires you to build and operate that tuning yourself.

Trade-offs and pitfalls. A common mistake is defaulting to Spark for everything because a team is comfortable with it, even when the workload is simple, SQL-expressible ETL that a warehouse's native transform capability would handle with far less operational overhead. The opposite mistake is trying to force complex, multi-language, or streaming-heavy processing into warehouse SQL, which usually produces convoluted, hard-to-maintain queries. Maintenance burden compounds this: a self-run Spark or Presto/Trino cluster requires ongoing tuning and version upgrades that a managed warehouse eliminates entirely, so factor the team's appetite for that ongoing operational work into the decision, not just which engine is technically capable of the workload.

Direct answer. At 10,000 events/second averaging 2KB each, you need 20 shards, and the binding constraint is throughput, not record count.

Structured elaboration. Each Kinesis shard has two independent write limits, and whichever one you hit first is the one that matters:

$\text{limit}_1 = 1{,}000 \text{ records/sec}$
$\text{limit}_2 = 1 \text{ MiB/sec}$

With a 2KB average record size, the throughput limit binds well before the record-count limit does. Convert the throughput limit into an equivalent records/sec figure at this record size:

$\text{records/sec before hitting the 1 MiB/sec cap} = \frac{1 \times 1024 \text{ KB}}{2 \text{ KB}} = 512 \text{ records/sec per shard}$

Since 512 records/sec is well below the 1,000 records/sec cap, throughput is the binding constraint per shard, not record count. Total required shard count:

$\text{shards} = \left\lceil \frac{10{,}000 \text{ events/sec}}{512 \text{ events/sec per shard}} \right\rceil = 20$

Cross-checking directly in raw bytes: total ingest is $10{,}000 \times 2\text{KB} = 20{,}000 \text{ KB/sec} \approx 19.53 \text{ MiB/sec}$, and $19.53 / 1 = 19.53$, which rounds up to 20 shards, matching the record-based calculation exactly (both approaches must agree, since they measure the same binding constraint two ways).

events_per_sec = 10_000
avg_kb = 2
bytes_per_sec = events_per_sec * avg_kb * 1024
mb_per_sec = bytes_per_sec / (1024 * 1024)
print(f"total throughput: {mb_per_sec:.2f} MiB/sec")  # 19.53 MiB/sec

shard_mb_limit = 1.0
shard_record_limit = 1000
records_per_shard_before_mb_cap = (shard_mb_limit * 1024) / avg_kb
effective_capacity_per_shard = min(records_per_shard_before_mb_cap, shard_record_limit)
print(f"binding per-shard capacity: {effective_capacity_per_shard:.0f} records/sec")  # 512

import math
shards_needed = math.ceil(events_per_sec / effective_capacity_per_shard)
print(f"shards needed: {shards_needed}")  # 20

Running this script prints total throughput: 19.53 MiB/sec, binding per-shard capacity: 512 records/sec, and shards needed: 20, confirming both derivations.

Worked example: scaling shard count without losing data. To scale up, split individual shards (a shard-split operation) rather than tearing down and rebuilding the stream. A split takes one shard and divides its hash-key range into two child shards, each inheriting half the original's capacity; Kinesis keeps the parent shard readable until all its existing records have been consumed, so no data is lost during the transition. Well-behaved consumers using the Kinesis Client Library detect the parent shard's CLOSED status, finish draining it, then automatically discover and start consuming the new child shards, so a properly implemented consumer experiences no data loss and only a brief period of reduced parallelism while it catches up on the closing parent shard. Plan splits ahead of an anticipated growth event (do not wait until you are already throttling), since a split takes a small amount of time to propagate and briefly affects write availability on the shard being split.

Trade-offs and pitfalls. This calculation assumes traffic is spread evenly across shards, but an uneven partition-key distribution can create a hot shard that throttles well before the stream's aggregate 20-shard capacity is reached; monitor per-shard WriteProvisionedThroughputExceeded metrics, not just aggregate throughput, to catch this. Also budget headroom above the bare-minimum 20 shards, since traffic bursts above the 2KB average or above 10,000 events/sec will throttle a stream sized to the exact average; many teams provision 20-30% above the calculated minimum specifically to absorb normal variance without constant resharding.

Direct answer. A fully managed streaming service (AWS MSK Serverless, Kinesis Data Streams, Confluent Cloud, GCP Pub/Sub) removes cluster operations entirely at the cost of some flexibility and potential vendor lock-in; a self-managed Kafka cluster on VMs or Kubernetes gives full control and avoids per-message pricing at the cost of running the cluster yourself. Recommend the managed option unless the team has genuine Kafka-specific requirements and the operational capacity to run it well.

Structured elaboration. Five decision criteria:

1. Operational overhead. Managed services eliminate broker provisioning, patching, and rebalancing operations; self-managed Kafka requires a team that understands partition rebalancing, ISR (in-sync replica) management, and broker capacity planning.
2. Feature parity. Managed services vary in how closely they match Kafka's actual protocol and ecosystem. Kinesis and Pub/Sub are Kafka-like but not Kafka-compatible, so existing Kafka client code and tooling (Kafka Streams, a library for processing data as it continuously flows through Kafka; ksqlDB, which lets you write SQL directly against a stream instead of custom code; Kafka Connect, a set of pre-built connectors for moving data in and out of Kafka without hand-written integration code) will not work unchanged; MSK and Confluent Cloud run real Kafka, preserving full ecosystem compatibility.
3. Ordering and delivery guarantees. All of these services provide per-partition (or per-shard) ordering, but exactly-once semantics and cross-partition ordering guarantees differ in detail between services, so verify the specific guarantee against your correctness requirements rather than assuming they are interchangeable.
4. Scaling. Managed services generally auto-scale (Kinesis On-Demand mode, Pub/Sub) or scale with a support ticket and a few clicks (MSK); self-managed Kafka scaling requires manually adding brokers and rebalancing partitions, which is operationally delicate at high throughput.
5. Cost predictability. Managed streaming is billed per-shard-hour or per-message, which is predictable at moderate volume but can become expensive at very high sustained throughput; self-managed Kafka's cost is dominated by compute and storage infrastructure, which can be cheaper at large, steady scale if the team already has the expertise to run it efficiently.

Worked example. A team building on AWS with no existing Kafka investment and a small platform team should default to Kinesis Data Streams or MSK Serverless: the operational savings alone typically outweigh the marginal cost difference, and neither requires committing to Kafka-specific tooling if the team has no existing dependency on it. A team migrating an existing, large Kafka-based system with deep dependencies on Kafka Streams and Kafka Connect should lean toward MSK or Confluent Cloud rather than Kinesis or Pub/Sub, since switching away from real Kafka would mean rewriting that tooling; in this case "managed vs self-managed" is really "managed Kafka vs self-managed Kafka," a narrower and lower-risk decision than replacing the whole streaming platform.

Trade-offs and pitfalls. The most common mistake is treating "managed streaming" as one interchangeable category: Kinesis and a managed Kafka service solve overlapping problems but are not drop-in replacements for each other's ecosystems. Also verify exactly-once semantics claims against your specific correctness requirements rather than assuming any managed service delivers the same guarantee as a properly configured self-managed Kafka cluster; the details of idempotent producers (a producer mode that stops a retried write from being counted twice if the original acknowledgment was lost) and transactional consumers (consumers that only see a group of related writes once every write in that group has committed, avoiding a partial view mid-transaction) vary by platform.

Direct answer. AWS EMR, AWS Glue, and GCP Dataproc all run managed Spark, but they differ sharply in operational overhead and pricing model: EMR gives the most control over cluster configuration at the highest operational overhead, Glue is the most serverless and hands-off but the least flexible, and Dataproc sits in between with fast cluster startup and strong GCP-ecosystem integration.

Structured elaboration.

Integration with cloud storage and IAM follows each provider's native model: EMR and Glue integrate tightly with S3 and IAM roles; Dataproc integrates tightly with Cloud Storage and GCP IAM. Moving a Spark workload between these services generally requires re-plumbing storage paths and access policies, even though the Spark code itself is largely portable.

Worked example. A team running dozens of small, independent, scheduled nightly ETL jobs (each transforming one table, running for a few minutes) is well served by Glue: the per-job serverless billing avoids paying for idle cluster time between jobs, and there is no cluster to size or manage across dozens of small, independent workloads. A team running a small number of large, long-lived, tightly-tuned Spark jobs, or that needs non-Spark engines like Presto or Flink on the same infrastructure, benefits more from EMR's configurability, accepting the higher operational overhead in exchange for control over exactly how the cluster is tuned. A GCP-native team wanting Spark without EMR's configuration burden, especially for ephemeral, per-job clusters that spin up and tear down automatically, is typically best served by Dataproc.

Trade-offs and pitfalls. A common mistake is choosing Glue for a workload that needs fine-grained Spark configuration tuning (custom JVM flags, specific executor memory layouts) that Glue's serverless model restricts, forcing an awkward workaround. The opposite mistake is running EMR for many small, independent jobs, which pays for cluster provisioning and idle time repeatedly instead of using a serverless model's per-job billing. Match the service to the job shape (many small vs few large, tightly-tuned vs default-configuration-acceptable) rather than standardizing on one service for convenience.

Direct answer. A fair cross-platform benchmark comparing Redshift, Snowflake, and BigQuery needs a dataset large enough to expose real concurrency and I/O behavior, a query set representative of actual production patterns rather than synthetic microbenchmarks, matched compute sizing across platforms, and a documented, repeatable methodology so results can be reproduced and trusted.

Structured elaboration.

• Dataset size. Use a dataset at or near your actual production scale (or a representative fraction with the same skew and cardinality characteristics), since small benchmark datasets fit entirely in cache on all three platforms and mask exactly the I/O and concurrency differences that matter at real scale.
• Sample queries. Draw the query set from actual production query logs, covering a mix of simple point lookups, moderate aggregations, and the heaviest multi-table joins your workload actually runs, rather than a generic industry-standard benchmark that may not reflect your specific access patterns.
• Concurrency profiles. Test at multiple concurrency levels (single query, moderate concurrent load, peak concurrent load matching your busiest observed period), since a platform that wins at low concurrency can lose badly at high concurrency or vice versa, and a single-query benchmark tells you nothing about how the platform behaves under real, contended load.
• Cluster/compute sizing. Match compute cost, not instance count or a vendor's marketing-recommended size, across platforms: normalize on dollar-per-hour spend so the comparison answers "what do I get for the same budget" rather than comparing an oversized cluster on one platform against an undersized one on another.
• Reproducibility. Document exact dataset version, query text, concurrency-generation tooling, and timing methodology; run each test multiple times and report variance, not just a single run's number, since cloud platforms exhibit real run-to-run variance that a single measurement can misrepresent as a stable result.

Worked example. A reproducible cost-per-query benchmark would: load an identical dataset (same row counts, same skew) into all three platforms; select roughly 20-30 queries spanning simple, moderate, and heavy complexity from actual production logs; run each query set at three concurrency levels (1, 10, and 50 concurrent users, chosen to bracket the real observed range) using an open-source load-generation tool configured identically against all three platforms; normalize compute spend to the same dollar-per-hour rate across platforms before comparing; and run the full suite three times on different days to capture variance from each platform's shared-infrastructure noise. Report cost-per-query, p50 and p95 latency, and query-failure rate under load for each concurrency level, since a platform that is cheap per query at low concurrency but degrades sharply at high concurrency tells a very different story than the single-query number alone would suggest.

Trade-offs and pitfalls. The most common benchmarking mistake is testing only at low concurrency with a small dataset, which produces a clean, easy-to-report number that fails to predict real production behavior; the whole point of benchmarking at scale and under contention is to surface exactly the differences that a small, single-query test hides. A second common mistake is normalizing on instance count or a vendor's suggested default size instead of actual dollar cost, which silently favors whichever platform's default sizing happens to be more generous rather than answering the real question of value per dollar spent.