Requirements (clarify)
- Functional: load 3rd-party ETL transformation plugins dynamically, run per-job transforms, support hot-reload.
- Non-functional: secure sandboxing, resource limits (CPU, memory, I/O), validation, observability, backward compatibility for plugin API versions.
High-level architecture
ETL Orchestrator ←→ Plugin Manager ←→ Plugin Runtimes (isolated processes/containers)
Plugin Registry (metadata, versions, signatures) + CI validation pipeline + Metrics/Alerting
Key components
- Plugin package format: Python wheel + manifest (plugin.yml) with name, version, api_version, entrypoint, required_permissions, schema.
- Plugin Manager: installs, validates signatures, verifies manifest, schedules plugin into a runtime.
- Plugin Runtime (sandbox): per-plugin process or lightweight container (gVisor, Firecracker, Docker) with constrained cgroups, seccomp, chroot; communicates over gRPC/HTTP with mutual TLS.
- API Gateway: stable RPC surface exposing transform(input, context) and lifecycle hooks (init, shutdown, health).
- Registry + Compatibility Layer: maps api_version to adapter shims.
API sketch (Python client used by runtimes)
class PluginInterface(Protocol):
def init(self, config: dict) -> None: ...
def transform(self, batch: List[dict], ctx: dict) -> List[dict]: ...
def shutdown(self) -> None: ...
Manifest example (plugin.yml)
name: "normalize-addr"
version: "1.2.0"
api_version: "v2"
entrypoint: "normalize:Normalize"
permissions:
network: false
file_system: read:/etc/geo
cpu: 1
memory_mb: 256
Validation steps
- Static checks: manifest schema, required entrypoint, type hints, banned modules list.
- Security checks: signature verification (publisher key), dependency vulnerability scan (Snyk/OSV), ensure no native-code wheels unless allowed.
- Behavioral tests: run unit integration tests in CI using a test harness and sample data; enforce determinism/timeouts.
- API compatibility: use automated adapter tests against supported api_versions.
Sandboxing & resource enforcement
- Run each plugin in isolated process/container with:
- cgroups for CPU and memory limits
- seccomp to whitelist syscalls
- Linux namespaces / no network by default; grant per-manifest permissions
- Read-only filesystem mounts; sensitive mounts forbidden
- Timeout per call and execution watchdog; kill/restart policy
- Communications via protobuf/gRPC over unix socket; only structured data allowed (no pickles).
Hot-reload
- Orchestrator watches registry; when new version arrives:
- Validate and stage
- Start new runtime with new version and run health checks
- Drain existing tasks, switch traffic to new instance
- Rollback on failure
- Support in-process lightweight reload only for trusted internal plugins; otherwise use process swap.
Backward compatibility
- Semantic versioning + api_version field.
- Compatibility Layer: adapters that translate old api -> new api (auto-generated stubs where possible).
- Deprecation policy: maintain vN for at least X releases; log warnings in orchestration.
- Feature negotiation at init time; if not compatible, orchestration refuses load.
Observability & governance
- Per-plugin metrics (latency, error rate, memory/CPU), structured logs, traces (OpenTelemetry).
- RBAC for plugin publishing, cryptographic signing, audit logs.
- Quota enforcement per-tenant.
Trade-offs
- Process/container isolation increases complexity and cost but is necessary for security. Lightweight processes with seccomp suffice for internal-only plugins.
- Binary/native extensions are risky—prefer pure-Python or explicit approval.
Example flow
- Developer publishes wheel + manifest + signature to registry.
- CI runs static/security/tests; registry marks it certified.
- Orchestrator pulls plugin, Plugin Manager validates, starts sandboxed runtime.
- ETL job invokes transform via gRPC; runtime enforces limits and returns results.
- New version triggers blue-green swap with health checks and metrics-based rollback.
This design balances extensibility, security, operational safety, and developer ergonomics for production ETL pipelines.
