Build Spark pipelines you can trust before they run.
Draw the DAG on a visual canvas or write YAML — both compile to the same typed Scala engine. Every step is validated, previewed and measured before a single Spark core is spent.
Canvas and YAML, one engine
Every node on the canvas is a typed step — JDBC and Kafka ingestion, JSON/Avro parsing, SQL transforms, variables and UDFs. Prefer code? The same pipeline is plain YAML, reviewable in a pull request.
Predictive dry-run
Sample-execute each step against the live source via Trino before running Spark. You see row estimates, output schema and a data preview per node — schema mistakes die on the canvas, not in production.
Run time-machine
Scrub through any past run on its step graph: nodes light up in execution order with their record and byte counters, at 1×, 2× or 4× playback. Debugging a run becomes watching it.
Run data-diff
Compare two runs by what changed in the data — rows, bytes, duration and per-step deltas — not just what changed in the code. Regressions surface as numbers, not surprises.
Governed recovery
On a failed run, deterministic diagnostics classify stored evidence and calculate downstream impact. When the AI repair service is configured, a complete patch is validated on an isolated branch, applied as a version, verified, and rolled back on failure.
Real metrics per step
Rows, bytes, shuffle, memory peaks and wall-clock per step, pulled from the live Spark API during the run. No estimates, no sampling — the actual numbers.