DE / Done
DataLoom Data Operator Workspace
Overview
Data operator workspace to build, run, and monitor end to end pipelines. Raw data lands in Bronze, is cleaned deterministically into Silver, passes a four rule quality gate, and is promoted to Gold only when fit for use. Every rejected row lands in Quarantine with a reason code, agents only propose while humans approve, and validation rules plus project thresholds are readable in the data contract page.
Challenge
Guaranteeing that bad data never reaches reports: rejected rows need recorded reasons instead of silent drops, promotion needs deterministic reconciliation instead of model opinion, and rule changes need versioning plus human approval instead of silent formula edits.
Result
Shipped a live platform at dataloom.web.id with a demo mode that runs fully in the browser: guided setup, DAG canvas, run monitor with gate checklist, quarantine with reason codes, data contract, agent approval workspace, read only text to SQL, and usage billing. Verified per README: web typecheck and lint pass with 25 of 25 pages built, 320 backend unit tests pass, and a 58 step browser suite passes with zero console errors.
Key Statistics
320 passed, 0 failed
Backend unit tests
58 steps passed, zero console errors
Browser suite
25 of 25 pages, typecheck and lint pass
Web build
33 demo captures in docs/screenshots
Screenshots
Technologies
Frontend
-
Next.js 15 -
React 19 -
TypeScript -
Tailwind CSS - React Flow
- Recharts
Backend
-
FastAPI - SQLAlchemy 2
- Pydantic v2
- Polars
- DuckDB
- Redis Streams
Agents
- LangGraph
- sqlglot guard
- Pydantic output
Data
-
Postgres - Redis 7
- S3-compatible storage
Quality
- pytest
- Playwright
Deployment
-
Docker Compose - Caddy
- Prometheus
Gallery
Landing
Public landing: quality gates, quarantine, reconciliation, and data contract as the product promise.
Dashboard
24 hour summary, pipeline health, and latest runs across three demo projects.
Pipeline Canvas
Source, validate, clean, transform, aggregate, and sink nodes with deterministic validation.
Run Monitor
Node timeline, rows in and out, quarantine counts, gate checklist, and live logs.
Quarantine and Reconciliation
Rejected rows with reason codes, ratio against project threshold, and reconciliation status.
Data Contract
Project policy, four gate rules, and per pipeline row validation rules with enforcement links.
