Data Flow

Portal sends run request

The user clicks “Run” in the portal. The portal sends POST /api/v1/runs with the pipeline identifier (namespace, layer, name) and trigger type (manual).

ratd creates a pending run

ratd inserts a new row in the runs table with status=pending and dispatches the execution to the runner via the SubmitPipeline gRPC call. The runner returns a run handle immediately --- execution is asynchronous.

Runner executes the 5-phase pipeline

The runner performs branch creation, config loading, SQL execution, Iceberg writes, and quality testing. See Pipeline Execution for the full breakdown.

Runner calls back to ratd

On completion (success or failure), the runner pushes the terminal status to ratd via an HTTP POST to the callback URL. This is faster than polling.

Portal picks up the result

The portal’s SWR hooks automatically revalidate the run status. The UI updates to show the completed run with duration, rows written, and any quality test results.

Portal sends the query

The user types SQL in the CodeMirror editor and clicks “Run”. The portal sends POST /api/v1/query/execute with the raw SQL text.

ratd validates and proxies

ratd performs basic validation (query size under 100 KB, timeout set to 30 seconds) and forwards the query to ratq via the ExecuteQuery gRPC call. ratd acts as a proxy --- it does not execute SQL itself.

ratq enforces read-only

Before execution, ratq scans the SQL for dangerous patterns. It blocks 25+ SQL statements (CREATE, DROP, INSERT, etc.) and 20+ functions (write_parquet, read_csv, etc.). If any blocked pattern is detected, it returns an error immediately.

ratq executes via DuckDB

The query runs against ratq’s persistent DuckDB connection, which has Iceberg tables pre-registered from the last catalog refresh (every 30 seconds). DuckDB reads Parquet files from MinIO via iceberg_scan().

Results return to the portal

Results are serialized as columnar data (Arrow format) from ratq to ratd, then converted to JSON for the portal. The portal renders results in a data table with type-colored values, row numbers, and zebra striping.

User uploads a file

The user drags a file into a landing zone in the portal. The portal sends a multipart upload to POST /api/v1/storage/upload. ratd validates the file (size limits, allowed content types, no path traversal) and writes it to MinIO at s3://rat/{namespace}/landing/{zone_name}/{filename}.

File is registered in Postgres

ratd inserts a row in the landing_files table with the file’s S3 path, size, content type, and upload timestamp.

Pipeline reads from landing zone

When a bronze pipeline uses {{ landing_zone("orders") }}, the Jinja template resolves to the S3 glob pattern s3://rat/{namespace}/landing/orders/**. DuckDB’s read_csv_auto() or read_parquet() reads all files matching the pattern.

Post-run archival (optional)

If the pipeline has @archive_landing_zones: true, after a successful run, the source files are moved from the landing zone to _processed/{run_id}/. This prevents reprocessing on subsequent runs.

Scheduler ticks every 30 seconds

The ratd scheduler runs a background goroutine that ticks every 30 seconds. On each tick, it queries the schedules table for rows where enabled = true AND next_run_at <= now().

Advisory lock prevents duplicate execution

Before processing due schedules, the scheduler acquires a Postgres advisory lock. This ensures that only one ratd instance (in a multi-replica scenario) evaluates schedules. Other instances skip the tick.

Run is created and dispatched

For each due schedule, the scheduler creates a run record with the trigger set to the schedule type (e.g., schedule:hourly) and dispatches it to the runner via the WarmPoolExecutor.

Schedule is updated

After dispatching, the scheduler updates the schedule’s last_run_at to now and computes the next_run_at using the cron expression via robfig/cron/v3.

Catch-up policy

If ratd was down and missed a scheduled run, the scheduler uses a catch-up-once policy: it fires the pipeline once (not once per missed tick). This prevents a flood of duplicate runs after a restart.