Part 1: Your First Pipeline

Open the Pipelines page

Click Pipelines in the left sidebar. The page is empty since you have not created any pipelines yet.

Create a new pipeline

Click the + New Pipeline button in the top-right corner. A dialog will appear:

Fill in the following:

Click Create.

Write the SQL

You are now on the pipeline detail page with the Code tab open. You will see a CodeMirror SQL editor. Replace any placeholder content with:

-- @description: Sample space launch data
 
SELECT *
FROM (VALUES
    ('L001', 'Starlink Group 6-1', '2023-02-17', 'Falcon 9', 'Cape Canaveral SLC-40', 'USA', 'LEO', 'success', 17400, 'Communication'),
    ('L002', 'Crew-6', '2023-03-02', 'Falcon 9', 'Kennedy Space Center LC-39A', 'USA', 'LEO', 'success', 12055, 'Crewed'),
    ('L007', 'Euclid', '2023-07-01', 'Falcon 9', 'Cape Canaveral SLC-40', 'USA', 'L2', 'success', 2160, 'Science'),
    ('L008', 'Luna-25', '2023-08-10', 'Soyuz-2.1b', 'Vostochny Site 1S', 'Russia', 'Lunar', 'failure', 1750, 'Science'),
    ('L014', 'Starship IFT-2', '2023-11-18', 'Starship', 'Starbase OLP-A', 'USA', 'Suborbital', 'partial_failure', 0, 'Test')
) AS t(launch_id, mission_name, launch_date, vehicle, launch_site, country, orbit, outcome, payload_mass_kg, mission_type)

This generates 5 sample launches — a mix of successes, a failure, and a partial failure. We are keeping it small for now. Part 2 will bring in the full 25-row dataset.

Save the pipeline

Press Ctrl+S (or Cmd+S on macOS). A confirmation toast will appear.

Preview the output

Before committing to a real run, preview what your SQL produces. Press Ctrl+Shift+Enter (or Cmd+Shift+Enter).

The preview panel opens below the editor, showing a results table:

Publish the pipeline

Click the Publish button in the toolbar. Publishing creates a versioned snapshot of your code. This is the version that will execute when you trigger a run.

Run the pipeline

Click the Run button (play icon) in the toolbar. A confirmation dialog may appear — click Confirm.

Here is what happens behind the scenes:

1. ratd (API server) creates a run record in Postgres
2. ratd dispatches the run to the runner via gRPC
3. The runner asks Nessie to create an isolated branch
4. DuckDB executes your SQL and produces an Arrow table (5 rows)
5. PyIceberg writes the Arrow table as Parquet files to MinIO (S3)
6. Quality tests run (none configured yet — skip)
7. Nessie merges the branch back to main
8. The run is marked success

Watch the run

You will be taken to the Run Detail page. Here you can see:

• Status badge — transitions from pending → running → success
• Live logs — streamed in real-time via SSE
• Duration — how long the run took
• Trigger — shows manual

The logs will look something like:

[INFO]  Starting pipeline: default.bronze.raw_launches
[INFO]  Creating branch: run-abc123
[INFO]  Executing SQL pipeline...
[INFO]  Query produced 5 rows
[INFO]  Writing to Iceberg table: default.bronze.raw_launches
[INFO]  No quality tests configured — skipping
[INFO]  Merging branch: run-abc123 → main
[INFO]  Pipeline completed successfully in 2.1s

Query the results

Navigate to Query in the left sidebar. This opens the interactive SQL editor powered by DuckDB (via ratq).

Type the following and press Ctrl+Enter:

SELECT * FROM "default"."bronze"."raw_launches"

You should see your 5 rows:

Try a more interesting query:

SELECT
    outcome,
    COUNT(*) AS launches,
    SUM(payload_mass_kg) AS total_payload_kg
FROM "default"."bronze"."raw_launches"
GROUP BY outcome
ORDER BY launches DESC

Explore the table

Navigate to Explorer in the left sidebar. Your table appears in the tree:

default/
  bronze/
    raw_launches (5 rows)

Click raw_launches to see the table detail with three tabs:

• Schema — column names, types, and nullability
• Docs — editable documentation (add descriptions for each column)
• Preview — quick look at sample data