Your First Pipeline

Open the Portal

Navigate to http://localhost:3000 in your browser. You should see the RAT dashboard.

Go to the Pipelines page

Click Pipelines in the left sidebar. This page lists all pipelines in your platform. Right now, it should be empty.

Create a new pipeline

Click the + New Pipeline button in the top-right corner. A dialog will appear asking for the pipeline details:

Click Create to confirm.

Write the pipeline SQL

After creation, you are taken to the pipeline detail page with the Code tab open. You will see a CodeMirror SQL editor. Replace any placeholder content with the following:

SELECT
    1 AS order_id,
    'Alice' AS customer,
    99.99 AS amount,
    current_timestamp AS created_at
 
UNION ALL
 
SELECT
    2 AS order_id,
    'Bob' AS customer,
    149.50 AS amount,
    current_timestamp AS created_at
 
UNION ALL
 
SELECT
    3 AS order_id,
    'Charlie' AS customer,
    75.00 AS amount,
    current_timestamp AS created_at

This SQL generates three sample order rows. In a real pipeline, you would typically read from a Bronze table (using ref()) or ingest files from a landing zone (using landing_zone()), but static data works perfectly for learning the basics.

Save the pipeline

Press Ctrl+S (or Cmd+S on macOS) to save. You should see a brief confirmation toast in the bottom-right corner.

Preview the output

Before running the pipeline for real, you can preview what it will produce. Press Ctrl+Shift+Enter (or Cmd+Shift+Enter on macOS).

The preview panel will open below the editor, showing a table with your three rows:

Publish the pipeline

Click the Publish button in the toolbar above the editor. Publishing creates a versioned snapshot of your pipeline code. This is the version that will execute when you trigger a run.

Run the pipeline

Click the Run button (the play icon) in the toolbar. A confirmation dialog may appear asking you to confirm a manual trigger. Click Confirm.

RAT will:

1. Create a new run with status pending
2. Send the pipeline to the runner service for execution
3. The runner creates an isolated Nessie branch
4. Execute your SQL via DuckDB
5. Write the results as an Iceberg table in MinIO
6. Run any quality tests (none yet for this pipeline)
7. Merge the branch into the main catalog
8. Mark the run as success

Watch the run

You will be automatically taken to the Run Detail page (or navigate there via Runs in the sidebar). Here you can see:

• Status badge — transitions from pending to running to success
• Live logs — streamed in real-time via SSE (Server-Sent Events)
• Duration — how long the run took
• Trigger — shows manual since you clicked Run

The logs will look something like:

[INFO]  Starting pipeline: default.bronze.raw_orders
[INFO]  Creating branch: run-abc123
[INFO]  Executing SQL pipeline...
[INFO]  Query produced 3 rows
[INFO]  Writing to Iceberg table: default.bronze.raw_orders
[INFO]  No quality tests configured — skipping
[INFO]  Merging branch: run-abc123 → main
[INFO]  Pipeline completed successfully in 2.3s

Query the results

Navigate to Query Console in the left sidebar. This opens the interactive SQL editor powered by DuckDB (via the ratq service).

Type the following query and press Ctrl+Enter (or click Run Query):

SELECT * FROM "default"."bronze"."raw_orders"

You should see the three rows you created:

You can run any SQL against your tables — aggregations, joins, filters:

SELECT
    customer,
    amount,
    RANK() OVER (ORDER BY amount DESC) AS rank
FROM "default"."bronze"."raw_orders"
ORDER BY amount DESC

Explore the table

Navigate to Explorer in the left sidebar. You will see your tables grouped by namespace and layer:

default/
  bronze/
    raw_orders (3 rows)

Click on raw_orders to open the Table Detail page. Here you can see:

• Schema tab — column names, types, and nullability
• Docs tab — editable documentation for the table (add descriptions for each column)
• Preview tab — a quick look at sample data