Part 3: Building Layers with ref() 🏗️

Time: ~10 minutes | Prerequisites: Part 2 completed (bronze pipelines running)

In this part, you will create your first Silver pipeline that joins two Bronze tables together. You will learn the ref() syntax for declaring dependencies between pipelines and explore the Lineage page to see your growing DAG.

The Medallion Architecture

RAT uses a three-layer approach to organize your data, commonly known as the medallion architecture:

Each layer builds on the one before it. Bronze pipelines read from external sources (landing zones, APIs, files). Silver pipelines read from Bronze tables. Gold pipelines read from Silver tables. This separation keeps your data organized and your pipelines easy to reason about.

Create the Silver Pipeline

Let’s create enriched_launches — a Silver pipeline that joins launch missions with their vehicle specifications.

-- @description: Launches enriched with vehicle specifications
 
SELECT
    l.launch_id,
    l.mission_name,
    l.launch_date,
    l.vehicle,
    l.launch_site,
    l.country,
    l.orbit,
    l.outcome,
    l.payload_mass_kg,
    l.mission_type,
    v.manufacturer,
    v.height_m,
    v.thrust_kn,
    v.stages,
    v.mass_kg AS vehicle_mass_kg
FROM {{ ref('bronze.mission_log') }} l
LEFT JOIN {{ ref('bronze.rocket_registry') }} v
    ON l.vehicle = v.vehicle_name

Understanding ref()

The ref() function is how you declare dependencies between pipelines. When RAT sees {{ ref('bronze.mission_log') }}, it does two things:

1. Resolves the table path — replaces the ref with the actual Iceberg table reference
2. Registers a dependency — RAT now knows this Silver pipeline depends on that Bronze pipeline

Two-part vs three-part syntax

In most cases, the 2-part syntax is what you want. It references a pipeline within the same namespace as the current pipeline. Use 3-part syntax only when you need to read from a different namespace.

Explore the Lineage DAG

Now that your Silver pipeline references two Bronze pipelines, RAT has built a dependency graph. Let’s see it.

Verify the Data

Let’s confirm the enriched data looks correct.

SELECT
    mission_name,
    vehicle,
    manufacturer,
    thrust_kn,
    outcome
FROM default.silver.enriched_launches
ORDER BY launch_date DESC
LIMIT 10

SELECT DISTINCT vehicle
FROM default.silver.enriched_launches
WHERE manufacturer IS NULL

What You Built

In this part, you:

• ✅ Created a Silver pipeline (enriched_launches) that joins two Bronze tables
• ✅ Learned the ref() syntax for declaring pipeline dependencies
• ✅ Explored the Lineage DAG to visualize data flow
• ✅ Verified the enriched data in the Query page

Your pipeline graph now has three nodes across two layers. In the next part, you’ll learn how to make pipelines incremental so they don’t reprocess the entire dataset on every run.

Next: In Part 4, you’ll master merge strategies and make your pipelines load data incrementally.