Part 8: Gold Layer & Complete Platform 🏆

Create a new pipeline

Navigate to Pipelines → click New Pipeline.

Write the SQL

-- @description: Country-level launch statistics by year
-- @merge_strategy: full_refresh
 
SELECT
    country,
    EXTRACT(YEAR FROM launch_date::DATE) AS launch_year,
    COUNT(*) AS total_launches,
    SUM(CASE WHEN outcome = 'success' THEN 1 ELSE 0 END) AS successes,
    ROUND(100.0 * SUM(CASE WHEN outcome = 'success' THEN 1 ELSE 0 END) / COUNT(*), 1) AS success_rate_pct,
    SUM(payload_mass_kg) AS total_payload_kg,
    COUNT(DISTINCT vehicle) AS unique_vehicles,
    STRING_AGG(DISTINCT mission_type, ', ' ORDER BY mission_type) AS mission_types
FROM {{ ref('silver.enriched_launches') }}
GROUP BY country, launch_year
ORDER BY launch_year DESC, total_launches DESC

This query:

1. Groups by country and year — giving you a row per country per year
2. Counts launches and successes — with a computed success rate percentage
3. Sums payload mass — total kg launched per country per year
4. Counts unique vehicles — how many different rockets each country used
5. Aggregates mission types — comma-separated list of mission types per group

Preview

Click Preview to verify the results. You should see rows like USA, 2024, 8 launches, 87.5% success rate — one row per country-year combination.

Publish

Click Publish. Enter version message: "Initial gold aggregation".

Run

Click Run to execute the pipeline and write the results to Iceberg.

Open Gold pipeline settings

Navigate to Pipelines → launch_summary → Settings tab.

Add a pipeline_success trigger

Click Add Trigger → select pipeline_success.

Save

Click Save. The complete chain is now wired.

Trigger the chain

You have two options:

Watch the domino effect

Go to the Runs page and watch the cascade:

Three pipelines, zero manual clicks after the initial trigger.

Verify the Gold data

Navigate to Query and run:

SELECT *
FROM default.gold.launch_summary
ORDER BY launch_year DESC, total_launches DESC

You should see country-level statistics — the final analytics-ready output.

Open a pipeline overview

Navigate to Pipelines → enriched_launches → Overview tab.

View the version history

Scroll down to the Version History section. You should see at least one version:

Version 1 — "Initial pipeline"
            Published during Part 3

Edit the code

Go to the Code tab. Add a comment at the top:

-- @description: Launches enriched with vehicle specifications (improved)

Save the file. Notice the Unpublished Changes badge appears on the Overview tab.

Publish a new version

Go back to Overview → click Publish. Enter: "Update description annotation".

Now you have two versions:

Version 2 — "Update description annotation"
            Published just now

Version 1 — "Initial pipeline"
            Published earlier

Find the target version

In the version history, locate Version 1.

Click Rollback

Click the Rollback button next to Version 1. RAT creates:

Version 3 — "Rollback to version 1"
            Published just now

Version 2 — "Update description annotation"
            Still in history

Version 1 — "Initial pipeline"
            The rollback target

Navigate to Settings

Click Settings in the sidebar navigation.

Open the Retention tab

You will see all retention configuration fields with their current values:

Review the defaults

For this tutorial, the defaults are fine. In a production deployment, you might lower runs_max_per_pipeline for high-frequency pipelines or increase runs_max_age_days for regulated data.

Navigate to Explorer

Click Explorer in the sidebar. You will see a tree of all namespaces, layers, and tables.

Click a table

Click default → gold → launch_summary to open the table detail view.

Open the Docs tab

Click the Docs tab. You will see all columns listed with empty description fields.

Add column descriptions

Fill in descriptions for key columns:

Save

Click Save. These descriptions are now visible to anyone browsing the Explorer — and they appear in the schema sidebar of the Query page.

Open the Query page

Navigate to Query in the sidebar.

Browse the schema sidebar

On the left, you will see a schema browser listing all tables organized by namespace → layer → table. Each table shows its columns and types. Click a table name to insert it into the editor at the cursor position.

Write a rich query

Let’s write a comprehensive query against the Gold table:

SELECT
    ls.country,
    ls.launch_year,
    ls.total_launches,
    ls.successes,
    ls.success_rate_pct,
    ls.total_payload_kg,
    ls.unique_vehicles,
    ls.mission_types,
    -- Compare to previous year
    LAG(ls.total_launches) OVER (
        PARTITION BY ls.country ORDER BY ls.launch_year
    ) AS prev_year_launches,
    ls.total_launches - COALESCE(LAG(ls.total_launches) OVER (
        PARTITION BY ls.country ORDER BY ls.launch_year
    ), 0) AS yoy_change
FROM default.gold.launch_summary ls
ORDER BY ls.country, ls.launch_year DESC

This query adds year-over-year comparisons using window functions — the kind of analysis that Gold tables are built for.

Run with Ctrl+Enter

Press Ctrl+Enter (or Cmd+Enter on Mac) to execute the query. Results appear in the table below the editor with sortable columns and type-colored values.

Explore the results

The data table supports:

• Column sorting — click any column header
• Row numbers — displayed in the leftmost column
• Type-colored values — numbers, strings, dates, and nulls are visually distinct
• Copy to clipboard — select rows or the entire result set