Part 7: Triggers & Automation ⚡

Time: ~15 minutes | Prerequisites: Part 6 completed (Python pipeline running)

Up to now, you have been running every pipeline manually — click Run, wait, repeat. That works for development, but a real data platform needs automation. RAT’s trigger system lets you wire pipelines together so they fire automatically based on events: a file upload, a cron schedule, an upstream pipeline completing, or an external HTTP call.

In this part, you will set up four different trigger types and watch your pipeline chain execute hands-free.

Trigger Types Overview

RAT supports 6 trigger types. Here is the full reference:

Every trigger also has these common fields:

1. Cron Trigger — Schedule mission_log

Let’s start simple. We will schedule the mission_log bronze pipeline to run every hour.

2. Landing Zone Upload Trigger

Even better than a schedule: trigger mission_log immediately when someone uploads new CSV data to the launch-data landing zone.

3. Pipeline Success Trigger — Chain Silver After Bronze

Now for the powerful part: chaining layers together. We want enriched_launches (Silver) to run automatically every time mission_log (Bronze) succeeds.

Now the chain is wired: upload a CSV → Bronze runs → Silver runs automatically. Let’s test it.

4. Webhook Trigger — Fire from External Systems

Webhooks let external systems (CI/CD, Airflow, custom scripts) trigger RAT pipelines programmatically via HTTP.

curl -X POST http://localhost:8080/api/v1/triggers/{trigger-id}/fire \
  -H "Authorization: Bearer rat_wh_your_token_here"

Cooldown Configuration

The cooldown_seconds field prevents a trigger from firing more frequently than the specified interval. Here is a practical guide:

Example: A landing zone receives 20 CSV files over 2 minutes. With cooldown_seconds: 120, the trigger fires once for the first file, then waits 2 minutes before it can fire again. By that time, all 20 files are present and the pipeline processes them all in a single run.

Other Trigger Types

We set up four trigger types hands-on. Here is a quick overview of the remaining two:

File Pattern

Like landing_zone_upload, but more selective. It fires only when the uploaded filename matches a glob pattern.

{
  "type": "file_pattern",
  "config": {
    "zone_namespace": "default",
    "zone_name": "raw_events",
    "pattern": "launches_*.csv"
  },
  "cooldown_seconds": 60,
  "enabled": true
}

Use case: A landing zone receives both .csv and .parquet files, but you only want to trigger on CSV uploads.

Cron Dependency

The most sophisticated trigger. It combines a cron schedule with dependency checking: the trigger fires only when both conditions are met:

1. The cron schedule is due
2. All listed dependency pipelines have completed successfully since the trigger last fired

{
  "type": "cron_dependency",
  "config": {
    "cron_expr": "0 * * * *",
    "dependencies": [
      {
        "namespace": "default",
        "layer": "silver",
        "name": "enriched_launches"
      }
    ]
  },
  "enabled": true
}

Use case: A Gold pipeline that should run hourly, but only if Silver has produced fresh data. This prevents wasteful re-runs when nothing has changed.

Building Trigger Chains

With what you have learned, you can build sophisticated automation chains:

One file upload cascades through the entire pipeline graph. Zero manual intervention.

What You Built

In this part, you:

• ✅ Created a cron trigger to schedule mission_log hourly
• ✅ Created a landing zone upload trigger for event-driven ingestion
• ✅ Created a pipeline_success trigger to chain Silver after Bronze
• ✅ Created a webhook trigger and fired it with curl
• ✅ Learned about cooldown configuration for batching rapid events
• ✅ Explored file_pattern and cron_dependency triggers

Your pipelines now run themselves. In the final part, you will complete the platform with a Gold layer, explore versioning, data retention, and the full lineage DAG.

Next: In Part 8, you’ll build the Gold layer, wire the complete trigger chain, and explore versioning, data retention, table documentation, and the query editor.