Pipeline Triggers

Pipeline triggers are the event-driven automation system in RAT. Unlike schedules (which are time-based only), triggers respond to a variety of events: file uploads, cron ticks, upstream pipeline completions, webhooks, file pattern matches, and cron-with-dependency checks.

Trigger Types Overview

RAT supports 6 trigger types:

Every trigger also has these common fields:

Creating Triggers

Trigger Type Details

1. Landing Zone Upload

Fires when any file is uploaded to a specific landing zone. Ideal for event-driven ingestion pipelines.

{
  "type": "landing_zone_upload",
  "config": {
    "zone_namespace": "default",
    "zone_name": "raw_orders"
  },
  "cooldown_seconds": 60,
  "enabled": true
}

Use case: A Bronze pipeline that ingests CSV files as they arrive in the raw_orders landing zone.

2. Cron

Fires on a cron schedule. Functionally equivalent to the legacy schedule system, but uses the unified trigger infrastructure with cooldown support.

{
  "type": "cron",
  "config": {
    "cron_expr": "0 */6 * * *"
  },
  "enabled": true
}

The cron_expr field uses the standard 5-field cron format (minute, hour, day-of-month, month, day-of-week).

Use case: Run a Silver pipeline every 6 hours to process accumulated Bronze data.

3. Pipeline Success

Fires when a specific upstream pipeline completes successfully. Failed or cancelled runs do not trigger downstream pipelines.

{
  "type": "pipeline_success",
  "config": {
    "pipeline_namespace": "default",
    "pipeline_layer": "bronze",
    "pipeline_name": "raw_orders"
  },
  "cooldown_seconds": 0,
  "enabled": true
}

Use case: A Silver pipeline that cleans orders should run immediately after the Bronze ingestion pipeline succeeds.

4. Webhook

Creates an HTTP endpoint that fires the pipeline when called with a valid token. RAT auto-generates the URL and bearer token when you create this trigger type.

{
  "type": "webhook",
  "config": {},
  "enabled": true
}

After creation, the API returns:

{
  "id": "trigger-uuid",
  "webhook_url": "http://localhost:8080/api/v1/triggers/trigger-uuid/fire",
  "webhook_token": "rat_wh_a1b2c3d4e5f6..."
}

Call the webhook:

curl -X POST http://localhost:8080/api/v1/triggers/{trigger-id}/fire \
  -H "Authorization: Bearer rat_wh_a1b2c3d4e5f6..."

Use case: External systems (CI/CD, Airflow, custom scripts) that need to trigger RAT pipelines programmatically.

5. File Pattern

Fires when an uploaded filename matches a glob pattern within a landing zone. More selective than landing_zone_upload, which fires on any upload.

{
  "type": "file_pattern",
  "config": {
    "zone_namespace": "default",
    "zone_name": "raw_events",
    "pattern": "events_*.parquet"
  },
  "cooldown_seconds": 30,
  "enabled": true
}

The pattern field supports standard glob syntax:

• * matches any characters (except /)
• ? matches a single character
• [abc] matches any character in the set

Use case: A landing zone receives both .parquet and .csv files, but you only want to trigger on Parquet uploads.

6. Cron Dependency

The most sophisticated trigger type. It combines a cron schedule with dependency checking: the trigger fires only when both conditions are met:

1. The cron schedule is due
2. All listed dependency pipelines have completed successfully since the trigger last fired

{
  "type": "cron_dependency",
  "config": {
    "cron_expr": "0 * * * *",
    "dependencies": [
      {
        "namespace": "default",
        "layer": "silver",
        "name": "clean_orders"
      },
      {
        "namespace": "default",
        "layer": "silver",
        "name": "clean_customers"
      }
    ]
  },
  "enabled": true
}

How dependency checking works:

1. At each cron tick, the evaluator checks if the schedule is due.
2. If due, it looks up the latest successful run of each dependency pipeline.
3. For each dependency, it checks if that run’s finished_at is after the trigger’s last_triggered_at.
4. If all dependencies have new successful data, the trigger fires.
5. If any dependency has not produced new data since the last trigger, it skips.

Use case: A Gold aggregation pipeline that should run hourly, but only if both upstream Silver pipelines have produced fresh data.

Trigger Evaluation Flow

Cooldown Periods

The cooldown_seconds field prevents a trigger from firing more frequently than the specified interval. This is evaluated by comparing last_triggered_at + cooldown_seconds against the current time.

When to use cooldown:

Example: A landing zone receives 20 CSV files over 2 minutes. With cooldown_seconds: 120, the trigger fires once for the first file, then waits 2 minutes before it can fire again. By that time, all 20 files are present and the pipeline processes them all in a single run.

Managing Triggers via API

Create a trigger

curl -X POST http://localhost:8080/api/v1/pipelines/{ns}/{layer}/{name}/triggers \
  -H "Content-Type: application/json" \
  -d '{
    "type": "pipeline_success",
    "config": {
      "pipeline_namespace": "default",
      "pipeline_layer": "bronze",
      "pipeline_name": "raw_orders"
    },
    "cooldown_seconds": 0,
    "enabled": true
  }'

List triggers for a pipeline

curl http://localhost:8080/api/v1/pipelines/{ns}/{layer}/{name}/triggers

Update a trigger

curl -X PATCH http://localhost:8080/api/v1/triggers/{trigger-id} \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": false,
    "cooldown_seconds": 120
  }'

Delete a trigger

curl -X DELETE http://localhost:8080/api/v1/triggers/{trigger-id}

Building Trigger Chains

Triggers can be combined to create sophisticated automation pipelines:

In this pattern:

1. A file upload triggers the Bronze pipeline immediately.
2. A pipeline_success trigger chains the Silver pipeline to run after Bronze succeeds.
3. A cron_dependency trigger runs the Gold pipeline hourly, but only if Silver has produced new data.

Troubleshooting

Trigger is enabled but never fires

1. Check cooldown — if cooldown_seconds is high, the trigger may be in its cooldown window
2. Check last_triggered_at — a very recent timestamp means cooldown is active
3. For cron triggers — verify the cron expression at crontab.guru
4. For cron_dependency — verify that all dependency pipelines have recent successful runs
5. Check ratd logs — docker compose logs ratd | grep "trigger evaluator"

Trigger fires too often

Increase the cooldown_seconds value. For landing zone triggers, a cooldown of 60-300 seconds is typical.

Webhook returns 401

The token is invalid or missing. Webhook tokens are shown only once at creation time. Delete the trigger and create a new one if you have lost the token.