Annotations

Annotations are metadata directives embedded in SQL or Python comments at the top of pipeline and quality test files. They configure how RAT processes the pipeline output — merge strategy, deduplication keys, watermark tracking, retry behavior, and more.

Syntax

Annotations use the @key: value format inside comments:

-- @merge_strategy: incremental
-- @unique_key: order_id
-- @watermark_column: updated_at
-- @description: Deduplicated orders from the raw feed

# @merge_strategy: full_refresh
# @description: Fetch and materialize exchange rates from API
# @max_retries: 3
# @retry_delay_seconds: 30

Parsing Rules

1. Comment prefix: SQL files use --, Python files use #.
2. Top of file: Annotations must appear at the very top of the file, before any code.
3. First non-comment line stops parsing: Once the parser encounters a line that is not a comment or is blank after annotations, it stops looking for more annotations.
4. One per line: Each annotation must be on its own line.
5. Whitespace is trimmed: Leading/trailing spaces around values are stripped.
6. Keys are case-insensitive: @Merge_Strategy and @merge_strategy are equivalent (normalized to lowercase internally).
7. Order does not matter: Annotations can appear in any order, though by convention they follow the order listed in this reference.

What Stops Parsing

-- @merge_strategy: incremental    ← PARSED
-- @unique_key: order_id           ← PARSED
--                                 ← skipped (blank comment)
-- @watermark_column: updated_at   ← PARSED (blank comments don't stop parsing)
 
SELECT * FROM {{ ref('bronze.data') }}   ← parsing stops here (non-comment line)
 
-- @description: too late          ← NOT PARSED (after SQL)

-- @merge_strategy: incremental
-- @unique_key: order_id
-- This pipeline cleans raw data     ← stops parsing (no @key pattern)
-- @watermark_column: updated_at     ← NOT PARSED
 
SELECT ...

Pipeline Annotations

These annotations configure how a pipeline processes and writes data.

@description

Human-readable description of what the pipeline does. Displayed in the portal’s pipeline list and detail views.

-- @description: Daily revenue aggregation by currency and region

@materialized

Controls how the pipeline output is materialized. Currently only table is supported.

-- @materialized: table

@merge_strategy

Defines how new data is merged with existing data in the target table.

-- @merge_strategy: incremental

Strategy behaviors:

@unique_key

The column(s) used as the primary key for deduplication. Required for incremental, delete_insert, and scd2 strategies.

-- Single key
-- @unique_key: order_id
 
-- Composite key
-- @unique_key: customer_id, product_id, order_date

For composite keys, separate column names with commas. Whitespace around column names is trimmed.

@watermark_column

The column used to track incremental progress. RAT computes MAX(watermark_column) from the existing table to determine which rows are new.

-- @watermark_column: updated_at

The watermark column should be a monotonically increasing value — typically a timestamp (updated_at, created_at) or a sequential integer (id, version).

@partition_column

The column used for partitioning in snapshot strategy. Each run writes a new partition value.

-- @merge_strategy: snapshot
-- @partition_column: snapshot_date

@scd_valid_from

The column name for the SCD2 record validity start date.

-- @scd_valid_from: effective_date

@scd_valid_to

The column name for the SCD2 record validity end date.

-- @scd_valid_to: expiry_date

@archive_landing_zones

When set to true, RAT moves landing zone files to a _processed/ subdirectory after a successful run. This prevents reprocessing the same files on the next run.

-- @archive_landing_zones: true

@max_retries

Maximum number of retry attempts if the pipeline run fails. Applies to transient errors (network timeouts, temporary S3 unavailability).

-- @max_retries: 3

@retry_delay_seconds

Delay in seconds between retry attempts. Uses a fixed delay (not exponential backoff).

-- @retry_delay_seconds: 30

Pipeline Annotations Summary

Quality Test Annotations

Quality tests are SQL files in the pipeline’s tests/quality/ directory. They use the same annotation syntax but with a different set of keys.

@severity

Determines what happens when the quality test fails.

-- @severity: error

• error: The run fails and the ephemeral Nessie branch is deleted. Bad data never reaches production.
• warn: The run succeeds with a warning. The branch is merged despite the quality issue. The warning is visible in the portal and run logs.

@description

Human-readable description of what the quality test checks. Displayed in the portal’s quality test results.

-- @description: Ensure no orders have negative total amounts

@tags

Comma-separated tags for organizing and filtering quality tests.

-- @tags: finance, critical, orders

@remediation

Instructions for how to fix the issue when this quality test fails. Shown in the portal alongside the failure.

-- @remediation: Check the source system for orders with negative amounts. These are usually refunds that should have status='refunded'.

Quality Test Example

-- @severity: error
-- @description: Orders must not have negative total amounts
-- @tags: finance, data-integrity
-- @remediation: Negative amounts indicate refunds — filter them out or set status to 'refunded'
 
SELECT *
FROM {{ this }}
WHERE total_amount < 0

A quality test passes when it returns zero rows. If the query returns any rows, those rows represent violations and the test fails.

Quality Test Annotations Summary

Annotations vs config.yaml

Pipeline configuration can be defined in two places:

1. Annotations in the pipeline file (pipeline.sql or pipeline.py)
2. config.yaml in the pipeline directory

When both exist, annotations take precedence. The resolution order is:

config.yaml Format

merge_strategy: incremental
unique_key: order_id
watermark_column: updated_at
description: Deduplicated orders with latest status
max_retries: 2
retry_delay_seconds: 15

When to Use Which

Annotation Validation

RAT validates annotations at pipeline registration time and again at the start of each run: