ReferenceAPI ReferenceStorage

Storage

Storage endpoints manage pipeline source files in S3 (MinIO). These endpoints power the portal’s code editor — every file you see in the editor is stored as an S3 object. Writing a file under a pipeline’s prefix automatically marks the pipeline as draft-dirty.

Endpoints

MethodEndpointDescription
GET/api/v1/filesList files by prefix
GET/api/v1/files/{path}Read file content
PUT/api/v1/files/{path}Write or update a file
DELETE/api/v1/files/{path}Delete a file
POST/api/v1/files/uploadMultipart file upload

List Files

GET /api/v1/files

Lists files under an S3 prefix. The prefix must start with a valid namespace.

Query Parameters

ParameterTypeRequiredDescription
prefixstringYesS3 prefix to list (e.g., default/pipelines/silver/orders/)
namespacestringNoFilter by namespace
excludestringNoComma-separated path segments to filter out (e.g., landing,data)

Request

curl "http://localhost:8080/api/v1/files?prefix=default/pipelines/silver/orders/&exclude=landing,data"

Response — 200 OK

{
  "files": [
    {
      "path": "default/pipelines/silver/orders/pipeline.sql",
      "size": 245,
      "modified": "2026-02-12T10:00:00Z",
      "type": "pipeline-sql",
      "version_id": "abc123"
    },
    {
      "path": "default/pipelines/silver/orders/config.yaml",
      "size": 180,
      "modified": "2026-02-12T10:00:00Z",
      "type": "config",
      "version_id": "def456"
    }
  ]
}

Response Fields

FieldTypeDescription
filesarrayList of file objects
files[].pathstringFull S3 path
files[].sizeintegerFile size in bytes
files[].modifiedstringISO 8601 last modified timestamp
files[].typestringAuto-detected file type (see File Type Classification below)
files[].version_idstringS3 version ID

File Type Classification

The type field is auto-detected from the file path:

TypeMatches
pipeline-sqlpipeline.sql, any *.sql file
pipeline-pypipeline.py
configconfig.yaml, config.yml, schema.yaml, schema.yml
meta*.meta.yaml, *.meta.yml, meta.yaml
doc*.md, *.txt, *.rst, README
testFiles in tests/ or test/ dirs, test_* prefix, *_test.go, *_test.py
hookFiles in hooks/ dir, hook_*, pre_*, post_* prefixes
(empty string)Unrecognized files

Read File

GET /api/v1/files/{path}

Returns the content of a single file from S3.

Path Parameters

ParameterTypeDescription
pathstringFull S3 file path (e.g., default/pipelines/silver/orders/pipeline.sql)

Request

curl http://localhost:8080/api/v1/files/default/pipelines/silver/orders/pipeline.sql

Response — 200 OK

{
  "path": "default/pipelines/silver/orders/pipeline.sql",
  "content": "SELECT * FROM {{ ref('bronze.raw_orders') }}\nWHERE id IS NOT NULL",
  "size": 245,
  "modified": "2026-02-12T10:00:00Z",
  "version_id": "abc123"
}

Response Fields

FieldTypeDescription
pathstringFull S3 path
contentstringFile content as a string
sizeintegerFile size in bytes
modifiedstringISO 8601 last modified timestamp
version_idstringS3 version ID

Error Responses

StatusCodeDescription
404NOT_FOUNDFile not found
500INTERNALS3 read error

Write File

PUT /api/v1/files/{path}

Creates or updates a file in S3. Writing a file under a pipeline’s S3 prefix automatically marks the pipeline as draft-dirty (unpublished changes).

Path Parameters

ParameterTypeDescription
pathstringFull S3 file path

Request Body

FieldTypeRequiredDescription
contentstringYesFile content to write

Request

curl -X PUT http://localhost:8080/api/v1/files/default/pipelines/silver/orders/pipeline.sql \
  -H "Content-Type: application/json" \
  -d '{
    "content": "SELECT * FROM {{ ref('\''bronze.raw_orders'\'') }}"
  }'

Response — 200 OK

{
  "path": "default/pipelines/silver/orders/pipeline.sql",
  "status": "written",
  "version_id": "abc123"
}

Error Responses

StatusCodeDescription
400INVALID_ARGUMENTInvalid request body (missing content field)
400INVALID_ARGUMENTPath validation failure (see Path Validation below)
500INTERNALS3 write error

Delete File

DELETE /api/v1/files/{path}

Deletes a file from S3.

Path Parameters

ParameterTypeDescription
pathstringFull S3 file path

Request

curl -X DELETE http://localhost:8080/api/v1/files/default/pipelines/silver/orders/old_file.sql

Response — 204 No Content

No response body.

Error Responses

StatusCodeDescription
500INTERNALS3 delete error

Upload File

POST /api/v1/files/upload

Uploads a file via multipart form data. Maximum file size is 32 MB.

Form Fields

FieldTypeRequiredDescription
pathstringYesDestination S3 path
filebinaryYesFile content

Request

curl -X POST http://localhost:8080/api/v1/files/upload \
  -F "path=default/pipelines/silver/orders/data.csv" \
  -F "file=@/local/path/to/data.csv"

Response — 201 Created

{
  "path": "default/pipelines/silver/orders/data.csv",
  "filename": "data.csv",
  "size": 1024,
  "status": "uploaded",
  "version_id": "abc123"
}

Response Fields

FieldTypeDescription
pathstringDestination S3 path
filenamestringOriginal filename
sizeintegerFile size in bytes
statusstringAlways uploaded
version_idstringS3 version ID

Error Responses

StatusCodeDescription
400INVALID_ARGUMENTMissing path form field
400INVALID_ARGUMENTMissing file form field
400INVALID_ARGUMENTPath validation failure (e.g., path traversal attempt)
413INVALID_ARGUMENTFile exceeds 32 MB size limit

Path Validation

All file paths are validated against the following rules:

RuleExample Invalid Path
Must not be empty(empty string)
Must not contain .. (path traversal)default/../etc/passwd
Must be relative (no leading /)/default/pipelines/silver/orders/pipeline.sql
Must not contain null bytesdefault/pipelines\x00/file
Must not contain backslashesdefault\pipelines\file
Must start with a valid namespace segmentinvalid_ns/pipelines/silver/orders/pipeline.sql
⚠️

Path traversal attempts (paths containing ..) are rejected with a 400 error. This prevents unauthorized access to files outside the intended S3 prefix.

TablesLanding Zones