deriva-ml Schema Reference

This document is the authoritative description of the deriva-ml schema. It is kept in sync with src/deriva_ml/schema/create_schema.py by the deriva-ml-validate-schema CI check.

Editing this doc

Schema changes are doc-first per Phase-2-Subsystem-0. To change the schema:

1. Edit the relevant ## <Table> section below (or add a new one).
2. Edit src/deriva_ml/schema/create_schema.py to match.
3. Run uv run deriva-ml-validate-schema locally to verify agreement.
4. Commit both files together. CI re-runs the validator.

What is and isn't validated

The validator enforces: table names, columns (name + type), foreign keys, vocabulary seeded terms, and association-table endpoints.

The validator does NOT enforce: descriptions (narrative on both sides), ERMrest annotations, curie_template values, indexes, or display configs. These may differ between doc and code without CI failure.

The referenced_schema field in foreign keys is often written in the code as a parameter (sname) or attribute (schema.name). The validator's AST parser resolves these to a <dynamic> sentinel and treats them as matching any doc-side value. In this doc we write the literal schema name (deriva-ml) for clarity.

See docs/superpowers/specs/2026-04-21-schema-doc-source-of-truth-design.md for the full rationale.

Conventions

Foreign keys to vocabulary tables reference Name, not RID

In the deriva-ml schema, every FK that references a vocabulary table references the vocabulary's Name column — never RID or another column. This applies to both explicit ForeignKeyDef call sites (e.g. Execution.Status → Execution_Status.Name) and to FKs generated by Table.define_association when one of the endpoints is a vocabulary table (e.g. Dataset_Dataset_Type(Dataset_Type) → Dataset_Type.Name).

Rationale. A Name-valued FK stores a human-readable string, not an opaque RID. Raw catalog exports are self-describing; EnumClass(row["Col"]) parses directly; migration from a plain-text column to an FK'd column requires no data conversion.

Enforcement. tests/schema/test_vocab_fk_convention.py creates a fresh catalog via create_ml_catalog, introspects every FK in the deriva-ml schema, and fails if any vocabulary-targeting FK references a column other than Name. Deriva-py's Table.define_association default heuristic (key_column_search_order=['Name', 'name', 'ID', 'id'] with RID as fallback) already picks Name for vocabulary endpoints; this convention makes the implicit behavior explicit and enforces it for new explicit FKs.

Table of contents

Ordered as: core entities, then vocabularies, then association tables.

Core entities: Dataset, Dataset_Version, Execution, Workflow Vocabularies: Dataset_Type, Workflow_Type, Feature_Name, Asset_Type, Asset_Role, Execution_Status Associations: Dataset_Dataset_Type, Dataset_Nested_Dataset, Dataset_Execution, Dataset_File, Execution_Nested_Execution, Workflow_Workflow_Type, Directory_Dataset

---

Dataset

A versioned collection of records identified for training, validation, testing, or other ML purposes. The primary grouping unit for ML data.

table: Dataset
kind: table
columns:
- name: Description
  type: markdown
- name: Deleted
  type: boolean
foreign_keys: []

Dataset_Version

Version history for a Dataset. Each row records one semantic version (MAJOR.MINOR.PATCH) of a dataset, pinned to a catalog snapshot and optionally a MINID for durable external reference.

table: Dataset_Version
kind: table
columns:
- name: Version
  type: text
- name: Description
  type: markdown
- name: Dataset
  type: text
- name: Execution
  type: text
- name: Minid
  type: text
- name: Minid_Spec_Hash
  type: text
- name: Snapshot
  type: text
foreign_keys:
- columns:
  - Dataset
  referenced_schema: deriva-ml
  referenced_table: Dataset
  referenced_columns:
  - RID
- columns:
  - Execution
  referenced_schema: deriva-ml
  referenced_table: Execution
  referenced_columns:
  - RID

Directory_Dataset

Source folder a directory dataset (auto-created by add_files) represents, as a path relative to the ingest root. One row per directory dataset; absent for datasets not built from a directory tree.

table: Directory_Dataset
kind: table
columns:
- name: Dataset
  type: text
- name: Path
  type: text
foreign_keys:
- columns:
  - Dataset
  referenced_schema: deriva-ml
  referenced_table: Dataset
  referenced_columns:
  - RID

Execution

Per-execution lifecycle row. Created once per workflow run; status transitions (Created → Running → Stopped → Pending_Upload → Uploaded or Failed) are managed by the Phase-1 state machine in src/deriva_ml/execution/state_machine.py. The Status column is FK-constrained to the Execution_Status vocabulary (7 canonical lifecycle values, seeded at schema creation); invalid status writes are rejected by the catalog.

table: Execution
kind: table
columns:
- name: Workflow
  type: text
- name: Description
  type: markdown
- name: Execution_Duration
  type: text
- name: Download_Duration
  type: text
- name: Upload_Duration
  type: text
- name: Status
  type: text
- name: Status_Detail
  type: text
foreign_keys:
- columns:
  - Workflow
  referenced_schema: deriva-ml
  referenced_table: Workflow
  referenced_columns:
  - RID
- columns:
  - Status
  referenced_schema: deriva-ml
  referenced_table: Execution_Status
  referenced_columns:
  - Name

Workflow

A computational pipeline or procedure definition — the "program" an Execution runs. Identified by Checksum (content hash of the code) for deduplication.

table: Workflow
kind: table
columns:
- name: Name
  type: text
- name: Description
  type: markdown
- name: URL
  type: ermrest_uri
- name: Checksum
  type: text
- name: Version
  type: text
foreign_keys: []

Dataset_Type

Controlled vocabulary classifying a Dataset's purpose. Multiple types can apply (e.g., Training + Labeled). Terms are orthogonal tags.

table: Dataset_Type
kind: vocabulary
terms:
- name: Complete
- name: File
- name: Directory
- name: Training
- name: Testing
- name: Validation
- name: Split
- name: Labeled
- name: Unlabeled

Workflow_Type

Controlled vocabulary classifying a Workflow by its computational purpose.

table: Workflow_Type
kind: vocabulary
terms:
- name: Training
- name: Testing
- name: Prediction
- name: Feature_Creation
- name: Visualization
- name: Analysis
- name: Ingest
- name: Data_Cleaning
- name: Dataset_Management

Feature_Name

Controlled vocabulary of feature identifiers. Each domain-specific Feature table references its name here. No starter terms — projects define their own.

table: Feature_Name
kind: vocabulary

Asset_Type

Controlled vocabulary classifying file assets by purpose. Drives which upload-spec regex applies and which Execution_* association table an asset flows into.

table: Asset_Type
kind: vocabulary
terms:
- name: Execution_Config
- name: Runtime_Env
- name: Hydra_Config
- name: Deriva_Config
- name: Metrics_File
- name: Execution_Metadata
- name: Execution_Asset
- name: File
- name: Input_File
- name: Output_File
- name: Model_File
- name: Notebook_Output

Asset_Role

Controlled vocabulary classifying an asset's role in an Execution — input data or output artifact.

table: Asset_Role
kind: vocabulary
terms:
- name: Input
- name: Output

Execution_Status

Controlled vocabulary for the Execution.Status column — the lifecycle state of an Execution row. Seeded with the 7 canonical values matching the ExecutionStatus StrEnum in src/deriva_ml/execution/state_store.py. New terms should NOT be added without updating that enum in tandem.

table: Execution_Status
kind: vocabulary
terms:
- name: Created
- name: Running
- name: Stopped
- name: Failed
- name: Pending_Upload
- name: Uploaded
- name: Aborted

Dataset_Dataset_Type

Association linking a Dataset to its zero-or-more Dataset_Type classifications.

table: Dataset_Dataset_Type
kind: association
associates:
- table: Dataset
- table: Dataset_Type

Dataset_Nested_Dataset

Self-association: one Dataset can contain nested Dataset children (e.g., a Split dataset has Training/Testing/Validation children).

table: Dataset_Nested_Dataset
kind: association
associates:
- table: Dataset
- table: Nested_Dataset

Dataset_Execution

Association linking a Dataset to the Execution(s) that produced or consumed it. Drives provenance queries.

table: Dataset_Execution
kind: association
associates:
- table: Dataset
- table: Execution

Dataset_File

Association linking a Dataset to raw File members (files that aren't tracked as catalog Asset rows).

table: Dataset_File
kind: association
associates:
- table: Dataset
- table: File

Execution_Nested_Execution

Self-association for hierarchical runs: a sweep/multirun Execution is the parent; individual trials are children. The optional Sequence column records ordering (null for parallel).

table: Execution_Nested_Execution
kind: association
associates:
- table: Execution
- table: Nested_Execution

Workflow_Workflow_Type

Association linking a Workflow to its zero-or-more Workflow_Type classifications.

table: Workflow_Workflow_Type
kind: association
associates:
- table: Workflow
- table: Workflow_Type