`airflow.models.dataset`¶

Module Contents¶

Classes¶

`DatasetModel`	A table to store datasets.
`DagScheduleDatasetReference`	References from a DAG to a dataset of which it is a consumer.
`TaskOutletDatasetReference`	References from a task to a dataset that it updates / produces.
`DatasetDagRunQueue`	Model for storing dataset events that need processing.
`DatasetEvent`	A table to store datasets events.

Attributes¶

association_table

class airflow.models.dataset.DatasetModel(uri, **kwargs)[source]¶

Bases: airflow.models.base.Base

A table to store datasets.

Parameters

uri (str) – a string that uniquely identifies the dataset
extra – JSON field for arbitrary extra info

id[source]¶

uri[source]¶

extra[source]¶

created_at[source]¶

updated_at[source]¶

is_orphaned[source]¶

consuming_dags[source]¶

producing_tasks[source]¶

__tablename__ = dataset[source]¶

__table_args__[source]¶

classmethod from_public(obj)[source]¶

__eq__(other)[source]¶

__hash__()[source]¶

__repr__()[source]¶

class airflow.models.dataset.DagScheduleDatasetReference[source]¶

Bases: airflow.models.base.Base

References from a DAG to a dataset of which it is a consumer.

dataset_id[source]¶

dag_id[source]¶

created_at[source]¶

updated_at[source]¶

dataset[source]¶

queue_records[source]¶

__tablename__ = dag_schedule_dataset_reference[source]¶

__table_args__[source]¶

__eq__(other)[source]¶

__hash__()[source]¶

__repr__()[source]¶

class airflow.models.dataset.TaskOutletDatasetReference[source]¶

Bases: airflow.models.base.Base

References from a task to a dataset that it updates / produces.

dataset_id[source]¶

dag_id[source]¶

task_id[source]¶

created_at[source]¶

updated_at[source]¶

dataset[source]¶

__tablename__ = task_outlet_dataset_reference[source]¶

__table_args__[source]¶

__eq__(other)[source]¶

__hash__()[source]¶

__repr__()[source]¶

class airflow.models.dataset.DatasetDagRunQueue[source]¶

Bases: airflow.models.base.Base

Model for storing dataset events that need processing.

dataset_id[source]¶

target_dag_id[source]¶

created_at[source]¶

__tablename__ = dataset_dag_run_queue[source]¶

__table_args__[source]¶

__eq__(other)[source]¶

__hash__()[source]¶

__repr__()[source]¶

airflow.models.dataset.association_table[source]¶

class airflow.models.dataset.DatasetEvent[source]¶

Bases: airflow.models.base.Base

A table to store datasets events.

Parameters

dataset_id – reference to DatasetModel record
extra – JSON field for arbitrary extra info
source_task_id – the task_id of the TI which updated the dataset
source_dag_id – the dag_id of the TI which updated the dataset
source_run_id – the run_id of the TI which updated the dataset
source_map_index – the map_index of the TI which updated the dataset
timestamp – the time the event was logged

We use relationships instead of foreign keys so that dataset events are not deleted even if the foreign key object is.

property uri[source]¶

id[source]¶

dataset_id[source]¶

extra[source]¶

source_task_id[source]¶

source_dag_id[source]¶

source_run_id[source]¶

source_map_index[source]¶

timestamp[source]¶

__tablename__ = dataset_event[source]¶

__table_args__[source]¶

created_dagruns[source]¶

source_task_instance[source]¶

source_dag_run[source]¶

dataset[source]¶

__repr__()[source]¶

Was this entry helpful?