airflow.models.dagbag

Module Contents

class airflow.models.dagbag.FileLoadStat[source]

Bases: typing.NamedTuple

Information about single file

file :str[source]
duration :timedelta[source]
dag_num :int[source]
task_num :int[source]
dags :str[source]
class airflow.models.dagbag.DagBag(dag_folder: Union[str, ‘pathlib.Path’, None] = None, include_examples: bool = conf.getboolean('core', 'LOAD_EXAMPLES'), include_smart_sensor: bool = conf.getboolean('smart_sensor', 'USE_SMART_SENSOR'), safe_mode: bool = conf.getboolean('core', 'DAG_DISCOVERY_SAFE_MODE'), read_dags_from_db: bool = False, store_serialized_dags: Optional[bool] = None, load_op_links: bool = True)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

A dagbag is a collection of dags, parsed out of a folder tree and has high level configuration settings, like what database to use as a backend and what executor to use to fire off tasks. This makes it easier to run distinct environments for say production and development, tests, or for different teams or security profiles. What would have been system level settings are now dagbag level so that one system can run multiple, independent settings sets.

Parameters

  • dag_folder (unicode) – the folder to scan to find DAGs

  • include_examples (bool) – whether to include the examples that ship with airflow or not

  • include_smart_sensor (bool) – whether to include the smart sensor native DAGs that create the smart sensor operators for whole cluster

  • read_dags_from_db (bool) – Read DAGs from DB if True is passed. If False DAGs are read from python files.

  • load_op_links (bool) – Should the extra operator link be loaded via plugins when de-serializing the DAG? This flag is set to False in Scheduler so that Extra Operator links are not loaded to not run User code in Scheduler.

DAGBAG_IMPORT_TIMEOUT[source]
SCHEDULER_ZOMBIE_TASK_THRESHOLD[source]
store_serialized_dags[source]

Whether or not to read dags from DB

dag_ids[source]
Returns

a list of DAG IDs in this bag

Return type

List[unicode]

size(self)[source]
Returns

the amount of dags contained in this dagbag

get_dag(self, dag_id, session: Session = None)[source]

Gets the DAG out of the dictionary, and refreshes it if expired

Parameters

dag_id (str) – DAG Id

_add_dag_from_db(self, dag_id: str, session: Session)[source]

Add DAG to DagBag from DB

process_file(self, filepath, only_if_updated=True, safe_mode=True)[source]

Given a path to a python module or zip file, this method imports the module and look for dag objects within it.

_load_modules_from_file(self, filepath, safe_mode)[source]
_load_modules_from_zip(self, filepath, safe_mode)[source]
_process_modules(self, filepath, mods, file_last_changed_on_disk)[source]
bag_dag(self, dag, root_dag)[source]

Adds the DAG into the bag, recurses into sub dags.

Raises

AirflowDagCycleException if a cycle is detected in this dag or its subdags.

Raises

AirflowDagDuplicatedIdException if this dag or its subdags already exists in the bag.

_bag_dag(self, *, dag, root_dag, recursive)[source]

Actual implementation of bagging a dag.

The only purpose of this is to avoid exposing recursive in bag_dag(), intended to only be used by the _bag_dag() implementation.

collect_dags(self, dag_folder: Union[str, ‘pathlib.Path’, None] = None, only_if_updated: bool = True, include_examples: bool = conf.getboolean('core', 'LOAD_EXAMPLES'), include_smart_sensor: bool = conf.getboolean('smart_sensor', 'USE_SMART_SENSOR'), safe_mode: bool = conf.getboolean('core', 'DAG_DISCOVERY_SAFE_MODE'))[source]

Given a file path or a folder, this method looks for python modules, imports them and adds them to the dagbag collection.

Note that if a .airflowignore file is found while processing the directory, it will behave much like a .gitignore, ignoring files that match any of the regex patterns specified in the file.

Note: The patterns in .airflowignore are treated as un-anchored regexes, not shell-like glob patterns.

collect_dags_from_db(self)[source]

Collects DAGs from database.

dagbag_report(self)[source]

Prints a report around DagBag loading stats

sync_to_db(self, session: Optional[Session] = None)[source]

Save attributes about list of DAG to the DB.

_sync_perm_for_dag(self, dag, session: Optional[Session] = None)[source]

Sync DAG specific permissions, if necessary

Was this entry helpful?