• Home
• airflow.providers.openlineage
• airflow.providers.openlineage.utils
• airflow.providers.openlineage.utils.emission_policy

↑↓ Navigate ⏎ Select Esc Close

airflow.providers.openlineage.utils.emission_policy

Per-scope emission policy resolution for the OpenLineage provider.

The emission_policy Airflow configuration option ([openlineage] section) accepts a JSON array of rule objects. Each rule has the shape:

{
    "scope":      {<scope keys>},        # required, may be empty for "global"
    "match_mode": "exact" | "regex",     # optional, default "exact"
    "controls":   {<control flags>},     # required, must be non-empty
    "locked":     true | false           # optional, default false
}

Top-level keys outside the four above are rejected with a WARNING and the rule is skipped.

Scope keys (all optional inside scope):

• operator — fully-qualified operator class name; matches task events for that operator type.

• dag_id — matches task and/or dag run events for all tasks / dag runs belonging to the named DAG. Use emit_task_events / emit_dag_events to target one event type selectively.

• task_id — only valid alongside dag_id; targets a specific task.

• hook: fully-qualified class name of the object that reported a hook-collected dataset (usually a hook). Can be added to any of the scopes above, and narrows the rule to datasets reported by that hook. Only exclude_datasets and hook_lineage are valid controls for a hook-scoped rule.

• (empty ``scope: {}``) — global default override; applies to every task and DAG event.

operator cannot be combined with dag_id or task_id (scope is one of: global, operator-only, dag-only, dag+task, each optionally with hook). task_id without dag_id and operator combined with emit_dag_events (or task_id combined with emit_dag_events) are also rejected with a WARNING.

match_mode lives at the top level: "exact" (default) or "regex". When set to "regex", every value inside scope (dag_id, task_id, operator, hook) is treated as a re.fullmatch pattern.

Control flag keys (all optional inside controls; the dict must be non-empty):

• emit — shorthand: disable ALL OpenLineage events in scope (both task and dag run events). Default: true.

• emit_task_events — disable task-level events only; takes precedence over emit for task event decisions. Default: true.

• emit_dag_events — disable dag-run-level events only; takes precedence over emit for dag event decisions. Default: true.

• extract_operator_metadata — whether to run operator-specific extractor-based metadata collection. When true, the extractor manager calls the operator’s OpenLineage extractor (if registered), which may produce dataset inputs/outputs, job facets, run facets, and other operator-specific metadata. When false, the entire extraction pipeline is skipped and a minimal event is emitted. Only meaningful for task events. Default: true.

• include_source_code — whether to include operator source code in the SourceCodeJobFacet for Python and Bash operators. Only meaningful for task events when extract_operator_metadata is also true. Default: true.

• hook_lineage — whether to use HookLineageCollector as a fallback when the extractor finds no inputs/outputs. Has no effect when ``extract_operator_metadata`` is ``false`` — the entire extraction pipeline (including hook lineage) is skipped. Only meaningful for task events. Default: true. In a hook-scoped rule it instead keeps (true) or drops (false) every dataset reported by the matching hook, and it cannot be combined with exclude_datasets in the same rule.

• include_full_task_info — whether to include the full serialized operator state in the AirflowRunFacet. When false (default), only a curated subset of task attributes is sent. Only meaningful for task events. Default: false.

• exclude_datasets: list of re.fullmatch patterns (independent of match_mode) matched against "<namespace>/<name>" of each OpenLineage dataset (without a doubled / when the name starts with one); matching datasets are dropped from task events and from the SQL hook lineage events of the task. Applies to extractor, hook-collected and manually declared lineage. Default: [].

Unlike the boolean controls, exclude_datasets is resolved per dataset, and the most specific matching rule’s list replaces broader lists instead of merging with them, so a narrower rule can shorten or clear ([]) a broader exclusion.

locked: true is an admin floor lock that prevents per-Dag / per-task authoring overrides from changing the field(s) carried by this rule’s controls dict.

Flag hierarchy

Flags are not fully independent — some only take effect when a higher-level flag is enabled:

• extract_operator_metadata: false skips the entire operator extraction pipeline. When this is false, both include_source_code and hook_lineage have no effect regardless of their values, because the code paths they control are never reached.

• include_source_code only applies inside Python and Bash operator extractors; setting it to false on other operator types is a no-op.

Priority for task events (most specific tier wins; within a tier, last matching rule wins):

1. dag_id + task_id

2. dag_id

3. operator

4. Global (no match keys)

5. Built-in defaults

For emit at the task level: emit_task_events beats emit within the same rule.

For exclude_datasets on a dataset reported by a hook, each tier above is preceded by its hook-scoped variant: dag_id + task_id + hook, dag_id + task_id, dag_id + hook, dag_id, operator + hook, operator, hook, global. Datasets from other sources only see the tiers without hook.

Priority for DAG run events:

1. dag_id rule (using emit_dag_events or emit)

2. Global (no match keys)

3. Built-in defaults

For emit at the dag level: emit_dag_events beats emit within the same rule.

Legacy config translation

Any active legacy config option (disabled_for_operators, disable_source_code, include_full_task_info, selective_enable) is always translated into equivalent emission_policy rules, regardless of whether emission_policy itself is set. The translated rules are prepended before any user-provided rules, so user rules win within each priority tier (last-wins). A DeprecationWarning is issued listing every translated option — silence it by migrating those options into emission_policy exclusively.

When no legacy option is active and emission_policy is empty, the resolver returns the built-in defaults with no warnings.

Audit logging

Every resolved field that is non-default — whether from a rule or from a legacy translation — is logged at INFO level, identifying the field, the event context, and the exact rule that caused the change.

Attributes

DatasetT
log
V
ControlValue
EMIT
EMIT_TASK_EVENTS
EMIT_DAG_EVENTS
EXTRACT_OPERATOR_METADATA
INCLUDE_SOURCE_CODE
HOOK_LINEAGE
INCLUDE_FULL_TASK_INFO
EXCLUDE_DATASETS
SCOPE_DAG_ID
SCOPE_TASK_ID
SCOPE_OPERATOR
SCOPE_HOOK
RULE_SCOPE
RULE_CONTROLS
RULE_MATCH_MODE
RULE_LOCKED
OL_EMISSION_POLICY_PARAM

Classes

EmissionPolicy	Resolved emission policy for a specific event context.
Rule	A single, fully-validated emission_policy rule.
DatasetFilter	exclude_datasets resolution for the datasets of one task.

Functions

get_hook_class_name(hook)	Return the fully-qualified class name that the hook scope key matches, if any.
find_invalid_pattern(field_name, value)	Return an error message if value is not a list of valid regex patterns, else None.
resolve_task_emission_policy(operator, dag_id, task_id)	Resolve the emission policy for a task-level event.
resolve_dag_emission_policy(dag_id[, dag])	Resolve the emission policy for a DAG-level event.

Module Contents

airflow.providers.openlineage.utils.emission_policy.DatasetT

airflow.providers.openlineage.utils.emission_policy.log

airflow.providers.openlineage.utils.emission_policy.V

airflow.providers.openlineage.utils.emission_policy.ControlValue

airflow.providers.openlineage.utils.emission_policy.EMIT = 'emit'

airflow.providers.openlineage.utils.emission_policy.EMIT_TASK_EVENTS = 'emit_task_events'

airflow.providers.openlineage.utils.emission_policy.EMIT_DAG_EVENTS = 'emit_dag_events'

airflow.providers.openlineage.utils.emission_policy.EXTRACT_OPERATOR_METADATA = 'extract_operator_metadata'

airflow.providers.openlineage.utils.emission_policy.INCLUDE_SOURCE_CODE = 'include_source_code'

airflow.providers.openlineage.utils.emission_policy.HOOK_LINEAGE = 'hook_lineage'

airflow.providers.openlineage.utils.emission_policy.INCLUDE_FULL_TASK_INFO = 'include_full_task_info'

airflow.providers.openlineage.utils.emission_policy.EXCLUDE_DATASETS = 'exclude_datasets'

airflow.providers.openlineage.utils.emission_policy.SCOPE_DAG_ID = 'dag_id'

airflow.providers.openlineage.utils.emission_policy.SCOPE_TASK_ID = 'task_id'

airflow.providers.openlineage.utils.emission_policy.SCOPE_OPERATOR = 'operator'

airflow.providers.openlineage.utils.emission_policy.SCOPE_HOOK = 'hook'

airflow.providers.openlineage.utils.emission_policy.RULE_SCOPE = 'scope'

airflow.providers.openlineage.utils.emission_policy.RULE_CONTROLS = 'controls'

airflow.providers.openlineage.utils.emission_policy.RULE_MATCH_MODE = 'match_mode'

airflow.providers.openlineage.utils.emission_policy.RULE_LOCKED = 'locked'

airflow.providers.openlineage.utils.emission_policy.OL_EMISSION_POLICY_PARAM = '_openlineage_emission_policy'

class airflow.providers.openlineage.utils.emission_policy.EmissionPolicy

Resolved emission policy for a specific event context.

emit: bool

extract_operator_metadata: bool

include_source_code: bool

hook_lineage: bool

include_full_task_info: bool

dataset_filter: DatasetFilter

classmethod defaults()

Return the default policy (all controls at their built-in defaults).

class airflow.providers.openlineage.utils.emission_policy.Rule

A single, fully-validated emission_policy rule.

Instances are produced exclusively by _parse_rule(), so holding a Rule is itself the “this passed schema validation” guarantee — downstream resolution never re-validates. scope / controls are the raw (validated) sub-dicts.

scope: dict[str, str]

controls: dict[str, ControlValue]

match_mode: str = 'exact'

locked: bool = False

property has_dag: bool

property has_task: bool

property has_op: bool

property has_hook: bool

airflow.providers.openlineage.utils.emission_policy.get_hook_class_name(hook)

Return the fully-qualified class name that the hook scope key matches, if any.

class airflow.providers.openlineage.utils.emission_policy.DatasetFilter

exclude_datasets resolution for the datasets of one task.

Hook-scoped rules can only be matched once the reporting hook is known, so the winning patterns are resolved lazily per hook (None for datasets not reported by a hook) and cached. tiers holds only rules that set patterns, most specific first: dag+task+hook, dag+task, dag+hook, dag, operator+hook, operator, hook, global.

tiers: tuple[tuple[Rule, ...], ...] = ()

authoring_patterns: tuple[str, ...] | None = None

context: str = ''

get_patterns(hook=None)

Return the exclusion patterns that apply to datasets reported by hook.

is_excluded(dataset, hook=None)

Return True if dataset, reported by hook (if any), matches an exclusion pattern.

exclude(datasets, hook=None)

Return datasets without the ones excluded for hook.

airflow.providers.openlineage.utils.emission_policy.find_invalid_pattern(field_name, value)

Return an error message if value is not a list of valid regex patterns, else None.

Shared by the conf parser (which skips the offending rule) and the authoring API (which raises), so every pattern that reaches dataset matching has compiled at least once.

airflow.providers.openlineage.utils.emission_policy.resolve_task_emission_policy(operator, dag_id, task_id)

Resolve the emission policy for a task-level event.

This is the single authoritative entry point for task event emission decisions.

Any active legacy options (disabled_for_operators, disable_source_code, include_full_task_info, selective_enable) are always translated into equivalent emission_policy rules, prepended to the user-provided rules so user rules win within each tier (last-wins). A DeprecationWarning is issued whenever a legacy option contributes a rule — silence it by migrating those options into emission_policy exclusively.

Never raises: any failure (e.g. a misconfigured emission_policy) is logged and the built-in defaults are returned, so a bad config degrades to “emit with defaults” rather than crashing the listener’s task-notification path.

Parameters:

• operator (airflow.providers.openlineage.utils.utils.AnyOperator) – The Airflow operator/task object.

• dag_id (str) – The DAG ID for this task instance.

• task_id (str) – The task ID for this task instance.

airflow.providers.openlineage.utils.emission_policy.resolve_dag_emission_policy(dag_id, dag=None)

Resolve the emission policy for a DAG-level event.

Returns a full EmissionPolicy for a uniform resolver API across task and DAG scopes. Today only emit governs DAG-run events; the remaining fields (extract_operator_metadata, include_source_code, hook_lineage, include_full_task_info) are reserved — they carry their built-in defaults and are not yet meaningful at DAG scope (no extraction happens at the DAG level). The unified return type keeps room for future DAG-level controls (e.g. terminal-event filtering) without re-widening the API.

Any active legacy options (currently only selective_enable affects DAG events) are always translated into equivalent emission_policy rules, prepended to the user-provided rules so user rules win within each tier (last-wins). A DeprecationWarning is issued whenever a legacy option contributes a rule.

Never raises: any failure (e.g. a misconfigured emission_policy) is logged and the built-in defaults are returned, so a bad config degrades to “emit with defaults” rather than crashing the listener’s dag-run-notification path.

Parameters:

• dag_id (str) – The DAG ID for the dag run event.

• dag (object | None) – Optional DAG object used for the selective_enable check / translation.

Previous Next

Was this entry helpful?