airflow.providers.openlineage.utils.utils

Module Contents

Classes

InfoJsonEncodable

Airflow objects might not be json-encodable overall.

DagInfo

Defines encoding DAG object to JSON.

DagRunInfo

Defines encoding DagRun object to JSON.

TaskInstanceInfo

Defines encoding TaskInstance object to JSON.

TaskInfo

Defines encoding BaseOperator/AbstractOperator object to JSON.

TaskGroupInfo

Defines encoding TaskGroup object to JSON.

OpenLineageRedactor

This class redacts sensitive data similar to SecretsMasker in Airflow logs.

Functions

get_operator_class(task)

get_job_name(task)

get_custom_facets([task_instance])

get_fully_qualified_class_name(operator)

is_operator_disabled(operator)

is_selective_lineage_enabled(obj)

If selective enable is active check if DAG or Task is enabled to emit events.

get_airflow_run_facet(dag_run, dag, task_instance, ...)

get_unknown_source_attribute_run_facet(task[, name])

is_json_serializable(item)

print_warning(log)

get_filtered_unknown_operator_keys(operator)

normalize_sql(sql)

Attributes

log

airflow.providers.openlineage.utils.utils.log[source]
airflow.providers.openlineage.utils.utils.get_operator_class(task)[source]
airflow.providers.openlineage.utils.utils.get_job_name(task)[source]
airflow.providers.openlineage.utils.utils.get_custom_facets(task_instance=None)[source]
airflow.providers.openlineage.utils.utils.get_fully_qualified_class_name(operator)[source]
airflow.providers.openlineage.utils.utils.is_operator_disabled(operator)[source]
airflow.providers.openlineage.utils.utils.is_selective_lineage_enabled(obj)[source]

If selective enable is active check if DAG or Task is enabled to emit events.

class airflow.providers.openlineage.utils.utils.InfoJsonEncodable(obj)[source]

Bases: dict

Airflow objects might not be json-encodable overall.

The class provides additional attributes to control what and how is encoded:

  • renames: a dictionary of attribute name changes

  • casts: a dictionary consisting of attribute names
    and corresponding methods that should change
    object value

  • includes: list of attributes to be included in encoding

  • excludes: list of attributes to be excluded from encoding

Don’t use both includes and excludes.

renames: dict[str, str][source]
casts: dict[str, Any][source]
includes: list[str] = [][source]
excludes: list[str] = [][source]
class airflow.providers.openlineage.utils.utils.DagInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding DAG object to JSON.

includes = ['dag_id', 'schedule_interval', 'tags', 'start_date'][source]
casts[source]
renames[source]
class airflow.providers.openlineage.utils.utils.DagRunInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding DagRun object to JSON.

includes = ['conf', 'dag_id', 'data_interval_start', 'data_interval_end', 'external_trigger', 'run_id',...[source]
class airflow.providers.openlineage.utils.utils.TaskInstanceInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding TaskInstance object to JSON.

includes = ['duration', 'try_number', 'pool'][source]
casts[source]
class airflow.providers.openlineage.utils.utils.TaskInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding BaseOperator/AbstractOperator object to JSON.

renames[source]
includes = ['depends_on_past', 'downstream_task_ids', 'execution_timeout', 'executor_config',...[source]
casts[source]
class airflow.providers.openlineage.utils.utils.TaskGroupInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding TaskGroup object to JSON.

renames[source]
includes = ['downstream_group_ids', 'downstream_task_ids', 'prefix_group_id', 'tooltip',...[source]
airflow.providers.openlineage.utils.utils.get_airflow_run_facet(dag_run, dag, task_instance, task, task_uuid)[source]
airflow.providers.openlineage.utils.utils.get_unknown_source_attribute_run_facet(task, name=None)[source]
class airflow.providers.openlineage.utils.utils.OpenLineageRedactor[source]

Bases: airflow.utils.log.secrets_masker.SecretsMasker

This class redacts sensitive data similar to SecretsMasker in Airflow logs.

The difference is that our default max recursion depth is way higher - due to the structure of OL events we need more depth. Additionally, we allow data structures to specify data that needs not to be redacted by specifying _skip_redact list by deriving RedactMixin.

classmethod from_masker(other)[source]
airflow.providers.openlineage.utils.utils.is_json_serializable(item)[source]
airflow.providers.openlineage.utils.utils.print_warning(log)[source]
airflow.providers.openlineage.utils.utils.get_filtered_unknown_operator_keys(operator)[source]
airflow.providers.openlineage.utils.utils.normalize_sql(sql)[source]

Was this entry helpful?