airflow.providers.openlineage.utils.utils

Module Contents

Classes

InfoJsonEncodable

Airflow objects might not be json-encodable overall.

DagInfo

Defines encoding DAG object to JSON.

DagRunInfo

Defines encoding DagRun object to JSON.

TaskInstanceInfo

Defines encoding TaskInstance object to JSON.

DatasetInfo

Defines encoding Airflow Dataset object to JSON.

TaskInfo

Defines encoding BaseOperator/AbstractOperator object to JSON.

TaskGroupInfo

Defines encoding TaskGroup object to JSON.

OpenLineageRedactor

This class redacts sensitive data similar to SecretsMasker in Airflow logs.

Functions

try_import_from_string(string)

get_operator_class(task)

get_job_name(task)

get_custom_facets([task_instance])

get_fully_qualified_class_name(operator)

is_operator_disabled(operator)

is_selective_lineage_enabled(obj)

If selective enable is active check if DAG or Task is enabled to emit events.

get_airflow_run_facet(dag_run, dag, task_instance, ...)

get_airflow_job_facet(dag_run)

get_airflow_state_run_facet(dag_run)

get_unknown_source_attribute_run_facet(task[, name])

is_json_serializable(item)

print_warning(log)

get_filtered_unknown_operator_keys(operator)

normalize_sql(sql)

should_use_external_connection(hook)

Attributes

log

airflow.providers.openlineage.utils.utils.log[source]
airflow.providers.openlineage.utils.utils.try_import_from_string(string)[source]
airflow.providers.openlineage.utils.utils.get_operator_class(task)[source]
airflow.providers.openlineage.utils.utils.get_job_name(task)[source]
airflow.providers.openlineage.utils.utils.get_custom_facets(task_instance=None)[source]
airflow.providers.openlineage.utils.utils.get_fully_qualified_class_name(operator)[source]
airflow.providers.openlineage.utils.utils.is_operator_disabled(operator)[source]
airflow.providers.openlineage.utils.utils.is_selective_lineage_enabled(obj)[source]

If selective enable is active check if DAG or Task is enabled to emit events.

class airflow.providers.openlineage.utils.utils.InfoJsonEncodable(obj)[source]

Bases: dict

Airflow objects might not be json-encodable overall.

The class provides additional attributes to control what and how is encoded:

  • renames: a dictionary of attribute name changes

  • casts: a dictionary consisting of attribute names
    and corresponding methods that should change
    object value

  • includes: list of attributes to be included in encoding

  • excludes: list of attributes to be excluded from encoding

Don’t use both includes and excludes.

renames: dict[str, str][source]
casts: dict[str, Any][source]
includes: list[str] = [][source]
excludes: list[str] = [][source]
class airflow.providers.openlineage.utils.utils.DagInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding DAG object to JSON.

includes = ['dag_id', 'description', 'owner', 'schedule_interval', 'start_date', 'tags'][source]
casts[source]
renames[source]
class airflow.providers.openlineage.utils.utils.DagRunInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding DagRun object to JSON.

includes = ['conf', 'dag_id', 'data_interval_start', 'data_interval_end', 'external_trigger', 'run_id',...[source]
class airflow.providers.openlineage.utils.utils.TaskInstanceInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding TaskInstance object to JSON.

includes = ['duration', 'try_number', 'pool', 'queued_dttm'][source]
casts[source]
class airflow.providers.openlineage.utils.utils.DatasetInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding Airflow Dataset object to JSON.

includes = ['uri', 'extra'][source]
class airflow.providers.openlineage.utils.utils.TaskInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding BaseOperator/AbstractOperator object to JSON.

renames[source]
includes = ['depends_on_past', 'downstream_task_ids', 'execution_timeout', 'executor_config',...[source]
casts[source]
class airflow.providers.openlineage.utils.utils.TaskGroupInfo(obj)[source]

Bases: InfoJsonEncodable

Defines encoding TaskGroup object to JSON.

renames[source]
includes = ['downstream_group_ids', 'downstream_task_ids', 'prefix_group_id', 'tooltip',...[source]
airflow.providers.openlineage.utils.utils.get_airflow_run_facet(dag_run, dag, task_instance, task, task_uuid)[source]
airflow.providers.openlineage.utils.utils.get_airflow_job_facet(dag_run)[source]
airflow.providers.openlineage.utils.utils.get_airflow_state_run_facet(dag_run)[source]
airflow.providers.openlineage.utils.utils.get_unknown_source_attribute_run_facet(task, name=None)[source]
class airflow.providers.openlineage.utils.utils.OpenLineageRedactor[source]

Bases: airflow.utils.log.secrets_masker.SecretsMasker

This class redacts sensitive data similar to SecretsMasker in Airflow logs.

The difference is that our default max recursion depth is way higher - due to the structure of OL events we need more depth. Additionally, we allow data structures to specify data that needs not to be redacted by specifying _skip_redact list by deriving RedactMixin.

classmethod from_masker(other)[source]
airflow.providers.openlineage.utils.utils.is_json_serializable(item)[source]
airflow.providers.openlineage.utils.utils.print_warning(log)[source]
airflow.providers.openlineage.utils.utils.get_filtered_unknown_operator_keys(operator)[source]
airflow.providers.openlineage.utils.utils.normalize_sql(sql)[source]
airflow.providers.openlineage.utils.utils.should_use_external_connection(hook)[source]

Was this entry helpful?