`airflow.providers.openlineage.utils.utils`¶

Module Contents¶

Classes¶

`InfoJsonEncodable`	Airflow objects might not be json-encodable overall.
`DagInfo`	Defines encoding DAG object to JSON.
`DagRunInfo`	Defines encoding DagRun object to JSON.
`TaskInstanceInfo`	Defines encoding TaskInstance object to JSON.
`TaskInfo`	Defines encoding BaseOperator/AbstractOperator object to JSON.
`TaskGroupInfo`	Defines encoding TaskGroup object to JSON.
`OpenLineageRedactor`	This class redacts sensitive data similar to SecretsMasker in Airflow logs.

Functions¶

`openlineage_job_name`(dag_id, task_id)
`get_operator_class`(task)
`to_json_encodable`(task)
`url_to_https`(url)
`redacted_connection_uri`(conn[, filtered_params, ...])	Return the connection URI for the given Connection.
`get_connection`(conn_id)
`get_job_name`(task)
`get_custom_facets`([task_instance])
`get_airflow_run_facet`(dag_run, dag, task_instance, ...)
`is_json_serializable`(item)
`print_warning`(log)
`is_source_enabled`()
`get_filtered_unknown_operator_keys`(operator)
`normalize_sql`(sql)

Attributes¶

log

airflow.providers.openlineage.utils.utils.log[source]¶

airflow.providers.openlineage.utils.utils.openlineage_job_name(dag_id, task_id)[source]¶

airflow.providers.openlineage.utils.utils.get_operator_class(task)[source]¶

airflow.providers.openlineage.utils.utils.to_json_encodable(task)[source]¶

airflow.providers.openlineage.utils.utils.url_to_https(url)[source]¶

airflow.providers.openlineage.utils.utils.redacted_connection_uri(conn, filtered_params=None, filtered_prefixes=None)[source]¶

Return the connection URI for the given Connection.

This method additionally filters URI by removing query parameters that are known to carry sensitive data like username, password, access key.

airflow.providers.openlineage.utils.utils.get_connection(conn_id)[source]¶

airflow.providers.openlineage.utils.utils.get_job_name(task)[source]¶

airflow.providers.openlineage.utils.utils.get_custom_facets(task_instance=None)[source]¶

class airflow.providers.openlineage.utils.utils.InfoJsonEncodable(obj)[source]¶

Bases: dict

Airflow objects might not be json-encodable overall.

The class provides additional attributes to control what and how is encoded:

renames: a dictionary of attribute name changes
casts: a dictionary consisting of attribute names

and corresponding methods that should change

object value
includes: list of attributes to be included in encoding
excludes: list of attributes to be excluded from encoding

Don’t use both includes and excludes.

renames: dict[str, str][source]¶

casts: dict[str, Any][source]¶

includes: list[str] = [][source]¶

excludes: list[str] = [][source]¶

class airflow.providers.openlineage.utils.utils.DagInfo(obj)[source]¶

Bases: InfoJsonEncodable

Defines encoding DAG object to JSON.

includes = ['dag_id', 'schedule_interval', 'tags', 'start_date'][source]¶

casts[source]¶

renames[source]¶

class airflow.providers.openlineage.utils.utils.DagRunInfo(obj)[source]¶

Bases: InfoJsonEncodable

Defines encoding DagRun object to JSON.

includes = ['conf', 'dag_id', 'data_interval_start', 'data_interval_end', 'external_trigger', 'run_id',...[source]¶

class airflow.providers.openlineage.utils.utils.TaskInstanceInfo(obj)[source]¶

Bases: InfoJsonEncodable

Defines encoding TaskInstance object to JSON.

includes = ['duration', 'try_number', 'pool'][source]¶

casts[source]¶

class airflow.providers.openlineage.utils.utils.TaskInfo(obj)[source]¶

Bases: InfoJsonEncodable

Defines encoding BaseOperator/AbstractOperator object to JSON.

renames[source]¶

excludes = ['_BaseOperator__instantiated', '_dag', '_hook', '_log', '_outlets', '_inlets',...[source]¶

casts[source]¶

class airflow.providers.openlineage.utils.utils.TaskGroupInfo(obj)[source]¶

Bases: InfoJsonEncodable

Defines encoding TaskGroup object to JSON.

renames[source]¶

includes = ['downstream_group_ids', 'downstream_task_ids', 'prefix_group_id', 'tooltip',...[source]¶

airflow.providers.openlineage.utils.utils.get_airflow_run_facet(dag_run, dag, task_instance, task, task_uuid)[source]¶

class airflow.providers.openlineage.utils.utils.OpenLineageRedactor[source]¶

Bases: airflow.utils.log.secrets_masker.SecretsMasker

This class redacts sensitive data similar to SecretsMasker in Airflow logs.

The difference is that our default max recursion depth is way higher - due to the structure of OL events we need more depth. Additionally, we allow data structures to specify data that needs not to be redacted by specifying _skip_redact list by deriving RedactMixin.

classmethod from_masker(other)[source]¶

airflow.providers.openlineage.utils.utils.is_json_serializable(item)[source]¶

airflow.providers.openlineage.utils.utils.print_warning(log)[source]¶

airflow.providers.openlineage.utils.utils.is_source_enabled()[source]¶

airflow.providers.openlineage.utils.utils.get_filtered_unknown_operator_keys(operator)[source]¶

airflow.providers.openlineage.utils.utils.normalize_sql(sql)[source]¶

airflow.providers.openlineage.utils.utils¶

Module Contents¶

Classes¶

Functions¶

Attributes¶

`airflow.providers.openlineage.utils.utils`¶