airflow.providers.common.sql.operators.generic_transfer¶

Classes¶

GenericTransfer

Moves data from a connection to another.

Module Contents¶

class airflow.providers.common.sql.operators.generic_transfer.GenericTransfer(*, sql, destination_table, source_conn_id, source_hook_params=None, destination_conn_id, destination_hook_params=None, preoperator=None, insert_args=None, page_size=None, paginated_sql_statement_clause=None, **kwargs)[source]¶

Bases: airflow.providers.common.sql.version_compat.BaseOperator

Moves data from a connection to another.

Assuming that they both provide the required methods in their respective hooks. The source hook needs to expose a get_records method, and the destination a insert_rows method.

This is meant to be used on small-ish datasets that fit in memory.

Parameters:

sql (str) – SQL query to execute against the source database. (templated)
destination_table (str) – target table. (templated)
source_conn_id (str) – source connection. (templated)
source_hook_params (dict | None) – source hook parameters.
destination_conn_id (str) – destination connection. (templated)
destination_hook_params (dict | None) – destination hook parameters.
preoperator (str | list[str] | None) – sql statement or list of statements to be executed prior to loading the data. (templated)
insert_args (dict | None) – extra params for insert_rows method.
page_size (int | None) – number of records to be read in paginated mode (optional).
paginated_sql_statement_clause (str | None) – SQL statement clause to be used for pagination (optional).

template_fields: collections.abc.Sequence[str] = ('source_conn_id', 'destination_conn_id', 'sql', 'destination_table', 'preoperator',...[source]¶

template_ext: collections.abc.Sequence[str] = ('.sql', '.hql')[source]¶

template_fields_renderers[source]¶

ui_color = '#b0f07c'[source]¶

sql[source]¶

destination_table[source]¶

source_conn_id[source]¶

source_hook_params = None[source]¶

destination_conn_id[source]¶

destination_hook_params = None[source]¶

preoperator = None[source]¶

insert_args[source]¶

page_size = None[source]¶

paginated_sql_statement_clause = '{} LIMIT {} OFFSET {}'[source]¶

classmethod get_hook(conn_id, hook_params=None)[source]¶

Return DbApiHook for this connection id.

Parameters:

conn_id (str) – connection id
hook_params (dict | None) – hook parameters

Returns:

DbApiHook for this connection

Return type:

airflow.providers.common.sql.hooks.sql.DbApiHook

property source_hook: airflow.providers.common.sql.hooks.sql.DbApiHook[source]¶

property destination_hook: airflow.providers.common.sql.hooks.sql.DbApiHook[source]¶

get_paginated_sql(offset)[source]¶

Format the paginated SQL statement using the current format.

render_template_fields(context, jinja_env=None)[source]¶

Template all attributes listed in self.template_fields.

This mutates the attributes in-place and is irreversible.

Parameters:

context (airflow.sdk.definitions.context.Context) – Context dict with values to apply on content.
jinja_env (jinja2.Environment | None) – Jinja’s environment to use for rendering.

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶