airflow.providers.snowflake.transfers.copy_into_snowflake

Abstract operator that child classes implement COPY INTO <TABLE> SQL in Snowflake.

Module Contents

Classes

CopyFromExternalStageToSnowflakeOperator

Executes a COPY INTO command to load files from an external stage from clouds to Snowflake.
class airflow.providers.snowflake.transfers.copy_into_snowflake.CopyFromExternalStageToSnowflakeOperator(*, files=None, table, stage, prefix=None, file_format, schema=None, columns_array=None, pattern=None, warehouse=None, database=None, autocommit=True, snowflake_conn_id='snowflake_default', role=None, authenticator=None, session_parameters=None, copy_options=None, validation_mode=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Executes a COPY INTO command to load files from an external stage from clouds to Snowflake.

This operator requires the snowflake_conn_id connection. The snowflake host, login, and, password field must be setup in the connection. Other inputs can be defined in the connection or hook instantiation.

Parameters

  • namespace – snowflake namespace

  • table (str) – snowflake table

  • file_format (str) – file format name i.e. CSV, AVRO, etc

  • stage (str) – reference to a specific snowflake stage. If the stage’s schema is not the same as the table one, it must be specified

  • prefix (str | None) – cloud storage location specified to limit the set of files to load

  • files (list | None) – files to load into table

  • pattern (str | None) – pattern to load files from external location to table

  • copy_into_postifx – optional sql postfix for INSERT INTO query such as formatTypeOptions and copyOptions

  • snowflake_conn_id (str) – Reference to Snowflake connection id

  • account – snowflake account name

  • warehouse (str | None) – name of snowflake warehouse

  • database (str | None) – name of snowflake database

  • region – name of snowflake region

  • role (str | None) – name of snowflake role

  • schema (str | None) – name of snowflake schema

  • authenticator (str | None) – authenticator for Snowflake. ‘snowflake’ (default) to use the internal Snowflake authenticator ‘externalbrowser’ to authenticate using your web browser and Okta, ADFS or any other SAML 2.0-compliant identify provider (IdP) that has been defined for your account https://<your_okta_account_name>.okta.com to authenticate through native Okta.

  • session_parameters (dict | None) – You can set session-level parameters at the time you connect to Snowflake

  • copy_options (str | None) – snowflake COPY INTO syntax copy options

  • validation_mode (str | None) – snowflake COPY INTO syntax validation mode

template_fields: Sequence[str] = ('files',)[source]
template_fields_renderers[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_openlineage_facets_on_complete(task_instance)[source]

Implement _on_complete because we rely on return value of a query.

Was this entry helpful?