airflow.contrib.operators.gcp_sql_operator

Module Contents

airflow.contrib.operators.gcp_sql_operator.SETTINGS = settings

airflow.contrib.operators.gcp_sql_operator.SETTINGS_VERSION = settingsVersion

airflow.contrib.operators.gcp_sql_operator.CLOUD_SQL_CREATE_VALIDATION

airflow.contrib.operators.gcp_sql_operator.CLOUD_SQL_EXPORT_VALIDATION

airflow.contrib.operators.gcp_sql_operator.CLOUD_SQL_IMPORT_VALIDATION

airflow.contrib.operators.gcp_sql_operator.CLOUD_SQL_DATABASE_CREATE_VALIDATION

airflow.contrib.operators.gcp_sql_operator.CLOUD_SQL_DATABASE_PATCH_VALIDATION

class airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator(instance, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', *args, **kwargs)

Bases: airflow.models.BaseOperator

Abstract base operator for Google Cloud SQL operators to inherit from.

Parameters

• instance (str) – Cloud SQL instance ID. This does not include the project ID.

• project_id (str) – Optional, Google Cloud Platform Project ID. f set to None or missing, the default project_id from the GCP connection is used.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

_validate_inputs(self)

_check_if_instance_exists(self, instance)

_check_if_db_exists(self, db_name)

execute(self, context)

static _get_settings_version(instance)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlInstanceCreateOperator(body, instance, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs)

Bases: airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator

Creates a new Cloud SQL instance. If an instance with the same name exists, no action will be taken and the operator will succeed.

See also

For more information on how to use this operator, take a look at the guide: CloudSqlInstanceCreateOperator

Parameters

• body (dict) – Body required by the Cloud SQL insert API, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/insert #request-body

• instance (str) – Cloud SQL instance ID. This does not include the project ID.

• project_id (str) – Optional, Google Cloud Platform Project ID. If set to None or missing, the default project_id from the GCP connection is used.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

• validate_body (bool) – True if body should be validated, False otherwise.

template_fields = ['project_id', 'instance', 'gcp_conn_id', 'api_version']

_validate_inputs(self)

_validate_body_fields(self)

execute(self, context)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlInstancePatchOperator(body, instance, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', *args, **kwargs)

Bases: airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator

Updates settings of a Cloud SQL instance.

Caution: This is a partial update, so only included values for the settings will be updated.

In the request body, supply the relevant portions of an instance resource, according to the rules of patch semantics. https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch

See also

For more information on how to use this operator, take a look at the guide: CloudSqlInstancePatchOperator

Parameters

• body (dict) – Body required by the Cloud SQL patch API, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/patch#request-body

• instance (str) – Cloud SQL instance ID. This does not include the project ID.

• project_id (str) – Optional, Google Cloud Platform Project ID. If set to None or missing, the default project_id from the GCP connection is used.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

template_fields = ['project_id', 'instance', 'gcp_conn_id', 'api_version']

_validate_inputs(self)

execute(self, context)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlInstanceDeleteOperator(instance, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', *args, **kwargs)

Bases: airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator

Deletes a Cloud SQL instance.

See also

For more information on how to use this operator, take a look at the guide: CloudSqlInstanceDeleteOperator

Parameters

• instance (str) – Cloud SQL instance ID. This does not include the project ID.

• project_id (str) – Optional, Google Cloud Platform Project ID. If set to None or missing, the default project_id from the GCP connection is used.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

template_fields = ['project_id', 'instance', 'gcp_conn_id', 'api_version']

execute(self, context)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlInstanceDatabaseCreateOperator(instance, body, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs)

Bases: airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator

Creates a new database inside a Cloud SQL instance.

See also

For more information on how to use this operator, take a look at the guide: CloudSqlInstanceDatabaseCreateOperator

Parameters

• instance (str) – Database instance ID. This does not include the project ID.

• body (dict) – The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body

• project_id (str) – Optional, Google Cloud Platform Project ID. If set to None or missing, the default project_id from the GCP connection is used.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

• validate_body (bool) – Whether the body should be validated. Defaults to True.

template_fields = ['project_id', 'instance', 'gcp_conn_id', 'api_version']

_validate_inputs(self)

_validate_body_fields(self)

execute(self, context)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlInstanceDatabasePatchOperator(instance, database, body, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs)

Bases: airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator

Updates a resource containing information about a database inside a Cloud SQL instance using patch semantics. See: https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch

See also

For more information on how to use this operator, take a look at the guide: CloudSqlInstanceDatabasePatchOperator

Parameters

• instance (str) – Database instance ID. This does not include the project ID.

• database (str) – Name of the database to be updated in the instance.

• body (dict) – The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/patch#request-body

• project_id (str) – Optional, Google Cloud Platform Project ID.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

• validate_body (bool) – Whether the body should be validated. Defaults to True.

template_fields = ['project_id', 'instance', 'database', 'gcp_conn_id', 'api_version']

_validate_inputs(self)

_validate_body_fields(self)

execute(self, context)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlInstanceDatabaseDeleteOperator(instance, database, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', *args, **kwargs)

Bases: airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator

Deletes a database from a Cloud SQL instance.

See also

For more information on how to use this operator, take a look at the guide: CloudSqlInstanceDatabaseDeleteOperator

Parameters

• instance (str) – Database instance ID. This does not include the project ID.

• database (str) – Name of the database to be deleted in the instance.

• project_id (str) – Optional, Google Cloud Platform Project ID. If set to None or missing, the default project_id from the GCP connection is used.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

template_fields = ['project_id', 'instance', 'database', 'gcp_conn_id', 'api_version']

_validate_inputs(self)

execute(self, context)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlInstanceExportOperator(instance, body, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs)

Bases: airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator

Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.

Note: This operator is idempotent. If executed multiple times with the same export file URI, the export file in GCS will simply be overridden.

See also

For more information on how to use this operator, take a look at the guide: CloudSqlInstanceImportOperator

Parameters

• instance (str) – Cloud SQL instance ID. This does not include the project ID.

• body (dict) – The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body

• project_id (str) – Optional, Google Cloud Platform Project ID. If set to None or missing, the default project_id from the GCP connection is used.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

• validate_body (bool) – Whether the body should be validated. Defaults to True.

template_fields = ['project_id', 'instance', 'gcp_conn_id', 'api_version']

_validate_inputs(self)

_validate_body_fields(self)

execute(self, context)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlInstanceImportOperator(instance, body, project_id=None, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, *args, **kwargs)

Bases: airflow.contrib.operators.gcp_sql_operator.CloudSqlBaseOperator

Imports data into a Cloud SQL instance from a SQL dump or CSV file in Cloud Storage.

CSV IMPORT:

This operator is NOT idempotent for a CSV import. If the same file is imported multiple times, the imported data will be duplicated in the database. Moreover, if there are any unique constraints the duplicate import may result in an error.

SQL IMPORT:

This operator is idempotent for a SQL import if it was also exported by Cloud SQL. The exported SQL contains ‘DROP TABLE IF EXISTS’ statements for all tables to be imported.

If the import file was generated in a different way, idempotence is not guaranteed. It has to be ensured on the SQL file level.

See also

For more information on how to use this operator, take a look at the guide: CloudSqlInstanceImportOperator

Parameters

• instance (str) – Cloud SQL instance ID. This does not include the project ID.

• body (dict) – The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body

• project_id (str) – Optional, Google Cloud Platform Project ID. If set to None or missing, the default project_id from the GCP connection is used.

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform.

• api_version (str) – API version used (e.g. v1beta4).

• validate_body (bool) – Whether the body should be validated. Defaults to True.

template_fields = ['project_id', 'instance', 'gcp_conn_id', 'api_version']

_validate_inputs(self)

_validate_body_fields(self)

execute(self, context)

class airflow.contrib.operators.gcp_sql_operator.CloudSqlQueryOperator(sql, autocommit=False, parameters=None, gcp_conn_id='google_cloud_default', gcp_cloudsql_conn_id='google_cloud_sql_default', *args, **kwargs)

Bases: airflow.models.BaseOperator

Performs DML or DDL query on an existing Cloud Sql instance. It optionally uses cloud-sql-proxy to establish secure connection with the database.

See also

For more information on how to use this operator, take a look at the guide: CloudSqlQueryOperator

Parameters

• sql (str or list[str]) – SQL query or list of queries to run (should be DML or DDL query - this operator does not return any data from the database, so it is useless to pass it DQL queries. Note that it is responsibility of the author of the queries to make sure that the queries are idempotent. For example you can use CREATE TABLE IF NOT EXISTS to create a table.

• parameters (mapping or iterable) – (optional) the parameters to render the SQL query with.

• autocommit (bool) – if True, each command is automatically committed. (default value: False)

• gcp_conn_id (str) – The connection ID used to connect to Google Cloud Platform for cloud-sql-proxy authentication.

• gcp_cloudsql_conn_id (str) – The connection ID used to connect to Google Cloud SQL its schema should be gcpcloudsql://. See CloudSqlDatabaseHook for details on how to define gcpcloudsql:// connection.

template_fields = ['sql', 'gcp_cloudsql_conn_id', 'gcp_conn_id']

template_ext = ['.sql']

execute(self, context)