airflow.providers.google.cloud.hooks.cloud_sql

This module contains a Google Cloud SQL Hook.

Module Contents

Classes

CloudSqlOperationStatus

Helper class with operation statuses.

CloudSQLHook

Hook for Google Cloud SQL APIs.

CloudSQLAsyncHook

Class to get asynchronous hook for Google Cloud SQL.

CloudSqlProxyRunner

Downloads and runs cloud-sql-proxy as subprocess of the Python process.

CloudSQLDatabaseHook

Serves DB connection configuration for Google Cloud SQL (Connections of gcpcloudsqldb:// type).

Attributes

UNIX_PATH_MAX

TIME_TO_SLEEP_IN_SECONDS

CLOUD_SQL_PROXY_VERSION_REGEX

CLOUD_SQL_PROXY_DOWNLOAD_URL

CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL

CONNECTION_URIS

CLOUD_SQL_VALID_DATABASE_TYPES

airflow.providers.google.cloud.hooks.cloud_sql.UNIX_PATH_MAX = 108[source]
airflow.providers.google.cloud.hooks.cloud_sql.TIME_TO_SLEEP_IN_SECONDS = 20[source]
airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_PROXY_VERSION_REGEX[source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSqlOperationStatus[source]

Helper class with operation statuses.

PENDING = 'PENDING'[source]
RUNNING = 'RUNNING'[source]
DONE = 'DONE'[source]
UNKNOWN = 'UNKNOWN'[source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook(api_version, gcp_conn_id=default_conn_name, impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Cloud SQL APIs.

All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.

Parameters

  • api_version (str) – This is the version of the api.

  • gcp_conn_id (str) – The Airflow connection used for GCP credentials.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – This is the optional service account to impersonate using short term credentials.

conn_name_attr = 'gcp_conn_id'[source]
default_conn_name = 'google_cloud_sql_default'[source]
conn_type = 'gcpcloudsql'[source]
hook_name = 'Google Cloud SQL'[source]
get_conn()[source]

Retrieve connection to Cloud SQL.

Returns

Google Cloud SQL services object.

Return type

googleapiclient.discovery.Resource

get_instance(instance, project_id)[source]

Retrieve a resource containing information about a Cloud SQL instance.

Parameters

  • instance (str) – Database instance ID. This does not include the project ID.

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

A Cloud SQL instance resource.

Return type

dict

create_instance(body, project_id)[source]

Create a new Cloud SQL instance.

Parameters
Returns

None

Return type

None

patch_instance(body, instance, project_id)[source]

Update settings of a Cloud SQL instance.

Caution: This is not a partial update, so you must include values for all the settings that you want to retain.

Parameters
Returns

None

Return type

None

delete_instance(instance, project_id)[source]

Delete a Cloud SQL instance.

Parameters

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • instance (str) – Cloud SQL instance ID. This does not include the project ID.

Returns

None

Return type

None

get_database(instance, database, project_id)[source]

Retrieve a database resource from a Cloud SQL instance.

Parameters

  • instance (str) – Database instance ID. This does not include the project ID.

  • database (str) – Name of the database in the instance.

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

A Cloud SQL database resource, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases#resource.

Return type

dict

create_database(instance, body, project_id)[source]

Create a new database inside a Cloud SQL instance.

Parameters
Returns

None

Return type

None

patch_database(instance, database, body, project_id)[source]

Update a database resource inside a Cloud SQL instance.

This method supports patch semantics. See https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch.

Parameters
Returns

None

Return type

None

delete_database(instance, database, project_id)[source]

Delete a database from a Cloud SQL instance.

Parameters

  • instance (str) – Database instance ID. This does not include the project ID.

  • database (str) – Name of the database to be deleted in the instance.

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

None

Return type

None

export_instance(instance, body, project_id)[source]

Export data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.

Parameters
Returns

None

import_instance(instance, body, project_id)[source]

Import data into a Cloud SQL instance from a SQL dump or CSV file in Cloud Storage.

Parameters
Returns

None

Return type

None

clone_instance(instance, body, project_id)[source]

Clones an instance to a target instance.

Parameters

  • instance (str) – Database instance ID to be cloned. This does not include the project ID.

  • instance – Database instance ID to be used for the clone. This does not include the project ID.

  • body (dict) – The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/rest/v1beta4/instances/clone

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

None

Return type

None

create_ssl_certificate(instance, body, project_id)[source]

Create SSL certificate for a Cloud SQL instance.

Parameters
Returns

SslCert insert response. For more details see: https://cloud.google.com/sql/docs/mysql/admin-api/rest/v1/sslCerts/insert#response-body

airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_PROXY_DOWNLOAD_URL = 'https://dl.google.com/cloudsql/cloud_sql_proxy.{}.{}'[source]
airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL = 'https://storage.googleapis.com/cloudsql-proxy/{}/cloud_sql_proxy.{}.{}'[source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLAsyncHook(**kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseAsyncHook

Class to get asynchronous hook for Google Cloud SQL.

sync_hook_class[source]
async get_operation_name(project_id, operation_name, session)[source]
async get_operation(project_id, operation_name)[source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSqlProxyRunner(path_prefix, instance_specification, gcp_conn_id='google_cloud_default', project_id=PROVIDE_PROJECT_ID, sql_proxy_version=None, sql_proxy_binary_path=None)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Downloads and runs cloud-sql-proxy as subprocess of the Python process.

The cloud-sql-proxy needs to be downloaded and started before we can connect to the Google Cloud SQL instance via database connection. It establishes secure tunnel connection to the database. It authorizes using the Google Cloud credentials that are passed by the configuration.

More details about the proxy can be found here: https://cloud.google.com/sql/docs/mysql/sql-proxy

Parameters

  • path_prefix (str) – Unique path prefix where proxy will be downloaded and directories created for unix sockets.

  • instance_specification (str) – Specification of the instance to connect the proxy to. It should be specified in the form that is described in https://cloud.google.com/sql/docs/mysql/sql-proxy#multiple-instances in -instances parameter (typically in the form of <project>:<region>:<instance> for UNIX socket connections and in the form of <project>:<region>:<instance>=tcp:<port> for TCP connections.

  • gcp_conn_id (str) – Id of Google Cloud connection to use for authentication

  • project_id (str) – Optional id of the Google Cloud project to connect to - it overwrites default project id taken from the Google Cloud connection.

  • sql_proxy_version (str | None) – Specific version of SQL proxy to download (for example ‘v1.13’). By default latest version is downloaded.

  • sql_proxy_binary_path (str | None) – If specified, then proxy will be used from the path specified rather than dynamically generated. This means that if the binary is not present in that path it will also be downloaded.

start_proxy()[source]

Start Cloud SQL Proxy.

You have to remember to stop the proxy if you started it!

stop_proxy()[source]

Stop running proxy.

You should stop the proxy after you stop using it.

get_proxy_version()[source]

Return version of the Cloud SQL Proxy.

get_socket_path()[source]

Retrieve UNIX socket path used by Cloud SQL Proxy.

Returns

The dynamically generated path for the socket created by the proxy.

Return type

str

airflow.providers.google.cloud.hooks.cloud_sql.CONNECTION_URIS: dict[str, dict[str, dict[str, str]]][source]
airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_VALID_DATABASE_TYPES = ['postgres', 'mysql'][source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook(gcp_cloudsql_conn_id='google_cloud_sql_default', gcp_conn_id='google_cloud_default', impersonation_chain=None, default_gcp_project_id=None, sql_proxy_binary_path=None, ssl_cert=None, ssl_key=None, ssl_root_cert=None, ssl_secret_id=None, **kwargs)[source]

Bases: airflow.hooks.base.BaseHook

Serves DB connection configuration for Google Cloud SQL (Connections of gcpcloudsqldb:// type).

The hook is a “meta” one. It does not perform an actual connection.

It is there to retrieve all the parameters configured in gcpcloudsql:// connection, start/stop Cloud SQL Proxy if needed, dynamically generate Postgres or MySQL connection in the database and return an actual Postgres or MySQL hook. The returned Postgres/MySQL hooks are using direct connection or Cloud SQL Proxy socket/TCP as configured.

Main parameters of the hook are retrieved from the standard URI components:

  • user - User name to authenticate to the database (from login of the URI).

  • password - Password to authenticate to the database (from password of the URI).

  • public_ip - IP to connect to for public connection (from host of the URI).

  • public_port - Port to connect to for public connection (from port of the URI).

  • database - Database to connect to (from schema of the URI).

  • sql_proxy_binary_path - Optional path to Cloud SQL Proxy binary. If the binary is not specified or the binary is not present, it is automatically downloaded.

Remaining parameters are retrieved from the extras (URI query parameters):

  • project_id - Optional, Google Cloud project where the Cloud SQL

    instance exists. If missing, default project id passed is used.

  • instance - Name of the instance of the Cloud SQL database instance.

  • location - The location of the Cloud SQL instance (for example europe-west1).

  • database_type - The type of the database instance (MySQL or Postgres).

  • use_proxy - (default False) Whether SQL proxy should be used to connect to Cloud SQL DB.

  • use_ssl - (default False) Whether SSL should be used to connect to Cloud SQL DB. You cannot use proxy and SSL together.

  • use_iam - (default False) Whether IAM should be used to connect to Cloud SQL DB. With using IAM password field should be empty string.

  • sql_proxy_use_tcp - (default False) If set to true, TCP is used to connect via proxy, otherwise UNIX sockets are used.

  • sql_proxy_version - Specific version of the proxy to download (for example v1.13). If not specified, the latest version is downloaded.

  • sslcert - Path to client certificate to authenticate when SSL is used.

  • sslkey - Path to client private key to authenticate when SSL is used.

  • sslrootcert - Path to server’s certificate to authenticate when SSL is used.

Parameters

  • gcp_cloudsql_conn_id (str) – URL of the connection

  • gcp_conn_id (str) – The connection ID used to connect to Google Cloud for cloud-sql-proxy authentication.

  • default_gcp_project_id (str | None) – Default project id used if project_id not specified in the connection URL

  • ssl_cert (str | None) – Optional. Path to client certificate to authenticate when SSL is used. Overrides the connection field sslcert.

  • ssl_key (str | None) – Optional. Path to client private key to authenticate when SSL is used. Overrides the connection field sslkey.

  • ssl_root_cert (str | None) – Optional. Path to server’s certificate to authenticate when SSL is used. Overrides the connection field sslrootcert.

  • ssl_secret_id (str | None) –

    Optional. ID of the secret in Google Cloud Secret Manager that stores SSL certificate in the format below:

    {‘sslcert’: ‘’,

    ’sslkey’: ‘’, ‘sslrootcert’: ‘’}

    Overrides the connection fields sslcert, sslkey, sslrootcert. Note that according to the Secret Manager requirements, the mentioned dict should be saved as a string, and encoded with base64. Note that this parameter is incompatible with parameters ssl_cert, ssl_key, ssl_root_cert.

property sslcert: str | None[source]
property sslkey: str | None[source]
property sslrootcert: str | None[source]
conn_name_attr = 'gcp_cloudsql_conn_id'[source]
default_conn_name = 'google_cloud_sqldb_default'[source]
conn_type = 'gcpcloudsqldb'[source]
hook_name = 'Google Cloud SQL Database'[source]
validate_ssl_certs()[source]

SSL certificates validator.

Returns

None

Return type

None

validate_socket_path_length()[source]

Validate sockets path length.

Returns

None or rises AirflowException

Return type

None

create_connection()[source]

Create a connection.

Connection ID will be randomly generated according to whether it uses proxy, TCP, UNIX sockets, SSL.

get_sqlproxy_runner()[source]

Retrieve Cloud SQL Proxy runner.

It is used to manage the proxy lifecycle per task.

Returns

The Cloud SQL Proxy runner.

Return type

CloudSqlProxyRunner

get_database_hook(connection)[source]

Retrieve database hook.

This is the actual Postgres or MySQL database hook that uses proxy or connects directly to the Google Cloud SQL database.

cleanup_database_hook()[source]

Clean up database hook after it was used.

reserve_free_tcp_port()[source]

Reserve free TCP port to be used by Cloud SQL Proxy.

free_reserved_port()[source]

Free TCP port.

Makes it immediately ready to be used by Cloud SQL Proxy.

Was this entry helpful?