airflow.providers.google.cloud.hooks.cloud_sql

This module contains a Google Cloud SQL Hook.

Module Contents

airflow.providers.google.cloud.hooks.cloud_sql.UNIX_PATH_MAX = 108[source]
airflow.providers.google.cloud.hooks.cloud_sql.TIME_TO_SLEEP_IN_SECONDS = 20[source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSqlOperationStatus[source]

Helper class with operation statuses.

PENDING = PENDING[source]
RUNNING = RUNNING[source]
DONE = DONE[source]
UNKNOWN = UNKNOWN[source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook(api_version: str, gcp_conn_id: str = default_conn_name, delegate_to: Optional[str] = None, impersonation_chain: Optional[Union[str, Sequence[str]]] = None)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Cloud SQL APIs.

All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.

conn_name_attr = gcp_conn_id[source]
default_conn_name = google_cloud_default[source]
conn_type = gcpcloudsql[source]
hook_name = Google Cloud SQL[source]
get_conn(self)[source]

Retrieves connection to Cloud SQL.

Returns

Google Cloud SQL services object.

Return type

dict

get_instance(self, instance: str, project_id: str)[source]

Retrieves a resource containing information about a Cloud SQL instance.

Parameters

  • instance (str) – Database instance ID. This does not include the project ID.

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

A Cloud SQL instance resource.

Return type

dict

create_instance(self, body: Dict, project_id: str)[source]

Creates a new Cloud SQL instance.

Parameters
Returns

None

patch_instance(self, body: dict, instance: str, project_id: str)[source]

Updates settings of a Cloud SQL instance.

Caution: This is not a partial update, so you must include values for all the settings that you want to retain.

Parameters
Returns

None

delete_instance(self, instance: str, project_id: str)[source]

Deletes a Cloud SQL instance.

Parameters

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • instance (str) – Cloud SQL instance ID. This does not include the project ID.

Returns

None

get_database(self, instance: str, database: str, project_id: str)[source]

Retrieves a database resource from a Cloud SQL instance.

Parameters

  • instance (str) – Database instance ID. This does not include the project ID.

  • database (str) – Name of the database in the instance.

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

A Cloud SQL database resource, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases#resource.

Return type

dict

create_database(self, instance: str, body: Dict, project_id: str)[source]

Creates a new database inside a Cloud SQL instance.

Parameters
Returns

None

patch_database(self, instance: str, database: str, body: Dict, project_id: str)[source]

Updates a database resource inside a Cloud SQL instance.

This method supports patch semantics. See https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch.

Parameters
Returns

None

delete_database(self, instance: str, database: str, project_id: str)[source]

Deletes a database from a Cloud SQL instance.

Parameters

  • instance (str) – Database instance ID. This does not include the project ID.

  • database (str) – Name of the database to be deleted in the instance.

  • project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.

Returns

None

export_instance(self, instance: str, body: Dict, project_id: str)[source]

Exports data from a Cloud SQL instance to a Cloud Storage bucket as a SQL dump or CSV file.

Parameters
Returns

None

import_instance(self, instance: str, body: Dict, project_id: str)[source]

Imports data into a Cloud SQL instance from a SQL dump or CSV file in Cloud Storage.

Parameters
Returns

None

_wait_for_operation_to_complete(self, project_id: str, operation_name: str)[source]

Waits for the named operation to complete - checks status of the asynchronous call.

Parameters

  • project_id (str) – Project ID of the project that contains the instance.

  • operation_name (str) – Name of the operation.

Returns

None

airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_PROXY_DOWNLOAD_URL = https://dl.google.com/cloudsql/cloud_sql_proxy.{}.{}[source]
airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_PROXY_VERSION_DOWNLOAD_URL = https://storage.googleapis.com/cloudsql-proxy/{}/cloud_sql_proxy.{}.{}[source]
airflow.providers.google.cloud.hooks.cloud_sql.GCP_CREDENTIALS_KEY_PATH = extra__google_cloud_platform__key_path[source]
airflow.providers.google.cloud.hooks.cloud_sql.GCP_CREDENTIALS_KEYFILE_DICT = extra__google_cloud_platform__keyfile_dict[source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSqlProxyRunner(path_prefix: str, instance_specification: str, gcp_conn_id: str = 'google_cloud_default', project_id: Optional[str] = None, sql_proxy_version: Optional[str] = None, sql_proxy_binary_path: Optional[str] = None)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Downloads and runs cloud-sql-proxy as subprocess of the Python process.

The cloud-sql-proxy needs to be downloaded and started before we can connect to the Google Cloud SQL instance via database connection. It establishes secure tunnel connection to the database. It authorizes using the Google Cloud credentials that are passed by the configuration.

More details about the proxy can be found here: https://cloud.google.com/sql/docs/mysql/sql-proxy

Parameters

  • path_prefix (str) – Unique path prefix where proxy will be downloaded and directories created for unix sockets.

  • instance_specification (str) – Specification of the instance to connect the proxy to. It should be specified in the form that is described in https://cloud.google.com/sql/docs/mysql/sql-proxy#multiple-instances in -instances parameter (typically in the form of <project>:<region>:<instance> for UNIX socket connections and in the form of <project>:<region>:<instance>=tcp:<port> for TCP connections.

  • gcp_conn_id (str) – Id of Google Cloud connection to use for authentication

  • project_id (str) – Optional id of the Google Cloud project to connect to - it overwrites default project id taken from the Google Cloud connection.

  • sql_proxy_version (str) – Specific version of SQL proxy to download (for example ‘v1.13’). By default latest version is downloaded.

  • sql_proxy_binary_path (str) – If specified, then proxy will be used from the path specified rather than dynamically generated. This means that if the binary is not present in that path it will also be downloaded.

_build_command_line_parameters(self)[source]
static _is_os_64bit()[source]
_download_sql_proxy_if_needed(self)[source]
_get_credential_parameters(self, session: Session)[source]
start_proxy(self)[source]

Starts Cloud SQL Proxy.

You have to remember to stop the proxy if you started it!

stop_proxy(self)[source]

Stops running proxy.

You should stop the proxy after you stop using it.

get_proxy_version(self)[source]

Returns version of the Cloud SQL Proxy.

get_socket_path(self)[source]

Retrieves UNIX socket path used by Cloud SQL Proxy.

Returns

The dynamically generated path for the socket created by the proxy.

Return type

str

airflow.providers.google.cloud.hooks.cloud_sql.CONNECTION_URIS :Dict[str, Dict[str, Dict[str, str]]][source]
airflow.providers.google.cloud.hooks.cloud_sql.CLOUD_SQL_VALID_DATABASE_TYPES = ['postgres', 'mysql'][source]
class airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook(gcp_cloudsql_conn_id: str = 'google_cloud_sql_default', gcp_conn_id: str = 'google_cloud_default', default_gcp_project_id: Optional[str] = None)[source]

Bases: airflow.hooks.base.BaseHook

Serves DB connection configuration for Google Cloud SQL (Connections of gcpcloudsqldb:// type).

The hook is a “meta” one. It does not perform an actual connection. It is there to retrieve all the parameters configured in gcpcloudsql:// connection, start/stop Cloud SQL Proxy if needed, dynamically generate Postgres or MySQL connection in the database and return an actual Postgres or MySQL hook. The returned Postgres/MySQL hooks are using direct connection or Cloud SQL Proxy socket/TCP as configured.

Main parameters of the hook are retrieved from the standard URI components:

  • user - User name to authenticate to the database (from login of the URI).

  • password - Password to authenticate to the database (from password of the URI).

  • public_ip - IP to connect to for public connection (from host of the URI).

  • public_port - Port to connect to for public connection (from port of the URI).

  • database - Database to connect to (from schema of the URI).

Remaining parameters are retrieved from the extras (URI query parameters):

  • project_id - Optional, Google Cloud project where the Cloud SQL

    instance exists. If missing, default project id passed is used.

  • instance - Name of the instance of the Cloud SQL database instance.

  • location - The location of the Cloud SQL instance (for example europe-west1).

  • database_type - The type of the database instance (MySQL or Postgres).

  • use_proxy - (default False) Whether SQL proxy should be used to connect to Cloud SQL DB.

  • use_ssl - (default False) Whether SSL should be used to connect to Cloud SQL DB. You cannot use proxy and SSL together.

  • sql_proxy_use_tcp - (default False) If set to true, TCP is used to connect via proxy, otherwise UNIX sockets are used.

  • sql_proxy_binary_path - Optional path to Cloud SQL Proxy binary. If the binary is not specified or the binary is not present, it is automatically downloaded.

  • sql_proxy_version - Specific version of the proxy to download (for example v1.13). If not specified, the latest version is downloaded.

  • sslcert - Path to client certificate to authenticate when SSL is used.

  • sslkey - Path to client private key to authenticate when SSL is used.

  • sslrootcert - Path to server’s certificate to authenticate when SSL is used.

Parameters

  • gcp_cloudsql_conn_id (str) – URL of the connection

  • gcp_conn_id (str) – The connection ID used to connect to Google Cloud for cloud-sql-proxy authentication.

  • default_gcp_project_id (str) – Default project id used if project_id not specified in the connection URL

conn_name_attr = gcp_cloudsql_conn_id[source]
default_conn_name = google_cloud_sql_default[source]
conn_type = gcpcloudsqldb[source]
hook_name = Google Cloud SQL Database[source]
_conn :Optional[Any][source]
static _get_bool(val: Any)[source]
static _check_ssl_file(file_to_check, name)[source]
_validate_inputs(self)[source]
validate_ssl_certs(self)[source]

SSL certificates validator.

Returns

None

validate_socket_path_length(self)[source]

Validates sockets path length.

Returns

None or rises AirflowException

static _generate_unique_path()[source]

We are not using mkdtemp here as the path generated with mkdtemp can be close to 60 characters and there is a limitation in length of socket path to around 100 characters in total. We append project/location/instance to it later and postgres appends its own prefix, so we chose a shorter “/tmp/[8 random characters]”

static _quote(value)[source]
_generate_connection_uri(self)[source]
_get_instance_socket_name(self)[source]
_get_sqlproxy_instance_specification(self)[source]
create_connection(self)[source]

Create Connection object, according to whether it uses proxy, TCP, UNIX sockets, SSL. Connection ID will be randomly generated.

get_sqlproxy_runner(self)[source]

Retrieve Cloud SQL Proxy runner. It is used to manage the proxy lifecycle per task.

Returns

The Cloud SQL Proxy runner.

Return type

CloudSqlProxyRunner

get_database_hook(self, connection: Connection)[source]

Retrieve database hook. This is the actual Postgres or MySQL database hook that uses proxy or connects directly to the Google Cloud SQL database.

cleanup_database_hook(self)[source]

Clean up database hook after it was used.

reserve_free_tcp_port(self)[source]

Reserve free TCP port to be used by Cloud SQL Proxy

free_reserved_port(self)[source]

Free TCP port. Makes it immediately ready to be used by Cloud SQL Proxy.

Was this entry helpful?