Google Cloud SQL Connection¶

The gcpcloudsql:// connection is used by airflow.providers.google.cloud.operators.cloud_sql.CloudSQLExecuteQueryOperator to perform query on a Google Cloud SQL database. Google Cloud SQL database can be either Postgres or MySQL, so this is a “meta” connection type. It introduces common schema for both MySQL and Postgres, including what kind of connectivity should be used. Google Cloud SQL supports connecting via public IP or via Cloud SQL Proxy. In the latter case the CloudSQLHook uses CloudSqlProxyRunner to automatically prepare and use temporary Postgres or MySQL connection that will use the proxy to connect (either via TCP or UNIX socket.

Configuring the Connection¶

Host (required)

The host to connect to.

Schema (optional)

Specify the schema name to be used in the database.

Login (required)

Specify the user name to connect.

Password (required)

Specify the password to connect.

Extra (optional)

Specify the extra parameters (as JSON dictionary) that can be used in Google Cloud SQL connection.

Details of all the parameters supported in extra field can be found in CloudSQLHook.

Example “extras” field:

{
   "database_type": "mysql",
   "project_id": "example-project",
   "location": "europe-west1",
   "instance": "testinstance",
   "use_proxy": true,
   "sql_proxy_use_tcp": false
}

When specifying the connection as URI (in AIRFLOW_CONN_{CONN_ID} variable), you should specify it following the standard syntax of DB connection, where extras are passed as parameters of the URI. Note that all components of the URI should be URL-encoded.

For example:

export AIRFLOW_CONN_GOOGLE_CLOUD_SQL_DEFAULT='gcpcloudsql://user:XXXXXXXXX@1.1.1.1:3306/mydb?database_type=mysql&project_id=example-project&location=europe-west1&instance=testinstance&use_proxy=True&sql_proxy_use_tcp=False'

Configuring and using IAM authentication¶

Warning

This functionality requires gcloud command (Google Cloud SDK) must be installed on the Airflow worker.

Warning

IAM authentication working only for Google Service Accounts.

Configure Service Accounts on Google Cloud IAM side¶

For connecting via IAM you need to use Service Account. It can be the same service account which you use for the gcloud authentication or an another account. If you decide to use a different account then this account should be impersonated from the account which used for gcloud authentication and granted a Service Account Token Creator role. More information how to grant a role here.

Also the Service Account should be configured for working with IAM. Here are links describing what should be done before the start: PostgreSQL and MySQL.

Configure `gcpcloudsql` connection with IAM enabling¶

For using IAM you need to enable "use_iam": "True" in the extra field. And specify IAM account in this format USERNAME@PROJECT_ID.iam.gserviceaccount.com in login field and empty string in the password field.

For example:

tests/system/google/cloud/cloud_sql/example_cloud_sql_query_iam.py[source]

CONNECTION_WITH_IAM_KWARGS = {
    "conn_type": "gcpcloudsql",
    "login": CLOUD_IAM_SA,
    "password": "",
    "host": CLOUD_SQL_IP_ADDRESS,
    "port": CLOUD_SQL_PUBLIC_PORT,
    "schema": CLOUD_SQL_DATABASE_NAME,
    "extra": {
        "database_type": DATABASE_TYPE,
        "project_id": PROJECT_ID,
        "location": REGION,
        "instance": CLOUD_SQL_INSTANCE_NAME,
        "use_proxy": "False",
        "use_ssl": "True",
        "use_iam": "True",
    },
}