Google Cloud Build Operators

The GoogleCloud Build is a service that executes your builds on Google Cloud infrastructure. Cloud Build can import source code from Google Cloud Storage, Cloud Source Repositories, execute a build to your specifications, and produce artifacts such as Docker containers or Java archives.

Build configuration overview

In order to trigger a build, it is necessary to pass the build configuration.

airflow/providers/google/cloud/example_dags/example_cloud_build.py

create_build_from_storage_body = {
    "source": {"storageSource": GCP_SOURCE_ARCHIVE_URL},
    "steps": [
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": ["build", "-t", f"gcr.io/$PROJECT_ID/{GCP_SOURCE_BUCKET_NAME}", "."],
        }
    ],
    "images": [f"gcr.io/$PROJECT_ID/{GCP_SOURCE_BUCKET_NAME}"],
}

The source code for the build can come from Google Cloud Build Storage:

"source": {"storageSource": {"bucket": "bucket-name", "object": "object-name.tar.gz"}},

It is also possible to specify it using the URL:

"source": {"storageSource": "gs://bucket-name/object-name.tar.gz"},

In addition, a build can refer to source stored in Google Cloud Source Repositories.

"source": {
    "repoSource": {
        "projectId": "airflow-project",
        "repoName": "airflow-repo",
        "branchName": "master",
    }
},

It is also possible to specify it using the URL:

"source": {"repoSource": "https://source.developers.google.com/p/airflow-project/r/airflow-repo"},

It is also possible to specify it using a YAML or JSON format.

airflow/providers/google/cloud/example_dags/example_cloud_build.py

    create_build_from_file = CloudBuildCreateBuildOperator(
        task_id="create_build_from_file",
        project_id=GCP_PROJECT_ID,
        body=str(CURRENT_FOLDER.joinpath('example_cloud_build.yaml')),
        params={'name': 'Airflow'},
    )

Read Build Configuration Overview to understand all the fields you can include in a build config file.

Trigger a build

Trigger a build is performed with the CloudBuildCreateBuildOperator operator.

airflow/providers/google/cloud/example_dags/example_cloud_build.py

create_build_from_storage = CloudBuildCreateBuildOperator(
    task_id="create_build_from_storage", project_id=GCP_PROJECT_ID, body=create_build_from_storage_body
)

You can use Jinja templating with body, gcp_conn_id, api_version, impersonation_chain parameters which allows you to dynamically determine values. The result is saved to XCom, which allows it to be used by other operators.

airflow/providers/google/cloud/example_dags/example_cloud_build.py

create_build_from_storage_result = BashOperator(
    bash_command="echo '{{ task_instance.xcom_pull('create_build_from_storage')['images'][0] }}'",
    task_id="create_build_from_storage_result",
)

Was this entry helpful?