Google Cloud Build Operators¶
The GoogleCloud Build is a service that executes your builds on Google Cloud infrastructure. Cloud Build can import source code from Google Cloud Storage, Cloud Source Repositories, execute a build to your specifications, and produce artifacts such as Docker containers or Java archives.
Build configuration overview¶
In order to trigger a build, it is necessary to pass the build configuration.
airflow/providers/google/cloud/example_dags/example_cloud_build.py
create_build_from_storage_body = {
"source": {"storageSource": GCP_SOURCE_ARCHIVE_URL},
"steps": [
{
"name": "gcr.io/cloud-builders/docker",
"args": ["build", "-t", f"gcr.io/$PROJECT_ID/{GCP_SOURCE_BUCKET_NAME}", "."],
}
],
"images": [f"gcr.io/$PROJECT_ID/{GCP_SOURCE_BUCKET_NAME}"],
}
The source code for the build can come from Google Cloud Build Storage:
"source": {"storageSource": {"bucket": "bucket-name", "object": "object-name.tar.gz"}},
It is also possible to specify it using the URL:
"source": {"storageSource": "gs://bucket-name/object-name.tar.gz"},
In addition, a build can refer to source stored in Google Cloud Source Repositories.
"source": {
"repoSource": {
"projectId": "airflow-project",
"repoName": "airflow-repo",
"branchName": "master",
}
},
It is also possible to specify it using the URL:
"source": {"repoSource": "https://source.developers.google.com/p/airflow-project/r/airflow-repo"},
It is also possible to specify it using a YAML or JSON format.
airflow/providers/google/cloud/example_dags/example_cloud_build.py
create_build_from_file = CloudBuildCreateBuildOperator(
task_id="create_build_from_file",
project_id=GCP_PROJECT_ID,
body=str(CURRENT_FOLDER.joinpath('example_cloud_build.yaml')),
params={'name': 'Airflow'},
)
Read Build Configuration Overview to understand all the fields you can include in a build config file.
Trigger a build¶
Trigger a build is performed with the
CloudBuildCreateBuildOperator operator.
airflow/providers/google/cloud/example_dags/example_cloud_build.py
create_build_from_storage = CloudBuildCreateBuildOperator(
task_id="create_build_from_storage", project_id=GCP_PROJECT_ID, body=create_build_from_storage_body
)
You can use Jinja templating with
body, gcp_conn_id, api_version, impersonation_chain
parameters which allows you to dynamically determine values. The result is saved to XCom, which allows it
to be used by other operators.
airflow/providers/google/cloud/example_dags/example_cloud_build.py
create_build_from_storage_result = BashOperator(
bash_command="echo '{{ task_instance.xcom_pull('create_build_from_storage')['images'][0] }}'",
task_id="create_build_from_storage_result",
)