Google Compute Engine Operators¶
Prerequisite Tasks¶
To use these operators, you must do a few things:
Select or create a Cloud Platform project using Cloud Console.
Enable billing for your project, as described in Google Cloud documentation.
Enable API, as described in Cloud Console documentation.
Install API libraries via pip.
pip install 'apache-airflow[gcp]'Detailed information is available Installation
GceInstanceStartOperator¶
Use the
GceInstanceStartOperator
to start an existing Google Compute Engine instance.
Arguments¶
The following examples of OS environment variables used to pass arguments to the operator:
GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')
GCE_INSTANCE = os.environ.get('GCE_INSTANCE', 'testinstance')
Using the operator¶
The code to create the operator:
gce_instance_start = GceInstanceStartOperator(
project_id=GCP_PROJECT_ID,
zone=GCE_ZONE,
resource_id=GCE_INSTANCE,
task_id='gcp_compute_start_task'
)
You can also create the operator without project id - project id will be retrieved from the GCP connection id used:
gce_instance_start2 = GceInstanceStartOperator(
zone=GCE_ZONE,
resource_id=GCE_INSTANCE,
task_id='gcp_compute_start_task2'
)
Templating¶
template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')
More information¶
See Google Compute Engine API documentation to start an instance.
GceInstanceStopOperator¶
Use the operator to stop Google Compute Engine instance.
For parameter definition, take a look at
GceInstanceStopOperator
Arguments¶
The following examples of OS environment variables used to pass arguments to the operator:
GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')
GCE_INSTANCE = os.environ.get('GCE_INSTANCE', 'testinstance')
Using the operator¶
The code to create the operator:
gce_instance_stop = GceInstanceStopOperator(
project_id=GCP_PROJECT_ID,
zone=GCE_ZONE,
resource_id=GCE_INSTANCE,
task_id='gcp_compute_stop_task'
)
You can also create the operator without project id - project id will be retrieved from the GCP connection used:
gce_instance_stop2 = GceInstanceStopOperator(
zone=GCE_ZONE,
resource_id=GCE_INSTANCE,
task_id='gcp_compute_stop_task2'
)
Templating¶
template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')
More information¶
See Google Compute Engine API documentation to stop an instance.
GceSetMachineTypeOperator¶
Use the operator to change machine type of a Google Compute Engine instance.
For parameter definition, take a look at
GceSetMachineTypeOperator.
Arguments¶
The following examples of OS environment variables used to pass arguments to the operator:
GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')
GCE_INSTANCE = os.environ.get('GCE_INSTANCE', 'testinstance')
GCE_SHORT_MACHINE_TYPE_NAME = os.environ.get('GCE_SHORT_MACHINE_TYPE_NAME', 'n1-standard-1')
SET_MACHINE_TYPE_BODY = {
'machineType': 'zones/{}/machineTypes/{}'.format(GCE_ZONE, GCE_SHORT_MACHINE_TYPE_NAME)
}
Using the operator¶
The code to create the operator:
gce_set_machine_type = GceSetMachineTypeOperator(
project_id=GCP_PROJECT_ID,
zone=GCE_ZONE,
resource_id=GCE_INSTANCE,
body=SET_MACHINE_TYPE_BODY,
task_id='gcp_compute_set_machine_type'
)
You can also create the operator without project id - project id will be retrieved from the GCP connection used:
gce_set_machine_type2 = GceSetMachineTypeOperator(
zone=GCE_ZONE,
resource_id=GCE_INSTANCE,
body=SET_MACHINE_TYPE_BODY,
task_id='gcp_compute_set_machine_type2'
)
Templating¶
template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')
More information¶
See Google Compute Engine API documentation to set the machine type.
GceInstanceTemplateCopyOperator¶
Use the operator to copy an existing Google Compute Engine instance template applying a patch to it.
For parameter definition, take a look at
GceInstanceTemplateCopyOperator.
Arguments¶
The following examples of OS environment variables used to pass arguments to the operator:
GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')
GCE_TEMPLATE_NAME = os.environ.get('GCE_TEMPLATE_NAME', 'instance-template-test')
GCE_NEW_TEMPLATE_NAME = os.environ.get('GCE_NEW_TEMPLATE_NAME',
'instance-template-test-new')
GCE_NEW_DESCRIPTION = os.environ.get('GCE_NEW_DESCRIPTION', 'Test new description')
GCE_INSTANCE_TEMPLATE_BODY_UPDATE = {
"name": GCE_NEW_TEMPLATE_NAME,
"description": GCE_NEW_DESCRIPTION,
"properties": {
"machineType": "n1-standard-2"
}
}
Using the operator¶
The code to create the operator:
gce_instance_template_copy = GceInstanceTemplateCopyOperator(
project_id=GCP_PROJECT_ID,
resource_id=GCE_TEMPLATE_NAME,
body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
task_id='gcp_compute_igm_copy_template_task'
)
You can also create the operator without project id - project id will be retrieved from the GCP connection used:
gce_instance_template_copy2 = GceInstanceTemplateCopyOperator(
resource_id=GCE_TEMPLATE_NAME,
body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
task_id='gcp_compute_igm_copy_template_task_2'
)
Templating¶
template_fields = ('project_id', 'resource_id', 'request_id',
'gcp_conn_id', 'api_version')
More information¶
See Google Compute Engine API documentation to create a new instance with an existing template.
GceInstanceGroupManagerUpdateTemplateOperator¶
Use the operator to update a template in Google Compute Engine Instance Group Manager.
For parameter definition, take a look at
GceInstanceGroupManagerUpdateTemplateOperator.
Arguments¶
The following examples of OS environment variables used to pass arguments to the operator:
GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')
GCE_INSTANCE_GROUP_MANAGER_NAME = os.environ.get('GCE_INSTANCE_GROUP_MANAGER_NAME',
'instance-group-test')
SOURCE_TEMPLATE_URL = os.environ.get(
'SOURCE_TEMPLATE_URL',
"https://www.googleapis.com/compute/beta/projects/" + GCP_PROJECT_ID +
"/global/instanceTemplates/instance-template-test")
DESTINATION_TEMPLATE_URL = os.environ.get(
'DESTINATION_TEMPLATE_URL',
"https://www.googleapis.com/compute/beta/projects/" + GCP_PROJECT_ID +
"/global/instanceTemplates/" + GCE_NEW_TEMPLATE_NAME)
UPDATE_POLICY = {
"type": "OPPORTUNISTIC",
"minimalAction": "RESTART",
"maxSurge": {
"fixed": 1
},
"minReadySec": 1800
}
Using the operator¶
The code to create the operator:
gce_instance_group_manager_update_template = \
GceInstanceGroupManagerUpdateTemplateOperator(
project_id=GCP_PROJECT_ID,
resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
zone=GCE_ZONE,
source_template=SOURCE_TEMPLATE_URL,
destination_template=DESTINATION_TEMPLATE_URL,
update_policy=UPDATE_POLICY,
task_id='gcp_compute_igm_group_manager_update_template'
)
You can also create the operator without project id - project id will be retrieved from the GCP connection used:
gce_instance_group_manager_update_template2 = \
GceInstanceGroupManagerUpdateTemplateOperator(
resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
zone=GCE_ZONE,
source_template=SOURCE_TEMPLATE_URL,
destination_template=DESTINATION_TEMPLATE_URL,
task_id='gcp_compute_igm_group_manager_update_template_2'
)
Templating¶
template_fields = ('project_id', 'resource_id', 'zone', 'request_id',
'source_template', 'destination_template',
'gcp_conn_id', 'api_version')
Troubleshooting¶
You might find that your GceInstanceGroupManagerUpdateTemplateOperator fails with missing permissions. To execute the operation, the service account requires the permissions that theService Account User role provides (assigned via Google Cloud IAM).
More information¶
See Google Compute Engine API documentation to manage a group instance.