Google Compute Engine Operators

Prerequisite Tasks

ComputeEngineStartInstanceOperator

Use the ComputeEngineStartInstanceOperator to start an existing Google Compute Engine instance.

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_instance_start = ComputeEngineStartInstanceOperator(
    project_id=GCP_PROJECT_ID, zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_start_task'
)
Copy to clipboard

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection id used:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_instance_start2 = ComputeEngineStartInstanceOperator(
    zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_start_task2'
)
Copy to clipboard

Templating

template_fields = (
    'project_id',
    'zone',
    'resource_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)
Copy to clipboard

More information

See Google Compute Engine API documentation to start an instance.

ComputeEngineStopInstanceOperator

Use the operator to stop Google Compute Engine instance.

For parameter definition, take a look at ComputeEngineStopInstanceOperator

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_instance_stop = ComputeEngineStopInstanceOperator(
    project_id=GCP_PROJECT_ID, zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_stop_task'
)
Copy to clipboard

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_instance_stop2 = ComputeEngineStopInstanceOperator(
    zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_stop_task2'
)
Copy to clipboard

Templating

template_fields = (
    'project_id',
    'zone',
    'resource_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)
Copy to clipboard

More information

See Google Compute Engine API documentation to stop an instance.

ComputeEngineSetMachineTypeOperator

Use the operator to change machine type of a Google Compute Engine instance.

For parameter definition, take a look at ComputeEngineSetMachineTypeOperator.

Arguments

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_set_machine_type = ComputeEngineSetMachineTypeOperator(
    project_id=GCP_PROJECT_ID,
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    body={'machineType': f'zones/{GCE_ZONE}/machineTypes/{GCE_SHORT_MACHINE_TYPE_NAME}'},
    task_id='gcp_compute_set_machine_type',
)
Copy to clipboard

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_set_machine_type2 = ComputeEngineSetMachineTypeOperator(
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    body={'machineType': f'zones/{GCE_ZONE}/machineTypes/{GCE_SHORT_MACHINE_TYPE_NAME}'},
    task_id='gcp_compute_set_machine_type2',
)
Copy to clipboard

Templating

template_fields = (
    'project_id',
    'zone',
    'resource_id',
    'body',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)
Copy to clipboard

More information

See Google Compute Engine API documentation to set the machine type.

ComputeEngineCopyInstanceTemplateOperator

Use the operator to copy an existing Google Compute Engine instance template applying a patch to it.

For parameter definition, take a look at ComputeEngineCopyInstanceTemplateOperator.

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

GCE_TEMPLATE_NAME = os.environ.get('GCE_TEMPLATE_NAME', 'instance-template-test')
GCE_NEW_TEMPLATE_NAME = os.environ.get('GCE_NEW_TEMPLATE_NAME', 'instance-template-test-new')
GCE_NEW_DESCRIPTION = os.environ.get('GCE_NEW_DESCRIPTION', 'Test new description')
GCE_INSTANCE_TEMPLATE_BODY_UPDATE = {
    "name": GCE_NEW_TEMPLATE_NAME,
    "description": GCE_NEW_DESCRIPTION,
    "properties": {"machineType": "n1-standard-2"},
}
Copy to clipboard

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

gce_instance_template_copy = ComputeEngineCopyInstanceTemplateOperator(
    project_id=GCP_PROJECT_ID,
    resource_id=GCE_TEMPLATE_NAME,
    body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
    task_id='gcp_compute_igm_copy_template_task',
)
Copy to clipboard

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

gce_instance_template_copy2 = ComputeEngineCopyInstanceTemplateOperator(
    resource_id=GCE_TEMPLATE_NAME,
    body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
    task_id='gcp_compute_igm_copy_template_task_2',
)
Copy to clipboard

Templating

template_fields = (
    'project_id',
    'resource_id',
    'request_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)
Copy to clipboard

More information

See Google Compute Engine API documentation to create a new instance with an existing template.

ComputeEngineInstanceGroupUpdateManagerTemplateOperator

Use the operator to update a template in Google Compute Engine Instance Group Manager.

For parameter definition, take a look at ComputeEngineInstanceGroupUpdateManagerTemplateOperator.

Arguments

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

GCE_INSTANCE_GROUP_MANAGER_NAME = os.environ.get('GCE_INSTANCE_GROUP_MANAGER_NAME', 'instance-group-test')

SOURCE_TEMPLATE_URL = os.environ.get(
    'SOURCE_TEMPLATE_URL',
    "https://www.googleapis.com/compute/beta/projects/"
    + GCP_PROJECT_ID
    + "/global/instanceTemplates/instance-template-test",
)

DESTINATION_TEMPLATE_URL = os.environ.get(
    'DESTINATION_TEMPLATE_URL',
    "https://www.googleapis.com/compute/beta/projects/"
    + GCP_PROJECT_ID
    + "/global/instanceTemplates/"
    + GCE_NEW_TEMPLATE_NAME,
)

UPDATE_POLICY = {
    "type": "OPPORTUNISTIC",
    "minimalAction": "RESTART",
    "maxSurge": {"fixed": 1},
    "minReadySec": 1800,
}

Copy to clipboard

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

gce_instance_group_manager_update_template = ComputeEngineInstanceGroupUpdateManagerTemplateOperator(
    project_id=GCP_PROJECT_ID,
    resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
    zone=GCE_ZONE,
    source_template=SOURCE_TEMPLATE_URL,
    destination_template=DESTINATION_TEMPLATE_URL,
    update_policy=UPDATE_POLICY,
    task_id='gcp_compute_igm_group_manager_update_template',
)
Copy to clipboard

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

gce_instance_group_manager_update_template2 = ComputeEngineInstanceGroupUpdateManagerTemplateOperator(
    resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
    zone=GCE_ZONE,
    source_template=SOURCE_TEMPLATE_URL,
    destination_template=DESTINATION_TEMPLATE_URL,
    task_id='gcp_compute_igm_group_manager_update_template_2',
)
Copy to clipboard

Templating

template_fields = (
    'project_id',
    'resource_id',
    'zone',
    'request_id',
    'source_template',
    'destination_template',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)
Copy to clipboard

Troubleshooting

You might find that your ComputeEngineInstanceGroupUpdateManagerTemplateOperator fails with missing permissions. To execute the operation, the service account requires the permissions that theService Account User role provides (assigned via Google Cloud IAM).

More information

See Google Compute Engine API documentation to manage a group instance.

Reference

For further information, look at:

Was this entry helpful?