Google Compute Engine Operators

Prerequisite Tasks

ComputeEngineStartInstanceOperator

Use the ComputeEngineStartInstanceOperator to start an existing Google Compute Engine instance.

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_instance_start = ComputeEngineStartInstanceOperator(
    project_id=GCP_PROJECT_ID, zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_start_task'
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection id used:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_instance_start2 = ComputeEngineStartInstanceOperator(
    zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_start_task2'
)

Templating

template_fields = (
    'project_id',
    'zone',
    'resource_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

More information

See Google Compute Engine API documentation to start an instance.

ComputeEngineStopInstanceOperator

Use the operator to stop Google Compute Engine instance.

For parameter definition, take a look at ComputeEngineStopInstanceOperator

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_instance_stop = ComputeEngineStopInstanceOperator(
    project_id=GCP_PROJECT_ID, zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_stop_task'
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_instance_stop2 = ComputeEngineStopInstanceOperator(
    zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_stop_task2'
)

Templating

template_fields = (
    'project_id',
    'zone',
    'resource_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

More information

See Google Compute Engine API documentation to stop an instance.

ComputeEngineSetMachineTypeOperator

Use the operator to change machine type of a Google Compute Engine instance.

For parameter definition, take a look at ComputeEngineSetMachineTypeOperator.

Arguments

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_set_machine_type = ComputeEngineSetMachineTypeOperator(
    project_id=GCP_PROJECT_ID,
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    body={'machineType': f'zones/{GCE_ZONE}/machineTypes/{GCE_SHORT_MACHINE_TYPE_NAME}'},
    task_id='gcp_compute_set_machine_type',
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute.pyView Source

gce_set_machine_type2 = ComputeEngineSetMachineTypeOperator(
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    body={'machineType': f'zones/{GCE_ZONE}/machineTypes/{GCE_SHORT_MACHINE_TYPE_NAME}'},
    task_id='gcp_compute_set_machine_type2',
)

Templating

template_fields = (
    'project_id',
    'zone',
    'resource_id',
    'body',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

More information

See Google Compute Engine API documentation to set the machine type.

ComputeEngineCopyInstanceTemplateOperator

Use the operator to copy an existing Google Compute Engine instance template applying a patch to it.

For parameter definition, take a look at ComputeEngineCopyInstanceTemplateOperator.

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

GCE_TEMPLATE_NAME = os.environ.get('GCE_TEMPLATE_NAME', 'instance-template-test')
GCE_NEW_TEMPLATE_NAME = os.environ.get('GCE_NEW_TEMPLATE_NAME', 'instance-template-test-new')
GCE_NEW_DESCRIPTION = os.environ.get('GCE_NEW_DESCRIPTION', 'Test new description')
GCE_INSTANCE_TEMPLATE_BODY_UPDATE = {
    "name": GCE_NEW_TEMPLATE_NAME,
    "description": GCE_NEW_DESCRIPTION,
    "properties": {"machineType": "n1-standard-2"},
}

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

gce_instance_template_copy = ComputeEngineCopyInstanceTemplateOperator(
    project_id=GCP_PROJECT_ID,
    resource_id=GCE_TEMPLATE_NAME,
    body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
    task_id='gcp_compute_igm_copy_template_task',
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

gce_instance_template_copy2 = ComputeEngineCopyInstanceTemplateOperator(
    resource_id=GCE_TEMPLATE_NAME,
    body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
    task_id='gcp_compute_igm_copy_template_task_2',
)

Templating

template_fields = (
    'project_id',
    'resource_id',
    'request_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

More information

See Google Compute Engine API documentation to create a new instance with an existing template.

ComputeEngineInstanceGroupUpdateManagerTemplateOperator

Use the operator to update a template in Google Compute Engine Instance Group Manager.

For parameter definition, take a look at ComputeEngineInstanceGroupUpdateManagerTemplateOperator.

Arguments

Using the operator

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

GCE_INSTANCE_GROUP_MANAGER_NAME = os.environ.get('GCE_INSTANCE_GROUP_MANAGER_NAME', 'instance-group-test')

SOURCE_TEMPLATE_URL = os.environ.get(
    'SOURCE_TEMPLATE_URL',
    "https://www.googleapis.com/compute/beta/projects/"
    + GCP_PROJECT_ID
    + "/global/instanceTemplates/instance-template-test",
)

DESTINATION_TEMPLATE_URL = os.environ.get(
    'DESTINATION_TEMPLATE_URL',
    "https://www.googleapis.com/compute/beta/projects/"
    + GCP_PROJECT_ID
    + "/global/instanceTemplates/"
    + GCE_NEW_TEMPLATE_NAME,
)

UPDATE_POLICY = {
    "type": "OPPORTUNISTIC",
    "minimalAction": "RESTART",
    "maxSurge": {"fixed": 1},
    "minReadySec": 1800,
}

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

gce_instance_group_manager_update_template = ComputeEngineInstanceGroupUpdateManagerTemplateOperator(
    project_id=GCP_PROJECT_ID,
    resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
    zone=GCE_ZONE,
    source_template=SOURCE_TEMPLATE_URL,
    destination_template=DESTINATION_TEMPLATE_URL,
    update_policy=UPDATE_POLICY,
    task_id='gcp_compute_igm_group_manager_update_template',
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute_igm.pyView Source

gce_instance_group_manager_update_template2 = ComputeEngineInstanceGroupUpdateManagerTemplateOperator(
    resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
    zone=GCE_ZONE,
    source_template=SOURCE_TEMPLATE_URL,
    destination_template=DESTINATION_TEMPLATE_URL,
    task_id='gcp_compute_igm_group_manager_update_template_2',
)

Templating

template_fields = (
    'project_id',
    'resource_id',
    'zone',
    'request_id',
    'source_template',
    'destination_template',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

Troubleshooting

You might find that your ComputeEngineInstanceGroupUpdateManagerTemplateOperator fails with missing permissions. To execute the operation, the service account requires the permissions that theService Account User role provides (assigned via Google Cloud IAM).

More information

See Google Compute Engine API documentation to manage a group instance.

Reference

For further information, look at:

Was this entry helpful?