Google Compute Engine Operators¶

Prerequisite Tasks¶

ComputeEngineStartInstanceOperator¶

Use the ComputeEngineStartInstanceOperator to start an existing Google Compute Engine instance.

Using the operator¶

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.py[source]

gce_instance_start = ComputeEngineStartInstanceOperator(
    project_id=GCP_PROJECT_ID, zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_start_task'
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection id used:

airflow/providers/google/cloud/example_dags/example_compute.py[source]

gce_instance_start2 = ComputeEngineStartInstanceOperator(
    zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_start_task2'
)

Templating¶

template_fields: Sequence[str] = (
    'project_id',
    'zone',
    'resource_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

More information¶

See Google Compute Engine API documentation to start an instance.

ComputeEngineStopInstanceOperator¶

Use the operator to stop Google Compute Engine instance.

For parameter definition, take a look at ComputeEngineStopInstanceOperator

Using the operator¶

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.py[source]

gce_instance_stop = ComputeEngineStopInstanceOperator(
    project_id=GCP_PROJECT_ID, zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_stop_task'
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute.py[source]

gce_instance_stop2 = ComputeEngineStopInstanceOperator(
    zone=GCE_ZONE, resource_id=GCE_INSTANCE, task_id='gcp_compute_stop_task2'
)

Templating¶

template_fields: Sequence[str] = (
    'project_id',
    'zone',
    'resource_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

More information¶

See Google Compute Engine API documentation to stop an instance.

ComputeEngineSetMachineTypeOperator¶

Use the operator to change machine type of a Google Compute Engine instance.

For parameter definition, take a look at ComputeEngineSetMachineTypeOperator.

Arguments¶

Using the operator¶

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute.py[source]

gce_set_machine_type = ComputeEngineSetMachineTypeOperator(
    project_id=GCP_PROJECT_ID,
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    body={'machineType': f'zones/{GCE_ZONE}/machineTypes/{GCE_SHORT_MACHINE_TYPE_NAME}'},
    task_id='gcp_compute_set_machine_type',
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute.py[source]

gce_set_machine_type2 = ComputeEngineSetMachineTypeOperator(
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    body={'machineType': f'zones/{GCE_ZONE}/machineTypes/{GCE_SHORT_MACHINE_TYPE_NAME}'},
    task_id='gcp_compute_set_machine_type2',
)

Templating¶

template_fields: Sequence[str] = (
    'project_id',
    'zone',
    'resource_id',
    'body',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

More information¶

See Google Compute Engine API documentation to set the machine type.

ComputeEngineCopyInstanceTemplateOperator¶

Use the operator to copy an existing Google Compute Engine instance template applying a patch to it.

For parameter definition, take a look at ComputeEngineCopyInstanceTemplateOperator.

Using the operator¶

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute_igm.py[source]

GCE_TEMPLATE_NAME = os.environ.get('GCE_TEMPLATE_NAME', 'instance-template-test')
GCE_NEW_TEMPLATE_NAME = os.environ.get('GCE_NEW_TEMPLATE_NAME', 'instance-template-test-new')
GCE_NEW_DESCRIPTION = os.environ.get('GCE_NEW_DESCRIPTION', 'Test new description')
GCE_INSTANCE_TEMPLATE_BODY_UPDATE = {
    "name": GCE_NEW_TEMPLATE_NAME,
    "description": GCE_NEW_DESCRIPTION,
    "properties": {"machineType": "n1-standard-2"},
}

airflow/providers/google/cloud/example_dags/example_compute_igm.py[source]

gce_instance_template_copy = ComputeEngineCopyInstanceTemplateOperator(
    project_id=GCP_PROJECT_ID,
    resource_id=GCE_TEMPLATE_NAME,
    body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
    task_id='gcp_compute_igm_copy_template_task',
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute_igm.py[source]

gce_instance_template_copy2 = ComputeEngineCopyInstanceTemplateOperator(
    resource_id=GCE_TEMPLATE_NAME,
    body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
    task_id='gcp_compute_igm_copy_template_task_2',
)

Templating¶

template_fields: Sequence[str] = (
    'project_id',
    'resource_id',
    'request_id',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

More information¶

See Google Compute Engine API documentation to create a new instance with an existing template.

ComputeEngineInstanceGroupUpdateManagerTemplateOperator¶

Use the operator to update a template in Google Compute Engine Instance Group Manager.

For parameter definition, take a look at ComputeEngineInstanceGroupUpdateManagerTemplateOperator.

Arguments¶

Using the operator¶

The code to create the operator:

airflow/providers/google/cloud/example_dags/example_compute_igm.py[source]

GCE_INSTANCE_GROUP_MANAGER_NAME = os.environ.get('GCE_INSTANCE_GROUP_MANAGER_NAME', 'instance-group-test')

SOURCE_TEMPLATE_URL = os.environ.get(
    'SOURCE_TEMPLATE_URL',
    "https://www.googleapis.com/compute/beta/projects/"
    + GCP_PROJECT_ID
    + "/global/instanceTemplates/instance-template-test",
)

DESTINATION_TEMPLATE_URL = os.environ.get(
    'DESTINATION_TEMPLATE_URL',
    "https://www.googleapis.com/compute/beta/projects/"
    + GCP_PROJECT_ID
    + "/global/instanceTemplates/"
    + GCE_NEW_TEMPLATE_NAME,
)

UPDATE_POLICY = {
    "type": "OPPORTUNISTIC",
    "minimalAction": "RESTART",
    "maxSurge": {"fixed": 1},
    "minReadySec": 1800,
}

airflow/providers/google/cloud/example_dags/example_compute_igm.py[source]

gce_instance_group_manager_update_template = ComputeEngineInstanceGroupUpdateManagerTemplateOperator(
    project_id=GCP_PROJECT_ID,
    resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
    zone=GCE_ZONE,
    source_template=SOURCE_TEMPLATE_URL,
    destination_template=DESTINATION_TEMPLATE_URL,
    update_policy=UPDATE_POLICY,
    task_id='gcp_compute_igm_group_manager_update_template',
)

You can also create the operator without project id - project id will be retrieved from the Google Cloud connection used:

airflow/providers/google/cloud/example_dags/example_compute_igm.py[source]

gce_instance_group_manager_update_template2 = ComputeEngineInstanceGroupUpdateManagerTemplateOperator(
    resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
    zone=GCE_ZONE,
    source_template=SOURCE_TEMPLATE_URL,
    destination_template=DESTINATION_TEMPLATE_URL,
    task_id='gcp_compute_igm_group_manager_update_template_2',
)

Templating¶

template_fields: Sequence[str] = (
    'project_id',
    'resource_id',
    'zone',
    'request_id',
    'source_template',
    'destination_template',
    'gcp_conn_id',
    'api_version',
    'impersonation_chain',
)

Troubleshooting¶

You might find that your ComputeEngineInstanceGroupUpdateManagerTemplateOperator fails with missing permissions. To execute the operation, the service account requires the permissions that theService Account User role provides (assigned via Google Cloud IAM).

More information¶

See Google Compute Engine API documentation to manage a group instance.

Reference¶

For further information, look at: