airflow.contrib.operators.azure_container_instances_operator

Module Contents

airflow.contrib.operators.azure_container_instances_operator.Volume[source]
airflow.contrib.operators.azure_container_instances_operator.DEFAULT_ENVIRONMENT_VARIABLES :Dict[str, str][source]
airflow.contrib.operators.azure_container_instances_operator.DEFAULT_SECURED_VARIABLES :Sequence[str] = [][source]
airflow.contrib.operators.azure_container_instances_operator.DEFAULT_VOLUMES :Sequence[Volume] = [][source]
airflow.contrib.operators.azure_container_instances_operator.DEFAULT_MEMORY_IN_GB = 2.0[source]
airflow.contrib.operators.azure_container_instances_operator.DEFAULT_CPU = 1.0[source]
class airflow.contrib.operators.azure_container_instances_operator.AzureContainerInstancesOperator(ci_conn_id, registry_conn_id, resource_group, name, image, region, environment_variables=None, secured_variables=None, volumes=None, memory_in_gb=None, cpu=None, gpu=None, command=None, remove_on_error=True, fail_if_exists=True, tags=None, *args, **kwargs)[source]

Bases: airflow.models.BaseOperator

Start a container on Azure Container Instances

Parameters

  • ci_conn_id (str) – connection id of a service principal which will be used to start the container instance

  • registry_conn_id (str) – connection id of a user which can login to a private docker registry. If None, we assume a public registry

  • resource_group (str) – name of the resource group wherein this container instance should be started

  • name (str) – name of this container instance. Please note this name has to be unique in order to run containers in parallel.

  • image (str) – the docker image to be used

  • region (str) – the region wherein this container instance should be started

  • environment_variables (dict) – key,value pairs containing environment variables which will be passed to the running container

  • secured_variables ([str]) – names of environmental variables that should not be exposed outside the container (typically passwords).

  • volumes (list[<conn_id, account_name, share_name, mount_path, read_only>]) – list of Volume tuples to be mounted to the container. Currently only Azure Fileshares are supported.

  • memory_in_gb (double) – the amount of memory to allocate to this container

  • cpu (double) – the number of cpus to allocate to this container

  • gpu (azure.mgmt.containerinstance.models.GpuResource) – GPU Resource for the container.

  • command ([str]) – the command to run inside the container

  • container_timeout (datetime.timedelta) – max time allowed for the execution of the container instance.

  • tags (dict[str, str]) – azure tags as dict of str:str

Example:

AzureContainerInstancesOperator(
    "azure_service_principal",
    "azure_registry_user",
    "my-resource-group",
    "my-container-name-{{ ds }}",
    "myprivateregistry.azurecr.io/my_container:latest",
    "westeurope",
    {"MODEL_PATH":  "my_value",
     "POSTGRES_LOGIN": "{{ macros.connection('postgres_default').login }}"
     "POSTGRES_PASSWORD": "{{ macros.connection('postgres_default').password }}",
     "JOB_GUID": "{{ ti.xcom_pull(task_ids='task1', key='guid') }}" },
    ['POSTGRES_PASSWORD'],
    [("azure_wasb_conn_id",
    "my_storage_container",
    "my_fileshare",
    "/input-data",
    True),],
    memory_in_gb=14.0,
    cpu=4.0,
    gpu=GpuResource(count=1, sku='K80'),
    command=["/bin/echo", "world"],
    container_timeout=timedelta(hours=2),
    task_id="start_container"
)
template_fields = ['name', 'image', 'command', 'environment_variables'][source]
execute(self, context)[source]
on_kill(self)[source]
_monitor_logging(self, ci_hook, resource_group, name)[source]
_log_last(self, logs, last_line_logged)[source]
static _check_name(name)[source]

Was this entry helpful?