Module Contents



Start a container on Azure Container Instances







DEFAULT_CPU[source] :Dict[str, str][source] :Sequence[str] = [][source] :Sequence[Volume] = [][source] = 2.0[source] = 1.0[source]
class*, ci_conn_id, registry_conn_id, resource_group, name, image, region, environment_variables=None, secured_variables=None, volumes=None, memory_in_gb=None, cpu=None, gpu=None, command=None, remove_on_error=True, fail_if_exists=True, tags=None, os_type='Linux', restart_policy='Never', ip_address=None, ports=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Start a container on Azure Container Instances

  • ci_conn_id (str) – connection id of a service principal which will be used to start the container instance

  • registry_conn_id (Optional[str]) – connection id of a user which can login to a private docker registry. For Azure use Azure connection id

  • resource_group (str) – name of the resource group wherein this container instance should be started

  • name (str) – name of this container instance. Please note this name has to be unique in order to run containers in parallel.

  • image (str) – the docker image to be used

  • region (str) – the region wherein this container instance should be started

  • environment_variables (Optional[dict]) – key,value pairs containing environment variables which will be passed to the running container

  • secured_variables (Optional[str]) – names of environmental variables that should not be exposed outside the container (typically passwords).

  • volumes (Optional[list]) – list of Volume tuples to be mounted to the container. Currently only Azure Fileshares are supported.

  • memory_in_gb (Optional[Any]) – the amount of memory to allocate to this container

  • cpu (Optional[Any]) – the number of cpus to allocate to this container

  • gpu (Optional[Any]) – GPU Resource for the container.

  • command (Optional[List[str]]) – the command to run inside the container

  • container_timeout – max time allowed for the execution of the container instance.

  • tags (Optional[Dict[str, str]]) – azure tags as dict of str:str

  • os_type (str) – The operating system type required by the containers in the container group. Possible values include: ‘Windows’, ‘Linux’

  • restart_policy (str) – Restart policy for all containers within the container group. Possible values include: ‘Always’, ‘OnFailure’, ‘Never’

  • ip_address (Optional[azure.mgmt.containerinstance.models.IpAddress]) – The IP address type of the container group.


    ci_conn_id = "azure_service_principal",
    registry_conn_id = "azure_registry_user",
    resource_group = "my-resource-group",
    name = "my-container-name-{{ ds }}",
    image = "",
    region = "westeurope",
    environment_variables = {"MODEL_PATH":  "my_value",
     "POSTGRES_LOGIN": "{{ macros.connection('postgres_default').login }}",
     "POSTGRES_PASSWORD": "{{ macros.connection('postgres_default').password }}",
     "JOB_GUID": "{{ ti.xcom_pull(task_ids='task1', key='guid') }}" },
    secured_variables = ['POSTGRES_PASSWORD'],
    volumes = [("azure_container_instance_conn_id",
    gpu=GpuResource(count=1, sku='K80'),
    command=["/bin/echo", "world"],
template_fields :Sequence[str] = ['name', 'image', 'command', 'environment_variables'][source]
execute(self, context)[source]

This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.


Override this method to cleanup subprocesses when a task instance gets killed. Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up or it will leave ghost processes behind.

Was this entry helpful?