airflow.operators.docker_operator

Module Contents

class airflow.operators.docker_operator.DockerOperator(image, api_version=None, command=None, cpus=1.0, docker_url='unix://var/run/docker.sock', environment=None, force_pull=False, mem_limit=None, host_tmp_dir=None, network_mode=None, tls_ca_cert=None, tls_client_cert=None, tls_client_key=None, tls_hostname=None, tls_ssl_version=None, tmp_dir='/tmp/airflow', user=None, volumes=None, working_dir=None, xcom_push=False, xcom_all=False, docker_conn_id=None, dns=None, dns_search=None, auto_remove=False, shm_size=None, *args, **kwargs)

Bases: airflow.models.BaseOperator

Execute a command inside a docker container.

A temporary directory is created on the host and mounted into a container to allow storing files that together exceed the default disk size of 10GB in a container. The path to the mounted directory can be accessed via the environment variable AIRFLOW_TMP_DIR.

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

Parameters

• image (str) – Docker image from which to create the container. If image tag is omitted, “latest” will be used.

• api_version (str) – Remote API version. Set to auto to automatically detect the server’s version.

• auto_remove (bool) – Auto-removal of the container on daemon side when the container’s process exits. The default is False.

• command (str or list) – Command to be run in the container. (templated)

• cpus (float) – Number of CPUs to assign to the container. This value gets multiplied with 1024. See https://docs.docker.com/engine/reference/run/#cpu-share-constraint

• dns (list[str]) – Docker custom DNS servers

• dns_search (list[str]) – Docker custom DNS search domain

• docker_url (str) – URL of the host running the docker daemon. Default is unix://var/run/docker.sock

• environment (dict) – Environment variables to set in the container. (templated)

• force_pull (bool) – Pull the docker image on every run. Default is False.

• mem_limit (float or str) – Maximum amount of memory the container can use. Either a float value, which represents the limit in bytes, or a string like 128m or 1g.

• host_tmp_dir (str) – Specify the location of the temporary directory on the host which will be mapped to tmp_dir. If not provided defaults to using the standard system temp directory.

• network_mode (str) – Network mode for the container.

• tls_ca_cert (str) – Path to a PEM-encoded certificate authority to secure the docker connection.

• tls_client_cert (str) – Path to the PEM-encoded certificate used to authenticate docker client.

• tls_client_key (str) – Path to the PEM-encoded key used to authenticate docker client.

• tls_hostname (str or bool) – Hostname to match against the docker server certificate or False to disable the check.

• tls_ssl_version (str) – Version of SSL to use when communicating with docker daemon.

• tmp_dir (str) – Mount point inside the container to a temporary directory created on the host by the operator. The path is also made available via the environment variable AIRFLOW_TMP_DIR inside the container.

• user (int or str) – Default user inside the docker container.

• volumes (list) – List of volumes to mount into the container, e.g. ['/host/path:/container/path', '/host/path2:/container/path2:ro'].

• working_dir (str) – Working directory to set on the container (equivalent to the -w switch the docker client)

• xcom_push (bool) – Does the stdout will be pushed to the next step using XCom. The default is False.

• xcom_all (bool) – Push all the stdout or just the last line. The default is False (last line).

• docker_conn_id (str) – ID of the Airflow connection to use

• shm_size (int) – Size of /dev/shm in bytes. The size must be greater than 0. If omitted uses system default.

template_fields = ['command', 'environment']

template_ext = ['.sh', '.bash']

get_hook(self)

execute(self, context)

get_command(self)

on_kill(self)

__get_tls_config(self)