Implements Docker operator
DockerOperator(*, image: str, api_version: Optional[str] = None, command: Optional[Union[str, List[str]]] = None, container_name: Optional[str] = None, cpus: float = 1.0, docker_url: str = 'unix://var/run/docker.sock', environment: Optional[Dict] = None, private_environment: Optional[Dict] = None, force_pull: bool = False, mem_limit: Optional[Union[float, str]] = None, host_tmp_dir: Optional[str] = None, network_mode: Optional[str] = None, tls_ca_cert: Optional[str] = None, tls_client_cert: Optional[str] = None, tls_client_key: Optional[str] = None, tls_hostname: Optional[Union[str, bool]] = None, tls_ssl_version: Optional[str] = None, mount_tmp_dir: bool = True, tmp_dir: str = '/tmp/airflow', user: Optional[Union[str, int]] = None, mounts: Optional[List[Mount]] = None, entrypoint: Optional[Union[str, List[str]]] = None, working_dir: Optional[str] = None, xcom_all: bool = False, docker_conn_id: Optional[str] = None, dns: Optional[List[str]] = None, dns_search: Optional[List[str]] = None, auto_remove: bool = False, shm_size: Optional[int] = None, tty: bool = False, privileged: bool = False, cap_add: Optional[Iterable[str]] = None, extra_hosts: Optional[Dict[str, str]] = None, **kwargs)¶
Execute a command inside a docker container.
By default, a temporary directory is created on the host and mounted into a container to allow storing files that together exceed the default disk size of 10GB in a container. In this case The path to the mounted directory can be accessed via the environment variable
If the volume cannot be mounted, warning is printed and an attempt is made to execute the docker command without the temporary folder mounted. This is to make it works by default with remote docker engine or when you run docker-in-docker solution and temporary directory is not shared with the docker engine. Warning is printed in logs in this case.
If you know you run DockerOperator with remote engine or via docker-in-docker you should set
mount_tmp_dirparameter to False. In this case, you can still use
mountsparameter to mount already existing named volumes in your Docker Engine to achieve similar capability where you can store files exceeding default disk size of the container,
If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter
image (str) -- Docker image from which to create the container. If image tag is omitted, "latest" will be used.
api_version (str) -- Remote API version. Set to
autoto automatically detect the server's version.
cpus (float) -- Number of CPUs to assign to the container. This value gets multiplied with 1024. See https://docs.docker.com/engine/reference/run/#cpu-share-constraint
docker_url (str) -- URL of the host running the docker daemon. Default is unix://var/run/docker.sock
environment (dict) -- Environment variables to set in the container. (templated)
private_environment (dict) -- Private environment variables to set in the container. These are not templated, and hidden from the website.
force_pull (bool) -- Pull the docker image on every run. Default is False.
host_tmp_dir (str) -- Specify the location of the temporary directory on the host which will be mapped to tmp_dir. If not provided defaults to using the standard system temp directory.
network_mode (str) -- Network mode for the container.
tls_ca_cert (str) -- Path to a PEM-encoded certificate authority to secure the docker connection.
tls_client_cert (str) -- Path to the PEM-encoded certificate used to authenticate docker client.
tls_client_key (str) -- Path to the PEM-encoded key used to authenticate docker client.
tls_ssl_version (str) -- Version of SSL to use when communicating with docker daemon.
mount_tmp_dir (bool) -- Specify whether the temporary directory should be bind-mounted from the host to the container. Defaults to True
tmp_dir (str) -- Mount point inside the container to a temporary directory created on the host by the operator. The path is also made available via the environment variable
AIRFLOW_TMP_DIRinside the container.
working_dir (str) -- Working directory to set on the container (equivalent to the -w switch the docker client)
xcom_all (bool) -- Push all the stdout or just the last line. The default is False (last line).
auto_remove (bool) -- Auto-removal of the container on daemon side when the container's process exits. The default is False.
shm_size (int) -- Size of
/dev/shmin bytes. The size must be greater than 0. If omitted uses system default.
tty (bool) -- Allocate pseudo-TTY to the container This needs to be set see logs of the Docker container.
privileged (bool) -- Give extended privileges to this container.
template_fields= ['command', 'environment', 'container_name']¶
template_ext= ['.sh', '.bash']¶
Retrieves hook for the operator.
The Docker Hook
format_command(command: Union[str, List[str]])¶
Retrieve command(s). if command string starts with [, it returns the command list)