airflow.providers.docker.operators.docker
¶
Implements Docker operator.
Module Contents¶
Classes¶
Execute a command inside a docker container. |
Functions¶
|
Make sure string is returned even if bytes are passed. Docker stream can return bytes. |
- airflow.providers.docker.operators.docker.stringify(line)[source]¶
Make sure string is returned even if bytes are passed. Docker stream can return bytes.
- class airflow.providers.docker.operators.docker.DockerOperator(*, image, api_version=None, command=None, container_name=None, cpus=1.0, docker_url=None, environment=None, private_environment=None, env_file=None, force_pull=False, mem_limit=None, host_tmp_dir=None, network_mode=None, tls_ca_cert=None, tls_client_cert=None, tls_client_key=None, tls_verify=True, tls_hostname=None, tls_ssl_version=None, mount_tmp_dir=True, tmp_dir='/tmp/airflow', user=None, mounts=None, entrypoint=None, working_dir=None, xcom_all=False, docker_conn_id=None, dns=None, dns_search=None, auto_remove='never', shm_size=None, tty=False, hostname=None, privileged=False, cap_add=None, extra_hosts=None, retrieve_output=False, retrieve_output_path=None, timeout=DEFAULT_TIMEOUT_SECONDS, device_requests=None, log_opts_max_size=None, log_opts_max_file=None, ipc_mode=None, skip_on_exit_code=None, port_bindings=None, ulimits=None, skip_exit_code=NOTSET, **kwargs)[source]¶
Bases:
airflow.models.BaseOperator
Execute a command inside a docker container.
By default, a temporary directory is created on the host and mounted into a container to allow storing files that together exceed the default disk size of 10GB in a container. In this case The path to the mounted directory can be accessed via the environment variable
AIRFLOW_TMP_DIR
.If the volume cannot be mounted, warning is printed and an attempt is made to execute the docker command without the temporary folder mounted. This is to make it works by default with remote docker engine or when you run docker-in-docker solution and temporary directory is not shared with the docker engine. Warning is printed in logs in this case.
If you know you run DockerOperator with remote engine or via docker-in-docker you should set
mount_tmp_dir
parameter to False. In this case, you can still usemounts
parameter to mount already existing named volumes in your Docker Engine to achieve similar capability where you can store files exceeding default disk size of the container,If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter
docker_conn_id
.- Parameters
image (str) – Docker image from which to create the container. If image tag is omitted, “latest” will be used. (templated)
api_version (str | None) – Remote API version. Set to
auto
to automatically detect the server’s version.command (str | list[str] | None) – Command to be run in the container. (templated)
container_name (str | None) – Name of the container. Optional (templated)
cpus (float) – Number of CPUs to assign to the container. This value gets multiplied with 1024. See https://docs.docker.com/engine/reference/run/#cpu-share-constraint
docker_url (str | list[str] | None) – URL or list of URLs of the host(s) running the docker daemon. Default is the value of the
DOCKER_HOST
environment variable or unix://var/run/docker.sock if it is unset.environment (dict | None) – Environment variables to set in the container. (templated)
private_environment (dict | None) – Private environment variables to set in the container. These are not templated, and hidden from the website.
env_file (str | None) – Relative path to the
.env
file with environment variables to set in the container. Overridden by variables in the environment parameter. (templated)force_pull (bool) – Pull the docker image on every run. Default is False.
mem_limit (float | str | None) – Maximum amount of memory the container can use. Either a float value, which represents the limit in bytes, or a string like
128m
or1g
.host_tmp_dir (str | None) – Specify the location of the temporary directory on the host which will be mapped to tmp_dir. If not provided defaults to using the standard system temp directory.
network_mode (str | None) –
Network mode for the container. It can be one of the following:
"bridge"
: Create new network stack for the container with default docker bridge network"none"
: No networking for this container"container:<name|id>"
: Use the network stack of another container specified via <name|id>"host"
: Use the host network stack. Incompatible with port_bindings"<network-name>|<network-id>"
: Connects the container to user created network (usingdocker network create
command)
tls_ca_cert (str | None) – Path to a PEM-encoded certificate authority to secure the docker connection.
tls_client_cert (str | None) – Path to the PEM-encoded certificate used to authenticate docker client.
tls_client_key (str | None) – Path to the PEM-encoded key used to authenticate docker client.
tls_verify (bool) – Set
True
to verify the validity of the provided certificate.tls_hostname (str | bool | None) – Hostname to match against the docker server certificate or False to disable the check.
tls_ssl_version (str | None) – Version of SSL to use when communicating with docker daemon.
mount_tmp_dir (bool) – Specify whether the temporary directory should be bind-mounted from the host to the container. Defaults to True
tmp_dir (str) – Mount point inside the container to a temporary directory created on the host by the operator. The path is also made available via the environment variable
AIRFLOW_TMP_DIR
inside the container.user (str | int | None) – Default user inside the docker container.
mounts (list[docker.types.Mount] | None) – List of volumes to mount into the container. Each item should be a
docker.types.Mount
instance.entrypoint (str | list[str] | None) – Overwrite the default ENTRYPOINT of the image
working_dir (str | None) – Working directory to set on the container (equivalent to the -w switch the docker client)
xcom_all (bool) – Push all the stdout or just the last line. The default is False (last line).
docker_conn_id (str | None) – The Docker connection id
dns_search (list[str] | None) – Docker custom DNS search domain
auto_remove (typing_extensions.Literal[never, success, force]) –
Enable removal of the container when the container’s process exits. Possible values:
never
: (default) do not remove containersuccess
: remove on successforce
: always remove container
shm_size (int | None) – Size of
/dev/shm
in bytes. The size must be greater than 0. If omitted uses system default.tty (bool) – Allocate pseudo-TTY to the container This needs to be set see logs of the Docker container.
hostname (str | None) – Optional hostname for the container.
privileged (bool) – Give extended privileges to this container.
cap_add (Iterable[str] | None) – Include container capabilities
extra_hosts (dict[str, str] | None) – Additional hostnames to resolve inside the container, as a mapping of hostname to IP address.
retrieve_output (bool) – Should this docker image consistently attempt to pull from and output file before manually shutting down the image. Useful for cases where users want a pickle serialized output that is not posted to logs
retrieve_output_path (str | None) – path for output file that will be retrieved and passed to xcom
timeout (int) – Timeout for API calls, in seconds. Default is 60 seconds.
device_requests (list[docker.types.DeviceRequest] | None) – Expose host resources such as GPUs to the container.
log_opts_max_size (str | None) – The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Eg: 10m or 1g Defaults to -1 (unlimited).
log_opts_max_file (str | None) – The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1.
ipc_mode (str | None) – Set the IPC mode for the container.
skip_on_exit_code (int | collections.abc.Container[int] | None) – If task exits with this exit code, leave the task in
skipped
state (default: None). If set toNone
, any non-zero exit code will be treated as a failure.port_bindings (dict | None) – Publish a container’s port(s) to the host. It is a dictionary of value where the key indicates the port to open inside the container and value indicates the host port that binds to the container port. Incompatible with
"host"
innetwork_mode
.ulimits (list[docker.types.Ulimit] | None) – List of ulimit options to set for the container. Each item should be a
docker.types.Ulimit
instance.
- template_fields: Sequence[str] = ('image', 'command', 'environment', 'env_file', 'container_name')[source]¶
- execute(context)[source]¶
Derive when creating an operator.
Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
- static format_command(command)[source]¶
Retrieve command(s).
If command string starts with
[
, the string is treated as a Python literal and parsed into a list of commands.