airflow.providers.docker.operators.docker_swarm

Run ephemeral Docker Swarm services

Module Contents

class airflow.providers.docker.operators.docker_swarm.DockerSwarmOperator(*, image: str, enable_logging: bool = True, configs: Optional[List[types.ConfigReference]] = None, secrets: Optional[List[types.SecretReference]] = None, mode: Optional[types.ServiceMode] = None, networks: Optional[List[Union[str, types.NetworkAttachmentConfig]]] = None, **kwargs)[source]

Bases: airflow.providers.docker.operators.docker.DockerOperator

Execute a command as an ephemeral docker swarm service. Example use-case - Using Docker Swarm orchestration to make one-time scripts highly available.

A temporary directory is created on the host and mounted into a container to allow storing files that together exceed the default disk size of 10GB in a container. The path to the mounted directory can be accessed via the environment variable AIRFLOW_TMP_DIR.

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

Parameters
  • image (str) -- Docker image from which to create the container. If image tag is omitted, "latest" will be used.

  • api_version (str) -- Remote API version. Set to auto to automatically detect the server's version.

  • auto_remove (bool) -- Auto-removal of the container on daemon side when the container's process exits. The default is False.

  • command (str or list) -- Command to be run in the container. (templated)

  • docker_url (str) -- URL of the host running the docker daemon. Default is unix://var/run/docker.sock

  • environment (dict) -- Environment variables to set in the container. (templated)

  • force_pull (bool) -- Pull the docker image on every run. Default is False.

  • mem_limit (float or str) -- Maximum amount of memory the container can use. Either a float value, which represents the limit in bytes, or a string like 128m or 1g.

  • tls_ca_cert (str) -- Path to a PEM-encoded certificate authority to secure the docker connection.

  • tls_client_cert (str) -- Path to the PEM-encoded certificate used to authenticate docker client.

  • tls_client_key (str) -- Path to the PEM-encoded key used to authenticate docker client.

  • tls_hostname (str or bool) -- Hostname to match against the docker server certificate or False to disable the check.

  • tls_ssl_version (str) -- Version of SSL to use when communicating with docker daemon.

  • tmp_dir (str) -- Mount point inside the container to a temporary directory created on the host by the operator. The path is also made available via the environment variable AIRFLOW_TMP_DIR inside the container.

  • user (int or str) -- Default user inside the docker container.

  • docker_conn_id (str) -- The Docker connection id

  • tty (bool) -- Allocate pseudo-TTY to the container of this service This needs to be set see logs of the Docker container / service.

  • enable_logging (bool) -- Show the application's logs in operator's logs. Supported only if the Docker engine is using json-file or journald logging drivers. The tty parameter should be set to use this with Python applications.

  • configs (List[docker.types.ConfigReference]) -- List of docker configs to be exposed to the containers of the swarm service. The configs are ConfigReference objects as per the docker api [https://docker-py.readthedocs.io/en/stable/services.html#docker.models.services.ServiceCollection.create]_

  • secrets (List[docker.types.SecretReference]) -- List of docker secrets to be exposed to the containers of the swarm service. The secrets are SecretReference objects as per the docker create_service api. [https://docker-py.readthedocs.io/en/stable/services.html#docker.models.services.ServiceCollection.create]_

  • mode (docker.types.ServiceMode) -- Indicate whether a service should be deployed as a replicated or global service, and associated parameters

  • networks (List[Union[str, NetworkAttachmentConfig]]) -- List of network names or IDs or NetworkAttachmentConfig to attach the service to.

execute(self, context)[source]
on_kill(self)[source]

Was this entry helpful?