Dockerfile Changelog¶
The Dockerfile
does not strictly follow the SemVer approach of
Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly
follows it, the Dockerfile
is really a way to give users a conveniently packaged Airflow
using standard container approach, so occasionally there are some changes in the building process
or in the entrypoint of the image that require slight adaptation of how it is used or built.
The Changelog below describes the changes introduced in each version of the docker images released by the Airflow team.
- note
The Changelog below concerns only the convenience production images released at Airflow DockerHub . The images that are released there, are usually built using the
Dockerfile
released together with Airflow. However You are free to take latest releasedDockerfile
from Airflow and use it to build an image for any Airflow version from theAirflow 2
line. There is no guarantee that it works, but if it does, then you can use latest features from that image to build the previous Airflow versions.- warning
Some of the images below (as noted in the Changelog) have been regenerated using newer
Dockerfiles
. This happens when there is a breaking change that invalidates already released images and the images need regeneration. This has happened already when MySQL changed the keys they used to release their packages: #20911 and 2.1 images were all regenerated using the 2.2Dockerfile
. This is a rare event and we do it only when we have no choice because of external factors. In such case, the newer version of the image might contain breaking changes when it comes to running or building the image (but we try to avoid those).
Airflow 2.3¶
2.3.0
Airflow 2.3
Dockerfile
is now better optimized for caching and “standalone” which means that you can copy just theDockerfile
to any folder and start building custom images. This however requires Buildkit to build the image because we started using features that are only available inBuildkit
. This can be done by settingDOCKER_BUILDKIT=1
as an environment variable or by installing the buildx plugin and runningdocker buildx build
command.Add Python 3.10 support
Add support for Bullseye Debian release (Debian Buster is deprecated)
Add Multi-Platform support (AMD64/ARM64) in order to accommodate MacOS M1 users
Build parameters which control if packages and Airflow should be installed from context file were unified
The
INSTALL_FROM_PYPI
arg was replaced toAIRFLOW_IS_IN_CONTEXT
(with reverse meaning and default changed to false)The
INSTALL_FROM_DOCKER_CONTEXT_FILES
arg changed toINSTALL_PACKAGES_FROM_CONTEXT
Airflow 2.2¶
MySQL changed the keys to sign their packages on 17 Feb 2022. This caused all released images to fail when being extended. As result, on 18 Feb 2021 we re-released all the
2.2
and2.1
images with latest versions ofDockerfile
containing the new signing keys. The detailed issue here2.2.4 * Add support for both
.piprc
andpip.conf
customizations * Add ArtifactHub labels for better discovery of the images * Update default Python image to be 3.7 * Build images withBuildkit
(optional) * Fix building the image on Azure withtext file busy
error2.2.3 * No changes
2.2.2 * No changes
2.2.1 * Workaround the problem with
libstdcpp
TLS error2.2.0 * Remove AIRFLOW_GID (5000) from Airflow images (potentially breaking change for users using it) * Added warnings for Quick-start docker compose * Fix warm shutdown for celery worker (signal propagation) * Add Oauth libraries to PROD images * Add Python 3.9 support
Airflow 2.1¶
MySQL changed the keys to sign their packages on 17 Feb 2022. This caused all released images to fail when being extended. As result, on 18 Feb 2021 we re-released all the
2.2
and2.1
images with latest versions ofDockerfile
containing the new signing key.There were subtle changes in the behaviour of some 2.1 images due to that (more details below) Detailed issue here
- note
that the changes below were valid before image refreshing on 18 Feb 2022. Since all the images were refreshed on 18 Feb with the same
Dockerfile
as 2.1.4, the changes 2.1.1 -> 2.1.3 are effectively applied to all the images in 2.1.* line. The images refreshed have also those fixes added:
All 2.1.* image versions refreshed on 18 Feb 2022 have those fixes applied: * Fix building the image on Azure with
text file busy
error * Workaround the problem withlibstdcpp
TLS error * Remove AIRFLOW_GID (5000) from Airflow images (potentially breaking change for users using it) * Added warnings for Quick-start docker compose * Add Oauth libraries to PROD images
Original image Changelog (before the refresh on 18 Feb 2022):
- 2.1.4
Workaround the problem with
libstdcpp
TLS errorfixed detection of port number in connection URL
Improve warnings for quick-start-docker compose
Fix warm shutdown for celery worker (signal propagation)
- 2.1.3
fixed auto-creation of user to use non-deprecated
create user
rather thanuser_create
remove waiting for celery backend for
worker
andflower
commands rather thanscheduler
andcelery
onlyremove deprecated
airflow upgradedb
command from Airflow 1.10 in case upgrade is requestedAdd Python 3.9 support
- 2.1.2
No changes
- 2.1.1
Fix failure of lack of default commands (failed when no commands were passed)
Added
_PIP_ADDITIONAL_REQUIREMENTS
development feature
- 2.1.0
Unset default
PIP_USER
variable - which caused PythonVirtualEnv to fail
Airflow 2.0¶
MySQL changed the keys to sign their packages on 17 Feb 2022. This caused all released images to fail when being extended. As result, on 18 Feb 2021 we re-released all the
2.2
and2.1
images with latest versions ofDockerfile
containing the new signing key.There were no changes in the behaviour of 2.0.2 image due to that Detailed issue here . Only 2.0.2 image was regenerated, as 2.0.1 and 2.0.0 versions are hardly used and it is unlikely someone would like to extend those images. Extending 2.0.1 and 2.0.0 images will lead to failures of “missing key”.
- 2.0.2
Set correct PYTHONPATH for
root
user. Allows to run the image as rootWarn if the deprecated 5000 group ID was used for airflow user when running the image (should be 0 for the OpenShift compatibility). Fails if the group 5000 was used with any other user (it would not work anyway but with cryptic errors)
Set umask as 002 by default, so that you can actually change the user id used to run the image (required for OpenShift compatibility)
Skip checking the DB and celery backend if CONNECTION_CHECK_MAX_COUNT is equal to 0
- 2.0.1
Avoid reverse IP lookup when checking DB availability. This helped to solve long delays on misconfigured docker engines
Add auto-detection of redis and amqp broker ports
Fixed detection of all user/password combinations in URLs - helps in auto-detecting ports and testing connectivity
Add possibility to create Admin user automatically when entering the image
Automatically create system user when different user than
airflow
is used. Needed for OpenShift compatibilityAllows to exec to
bash
orpython
if specified as parametersRemove
airflow
command if it is specified as first parameter of therun
command
- 2.0.0
Initial release of the image based on Debian Buster