Installation of airflowctl

This page describes installations options that you might use when considering how to install Airflow®. Airflow consists of many components, often distributed among many physical or virtual machines, therefore installation of Airflow might be quite complex, depending on the options you choose.

Using released sources

More details: Installing from Sources

When this option works best

Intended users

  • Users who are familiar with installing and building software from sources and are conscious about integrity and provenance of the software they use down to the lowest level possible.

What are you expected to handle

  • You are expected to build and install airflow and its components on your own.

  • You should develop and handle the deployment for all components of Airflow.

  • You are responsible for setting up database, creating and managing database schema with airflow db commands, automated startup and recovery, maintenance, cleanup and upgrades of Airflow and the Airflow Providers.

  • You need to setup monitoring of your system allowing you to observe resources and react to problems.

  • You are expected to configure and manage appropriate resources for the installation (memory, CPU, etc) based on the monitoring of your installation and feedback loop. See the notes about requirements.

What Apache Airflow Community provides for that method

  • You have instructions on how to build the software but due to various environments and tools you might want to use, you might expect that there will be problems which are specific to your deployment and environment you will have to diagnose and solve.

Where to ask for help

  • The #user-troubleshooting channel on slack can be used for quick general troubleshooting questions. The GitHub discussions if you look for longer discussion and have more information to share.

  • The #user-best-practices channel on slack can be used to ask for and share best practices on using and deploying airflow.

  • If you can provide description of a reproducible problem with Airflow software, you can open issue at GitHub issues

  • If you want to contribute back to Airflow, the #contributors slack channel for building the Airflow itself

Using PyPI

More details: Installation from PyPI

When this option works best

  • This installation method is useful when you are not familiar with Containers and Docker and want to install Apache Airflow on physical or virtual machines and you are used to installing and running software using custom deployment mechanism.

  • The only officially supported mechanism of installation is via pip using constraint mechanisms. The constraint files are managed by Apache Airflow release managers to make sure that you can repeatably install Airflow from PyPI with all Providers and required dependencies.

  • In case of PyPI installation you could also verify integrity and provenance of the packages downloaded from PyPI as described at the installation page, but software you download from PyPI is pre-built for you so that you can install it without building, and you do not build the software from sources.

Intended users

  • Users who are familiar with installing and configuring Python applications, managing Python environments, dependencies and running software with their custom deployment mechanisms.

What are you expected to handle

  • You are expected to install airflowctl.

  • You should run the Airflow API server.

  • You need to setup monitoring of your system allowing you to observe resources and react to problems.

What Apache Airflow Community provides for that method

  • You have Installation from PyPI on how to install the software but due to various environments and tools you might want to use, you might expect that there will be problems which are specific to your deployment and environment you will have to diagnose and solve.

  • You have Quick Start where you can see an example of Quick Start with running Airflow locally which you can use to start Airflow quickly for local testing and development. However, this is just for inspiration. Do not expect Quick Start is ready for production installation, you need to build your own production-ready deployment if you follow this approach.

Where to ask for help

  • The #user-troubleshooting channel on Airflow Slack for quick general troubleshooting questions. The GitHub discussions if you look for longer discussion and have more information to share.

  • The #user-best-practices channel on slack can be used to ask for and share best practices on using and deploying airflow.

  • If you can provide description of a reproducible problem with Airflow software, you can open issue at GitHub issues

Was this entry helpful?