Extra Packages Reference

Here's the list of the subpackages and what they enable:

Fundamentals:

subpackage

install command

enables

all

pip install 'apache-airflow[all]'

All Airflow user facing features (no devel and doc requirements)

all_dbs

pip install 'apache-airflow[all_dbs]'

All databases integrations

devel

pip install 'apache-airflow[devel]'

Minimum dev tools requirements (without all)

devel_hadoop

pip install 'apache-airflow[devel_hadoop]'

Same as devel + dependencies for developing the Hadoop stack

devel_all

pip install 'apache-airflow[devel_all]'

Everything needed for development (devel_hadoop + all)

devel_ci

pip install 'apache-airflow[devel_ci]'

All dependencies required for CI build.

doc

pip install 'apache-airflow[doc]'

Packages needed to build docs (included in devel)

password

pip install 'apache-airflow[password]'

Password authentication for users

Apache Software:

subpackage

install command

enables

atlas

pip install 'apache-airflow[apache.atlas]'

Apache Atlas to use Data Lineage feature

beam

pip install 'apache-airflow[apache.beam]'

Apache Beam operators & hooks

cassandra

pip install 'apache-airflow[apache.cassandra]'

Cassandra related operators & hooks

druid

pip install 'apache-airflow[apache.druid]'

Druid related operators & hooks

hdfs

pip install 'apache-airflow[apache.hdfs]'

HDFS hooks and operators

hive

pip install 'apache-airflow[apache.hive]'

All Hive related operators

kylin

pip install 'apache-airflow[apache.kylin]'

All Kylin related operators & hooks

livy

pip install 'apache-airflow[apache.livy]'

All Livy related operators, hooks & sensors

pig

pip install 'apache-airflow[apache.pig]'

All Pig related operators & hooks

pinot

pip install 'apache-airflow[apache.pinot]'

All Pinot related hooks

spark

pip install 'apache-airflow[apache.spark]'

All Spark related operators & hooks

sqoop

pip install 'apache-airflow[apache.sqoop]'

All Sqoop related operators & hooks

webhdfs

pip install 'apache-airflow[apache.webhdfs]'

HDFS hooks and operators

Services:

subpackage

install command

enables

aws

pip install 'apache-airflow[amazon]'

Amazon Web Services

azure

pip install 'apache-airflow[microsoft.azure]'

Microsoft Azure

cloudant

pip install 'apache-airflow[cloudant]'

Cloudant hook

databricks

pip install 'apache-airflow[databricks]'

Databricks hooks and operators

datadog

pip install 'apache-airflow[datadog]'

Datadog hooks and sensors

dingding

pip install 'apache-airflow[dingding]'

Dingding hooks and sensors

discord

pip install 'apache-airflow[discord]'

Discord hooks and sensors

facebook

pip install 'apache-airflow[facebook]'

Facebook Social

gcp

pip install 'apache-airflow[google]'

Google Cloud

github_enterprise

pip install 'apache-airflow[github_enterprise]'

GitHub Enterprise auth backend

google_auth

pip install 'apache-airflow[google_auth]'

Google auth backend

hashicorp

pip install 'apache-airflow[hashicorp]'

Hashicorp Services (Vault)

jira

pip install 'apache-airflow[jira]'

Jira hooks and operators

opsgenie

pip install 'apache-airflow[opsgenie]'

OpsGenie hooks and operators

pagerduty

pip install 'apache-airflow[pagerduty]'

Pagerduty hook

plexus

pip install 'apache-airflow[plexus]'

Plexus service of CoreScientific.com AI platform

qds

pip install 'apache-airflow[qds]'

Enable QDS (Qubole Data Service) support

salesforce

pip install 'apache-airflow[salesforce]'

Salesforce hook

sendgrid

pip install 'apache-airflow[sendgrid]'

Send email using sendgrid

segment

pip install 'apache-airflow[segment]'

Segment hooks and sensors

sentry

pip install 'apache-airflow[sentry]'

Sentry service for application logging and monitoring

slack

pip install 'apache-airflow[slack]'

airflow.providers.slack.operators.slack.SlackAPIOperator

snowflake

pip install 'apache-airflow[snowflake]'

Snowflake hooks and operators

telegram

pip install 'apache-airflow[telegram]'

airflow.providers.telegram.operators.telegram.TelegramOperator

vertica

pip install 'apache-airflow[vertica]'

Vertica hook support as an Airflow backend

yandex

pip install 'apache-airflow[yandex]'

Yandex.cloud hooks and operators

zendesk

pip install 'apache-airflow[zendesk]'

Zendesk hooks

Software:

subpackage

install command

enables

async

pip install 'apache-airflow[async]'

Async worker classes for Gunicorn

celery

pip install 'apache-airflow[celery]'

CeleryExecutor

dask

pip install 'apache-airflow[dask]'

DaskExecutor

docker

pip install 'apache-airflow[docker]'

Docker hooks and operators

elasticsearch

pip install 'apache-airflow[elasticsearch]'

Elasticsearch hooks and Log Handler

exasol

pip install 'apache-airflow[exasol]'

Exasol hooks and operators

jenkins

pip install 'apache-airflow[jenkins]'

Jenkins hooks and operators

kubernetes

pip install 'apache-airflow[cncf.kubernetes]'

Kubernetes Executor and operator

mongo

pip install 'apache-airflow[mongo]'

Mongo hooks and operators

mssql (deprecated)

pip install 'apache-airflow[microsoft.mssql]'

Microsoft SQL Server operators and hook, support as an Airflow backend. Uses pymssql. Will be replaced by subpackage odbc.

mysql

pip install 'apache-airflow[mysql]'

MySQL operators and hook, support as an Airflow backend. The version of MySQL server has to be 5.6.4+. The exact version upper bound depends on version of mysqlclient package. For example, mysqlclient 1.3.12 can only be used with MySQL server 5.6.4 through 5.7.

odbc

pip install 'apache-airflow[odbc]'

ODBC data sources including MS SQL Server. Can use MsSqlOperator, or as metastore database backend. Uses pyodbc. See apache-airflow-providers-odbc for more info.

openfaas

pip install 'apache-airflow[openfaas]'

OpenFaaS hooks

oracle

pip install 'apache-airflow[oracle]'

Oracle hooks and operators

postgres

pip install 'apache-airflow[postgres]'

PostgreSQL operators and hook, support as an Airflow backend

presto

pip install 'apache-airflow[presto]'

All Presto related operators & hooks

rabbitmq

pip install 'apache-airflow[rabbitmq]'

RabbitMQ support as a Celery backend

redis

pip install 'apache-airflow[redis]'

Redis hooks and sensors

samba

pip install 'apache-airflow[samba]'

airflow.providers.apache.hive.transfers.hive_to_samba.HiveToSambaOperator

singularity

pip install 'apache-airflow[singularity]'

Singularity container operator

statsd

pip install 'apache-airflow[statsd]'

Needed by StatsD metrics

tableau

pip install 'apache-airflow[tableau]'

Tableau visualization integration

virtualenv

pip install 'apache-airflow[virtualenv]'

Running python tasks in local virtualenv

Other:

subpackage

install command

enables

cgroups

pip install 'apache-airflow[cgroups]'

Needed To use CgroupTaskRunner

ftp

pip install 'apache-airflow[ftp]'

FTP hooks and operators

grpc

pip install 'apache-airflow[grpc]'

Grpc hooks and operators

http

pip install 'apache-airflow[http]'

HTTP hooks, operators and sensors

imap

pip install 'apache-airflow[imap]'

IMAP hooks and sensors

jdbc

pip install 'apache-airflow[jdbc]'

JDBC hooks and operators

kerberos

pip install 'apache-airflow[kerberos]'

Kerberos integration for Kerberized Hadoop

ldap

pip install 'apache-airflow[ldap]'

LDAP authentication for users

papermill

pip install 'apache-airflow[papermill]'

Papermill hooks and operators

sftp

pip install 'apache-airflow[sftp]'

SFTP hooks, operators and sensors

sqlite

pip install 'apache-airflow[sqlite]'

SQLite hooks and operators

ssh

pip install 'apache-airflow[ssh]'

SSH hooks and operators

winrm

pip install 'apache-airflow[microsoft.winrm]'

WinRM hooks and operators

** Deprecated 1.10 Extras **

Those are the extras that have been deprecated in 2.0 and will be removed (current plan is to remove them in 2.1):

Deprecated extra

Extra to be used instead

atlas

apache.atlas

aws

amazon

azure

microsoft.azure

cassandra

apache.cassandra

crypto

druid

apache.druid

gcp

google

gcp_api

google

hdfs

apache.hdfs

hive

apache.hive

kubernetes

cncf.kubernetes

mssql

microsoft.mssql

pinot

apache.pinot

qds

qubole

s3

amazon

webhdfs

apache.webhdfs

winrm

microsoft.winrm

  • crypto extra is not needed as cryptography is installed by default in Airflow 2.0

Was this entry helpful?