Reference for package extras¶
Here’s the list of all the extra dependencies.
Most of the the extras result in installing additional provider packages.
The entries with *
in the Providers
column indicate that one or more provider packages will be
installed automatically when those extras are used.
For provider
extras - they usually install single provider package.
For bundle
extras - they usually install a group of extras (for example all
or devel_all
or all_dbs
), and thus it will include two or more provider extras.
The entries with *
in the Preinstalled
column indicate that those extras (with providers) are always
pre-installed when Airflow is installed.
Note
You can disable automated installation of the providers with extras when installing Airflow. You need to
have INSTALL_PROVIDERS_FROM_SOURCES
environment variable to true
before running pip install
command. Contributors need to set it, if they are installing Airflow locally, and want to develop
providers directly via Airflow sources. This variable is automatically set in Breeze
development environment. Setting this variable is not needed in editable mode (pip install -e
).
Apache Software extras:
Those are extras that add dependencies needed for integration with other Apache projects.
extra |
install command |
enables |
Providers |
---|---|---|---|
apache.atlas |
|
Apache Atlas to use Data Lineage feature |
|
apache.beam |
|
Apache Beam operators & hooks |
|
apache.cassandra |
|
Cassandra related operators & hooks |
|
apache.druid |
|
Druid related operators & hooks |
|
apache.hdfs |
|
HDFS hooks and operators |
|
apache.hive |
|
All Hive related operators |
|
apache.kylin |
|
All Kylin related operators & hooks |
|
apache.livy |
|
All Livy related operators, hooks & sensors |
|
apache.pig |
|
All Pig related operators & hooks |
|
apache.pinot |
|
All Pinot related hooks |
|
apache.spark |
|
All Spark related operators & hooks |
|
apache.sqoop |
|
All Sqoop related operators & hooks |
|
apache.webhdfs |
|
HDFS hooks and operators |
Services extras:
Those are extras that add dependencies needed for integration with external services - either cloud based or on-premises.
extra |
install command |
enables |
Providers |
---|---|---|---|
amazon |
|
Amazon Web Services |
|
azure |
|
Microsoft Azure |
|
cloudant |
|
Cloudant hook |
|
databricks |
|
Databricks hooks and operators |
|
datadog |
|
Datadog hooks and sensors |
|
dask |
|
DaskExecutor |
|
dingding |
|
Dingding hooks and sensors |
|
discord |
|
Discord hooks and sensors |
|
|
Facebook Social |
||
|
Google Cloud |
||
github_enterprise |
|
GitHub Enterprise auth backend |
|
google_auth |
|
Google auth backend |
|
hashicorp |
|
Hashicorp Services (Vault) |
|
jira |
|
Jira hooks and operators |
|
opsgenie |
|
OpsGenie hooks and operators |
|
pagerduty |
|
Pagerduty hook |
|
plexus |
|
Plexus service of CoreScientific.com AI platform |
|
qubole |
|
Enable QDS (Qubole Data Service) support |
|
salesforce |
|
Salesforce hook |
|
sendgrid |
|
Send email using sendgrid |
|
segment |
|
Segment hooks and sensors |
|
sentry |
|
Sentry service for application logging and monitoring |
|
slack |
|
Slack hooks and operators |
|
snowflake |
|
Snowflake hooks and operators |
|
telegram |
|
Telegram hooks and operators |
|
vertica |
|
Vertica hook support as an Airflow backend |
|
yandex |
|
Yandex.cloud hooks and operators |
|
zendesk |
|
Zendesk hooks |
Software extras:
Those are extras that add dependencies needed for integration with other software packages installed usually as part of the deployment of Airflow.
extra |
install command |
enables |
Providers |
---|---|---|---|
async |
|
Async worker classes for Gunicorn |
|
celery |
|
CeleryExecutor |
|
cncf.kubernetes |
|
Kubernetes Executor and operator |
|
docker |
|
Docker hooks and operators |
|
elasticsearch |
|
Elasticsearch hooks and Log Handler |
|
exasol |
|
Exasol hooks and operators |
|
jenkins |
|
Jenkins hooks and operators |
|
ldap |
|
LDAP authentication for users |
|
mongo |
|
Mongo hooks and operators |
|
microsoft.mssql |
|
Microsoft SQL Server operators and hook. |
|
mysql |
|
MySQL operators and hook |
|
odbc |
|
ODBC data sources including MS SQL Server |
|
openfaas |
|
OpenFaaS hooks |
|
oracle |
|
Oracle hooks and operators |
|
postgres |
|
PostgreSQL operators and hook |
|
password |
|
Password authentication for users |
|
presto |
|
All Presto related operators & hooks |
|
rabbitmq |
|
RabbitMQ support as a Celery backend |
|
redis |
|
Redis hooks and sensors |
|
samba |
|
Samba hooks and operators |
|
singularity |
|
Singularity container operator |
|
statsd |
|
Needed by StatsD metrics |
|
tableau |
|
Tableau visualization integration |
|
virtualenv |
|
Running python tasks in local virtualenv |
Other extras:
Those are extras that provide support for integration with external systems via some - usually - standard protocols.
extra |
install command |
enables |
Providers |
Preinstalled |
---|---|---|---|---|
cgroups |
|
Needed To use CgroupTaskRunner |
||
ftp |
|
FTP hooks and operators |
||
grpc |
|
Grpc hooks and operators |
||
http |
|
HTTP hooks, operators and sensors |
||
imap |
|
IMAP hooks and sensors |
||
jdbc |
|
JDBC hooks and operators |
||
kerberos |
|
Kerberos integration for Kerberized services (Hadoop, Presto) |
||
papermill |
|
Papermill hooks and operators |
||
sftp |
|
SFTP hooks, operators and sensors |
||
sqlite |
|
SQLite hooks and operators |
||
ssh |
|
SSH hooks and operators |
||
microsoft.winrm |
|
WinRM hooks and operators |
Bundle extras:
Those are extras that install one ore more extras.
extra |
install command |
enables |
Providers |
---|---|---|---|
all |
|
All Airflow user facing features (no devel and doc requirements) |
|
all_dbs |
|
All databases integrations |
|
devel |
|
Minimum dev tools requirements (without providers) |
|
devel_hadoop |
|
Same as |
|
devel_all |
|
Everything needed for development ( |
|
devel_ci |
|
All dependencies required for CI build. |
Doc extras:
This is the extra that is needed to generated documentation for Airflow. This is used for development time only
doc |
|
Packages needed to build docs (included in |
Deprecated 1.10 extras:
Those are the extras that have been deprecated in 2.0 and will be removed in Airflow 3.0.0. They were all replaced by new extras, which have naming consistent with the names of provider packages.
The crypto
extra is not needed any more, because all crypto dependencies are part of airflow package,
so there is no replacement for crypto
extra.
Deprecated extra |
Extra to be used instead |
---|---|
atlas |
apache.atlas |
aws |
amazon |
azure |
microsoft.azure |
cassandra |
apache.cassandra |
crypto |
|
druid |
apache.druid |
gcp |
|
gcp_api |
|
hdfs |
apache.hdfs |
hive |
apache.hive |
kubernetes |
cncf.kubernetes |
mssql |
microsoft.mssql |
pinot |
apache.pinot |
qds |
qubole |
s3 |
amazon |
spark |
apache.spark |
webhdfs |
apache.webhdfs |
winrm |
microsoft.winrm |