Content¶
References
Commits
apache-airflow-providers-microsoft-azure¶
Provider package¶
This is a provider package for microsoft.azure provider. All classes for this provider package
are in airflow.providers.microsoft.azure python package.
Installation¶
You can install this package on top of an existing Airflow 2 installation (see Requirements below)
for the minimum Airflow version supported) via
pip install apache-airflow-providers-microsoft-azure
Requirements¶
The minimum Apache Airflow version supported by this provider package is 2.4.0.
PIP package |
Version required |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Cross provider package dependencies¶
Those are dependencies that might be needed in order to use all the features of the package. You need to install the specified provider packages in order to use them.
You can install such cross-provider dependencies when installing from PyPI. For example:
pip install apache-airflow-providers-microsoft-azure[google]
Dependent package |
Extra |
|---|---|
|
|
|
|
|
Downloading official packages¶
You can download officially released packages and verify their checksums and signatures from the Official Apache Download site
The apache-airflow-providers-microsoft-azure 6.1.1 sdist package (asc, sha512)
The apache-airflow-providers-microsoft-azure 6.1.1 wheel package (asc, sha512)
Changelog¶
6.1.1¶
Bug Fixes¶
Fix deferrable mode execution in WasbPrefixSensor (#31411)
Misc¶
Optimize deferred mode execution for wasb sensors (#31009)
6.1.0¶
Note
This release of provider is only available for Airflow 2.4+ as explained in the Apache Airflow providers support policy.
Features¶
Add deferrable mode to 'WasbPrefixSensor' (#30252)
Misc¶
Bump minimum Airflow version in providers (#30917)Optimize deferrable execution mode 'AzureDataFactoryPipelineRunStatusSensor' (#30983)Optimize deferred execution for AzureDataFactoryRunPipelineOperator (#31214)
6.0.0¶
Breaking changes¶
Warning
In this version of the provider, deprecated GCS hook’s param delegate_to is removed from AzureBlobStorageToGCSOperator.
Impersonation can be achieved instead by utilizing the impersonation_chain param.
remove delegate_to from GCP operators and hooks (#30748)
Misc¶
Merge WasbBlobAsyncSensor to WasbBlobSensor (#30488)
5.3.1¶
Bug Fixes¶
Fix AzureDataFactoryPipelineRunLink get_link method (#30514)Load subscription_id from extra__azure__subscriptionId (#30556)
5.3.0¶
Features¶
Add deferrable 'AzureDataFactoryRunPipelineOperator' (#30147)Add deferrable 'AzureDataFactoryPipelineRunStatusSensor' (#29801)Support deleting the local log files when using remote logging (#29772)
Bug Fixes¶
Fix ADF job failure during deferral (#30248)Fix AzureDataLakeStorageV2Hook 'account_url' with Active Directory authentication (#29980) (#29981)
Misc¶
merge AzureDataFactoryPipelineRunStatusAsyncSensor to AzureDataFactoryPipelineRunStatusSensor (#30250)Expose missing params in AzureSynapseHook API docs (#30099)
5.2.0¶
Features¶
Enable individual trigger logging (#27758)
Bug Fixes¶
Fix params rendering in AzureSynapseHook Python API docs (#29041)
Misc¶
Deprecate 'delegate_to' param in GCP operators and update docs (#29088)
5.1.0¶
Features¶
Add hook for Azure Data Lake Storage Gen2 (#28262)
Bug Fixes¶
Hide 'extra' field in WASB connection form (#28914)
5.0.0¶
Note
This release of provider is only available for Airflow 2.3+ as explained in the Apache Airflow providers support policy.
Breaking changes¶
In AzureFileShareHook, if both
extra__azure_fileshare__fooandfooexisted in connection extra dict, the prefixed version would be used; now, the non-prefixed version will be preferred.Remove deprecated classes (#27417)In Azure Batch
vm_sizeandvm_node_agent_sku_idparameters are required.
Misc¶
Move min airflow version to 2.3.0 for all providers (#27196)
Features¶
Add azure, google, authentication library limits to eaager upgrade (#27535)Allow and prefer non-prefixed extra fields for remaining azure (#27220)Allow and prefer non-prefixed extra fields for AzureFileShareHook (#27041)Allow and prefer non-prefixed extra fields for AzureDataExplorerHook (#27219)Allow and prefer non-prefixed extra fields for AzureDataFactoryHook (#27047)Update WasbHook to reflect preference for unprefixed extra (#27024)Look for 'extra__' instead of 'extra_' in 'get_field' (#27489)
Bug Fixes¶
Fix Azure Batch errors revealed by added typing to azure batch lib (#27601)Fix separator getting added to variables_prefix when empty (#26749)
4.3.0¶
Features¶
Add DataFlow operations to Azure DataFactory hook (#26345)Add network_profile param in AzureContainerInstancesOperator (#26117)Add Azure synapse operator (#26038)Auto tail file logs in Web UI (#26169)Implement Azure Service Bus Topic Create, Delete Operators (#25436)
Bug Fixes¶
Fix AzureBatchOperator false negative task status (#25844)
4.2.0¶
Features¶
Add 'test_connection' method to AzureContainerInstanceHook (#25362)Add test_connection to Azure Batch hook (#25235)Bump typing-extensions and mypy for ParamSpec (#25088)Implement Azure Service Bus (Update and Receive) Subscription Operator (#25029)Set default wasb Azure http logging level to warning; fixes #16224 (#18896)
4.1.0¶
Features¶
Add 'test_connection' method to AzureCosmosDBHook (#25018)Add test_connection method to AzureFileShareHook (#24843)Add test_connection method to Azure WasbHook (#24771)Implement Azure service bus subscription Operators (#24625)Implement Azure Service Bus Queue Operators (#24038)
Bug Fixes¶
Update providers to use functools compat for ''cached_property'' (#24582)
4.0.0¶
Breaking changes¶
Note
This release of provider is only available for Airflow 2.2+ as explained in the Apache Airflow providers support policy.
Features¶
Pass connection extra parameters to wasb BlobServiceClient (#24154)
Misc¶
Apply per-run log templates to log handlers (#24153)Migrate Microsoft example DAGs to new design #22452 - azure (#24141)Add typing to Azure Cosmos Client Hook (#23941)
3.9.0¶
Features¶
wasb hook: user defaultAzureCredentials instead of managedIdentity (#23394)
Misc¶
Replace usage of 'DummyOperator' with 'EmptyOperator' (#22974)
3.8.0¶
Features¶
Update secrets backends to use get_conn_value instead of get_conn_uri (#22348)
Misc¶
Docs: Fix example usage for 'AzureCosmosDocumentSensor' (#22735)
3.7.0¶
Features¶
Add 'test_connection' method to 'AzureDataFactoryHook' (#21924)Add pre-commit check for docstring param types (#21398)Make container creation configurable when uploading files via WasbHook (#20510)
Misc¶
Support for Python 3.10(AzureCosmosDBHook) Update to latest Cosmos API (#21514)
3.6.0¶
Features¶
Add optional features in providers. (#21074)
Misc¶
Refactor operator links to not create ad hoc TaskInstances (#21285)
3.5.0¶
Features¶
Azure: New sftp to wasb operator (#18877)Removes InputRequired validation with azure extra (#20084)Add operator link to monitor Azure Data Factory pipeline runs (#20207)
3.4.0¶
Features¶
Remove unnecessary connection form customizations in Azure (#19595)Update Azure modules to comply with AIP-21 (#19431)Remove 'host' from hidden fields in 'WasbHook' (#19475)use DefaultAzureCredential if login not provided for Data Factory (#19079)
Bug Fixes¶
Fix argument error in AzureContainerInstancesOperator (#19668)
3.3.0¶
Features¶
update azure cosmos to latest version (#18695)Added sas_token var to BlobServiceClient return. Updated tests (#19234)Add pre-commit hook for common misspelling check in files (#18964)
Bug Fixes¶
Fix changelog for Azure Provider (#18736)
Other¶
Expanding docs on client auth for AzureKeyVaultBackend (#18659)Static start_date and default arg cleanup for Microsoft providers example DAGs (#19062)
3.2.0¶
Features¶
Rename AzureDataLakeStorage to ADLS (#18493)Creating ADF pipeline run operator, sensor + ADF custom conn fields (#17885)Rename LocalToAzureDataLakeStorageOperator to LocalFilesystemToADLSOperator (#18168)Rename FileToWasbOperator to LocalFilesystemToWasbOperator (#18109)
Bug Fixes¶
Fixed wasb hook attempting to create container when getting a blob client (#18287)Removing redundant relabeling of password conn field (#18386)Proper handling of Account URL custom conn field in AzureBatchHook (#18456)Proper handling of custom conn field values in the AzureDataExplorerHook (#18203)
Main¶
Changes in operators names and import paths are listed in the following table This is a backward compatible change. Deprecated operators will be removed in the next major release.
Deprecated operator name |
New operator name |
Deprecated path |
New path |
AzureDataLakeStorageListOperator |
ADLSListOperator |
airflow.providers.microsoft.azure.operators.adls_list |
airflow.providers.microsoft.azure.operators.adls |
AzureDataLakeStorageDeleteOperator |
ADLSDeleteOperator |
airflow.providers.microsoft.azure.operators.adls_delete |
airflow.providers.microsoft.azure.operators.adls |
3.1.1¶
Misc¶
Optimise connection importing for Airflow 2.2.0Adds secrets backend/logging/auth information to provider yaml (#17625)
3.1.0¶
Features¶
Add support for managed identity in WASB hook (#16628)Reduce log messages for happy path (#16626)
Bug Fixes¶
Fix multiple issues in Microsoft AzureContainerInstancesOperator (#15634)
3.0.0¶
Breaking changes¶
Auto-apply apply_default decorator (#15667)
Warning
Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+.
If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade
Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded
automatically and you will have to manually run airflow upgrade db to complete the migration.
Fixes AzureFileShare connection extras (#16388)
Azure Container Volume and Azure File Share have now dedicated connection types with editable
UI fields. You should not use Wasb connection type any more for those connections. Names of
connection ids for those hooks/operators were changed to reflect that.
Features¶
add oracle connection link (#15632)Add delimiter argument to WasbHook delete_file method (#15637)
Bug Fixes¶
Fix colon spacing in ``AzureDataExplorerHookdocstring (#15841)``fix wasb remote logging when blob already exists (#16280)
2.0.0¶
Breaking changes¶
Removes unnecessary AzureContainerInstance connection type (#15514)
This change removes azure_container_instance_default connection type and replaces it with the
azure_default. The problem was that AzureContainerInstance was not needed as it was exactly the
same as the plain “azure” connection, however it’s presence caused duplication in the field names
used in the UI editor for connections and unnecessary warnings generated. This version uses
plain Azure Hook and connection also for Azure Container Instance. If you already have
azure_container_instance_default connection created in your DB, it will continue to work, but
the first time you edit it with the UI you will have to change it’s type to azure_default.
Features¶
Add dynamic connection fields to Azure Connection (#15159)
Bug fixes¶
Fix 'logging.exception' redundancy (#14823)
1.3.0¶
Features¶
A bunch of template_fields_renderers additions (#15130)
Bug fixes¶
Fix attributes for AzureDataFactory hook (#14704)
1.2.0¶
Features¶
Add Azure Data Factory hook (#11015)
Bug fixes¶
BugFix: Fix remote log in azure storage blob displays in one line (#14313)Fix AzureDataFactoryHook failing to instantiate its connection (#14565)
1.1.0¶
Updated documentation and readme files.
Features¶
Upgrade azure blob to v12 (#12188)Fix Azure Data Explorer Operator (#13520)add AzureDatalakeStorageDeleteOperator (#13206)
1.0.0¶
Initial version of the provider.