Stable (2.10.5)2.10.52.10.42.10.32.10.22.10.12.10.02.9.32.9.22.9.12.9.02.8.42.8.32.8.22.8.12.8.02.7.32.7.22.7.12.7.02.6.32.6.22.6.12.6.02.5.32.5.22.5.12.5.02.4.32.4.22.4.12.4.02.3.42.3.32.3.22.3.12.3.02.2.52.2.42.2.32.2.22.2.12.2.02.1.42.1.32.1.22.1.12.1.02.0.32.0.22.0.12.0.01.10.151.10.141.10.131.10.121.10.111.10.101.10.91.10.81.10.71.10.61.10.51.10.41.10.31.10.21.10.1
Stable (2.10.5)2.10.52.10.42.10.32.10.22.10.12.10.02.9.32.9.22.9.12.9.02.8.42.8.32.8.22.8.12.8.02.7.32.7.22.7.12.7.02.6.32.6.22.6.12.6.02.5.32.5.22.5.12.5.02.4.32.4.22.4.12.4.02.3.42.3.32.3.22.3.12.3.02.2.52.2.42.2.32.2.22.2.12.2.02.1.42.1.32.1.22.1.12.1.02.0.32.0.22.0.12.0.01.10.151.10.141.10.131.10.121.10.111.10.101.10.91.10.81.10.71.10.61.10.51.10.41.10.31.10.21.10.1

Changelog

Airflow 2.2.0, 2021-10-11

New Features

  • AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030, #16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434, #18421, #18475, #18499, #18573, #18522, #18729, #18706, #18742, #18786, #18804)

  • AIP-40: Add Deferrable "Async" Tasks (#15389, #17564, #17565, #17601, #17745, #17747, #17748, #17875, #17876, #18129, #18210, #18214, #18552, #18728, #18414)

  • Add a Docker Taskflow decorator (#15330, #18739)

  • Add Airflow Standalone command (#15826)

  • Display alert messages on dashboard from local settings (#18284)

  • Advanced Params using json-schema (#17100)

  • Ability to test connections from UI or API (#15795, #18750)

  • Add Next Run to UI (#17732)

  • Add default weight rule configuration option (#18627)

  • Add a calendar field to choose the execution date of the DAG when triggering it (#16141)

  • Allow setting specific cwd for BashOperator (#17751)

  • Show import errors in DAG views (#17818)

  • Add pre/post execution hooks [Experimental] (#17576)

  • Added table to view providers in Airflow ui under admin tab (#15385)

  • Adds secrets backend/logging/auth information to provider yaml (#17625)

  • Add date format filters to Jinja environment (#17451)

  • Introduce RESTARTING state (#16681)

  • Webserver: Unpause DAG on manual trigger (#16569)

  • API endpoint to create new user (#16609)

  • Add insert_args for support transfer replace (#15825)

  • Add recursive flag to glob in filesystem sensor (#16894)

  • Add conn to jinja template context (#16686)

  • Add default_args for TaskGroup (#16557)

  • Allow adding duplicate connections from UI (#15574)

  • Allow specifying multiple URLs via the CORS config option (#17941)

  • Implement API endpoint for DAG deletion (#17980)

  • Add DAG run endpoint for marking a dagrun success or failed(#17839)

  • Add support for kinit options [-f|-F] and [-a|-A] (#17816)

  • Queue support for DaskExecutor using Dask Worker Resources (#16829, #18720)

  • Make auto refresh interval configurable (#18107)

Improvements

  • Small improvements for Airflow UI (#18715, #18795)

  • Rename processor_poll_interval to scheduler_idle_sleep_time (#18704)

  • Check the allowed values for the logging level (#18651)

  • Fix error on triggering a dag that doesn't exist using dagrun_conf (#18655)

  • Add muldelete action to TaskInstanceModelView (#18438)

  • Avoid importing DAGs during clean DB installation (#18450)

  • Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634)

  • Make Kubernetes job description fit on one log line (#18377)

  • Always draw borders if task instance state is null or undefined (#18033)

  • Inclusive Language (#18349)

  • Improved log handling for zombie tasks (#18277)

  • Adding Variable.update method and improving detection of variable key collisions (#18159)

  • Add note about params on trigger DAG page (#18166)

  • Change TaskInstance and TaskReschedule PK from execution_date to run_id (#17719)

  • Adding TaskGroup support in BaseOperator.chain() (#17456)

  • Advises the kernel to not cache log files generated by Airflow (#18054)

  • Allow filtering DAGS by tags in the REST API (#18090)

  • Optimize imports of Providers Manager (#18052)

  • Adds capability of Warnings for incompatible community providers (#18020)

  • Serialize the template_ext attribute to show it in UI (#17985)

  • Add robots.txt and X-Robots-Tag header (#17946)

  • Refactor BranchDayOfWeekOperator, DayOfWeekSensor (#17940)

  • Update error message to guide the user into self-help mostly (#17929)

  • Update to Celery 5 (#17397)

  • Avoid redirect loop for users with no permissions (#17838)

  • Improve graph view load time for dags with open groups (#17821)

  • Add links to provider's documentation (#17736)

  • Remove Marshmallow schema warnings (#17753)

  • Rename none_failed_or_skipped by none_failed_min_one_success trigger rule (#17683)

  • Remove [core] store_dag_code & use DB to get Dag Code (#16342)

  • Rename task_concurrency to max_active_tis_per_dag (#17708)

  • Import Hooks lazily individually in providers manager (#17682)

  • Adding support for multiple task-ids in the external task sensor (#17339)

  • Replace execution_date with run_id in airflow tasks run command (#16666)

  • Avoid endless redirect loop when user has no roles (#17613)

  • Make output from users cli command more consistent (#17642)

  • Open relative extra links in place (#17477)

  • Move worker_log_server_port option to the logging section (#17621)

  • Use gunicorn to serve logs generated by worker (#17591)

  • Improve validation of Group id (#17578)

  • Simplify 404 page (#17501)

  • Add XCom.clear so it's hookable in custom XCom backend (#17405)

  • Add deprecation notice for SubDagOperator (#17488)

  • Support DAGS folder being in different location on scheduler and runners (#16860)

  • Remove /dagrun/create and disable edit form generated by F.A.B (#17376)

  • Enable specifying dictionary paths in template_fields_renderers (#17321)

  • error early if virtualenv is missing (#15788)

  • Handle connection parameters added to Extra and custom fields (#17269)

  • Fix airflow celery stop to accept the pid file. (#17278)

  • Remove DAG refresh buttons (#17263)

  • Deprecate dummy trigger rule in favor of always (#17144)

  • Be verbose about failure to import airflow_local_settings (#17195)

  • Include exit code in AirflowException str when BashOperator fails. (#17151)

  • Adding EdgeModifier support for chain() (#17099)

  • Only allows supported field types to be used in custom connections (#17194)

  • Secrets backend failover (#16404)

  • Warn on Webserver when using SQLite or SequentialExecutor (#17133)

  • Extend init_containers defined in pod_override (#17537)

  • Client-side filter dag dependencies (#16253)

  • Improve executor validation in CLI (#17071)

  • Prevent running airflow db init/upgrade migrations and setup in parallel. (#17078)

  • Update chain() and cross_downstream() to support XComArgs (#16732)

  • Improve graph view refresh (#16696)

  • When a task instance fails with exception, log it (#16805)

  • Set process title for serve-logs and LocalExecutor (#16644)

  • Rename test_cycle to check_cycle (#16617)

  • Set process title for Worker when using LocalExecutor (#16623)

  • Add schema as DbApiHook instance attribute (#16521, #17423)

  • Improve compatibility with MSSQL (#9973)

  • Add transparency for unsupported connection type (#16220)

  • Call resource based fab methods (#16190)

  • Format more dates with timezone (#16129)

  • Replace deprecated dag.sub_dag with dag.partial_subset (#16179)

  • Treat AirflowSensorTimeout as immediate failure without retrying (#12058)

  • Marking success/failed automatically clears failed downstream tasks (#13037)

  • Add close/open indicator for import dag errors (#16073)

  • Add collapsible import errors (#16072)

  • Always return a response in TI's action_clear view (#15980)

  • Add cli command to delete user by email (#15873)

  • Use resource and action names for FAB permissions (#16410)

  • Rename DAG concurrency ([core] dag_concurrency) settings for easier understanding (#16267, #18730)

  • Calendar UI improvements (#16226)

  • Refactor: SKIPPED should not be logged again as SUCCESS (#14822)

  • Remove version limits for dnspython (#18046, #18162)

  • Accept custom run ID in TriggerDagRunOperator (#18788)

Bug Fixes

  • Make REST API patch user endpoint work the same way as the UI (#18757)

  • Properly set start_date for cleared tasks (#18708)

  • Ensure task_instance exists before running update on its state(REST API) (#18642)

  • Make AirflowDateTimePickerWidget a required field (#18602)

  • Retry deadlocked transactions on deleting old rendered task fields (#18616)

  • Fix retry_exponential_backoff divide by zero error when retry delay is zero (#17003)

  • Improve how UI handles datetimes (#18611, #18700)

  • Bugfix: dag_bag.get_dag should return None, not raise exception (#18554)

  • Only show the task modal if it is a valid instance (#18570)

  • Fix accessing rendered {{ task.x }} attributes from within templates (#18516)

  • Add missing email type of connection (#18502)

  • Don't use flash for "same-page" UI messages. (#18462)

  • Fix task group tooltip (#18406)

  • Properly fix dagrun update state endpoint (#18370)

  • Properly handle ti state difference between executor and scheduler (#17819)

  • Fix stuck "queued" tasks in KubernetesExecutor (#18152)

  • Don't permanently add zip DAGs to sys.path (#18384)

  • Fix random deadlocks in MSSQL database (#18362)

  • Deactivating DAGs which have been removed from files (#17121)

  • When syncing dags to db remove dag_tag rows that are now unused (#8231)

  • Graceful scheduler shutdown on error (#18092)

  • Fix mini scheduler not respecting wait_for_downstream dep (#18338)

  • Pass exception to run_finished_callback for Debug Executor (#17983)

  • Make XCom.get_one return full, not abbreviated values (#18274)

  • Use try/except when closing temporary file in task_runner (#18269)

  • show next run if not none (#18273)

  • Fix DB session handling in XCom.set (#18240)

  • Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)

  • Fix external_executor_id not being set for manually run jobs (#17207)

  • Fix deleting of zipped Dags in Serialized Dag Table (#18243)

  • Return explicit error on user-add for duplicated email (#18224)

  • Remove loading dots even when last run data is empty (#18230)

  • Swap dag import error dropdown icons (#18207)

  • Automatically create section when migrating config (#16814)

  • Set encoding to utf-8 by default while reading task logs (#17965)

  • Set task state to failed when pod is DELETED while running (#18095)

  • Apply parent dag permissions to subdags (#18160)

  • Change id collation for MySQL to case-sensitive (#18072)

  • Logs task launch exception in StandardTaskRunner (#17967)

  • Applied permissions to self._error_file (#15947)

  • Hide variable import form if user lacks permission (#18000)

  • Fix blank dag dependencies view (#17990)

  • Add missing menu access for dag dependencies and configurations pages (#17450)

  • Fix passing Jinja templates in DateTimeSensor (#17959)

  • Fixing bug which restricted the visibility of ImportErrors (#17924)

  • Fix grammar in traceback.html (#17942)

  • Fix DagRunState enum query for MySQLdb driver (#17886)

  • Fixed button size in "Actions" group. (#17902)

  • Fix log links on graph TI modal (#17862)

  • Only show import errors for DAGs a user can access (#17835)

  • Show all import_errors from zip files (#17759)

  • fix EXTRA_LOGGER_NAMES param and related docs (#17808)

  • Use one interpreter for Airflow and gunicorn (#17805)

  • Fix: Mysql 5.7 id utf8mb3 (#14535)

  • Fix dag_processing.last_duration metric random holes (#17769)

  • Automatically use utf8mb3_general_ci collation for MySQL (#17729)

  • Fix wrong query on running tis (#17631)

  • Add root to tree refresh url (#17633)

  • Do not delete running DAG from the UI (#17630)

  • Rescue if a DagRun's DAG was removed from db (#17544)

  • fix: filter condition of TaskInstance does not work #17535 (#17548)

  • Regression on pid reset to allow task start after heartbeat (#17333)

  • Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248)

  • Remove locks for upgrades in MSSQL (#17213)

  • Create virtualenv via python call (#17156)

  • Ensure a DAG is acyclic when running DAG.cli() (#17105)

  • Translate non-ascii characters (#17057)

  • Change the logic of None comparison in model_list template (#16893)

  • Have UI and POST /task_instances_state API endpoint have same behaviour (#16539)

  • ensure task is skipped if missing sla (#16719)

  • Fix direct use of cached_property module (#16710)

  • Fix TI success confirm page (#16650)

  • Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)

  • Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)

  • Modify return value check in python virtualenv jinja template (#16049)

  • Fix dag dependency search (#15924)

  • Make custom JSON encoder support Decimal (#16383)

  • Bugfix: Allow clearing tasks with just dag_id and empty subdir (#16513)

  • Convert port value to a number before calling test connection (#16497)

  • Handle missing/null serialized DAG dependencies (#16393)

  • Correctly set dag.fileloc when using the @dag decorator (#16384)

  • Fix TI success/failure links (#16233)

  • Correctly implement autocomplete early return in airflow/www/views.py (#15940)

  • Backport fix to allow pickling of Loggers to Python 3.6 (#18798)

  • Fix bug that Backfill job fail to run when there are tasks run into reschedule state (#17305, #18806)

Doc only changes

  • Update dagbag_size documentation (#18824)

  • Update documentation about bundle extras (#18828)

  • Fix wrong Postgres search_path set up instructions (#17600)

  • Remove AIRFLOW_GID from Docker images (#18747)

  • Improve error message for BranchPythonOperator when no task_id to follow (#18471)

  • Improve guidance to users telling them what to do on import timeout (#18478)

  • Explain scheduler fine-tuning better (#18356)

  • Added example JSON for airflow pools import (#18376)

  • Add sla_miss_callback section to the documentation (#18305)

  • Explain sentry default environment variable for subprocess hook (#18346)

  • Refactor installation pages (#18282)

  • Improves installing from sources pages for all components (#18251)

  • Doc: Use closer.lua script for downloading sources (#18179)

  • Improves quick-start docker-compose warnings and documentation (#18164)

  • Production-level support for MSSQL (#18382)

  • Update non-working example in documentation (#18067)

  • Improve the description of how to handle dynamic task generation (#17963)

  • Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866)

  • Update max_tis_per_query to better render on the webpage (#17971)

  • Adds Github Oauth example with team based authorization (#17896)

  • Update docker.rst (#17882)

  • Example xcom update (#17749)

  • Add doc warning about connections added via envvars (#17915)

  • fix wrong documents around upgrade-check.rst (#17903)

  • Add Brent to Committers list (#17873)

  • Improves documentation about modules management (#17757)

  • Remove deprecated metrics from metrics.rst (#17772)

  • Make sure "production-readiness" of docker-compose is well explained (#17731)

  • Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710)

  • Doc: Replace deprecated param from docstrings (#17709)

  • docs(dagowner): describe dag owner more carefully (#17699)

  • docs(impersonation): update note so avoid misintrepretation (#17701)

  • Docs: Make DAG.is_active read-only in API (#17667)

  • Improve cross-links to operators and hooks references (#17622)

  • Update documentation regarding Python 3.9 support (#17611)

  • Fix MySQL database character set instruction (#17603)

  • Document overriding XCom.clear for data lifecycle management (#17589)

  • Path correction in docs for airflow core (#17567)

  • docs(celery): reworded, add actual multiple queues example (#17541)

  • Doc: Add FAQ to speed up parsing with tons of dag files (#17519)

  • Improve image building documentation for new users (#17409)

  • Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296)

  • Update warning about MariaDB and multiple schedulers (#17287)

  • Doc: Recommend using same configs on all Airflow components (#17146)

  • Move docs about masking to a new page (#17007)

  • Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926)

  • Docs: Better description for pod_template_file (#16861)

  • Add Aneesh Joseph as Airflow Committer (#16835)

  • Docs: Added new pipeline example for the tutorial docs (#16548)

  • Remove upstart from docs (#16672)

  • Add new committers: Jed and TP (#16671)

  • Docs: Fix flask-ouathlib to flask-oauthlib in Upgrading docs (#16320)

  • Docs: Fix creating a connection docs (#16312)

  • Docs: Fix url for Elasticsearch (#16275)

  • Small improvements for README.md files (#16244)

  • Fix docs for dag_concurrency (#16177)

  • Check syntatic correctness for code-snippets (#16005)

  • Improvements for Docker Image docs (#14843)

  • Add proper link for wheel packages in docs. (#15999)

  • Add Docs for default_pool slots (#15997)

  • Add memory usage warning in quick-start documentation (#15967)

  • Update example KubernetesExecutor git-sync pod template file (#15904)

  • Docs: Fix Taskflow API docs (#16574)

  • Added new pipeline example for the tutorial docs (#16084)

  • Updating the DAG docstring to include render_template_as_native_obj (#16534)

  • Update docs on setting up SMTP (#16523)

  • Docs: Fix API verb from POST to PATCH (#16511)

Misc/Internal

  • Renaming variables to be consistent with code logic (#18685)

  • Simplify strings previously split across lines (#18679)

  • fix exception string of BranchPythonOperator (#18623)

  • Add multiple roles when creating users (#18617)

  • Move FABs base Security Manager into Airflow. (#16647)

  • Remove unnecessary css state colors (#18461)

  • Update boto3 to <1.19 (#18389)

  • Improve coverage for airflow.security.kerberos module (#18258)

  • Fix kinesis test (#18337)

  • Fix provider test acessing importlib-resources (#18228)

  • Silence warnings in tests from using SubDagOperator (#18275)

  • Fix usage of range(len()) to enumerate (#18174)

  • Test coverage on the autocomplete view (#15943)

  • Update version added fields in airflow/config_templates/config.yml (#18128)

  • Add "packaging" to core requirements (#18122)

  • Adds LoggingMixins to BaseTrigger (#18106)

  • Limit colorlog version (6.x is incompatible) (#18099)

  • Fix building docs in main builds (#18035)

  • Remove upper-limit on tenacity (#17593)

  • Remove redundant numpy dependency (#17594)

  • Bump mysql-connector-python to latest version (#17596)

  • Make pandas an optional core dependency (#17575)

  • Add more typing to airflow.utils.helpers (#15582)

  • Chore: Some code cleanup in airflow/utils/db.py (#17090)

  • Refactor: Remove processor_factory from DAG processing (#16659)

  • Remove AbstractDagFileProcessorProcess from dag processing (#16816)

  • Update TaskGroup typing (#16811)

  • Update click to 8.x (#16779)

  • Remove remaining Pylint disables (#16760)

  • Remove duplicated try, there is already a try in create_session (#16701)

  • Removes pylint from our toolchain (#16682)

  • Refactor usage of unneeded function call (#16653)

  • Add type annotations to setup.py (#16658)

  • Remove SQLAlchemy <1.4 constraint (#16630) (Note: our dependencies still have a requirement on <1.4)

  • Refactor dag.clear method (#16086)

  • Use DAG_ACTIONS constant (#16232)

  • Use updated _get_all_non_dag_permissions method (#16317)

  • Add updated-name wrappers for built-in FAB methods (#16077)

  • Pins docutils to <0.17 until breaking behaviour is fixed (#16133)

  • Remove TaskInstance.log_filepath attribute (#15217)

  • Removes unnecessary function call in airflow/www/app.py (#15956)

  • Move plyvel to google provider extra (#15812)

  • Update permission migrations to use new naming scheme (#16400)

  • Use resource and action names for FAB (#16380)

  • Swap out calls to find_permission_view_menu for get_permission wrapper (#16377)

  • Fix deprecated default for fab_logging_level to WARNING (#18783)

  • Allow running tasks from UI when using CeleryKubernetesExecutor (#18441)

Airflow 2.1.4, 2021-09-18

Bug Fixes

  • Fix deprecation error message rather than silencing it (#18126)

  • Limit the number of queued dagruns created by the Scheduler (#18065)

  • Fix DagRun execution order from queued to running not being properly followed (#18061)

  • Fix max_active_runs not allowing moving of queued dagruns to running (#17945)

  • Avoid redirect loop for users with no permissions (#17838)

  • Avoid endless redirect loop when user has no roles (#17613)

  • Fix log links on graph TI modal (#17862)

  • Hide variable import form if user lacks permission (#18000)

  • Improve dag/task concurrency check (#17786)

  • Fix Clear task instances endpoint resets all DAG runs bug (#17961)

  • Fixes incorrect parameter passed to views (#18083) (#18085)

  • Fix Sentry handler from LocalTaskJob causing error (#18119)

  • Limit colorlog version (6.x is incompatible) (#18099)

  • Only show Pause/Unpause tooltip on hover (#17957)

  • Improve graph view load time for dags with open groups (#17821)

  • Increase width for Run column (#17817)

  • Fix wrong query on running tis (#17631)

  • Add root to tree refresh url (#17633)

  • Do not delete running DAG from the UI (#17630)

  • Improve discoverability of Provider packages' functionality

  • Do not let create_dagrun overwrite explicit run_id (#17728)

  • BugFix: Regression on pid reset to allow task start after heartbeat (#17333)

  • Set task state to failed when pod is DELETED while running (#18095)

  • Advises the kernel to not cache log files generated by Airflow (#18054)

  • Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)

Doc only changes

  • Update version added fields in airflow/config_templates/config.yml (#18128)

  • Improve the description of how to handle dynamic task generation (#17963)

  • Improve cross-links to operators and hooks references (#17622)

  • Doc: Fix replacing Airflow version for Docker stack (#17711)

  • Make the providers operators/hooks reference much more usable (#17768)

  • Update description about the new connection-types provider meta-data

  • Suggest to use secrets backend for variable when it contains sensitive data (#17319)

  • Improves quick-start docker-compose warnings and documentation (#18164)

  • Separate Installing from sources section and add more details (#18171)

  • Doc: Use closer.lua script for downloading sources (#18179)

  • Doc: Improve installing from sources (#18194)

  • Improves installing from sources pages for all components (#18251)

Airflow 2.1.3, 2021-08-23

Bug Fixes

  • Fix task retries when they receive sigkill and have retries and properly handle sigterm (#16301)

  • Fix redacting secrets in context exceptions. (#17618)

  • Fix race condition with dagrun callbacks (#16741)

  • Add 'queued' to DagRunState (#16854)

  • Add 'queued' state to DagRun (#16401)

  • Fix external elasticsearch logs link (#16357)

  • Add proper warning message when recorded PID is different from current PID (#17411)

  • Fix running tasks with default_impersonation config (#17229)

  • Rescue if a DagRun's DAG was removed from db (#17544)

  • Fixed broken json_client (#17529)

  • Handle and log exceptions raised during task callback (#17347)

  • Fix CLI kubernetes cleanup-pods which fails on invalid label key (#17298)

  • Show serialization exceptions in DAG parsing log (#17277)

  • Fix: TaskInstance does not show queued_by_job_id & external_executor_id (#17179)

  • Adds more explanatory message when SecretsMasker is not configured (#17101)

  • Enable the use of __init_subclass__ in subclasses of BaseOperator (#17027)

  • Fix task instance retrieval in XCom view (#16923)

  • Validate type of priority_weight during parsing (#16765)

  • Correctly handle custom deps` and ``task_group during DAG Serialization (#16734)

  • Fix slow (cleared) tasks being be adopted by Celery worker. (#16718)

  • Fix calculating duration in tree view (#16695)

  • Fix AttributeError: datetime.timezone object has no attribute name (#16599)

  • Redact conn secrets in webserver logs (#16579)

  • Change graph focus to top of view instead of center (#16484)

  • Fail tasks in scheduler when executor reports they failed (#15929)

  • fix(smart_sensor): Unbound variable errors (#14774)

  • Add back missing permissions to UserModelView controls. (#17431)

  • Better diagnostics and self-healing of docker-compose (#17484)

  • Improve diagnostics message when users have secret_key mis-configured (#17410)

  • Stop checking execution_date in task_instance.refresh_from_db (#16809)

Improvements

  • Run mini scheduler in LocalTaskJob during task exit (#16289)

  • Remove SQLAlchemy<1.4 constraint (#16630)

  • Bump Jinja2 upper-bound from 2.12.0 to 4.0.0 (#16595)

  • Bump dnspython (#16698)

  • Updates to FlaskAppBuilder 3.3.2+ (#17208)

  • Add State types for tasks and DAGs (#15285)

  • Set Process title for Worker when using LocalExecutor (#16623)

  • Move DagFileProcessor and DagFileProcessorProcess out of scheduler_job.py (#16581)

Doc only changes

  • Fix inconsistencies in configuration docs (#17317)

  • Fix docs link for using SQLite as Metadata DB (#17308)

Misc

  • Switch back http provider after requests removes LGPL dependency (#16974)

Airflow 2.1.2, 2021-07-14

Bug Fixes

  • Only allow webserver to request from the worker log server (#16754)

  • Fix "Invalid JSON configuration, must be a dict" bug (#16648)

  • Fix CeleryKubernetesExecutor (#16700)

  • Mask value if the key is token (#16474)

  • Fix impersonation issue with LocalTaskJob (#16852)

  • Resolve all npm vulnerabilities including bumping jQuery to 3.5 (#16440)

Misc

Airflow 2.1.1, 2021-07-02

Bug Fixes

  • Don't crash attempting to mask secrets in dict with non-string keys (#16601)

  • Always install sphinx_airflow_theme from PyPI (#16594)

  • Remove limitation for elasticsearch library (#16553)

  • Adding extra requirements for build and runtime of the PROD image. (#16170)

  • Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)

  • Removes unnecessary packages from setup_requires (#16139)

  • Pins docutils to <0.17 until breaking behaviour is fixed (#16133)

  • Improvements for Docker Image docs (#14843)

  • Ensure that dag_run.conf is a dict (#15057)

  • Fix CLI connections import and migrate logic from secrets to Connection model (#15425)

  • Fix Dag Details start date bug (#16206)

  • Fix DAG run state not updated while DAG is paused (#16343)

  • Allow null value for operator field in task_instance schema(REST API) (#16516)

  • Avoid recursion going too deep when redacting logs (#16491)

  • Backfill: Don't create a DagRun if no tasks match task regex (#16461)

  • Tree View UI for larger DAGs & more consistent spacing in Tree View (#16522)

  • Correctly handle None returns from Query.scalar() (#16345)

  • Adding only_active parameter to /dags endpoint (#14306)

  • Don't show stale Serialized DAGs if they are deleted in DB (#16368)

  • Make REST API List DAGs endpoint consistent with UI/CLI behaviour (#16318)

  • Support remote logging in elasticsearch with filebeat 7 (#14625)

  • Queue tasks with higher priority and earlier execution_date first. (#15210)

  • Make task ID on legend have enough width and width of line chart to be 100%. (#15915)

  • Fix normalize-url vulnerability (#16375)

  • Validate retries value on init for better errors (#16415)

  • add num_runs query param for tree refresh (#16437)

  • Fix templated default/example values in config ref docs (#16442)

  • Add passphrase and private_key to default sensitive field names (#16392)

  • Fix tasks in an infinite slots pool were never scheduled (#15247)

  • Fix Orphaned tasks stuck in CeleryExecutor as running (#16550)

  • Don't fail to log if we can't redact something (#16118)

  • Set max tree width to 1200 pixels (#16067)

  • Fill the "job_id" field for airflow task run without --local/--raw for KubeExecutor (#16108)

  • Fixes problem where conf variable was used before initialization (#16088)

  • Fix apply defaults for task decorator (#16085)

  • Parse recently modified files even if just parsed (#16075)

  • Ensure that we don't try to mask empty string in logs (#16057)

  • Don't die when masking log.exception when there is no exception (#16047)

  • Restores apply_defaults import in base_sensor_operator (#16040)

  • Fix auto-refresh in tree view When webserver ui is not in / (#16018)

  • Fix dag.clear() to set multiple dags to running when necessary (#15382)

  • Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)

Airflow 2.1.0, 2021-05-21

New Features

  • Add PythonVirtualenvDecorator to Taskflow API (#14761)

  • Add Taskgroup decorator (#15034)

  • Create a DAG Calendar View (#15423)

  • Create cross-DAG dependencies view (#13199)

  • Add rest API to query for providers (#13394)

  • Mask passwords and sensitive info in task logs and UI (#15599)

  • Add SubprocessHook for running commands from operators (#13423)

  • Add DAG Timeout in UI page "DAG Details" (#14165)

  • Add WeekDayBranchOperator (#13997)

  • Add JSON linter to DAG Trigger UI (#13551)

  • Add DAG Description Doc to Trigger UI Page (#13365)

  • Add airflow webserver URL into SLA miss email. (#13249)

  • Add read only REST API endpoints for users (#14735)

  • Add files to generate Airflow's Python SDK (#14739)

  • Add dynamic fields to snowflake connection (#14724)

  • Add read only REST API endpoint for roles and permissions (#14664)

  • Add new datetime branch operator (#11964)

  • Add Google leveldb hook and operator (#13109) (#14105)

  • Add plugins endpoint to the REST API (#14280)

  • Add worker_pod_pending_timeout support (#15263)

  • Add support for labeling DAG edges (#15142)

  • Add CUD REST API endpoints for Roles (#14840)

  • Import connections from a file (#15177)

  • A bunch of template_fields_renderers additions (#15130)

  • Add REST API query sort and order to some endpoints (#14895)

  • Add timezone context in new ui (#15096)

  • Add query mutations to new UI (#15068)

  • Add different modes to sort dag files for parsing (#15046)

  • Auto refresh on Tree View (#15474)

  • BashOperator to raise AirflowSkipException on exit code 99 (by default, configurable) (#13421) (#14963)

  • Clear tasks by task ids in REST API (#14500)

  • Support jinja2 native Python types (#14603)

  • Allow celery workers without gossip or mingle modes (#13880)

  • Add airflow jobs check CLI command to check health of jobs (Scheduler etc) (#14519)

  • Rename DateTimeBranchOperator to BranchDateTimeOperator (#14720)

Improvements

  • Add optional result handler callback to DbApiHook (#15581)

  • Update Flask App Builder limit to recently released 3.3 (#15792)

  • Prevent creating flask sessions on REST API requests (#15295)

  • Sync DAG specific permissions when parsing (#15311)

  • Increase maximum length of pool name on Tasks to 256 characters (#15203)

  • Enforce READ COMMITTED isolation when using mysql (#15714)

  • Auto-apply apply_default to subclasses of BaseOperator (#15667)

  • Emit error on duplicated DAG ID (#15302)

  • Update KubernetesExecutor pod templates to allow access to IAM permissions (#15669)

  • More verbose logs when running airflow db check-migrations (#15662)

  • When one_success mark task as failed if no success (#15467)

  • Add an option to trigger a dag w/o changing conf (#15591)

  • Add Airflow UI instance_name configuration option (#10162)

  • Add a decorator to retry functions with DB transactions (#14109)

  • Add return to PythonVirtualenvOperator's execute method (#14061)

  • Add verify_ssl config for kubernetes (#13516)

  • Add description about secret_key when Webserver > 1 (#15546)

  • Add Traceback in LogRecord in JSONFormatter (#15414)

  • Add support for arbitrary json in conn uri format (#15100)

  • Adds description field in variable (#12413) (#15194)

  • Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134)

  • Execute on_failure_callback when SIGTERM is received (#15172)

  • Allow hiding of all edges when highlighting states (#15281)

  • Display explicit error in case UID has no actual username (#15212)

  • Serve logs with Scheduler when using Local or Sequential Executor (#15557)

  • Deactivate trigger, refresh, and delete controls on dag detail view. (#14144)

  • Turn off autocomplete for connection forms (#15073)

  • Increase default worker_refresh_interval to 6000 seconds (#14970)

  • Only show User's local timezone if it's not UTC (#13904)

  • Suppress LOG/WARNING for a few tasks CLI for better CLI experience (#14567)

  • Configurable API response (CORS) headers (#13620)

  • Allow viewers to see all docs links (#14197)

  • Update Tree View date ticks (#14141)

  • Make the tooltip to Pause / Unpause a DAG clearer (#13642)

  • Warn about precedence of env var when getting variables (#13501)

  • Move [celery] default_queue config to [operators] default_queue to re-use between executors (#14699)

Bug Fixes

  • Fix 500 error from updateTaskInstancesState API endpoint when dry_run not passed (#15889)

  • Ensure that task preceding a PythonVirtualenvOperator doesn't fail (#15822)

  • Prevent mixed case env vars from crashing processes like worker (#14380)

  • Fixed type annotations in DAG decorator (#15778)

  • Fix on_failure_callback when task receive SIGKILL (#15537)

  • Fix dags table overflow (#15660)

  • Fix changing the parent dag state on subdag clear (#15562)

  • Fix reading from zip package to default to text (#13962)

  • Fix wrong parameter for drawDagStatsForDag in dags.html (#13884)

  • Fix QueuedLocalWorker crashing with EOFError (#13215)

  • Fix typo in NotPreviouslySkippedDep (#13933)

  • Fix parallelism after KubeExecutor pod adoption (#15555)

  • Fix kube client on mac with keepalive enabled (#15551)

  • Fixes wrong limit for dask for python>3.7 (should be <3.7) (#15545)

  • Fix Task Adoption in KubernetesExecutor (#14795)

  • Fix timeout when using XCom with KubernetesPodOperator (#15388)

  • Fix deprecated provider aliases in "extras" not working (#15465)

  • Fixed default XCom deserialization. (#14827)

  • Fix used_group_ids in dag.partial_subset (#13700) (#15308)

  • Further fix trimmed pod_id for KubernetesPodOperator (#15445)

  • Bugfix: Invalid name when trimmed pod_id ends with hyphen in KubernetesPodOperator (#15443)

  • Fix incorrect slots stats when TI pool_slots > 1 (#15426)

  • Fix DAG last run link (#15327)

  • Fix sync-perm to work correctly when update_fab_perms = False (#14847)

  • Fixes limits on Arrow for plexus test (#14781)

  • Fix UI bugs in tree view (#14566)

  • Fix AzureDataFactoryHook failing to instantiate its connection (#14565)

  • Fix permission error on non-POSIX filesystem (#13121)

  • Fix spelling in "ignorable" (#14348)

  • Fix get_context_data doctest import (#14288)

  • Correct typo in GCSObjectsWtihPrefixExistenceSensor (#14179)

  • Fix order of failed deps (#14036)

  • Fix critical CeleryKubernetesExecutor bug (#13247)

  • Fix four bugs in StackdriverTaskHandler (#13784)

  • func.sum may return Decimal that break rest APIs (#15585)

  • Persist tags params in pagination (#15411)

  • API: Raise AlreadyExists exception when the execution_date is same (#15174)

  • Remove duplicate call to sync_metadata inside DagFileProcessorManager (#15121)

  • Extra docker-py update to resolve docker op issues (#15731)

  • Ensure executors end method is called (#14085)

  • Remove user_id from API schema (#15117)

  • Prevent clickable bad links on disabled pagination (#15074)

  • Acquire lock on db for the time of migration (#10151)

  • Skip SLA check only if SLA is None (#14064)

  • Print right version in airflow info command (#14560)

  • Make airflow info work with pipes (#14528)

  • Rework client-side script for connection form. (#14052)

  • API: Add CollectionInfo in all Collections that have total_entries (#14366)

  • Fix task_instance_mutation_hook when importing airflow.models.dagrun (#15851)

Doc only changes

  • Fix docstring of SqlSensor (#15466)

  • Small changes on "DAGs and Tasks documentation" (#14853)

  • Add note on changes to configuration options (#15696)

  • Add docs to the markdownlint and yamllint config files (#15682)

  • Rename old "Experimental" API to deprecated in the docs. (#15653)

  • Fix documentation error in git_sync_template.yaml (#13197)

  • Fix doc link permission name (#14972)

  • Fix link to Helm chart docs (#14652)

  • Fix docstrings for Kubernetes code (#14605)

  • docs: Capitalize & minor fixes (#14283) (#14534)

  • Fixed reading from zip package to default to text. (#13984)

  • An initial rework of the "Concepts" docs (#15444)

  • Improve docstrings for various modules (#15047)

  • Add documentation on database connection URI (#14124)

  • Add Helm Chart logo to docs index (#14762)

  • Create a new documentation package for Helm Chart (#14643)

  • Add docs about supported logging levels (#14507)

  • Update docs about tableau and salesforce provider (#14495)

  • Replace deprecated doc links to the correct one (#14429)

  • Refactor redundant doc url logic to use utility (#14080)

  • docs: NOTICE: Updated 2016-2019 to 2016-now (#14248)

  • Skip DAG perm sync during parsing if possible (#15464)

  • Add picture and examples for Edge Labels (#15310)

  • Add example DAG & how-to guide for sqlite (#13196)

  • Add links to new modules for deprecated modules (#15316)

  • Add note in Updating.md about FAB data model change (#14478)

Misc/Internal

  • Fix logging.exception redundancy (#14823)

  • Bump stylelint to remove vulnerable sub-dependency (#15784)

  • Add resolution to force dependencies to use patched version of lodash (#15777)

  • Update croniter to 1.0.x series (#15769)

  • Get rid of Airflow 1.10 in Breeze (#15712)

  • Run helm chart tests in parallel (#15706)

  • Bump ssri from 6.0.1 to 6.0.2 in /airflow/www (#15437)

  • Remove the limit on Gunicorn dependency (#15611)

  • Better "dependency already registered" warning message for tasks #14613 (#14860)

  • Pin pandas-gbq to <0.15.0 (#15114)

  • Use Pip 21.* to install airflow officially (#15513)

  • Bump mysqlclient to support the 1.4.x and 2.x series (#14978)

  • Finish refactor of DAG resource name helper (#15511)

  • Refactor/Cleanup Presentation of Graph Task and Path Highlighting (#15257)

  • Standardize default fab perms (#14946)

  • Remove datepicker for task instance detail view (#15284)

  • Turn provider's import warnings into debug logs (#14903)

  • Remove left-over fields from required in provider_info schema. (#14119)

  • Deprecate tableau extra (#13595)

  • Use built-in cached_property on Python 3.8 where possible (#14606)

  • Clean-up JS code in UI templates (#14019)

  • Bump elliptic from 6.5.3 to 6.5.4 in /airflow/www (#14668)

  • Switch to f-strings using flynt. (#13732)

  • use jquery ready instead of vanilla js (#15258)

  • Migrate task instance log (ti_log) js (#15309)

  • Migrate graph js (#15307)

  • Migrate dags.html javascript (#14692)

  • Removes unnecessary AzureContainerInstance connection type (#15514)

  • Separate Kubernetes pod_launcher from core airflow (#15165)

  • update remaining old import paths of operators (#15127)

  • Remove broken and undocumented "demo mode" feature (#14601)

  • Simplify configuration/legibility of Webpack entries (#14551)

  • remove inline tree js (#14552)

  • Js linting and inline migration for simple scripts (#14215)

  • Remove use of repeated constant in AirflowConfigParser (#14023)

  • Deprecate email credentials from environment variables. (#13601)

  • Remove unused 'context' variable in task_instance.py (#14049)

  • Disable suppress_logs_and_warning in cli when debugging (#13180)

Airflow 2.0.2, 2021-04-19

Bug Fixes

  • Bugfix: TypeError when Serializing & sorting iterable properties of DAGs (#15395)

  • Fix missing on_load trigger for folder-based plugins (#15208)

  • kubernetes cleanup-pods subcommand will only clean up Airflow-created Pods (#15204)

  • Fix password masking in CLI action_logging (#15143)

  • Fix url generation for TriggerDagRunOperatorLink (#14990)

  • Restore base lineage backend (#14146)

  • Unable to trigger backfill or manual jobs with Kubernetes executor. (#14160)

  • Bugfix: Task docs are not shown in the Task Instance Detail View (#15191)

  • Bugfix: Fix overriding pod_template_file in KubernetesExecutor (#15197)

  • Bugfix: resources in executor_config breaks Graph View in UI (#15199)

  • Fix celery executor bug trying to call len on map (#14883)

  • Fix bug in airflow.stats timing that broke dogstatsd mode (#15132)

  • Avoid scheduler/parser manager deadlock by using non-blocking IO (#15112)

  • Re-introduce dagrun.schedule_delay metric (#15105)

  • Compare string values, not if strings are the same object in Kube executor(#14942)

  • Pass queue to BaseExecutor.execute_async like in airflow 1.10 (#14861)

  • Scheduler: Remove TIs from starved pools from the critical path. (#14476)

  • Remove extra/needless deprecation warnings from airflow.contrib module (#15065)

  • Fix support for long dag_id and task_id in KubernetesExecutor (#14703)

  • Sort lists, sets and tuples in Serialized DAGs (#14909)

  • Simplify cleaning string passed to origin param (#14738) (#14905)

  • Fix error when running tasks with Sentry integration enabled. (#13929)

  • Webserver: Sanitize string passed to origin param (#14738)

  • Fix losing duration < 1 secs in tree (#13537)

  • Pin SQLAlchemy to <1.4 due to breakage of sqlalchemy-utils (#14812)

  • Fix KubernetesExecutor issue with deleted pending pods (#14810)

  • Default to Celery Task model when backend model does not exist (#14612)

  • Bugfix: Plugins endpoint was unauthenticated (#14570)

  • BugFix: fix DAG doc display (especially for TaskFlow DAGs) (#14564)

  • BugFix: TypeError in airflow.kubernetes.pod_launcher's monitor_pod (#14513)

  • Bugfix: Fix wrong output of tags and owners in dag detail API endpoint (#14490)

  • Fix logging error with task error when JSON logging is enabled (#14456)

  • Fix statsd metrics not sending when using daemon mode (#14454)

  • Gracefully handle missing start_date and end_date for DagRun (#14452)

  • BugFix: Serialize max_retry_delay as a timedelta (#14436)

  • Fix crash when user clicks on "Task Instance Details" caused by start_date being None (#14416)

  • BugFix: Fix TaskInstance API call fails if a task is removed from running DAG (#14381)

  • Scheduler should not fail when invalid executor_config is passed (#14323)

  • Fix bug allowing task instances to survive when dagrun_timeout is exceeded (#14321)

  • Fix bug where DAG timezone was not always shown correctly in UI tooltips (#14204)

  • Use Lax for cookie_samesite when empty string is passed (#14183)

  • [AIRFLOW-6076] fix dag.cli() KeyError (#13647)

  • Fix running child tasks in a subdag after clearing a successful subdag (#14776)

Improvements

  • Remove unused JS packages causing false security alerts (#15383)

  • Change default of [kubernetes] enable_tcp_keepalive for new installs to True (#15338)

  • Fixed #14270: Add error message in OOM situations (#15207)

  • Better compatibility/diagnostics for arbitrary UID in docker image (#15162)

  • Updates 3.6 limits for latest versions of a few libraries (#15209)

  • Adds Blinker dependency which is missing after recent changes (#15182)

  • Remove 'conf' from search_columns in DagRun View (#15099)

  • More proper default value for namespace in K8S cleanup-pods CLI (#15060)

  • Faster default role syncing during webserver start (#15017)

  • Speed up webserver start when there are many DAGs (#14993)

  • Much easier to use and better documented Docker image (#14911)

  • Use libyaml C library when available. (#14577)

  • Don't create unittest.cfg when not running in unit test mode (#14420)

  • Webserver: Allow Filtering TaskInstances by queued_dttm (#14708)

  • Update Flask-AppBuilder dependency to allow 3.2 (and all 3.x series) (#14665)

  • Remember expanded task groups in browser local storage (#14661)

  • Add plain format output to cli tables (#14546)

  • Make airflow dags show command display TaskGroups (#14269)

  • Increase maximum size of extra connection field. (#12944)

  • Speed up clear_task_instances by doing a single sql delete for TaskReschedule (#14048)

  • Add more flexibility with FAB menu links (#13903)

  • Add better description and guidance in case of sqlite version mismatch (#14209)

Doc only changes

  • Add documentation create/update community providers (#15061)

  • Fix mistake and typos in airflow.utils.timezone docstrings (#15180)

  • Replace new url for Stable Airflow Docs (#15169)

  • Docs: Clarify behavior of delete_worker_pods_on_failure (#14958)

  • Create a documentation package for Docker image (#14846)

  • Multiple minor doc (OpenAPI) fixes (#14917)

  • Replace Graph View Screenshot to show Auto-refresh (#14571)

Misc/Internal

  • Import Connection lazily in hooks to avoid cycles (#15361)

  • Rename last_scheduler_run into last_parsed_time, and ensure it's updated in DB (#14581)

  • Make TaskInstance.pool_slots not nullable with a default of 1 (#14406)

  • Log migrations info in consistent way (#14158)

Airflow 2.0.1, 2021-02-08

Bug Fixes

  • Bugfix: Return XCom Value in the XCom Endpoint API (#13684)

  • Bugfix: Import error when using custom backend and sql_alchemy_conn_secret (#13260)

  • Allow PID file path to be relative when daemonize a process (scheduler, kerberos, etc) (#13232)

  • Bugfix: no generic DROP CONSTRAINT in MySQL during airflow db upgrade (#13239)

  • Bugfix: Sync Access Control defined in DAGs when running sync-perm (#13377)

  • Stop sending Callback Requests if no callbacks are defined on DAG (#13163)

  • BugFix: Dag-level Callback Requests were not run (#13651)

  • Stop creating duplicate Dag File Processors (#13662)

  • Filter DagRuns with Task Instances in removed State while Scheduling (#13165)

  • Bump datatables.net from 1.10.21 to 1.10.22 in /airflow/www (#13143)

  • Bump datatables.net JS to 1.10.23 (#13253)

  • Bump dompurify from 2.0.12 to 2.2.6 in /airflow/www (#13164)

  • Update minimum cattrs version (#13223)

  • Remove inapplicable arg 'output' for CLI pools import/export (#13071)

  • Webserver: Fix the behavior to deactivate the authentication option and add docs (#13191)

  • Fix: add support for no-menu plugin views (#11742)

  • Add python-daemon limit for Python 3.8+ to fix daemon crash (#13540)

  • Change the default celery worker_concurrency to 16 (#13612)

  • Audit Log records View should not contain link if dag_id is None (#13619)

  • Fix invalid continue_token for cleanup list pods (#13563)

  • Switches to latest version of snowflake connector (#13654)

  • Fix backfill crash on task retry or reschedule (#13712)

  • Setting max_tis_per_query to 0 now correctly removes the limit (#13512)

  • Fix race conditions in task callback invocations (#10917)

  • Fix webserver exiting when gunicorn master crashes (#13518)(#13780)

  • Fix SQL syntax to check duplicate connections (#13783)

  • BaseBranchOperator will push to xcom by default (#13704) (#13763)

  • Fix Deprecation for configuration.getsection (#13804)

  • Fix TaskNotFound in log endpoint (#13872)

  • Fix race condition when using Dynamic DAGs (#13893)

  • Fix: Linux/Chrome window bouncing in Webserver

  • Fix db shell for sqlite (#13907)

  • Only compare updated time when Serialized DAG exists (#13899)

  • Fix dag run type enum query for mysqldb driver (#13278)

  • Add authentication to lineage endpoint for experimental API (#13870)

  • Do not add User role perms to custom roles. (#13856)

  • Do not add Website.can_read access to default roles. (#13923)

  • Fix invalid value error caused by long Kubernetes pod name (#13299)

  • Fix DB Migration for SQLite to upgrade to 2.0 (#13921)

  • Bugfix: Manual DagRun trigger should not skip scheduled runs (#13963)

  • Stop loading Extra Operator links in Scheduler (#13932)

  • Added missing return parameter in read function of FileTaskHandler (#14001)

  • Bugfix: Do not try to create a duplicate Dag Run in Scheduler (#13920)

  • Make v1/config endpoint respect webserver expose_config setting (#14020)

  • Disable row level locking for Mariadb and MySQL <8 (#14031)

  • Bugfix: Fix permissions to triggering only specific DAGs (#13922)

  • Fix broken SLA Mechanism (#14056)

  • Bugfix: Scheduler fails if task is removed at runtime (#14057)

  • Remove permissions to read Configurations for User and Viewer roles (#14067)

  • Fix DB Migration from 2.0.1rc1

Improvements

  • Increase the default min_file_process_interval to decrease CPU Usage (#13664)

  • Dispose connections when running tasks with os.fork & CeleryExecutor (#13265)

  • Make function purpose clearer in example_kubernetes_executor example dag (#13216)

  • Remove unused libraries - flask-swagger, funcsigs (#13178)

  • Display alternative tooltip when a Task has yet to run (no TI) (#13162)

  • User werkzeug's own type conversion for request args (#13184)

  • UI: Add queued_by_job_id & external_executor_id Columns to TI View (#13266)

  • Make json-merge-patch an optional library and unpin it (#13175)

  • Adds missing LDAP "extra" dependencies to ldap provider. (#13308)

  • Refactor setup.py to better reflect changes in providers (#13314)

  • Pin pyjwt and Add integration tests for Apache Pinot (#13195)

  • Removes provider-imposed requirements from setup.cfg (#13409)

  • Replace deprecated decorator (#13443)

  • Streamline & simplify __eq__ methods in models Dag and BaseOperator (#13449)

  • Additional properties should be allowed in provider schema (#13440)

  • Remove unused dependency - contextdecorator (#13455)

  • Remove 'typing' dependency (#13472)

  • Log migrations info in consistent way (#13458)

  • Unpin mysql-connector-python to allow 8.0.22 (#13370)

  • Remove thrift as a core dependency (#13471)

  • Add NotFound response for DELETE methods in OpenAPI YAML (#13550)

  • Stop Log Spamming when [core] lazy_load_plugins is False (#13578)

  • Display message and docs link when no plugins are loaded (#13599)

  • Unpin restriction for colorlog dependency (#13176)

  • Add missing Dag Tag for Example DAGs (#13665)

  • Support tables in DAG docs (#13533)

  • Add python3-openid dependency (#13714)

  • Add __repr__ for Executors (#13753)

  • Add description to hint if conn_type is missing (#13778)

  • Upgrade Azure blob to v12 (#12188)

  • Add extra field to get_connnection REST endpoint (#13885)

  • Make Smart Sensors DB Migration idempotent (#13892)

  • Improve the error when DAG does not exist when running dag pause command (#13900)

  • Update airflow_local_settings.py to fix an error message (#13927)

  • Only allow passing JSON Serializable conf to TriggerDagRunOperator (#13964)

  • Bugfix: Allow getting details of a DAG with null start_date (REST API) (#13959)

  • Add params to the DAG details endpoint (#13790)

  • Make the role assigned to anonymous users customizable (#14042)

  • Retry critical methods in Scheduler loop in case of OperationalError (#14032)

Doc only changes

  • Add Missing Statsd Metrics in Docs (#13708)

  • Add Missing Email configs in Configuration doc (#13709)

  • Add quick start for Airflow on Docker (#13660)

  • Describe which Python versions are supported (#13259)

  • Add note block to 2.x migration docs (#13094)

  • Add documentation about webserver_config.py (#13155)

  • Add missing version information to recently added configs (#13161)

  • API: Use generic information in UpdateMask component (#13146)

  • Add Airflow 2.0.0 to requirements table (#13140)

  • Avoid confusion in doc for CeleryKubernetesExecutor (#13116)

  • Update docs link in REST API spec (#13107)

  • Add link to PyPI Repository to provider docs (#13064)

  • Fix link to Airflow master branch documentation (#13179)

  • Minor enhancements to Sensors docs (#13381)

  • Use 2.0.0 in Airflow docs & Breeze (#13379)

  • Improves documentation regarding providers and custom connections (#13375)(#13410)

  • Fix malformed table in production-deployment.rst (#13395)

  • Update celery.rst to fix broken links (#13400)

  • Remove reference to scheduler run_duration param in docs (#13346)

  • Set minimum SQLite version supported (#13412)

  • Fix installation doc (#13462)

  • Add docs about mocking variables and connections (#13502)

  • Add docs about Flask CLI (#13500)

  • Fix Upgrading to 2 guide to use rbac UI (#13569)

  • Make docs clear that Auth can not be disabled for Stable API (#13568)

  • Remove archived links from docs & add link for AIPs (#13580)

  • Minor fixes in upgrading-to-2.rst (#13583)

  • Fix Link in Upgrading to 2.0 guide (#13584)

  • Fix heading for Mocking section in best-practices.rst (#13658)

  • Add docs on how to use custom operators within plugins folder (#13186)

  • Update docs to register Operator Extra Links (#13683)

  • Improvements for database setup docs (#13696)

  • Replace module path to Class with just Class Name (#13719)

  • Update DAG Serialization docs (#13722)

  • Fix link to Apache Airflow docs in webserver (#13250)

  • Clarifies differences between extras and provider packages (#13810)

  • Add information about all access methods to the environment (#13940)

  • Docs: Fix FAQ on scheduler latency (#13969)

  • Updated taskflow api doc to show dependency with sensor (#13968)

  • Add deprecated config options to docs (#13883)

  • Added a FAQ section to the Upgrading to 2 doc (#13979)

Airflow 2.0.0, 2020-12-18

The full changelog is about 3,000 lines long (already excluding everything backported to 1.10) so please check Airflow 2.0.0 Highligths Blog Post instead.

Airflow 1.10.15, 2021-03-17

Bug Fixes

  • Fix airflow db upgrade to upgrade db as intended (#13267)

  • Moved boto3 limitation to snowflake (#13286)

  • KubernetesExecutor should accept images from executor_config (#13074)

  • Scheduler should acknowledge active runs properly (#13803)

  • Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)

  • Include airflow/contrib/executors in the dist package

  • Pin Click version for Python 2.7 users

  • Ensure all statsd timers use millisecond values. (#10633)

  • [kubernetes_generate_dag_yaml] - Fix dag yaml generate function (#13816)

  • Fix airflow tasks clear` cli command with ``--yes (#14188)

  • Fix permission error on non-POSIX filesystem (#13121) (#14383)

  • Fixed deprecation message for "variables" command (#14457)

  • BugFix: fix the delete_dag function of json_client (#14441)

  • Fix merging of secrets and configmaps for KubernetesExecutor (#14090)

  • Fix webserver exiting when gunicorn master crashes (#13470)

  • Bump ini from 1.3.5 to 1.3.8 in airflow/www_rbac

  • Bump datatables.net from 1.10.21 to 1.10.23 in airflow/www_rbac

  • Webserver: Sanitize string passed to origin param (#14738)

  • Make rbac_app's db.session use the same timezone with @provide_session (#14025)

Improvements

  • Adds airflow as viable docker command in official image (#12878)

  • StreamLogWriter: Provide (no-op) close method (#10885)

  • Add 'airflow variables list' command for 1.10.x transition version (#14462)

Doc only changes

  • Update URL for Airflow docs (#13561)

  • Clarifies version args for installing 1.10 in Docker (#12875)

Airflow 1.10.14, 2020-12-10

Bug Fixes

  • BugFix: Tasks with depends_on_past or task_concurrency are stuck (#12663)

  • Fix issue with empty Resources in executor_config (#12633)

  • Fix: Deprecated config force_log_out_after was not used (#12661)

  • Fix empty asctime field in JSON formatted logs (#10515)

  • [AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY (#3651)

  • [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac (#3729)

  • [AIRFLOW-2886] Generate random Flask SECRET_KEY in default config (#3738)

  • Add missing comma in setup.py (#12790)

  • Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)

  • Fix setup.py missing comma in setup_requires (#12880)

  • Don't emit first_task_scheduling_delay metric for only-once dags (#12835)

Improvements

  • Update setup.py to get non-conflicting set of dependencies (#12636)

  • Rename [scheduler] max_threads to [scheduler] parsing_processes (#12605)

  • Add metric for scheduling delay between first run task & expected start time (#9544)

  • Add new-style 2.0 command names for Airflow 1.10.x (#12725)

  • Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)

  • Don't let webserver run with dangerous config (#12747)

  • Replace pkg_resources with importlib.metadata to avoid VersionConflict errors (#12694)

Doc only changes

  • Clarified information about supported Databases

Airflow 1.10.13, 2020-11-25

New Features

  • Add "already checked" to failed pods in K8sPodOperator (#11368)

  • Pass SQLAlchemy engine options to FAB based UI (#11395)

  • [AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)

  • Add permission "extra_links" for Viewer role and above (#10719)

  • Add generate_yaml command to easily test KubernetesExecutor before deploying pods (#10677)

  • Add Secrets backend for Microsoft Azure Key Vault (#10898)

Bug Fixes

  • SkipMixin: Handle empty branches (#11120)

  • [AIRFLOW-5274] dag loading duration metric name too long (#5890)

  • Handle no Dagrun in DagrunIdDep (#8389) (#11343)

  • Fix Kubernetes Executor logs for long dag names (#10942)

  • Add on_kill support for the KubernetesPodOperator (#10666)

  • KubernetesPodOperator template fix (#10963)

  • Fix displaying of add serialized_dag table migration

  • Fix Start Date tooltip on DAGs page (#10637)

  • URL encode execution date in the Last Run link (#10595)

  • Fixes issue with affinity backcompat in Airflow 1.10

  • Fix KubernetesExecutor import in views.py

  • Fix issues with Gantt View (#12419)

  • Fix Entrypoint and _CMD config variables (#12411)

  • Fix operator field update for SerializedBaseOperator (#10924)

  • Limited cryptography to < 3.2 for Python 2.7

  • Install cattr on Python 3.7 - Fix docs build on RTD (#12045)

  • Limit version of marshmallow-sqlalchemy

  • Pin kubernetes to a max version of 11.0.0 (#11974)

  • Use snakebite-py3 for HDFS dependency for Python3 (#12340)

  • Removes snakebite kerberos dependency (#10865)

  • Fix failing dependencies for FAB and Celery (#10828)

  • Fix pod_mutation_hook for 1.10.13 (#10850)

  • Fix formatting of Host information

  • Fix Logout Google Auth issue in Non-RBAC UI (#11890)

  • Add missing imports to app.py (#10650)

  • Show Generic Error for Charts & Query View in old UI (#12495)

  • TimeSensor should respect the default_timezone config (#9699)

  • TimeSensor should respect DAG timezone (#9882)

  • Unify user session lifetime configuration (#11970)

  • Handle outdated webserver session timeout gracefully. (#12332)

Improvements

  • Add XCom.deserialize_value to Airflow 1.10.13 (#12328)

  • Mount airflow.cfg to pod_template_file (#12311)

  • All k8s object must comply with JSON Schema (#12003)

  • Validate Airflow chart values.yaml & values.schema.json (#11990)

  • Pod template file uses custom custom env variable (#11480)

  • Bump attrs and cattrs dependencies (#11969)

  • Bump attrs to > 20.0 (#11799)

  • [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (#4751)

  • Rename task with duplicate task_id

  • Manage Flask AppBuilder Tables using Alembic Migrations (#12352)

  • airflow test only works for tasks in 1.10, not whole dags (#11191)

  • Improve warning messaging for duplicate task_ids in a DAG (#11126)

  • Pins moto to 1.3.14 (#10986)

  • DbApiHook: Support kwargs in get_pandas_df (#9730)

  • Make grace_period_seconds option on K8sPodOperator (#10727)

  • Fix syntax error in Dockerfile 'maintainer' Label (#10899)

  • The entrypoints in Docker Image should be owned by Airflow (#10853)

  • Make dockerfiles Google Shell Guide Compliant (#10734)

  • clean-logs script for Dockerfile: trim logs before sleep (#10685)

  • When sending tasks to celery from a sub-process, reset signal handlers (#11278)

  • SkipMixin: Add missing session.commit() and test (#10421)

  • Webserver: Further Sanitize values passed to origin param (#12459)

  • Security upgrade lodash from 4.17.19 to 4.17.20 (#11095)

  • Log instead of raise an Error for unregistered OperatorLinks (#11959)

  • Mask Password in Log table when using the CLI (#11468)

  • [AIRFLOW-3607] Optimize dep checking when depends on past set and concurrency limit

  • Execute job cancel HTTPRequest in Dataproc Hook (#10361)

  • Use rst lexer to format Airflow upgrade check output (#11259)

  • Remove deprecation warning from contrib/kubernetes/pod.py

  • adding body as templated field for CloudSqlImportOperator (#10510)

  • Change log level for User's session to DEBUG (#12414)

Deprecations

  • Deprecate importing Hooks from plugin-created module (#12133)

  • Deprecate adding Operators and Sensors via plugins (#12069)

Doc only changes

  • [Doc] Correct description for macro task_instance_key_str (#11062)

  • Checks if all the libraries in setup.py are listed in installation.rst file (#12023)

  • Revise "Project Focus" copy (#12011)

  • Move Project focus and Principles higher in the README (#11973)

  • Remove archived link from README.md (#11945)

  • Update download url for Airflow Version (#11800)

  • Add Project URLs for PyPI page (#11801)

  • Move Backport Providers docs to our docsite (#11136)

  • Refactor rebase copy (#11030)

  • Add missing images for kubernetes executor docs (#11083)

  • Fix indentation in executor_config example (#10467)

  • Enhanced the Kubernetes Executor doc (#10433)

  • Refactor content to a markdown table (#10863)

  • Rename "Beyond the Horizon" section and refactor content (#10802)

  • Refactor official source section to use bullets (#10801)

  • Add section for official source code (#10678)

  • Add redbubble link to Airflow merchandise (#10359)

  • README Doc: Link to Airflow directory in ASF Directory (#11137)

  • Fix the default value for VaultBackend's config_path (#12518)

Airflow 1.10.12, 2020-08-25

New Features

  • Add DateTimeSensor (#9697)

  • Add AirflowClusterPolicyViolation support to Airflow local settings (#10282)

  • Get Airflow configs with sensitive data from Secret Backends (#9645)

  • [AIRFLOW-4734] Upsert functionality for PostgresHook.insert_rows() (#8625)

  • Allow defining custom XCom class (#8560)

Bug Fixes

  • Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067)

  • Fixes PodMutationHook for backwards compatibility (#9903)

  • Fix bug in executor_config when defining resources (#9935)

  • Respect DAG Serialization setting when running sync_perm (#10321)

  • Show correct duration on graph view for running task (#8311) (#8675)

  • Fix regression in SQLThresholdCheckOperator (#9312)

  • [AIRFLOW-6931] Fixed migrations to find all dependencies for MSSQL (#9891)

  • Avoid sharing session with RenderedTaskInstanceFields write and delete (#9993)

  • Fix clear future recursive when ExternalTaskMarker is used (#9515)

  • Handle IntegrityError while creating TIs (#10136)

  • Fix airflow-webserver startup errors when using Kerberos Auth (#10047)

  • Fixes treatment of open slots in scheduler (#9316) (#9505)

  • Fix KubernetesPodOperator reattachment (#10230)

  • Fix more PodMutationHook issues for backwards compatibility (#10084)

  • [AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276)

  • Fix task_instance_mutation_hook (#9910)

  • Fixes failing formatting of DAG file containing {} in docstring (#9779)

  • Fix is_terminal_support_colors function (#9734)

  • Fix PythonVirtualenvOperator when using provide_context=True (#8256)

  • Fix issue with mounting volumes from secrets (#10366)

  • BugFix: K8s Executor Multinamespace mode is evaluated to true by default (#10410)

  • Make KubernetesExecutor recognize kubernetes_labels (#10412)

  • Fix broken Kubernetes PodRuntimeInfoEnv (#10478)

Improvements

  • Use Hash of Serialized DAG to determine DAG is changed or not (#10227)

  • Update Serialized DAGs in Webserver when DAGs are Updated (#9851)

  • Do not Update Serialized DAGs in DB if DAG did not change (#9850)

  • Add __repr__ to SerializedDagModel (#9862)

  • Update JS packages to latest versions (#9811) (#9921)

  • UI Graph View: Focus upstream / downstream task dependencies on mouseover (#9303)

  • Allow image in KubernetesPodOperator to be templated (#10068)

  • [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod (#7523)

  • Improve process terminating in scheduler_job (#8064)

  • Replace deprecated base classes used in bigquery_check_operator (#10272)

  • [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)

  • Limit all google-cloud api to <2.0.0 (#10317)

  • [AIRFLOW-6706] Lazy load operator extra links (#7327) (#10318)

  • Add Snowflake support to SQL operator and sensor (#9843)

  • Makes multi-namespace mode optional (#9570)

  • Pin pyarrow < 1.0

  • Pin pymongo version to <3.11.0

  • Pin google-cloud-container to <2 (#9901)

  • Dockerfile: Remove package.json and yarn.lock from the prod image (#9814)

  • Dockerfile: The group of embedded DAGs should be root to be OpenShift compatible (#9794)

  • Update upper limit of flask-swagger, gunicorn & jinja2 (#9684)

  • Webserver: Sanitize values passed to origin param (#10334)

  • Sort connection type list in add/edit page alphabetically (#8692)

Doc only changes

  • Add new committers: Ry Walker & Leah Cole to project.rst (#9892)

  • Add Qingping Hou to committers list (#9725)

  • Updated link to official documentation (#9629)

  • Create a short-link for Airflow Slack Invites (#10034)

  • Fix docstrings in BigQueryGetDataOperator (#10042)

  • Set language on code-block on docs/howto/email-config.rst (#10238)

  • Remove duplicate line from 1.10.10 CHANGELOG (#10289)

  • Improve heading on Email Configuration page (#10175)

  • Fix link for the Jinja Project in docs/tutorial.rst (#10245)

  • Create separate section for Cron Presets (#10247)

  • Add Syntax Highlights to code-blocks in docs/best-practices.rst (#10258)

  • Fix docstrings in BigQueryGetDataOperator (#10042)

  • Fix typo in Task Lifecycle section (#9867)

  • Make Secret Backend docs clearer about Variable & Connection View (#8913)

Airflow 1.10.11, 2020-07-10

New Features

  • Add task instance mutation hook (#8852)

  • Allow changing Task States Colors (#9520)

  • Add support for AWS Secrets Manager as Secrets Backend (#8186)

  • Add Airflow info command to the CLI (#8704)

  • Add Local Filesystem Secret Backend (#8596)

  • Add Airflow config CLI command (#8694)

  • Add Support for Python 3.8 (#8836)(#8823)

  • Allow K8S worker pod to be configured from JSON/YAML file (#6230)

  • Add quarterly to crontab presets (#6873)

  • Add support for ephemeral storage on KubernetesPodOperator (#6337)

  • Add AirflowFailException to fail without any retry (#7133)

  • Add SQL Branch Operator (#8942)

Bug Fixes

  • Use NULL as dag.description default value (#7593)

  • BugFix: DAG trigger via UI error in RBAC UI (#8411)

  • Fix logging issue when running tasks (#9363)

  • Fix JSON encoding error in DockerOperator (#8287)

  • Fix alembic crash due to typing import (#6547)

  • Correctly restore upstream_task_ids when deserializing Operators (#8775)

  • Correctly store non-default Nones in serialized tasks/dags (#8772)

  • Correctly deserialize dagrun_timeout field on DAGs (#8735)

  • Fix tree view if config contains " (#9250)

  • Fix Dag Run UI execution date with timezone cannot be saved issue (#8902)

  • Fix Migration for MSSQL (#8385)

  • RBAC ui: Fix missing Y-axis labels with units in plots (#8252)

  • RBAC ui: Fix missing task runs being rendered as circles instead (#8253)

  • Fix: DagRuns page renders the state column with artifacts in old UI (#9612)

  • Fix task and dag stats on home page (#8865)

  • Fix the trigger_dag api in the case of nested subdags (#8081)

  • UX Fix: Prevent undesired text selection with DAG title selection in Chrome (#8912)

  • Fix connection add/edit for spark (#8685)

  • Fix retries causing constraint violation on MySQL with DAG Serialization (#9336)

  • [AIRFLOW-4472] Use json.dumps/loads for templating lineage data (#5253)

  • Restrict google-cloud-texttospeech to <v2 (#9137)

  • Fix pickling failure when spawning processes (#8671)

  • Pin Version of azure-cosmos to <4 (#8956)

  • Azure storage 0.37.0 is not installable any more (#8833)

  • Fix modal_backdrop z-index in the UI (#7313)

  • Fix Extra Links in Gantt View (#8308)

  • Bug fix for EmrAddStepOperator init with cluster_name error (#9235)

  • Fix KubernetesPodOperator pod name length validation (#8829)

  • Fix non updating DAG code by checking against last modification time (#8266)

  • BugFix: Unpausing a DAG with catchup=False creates an extra DAG run (#8776)

Improvements

  • Improve add_dag_code_table migration (#8176)

  • Persistent display/filtering of DAG status (#8106)

  • Set unique logger names (#7330)

  • Update the version of cattrs from 0.9 to 1.0 to support Python 3.8 (#7100)

  • Reduce response payload size of /dag_stats and /task_stats (#8655)

  • Add authenticator parameter to snowflake_hook (#8642)

  • Show "Task Reschedule" table in Airflow Webserver (#9521)

  • Change worker_refresh_interval fallback to default of 30 (#9588)

  • Use pformat instead of str to render arguments in WebUI (#9587)

  • Simplify DagFileProcessorManager (#7521)

  • Reload gunicorn when plugins has been changed (#8997)

  • Fix the default value for store_dag_code (#9554)

  • Add support for fetching logs from running pods (#8626)

  • Persist start/end date and duration for DummyOperator Task Instance (#8663)

  • Ensure "started"/"ended" in tooltips are not shown if job not started (#8667)

  • Add context to execution_date_fn in ExternalTaskSensor (#8702)

  • Avoid color info in response of /dag_stats & /task_stats (#8742)

  • Make loading plugins from entrypoint fault-tolerant (#8732)

  • Refactor Kubernetes worker config (#7114)

  • Add default conf parameter to Spark JDBC Hook (#8787)

  • Allow passing backend_kwargs to AWS SSM client (#8802)

  • Filter dags by clicking on tag (#8897)

  • Support k8s auth method in Vault Secrets provider (#8640)

  • Monitor pods by labels instead of names (#6377)

  • Optimize count query on /home (#8729)

  • Fix JSON string escape in tree view (#8551)

  • Add TaskInstance state to TI Tooltip to be colour-blind friendlier (#8910)

  • Add a tip to trigger DAG screen (#9049)

  • Use Markup for htmlcontent for landing_times (#9242)

  • Pinning max pandas version to 2.0 (lesser than) to allow pandas 1.0 (#7954)

  • Update example webserver_config.py to show correct CSRF config (#8944)

  • Fix displaying Executor Class Name in "Base Job" table (#8679)

  • Use existing DagBag for 'dag_details' & 'trigger' Endpoints (#8501)

  • Flush pending Sentry exceptions before exiting (#7232)

  • Display DAG run conf in the list view (#6794)

  • Fix performance degradation when updating dagrun state (#8435)

  • Don't use the |safe filter in code, it's risky (#9180)

  • Validate only task commands are run by executors (#9178)

  • Show Deprecation warning on duplicate Task ids (#8728)

  • Move DAG._schedule_interval logic out of DAG.__init__ (#8225)

  • Make retrieving Paused Dag ids a separate method (#7587)

  • Bulk fetch paused_dag_ids (#7476)

  • Add a configurable DAGs volume mount path for Kubernetes (#8147)

  • Add schedulername option for KubernetesPodOperator (#6088)

  • Support running git sync container as root (#6312)

  • Add extra options for Slack Webhook operator and Slack hook (#9409)

  • Monkey patch greenlet Celery pools (#8559)

  • Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)

  • Prevent clickable sorting on non sortable columns in TI view (#8681)

  • Make hive macros py3 compatible (#8598)

  • Fix SVG tooltip positioning with custom scripting (#8269)

  • Avoid unnecessary sleep to maintain local task job heart rate (#6553)

  • Include some missing RBAC roles on User and Viewer roles (#9133)

  • Show Dag's Markdown docs on Tree View (#9448)

  • Improved compatibility with Python 3.5+ - Convert signal.SIGTERM to int (#9207)

  • Add 'main' param to template_fields in DataprocSubmitPySparkJobOperator (#9154)

  • Make it possible to silence warnings from Airflow (#9208)

  • Remove redundant count query in BaseOperator.clear() (#9362)

  • Fix DB migration message (#8988)

  • Fix awkward log info in dbapi_hook (#8482)

  • Fix Celery default to no longer allow pickle (#7205)

  • Further validation that only task commands are run by executors (#9240)

  • Remove vendored nvd3 and slugify libraries (#9136)

  • Enable configurable git sync depth (#9094)

  • Reduce the required resources for the Kubernetes's sidecar (#6062)

  • Refactor K8S codebase with k8s API models (#5481)

  • Move k8s executor out of contrib to closer match master (#8904)

  • Allow filtering using "event" and "owner" in "Log" view (#4881)

  • Add Yandex.Cloud custom connection to 1.10 (#8791)

  • Add table-hover css class to DAGs table (#5033)

  • Show un/pause errors in dags view. (#7669)

  • Restructure database queries on /home (#4872)

  • Add Cross Site Scripting defense (#6913)

  • Make Gantt tooltip the same as Tree and Graph view (#8220)

  • Add config to only delete worker pod on task failure (#7507)(#8312)

  • Remove duplicate error message on chart connection failure (#8476)

  • Remove default value spark_binary (#8508)

  • Expose Airflow Webserver Port in Production Docker Image (#8228)

  • Commit after each alembic migration (#4797)

  • KubernetesPodOperator fixes and test (#6524)

  • Docker Image: Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)

  • Docker Image: Add ADDITIONAL_PYTHON_DEPS (#9031)

  • Remove httplib2 from Google requirements (#9194)

  • Merging multiple SQL operators (#9124)

  • Adds hive as extra in pyhive dependency (#9075)

  • Change default auth for experimental backend to deny_all (#9611)

  • Restrict changing XCom values from the Webserver (#9614)

  • Add __repr__ for DagTag so tags display properly in /dagmodel/show (#8719)

  • Functionality to shuffle HMS connections used by HiveMetastoreHook facilitating load balancing (#9280)

  • Expose SQLAlchemy's connect_args and make it configurable (#6478)

Doc only changes

  • Add docs on using DAGRun.conf (#9578)

  • Enforce code-block directives in doc (#9443)

  • Carefully parse warning messages when building documentation (#8693)

  • Make KubernetesPodOperator clear in docs (#8444)

  • Improve language in Pod Mutation Hook docs (#8445)

  • Fix formatting of Pool docs in concepts.rst (#8443)

  • Make doc clearer about Airflow Variables using Environment Variables (#8427)

  • Fix pools doc for LocalExecutor (#7643)

  • Add Local and Sequential Executors to Doc (#8084)

  • Add documentation for CLI command Airflow dags test (#8251)

  • Fix typo in DAG Serialization documentation (#8317)

  • Add scheduler in production section (#7351)

  • Add a structural dag validation example (#6727)

  • Adding Task re-run documentation (#6295)

  • Fix outdated doc on settings.policy (#7532)

  • Add docs about reload_on_plugin_change option (#9575)

  • Add copy button to Code Blocks in Airflow Docs (#9450)

  • Update commands in docs for v1.10+ (#9585)

  • Add more info on dry-run CLI option (#9582)

  • Document default timeout value for SSHOperator (#8744)

  • Fix docs on creating CustomOperator (#8678)

  • Enhanced documentation around Cluster Policy (#8661)

  • Use sphinx syntax in concepts.rst (#7729)

  • Update README to remove Python 3.8 limitation for Master (#9451)

  • Add note about using dag_run.conf in BashOperator (#9143)

  • Improve tutorial - Include all imports statements (#8670)

  • Added more precise Python requirements to README.md (#8455)

  • Fix Airflow Stable version in README.md (#9360)

  • Update AWS connection example to show how to set from env var (#9191)

  • Fix list formatting of plugins doc. (#8873)

  • Add 'Version Added' on Secrets Backend docs (#8264)

  • Simplify language re roll-your-own secrets backend (#8257)

  • Add installation description for repeatable PyPi installation (#8513)

  • Add note extra links only render on when using RBAC webserver (#8788)

  • Remove unused Airflow import from docs (#9274)

  • Add PR/issue note in Contribution Workflow Example (#9177)

  • Don't use the term "whitelist" - language matters (#9174)

  • Add docs to change Colors on the Webserver (#9607)

  • Change 'initiate' to 'initialize' in installation.rst (#9619)

  • Replace old Variables View Screenshot with new (#9620)

  • Replace old SubDag zoom screenshot with new (#9621)

  • Update docs about the change to default auth for experimental API (#9617)

Airflow 1.10.10, 2020-04-09

New Features

  • [AIRFLOW-7048] Allow user to chose timezone to use in UI (#8046)

  • Add Production Docker image support (#7832)

  • Get Airflow Variables from Environment Variables (#7923)

  • Get Airflow Variables from Hashicorp Vault (#7944)

  • Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)

  • Get Airflow Variables from GCP Secrets Manager (#7946)

  • [AIRFLOW-5705] Add secrets backend and support for AWS SSM / Get Airflow Connections from AWS Parameter Store(#6376)

  • [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager / Get Airflow Connections from GCP Secrets Manager (#7795)

  • [AIRFLOW-7076] Add support for HashiCorp Vault as Secrets Backend / Get Airflow Connections from Hashicorp Vault (#7741)

  • [AIRFLOW-6685] Add ThresholdCheckOperator (#7353)

  • [AIRFLOW-7080] Add API endpoint to return a DAG's paused state (#7737)

Bug Fixes

Improvements

Misc/Internal

Doc only changes

  • Fix typo for store_serialized_dags config (#7952)

  • Fix broken link in README.md (#7893)

  • Separate supported Postgres versions with comma (#7892)

  • Fix grammar in setup.py (#7877)

  • Add Jiajie Zhong to committers list (#8047)

  • Update Security doc for 1.10.* for auth backends (#8072)

  • Fix Example in config_templates for Secrets Backend (#8074)

  • Add backticks in IMAGES.rst command description (#8075)

  • Change version_added for store_dag_code config (#8076)

  • [AIRFLOW-XXXX] Remove duplicate docs

  • [AIRFLOW-XXXX] Remove the defunct limitation of Dag Serialization (#7716)

  • [AIRFLOW-XXXX] Add prerequisite tasks for all GCP operators guide (#6049)

  • [AIRFLOW-XXXX] Simplify AWS/Azure/Databricks operators listing (#6047)

  • [AIRFLOW-XXXX] Add external reference to all GCP operator guide (#6048)

  • [AIRFLOW-XXXX] Simplify GCP operators listing

  • [AIRFLOW-XXXX] Simplify Qubole operators listing

  • [AIRFLOW-XXXX] Add autogenerated TOC (#6038)

  • [AIRFLOW-XXXX] Create "Using the CLI" page (#5823)

  • [AIRFLOW-XXXX] Group references in one section (#5776)

  • [AIRFLOW-XXXX] Fix analytics doc (#5885)

  • [AIRFLOW-XXXX] Add S3 Logging section (#6039)

  • [AIRFLOW-XXXX] Move Azure Logging section above operators (#6040)

  • [AIRFLOW-XXXX] Update temp link to a fixed link (#7715)

  • [AIRFLOW-XXXX] Add Updating.md section for 1.10.9 (#7385)

  • [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring (#7321)

  • [AIRFLOW-XXXX] Update tests info in CONTRIBUTING.rst (#7466)

  • [AIRFLOW-XXXX] Small BREEZE.rst update (#7487)

  • [AIRFLOW-XXXX] Add instructions for logging to localstack S3 (#7461)

  • [AIRFLOW-XXXX] Remove travis config warnings (#7467)

  • [AIRFLOW-XXXX] Add communication chapter to contributing (#7204)

  • [AIRFLOW-XXXX] Add known issue - example_dags/__init__.py (#7444)

  • [AIRFLOW-XXXX] Fix breeze build-docs (#7445)

  • [AIRFLOW-XXXX] Less verbose docker builds

  • [AIRFLOW-XXXX] Speed up mypy runs (#7421)

  • [AIRFLOW-XXXX] Fix location of kubernetes tests (#7373)

  • [AIRFLOW-XXXX] Remove quotes from domains in Google Oauth (#4226)

  • [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs (#7318)

  • [AIRFLOW-XXXX] Fix typo in the word committer (#7392)

  • [AIRFLOW-XXXX] Remove duplicated paragraph in docs (#7662)

  • Fix reference to KubernetesPodOperator (#8100)

Airflow 1.10.9, 2020-02-07

Bug Fixes

Airflow 1.10.8, 2020-02-07

New Features

Improvements

Bug Fixes

Misc/Internal

Doc only changes

  • [AIRFLOW-XXXX] Consistency fixes in new documentation (#7207)

  • [AIRFLOW-XXXX] Improve grammar and structure in FAQ doc (#7291)

  • [AIRFLOW-XXXX] Fix email configuration link in CONTRIBUTING.rst (#7311)

  • [AIRFLOW-XXXX] Update docs with new BranchPythonOperator behaviour (#4682)

  • [AIRFLOW-XXXX] Fix Typo in scripts/ci/ci_run_airflow_testing.sh (#7235)

  • [AIRFLOW-XXXX] Screenshot showing disk space configuration for OSX (#7226)

  • [AIRFLOW-XXXX] Add mentoring information to contributing docs (#7202)

  • [AIRFLOW-XXXX] Add rebase info to contributing (#7201)

  • [AIRFLOW-XXXX] Increase verbosity of static checks in CI (#7200)

  • [AIRFLOW-XXXX] Adds branching strategy to documentation (#7193)

  • [AIRFLOW-XXXX] Move email configuration from the concept page (#7189)

  • [AIRFLOW-XXXX] Update task lifecycle diagram (#7161)

  • [AIRFLOW-XXXX] Fix reference in concepts doc (#7135)

  • [AIRFLOW-XXXX] Clear debug docs (#7104)

  • [AIRFLOW-XXXX] Fix typos and broken links in development docs (#7086)

  • [AIRFLOW-XXXX] Clarify wait_for_downstream and execution_date (#6999)

  • [AIRFLOW-XXXX] Add airflow dags show command guide (#7014)

  • [AIRFLOW-XXXX] Update operation chaining documentation (#7018)

  • [AIRFLOW-XXXX] Add .autoenv_leave.zsh to .gitignore (#6986)

  • [AIRFLOW-XXXX] Fix development packages installation instructions (#6942)

  • [AIRFLOW-XXXX] Update committers list (#7212)

  • [AIRFLOW-XXXX] Move UPDATING changes into correct versions (#7166)

  • [AIRFLOW-XXXX] Add Documentation for check_slas flag (#6974)

  • [AIRFLOW-XXXX] Fix gcp keyfile_dict typo (#6962)

  • [AIRFLOW-XXXX] Add tips for writing a note in UPDATIND.md (#6960)

  • [AIRFLOW-XXXX] Add note warning that bash>4.0 is required for docs build script (#6947)

  • [AIRFLOW-XXXX] Add autoenv to gitignore (#6946)

  • [AIRFLOW-XXXX] Fix GCSTaskHandler Comment Typo (#6928)

  • [AIRFLOW-XXXX] Fix broken DAG Serialization Link (#6891)

  • [AIRFLOW-XXXX] Add versions_added field to configs

Airflow 1.10.7, 2019-12-24

New Features

Improvements

Bug Fixes

Misc/Internal

Doc only changes

  • [AIRFLOW-6211] Doc how to use conda for local virtualenv (#6766)

  • [AIRFLOW-5855] Fix broken reference in custom operator doc (#6508)

  • [AIRFLOW-5875] Fix typo in example_qubole_operator.py (#6525)

  • [AIRFLOW-5702] Fix common docstring issues (#6372)

  • [AIRFLOW-5640] Document and test email parameters of BaseOperator (#6315)

  • [AIRFLOW-XXX] Improve description OpenFaaS Hook (#6187)

  • [AIRFLOW-XXX] GSoD: How to make DAGs production ready (#6515)

  • [AIRFLOW-XXX] Use full command in examples (#5973)

  • [AIRFLOW-XXX] Update docs to accurately describe the precedence of remote and local logs (#5607)

  • [AIRFLOW-XXX] Fix example "extras" field in mysql connect doc (#5285)

  • [AIRFLOW-XXX] Fix wrong inline code highlighting in docs (#5309)

  • [AIRFLOW-XXX] Group executors in one section (#5834)

  • [AIRFLOW-XXX] Add task lifecycle diagram to documentation (#6762)

  • [AIRFLOW-XXX] Highlight code blocks (#6243)

  • [AIRFLOW-XXX] Documents about task_concurrency and pool (#5262)

  • [AIRFLOW-XXX] Fix incorrect docstring parameter (#6649)

  • [AIRFLOW-XXX] Add link to XCom section in concepts.rst (#6791)

  • [AIRFLOW-XXX] Update kubernetes doc with correct path (#6774)

  • [AIRFLOW-XXX] Add information how to configure pytest runner (#6736)

  • [AIRFLOW-XXX] More GSOD improvements (#6585)

  • [AIRFLOW-XXX] Clarified a grammatically incorrect sentence (#6667)

  • [AIRFLOW-XXX] Add notice for Mesos Executor deprecation in docs (#6712)

  • [AIRFLOW-XXX] Update list of pre-commits (#6603)

  • [AIRFLOW-XXX] Updates to Breeze documentation from GSOD (#6285)

  • [AIRFLOW-XXX] Clarify daylight savings time behavior (#6324)

  • [AIRFLOW-XXX] GSoD: Adding 'Create a custom operator' doc (#6348)

  • [AIRFLOW-XXX] Add resources & links to CONTRIBUTING.rst (#6405)

  • [AIRFLOW-XXX] Update chat channel details from gitter to slack (#4149)

  • [AIRFLOW-XXX] Add logo info to readme (#6349)

  • [AIRFLOW-XXX] Fixed case problem with CONTRIBUTING.rst (#6329)

  • [AIRFLOW-XXX] Google Season of Docs updates to CONTRIBUTING doc (#6283)

Airflow 1.10.6, 2019-10-28

New Features

Improvements

Bug Fixes

Doc-only changes

  • [AIRFLOW-5574] Fix Google Analytics script loading (#6218)

  • [AIRFLOW-5588] Add Celery's architecture diagram (#6247)

  • [AIRFLOW-5521] Fix link to GCP documentation (#6150)

  • [AIRFLOW-5398] Update contrib example DAGs to context manager (#5998)

  • [AIRFLOW-5268] Apply same DAG naming conventions as in literature (#5874)

  • [AIRFLOW-5101] Fix inconsistent owner value in examples (#5712)

  • [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319)

  • [AIRFLOW-XXX] Fix Documentation for adding extra Operator Links (#6301)

  • [AIRFLOW-XXX] Add section on task lifecycle & correct casing in docs (#4681)

  • [AIRFLOW-XXX] Make it clear that 1.10.5 was not accidentally omitted from UPDATING.md (#6240)

  • [AIRFLOW-XXX] Improve format in code-block directives (#6242)

  • [AIRFLOW-XXX] Format Sendgrid docs (#6245)

  • [AIRFLOW-XXX] Update to new logo (#6066)

  • [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291)

  • [AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances in 1.10.4 (#6226)

  • [AIRFLOW-XXX] Fix incorrect units in docs for metrics using Timers (#6152)

  • [AIRFLOW-XXX] Fix backtick issues in .rst files & Add Precommit hook (#6162)

  • [AIRFLOW-XXX] Update documentation about variables forcing answer (#6158)

  • [AIRFLOW-XXX] Add a third way to configure authorization (#6134)

  • [AIRFLOW-XXX] Add example of running pre-commit hooks on single file (#6143)

  • [AIRFLOW-XXX] Add information about default pool to docs (#6019)

  • [AIRFLOW-XXX] Make Breeze The default integration test environment (#6001)

Misc/Internal

Airflow 1.10.5, 2019-09-04

New Features

Improvements

Bug fixes

Misc/Internal

Doc-only changes

  • [AIRFLOW-XXX] Fixed Azkaban link (#5865)

  • [AIRFLOW-XXX] Remove duplicate lines from CONTRIBUTING.md (#5830)

  • [AIRFLOW-XXX] Fix incorrect docstring parameter in SchedulerJob (#5729)

Airflow 1.10.4, 2019-08-06

New Features

Improvement

Bug fixes

Misc/Internal

Doc-only changes

  • [AIRFLOW-XXX] Add missing doc for annotations param of KubernetesPodOperator (#5666)

  • [AIRFLOW-XXX] Fix typos in CONTRIBUTING.md (#5626)

  • [AIRFLOW-XXX] Correct BaseSensorOperator docs (#5562)

  • [AIRFLOW-4926] Fix example dags where its start_date is datetime.utcnow() (#5553)

  • [AIRFLOW-4860] Remove Redundant Information in Example Dags (#5497)

  • [AIRFLOW-4767] Fix errors in the documentation of Dataproc Operator (#5487)

  • [AIRFLOW-1684] Branching based on XCom variable (Docs) (#4365)

  • [AIRFLOW-3341] FAQ return DAG object example (#4605)

  • [AIRFLOW-4433] Add missing type in DockerOperator doc string (#5205)

  • [AIRFLOW-4321] Replace incorrect info of Max Size limit of GCS Object Size (#5106)

  • [AIRFLOW-XXX] Add information about user list (#5341)

  • [AIRFLOW-XXX] Clarify documentation related to autodetect parameter in GCS_to_BQ Op (#5294)

  • [AIRFLOW-XXX] Remove mention of pytz compatibility from timezone documentation (#5316)

  • [AIRFLOW-XXX] Add missing docs for GoogleCloudStorageDeleteOperator (#5274)

  • [AIRFLOW-XXX] Remove incorrect note about Scopes of GCP connection (#5242)

  • [AIRFLOW-XXX] Fix mistakes in docs of Dataproc operators (#5192)

  • [AIRFLOW-XXX] Link to correct class for timedelta in macros.rst (#5226)

  • [AIRFLOW-XXX] Add Kamil as committer (#5216)

  • [AIRFLOW-XXX] Add Joshua and Kevin as committer (#5207)

  • [AIRFLOW-XXX] Reduce log spam in tests (#5174)

  • [AIRFLOW-XXX] Speed up tests for PythonSensor (#5158)

  • [AIRFLOW-XXX] Add Bas Harenslak to committer list (#5157)

  • [AIRFLOW-XXX] Add Jarek Potiuk to committer list (#5132)

  • [AIRFLOW-XXX] Update docstring for SchedulerJob (#5105)

  • [AIRFLOW-XXX] Fix docstrings for CassandraToGoogleCloudStorageOperator (#5103)

  • [AIRFLOW-XXX] update SlackWebhookHook and SlackWebhookOperator docstring (#5074)

  • [AIRFLOW-XXX] Ignore Python files under node_modules in docs (#5063)

  • [AIRFLOW-XXX] Build a universal wheel with LICNESE files (#5052)

  • [AIRFLOW-XXX] Fix docstrings of SQSHook (#5099)

  • [AIRFLOW-XXX] Use Py3.7 on readthedocs

  • [AIRFLOW-4446] Fix typos (#5217)

Airflow 1.10.3, 2019-04-09

New Feature

Improvement

Bug fixes

Misc/Internal

Doc-only changes

  • [AIRFLOW-3996] Add view source link to included fragments

  • [AIRFLOW-3811] Automatic generation of API Reference in docs (#4788)

  • [AIRFLOW-3810] Remove duplicate autoclass directive (#4656)

  • [AIRFLOW-XXX] Mention that statsd must be installed to gather metrics (#5038)

  • [AIRFLOW-XXX] Add contents to cli (#4825)

  • [AIRFLOW-XXX] fix check docs failure on CI (#4998)

  • [AIRFLOW-XXX] Fix syntax docs errors (#4789)

  • [AIRFLOW-XXX] Docs rendering improvement (#4684)

  • [AIRFLOW-XXX] Automatically link Jira/GH on doc's changelog page (#4587)

  • [AIRFLOW-XXX] Mention Oracle in the Extra Packages documentation (#4987)

  • [AIRFLOW-XXX] Drop deprecated sudo option; use default docker compose on Travis. (#4732)

  • [AIRFLOW-XXX] Update kubernetes.rst docs (#3875)

  • [AIRFLOW-XXX] Improvements to formatted content in documentation (#4835)

  • [AIRFLOW-XXX] Add Daniel to committer list (#4961)

  • [AIRFLOW-XXX] Add Xiaodong Deng to committers list

  • [AIRFLOW-XXX] Add history become ASF top level project (#4757)

  • [AIRFLOW-XXX] Move out the examples from integration.rst (#4672)

  • [AIRFLOW-XXX] Extract reverse proxy info to a separate file (#4657)

  • [AIRFLOW-XXX] Reduction of the number of warnings in the documentation (#4585)

  • [AIRFLOW-XXX] Fix GCS Operator docstrings (#4054)

  • [AIRFLOW-XXX] Fix Docstrings in Hooks, Sensors & Operators (#4137)

  • [AIRFLOW-XXX] Split guide for operators to multiple files (#4814)

  • [AIRFLOW-XXX] Split connection guide to multiple files (#4824)

  • [AIRFLOW-XXX] Remove almost all warnings from building docs (#4588)

  • [AIRFLOW-XXX] Add backreference in docs between operator and integration (#4671)

  • [AIRFLOW-XXX] Improve linking to classes (#4655)

  • [AIRFLOW-XXX] Mock optional modules when building docs (#4586)

  • [AIRFLOW-XXX] Update plugin macros documentation (#4971)

  • [AIRFLOW-XXX] Add missing docstring for 'autodetect' in GCS to BQ Operator (#4979)

  • [AIRFLOW-XXX] Add missing GCP operators to Docs (#4260)

  • [AIRFLOW-XXX] Fixing the issue in Documentation (#3756)

  • [AIRFLOW-XXX] Add Hint at user defined macros (#4885)

  • [AIRFLOW-XXX] Correct schedule_interval in Scheduler docs (#4157)

  • [AIRFLOW-XXX] Improve airflow-jira script to make RelManager's life easier (#4857)

  • [AIRFLOW-XXX] Add missing class references to docs (#4644)

  • [AIRFLOW-XXX] Fix typo (#4564)

  • [AIRFLOW-XXX] Add a doc about fab security (#4595)

  • [AIRFLOW-XXX] Speed up DagBagTest cases (#3974)

  • [AIRFLOW-XXX] Reduction of the number of warnings in the documentation (#4585)

Airflow 1.10.2, 2019-01-19

New features

Improvements

Bug fixes

Doc-only changes

  • [AIRFLOW-XXX] GCP operators documentation clarifications (#4273)

  • [AIRFLOW-XXX] Docs: Fix paths to GCS transfer operator (#4479)

  • [AIRFLOW-XXX] Add missing GCP operators to Docs (#4260)

  • [AIRFLOW-XXX] Fix Docstrings for Operators (#3820)

  • [AIRFLOW-XXX] Fix inconsistent comment in example_python_operator.py (#4337)

  • [AIRFLOW-XXX] Fix incorrect parameter in SFTPOperator example (#4344)

  • [AIRFLOW-XXX] Add missing remote logging field (#4333)

  • [AIRFLOW-XXX] Revise template variables documentation (#4172)

  • [AIRFLOW-XXX] Fix typo in docstring of gcs_to_bq (#3833)

  • [AIRFLOW-XXX] Fix display of SageMaker operators/hook docs (#4263)

  • [AIRFLOW-XXX] Better instructions for Airflow flower (#4214)

  • [AIRFLOW-XXX] Make pip install commands consistent (#3752)

  • [AIRFLOW-XXX] Add BigQueryGetDataOperator to Integration Docs (#4063)

  • [AIRFLOW-XXX] Don't spam test logs with "bad cron expression" messages (#3973)

  • [AIRFLOW-XXX] Update committer list based on latest TLP discussion (#4427)

  • [AIRFLOW-XXX] Fix incorrect statement in contributing guide (#4104)

  • [AIRFLOW-XXX] Fix Broken Link in CONTRIBUTING.md

  • [AIRFLOW-XXX] Update Contributing Guide - Git Hooks (#4120)

  • [AIRFLOW-3426] Correct Python Version Documentation Reference (#4259)

  • [AIRFLOW-2663] Add instructions to install SSH dependencies

  • [AIRFLOW-XXX] Clean up installation extra packages table (#3750)

  • [AIRFLOW-XXX] Remove redundant space in Kerberos (#3866)

  • [AIRFLOW-3086] Add extras group for google auth to setup.py (#3917)

  • [AIRFLOW-XXX] Add Kubernetes Dependency in Extra Packages Doc (#4281)

  • [AIRFLOW-3696] Add Version info to Airflow Documentation (#4512)

  • [AIRFLOW-XXX] Correct Typo in sensor's exception (#4545)

  • [AIRFLOW-XXX] Fix a typo of config (#4544)

  • [AIRFLOW-XXX] Fix BashOperator Docstring (#4052)

  • [AIRFLOW-3018] Fix Minor issues in Documentation

  • [AIRFLOW-XXX] Fix Minor issues with Azure Cosmos Operator (#4289)

  • [AIRFLOW-3382] Fix incorrect docstring in DatastoreHook (#4222)

  • [AIRFLOW-XXX] Fix copy&paste mistake (#4212)

  • [AIRFLOW-3260] Correct misleading BigQuery error (#4098)

  • [AIRFLOW-XXX] Fix Typo in SFTPOperator docstring (#4016)

  • [AIRFLOW-XXX] Fixing the issue in Documentation (#3998)

  • [AIRFLOW-XXX] Fix undocumented params in S3_hook

  • [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment (#3963)

  • [AIRFLOW-3070] Refine web UI authentication-related docs (#3863)

Airflow 1.10.1, 2018-11-13

New features

Improvements

Doc-only changes

Bug fixes

  • [AIRFLOW-839] docker_operator.py attempts to log status key without first checking existence

  • [AIRFLOW-1104] Concurrency check in scheduler should count queued tasks as well as running

  • [AIRFLOW-1163] Add support for x-forwarded-* headers to support access behind AWS ELB

  • [AIRFLOW-1195] Cleared tasks in SubDagOperator do not trigger Parent dag_runs

  • [AIRFLOW-1508] Skipped state not part of State.task_states

  • [AIRFLOW-1762] Use key_file in SSHHook.create_tunnel()

  • [AIRFLOW-1837] Differing start_dates on tasks not respected by scheduler.

  • [AIRFLOW-1874] Support standard SQL in Check, ValueCheck and IntervalCheck BigQuery operators

  • [AIRFLOW-1917] print() from Python operators end up with extra new line

  • [AIRFLOW-1970] Database cannot be initialized if an invalid fernet key is provided

  • [AIRFLOW-2145] Deadlock after clearing a running task

  • [AIRFLOW-2216] Cannot specify a profile for AWS Hook to load with s3 config file

  • [AIRFLOW-2574] initdb fails when mysql password contains percent sign

  • [AIRFLOW-2707] Error accessing log files from web UI

  • [AIRFLOW-2716] Replace new Python 3.7 keywords

  • [AIRFLOW-2744] RBAC app doesn't integrate plugins (blueprints etc)

  • [AIRFLOW-2772] BigQuery hook does not allow specifying both the partition field name and table name at the same time

  • [AIRFLOW-2778] Bad Import in collect_dag in DagBag

  • [AIRFLOW-2786] Variables view fails to render if a variable has an empty key

  • [AIRFLOW-2799] Filtering UI objects by datetime is broken

  • [AIRFLOW-2800] Remove airflow/ low-hanging linting errors

  • [AIRFLOW-2825] S3ToHiveTransfer operator may not may able to handle GZIP file with uppercase ext in S3

  • [AIRFLOW-2848] dag_id is missing in metadata table "job" for LocalTaskJob

  • [AIRFLOW-2860] DruidHook: time variable is not updated correctly when checking for timeout

  • [AIRFLOW-2865] Race condition between on_success_callback and LocalTaskJob's cleanup

  • [AIRFLOW-2893] Stuck dataflow job due to jobName mismatch.

  • [AIRFLOW-2895] Prevent scheduler from spamming heartbeats/logs

  • [AIRFLOW-2900] Code not visible for Packaged DAGs

  • [AIRFLOW-2905] Switch to regional dataflow job service.

  • [AIRFLOW-2907] Sendgrid - Attachments - ERROR - Object of type 'bytes' is not JSON serializable

  • [AIRFLOW-2938] Invalid 'extra' field in connection can raise an AttributeError when attempting to edit

  • [AIRFLOW-2979] Deprecated Celery Option not in Options list

  • [AIRFLOW-2981] TypeError in dataflow operators when using GCS jar or py_file

  • [AIRFLOW-2984] Cannot convert naive_datetime when task has a naive start_date/end_date

  • [AIRFLOW-2994] flatten_results in BigQueryOperator/BigQueryHook should default to None

  • [AIRFLOW-3002] ValueError in dataflow operators when using GCS jar or py_file

  • [AIRFLOW-3012] Email on sla miss is send only to first address on the list

  • [AIRFLOW-3046] ECS Operator mistakenly reports success when task is killed due to EC2 host termination

  • [AIRFLOW-3064] No output from airflow test due to default logging config

  • [AIRFLOW-3072] Only admin can view logs in RBAC UI

  • [AIRFLOW-3079] Improve initdb to support MSSQL Server

  • [AIRFLOW-3089] Google auth doesn't work under http

  • [AIRFLOW-3099] Errors raised when some blocs are missing in airflow.cfg

  • [AIRFLOW-3109] Default user permission should contain 'can_clear'

  • [AIRFLOW-3111] Confusing comments and instructions for log templates in UPDATING.md and default_airflow.cfg

  • [AIRFLOW-3124] Broken webserver debug mode (RBAC)

  • [AIRFLOW-3136] Scheduler Failing the Task retries run while processing Executor Events

  • [AIRFLOW-3138] Migration cc1e65623dc7 creates issues with postgres

  • [AIRFLOW-3161] Log Url link does not link to task instance logs in RBAC UI

  • [AIRFLOW-3162] HttpHook fails to parse URL when port is specified

  • [AIRFLOW-3183] Potential Bug in utils/dag_processing/DagFileProcessorManager.max_runs_reached()

  • [AIRFLOW-3203] Bugs in DockerOperator & Some operator test scripts were named incorrectly

  • [AIRFLOW-3238] Dags, removed from the filesystem, are not deactivated on initdb

  • [AIRFLOW-3268] Cannot pass SSL dictionary to mysql connection via URL

  • [AIRFLOW-3277] Invalid timezone transition handling for cron schedules

  • [AIRFLOW-3295] Require encryption in DaskExecutor when certificates are configured.

  • [AIRFLOW-3297] EmrStepSensor marks cancelled step as successful

Airflow 1.10.0, 2018-08-03

Airflow 1.9.0, 2018-01-02

Airflow 1.8.2, 2017-09-04

Airflow 1.8.1, 2017-05-09

Airflow 1.8.0, 2017-03-12

Airflow 1.7.2

Airflow 1.7.1, 2016-05-19

  • Fix : Don't treat premature tasks as could_not_run tasks

  • AIRFLOW-92 Avoid unneeded upstream_failed session closes apache/airflow#1485

  • Add logic to lock DB and avoid race condition

  • Handle queued tasks from multiple jobs/executors

  • AIRFLOW-52 Warn about overwriting tasks in a DAG

  • Fix corner case with joining processes/queues (#1473)

  • [AIRFLOW-52] Fix bottlenecks when working with many tasks

  • Add columns to toggle extra detail in the connection list view.

  • Log the number of errors when importing DAGs

  • Log dagbag metrics duplicate messages in queue into Statsd (#1406)

  • Clean up issue template (#1419)

  • correct missed arg.foreground to arg.daemon in cli

  • Reinstate imports for github enterprise auth

  • Use os.execvp instead of subprocess.Popen for the webserver

  • Revert from using "--foreground" to "--daemon"

  • Implement a Cloudant hook

  • Add missing args to airflow clear

  • Fixed a bug in the scheduler: num_runs used where runs intended

  • Add multiprocessing support to the scheduler

  • Partial fix to make sure next_run_date cannot be None

  • Support list/get/set variables in the CLI

  • Properly handle BigQuery booleans in BigQuery hook.

  • Added the ability to view XCom variables in webserver

  • Change DAG.tasks from a list to a dict

  • Add support for zipped dags

  • Stop creating hook on instantiating of S3 operator

  • User subquery in views to find running DAGs

  • Prevent DAGs from being reloaded on every scheduler iteration

  • Add a missing word to docs

  • Document the parameters of DbApiHook

  • added oracle operator with existing oracle hook

  • Add PyOpenSSL to Google Cloud gcp_api.

  • Remove executor error unit test

  • Add DAG inference, deferral, and context manager

  • Don't return error when writing files to Google cloud storage.

  • Fix GCS logging for gcp_api.

  • Ensure attr is in scope for error message

  • Fixing misnamed PULL_REQUEST_TEMPLATE

  • Extract non_pooled_task_slot_count into a configuration param

  • Update plugins.rst for clarity on the example (#1309)

  • Fix s3 logging issue

  • Add twitter feed example dag

  • GitHub ISSUE_TEMPLATE & PR_TEMPLATE cleanup

  • Reduce logger verbosity

  • Adding a PR Template

  • Add Lucid to list of users

  • Fix usage of asciiart

  • Use session instead of outdated main_session for are_dependencies_met

  • Fix celery flower port allocation

  • Fix for missing edit actions due to flask-admin upgrade

  • Fix typo in comment in prioritize_queued method

  • Add HipchatOperator

  • Include all example dags in backfill unit test

  • Make sure skipped jobs are actually skipped

  • Fixing a broken example dag, example_skip_dag.py

  • Add consistent and thorough signal handling and logging

  • Allow Operators to specify SKIPPED status internally

  • Update docstring for executor trap unit test

  • Doc: explain the usage of Jinja templating for templated params

  • Don't schedule runs before the DAG's start_date

  • Fix infinite retries with pools, with test

  • Fix handling of deadlocked jobs

  • Show only Airflow's deprecation warnings

  • Set DAG_FOLDER for unit tests

  • Missing comma in setup.py

  • Deprecate args and kwargs in BaseOperator

  • Raise deep scheduler exceptions to force a process restart.

  • Change inconsistent example DAG owners

  • Fix module path of send_email_smtp in configuration

  • added Gentner Lab to list of users

  • Increase timeout time for unit test

  • Fix reading strings from conf

  • CHORE - Remove Trailing Spaces

  • Fix SSHExecuteOperator crash when using a custom ssh port

  • Add note about Airflow components to template

  • Rewrite BackfillJob logic for clarity

  • Add unit tests

  • Fix miscellaneous bugs and clean up code

  • Fix logic for determining DagRun states

  • Make SchedulerJob not run EVERY queued task

  • Improve BackfillJob handling of queued/deadlocked tasks

  • Introduce ignore_depends_on_past parameters

  • Use Popen with CeleryExecutor

  • Rename user table to users to avoid conflict with postgres

  • Beware of negative pool slots.

  • Add support for calling_format from boto to S3_Hook

  • Add PyPI meta data and sync version number

  • Set dags_are_paused_at_creation's default value to True

  • Resurface S3Log class eaten by rebase/push -f

  • Add missing session.commit() at end of initdb

  • Validate that subdag tasks have pool slots available, and test

  • Use urlparse for remote GCS logs, and add unit tests

  • Make webserver worker timeout configurable

  • Fixed scheduling for @once interval

  • Use psycopg2's API for serializing postgres cell values

  • Make the provide_session decorator more robust

  • update link to Lyft's website

  • use num_shards instead of partitions to be consistent with batch ingestion

  • Add documentation links to README

  • Update docs with separate configuration section

  • Fix airflow.utils deprecation warning code being Python 3 incompatible

  • Extract dbapi cell serialization into its own method

  • Set Postgres autocommit as supported only if server version is < 7.4

  • Use refactored utils module in unit test imports

  • Add changelog for 1.7.0

  • Use LocalExecutor on Travis if possible

  • remove unused logging,errno, MiniHiveCluster imports

  • remove extra import of logging lib

  • Fix required gcloud version

  • Refactoring utils into smaller submodules

  • Properly measure number of task retry attempts

  • Add function to get configuration as dict, plus unit tests

  • Merge branch 'master' into hivemeta_sasl

  • Add wiki link to README.md

  • [hotfix] make email.Utils > email.utils for py3

  • Add the missing "Date" header to the warning e-mails

  • Add the missing "Date" header to the warning e-mails

  • Check name of SubDag class instead of class itself

  • [hotfix] removing repo_token from .coveralls.yml

  • Set the service_name in coverals.yml

  • Fixes #1223

  • Update Airflow docs for remote logging

  • Add unit tests for trapping Executor errors

  • Make sure Executors properly trap errors

  • Fix HttpOpSensorTest to use fake request session

  • Linting

  • Add an example on pool usage in the documentation

  • Add two methods to bigquery hook's base cursor: run_table_upsert, which adds a table or updates an existing table; and run_grant_dataset_view_access, which grants view access to a given dataset for a given table.

  • Tasks references upstream and downstream tasks using strings instead of references

  • Fix typos in models.py

  • Fix broken links in documentation

  • [hotfix] fixing the Scheduler CLI to make dag_id optional

  • Update link to Common Pitfalls wiki page in README

  • Allow disabling periodic committing when inserting rows with DbApiHook

  • added Glassdoor to "who uses airflow"

  • Fix typo preventing from launching webserver

  • Documentation badge

  • Fixing ISSUE_TEMPLATE name to include .md suffix

  • Adding an ISSUE_TEMPLATE to ensure that issues are adequately defined

  • Linting & debugging

  • Refactoring the CLI to be data-driven

  • Updating the Bug Reporting protocol in the Contributing.md file

  • Fixing the docs

  • Clean up references to old session

  • remove session reference

  • resolve conflict

  • clear xcom data when task instance starts

  • replace main_session with @provide_session

  • Add extras to installation.rst

  • Changes to Contributing to reflect more closely the current state of development.

  • Modifying README to link to the wiki committer list

  • docs: fixes a spelling mistake in default config

  • Set killMode to 'control-group' for webservice.service

  • Set KillMode to 'control-group' for worker.service

  • Linting

  • Fix WebHdfsSensor

  • Adding more licenses to pass checks

  • fixing landscape's config

  • [hotfix] typo that made it in master

  • [hotfix] fixing landscape requirement detection

  • Make testing on hive conditional

  • Merge remote-tracking branch 'upstream/master' into minicluster

  • Update README.md

  • Throwing in a few license to pass the build

  • Adding a reqs.txt for landscape.io

  • Pointing to a reqs file

  • Some linting

  • Adding a .landscape.yml file

  • badge for PyPI version

  • Add license and ignore for sql and csv

  • Use correct connection id

  • Use correct table name

  • Provide data for ci tests

  • New badge for showing staleness of reqs

  • Removing requirements.txt as it is uni-dimensional

  • Make it work on py3

  • Remove decode for logging

  • Also keep py2 compatible

  • More py3 fixes

  • Convert to bytes for py3 compat

  • Make sure to be py3 compatible

  • Use unicodecsv to make it py3 compatible

  • Replace tab with spaces Remove unused import

  • Merge remote-tracking branch 'upstream/master'

  • Support decimal types in MySQL to GCS

  • Make sure to write binary as string can be unicode

  • Ignore metastore

  • More impyla fixes

  • Test HivemetaStore if Python 2

  • Allow users to set hdfs_namenode_principal in HDFSHook config

  • Add tests for Hiveserver2 and fix some issues from impyla

  • Merge branch 'impyla' into minicluster

  • This patch allows for testing of hive operators and hooks. Sasl is used (NoSasl in connection string is not possible). Tests have been adjusted.

  • Treat SKIPPED and SUCCESS the same way when evaluating depends_on_past=True

  • fix bigquery hook

  • Version cap for gcp_api

  • Fix typo when returning VerticaHook

  • Adding fernet key to use it as part of stdout commands

  • Adding support for ssl parameters. (picking up from jthomas123)

  • More detail in error message.

  • Make sure paths don't conflict bc of trailing /

  • Change gcs_hook to self.hook

  • Refactor remote log read/write and add GCS support

  • Only use multipart upload in S3Hook if file is large enough

  • Merge branch 'airbnb/master'

  • Add GSSAPI SASL to HiveMetaStoreHook.

  • Add warning for deprecated setting

  • Use kerberos_service_name = 'hive' as standard instead of 'impala'.

  • Use GSSAPI instead of KERBEROS and provide backwards compatibility

  • ISSUE-1123 Use impyla instead of pyhs2

  • Set celery_executor to use queue name as exchange

Was this entry helpful?