AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030, #16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434, #18421, #18475, #18499, #18573, #18522, #18729, #18706, #18742, #18786, #18804)
AIP-40: Add Deferrable "Async" Tasks (#15389, #17564, #17565, #17601, #17745, #17747, #17748, #17875, #17876, #18129, #18210, #18214, #18552, #18728, #18414)
Add a Docker Taskflow decorator (#15330, #18739)
Add Airflow Standalone command (#15826)
Display alert messages on dashboard from local settings (#18284)
Advanced Params using json-schema (#17100)
Ability to test connections from UI or API (#15795, #18750)
Add Next Run to UI (#17732)
Add default weight rule configuration option (#18627)
Add a calendar field to choose the execution date of the DAG when triggering it (#16141)
Allow setting specific cwd for BashOperator (#17751)
Show import errors in DAG views (#17818)
Add pre/post execution hooks Experimental
Added table to view providers in Airflow ui under admin tab (#15385)
Adds secrets backend/logging/auth information to provider yaml (#17625)
Add date format filters to Jinja environment (#17451)
Introduce RESTARTING state (#16681)
Webserver: Unpause DAG on manual trigger (#16569)
API endpoint to create new user (#16609)
Add insert_args for support transfer replace (#15825)
Add recursive flag to glob in filesystem sensor (#16894)
Add conn to jinja template context (#16686)
Add default_args for TaskGroup (#16557)
Allow adding duplicate connections from UI (#15574)
Allow specifying multiple URLs via the CORS config option (#17941)
Implement API endpoint for DAG deletion (#17980)
Add DAG run endpoint for marking a dagrun success or failed(#17839)
Add support for kinit options [-f|-F] and [-a|-A] (#17816)
Queue support for DaskExecutor using Dask Worker Resources (#16829, #18720)
Make auto refresh interval configurable (#18107)

Improvements

Small improvements for Airflow UI (#18715, #18795)
Rename processor_poll_interval to scheduler_idle_sleep_time (#18704)
Check the allowed values for the logging level (#18651)
Fix error on triggering a dag that doesn't exist using dagrun_conf (#18655)
Add muldelete action to TaskInstanceModelView (#18438)
Avoid importing DAGs during clean DB installation (#18450)
Require can_edit on DAG privileges to modify TaskInstances and DagRuns (#16634)
Make Kubernetes job description fit on one log line (#18377)
Always draw borders if task instance state is null or undefined (#18033)
Inclusive Language (#18349)
Improved log handling for zombie tasks (#18277)
Adding Variable.update method and improving detection of variable key collisions (#18159)
Add note about params on trigger DAG page (#18166)
Change TaskInstance and TaskReschedule PK from execution_date to run_id (#17719)
Adding TaskGroup support in BaseOperator.chain() (#17456)
Advises the kernel to not cache log files generated by Airflow (#18054)
Allow filtering DAGS by tags in the REST API (#18090)
Optimize imports of Providers Manager (#18052)
Adds capability of Warnings for incompatible community providers (#18020)
Serialize the template_ext attribute to show it in UI (#17985)
Add robots.txt and X-Robots-Tag header (#17946)
Refactor BranchDayOfWeekOperator, DayOfWeekSensor (#17940)
Update error message to guide the user into self-help mostly (#17929)
Update to Celery 5 (#17397)
Avoid redirect loop for users with no permissions (#17838)
Improve graph view load time for dags with open groups (#17821)
Add links to provider's documentation (#17736)
Remove Marshmallow schema warnings (#17753)
Rename none_failed_or_skipped by none_failed_min_one_success trigger rule (#17683)
Remove [core] store_dag_code & use DB to get Dag Code (#16342)
Rename task_concurrency to max_active_tis_per_dag (#17708)
Import Hooks lazily individually in providers manager (#17682)
Adding support for multiple task-ids in the external task sensor (#17339)
Replace execution_date with run_id in airflow tasks run command (#16666)
Avoid endless redirect loop when user has no roles (#17613)
Make output from users cli command more consistent (#17642)
Open relative extra links in place (#17477)
Move worker_log_server_port option to the logging section (#17621)
Use gunicorn to serve logs generated by worker (#17591)
Improve validation of Group id (#17578)
Simplify 404 page (#17501)
Add XCom.clear so it's hookable in custom XCom backend (#17405)
Add deprecation notice for SubDagOperator (#17488)
Support DAGS folder being in different location on scheduler and runners (#16860)
Remove /dagrun/create and disable edit form generated by F.A.B (#17376)
Enable specifying dictionary paths in template_fields_renderers (#17321)
error early if virtualenv is missing (#15788)
Handle connection parameters added to Extra and custom fields (#17269)
Fix airflow celery stop to accept the pid file. (#17278)
Remove DAG refresh buttons (#17263)
Deprecate dummy trigger rule in favor of always (#17144)
Be verbose about failure to import airflow_local_settings (#17195)
Include exit code in AirflowException str when BashOperator fails. (#17151)
Adding EdgeModifier support for chain() (#17099)
Only allows supported field types to be used in custom connections (#17194)
Secrets backend failover (#16404)
Warn on Webserver when using SQLite or SequentialExecutor (#17133)
Extend init_containers defined in pod_override (#17537)
Client-side filter dag dependencies (#16253)
Improve executor validation in CLI (#17071)
Prevent running airflow db init/upgrade migrations and setup in parallel. (#17078)
Update chain() and cross_downstream() to support XComArgs (#16732)
Improve graph view refresh (#16696)
When a task instance fails with exception, log it (#16805)
Set process title for serve-logs and LocalExecutor (#16644)
Rename test_cycle to check_cycle (#16617)
Set process title for Worker when using LocalExecutor (#16623)
Add schema as DbApiHook instance attribute (#16521, #17423)
Improve compatibility with MSSQL (#9973)
Add transparency for unsupported connection type (#16220)
Call resource based fab methods (#16190)
Format more dates with timezone (#16129)
Replace deprecated dag.sub_dag with dag.partial_subset (#16179)
Treat AirflowSensorTimeout as immediate failure without retrying (#12058)
Marking success/failed automatically clears failed downstream tasks (#13037)
Add close/open indicator for import dag errors (#16073)
Add collapsible import errors (#16072)
Always return a response in TI's action_clear view (#15980)
Add cli command to delete user by email (#15873)
Use resource and action names for FAB permissions (#16410)
Rename DAG concurrency ([core] dag_concurrency) settings for easier understanding (#16267, #18730)
Calendar UI improvements (#16226)
Refactor: SKIPPED should not be logged again as SUCCESS (#14822)
Remove version limits for dnspython (#18046, #18162)
Accept custom run ID in TriggerDagRunOperator (#18788)

Bug Fixes

Make REST API patch user endpoint work the same way as the UI (#18757)
Properly set start_date for cleared tasks (#18708)
Ensure task_instance exists before running update on its state(REST API) (#18642)
Make AirflowDateTimePickerWidget a required field (#18602)
Retry deadlocked transactions on deleting old rendered task fields (#18616)
Fix retry_exponential_backoff divide by zero error when retry delay is zero (#17003)
Improve how UI handles datetimes (#18611, #18700)
Bugfix: dag_bag.get_dag should return None, not raise exception (#18554)
Only show the task modal if it is a valid instance (#18570)
Fix accessing rendered {{ task.x }} attributes from within templates (#18516)
Add missing email type of connection (#18502)
Don't use flash for "same-page" UI messages. (#18462)
Fix task group tooltip (#18406)
Properly fix dagrun update state endpoint (#18370)
Properly handle ti state difference between executor and scheduler (#17819)
Fix stuck "queued" tasks in KubernetesExecutor (#18152)
Don't permanently add zip DAGs to sys.path (#18384)
Fix random deadlocks in MSSQL database (#18362)
Deactivating DAGs which have been removed from files (#17121)
When syncing dags to db remove dag_tag rows that are now unused (#8231)
Graceful scheduler shutdown on error (#18092)
Fix mini scheduler not respecting wait_for_downstream dep (#18338)
Pass exception to run_finished_callback for Debug Executor (#17983)
Make XCom.get_one return full, not abbreviated values (#18274)
Use try/except when closing temporary file in task_runner (#18269)
show next run if not none (#18273)
Fix DB session handling in XCom.set (#18240)
Sort adopted tasks in _check_for_stalled_adopted_tasks method (#18208)
Fix external_executor_id not being set for manually run jobs (#17207)
Fix deleting of zipped Dags in Serialized Dag Table (#18243)
Return explicit error on user-add for duplicated email (#18224)
Remove loading dots even when last run data is empty (#18230)
Swap dag import error dropdown icons (#18207)
Automatically create section when migrating config (#16814)
Set encoding to utf-8 by default while reading task logs (#17965)
Set task state to failed when pod is DELETED while running (#18095)
Apply parent dag permissions to subdags (#18160)
Change id collation for MySQL to case-sensitive (#18072)
Logs task launch exception in StandardTaskRunner (#17967)
Applied permissions to self._error_file (#15947)
Hide variable import form if user lacks permission (#18000)
Fix blank dag dependencies view (#17990)
Add missing menu access for dag dependencies and configurations pages (#17450)
Fix passing Jinja templates in DateTimeSensor (#17959)
Fixing bug which restricted the visibility of ImportErrors (#17924)
Fix grammar in traceback.html (#17942)
Fix DagRunState enum query for MySQLdb driver (#17886)
Fixed button size in "Actions" group. (#17902)
Fix log links on graph TI modal (#17862)
Only show import errors for DAGs a user can access (#17835)
Show all import_errors from zip files (#17759)
fix EXTRA_LOGGER_NAMES param and related docs (#17808)
Use one interpreter for Airflow and gunicorn (#17805)
Fix: Mysql 5.7 id utf8mb3 (#14535)
Fix dag_processing.last_duration metric random holes (#17769)
Automatically use utf8mb3_general_ci collation for MySQL (#17729)
Fix wrong query on running tis (#17631)
Add root to tree refresh url (#17633)
Do not delete running DAG from the UI (#17630)
Rescue if a DagRun's DAG was removed from db (#17544)
fix: filter condition of TaskInstance does not work #17535 (#17548)
Regression on pid reset to allow task start after heartbeat (#17333)
Dont use TaskInstance in CeleryExecutor.trigger_tasks (#16248)
Remove locks for upgrades in MSSQL (#17213)
Create virtualenv via python call (#17156)
Ensure a DAG is acyclic when running DAG.cli() (#17105)
Translate non-ascii characters (#17057)
Change the logic of None comparison in model_list template (#16893)
Have UI and POST /task_instances_state API endpoint have same behaviour (#16539)
ensure task is skipped if missing sla (#16719)
Fix direct use of cached_property module (#16710)
Fix TI success confirm page (#16650)
Cattrs 1.7.0 released by the end of May 2021 break lineage usage (#16173)
Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)
Modify return value check in python virtualenv jinja template (#16049)
Fix dag dependency search (#15924)
Make custom JSON encoder support Decimal (#16383)
Bugfix: Allow clearing tasks with just dag_id and empty subdir (#16513)
Convert port value to a number before calling test connection (#16497)
Handle missing/null serialized DAG dependencies (#16393)
Correctly set dag.fileloc when using the @dag decorator (#16384)
Fix TI success/failure links (#16233)
Correctly implement autocomplete early return in airflow/www/views.py (#15940)
Backport fix to allow pickling of Loggers to Python 3.6 (#18798)
Fix bug that Backfill job fail to run when there are tasks run into reschedule state (#17305, #18806)

Doc only changes

Update dagbag_size documentation (#18824)
Update documentation about bundle extras (#18828)
Fix wrong Postgres search_path set up instructions (#17600)
Remove AIRFLOW_GID from Docker images (#18747)
Improve error message for BranchPythonOperator when no task_id to follow (#18471)
Improve guidance to users telling them what to do on import timeout (#18478)
Explain scheduler fine-tuning better (#18356)
Added example JSON for airflow pools import (#18376)
Add sla_miss_callback section to the documentation (#18305)
Explain sentry default environment variable for subprocess hook (#18346)
Refactor installation pages (#18282)
Improves installing from sources pages for all components (#18251)
Doc: Use closer.lua script for downloading sources (#18179)
Improves quick-start docker-compose warnings and documentation (#18164)
Production-level support for MSSQL (#18382)
Update non-working example in documentation (#18067)
Improve the description of how to handle dynamic task generation (#17963)
Remove default_args pattern + added get_current_context() use for Core Airflow example DAGs (#16866)
Update max_tis_per_query to better render on the webpage (#17971)
Adds Github Oauth example with team based authorization (#17896)
Update docker.rst (#17882)
Example xcom update (#17749)
Add doc warning about connections added via envvars (#17915)
fix wrong documents around upgrade-check.rst (#17903)
Add Brent to Committers list (#17873)
Improves documentation about modules management (#17757)
Remove deprecated metrics from metrics.rst (#17772)
Make sure "production-readiness" of docker-compose is well explained (#17731)
Doc: Update Upgrade to v2 docs with Airflow 1.10.x EOL dates (#17710)
Doc: Replace deprecated param from docstrings (#17709)
docs(dagowner): describe dag owner more carefully (#17699)
docs(impersonation): update note so avoid misintrepretation (#17701)
Docs: Make DAG.is_active read-only in API (#17667)
Improve cross-links to operators and hooks references (#17622)
Update documentation regarding Python 3.9 support (#17611)
Fix MySQL database character set instruction (#17603)
Document overriding XCom.clear for data lifecycle management (#17589)
Path correction in docs for airflow core (#17567)
docs(celery): reworded, add actual multiple queues example (#17541)
Doc: Add FAQ to speed up parsing with tons of dag files (#17519)
Improve image building documentation for new users (#17409)
Doc: Strip unnecessary arguments from MariaDB JIRA URL (#17296)
Update warning about MariaDB and multiple schedulers (#17287)
Doc: Recommend using same configs on all Airflow components (#17146)
Move docs about masking to a new page (#17007)
Docs: Suggest use of Env vars instead of Airflow Vars in best practises doc (#16926)
Docs: Better description for pod_template_file (#16861)
Add Aneesh Joseph as Airflow Committer (#16835)
Docs: Added new pipeline example for the tutorial docs (#16548)
Remove upstart from docs (#16672)
Add new committers: Jed and TP (#16671)
Docs: Fix flask-ouathlib to flask-oauthlib in Upgrading docs (#16320)
Docs: Fix creating a connection docs (#16312)
Docs: Fix url for Elasticsearch (#16275)
Small improvements for README.md files (#16244)
Fix docs for dag_concurrency (#16177)
Check syntatic correctness for code-snippets (#16005)
Improvements for Docker Image docs (#14843)
Add proper link for wheel packages in docs. (#15999)
Add Docs for default_pool slots (#15997)
Add memory usage warning in quick-start documentation (#15967)
Update example KubernetesExecutor git-sync pod template file (#15904)
Docs: Fix Taskflow API docs (#16574)
Added new pipeline example for the tutorial docs (#16084)
Updating the DAG docstring to include render_template_as_native_obj (#16534)
Update docs on setting up SMTP (#16523)
Docs: Fix API verb from POST to PATCH (#16511)

Misc/Internal

Renaming variables to be consistent with code logic (#18685)
Simplify strings previously split across lines (#18679)
fix exception string of BranchPythonOperator (#18623)
Add multiple roles when creating users (#18617)
Move FABs base Security Manager into Airflow. (#16647)
Remove unnecessary css state colors (#18461)
Update boto3 to <1.19 (#18389)
Improve coverage for airflow.security.kerberos module (#18258)
Fix kinesis test (#18337)
Fix provider test acessing importlib-resources (#18228)
Silence warnings in tests from using SubDagOperator (#18275)
Fix usage of range(len()) to enumerate (#18174)
Test coverage on the autocomplete view (#15943)
Update version added fields in airflow/config_templates/config.yml (#18128)
Add "packaging" to core requirements (#18122)
Adds LoggingMixins to BaseTrigger (#18106)
Limit colorlog version (6.x is incompatible) (#18099)
Fix building docs in main builds (#18035)
Remove upper-limit on tenacity (#17593)
Remove redundant numpy dependency (#17594)
Bump mysql-connector-python to latest version (#17596)
Make pandas an optional core dependency (#17575)
Add more typing to airflow.utils.helpers (#15582)
Chore: Some code cleanup in airflow/utils/db.py (#17090)
Refactor: Remove processor_factory from DAG processing (#16659)
Remove AbstractDagFileProcessorProcess from dag processing (#16816)
Update TaskGroup typing (#16811)
Update click to 8.x (#16779)
Remove remaining Pylint disables (#16760)
Remove duplicated try, there is already a try in create_session (#16701)
Removes pylint from our toolchain (#16682)
Refactor usage of unneeded function call (#16653)
Add type annotations to setup.py (#16658)
Remove SQLAlchemy <1.4 constraint (#16630) (Note: our dependencies still have a requirement on <1.4)
Refactor dag.clear method (#16086)
Use DAG_ACTIONS constant (#16232)
Use updated _get_all_non_dag_permissions method (#16317)
Add updated-name wrappers for built-in FAB methods (#16077)
Pins docutils to <0.17 until breaking behaviour is fixed (#16133)
Remove TaskInstance.log_filepath attribute (#15217)
Removes unnecessary function call in airflow/www/app.py (#15956)
Move plyvel to google provider extra (#15812)
Update permission migrations to use new naming scheme (#16400)
Use resource and action names for FAB (#16380)
Swap out calls to find_permission_view_menu for get_permission wrapper (#16377)
Fix deprecated default for fab_logging_level to WARNING (#18783)
Allow running tasks from UI when using CeleryKubernetesExecutor (#18441)