Airflow: 2.4.0 Release

Release date:
September 15, 2022
Previous version:
2.4.0rc1 (released September 15, 2022)
Magnitude:
0 Diff Delta
Contributors:
0 total committers
Data confidence:
Commits:

Top Contributors in 2.4.0

Could not determine top contributors for this release.

Directory Browser for 2.4.0

We haven't yet finished calculating and confirming the files and directories changed in this release. Please check back soon.

Release Notes Published

New Features

  • Add Data-aware Scheduling (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-48+milestone%3A%22Airflow+2.4.0%22)
  • Add @task.short_circuit TaskFlow decorator (#25752)
  • Make execution_date_or_run_id optional in tasks test command (#26114)
  • Automatically register DAGs that are used in a context manager (#23592, #26398)
  • Add option of sending DAG parser logs to stdout. (#25754)
  • Support multiple DagProcessors parsing files from different locations. (#25935)
  • Implement ExternalPythonOperator (#25780)
  • Make execution_date optional for command dags test (#26111)
  • Implement expand_kwargs() against a literal list (#25925)
  • Add trigger rule tooltip (#26043)
  • Add conf parameter to CLI for airflow dags test (#25900)
  • Include scheduled slots in pools view (#26006)
  • Add output property to MappedOperator (#25604)
  • Add roles delete command to cli (#25854)
  • Add Airflow specific warning classes (#25799)
  • Add support for TaskGroup in ExternalTaskSensor (#24902)
  • Add @task.kubernetes taskflow decorator (#25663)
  • Add a way to import Airflow without side-effects (#25832)
  • Let timetables control generated run_ids. (#25795)
  • Allow per-timetable ordering override in grid view (#25633)
  • Grid logs for mapped instances (#25610, #25621, #25611)
  • Consolidate to one schedule param (#25410)
  • DAG regex flag in backfill command (#23870)
  • Adding support for owner links in the Dags view UI (#25280)
  • Ability to clear a specific DAG Run's task instances via REST API (#23516)
  • Possibility to document DAG with a separate markdown file (#25509)
  • Add parsing context to DAG Parsing (#25161)
  • Implement CronTriggerTimetable (#23662)
  • Add option to mask sensitive data in UI configuration page (#25346)
  • Create new databases from the ORM (#24156)
  • Implement XComArg.zip(*xcom_args) (#25176)
  • Introduce sla_miss metric (#23402)
  • Implement map() semantic (#25085)
  • Add override method to TaskGroupDecorator (#25160)
  • Implement expand_kwargs() (#24989)
  • Add parameter to turn off SQL query logging (#24570)
  • Add DagWarning model, and a check for missing pools (#23317)
  • Add Task Logs to Grid details panel (#24249)
  • Added small health check server and endpoint in scheduler(#23905)
  • Add built-in External Link for ExternalTaskMarker operator (#23964)
  • Add default task retry delay config (#23861)
  • Add clear DagRun endpoint. (#23451)
  • Add support for timezone as string in cron interval timetable (#23279)
  • Add auto-refresh to dags home page (#22900, #24770)

Improvements

  • Add more weekday operator and sensor examples #26071 (#26098)
  • Add subdir parameter to dags reserialize command (#26170)
  • Update zombie message to be more descriptive (#26141)
  • Only send an SlaCallbackRequest if the DAG is scheduled (#26089)
  • Promote Operator.output more (#25617)
  • Upgrade API files to typescript (#25098)
  • Less hacky double-rendering prevention in mapped task (#25924)
  • Improve Audit log (#25856)
  • Remove mapped operator validation code (#25870)
  • More DAG(schedule=...) improvements (#25648)
  • Reduce operator_name dupe in serialized JSON (#25819)
  • Make grid view group/mapped summary UI more consistent (#25723)
  • Remove useless statement in task_group_to_grid (#25654)
  • Add optional data interval to CronTriggerTimetable (#25503)
  • Remove unused code in /grid endpoint (#25481)
  • Add and document description fields (#25370)
  • Improve Airflow logging for operator Jinja template processing (#25452)
  • Update core example DAGs to use @task.branch decorator (#25242)
  • Update DAG audit_log route (#25415)
  • Change stdout and stderr access mode to append in commands (#25253)
  • Remove getTasks from Grid view (#25359)
  • Improve taskflow type hints with ParamSpec (#25173)
  • Use tables in grid details panes (#25258)
  • Explicitly list @dag arguments (#25044)
  • More typing in SchedulerJob and TaskInstance (#24912)
  • Patch getfqdn with more resilient version (#24981)
  • Replace all NBSP characters by whitespaces (#24797)
  • Re-serialize all DAGs on airflow db upgrade (#24518)
  • Rework contract of try_adopt_task_instances method (#23188)
  • Make expand() error vague so it's not misleading (#24018)
  • Add enum validation for [webserver]analytics_tool (#24032)
  • Add dttm searchable field in audit log (#23794)
  • Allow more parameters to be piped through via execute_in_subprocess (#23286)
  • Use func.count to count rows (#23657)
  • Remove stale serialized dags (#22917)
  • AIP45 Remove dag parsing in airflow run local (#21877)
  • Add support for queued state in DagRun update endpoint. (#23481)
  • Add fields to dagrun endpoint (#23440)
  • Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

  • Have consistent types between the ORM and the migration files (#24044, #25869)
  • Disallow any dag tags longer than 100 char (#25196)
  • Add the dag_id to AirflowDagCycleException message (#26204)
  • Properly build URL to retrieve logs independently from system (#26337)
  • For worker log servers only bind to IPV6 when dual stack is available (#26222)
  • Fix TaskInstance.task not defined before handle_failure (#26040)
  • Undo secrets backend config caching (#26223)
  • Fix faulty executor config serialization logic (#26191)
  • Show DAGs and Datasets menu links based on role permission (#26183)
  • Allow setting TaskGroup tooltip via function docstring (#26028)
  • Fix RecursionError on graph view of a DAG with many tasks (#26175)
  • Fix backfill occasional deadlocking (#26161)
  • Fix DagRun.start_date not set during backfill with --reset-dagruns True (#26135)
  • Use label instead of id for dynamic task labels in graph (#26108)
  • Don't fail DagRun when leaf mapped_task is SKIPPED (#25995)
  • Add group prefix to decorated mapped task (#26081)
  • Fix UI flash when triggering with dup logical date (#26094)
  • Fix Make items nullable for TaskInstance related endpoints to avoid API errors (#26076)
  • Fix BranchDateTimeOperator to be timezone-awreness-insensitive (#25944)
  • Fix legacy timetable schedule interval params (#25999)
  • Fix response schema for list-mapped-task-instance (#25965)
  • Properly check the existence of missing mapped TIs (#25788)
  • Fix broken auto-refresh on grid view (#25950)
  • Use per-timetable ordering in grid UI (#25880)
  • Rewrite recursion when parsing DAG into iteration (#25898)
  • Find cross-group tasks in iter_mapped_dependants (#25793)
  • Fail task if mapping upstream fails (#25757)
  • Support / in variable get endpoint (#25774)
  • Use cfg default_wrap value for grid logs (#25731)
  • Add origin request args when triggering a run (#25729)
  • Operator name separate from class (#22834)
  • Fix incorrect data interval alignment due to assumption on input time alignment (#22658)
  • Return None if an XComArg fails to resolve (#25661)
  • Correct json arg help in airflow variables set command (#25726)
  • Added MySQL index hint to use ti_state on find_zombies query (#25725)
  • Only excluded actually expanded fields from render (#25599)
  • Grid, fix toast for axios errors (#25703)
  • Fix UI redirect (#26409)
  • Require dag_id arg for dags list-runs (#26357)
  • Check for queued states for dags auto-refresh (#25695)
  • Fix upgrade code for the dag_owner_attributes table (#25579)
  • Add map index to task logs api (#25568)
  • Ensure that zombie tasks for dags with errors get cleaned up (#25550)
  • Make extra link work in UI (#25500)
  • Sync up plugin API schema and definition (#25524)
  • First/last names can be empty (#25476)
  • Refactor DAG pages to be consistent (#25402)
  • Check expand_kwargs() input type before unmapping (#25355)
  • Filter XCOM by key when calculating map lengths (#24530)
  • Fix ExternalTaskSensor not working with dynamic task (#25215)
  • Added exception catching to send default email if template file raises any exception (#24943)
  • Bring MappedOperator members in sync with BaseOperator (#24034)

Misc/Internal

  • Add automatically generated ERD schema for the MetaData DB (#26217)
  • Mark serialization functions as internal (#26193)
  • Remove remaining deprecated classes and replace them with PEP562 (#26167)
  • Move dag_edges and task_group_to_dict to corresponding util modules (#26212)
  • Lazily import many modules to improve import speed (#24486, #26239)
  • FIX Incorrect typing information (#26077)
  • Add missing contrib classes to deprecated dictionaries (#26179)
  • Re-configure/connect the ORM after forking to run a DAG processor (#26216)
  • Remove cattrs from lineage processing. (#26134)
  • Removed deprecated contrib files and replace them with PEP-562 getattr (#26153)
  • Make BaseSerialization.serialize "public" to other classes. (#26142)
  • Change the template to use human readable task_instance description (#25960)
  • Bump moment-timezone from 0.5.34 to 0.5.35 in /airflow/www (#26080)
  • Fix Flask deprecation warning (#25753)
  • Add CamelCase to generated operations types (#25887)
  • Fix migration issues and tighten the CI upgrade/downgrade test (#25869)
  • Fix type annotations in SkipMixin (#25864)
  • Workaround setuptools editable packages path issue (#25848)
  • Bump undici from 5.8.0 to 5.9.1 in /airflow/www (#25801)
  • Add custom_operator_name attr to _BranchPythonDecoratedOperator (#25783)
  • Clarify filename_template deprecation message (#25749)
  • Use ParamSpec to replace ... in Callable (#25658)
  • Remove deprecated modules (#25543)
  • Documentation on task mapping additions (#24489)
  • Remove Smart Sensors (#25507)
  • Fix elasticsearch test config to avoid warning on deprecated template (#25520)
  • Bump terser from 4.8.0 to 4.8.1 in /airflow/ui (#25178)
  • Generate typescript types from rest API docs (#25123)
  • Upgrade utils files to typescript (#25089)
  • Upgrade remaining context file to typescript. (#25096)
  • Migrate files to ts (#25267)
  • Upgrade grid Table component to ts. (#25074)
  • Skip mapping against mapped ti if it returns None (#25047)
  • Refactor js file structure (#25003)
  • Move mapped kwargs introspection to separate type (#24971)
  • Only assert stuff for mypy when type checking (#24937)
  • Bump moment from 2.29.3 to 2.29.4 in /airflow/www (#24885)
  • Remove "bad characters" from our codebase (#24841)
  • Remove xcom_push flag from BashOperator (#24824)
  • Move Flask hook registration to end of file (#24776)
  • Upgrade more javascript files to typescript (#24715)
  • Clean up task decorator type hints and docstrings (#24667)
  • Preserve original order of providers' connection extra fields in UI (#24425)
  • Rename charts.css to chart.css (#24531)
  • Rename grid.css to chart.css (#24529)
  • Misc: create new process group by set_new_process_group utility (#24371)
  • Airflow UI fix Prototype Pollution (#24201)
  • Bump moto version (#24222)
  • Remove unused [github_enterprise] from ref docs (#24033)
  • Clean up f-strings in logging calls (#23597)
  • Add limit for JPype1 (#23847)
  • Simply json responses (#25518)
  • Add min attrs version (#26408)

Doc only changes

  • Add url prefix setting for Celery Flower (#25986)
  • Updating deprecated configuration in examples (#26037)
  • Fix wrong link for taskflow tutorial (#26007)
  • Reorganize tutorials into a section (#25890)
  • Fix concept doc for dynamic task map (#26002)
  • Update code examples from "classic" operators to taskflow (#25845, #25657)
  • Add instructions on manually fixing MySQL Charset problems (#25938)
  • Prefer the local Quick Start in docs (#25888)
  • Fix broken link to Trigger Rules (#25840)
  • Improve docker documentation (#25735)
  • Correctly link to Dag parsing context in docs (#25722)
  • Add note on task_instance_mutation_hook usage (#25607)
  • Note that TaskFlow API automatically passes data between tasks (#25577)
  • Update DAG run to clarify when a DAG actually runs (#25290)
  • Update tutorial docs to include a definition of operators (#25012)
  • Rewrite the Airflow documentation home page (#24795)
  • Fix task-generated mapping example (#23424)
  • Add note on subtle logical date change in 2.2.0 (#24413)
  • Add missing import in best-practices code example (#25391)