Airflow: 2.3.0 Release

Release date:
April 27, 2022
Previous version:
2.3.0rc2 (released April 27, 2022)
Magnitude:
0 Diff Delta
Contributors:
0 total committers
Data confidence:
Commits:

Top Contributors in 2.3.0

Could not determine top contributors for this release.

Directory Browser for 2.3.0

We haven't yet finished calculating and confirming the files and directories changed in this release. Please check back soon.

Release Notes Published

New Features

  • Add dynamic task mapping (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-42+milestone%3A%22Airflow+2.3.0%22)
  • New Grid View replaces Tree View (#18675)
  • Templated requirements.txt in Python Operators (#17349)
  • Allow reuse of decorated tasks (#22941)
  • Move the database configuration to a new section (#22284)
  • Add SmoothOperator (#22813)
  • Make operator's execution_timeout configurable (#22389)
  • Events Timetable (#22332)
  • Support dag serialization with custom ti_deps rules (#22698)
  • Support log download in task log view (#22804)
  • support for continue backfill on failures (#22697)
  • Add dag-processor cli command (#22305)
  • Add possibility to create users in LDAP mode (#22619)
  • Add ignore_first_depends_on_past for scheduled jobs (#22491)
  • Update base sensor operator to support XCOM return value (#20656)
  • Add an option for run id in the ui trigger screen (#21851)
  • Enable JSON serialization for connections (#19857)
  • Add REST API endpoint for bulk update of DAGs (#19758)
  • Add queue button to click-on-DagRun interface. (#21555)
  • Add list-import-errors to airflow dags command (#22084)
  • Store callbacks in database if standalone_dag_processor config is True. (#21731)
  • Add LocalKubernetesExecutor (#19729)
  • Add celery.task_timeout_error metric (#21602)
  • Airflow db downgrade cli command (#21596)
  • Add ALL_SKIPPED trigger rule (#21662)
  • Add db clean CLI command for purging old data (#20838)
  • Add celery_logging_level (#21506)
  • Support different timeout value for dag file parsing (#21501)
  • Support generating SQL script for upgrades (#20962)
  • Add option to compress Serialized dag data (#21332)
  • Branch python operator decorator (#20860)
  • Add Audit Log View to Dag View (#20733)
  • Add missing StatsD metric for failing SLA Callback notification (#20924)
  • Add ShortCircuitOperator configurability for respecting downstream trigger rules (#20044)
  • Allow using Markup in page title in Webserver (#20888)
  • Add Listener Plugin API that tracks TaskInstance state changes (#20443)
  • Add context var hook to inject more env vars (#20361)
  • Add a button to set all tasks to skipped (#20455)
  • Cleanup pending pods (#20438)
  • Add config to warn public deployment exposure in UI (#18557)
  • Log filename template records (#20165)
  • Added windows extensions (#16110)
  • Showing approximate time until next dag_run in Airflow (#20273)
  • Extend config window on UI (#20052)
  • Add show dag dependencies feature to CLI (#19985)
  • Add cli command for 'airflow dags reserialize` (#19471)
  • Add missing description field to Pool schema(REST API) (#19841)
  • Introduce DagRun action to change state to queued. (#19353)
  • Add DAG run details page (#19705)
  • Add role export/import to cli tools (#18916)
  • Adding dag_id_pattern parameter to the /dags endpoint (#18924)

Improvements

  • Show schedule_interval/timetable description in UI (#16931)
  • Added column duration to DAG runs view (#19482)
  • Enable use of custom conn extra fields without prefix (#22607)
  • Initialize finished counter at zero (#23080)
  • Improve logging of optional provider features messages (#23037)
  • Meaningful error message in resolve_template_files (#23027)
  • Update ImportError items instead of deleting and recreating them (#22928)
  • Add option --skip-init to db reset command (#22989)
  • Support importing connections from files with ".yml" extension (#22872)
  • Support glob syntax in .airflowignore files (#21392) (#22051)
  • Hide pagination when data is a single page (#22963)
  • Support for sorting DAGs in the web UI (#22671)
  • Speed up has_access decorator by ~200ms (#22858)
  • Add XComArg to lazy-imported list of Airflow module (#22862)
  • Add more fields to REST API dags/dag_id/details endpoint (#22756)
  • Don't show irrelevant/duplicated/"internal" Task attrs in UI (#22812)
  • No need to load whole ti in current_state (#22764)
  • Pickle dag exception string fix (#22760)
  • Better verification of Localexecutor's parallelism option (#22711)
  • log backfill exceptions to sentry (#22704)
  • retry commit on MySQL deadlocks during backfill (#22696)
  • Add more fields to REST API get DAG(dags/dag_id) endpoint (#22637)
  • Use timetable to generate planned days for current year (#22055)
  • Disable connection pool for celery worker (#22493)
  • Make date picker label visible in trigger dag view (#22379)
  • Expose try_number in airflow vars (#22297)
  • Add generic connection type (#22310)
  • Add a few more fields to the taskinstance finished log message (#22262)
  • Pause auto-refresh if scheduler isn't running (#22151)
  • Show DagModel details. (#21868)
  • Add pip_install_options to PythonVirtualenvOperator (#22158)
  • Show import error for airflow dags list CLI command (#21991)
  • Pause auto-refresh when page is hidden (#21904)
  • Default args type check (#21809)
  • Enhance magic methods on XComArg for UX (#21882)
  • py files don't have to be checked is_zipfiles in refresh_dag (#21926)
  • Fix TaskDecorator type hints (#21881)
  • Add 'Show record' option for variables (#21342)
  • Use DB where possible for quicker airflow dag subcommands (#21793)
  • REST API: add rendered fields in task instance. (#21741)
  • Change the default auth backend to session (#21640)
  • Don't check if py DAG files are zipped during parsing (#21538)
  • Switch XCom implementation to use run_id (#20975)
  • Action log on Browse Views (#21569)
  • Implement multiple API auth backends (#21472)
  • Change logging level details of connection info in get_connection() (#21162)
  • Support mssql in airflow db shell (#21511)
  • Support config worker_enable_remote_control for celery (#21507)
  • Log memory usage in CgroupTaskRunner (#21481)
  • Modernize DAG-related URL routes and rename "tree" to "grid" (#20730)
  • Move Zombie detection to SchedulerJob (#21181)
  • Improve speed to run airflow by 6x (#21438)
  • Add more SQL template fields renderers (#21237)
  • Simplify fab has access lookup (#19294)
  • Log context only for default method (#21244)
  • Log trigger status only if at least one is running (#21191)
  • Add optional features in providers. (#21074)
  • Better multiple_outputs inferral for @task.python (#20800)
  • Improve handling of string type and non-attribute template_fields (#21054)
  • Remove un-needed deps/version requirements (#20979)
  • Correctly specify overloads for TaskFlow API for type-hinting (#20933)
  • Introduce notification_sent to SlaMiss view (#20923)
  • Rewrite the task decorator as a composition (#20868)
  • Add "Greater/Smaller than or Equal" to filters in the browse views (#20602) (#20798)
  • Rewrite DAG run retrieval in task command (#20737)
  • Speed up creation of DagRun for large DAGs (5k+ tasks) by 25-130% (#20722)
  • Make native environment Airflow-flavored like sandbox (#20704)
  • Better error when param value has unexpected type (#20648)
  • Add filter by state in DagRun REST API (List Dag Runs) (#20485)
  • Prevent exponential memory growth in Tasks with custom logging handler (#20541)
  • Set default logger in logging Mixin (#20355)
  • Reduce deprecation warnings from www (#20378)
  • Add hour and minute to time format on x-axis of all charts using nvd3.lineChart (#20002)
  • Add specific warning when Task asks for more slots than pool defined with (#20178)
  • UI: Update duration column for better human readability (#20112)
  • Use Viewer role as example public role (#19215)
  • Properly implement DAG param dict copying (#20216)
  • ShortCircuitOperator push XCom by returning python_callable result (#20071)
  • Add clear logging to tasks killed due to a Dagrun timeout (#19950)
  • Change log level for Zombie detection messages (#20204)
  • Better confirmation prompts (#20183)
  • Only execute TIs of running DagRuns (#20182)
  • Check and run migration in commands if necessary (#18439)
  • Log only when Zombies exists (#20118)
  • Increase length of the email and username (#19932)
  • Add more filtering options for TI's in the UI (#19910)
  • Dynamically enable "Test Connection" button by connection type (#19792)
  • Avoid littering postgres server logs with "could not obtain lock" with HA schedulers (#19842)
  • Renamed Connection.get_hook parameter to make it the same as in SqlSensor and SqlOperator. (#19849)
  • Add hook_params in SqlSensor using the latest changes from PR #18718. (#18431)
  • Speed up webserver boot time by delaying provider initialization (#19709)
  • Configurable logging of XCOM value in PythonOperator (#19378)
  • Minimize production js files (#19658)
  • Add hook_params in BaseSqlOperator (#18718)
  • Add missing "end_date" to hash components (#19281)
  • More friendly output of the airflow plugins command + add timetables (#19298)
  • Add sensor default timeout config (#19119)
  • Update taskinstance REST API schema to include dag_run_id field (#19105)
  • Adding feature in bash operator to append the user defined env variable to system env variable (#18944)
  • Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161)

Bug Fixes

  • Use inherited 'trigger_tasks' method (#23016)
  • In DAG dependency detector, use class type instead of class name (#21706)
  • Fix tasks being wrongly skipped by schedule_after_task_execution (#23181)
  • Fix X-Frame enabled behaviour (#23222)
  • Allow extra to be nullable in connection payload as per schema(REST API). (#23183)
  • Fix dag_id extraction for dag level access checks in web ui (#23015)
  • Fix timezone display for logs on UI (#23075)
  • Include message in graph errors (#23021)
  • Change trigger dropdown left position (#23013)
  • Don't add planned tasks for legacy DAG runs (#23007)
  • Add dangling rows check for TaskInstance references (#22924)
  • Validate the input params in connection CLI command (#22688)
  • Fix trigger event payload is not persisted in db (#22944)
  • Drop "airflow moved" tables in command db reset (#22990)
  • Add max width to task group tooltips (#22978)
  • Add template support for external_task_ids. (#22809)
  • Allow DagParam to hold falsy values (#22964)
  • Fix regression in pool metrics (#22939)
  • Priority order tasks even when using pools (#22483)
  • Do not clear XCom when resuming from deferral (#22932)
  • Handle invalid JSON metadata in get_logs_with_metadata endpoint. (#22898)
  • Fix pre-upgrade check for rows dangling w.r.t. dag_run (#22850)
  • Fixed backfill interference with scheduler (#22701)
  • Support conf param override for backfill runs (#22837)
  • Correctly interpolate pool name in PoolSlotsAvailableDep statues (#22807)
  • Fix email_on_failure with render_template_as_native_obj (#22770)
  • Fix processor cleanup on DagFileProcessorManager (#22685)
  • Prevent meta name clash for task instances (#22783)
  • remove json parse for gantt chart (#22780)
  • Check for missing dagrun should know version (#22752)
  • Fixes ScheduleInterval spec (#22635)
  • Fixing task status for non-running and non-committed tasks (#22410)
  • Do not log the hook connection details even at DEBUG level (#22627)
  • Stop crashing when empty logs are received from kubernetes client (#22566)
  • Fix bugs about timezone change (#22525)
  • Fix entire DAG stops when one task has end_date (#20920)
  • Use logger to print message during task execution. (#22488)
  • Make sure finalizers are not skipped during exception handling (#22475)
  • update smart sensor docs and minor fix on is_smart_sensor_compatible() (#22386)
  • Fix run_id k8s and elasticsearch compatibility with Airflow 2.1 (#22385)
  • Allow to except_skip None on BranchPythonOperator (#20411)
  • Fix incorrect datetime details (DagRun views) (#21357)
  • Remove incorrect deprecation warning in secrets backend (#22326)
  • Remove RefreshConfiguration workaround for K8s token refreshing (#20759)
  • Masking extras in GET /connections/<connection> endpoint (#22227)
  • Set queued_dttm when submitting task to directly to executor (#22259)
  • Addressed some issues in the tutorial mentioned in discussion #22233 (#22236)
  • Change default python executable to python3 for docker decorator (#21973)
  • Don't validate that Params are JSON when NOTSET (#22000)
  • Add per-DAG delete permissions (#21938)
  • Fix handling some None parameters in kubernetes 23 libs. (#21905)
  • Fix handling of empty (None) tags in bulk_write_to_db (#21757)
  • Fix DAG date range bug (#20507)
  • Removed request.referrer from views.py (#21751)
  • Make DbApiHook use get_uri from Connection (#21764)
  • Fix some migrations (#21670)
  • [de]serialize resources on task correctly (#21445)
  • Add params dag_id, task_id etc to XCom.serialize_value (#19505)
  • Update test connection functionality to use custom form fields (#21330)
  • fix all "high" npm vulnerabilities (#21526)
  • Fix bug incorrectly removing action from role, rather than permission. (#21483)
  • Fix relationship join bug in FAB/SecurityManager with SQLA 1.4 (#21296)
  • Use Identity instead of Sequence in SQLAlchemy 1.4 for MSSQL (#21238)
  • Ensure on_task_instance_running listener can get at task (#21157)
  • Return to the same place when triggering a DAG (#20955)
  • Fix task ID deduplication in @task_group (#20870)
  • Add downgrade to some FAB migrations (#20874)
  • Only validate Params when DAG is triggered (#20802)
  • Fix airflow trigger cli (#20781)
  • Fix task instances iteration in a pool to prevent blocking (#20816)
  • Allow depending to a @task_group as a whole (#20671)
  • Use original task's start_date if a task continues after deferral (#20062)
  • Disabled edit button in task instances list view page (#20659)
  • Fix a package name import error (#20519) (#20519)
  • Remove execution_date label when get cleanup pods list (#20417)
  • Remove unneeded FAB REST API endpoints (#20487)
  • Fix parsing of Cloudwatch log group arn containing slashes (#14667) (#19700)
  • Sanity check for MySQL's TIMESTAMP column (#19821)
  • Allow using default celery command group with executors subclassed from Celery-based executors. (#18189)
  • Move class_permission_name to mixin so it applies to all classes (#18749)
  • Adjust trimmed_pod_id and replace '.' with '-' (#19036)
  • Pass custom_headers to send_email and send_email_smtp (#19009)
  • Ensure catchup=False is used in example dags (#19396)
  • Edit permalinks in OpenApi description file (#19244)
  • Navigate directly to DAG when selecting from search typeahead list (#18991)
  • [Minor] Fix padding on home page (#19025)

Doc only changes

  • Update doc for DAG file processing (#23209)
  • Replace changelog/updating with release notes and towncrier now (#22003)
  • Fix wrong reference in tracking-user-activity.rst (#22745)
  • Remove references to rbac = True from docs (#22725)
  • Doc: Update description for executor-bound dependencies (#22601)
  • Update check-health.rst (#22372)
  • Stronger language about Docker Compose customizability (#22304)
  • Update logging-tasks.rst (#22116)
  • Add example config of sql_alchemy_connect_args (#22045)
  • Update best-practices.rst (#22053)
  • Add information on DAG pausing/deactivation/deletion (#22025)
  • Add brief examples of integration test dags you might want (#22009)
  • Run inclusive language check on CHANGELOG (#21980)
  • Add detailed email docs for Sendgrid (#21958)
  • Add docs for db upgrade / db downgrade (#21879)
  • Update modules_management.rst (#21889)
  • Fix UPDATING section on SqlAlchemy 1.4 scheme changes (#21887)
  • Update TaskFlow tutorial doc to show how to pass "operator-level" args. (#21446)
  • Fix doc - replace decreasing by increasing (#21805)
  • Add another way to dynamically generate DAGs to docs (#21297)
  • Add extra information about time synchronization needed (#21685)
  • Update debug.rst docs (#21246)
  • Replaces the usage of postgres:// with postgresql:// (#21205)
  • Fix task execution process in CeleryExecutor docs (#20783)

Misc/Internal

  • Bring back deprecated security manager functions (#23243)
  • Replace usage of DummyOperator with EmptyOperator (#22974)
  • Deprecate DummyOperator in favor of EmptyOperator (#22832)
  • Remove unnecessary python 3.6 conditionals (#20549)
  • Bump moment from 2.29.1 to 2.29.2 in /airflow/www (#22873)
  • Bump prismjs from 1.26.0 to 1.27.0 in /airflow/www (#22823)
  • Bump nanoid from 3.1.23 to 3.3.2 in /airflow/www (#22803)
  • Bump minimist from 1.2.5 to 1.2.6 in /airflow/www (#22798)
  • Remove dag parsing from db init command (#22531)
  • Update our approach for executor-bound dependencies (#22573)
  • Use Airflow.Base.metadata in FAB models (#22353)
  • Limit docutils to make our documentation pretty again (#22420)
  • Add Python 3.10 support (#22050)
  • [FEATURE] add 1.22 1.23 K8S support (#21902)
  • Remove pandas upper limit now that SQLA is 1.4+ (#22162)
  • Patch sql_alchemy_conn if old postgres scheme used (#22333)
  • Protect against accidental misuse of XCom.get_value() (#22244)
  • Order filenames for migrations (#22168)
  • Don't try to auto generate migrations for Celery tables (#22120)
  • Require SQLAlchemy 1.4 (#22114)
  • bump sphinx-jinja (#22101)
  • Add compat shim for SQLAlchemy to avoid warnings (#21959)
  • Rename xcom.dagrun_id to xcom.dag_run_id (#21806)
  • Deprecate non-JSON conn.extra (#21816)
  • Bump upper bound version of jsonschema to 5.0 (#21712)
  • Deprecate helper utility days_ago (#21653)
  • Remove :type lines now sphinx-autoapi supports type hints (#20951)
  • Silence deprecation warning in tests (#20900)
  • Use DagRun.run_id instead of execution_date when updating state of TIs (UI & REST API) (#18724)
  • Add Context stub to Airflow packages (#20817)
  • Update Kubernetes library version (#18797)
  • Rename PodLauncher to PodManager (#20576)
  • Removes Python 3.6 support (#20467)
  • Add deprecation warning for non-json-serializable params (#20174)
  • Rename TaskMixin to DependencyMixin (#20297)
  • Deprecate passing execution_date to XCom methods (#19825)
  • Remove get_readable_dags and get_editable_dags, and get_accessible_dags. (#19961)
  • Remove postgres 9.6 support (#19987)
  • Removed hardcoded connection types. Check if hook is instance of DbApiHook. (#19639)
  • add kubernetes 1.21 support (#19557)
  • Add FAB base class and set import_name explicitly. (#19667)
  • Removes unused state transitions to handle auto-changing view permissions. (#19153)
  • Chore: Use enum for __var and __type members (#19303)
  • Use fab models (#19121)
  • Consolidate method names between Airflow Security Manager and FAB default (#18726)
  • Remove distutils usages for Python 3.10 (#19064)
  • Removing redundant max_tis_per_query initialisation on SchedulerJob (#19020)
  • Remove deprecated usage of init_role() from API (#18820)
  • Remove duplicate code on dbapi hook (#18821)