-
Notifications
You must be signed in to change notification settings - Fork 13.7k
/
RELEASE_NOTES.rst
13192 lines (11307 loc) · 685 KB
/
RELEASE_NOTES.rst
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. contents:: Apache Airflow Releases
:local:
:depth: 1
.. towncrier release notes start
Airflow 2.8.2 (2024-02-26)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
The ``allowed_deserialization_classes`` flag now follows a glob pattern (#36147).
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
For example if one wants to add the class ``airflow.tests.custom_class`` to the
``allowed_deserialization_classes`` list, it can be done by writing the full class
name (``airflow.tests.custom_class``) or a pattern such as the ones used in glob
search (e.g., ``airflow.*``, ``airflow.tests.*``).
If you currently use a custom regexp path make sure to rewrite it as a glob pattern.
Alternatively, if you still wish to match it as a regexp pattern, add it under the new
list ``allowed_deserialization_classes_regexp`` instead.
The audit_logs permissions have been updated for heightened security (#37501).
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
This was done under the policy that we do not want users like Viewer, Ops,
and other users apart from Admin to have access to audit_logs. The intention behind
this change is to restrict users with less permissions from viewing user details
like First Name, Email etc. from the audit_logs when they are not permitted to.
The impact of this change is that the existing users with non admin rights won't be able
to view or access the audit_logs, both from the Browse tab or from the DAG run.
``AirflowTimeoutError`` is no longer ``except`` by default through ``Exception`` (#35653).
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The ``AirflowTimeoutError`` is now inheriting ``BaseException`` instead of
``AirflowException``->``Exception``.
See https://docs.python.org/3/library/exceptions.html#exception-hierarchy
This prevents code catching ``Exception`` from accidentally
catching ``AirflowTimeoutError`` and continuing to run.
``AirflowTimeoutError`` is an explicit intent to cancel the task, and should not
be caught in attempts to handle the error and return some default value.
Catching ``AirflowTimeoutError`` is still possible by explicitly ``except``ing
``AirflowTimeoutError`` or ``BaseException``.
This is discouraged, as it may allow the code to continue running even after
such cancellation requests.
Code that previously depended on performing strict cleanup in every situation
after catching ``Exception`` is advised to use ``finally`` blocks or
context managers. To perform only the cleanup and then automatically
re-raise the exception.
See similar considerations about catching ``KeyboardInterrupt`` in
https://docs.python.org/3/library/exceptions.html#KeyboardInterrupt
Bug Fixes
"""""""""
- Sort dag processing stats by last_runtime (#37302)
- Allow pre-population of trigger form values via URL parameters (#37497)
- Base date for fetching dag grid view must include selected run_id (#34887)
- Check permissions for ImportError (#37468)
- Move ``IMPORT_ERROR`` from DAG related permissions to view related permissions (#37292)
- Change ``AirflowTaskTimeout`` to inherit ``BaseException`` (#35653)
- Revert "Fix future DagRun rarely triggered by race conditions when max_active_runs reached its upper limit. (#31414)" (#37596)
- Change margin to padding so first task can be selected (#37527)
- Fix Airflow serialization for ``namedtuple`` (#37168)
- Fix bug with clicking url-unsafe tags (#37395)
- Set deterministic and new getter for ``Treeview`` function (#37162)
- Fix permissions of parent folders for log file handler (#37310)
- Fix permission check on DAGs when ``access_entity`` is specified (#37290)
- Fix the value of ``dateTimeAttrFormat`` constant (#37285)
- Resolve handler close race condition at triggerer shutdown (#37206)
- Fixing status icon alignment for various views (#36804)
- Remove superfluous ``@Sentry.enrich_errors`` (#37002)
- Use execution_date= param as a backup to base date for grid view (#37018)
- Handle SystemExit raised in the task. (#36986)
- Revoking audit_log permission from all users except admin (#37501)
- Fix broken regex for allowed_deserialization_classes (#36147)
- Fix the bug that affected the DAG end date. (#36144)
- Adjust node width based on task name length (#37254)
- fix: PythonVirtualenvOperator crashes if any python_callable function is defined in the same source as DAG (#37165)
- Fix collapsed grid width, line up selected bar with gantt (#37205)
- Adjust graph node layout (#37207)
- Revert the sequence of initializing configuration defaults (#37155)
- Displaying "actual" try number in TaskInstance view (#34635)
- Bugfix Triggering DAG with parameters is mandatory when show_trigger_form_if_no_params is enabled (#37063)
- Secret masker ignores passwords with special chars (#36692)
- Fix DagRuns with UPSTREAM_FAILED tasks get stuck in the backfill. (#36954)
- Disable ``dryrun`` auto-fetch (#36941)
- Fix copy button on a DAG run's config (#36855)
- Fix bug introduced by replacing spaces by + in run_id (#36877)
- Fix webserver always redirecting to home page if user was not logged in (#36833)
- REST API set description on POST to ``/variables`` endpoint (#36820)
- Sanitize the conn_id to disallow potential script execution (#32867)
- Fix task id copy button copying wrong id (#34904)
- Fix security manager inheritance in fab provider (#36538)
- Avoid ``pendulum.from_timestamp`` usage (#37160)
Miscellaneous
"""""""""""""
- Install latest docker ``CLI`` instead of specific one (#37651)
- Bump ``undici`` from ``5.26.3`` to ``5.28.3`` in ``/airflow/www`` (#37493)
- Add Python ``3.12`` exclusions in ``providers/pyproject.toml`` (#37404)
- Remove ``markdown`` from core dependencies (#37396)
- Remove unused ``pageSize`` method. (#37319)
- Add more-itertools as dependency of common-sql (#37359)
- Replace other ``Python 3.11`` and ``3.12`` deprecations (#37478)
- Include ``airflow_pre_installed_providers.txt`` into ``sdist`` distribution (#37388)
- Turn Pydantic into an optional dependency (#37320)
- Limit ``universal-pathlib to < 0.2.0`` (#37311)
- Allow running airflow against sqlite in-memory DB for tests (#37144)
- Add description to ``queue_when`` (#36997)
- Updated ``config.yml`` for environment variable ``sql_alchemy_connect_args`` (#36526)
- Bump min version of ``Alembic to 1.13.1`` (#36928)
- Limit ``flask-session`` to ``<0.6`` (#36895)
Doc Only Changes
""""""""""""""""
- Fix upgrade docs to reflect true ``CLI`` flags available (#37231)
- Fix a bug in fundamentals doc (#37440)
- Add redirect for deprecated page (#37384)
- Fix the ``otel`` config descriptions (#37229)
- Update ``Objectstore`` tutorial with ``prereqs`` section (#36983)
- Add more precise description on avoiding generic ``package/module`` names (#36927)
- Add airflow version substitution into Docker Compose Howto (#37177)
- Add clarification about DAG author capabilities to security model (#37141)
- Move docs for cron basics to Authoring and Scheduling section (#37049)
- Link to release notes in the upgrade docs (#36923)
- Prevent templated field logic checks in ``__init__`` of operators automatically (#33786)
Airflow 2.8.1 (2024-01-19)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
Target version for core dependency ``pendulum`` package set to 3 (#36281).
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Support for pendulum 2.1.2 will be saved for a while, presumably until the next feature version of Airflow.
It is advised to upgrade user code to use pendulum 3 as soon as possible.
Airflow packaging specification follows modern Python packaging standards (#36537).
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
We standardized Airflow dependency configuration to follow latest development in Python packaging by
using ``pyproject.toml``. Airflow is now compliant with those accepted PEPs:
* `PEP-440 Version Identification and Dependency Specification <https://www.python.org/dev/peps/pep-0440/>`__
* `PEP-517 A build-system independent format for source trees <https://www.python.org/dev/peps/pep-0517/>`__
* `PEP-518 Specifying Minimum Build System Requirements for Python Projects <https://www.python.org/dev/peps/pep-0518/>`__
* `PEP-561 Distributing and Packaging Type Information <https://www.python.org/dev/peps/pep-0561/>`__
* `PEP-621 Storing project metadata in pyproject.toml <https://www.python.org/dev/peps/pep-0621/>`__
* `PEP-660 Editable installs for pyproject.toml based builds (wheel based) <https://www.python.org/dev/peps/pep-0660/>`__
* `PEP-685 Comparison of extra names for optional distribution dependencies <https://www.python.org/dev/peps/pep-0685/>`__
Also we implement multiple license files support coming from Draft, not yet accepted (but supported by hatchling) PEP:
* `PEP 639 Improving License Clarity with Better Package Metadata <https://peps.python.org/pep-0639/>`__
This has almost no noticeable impact on users if they are using modern Python packaging and development tools, generally
speaking Airflow should behave as it did before when installing it from PyPI and it should be much easier to install
it for development purposes using ``pip install -e ".[devel]"``.
The differences from the user side are:
* Airflow extras now get extras normalized to ``-`` (following PEP-685) instead of ``_`` and ``.``
(as it was before in some extras). When you install airflow with such extras (for example ``dbt.core`` or
``all_dbs``) you should use ``-`` instead of ``_`` and ``.``.
In most modern tools this will work in backwards-compatible way, but in some old version of those tools you might need to
replace ``_`` and ``.`` with ``-``. You can also get warnings that the extra you are installing does not exist - but usually
this warning is harmless and the extra is installed anyway. It is, however, recommended to change to use ``-`` in extras in your dependency
specifications for all Airflow extras.
* Released airflow package does not contain ``devel``, ``devel-*``, ``doc`` and ``doc-gen`` extras.
Those extras are only available when you install Airflow from sources in ``--editable`` mode. This is
because those extras are only used for development and documentation building purposes and are not needed
when you install Airflow for production use. Those dependencies had unspecified and varying behaviour for
released packages anyway and you were not supposed to use them in released packages.
* The ``all`` and ``all-*`` extras were not always working correctly when installing Airflow using constraints
because they were also considered as development-only dependencies. With this change, those dependencies are
now properly handling constraints and they will install properly with constraints, pulling the right set
of providers and dependencies when constraints are used.
Graphviz dependency is now an optional one, not required one (#36647).
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The ``graphviz`` dependency has been problematic as Airflow required dependency - especially for
ARM-based installations. Graphviz packages require binary graphviz libraries - which is already a
limitation, but they also require to install graphviz Python bindings to be build and installed.
This does not work for older Linux installation but - more importantly - when you try to install
Graphviz libraries for Python 3.8, 3.9 for ARM M1 MacBooks, the packages fail to install because
Python bindings compilation for M1 can only work for Python 3.10+.
This is not a breaking change technically - the CLIs to render the DAGs is still there and IF you
already have graphviz installed, it will continue working as it did before. The only problem when it
does not work is where you do not have graphviz installed it will raise an error and inform that you need it.
Graphviz will remain to be installed for most users:
* the Airflow Image will still contain graphviz library, because
it is added there as extra
* when previous version of Airflow has been installed already, then
graphviz library is already installed there and Airflow will
continue working as it did
The only change will be a new installation of new version of Airflow from the scratch, where graphviz will
need to be specified as extra or installed separately in order to enable DAG rendering option.
Bug Fixes
"""""""""
- Fix airflow-scheduler exiting with code 0 on exceptions (#36800)
- Fix Callback exception when a removed task is the last one in the ``taskinstance`` list (#36693)
- Allow anonymous user edit/show resource when set ``AUTH_ROLE_PUBLIC=admin`` (#36750)
- Better error message when sqlite URL uses relative path (#36774)
- Explicit string cast required to force integer-type run_ids to be passed as strings instead of integers (#36756)
- Add log lookup exception for empty ``op`` subtypes (#35536)
- Remove unused index on task instance (#36737)
- Fix check on subclass for ``typing.Union`` in ``_infer_multiple_outputs`` for Python 3.10+ (#36728)
- Make sure ``multiple_outputs`` is inferred correctly even when using ``TypedDict`` (#36652)
- Add back FAB constant in legacy security manager (#36719)
- Fix AttributeError when using ``Dagrun.update_state`` (#36712)
- Do not let ``EventsTimetable`` schedule past events if ``catchup=False`` (#36134)
- Support encryption for triggers parameters (#36492)
- Fix the type hint for ``tis_query`` in ``_process_executor_events`` (#36655)
- Redirect to index when user does not have permission to access a page (#36623)
- Avoid using dict as default value in ``call_regular_interval`` (#36608)
- Remove option to set a task instance to running state in UI (#36518)
- Fix details tab not showing when using dynamic task mapping (#36522)
- Raise error when ``DagRun`` fails while running ``dag test`` (#36517)
- Refactor ``_manage_executor_state`` by refreshing TIs in batch (#36502)
- Add flask config: ``MAX_CONTENT_LENGTH`` (#36401)
- Fix get_leaves calculation for teardown in nested group (#36456)
- Stop serializing timezone-naive datetime to timezone-aware datetime with UTC tz (#36379)
- Make ``kubernetes`` decorator type annotation consistent with operator (#36405)
- Fix Webserver returning 500 for POST requests to ``api/dag/*/dagrun`` from anonymous user (#36275)
- Fix the required access for get_variable endpoint (#36396)
- Fix datetime reference in ``DAG.is_fixed_time_schedule`` (#36370)
- Fix AirflowSkipException message raised by BashOperator (#36354)
- Allow PythonVirtualenvOperator.skip_on_exit_code to be zero (#36361)
- Increase width of execution_date input in trigger.html (#36278)
- Fix logging for pausing DAG (#36182)
- Stop deserializing pickle when enable_xcom_pickling is False (#36255)
- Check DAG read permission before accessing DAG code (#36257)
- Enable mark task as failed/success always (#36254)
- Create latest log dir symlink as relative link (#36019)
- Fix Python-based decorators templating (#36103)
Miscellaneous
"""""""""""""
- Rename concurrency label to max active tasks (#36691)
- Restore function scoped ``httpx`` import in file_task_handler for performance (#36753)
- Add support of Pendulum 3 (#36281)
- Standardize airflow build process and switch to Hatchling build backend (#36537)
- Get rid of ``pyarrow-hotfix`` for ``CVE-2023-47248`` (#36697)
- Make ``graphviz`` dependency optional (#36647)
- Announce MSSQL support end in Airflow 2.9.0, add migration script hints (#36509)
- Set min ``pandas`` dependency to 1.2.5 for all providers and airflow (#36698)
- Bump follow-redirects from 1.15.3 to 1.15.4 in ``/airflow/www`` (#36700)
- Provide the logger_name param to base hook in order to override the logger name (#36674)
- Fix run type icon alignment with run type text (#36616)
- Follow BaseHook connection fields method signature in FSHook (#36444)
- Remove redundant ``docker`` decorator type annotations (#36406)
- Straighten typing in workday timetable (#36296)
- Use ``batch_is_authorized_dag`` to check if user has permission to read DAGs (#36279)
- Replace deprecated get_accessible_dag_ids and use get_readable_dags in get_dag_warnings (#36256)
Doc Only Changes
""""""""""""""""
- Metrics tagging documentation (#36627)
- In docs use logical_date instead of deprecated execution_date (#36654)
- Add section about live-upgrading Airflow (#36637)
- Replace ``numpy`` example with practical exercise demonstrating top-level code (#35097)
- Improve and add more complete description in the architecture diagrams (#36513)
- Improve the error message displayed when there is a webserver error (#36570)
- Update ``dags.rst`` with information on DAG pausing (#36540)
- Update installation prerequisites after upgrading to Debian Bookworm (#36521)
- Add description on the ways how users should approach DB monitoring (#36483)
- Add branching based on mapped task group example to dynamic-task-mapping.rst (#36480)
- Add further details to replacement documentation (#36485)
- Use cards when describing priority weighting methods (#36411)
- Update ``metrics.rst`` for param ``dagrun.schedule_delay`` (#36404)
- Update admonitions in Python operator doc to reflect sentiment (#36340)
- Improve audit_logs.rst (#36213)
- Remove Redshift mention from the list of managed Postgres backends (#36217)
Airflow 2.8.0 (2023-12-18)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
Raw HTML code in DAG docs and DAG params descriptions is disabled by default (#35460)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
To ensure that no malicious javascript can be injected with DAG descriptions or trigger UI forms by DAG authors
a new parameter ``webserver.allow_raw_html_descriptions`` was added with default value of ``False``.
If you trust your DAG authors code and want to allow using raw HTML in DAG descriptions and params, you can restore the previous
behavior by setting the configuration value to ``True``.
To ensure Airflow is secure by default, the raw HTML support in trigger UI has been super-seeded by markdown support via
the ``description_md`` attribute. If you have been using ``description_html`` please migrate to ``description_md``.
The ``custom_html_form`` is now deprecated.
New Features
""""""""""""
- AIP-58: Add Airflow ObjectStore (AFS) (`AIP-58 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-58+milestone%3A%22Airflow+2.8.0%22>`_)
- Add XCom tab to Grid (#35719)
- Add "literal" wrapper to disable field templating (#35017)
- Add task context logging feature to allow forwarding messages to task logs (#32646, #32693, #35857)
- Add Listener hooks for Datasets (#34418, #36247)
- Allow override of navbar text color (#35505)
- Add lightweight serialization for deltalake tables (#35462)
- Add support for serialization of iceberg tables (#35456)
- ``prev_end_date_success`` method access (#34528)
- Add task parameter to set custom logger name (#34964)
- Add pyspark decorator (#35247)
- Add trigger as a valid option for the db clean command (#34908)
- Add decorators for external and venv python branching operators (#35043)
- Allow PythonVenvOperator using other index url (#33017)
- Add Python Virtualenv Operator Caching (#33355)
- Introduce a generic export for containerized executor logging (#34903)
- Add ability to clear downstream tis in ``List Task Instances`` view (#34529)
- Attribute ``clear_number`` to track DAG run being cleared (#34126)
- Add BranchPythonVirtualenvOperator (#33356)
- Allow PythonVenvOperator using other index url (#33017)
- Add CLI notification commands to providers (#33116)
- Use dropdown instead of buttons when there are more than 10 retries in log tab (#36025)
Improvements
""""""""""""
- Add ``multiselect`` to run state in grid view (#35403)
- Fix warning message in ``Connection.get_hook`` in case of ImportError (#36005)
- Add processor_subdir to import_error table to handle multiple dag processors (#35956)
- Consolidate the call of change_state to fail or success in the core executors (#35901)
- Relax mandatory requirement for start_date when schedule=None (#35356)
- Use ExitStack to manage mutation of secrets_backend_list in dag.test (#34620)
- improved visibility of tasks in ActionModal for ``taskinstance`` (#35810)
- Create directories based on ``AIRFLOW_CONFIG`` path (#35818)
- Implements ``JSON-string`` connection representation generator (#35723)
- Move ``BaseOperatorLink`` into the separate module (#35032)
- Set mark_end_on_close after set_context (#35761)
- Move external logs links to top of react logs page (#35668)
- Change terminal mode to ``cbreak`` in ``execute_interactive`` and handle ``SIGINT`` (#35602)
- Make raw HTML descriptions configurable (#35460)
- Allow email field to be templated (#35546)
- Hide logical date and run id in trigger UI form (#35284)
- Improved instructions for adding dependencies in TaskFlow (#35406)
- Add optional exit code to list import errors (#35378)
- Limit query result on DB rather than client in ``synchronize_log_template`` function (#35366)
- Allow description to be passed in when using variables CLI (#34791)
- Allow optional defaults in required fields with manual triggered dags (#31301)
- Permitting airflow kerberos to run in different modes (#35146)
- Refactor commands to unify daemon context handling (#34945)
- Add extra fields to plugins endpoint (#34913)
- Add description to pools view (#34862)
- Move cli's Connection export and Variable export command print logic to a separate function (#34647)
- Extract and reuse get_kerberos_principle func from get_kerberos_principle (#34936)
- Change type annotation for ``BaseOperatorLink.operators`` (#35003)
- Optimise and migrate to ``SA2-compatible`` syntax for TaskReschedule (#33720)
- Consolidate the permissions name in SlaMissModelView (#34949)
- Add debug log saying what's being run to ``EventScheduler`` (#34808)
- Increase log reader stream loop sleep duration to 1 second (#34789)
- Resolve pydantic deprecation warnings re ``update_forward_refs`` (#34657)
- Unify mapped task group lookup logic (#34637)
- Allow filtering event logs by attributes (#34417)
- Make connection login and password TEXT (#32815)
- Ban import ``Dataset`` from ``airflow`` package in codebase (#34610)
- Use ``airflow.datasets.Dataset`` in examples and tests (#34605)
- Enhance task status visibility (#34486)
- Simplify DAG trigger UI (#34567)
- Ban import AirflowException from airflow (#34512)
- Add descriptions for airflow resource config parameters (#34438)
- Simplify trigger name expression (#34356)
- Move definition of Pod*Exceptions to pod_generator (#34346)
- Add deferred tasks to the cluster_activity view Pools Slots (#34275)
- heartbeat failure log message fix (#34160)
- Rename variables for dag runs (#34049)
- Clarify new_state in OpenAPI spec (#34056)
- Remove ``version`` top-level element from docker compose files (#33831)
- Remove generic trigger cancelled error log (#33874)
- Use ``NOT EXISTS`` subquery instead of ``tuple_not_in_condition`` (#33527)
- Allow context key args to not provide a default (#33430)
- Order triggers by - TI priority_weight when assign unassigned triggers (#32318)
- Add metric ``triggerer_heartbeat`` (#33320)
- Allow ``airflow variables export`` to print to stdout (#33279)
- Workaround failing deadlock when running backfill (#32991)
- add dag_run_ids and task_ids filter for the batch task instance API endpoint (#32705)
- Configurable health check threshold for triggerer (#33089)
- Rework provider manager to treat Airflow core hooks like other provider hooks (#33051)
- Ensure DAG-level references are filled on unmap (#33083)
- Affix webserver access_denied warning to be configurable (#33022)
- Add support for arrays of different data types in the Trigger Form UI (#32734)
- Add a mechanism to warn if executors override existing CLI commands (#33423)
Bug Fixes
"""""""""
- Account for change in UTC offset when calculating next schedule (#35887)
- Add read access to pools for viewer role (#35352)
- Fix gantt chart queued duration when queued_dttm is greater than start_date for deferred tasks (#35984)
- Avoid crushing container when directory is not found on rm (#36050)
- Update ``reset_user_sessions`` to work from either CLI or web (#36056)
- Fix UI Grid error when DAG has been removed. (#36028)
- Change Trigger UI to use HTTP POST in web ui (#36026)
- Fix airflow db shell needing an extra key press to exit (#35982)
- Change dag grid ``overscroll`` behaviour to auto (#35717)
- Run triggers inline with dag test (#34642)
- Add ``borderWidthRight`` to grid for Firefox ``scrollbar`` (#35346)
- Fix for infinite recursion due to secrets_masker (#35048)
- Fix write ``processor_subdir`` in serialized_dag table (#35661)
- Reload configuration for standalone dag file processor (#35725)
- Long custom operator name overflows in graph view (#35382)
- Add try_number to extra links query (#35317)
- Prevent assignment of non JSON serializable values to DagRun.conf dict (#35096)
- Numeric values in DAG details are incorrectly rendered as timestamps (#35538)
- Fix Scheduler and triggerer crashes in daemon mode when statsd metrics are enabled (#35181)
- Infinite UI redirection loop after deactivating an active user (#35486)
- Bug fix fetch_callback of Partial Subset DAG (#35256)
- Fix DagRun data interval for DeltaDataIntervalTimetable (#35391)
- Fix query in ``get_dag_by_pickle`` util function (#35339)
- Fix TriggerDagRunOperator failing to trigger subsequent runs when reset_dag_run=True (#35429)
- Fix weight_rule property type in ``mappedoperator`` (#35257)
- Bugfix/prevent concurrency with cached venv (#35258)
- Fix dag serialization (#34042)
- Fix py/url-redirection by replacing request.referrer by get_redirect() (#34237)
- Fix updating variables during variable imports (#33932)
- Use Literal from airflow.typing_compat in Airflow core (#33821)
- Always use ``Literal`` from ``typing_extensions`` (#33794)
Miscellaneous
"""""""""""""
- Change default MySQL client to MariaDB (#36243)
- Mark daskexecutor provider as removed (#35965)
- Bump FAB to ``4.3.10`` (#35991)
- Mark daskexecutor provider as removed (#35965)
- Rename ``Connection.to_json_dict`` to ``Connection.to_dict`` (#35894)
- Upgrade to Pydantic v2 (#35551)
- Bump ``moto`` version to ``>= 4.2.9`` (#35687)
- Use ``pyarrow-hotfix`` to mitigate CVE-2023-47248 (#35650)
- Bump ``axios`` from ``0.26.0 to 1.6.0`` in ``/airflow/www/`` (#35624)
- Make docker decorator's type annotation consistent with operator (#35568)
- Add default to ``navbar_text_color`` and ``rm`` condition in style (#35553)
- Avoid initiating session twice in ``dag_next_execution`` (#35539)
- Work around typing issue in examples and providers (#35494)
- Enable ``TCH004`` and ``TCH005`` rules (#35475)
- Humanize log output about retrieved DAG(s) (#35338)
- Switch from Black to Ruff formatter (#35287)
- Upgrade to Flask Application Builder 4.3.9 (#35085)
- D401 Support (#34932, #34933)
- Use requires_access to check read permission on dag instead of checking it explicitly (#34940)
- Deprecate lazy import ``AirflowException`` from airflow (#34541)
- View util refactoring on mapped stuff use cases (#34638)
- Bump ``postcss`` from ``8.4.25 to 8.4.31`` in ``/airflow/www`` (#34770)
- Refactor Sqlalchemy queries to 2.0 style (#34763, #34665, #32883, #35120)
- Change to lazy loading of io in pandas serializer (#34684)
- Use ``airflow.models.dag.DAG`` in examples (#34617)
- Use airflow.exceptions.AirflowException in core (#34510)
- Check that dag_ids passed in request are consistent (#34366)
- Refactors to make code better (#34278, #34113, #34110, #33838, #34260, #34409, #34377, #34350)
- Suspend qubole provider (#33889)
- Generate Python API docs for Google ADS (#33814)
- Improve importing in modules (#33812, #33811, #33810, #33806, #33807, #33805, #33804, #33803,
#33801, #33799, #33800, #33797, #33798, #34406, #33808)
- Upgrade Elasticsearch to 8 (#33135)
Doc Only Changes
""""""""""""""""
- Add support for tabs (and other UX components) to docs (#36041)
- Replace architecture diagram of Airflow with diagrams-generated one (#36035)
- Add the section describing the security model of DAG Author capabilities (#36022)
- Enhance docs for zombie tasks (#35825)
- Reflect drop/add support of DB Backends versions in documentation (#35785)
- More detail on mandatory task arguments (#35740)
- Indicate usage of the ``re2`` regex engine in the .airflowignore documentation. (#35663)
- Update ``best-practices.rst`` (#35692)
- Update ``dag-run.rst`` to mention Airflow's support for extended cron syntax through croniter (#35342)
- Update ``webserver.rst`` to include information of supported OAuth2 providers (#35237)
- Add back dag_run to docs (#35142)
- Fix ``rst`` code block format (#34708)
- Add typing to concrete taskflow examples (#33417)
- Add concrete examples for accessing context variables from TaskFlow tasks (#33296)
- Fix links in security docs (#33329)
Airflow 2.7.3 (2023-11-06)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
No significant changes.
Bug Fixes
"""""""""
- Fix pre-mature evaluation of tasks in mapped task group (#34337)
- Add TriggerRule missing value in rest API (#35194)
- Fix Scheduler crash looping when dagrun creation fails (#35135)
- Fix test connection with ``codemirror`` and extra (#35122)
- Fix usage of cron-descriptor since BC in v1.3.0 (#34836)
- Fix ``get_plugin_info`` for class based listeners. (#35022)
- Some improvements/fixes for dag_run and task_instance endpoints (#34942)
- Fix the dags count filter in webserver home page (#34944)
- Return only the TIs of the readable dags when ~ is provided as a dag_id (#34939)
- Fix triggerer thread crash in daemon mode (#34931)
- Fix wrong plugin schema (#34858)
- Use DAG timezone in TimeSensorAsync (#33406)
- Mark tasks with ``all_skipped`` trigger rule as ``skipped`` if any task is in ``upstream_failed`` state (#34392)
- Add read only validation to read only fields (#33413)
Misc/Internal
"""""""""""""
- Improve testing harness to separate DB and non-DB tests (#35160, #35333)
- Add pytest db_test markers to our tests (#35264)
- Add pip caching for faster build (#35026)
- Upper bound ``pendulum`` requirement to ``<3.0`` (#35336)
- Limit ``sentry_sdk`` to ``1.33.0`` (#35298)
- Fix subtle bug in mocking processor_agent in our tests (#35221)
- Bump ``@babel/traverse`` from ``7.16.0 to 7.23.2`` in ``/airflow/www`` (#34988)
- Bump ``undici`` from ``5.19.1 to 5.26.3`` in ``/airflow/www`` (#34971)
- Remove unused set from ``SchedulerJobRunner`` (#34810)
- Remove warning about ``max_tis per query > parallelism`` (#34742)
- Improve modules import in Airflow core by moving some of them into a type-checking block (#33755)
- Fix tests to respond to Python 3.12 handling of utcnow in sentry-sdk (#34946)
- Add ``connexion<3.0`` upper bound (#35218)
- Limit Airflow to ``< 3.12`` (#35123)
- update moto version (#34938)
- Limit WTForms to below ``3.1.0`` (#34943)
Doc Only Changes
""""""""""""""""
- Fix variables substitution in Airflow Documentation (#34462)
- Added example for defaults in ``conn.extras`` (#35165)
- Update datasets.rst issue with running example code (#35035)
- Remove ``mysql-connector-python`` from recommended MySQL driver (#34287)
- Fix syntax error in task dependency ``set_downstream`` example (#35075)
- Update documentation to enable test connection (#34905)
- Update docs errors.rst - Mention sentry "transport" configuration option (#34912)
- Update dags.rst to put SubDag deprecation note right after the SubDag section heading (#34925)
- Add info on getting variables and config in custom secrets backend (#34834)
- Document BaseExecutor interface in more detail to help users in writing custom executors (#34324)
- Fix broken link to ``airflow_local_settings.py`` template (#34826)
- Fixes python_callable function assignment context kwargs example in params.rst (#34759)
- Add missing multiple_outputs=True param in the TaskFlow example (#34812)
- Remove extraneous ``'>'`` in provider section name (#34813)
- Fix imports in extra link documentation (#34547)
Airflow 2.7.2 (2023-10-12)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
No significant changes
Bug Fixes
"""""""""
- Check if the lower of provided values are sensitives in config endpoint (#34712)
- Add support for ZoneInfo and generic UTC to fix datetime serialization (#34683, #34804)
- Fix AttributeError: 'Select' object has no attribute 'count' during the airflow db migrate command (#34348)
- Make dry run optional for patch task instance (#34568)
- Fix non deterministic datetime deserialization (#34492)
- Use iterative loop to look for mapped parent (#34622)
- Fix is_parent_mapped value by checking if any of the parent ``taskgroup`` is mapped (#34587)
- Avoid top-level airflow import to avoid circular dependency (#34586)
- Add more exemptions to lengthy metric list (#34531)
- Fix dag warning endpoint permissions (#34355)
- Fix task instance access issue in the batch endpoint (#34315)
- Correcting wrong time showing in grid view (#34179)
- Fix www ``cluster_activity`` view not loading due to ``standaloneDagProcessor`` templating (#34274)
- Set ``loglevel=DEBUG`` in 'Not syncing ``DAG-level`` permissions' (#34268)
- Make param validation consistent for DAG validation and triggering (#34248)
- Ensure details panel is shown when any tab is selected (#34136)
- Fix issues related to ``access_control={}`` (#34114)
- Fix not found ``ab_user`` table in the CLI session (#34120)
- Fix FAB-related logging format interpolation (#34139)
- Fix query bug in ``next_run_datasets_summary`` endpoint (#34143)
- Fix for TaskGroup toggles for duplicated labels (#34072)
- Fix the required permissions to clear a TI from the UI (#34123)
- Reuse ``_run_task_session`` in mapped ``render_template_fields`` (#33309)
- Fix scheduler logic to plan new dag runs by ignoring manual runs (#34027)
- Add missing audit logs for Flask actions add, edit and delete (#34090)
- Hide Irrelevant Dag Processor from Cluster Activity Page (#33611)
- Remove infinite animation for pinwheel, spin for 1.5s (#34020)
- Restore rendering of provider configuration with ``version_added`` (#34011)
Doc Only Changes
""""""""""""""""
- Clarify audit log permissions (#34815)
- Add explanation for Audit log users (#34814)
- Import ``AUTH_REMOTE_USER`` from FAB in WSGI middleware example (#34721)
- Add information about drop support MsSQL as DB Backend in the future (#34375)
- Document how to use the system's timezone database (#34667)
- Clarify what landing time means in doc (#34608)
- Fix screenshot in dynamic task mapping docs (#34566)
- Fix class reference in Public Interface documentation (#34454)
- Clarify var.value.get and var.json.get usage (#34411)
- Schedule default value description (#34291)
- Docs for triggered_dataset_event (#34410)
- Add DagRun events (#34328)
- Provide tabular overview about trigger form param types (#34285)
- Add link to Amazon Provider Configuration in Core documentation (#34305)
- Add "security infrastructure" paragraph to security model (#34301)
- Change links to SQLAlchemy 1.4 (#34288)
- Add SBOM entry in security documentation (#34261)
- Added more example code for XCom push and pull (#34016)
- Add state utils to Public Airflow Interface (#34059)
- Replace markdown style link with rst style link (#33990)
- Fix broken link to the "UPDATING.md" file (#33583)
Misc/Internal
"""""""""""""
- Update min-sqlalchemy version to account for latest features used (#34293)
- Fix SesssionExemptMixin spelling (#34696)
- Restrict ``astroid`` version < 3 (#34658)
- Fail dag test if defer without triggerer (#34619)
- Fix connections exported output (#34640)
- Don't run isort when creating new alembic migrations (#34636)
- Deprecate numeric type python version in PythonVirtualEnvOperator (#34359)
- Refactor ``os.path.splitext`` to ``Path.*`` (#34352, #33669)
- Replace = by is for type comparison (#33983)
- Refactor integer division (#34180)
- Refactor: Simplify comparisons (#34181)
- Refactor: Simplify string generation (#34118)
- Replace unnecessary dict comprehension with dict() in core (#33858)
- Change "not all" to "any" for ease of readability (#34259)
- Replace assert by if...raise in code (#34250, #34249)
- Move default timezone to except block (#34245)
- Combine similar if logic in core (#33988)
- Refactor: Consolidate import and usage of random (#34108)
- Consolidate importing of os.path.* (#34060)
- Replace sequence concatenation by unpacking in Airflow core (#33934)
- Refactor unneeded 'continue' jumps around the repo (#33849, #33845, #33846, #33848, #33839, #33844, #33836, #33842)
- Remove [project] section from ``pyproject.toml`` (#34014)
- Move the try outside the loop when this is possible in Airflow core (#33975)
- Replace loop by any when looking for a positive value in core (#33985)
- Do not create lists we don't need (#33519)
- Remove useless string join from core (#33969)
- Add TCH001 and TCH002 rules to pre-commit to detect and move type checking modules (#33865)
- Add cancel_trigger_ids to to_cancel dequeue in batch (#33944)
- Avoid creating unnecessary list when parsing stats datadog tags (#33943)
- Replace dict.items by dict.values when key is not used in core (#33940)
- Replace lambdas with comprehensions (#33745)
- Improve modules import in Airflow core by some of them into a type-checking block (#33755)
- Refactor: remove unused state - SHUTDOWN (#33746, #34063, #33893)
- Refactor: Use in-place .sort() (#33743)
- Use literal dict instead of calling dict() in Airflow core (#33762)
- remove unnecessary map and rewrite it using list in Airflow core (#33764)
- Replace lambda by a def method in Airflow core (#33758)
- Replace type func by ``isinstance`` in fab_security manager (#33760)
- Replace single quotes by double quotes in all Airflow modules (#33766)
- Merge multiple ``isinstance`` calls for the same object in a single call (#33767)
- Use a single statement with multiple contexts instead of nested statements in core (#33769)
- Refactor: Use f-strings (#33734, #33455)
- Refactor: Use random.choices (#33631)
- Use ``str.splitlines()`` to split lines (#33592)
- Refactor: Remove useless str() calls (#33629)
- Refactor: Improve detection of duplicates and list sorting (#33675)
- Simplify conditions on ``len()`` (#33454)
Airflow 2.7.1 (2023-09-07)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
CronTriggerTimetable is now less aggressive when trying to skip a run (#33404)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
When setting ``catchup=False``, CronTriggerTimetable no longer skips a run if
the scheduler does not query the timetable immediately after the previous run
has been triggered.
This should not affect scheduling in most cases, but can change the behaviour if
a DAG is paused-unpaused to manually skip a run. Previously, the timetable (with
``catchup=False``) would only start a run after a DAG is unpaused, but with this
change, the scheduler would try to look at little bit back to schedule the
previous run that covers a part of the period when the DAG was paused. This
means you will need to keep a DAG paused longer (namely, for the entire cron
period to pass) to really skip a run.
Note that this is also the behaviour exhibited by various other cron-based
scheduling tools, such as ``anacron``.
``conf.set()`` becomes case insensitive to match ``conf.get()`` behavior (#33452)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Also, ``conf.get()`` will now break if used with non-string parameters.
``conf.set(section, key, value)`` used to be case sensitive, i.e. ``conf.set("SECTION", "KEY", value)``
and ``conf.set("section", "key", value)`` were stored as two distinct configurations.
This was inconsistent with the behavior of ``conf.get(section, key)``, which was always converting the section and key to lower case.
As a result, configuration options set with upper case characters in the section or key were unreachable.
That's why we are now converting section and key to lower case in ``conf.set`` too.
We also changed a bit the behavior of ``conf.get()``. It used to allow objects that are not strings in the section or key.
Doing this will now result in an exception. For instance, ``conf.get("section", 123)`` needs to be replaced with ``conf.get("section", "123")``.
Bug Fixes
"""""""""
- Ensure that tasks wait for running indirect setup (#33903)
- Respect "soft_fail" for core async sensors (#33403)
- Differentiate 0 and unset as a default param values (#33965)
- Raise 404 from Variable PATCH API if variable is not found (#33885)
- Fix ``MappedTaskGroup`` tasks not respecting upstream dependency (#33732)
- Add limit 1 if required first value from query result (#33672)
- Fix UI DAG counts including deleted DAGs (#33778)
- Fix cleaning zombie RESTARTING tasks (#33706)
- ``SECURITY_MANAGER_CLASS`` should be a reference to class, not a string (#33690)
- Add back ``get_url_for_login`` in security manager (#33660)
- Fix ``2.7.0 db`` migration job errors (#33652)
- Set context inside templates (#33645)
- Treat dag-defined access_control as authoritative if defined (#33632)
- Bind engine before attempting to drop archive tables (#33622)
- Add a fallback in case no first name and last name are set (#33617)
- Sort data before ``groupby`` in TIS duration calculation (#33535)
- Stop adding values to rendered templates UI when there is no dagrun (#33516)
- Set strict to True when parsing dates in webserver views (#33512)
- Use ``dialect.name`` in custom SA types (#33503)
- Do not return ongoing dagrun when a ``end_date`` is less than ``utcnow`` (#33488)
- Fix a bug in ``formatDuration`` method (#33486)
- Make ``conf.set`` case insensitive (#33452)
- Allow timetable to slightly miss catchup cutoff (#33404)
- Respect ``soft_fail`` argument when ``poke`` is called (#33401)
- Create a new method used to resume the task in order to implement specific logic for operators (#33424)
- Fix DagFileProcessor interfering with dags outside its ``processor_subdir`` (#33357)
- Remove the unnecessary ``<br>`` text in Provider's view (#33326)
- Respect ``soft_fail`` argument when ExternalTaskSensor runs in deferrable mode (#33196)
- Fix handling of default value and serialization of Param class (#33141)
- Check if the dynamically-added index is in the table schema before adding (#32731)
- Fix rendering the mapped parameters when using ``expand_kwargs`` method (#32272)
- Fix dependencies for celery and opentelemetry for Python 3.8 (#33579)
Misc/Internal
"""""""""""""
- Bring back ``Pydantic`` 1 compatibility (#34081, #33998)
- Use a trimmed version of README.md for PyPI (#33637)
- Upgrade to ``Pydantic`` 2 (#33956)
- Reorganize ``devel_only`` extra in Airflow's setup.py (#33907)
- Bumping ``FAB`` to ``4.3.4`` in order to fix issues with filters (#33931)
- Add minimum requirement for ``sqlalchemy to 1.4.24`` (#33892)
- Update version_added field for configs in config file (#33509)
- Replace ``OrderedDict`` with plain dict (#33508)
- Consolidate import and usage of itertools (#33479)
- Static check fixes (#33462)
- Import utc from datetime and normalize its import (#33450)
- D401 Support (#33352, #33339, #33337, #33336, #33335, #33333, #33338)
- Fix some missing type hints (#33334)
- D205 Support - Stragglers (#33301, #33298, #33297)
- Refactor: Simplify code (#33160, #33270, #33268, #33267, #33266, #33264, #33292, #33453, #33476, #33567,
#33568, #33480, #33753, #33520, #33623)
- Fix ``Pydantic`` warning about ``orm_mode`` rename (#33220)
- Add MySQL 8.1 to supported versions. (#33576)
- Remove ``Pydantic`` limitation for version < 2 (#33507)
Doc only changes
"""""""""""""""""
- Add documentation explaining template_ext (and how to override it) (#33735)
- Explain how users can check if python code is top-level (#34006)
- Clarify that DAG authors can also run code in DAG File Processor (#33920)
- Fix broken link in Modules Management page (#33499)
- Fix secrets backend docs (#33471)
- Fix config description for base_log_folder (#33388)
Airflow 2.7.0 (2023-08-18)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
Remove Python 3.7 support (#30963)
""""""""""""""""""""""""""""""""""
As of now, Python 3.7 is no longer supported by the Python community.
Therefore, to use Airflow 2.7.0, you must ensure your Python version is
either 3.8, 3.9, 3.10, or 3.11.
Old Graph View is removed (#32958)
""""""""""""""""""""""""""""""""""
The old Graph View is removed. The new Graph View is the default view now.
The trigger UI form is skipped in web UI if no parameters are defined in a DAG (#33351)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
If you are using ``dag_run.conf`` dictionary and web UI JSON entry to run your DAG you should either:
* `Add params to your DAG <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.html#use-params-to-provide-a-trigger-ui-form>`_
* Enable the new configuration ``show_trigger_form_if_no_params`` to bring back old behaviour
The "db init", "db upgrade" commands and "[database] load_default_connections" configuration options are deprecated (#33136).
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Instead, you should use "airflow db migrate" command to create or upgrade database. This command will not create default connections.
In order to create default connections you need to run "airflow connections create-default-connections" explicitly,
after running "airflow db migrate".
In case of SMTP SSL connection, the context now uses the "default" context (#33070)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The "default" context is Python's ``default_ssl_contest`` instead of previously used "none". The
``default_ssl_context`` provides a balance between security and compatibility but in some cases,
when certificates are old, self-signed or misconfigured, it might not work. This can be configured
by setting "ssl_context" in "email" configuration of Airflow.
Setting it to "none" brings back the "none" setting that was used in Airflow 2.6 and before,
but it is not recommended due to security reasons ad this setting disables validation of certificates and allows MITM attacks.
Disable default allowing the testing of connections in UI, API and CLI(#32052)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
For security reasons, the test connection functionality is disabled by default across Airflow UI,
API and CLI. The availability of the functionality can be controlled by the
``test_connection`` flag in the ``core`` section of the Airflow
configuration (``airflow.cfg``). It can also be controlled by the
environment variable ``AIRFLOW__CORE__TEST_CONNECTION``.
The following values are accepted for this config param:
1. ``Disabled``: Disables the test connection functionality and
disables the Test Connection button in the UI.
This is also the default value set in the Airflow configuration.
2. ``Enabled``: Enables the test connection functionality and
activates the Test Connection button in the UI.
3. ``Hidden``: Disables the test connection functionality and
hides the Test Connection button in UI.
For more information on capabilities of users, see the documentation:
https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html#capabilities-of-authenticated-ui-users
It is strongly advised to **not** enable the feature until you make sure that only
highly trusted UI/API users have "edit connection" permissions.
The ``xcomEntries`` API disables support for the ``deserialize`` flag by default (#32176)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
For security reasons, the ``/dags/*/dagRuns/*/taskInstances/*/xcomEntries/*``
API endpoint now disables the ``deserialize`` option to deserialize arbitrary
XCom values in the webserver. For backward compatibility, server admins may set
the ``[api] enable_xcom_deserialize_support`` config to *True* to enable the
flag and restore backward compatibility.
However, it is strongly advised to **not** enable the feature, and perform
deserialization at the client side instead.
Change of the default Celery application name (#32526)
""""""""""""""""""""""""""""""""""""""""""""""""""""""
Default name of the Celery application changed from ``airflow.executors.celery_executor`` to ``airflow.providers.celery.executors.celery_executor``.
You should change both your configuration and Health check command to use the new name:
* in configuration (``celery_app_name`` configuration in ``celery`` section) use ``airflow.providers.celery.executors.celery_executor``
* in your Health check command use ``airflow.providers.celery.executors.celery_executor.app``
The default value for ``scheduler.max_tis_per_query`` is changed from 512 to 16 (#32572)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
This change is expected to make the Scheduler more responsive.
``scheduler.max_tis_per_query`` needs to be lower than ``core.parallelism``.
If both were left to their default value previously, the effective default value of ``scheduler.max_tis_per_query`` was 32
(because it was capped at ``core.parallelism``).
To keep the behavior as close as possible to the old config, one can set ``scheduler.max_tis_per_query = 0``,
in which case it'll always use the value of ``core.parallelism``.
Some executors have been moved to corresponding providers (#32767)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
In order to use the executors, you need to install the providers:
* for Celery executors you need to install ``apache-airflow-providers-celery`` package >= 3.3.0
* for Kubernetes executors you need to install ``apache-airflow-providers-cncf-kubernetes`` package >= 7.4.0
* For Dask executors you need to install ``apache-airflow-providers-daskexecutor`` package in any version
You can achieve it also by installing airflow with ``[celery]``, ``[cncf.kubernetes]``, ``[daskexecutor]`` extras respectively.
Users who base their images on the ``apache/airflow`` reference image (not slim) should be unaffected - the base
reference image comes with all the three providers installed.
Improvement Changes
^^^^^^^^^^^^^^^^^^^
PostgreSQL only improvement: Added index on taskinstance table (#30762)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
This index seems to have great positive effect in a setup with tens of millions such rows.
New Features
""""""""""""
- Add OpenTelemetry to Airflow (`AIP-49 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-49+milestone%3A%22Airflow+2.7.0%22>`_)
- Trigger Button - Implement Part 2 of AIP-50 (#31583)
- Removing Executor Coupling from Core Airflow (`AIP-51 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-51+milestone%3A%22Airflow+2.7.0%22>`_)
- Automatic setup and teardown tasks (`AIP-52 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-52+milestone%3A%22Airflow+2.7.0%22>`_)
- OpenLineage in Airflow (`AIP-53 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+milestone%3A%22Airflow+2.7.0%22+label%3Aprovider%3Aopenlineage>`_)
- Experimental: Add a cache to Variable and Connection when called at dag parsing time (#30259)
- Enable pools to consider deferred tasks (#32709)
- Allows to choose SSL context for SMTP connection (#33070)
- New gantt tab (#31806)
- Load plugins from providers (#32692)
- Add ``BranchExternalPythonOperator`` (#32787, #33360)
- Add option for storing configuration description in providers (#32629)
- Introduce Heartbeat Parameter to Allow ``Per-LocalTaskJob`` Configuration (#32313)
- Add Executors discovery and documentation (#32532)
- Add JobState for job state constants (#32549)
- Add config to disable the 'deserialize' XCom API flag (#32176)
- Show task instance in web UI by custom operator name (#31852)
- Add default_deferrable config (#31712)
- Introducing ``AirflowClusterPolicySkipDag`` exception (#32013)
- Use ``reactflow`` for datasets graph (#31775)
- Add an option to load the dags from db for command tasks run (#32038)
- Add version of ``chain`` which doesn't require matched lists (#31927)
- Use operator_name instead of task_type in UI (#31662)
- Add ``--retry`` and ``--retry-delay`` to ``airflow db check`` (#31836)
- Allow skipped task state task_instance_schema.py (#31421)
- Add a new config for celery result_backend engine options (#30426)
- UI Add Cluster Activity Page (#31123, #32446)
- Adding keyboard shortcuts to common actions (#30950)
- Adding more information to kubernetes executor logs (#29929)
- Add support for configuring custom alembic file (#31415)
- Add running and failed status tab for DAGs on the UI (#30429)
- Add multi-select, proposals and labels for trigger form (#31441)
- Making webserver config customizable (#29926)
- Render DAGCode in the Grid View as a tab (#31113)
- Add rest endpoint to get option of configuration (#31056)
- Add ``section`` query param in get config rest API (#30936)
- Create metrics to track ``Scheduled->Queued->Running`` task state transition times (#30612)
- Mark Task Groups as Success/Failure (#30478)
- Add CLI command to list the provider trigger info (#30822)
- Add Fail Fast feature for DAGs (#29406)
Improvements
""""""""""""
- Improve graph nesting logic (#33421)
- Configurable health check threshold for triggerer (#33089, #33084)
- add dag_run_ids and task_ids filter for the batch task instance API endpoint (#32705)
- Ensure DAG-level references are filled on unmap (#33083)
- Add support for arrays of different data types in the Trigger Form UI (#32734)
- Always show gantt and code tabs (#33029)
- Move listener success hook to after SQLAlchemy commit (#32988)
- Rename ``db upgrade`` to ``db migrate`` and add ``connections create-default-connections`` (#32810, #33136)
- Remove old gantt chart and redirect to grid views gantt tab (#32908)
- Adjust graph zoom based on selected task (#32792)
- Call listener on_task_instance_running after rendering templates (#32716)
- Display execution_date in graph view task instance tooltip. (#32527)
- Allow configuration to be contributed by providers (#32604, #32755, #32812)
- Reduce default for max TIs per query, enforce ``<=`` parallelism (#32572)
- Store config description in Airflow configuration object (#32669)
- Use ``isdisjoint`` instead of ``not intersection`` (#32616)
- Speed up calculation of leaves and roots for task groups (#32592)
- Kubernetes Executor Load Time Optimizations (#30727)
- Save DAG parsing time if dag is not schedulable (#30911)
- Updates health check endpoint to include ``dag_processor`` status. (#32382)
- Disable default allowing the testing of connections in UI, API and CLI (#32052, #33342)
- Fix config var types under the scheduler section (#32132)
- Allow to sort Grid View alphabetically (#32179)
- Add hostname to triggerer metric ``[triggers.running]`` (#32050)
- Improve DAG ORM cleanup code (#30614)
- ``TriggerDagRunOperator``: Add ``wait_for_completion`` to ``template_fields`` (#31122)
- Open links in new tab that take us away from Airflow UI (#32088)
- Only show code tab when a task is not selected (#31744)
- Add descriptions for celery and dask cert configs (#31822)
- ``PythonVirtualenvOperator`` termination log in alert (#31747)
- Migration of all DAG details to existing grid view dag details panel (#31690)
- Add a diagram to help visualize timer metrics (#30650)
- Celery Executor load time optimizations (#31001)
- Update code style for ``airflow db`` commands to SQLAlchemy 2.0 style (#31486)
- Mark uses of md5 as "not-used-for-security" in FIPS environments (#31171)
- Add pydantic support to serde (#31565)
- Enable search in note column in DagRun and TaskInstance (#31455)
- Save scheduler execution time by adding new Index idea for dag_run (#30827)
- Save scheduler execution time by caching dags (#30704)
- Support for sorting DAGs by Last Run Date in the web UI (#31234)
- Better typing for Job and JobRunners (#31240)
- Add sorting logic by created_date for fetching triggers (#31151)
- Remove DAGs.can_create on access control doc, adjust test fixture (#30862)
- Split Celery logs into stdout/stderr (#30485)
- Decouple metrics clients and ``validators`` into their own modules (#30802)
- Description added for pagination in ``get_log`` api (#30729)
- Optimize performance of scheduling mapped tasks (#30372)
- Add sentry transport configuration option (#30419)
- Better message on deserialization error (#30588)
Bug Fixes