Bot releases are hidden (Show)

airflow - Apache Airflow 2.6.3

Published by ephraimbuddy over 1 year ago

Bug Fixes

Use linear time regular expressions (#32303)
Fix triggerers alive check and add a new conf for triggerer heartbeat rate (#32123)
Catch the exception that triggerer initialization failed (#31999)
Hide sensitive values from extra in connection edit form (#32309)
Sanitize DagRun.run_id and allow flexibility (#32293)
Add triggerer canceled log (#31757)
Fix try number shown in the task view (#32361)
Retry transactions on occasional deadlocks for rendered fields (#32341)
Fix behaviour of LazyDictWithCache when import fails (#32248)
Remove executor_class from Job - fixing backfill for custom executors (#32219)
Fix bugged singleton implementation (#32218)
Use mapIndex to display extra links per mapped task. (#32154)
Ensure that main triggerer thread exits if the async thread fails (#32092)
Use re2 for matching untrusted regex (#32060)
Render list items in rendered fields view (#32042)
Fix hashing of dag_dependencies in serialized dag (#32037)
Return None if an XComArg fails to resolve in a multiple_outputs Task (#32027)
Check for DAG ID in query param from url as well as kwargs (#32014)
Flash an error message instead of failure in rendered-templates when map index is not found (#32011)
Fix ExternalTaskSensor when there is no task group TIs for the current execution date (#32009)
Fix number param html type in trigger template (#31980, #31946)
Fix masking nested variable fields (#31964)
Fix operator_extra_links property serialization in mapped tasks (#31904)
Decode old-style nested Xcom value (#31866)
Add a check for trailing slash in webserver base_url (#31833)
Fix connection uri parsing when the host includes a scheme (#31465)
Fix database session closing with xcom_pull and inlets (#31128)
Fix DAG's on_failure_callback is not invoked when task failed during testing dag. (#30965)
Fix airflow module version check when using ExternalPythonOperator and debug logging level (#30367)

Misc/Internal

Fix task.sensor annotation in type stub (#31954)
Limit Pydantic to < 2.0.0 until we solve 2.0.0 incompatibilities (#32312)
Fix Pydantic 2 pickiness about model definition (#32307)

Doc only changes

Add explanation about tag creation and cleanup (#32406)
Minor updates to docs (#32369, #32315, #32310, #31794)
Clarify Listener API behavior (#32269)
Add information for users who ask for requirements (#32262)
Add links to DAGRun / DAG / Task in Templates Reference (#32245)
Add comment to warn off a potential wrong fix (#32230)
Add a note that we'll need to restart triggerer to reflect any trigger change (#32140)
Adding missing hyperlink to the tutorial documentation (#32105)
Added difference between Deferrable and Non-Deferrable Operators (#31840)
Add comments explaining need for special "trigger end" log message (#31812)
Documentation update on Plugin updates. (#31781)
Fix SemVer link in security documentation (#32320)
Update security model of Airflow (#32098)
Update references to restructured documentation from Airflow core (#32282)
Separate out advanced logging configuration (#32131)
Add ™ to Airflow in prominent places (#31977)

airflow - Apache Airflow Helm Chart 1.10.0

Published by jedcunningham over 1 year ago

Significant Changes

Default Airflow image is updated to `2.6.2` (#31979)

The default Airflow image that is used with the Chart is now 2.6.2, previously it was 2.5.3.

New Features

Add support for container security context (#31043)

Improvements

Validate executor and config.core.executor match (#30693)
Support minAvailable property for PodDisruptionBudget (#30603)
Add volumeMounts to dag processor waitForMigrations (#30990)
Template extra volumes (#30773)

Bug Fixes

Fix webserver probes timeout and period (#30609)
Add missing waitForMigrations for workers (#31625)
Add missing priorityClassName to K8S worker pod template (#31328)
Adding log groomer sidecar to dag processor (#30726)
Do not propagate global security context to statsd and redis (#31865)

Misc

Default Airflow version to 2.6.2 (#31979)
Use template comments for the chart license header (#30569)
Align apiVersion and kind order in chart templates (#31850)
Cleanup Kubernetes < 1.23 support (#31847)

airflow - Apache Airflow 2.6.2

Published by eladkal over 1 year ago

Bug Fixes

Cascade update of TaskInstance to TaskMap table (#31445)
Fix Kubernetes executors detection of deleted pods (#31274)
Use keyword parameters for migration methods for mssql (#31309)
Control permissibility of driver config in extra from airflow.cfg (#31754)
Fixing broken links in openapi/v1.yaml (#31619)
Hide old alert box when testing connection with different value (#31606)
Add TriggererStatus to OpenAPI spec (#31579)
Resolving issue where Grid won't un-collapse when Details is collapsed (#31561)
Fix sorting of tags (#31553)
Add the missing map_index to the xcom key when skipping downstream tasks (#31541)
Fix airflow users delete CLI command (#31539)
Include triggerer health status in Airflow /health endpoint (#31529)
Remove dependency already registered for this task warning (#31502)
Use kube_client over default CoreV1Api for deleting pods (#31477)
Ensure min backoff in base sensor is at least 1 (#31412)
Fix max_active_tis_per_dagrun for Dynamic Task Mapping (#31406)
Fix error handling when pre-importing modules in DAGs (#31401)
Fix dropdown default and adjust tutorial to use 42 as default for proof (#31400)
Fix crash when clearing run with task from normal to mapped (#31352)
Make BaseJobRunner a generic on the job class (#31287)
Fix url_for_asset fallback and 404 on DAG Audit Log (#31233)
Don't present an undefined execution date (#31196)
Added spinner activity while the logs load (#31165)
Include rediss to the list of supported URL schemes (#31028)
Optimize scheduler by skipping "non-schedulable" DAGs (#30706)
Save scheduler execution time during search for queued dag_runs (#30699)
Fix ExternalTaskSensor to work correctly with task groups (#30742)
Fix DAG.access_control can't sync when clean access_control (#30340)
Fix failing get_safe_url tests for latest Python 3.8 and 3.9 (#31766)
Fix typing for POST user endpoint (#31767)
Fix wrong update for nested group default args (#31776)
Fix overriding default_args in nested task groups (#31608)
Mark [secrets] backend_kwargs as a sensitive config (#31788)
Executor events are not always "exited" here (#30859)
Validate connection IDs (#31140)

Misc/Internal

Add Python 3.11 support (#27264)
Replace unicodecsv with standard csv library (#31693)
Bring back unicodecsv as dependency of Airflow (#31814)
Remove found_descendents param from get_flat_relative_ids (#31559)
Fix typing in external task triggers (#31490)
Wording the next and last run DAG columns better (#31467)
Skip auto-document things with :meta private: (#31380)
Add an example for sql_alchemy_connect_args conf (#31332)
Convert dask upper-binding into exclusion (#31329)
Upgrade FAB to 4.3.1 (#31203)
Added metavar and choices to --state flag in airflow dags list-jobs CLI for suggesting valid state arguments. (#31308)
Use only one line for tmp dir log (#31170)
Rephrase comment in setup.py (#31312)
Add fullname to owner on logging (#30185)
Make connection id validation consistent across interface (#31282)
Use single source of truth for sensitive config items (#31820)

Doc only changes

Add docstring and signature for _read_remote_logs (#31623)
Remove note about triggerer being 3.7+ only (#31483)
Fix version support information (#31468)
Add missing BashOperator import to documentation example (#31436)
Fix task.branch error caused by incorrect initial parameter (#31265)
Update callbacks documentation (errors and context) (#31116)
Add an example for dynamic task mapping with non-TaskFlow operator (#29762)
Few doc fixes - links, grammar and wording (#31719)
Add description in a few more places about adding airflow to pip install (#31448)
Fix table formatting in docker build documentation (#31472)
Update documentation for constraints installation (#31882)

airflow - Apache Airflow 2.6.1

Published by ephraimbuddy over 1 year ago

Significant Changes

Clarifications of the external Health Check mechanism and using `Job` classes (#31277).

In the past SchedulerJob and other *Job classes are known to have been used to perform
external health checks for Airflow components. Those are, however, Airflow DB ORM related classes.
The DB models and database structure of Airflow are considered as internal implementation detail, following
public interface <https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html>_).
Therefore, they should not be used for external health checks. Instead, you should use the
airflow jobs check CLI command (introduced in Airflow 2.1) for that purpose.

Bug Fixes

Fix calculation of health check threshold for SchedulerJob (#31277)
Fix timestamp parse failure for k8s executor pod tailing (#31175)
Make sure that DAG processor job row has filled value in job_type column (#31182)
Fix section name reference for api_client_retry_configuration (#31174)
Ensure the KPO runs pod mutation hooks correctly (#31173)
Remove worrying log message about redaction from the OpenLineage plugin (#31149)
Move interleave_timestamp_parser config to the logging section (#31102)
Ensure that we check worker for served logs if no local or remote logs found (#31101)
Fix MappedTaskGroup import in taskinstance file (#31100)
Format DagBag.dagbag_report() Output (#31095)
Mask task attribute on task detail view (#31125)
Fix template error when iterating None value and fix params documentation (#31078)
Fix apache-hive extra so it installs the correct package (#31068)
Fix issue with zip files in DAGs folder when pre-importing Airflow modules (#31061)
Move TaskInstanceKey to a separate file to fix circular import (#31033, #31204)
Fix deleting DagRuns and TaskInstances that have a note (#30987)
Fix airflow providers get command output (#30978)
Fix Pool schema in the OpenAPI spec (#30973)
Add support for dynamic tasks with template fields that contain pandas.DataFrame (#30943)
Use the Task Group explicitly passed to 'partial' if any (#30933)
Fix order_by request in list DAG rest api (#30926)
Include node height/width in center-on-task logic (#30924)
Remove print from dag trigger command (#30921)
Improve task group UI in new graph (#30918)
Fix mapped states in grid view (#30916)
Fix problem with displaying graph (#30765)
Fix backfill KeyError when try_number out of sync (#30653)
Re-enable clear and setting state in the TaskInstance UI (#30415)
Prevent DagRun's state and start_date from being reset when clearing a task in a running DagRun (#30125)

Misc/Internal

Upper bind dask until they solve a side effect in their test suite (#31259)
Show task instances affected by clearing in a table (#30633)
Fix missing models in API documentation (#31021)

Doc only changes

Improve description of the dag_processing.processes metric (#30891)
Improve Quick Start instructions (#30820)
Add section about missing task logs to the FAQ (#30717)
Mount the config directory in docker compose (#30662)
Update version_added config field for might_contain_dag and metrics_allow_list (#30969)

airflow - Apache Airflow 2.6.0

Published by ephraimbuddy over 1 year ago

Significant Changes

Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506).

Default setting handles case where impersonation is needed and both users (airflow and the impersonated user)
have the same group set as main group. Previously the default was also other-writeable and the user might choose
to use the other-writeable setting if they wish by configuring file_task_handler_new_folder_permissions
and file_task_handler_new_file_permissions in logging section.

SLA callbacks no longer add files to the dag processor manager's queue (#30076)

This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU,
and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state.

The `cleanup()` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152).

This is potentially a breaking change for any custom trigger implementations that override the cleanup()
method and uses synchronous code, however using synchronous operations in cleanup was technically wrong,
because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays
impacting other triggers. The change is unlikely to affect any existing trigger implementations.

The gauge `scheduler.tasks.running` no longer exist (#30374)

The gauge has never been working and its value has always been 0. Having an accurate
value for this metric is complex so it has been decided that removing this gauge makes
more sense than fixing it with no certainty of the correctness of its value.

Consolidate handling of tasks stuck in queued under new `task_queued_timeout` config (#30375)

Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations
responsible for timing out stuck queued tasks have been deprecated and merged into
[scheduler] task_queued_timeout. The configurations that have been deprecated are
[kubernetes] worker_pods_pending_timeout, [celery] stalled_task_timeout, and
[celery] task_adoption_timeout. If any of these configurations are set, the longest timeout will be
respected. For example, if [celery] stalled_task_timeout is 1200, and [scheduler] task_queued_timeout
is 600, Airflow will set [scheduler] task_queued_timeout to 1200.

Improvement Changes

Display only the running configuration in configurations view (#28892)

The configurations view now only displays the running configuration. Previously, the default configuration
was displayed at the top but it was not obvious whether this default configuration was overridden or not.
Subsequently, the non-documented endpoint /configuration?raw=true is deprecated and will be removed in
Airflow 3.0. The HTTP response now returns an additional Deprecation header. The /config endpoint on
the REST API is the standard way to fetch Airflow configuration programmatically.

Explicit skipped states list for ExternalTaskSensor (#29933)

ExternalTaskSensor now has an explicit skipped_states list

Miscellaneous Changes

Handle OverflowError on exponential backoff in next_run_calculation (#28172)

Maximum retry task delay is set to be 24h (86400s) by default. You can change it globally via core.max_task_retry_delay
parameter.

Move Hive macros to the provider (#28538)

The Hive Macros (hive.max_partition, hive.closest_ds_partition) are available only when Hive Provider is
installed. Please install Hive Provider > 5.1.0 when using those macros.

New Features

Skip PythonVirtualenvOperator task when it returns a provided exit code (#30690)
rename skip_exit_code to skip_on_exit_code and allow providing multiple codes (#30692)
Add skip_on_exit_code also to ExternalPythonOperator (#30738)
Add max_active_tis_per_dagrun for Dynamic Task Mapping (#29094)
Add serializer for pandas dataframe (#30390)
Deferrable TriggerDagRunOperator (#30292)
Add command to get DAG Details via CLI (#30432)
Adding ContinuousTimetable and support for @continuous schedule_interval (#29909)
Allow customized rules to check if a file has dag (#30104)
Add a new Airflow conf to specify a SSL ca cert for Kubernetes client (#30048)
Bash sensor has an explicit retry code (#30080)
Add filter task upstream/downstream to grid view (#29885)
Add testing a connection via Airflow CLI (#29892)
Support deleting the local log files when using remote logging (#29772)
Blocklist to disable specific metric tags or metric names (#29881)
Add a new graph inside of the grid view (#29413)
Add database check_migrations config (#29714)
add output format arg for cli.dags.trigger (#29224)
Make json and yaml available in templates (#28930)
Enable tagged metric names for existing Statsd metric publishing events | influxdb-statsd support (#29093)
Add arg --yes to db export-archived command. (#29485)
Make the policy functions pluggable (#28558)
Add airflow db drop-archived command (#29309)
Enable individual trigger logging (#27758)
Implement new filtering options in graph view (#29226)
Add triggers for ExternalTask (#29313)
Add command to export purged records to CSV files (#29058)
Add FileTrigger (#29265)
Emit DataDog statsd metrics with metadata tags (#28961)
Add some statsd metrics for dataset (#28907)
Add --overwrite option to connections import CLI command (#28738)
Add general-purpose "notifier" concept to DAGs (#28569)
Add a new conf to wait past_deps before skipping a task (#27710)
Add Flink on K8s Operator (#28512)
Allow Users to disable SwaggerUI via configuration (#28354)
Show mapped task groups in graph (#28392)
Log FileTaskHandler to work with KubernetesExecutor's multi_namespace_mode (#28436)
Add a new config for adapting masked secrets to make it easier to prevent secret leakage in logs (#28239)
List specific config section and its values using the cli (#28334)
KubernetesExecutor multi_namespace_mode can use namespace list to avoid requiring cluster role (#28047)
Automatically save and allow restore of recent DAG run configs (#27805)
Added exclude_microseconds to cli (#27640)

Improvements

Rename most pod_id usage to pod_name in KubernetesExecutor (#29147)
Update the error message for invalid use of poke-only sensors (#30821)
Update log level in scheduler critical section edge case (#30694)
AIP-51 Removing Executor Coupling from Core Airflow (AIP-51 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-51+milestone%3A%22Airflow+2.6.0%22>_)
Add multiple exit code handling in skip logic for BashOperator (#30739)
Updated app to support configuring the caching hash method for FIPS v2 (#30675)
Preload airflow imports before dag parsing to save time (#30495)
Improve task & run actions UX in grid view (#30373)
Speed up TaskGroups with caching property of group_id (#30284)
Use the engine provided in the session (#29804)
Type related import optimization for Executors (#30361)
Add more type hints to the code base (#30503)
Always use self.appbuilder.get_session in security managers (#30233)
Update SQLAlchemy select() to new style (#30515)
Refactor out xcom constants from models (#30180)
Add exception class name to DAG-parsing error message (#30105)
Rename statsd_allow_list and statsd_block_list to metrics_*_list (#30174)
Improve serialization of tuples and sets (#29019)
Make cleanup method in trigger an async one (#30152)
Lazy load serialization modules (#30094)
SLA callbacks no longer add files to the dag_processing manager queue (#30076)
Add task.trigger rule to grid_data (#30130)
Speed up log template sync by avoiding ORM (#30119)
Separate cli_parser.py into two modules (#29962)
Explicit skipped states list for ExternalTaskSensor (#29933)
Add task state hover highlighting to new graph (#30100)
Store grid tabs in url params (#29904)
Use custom Connexion resolver to load lazily (#29992)
Delay Kubernetes import in secret masker (#29993)
Delay ConnectionModelView init until it's accessed (#29946)
Scheduler, make stale DAG deactivation threshold configurable instead of using dag processing timeout (#29446)
Improve grid view height calculations (#29563)
Avoid importing executor during conf validation (#29569)
Make permissions for FileTaskHandler group-writeable and configurable (#29506)
Add colors in help outputs of Airflow CLI commands #28789 (#29116)
Add a param for get_dags endpoint to list only unpaused dags (#28713)
Expose updated_at filter for dag run and task instance endpoints (#28636)
Increase length of user identifier columns (#29061)
Update gantt chart UI to display queued state of tasks (#28686)
Add index on log.dttm (#28944)
Display only the running configuration in configurations view (#28892)
Cap dropdown menu size dynamically (#28736)
Added JSON linter to connection edit / add UI for field extra. On connection edit screen, existing extra data will be displayed indented (#28583)
Use labels instead of pod name for pod log read in k8s exec (#28546)
Use time not tries for queued & running re-checks. (#28586)
CustomTTYColoredFormatter should inherit TimezoneAware formatter (#28439)
Improve past depends handling in Airflow CLI tasks.run command (#28113)
Support using a list of callbacks in on_*_callback/sla_miss_callbacks (#28469)
Better table name validation for db clean (#28246)
Use object instead of array in config.yml for config template (#28417)
Add markdown rendering for task notes. (#28245)
Show mapped task groups in grid view (#28208)
Add renamed and previous_name in config sections (#28324)
Speed up most Users/Role CLI commands (#28259)
Speed up Airflow role list command (#28244)
Refactor serialization (#28067, #30819, #30823)
Allow longer pod names for k8s executor / KPO (#27736)
Updates health check endpoint to include triggerer status (#27755)

Bug Fixes

Fix static_folder for cli app (#30952)
Initialize plugins for cli appbuilder (#30934)
Fix dag file processor heartbeat to run only if necessary (#30899)
Fix KubernetesExecutor sending state to scheduler (#30872)
Count mapped upstream only if all are finished (#30641)
ExternalTaskSensor: add external_task_group_id to template_fields (#30401)
Improve url detection for task instance details (#30779)
Use material icons for dag import error banner (#30771)
Fix misc grid/graph view UI bugs (#30752)
Add a collapse grid button (#30711)
Fix d3 dependencies (#30702)
Simplify logic to resolve tasks stuck in queued despite stalled_task_timeout (#30375)
When clearing task instances try to get associated DAGs from database (#29065)
Fix mapped tasks partial arguments when DAG default args are provided (#29913)
Deactivate DAGs deleted from within zip files (#30608)
Recover from too old resource version exception by retrieving the latest resource_version (#30425)
Fix possible race condition when refreshing DAGs (#30392)
Use custom validator for OpenAPI request body (#30596)
Fix TriggerDagRunOperator with deferrable parameter (#30406)
Speed up dag runs deletion (#30330)
Do not use template literals to construct html elements (#30447)
Fix deprecation warning in example_sensor_decorator DAG (#30513)
Avoid logging sensitive information in triggerer job log (#30110)
Add a new parameter for base sensor to catch the exceptions in poke method (#30293)
Fix dag run conf encoding with non-JSON serializable values (#28777)
Added fixes for Airflow to be usable on Windows Dask-Workers (#30249)
Force DAG last modified time to UTC (#30243)
Fix EmptySkipOperator in example dag (#30269)
Make the webserver startup respect update_fab_perms (#30246)
Ignore error when changing log folder permissions (#30123)
Disable ordering DagRuns by note (#30043)
Fix reading logs from finished KubernetesExecutor worker pod (#28817)
Mask out non-access bits when comparing file modes (#29886)
Remove Run task action from UI (#29706)
Fix log tailing issues with legacy log view (#29496)
Fixes to how DebugExecutor handles sensors (#28528)
Ensure that pod_mutation_hook is called before logging the pod name (#28534)
Handle OverflowError on exponential backoff in next_run_calculation (#28172)

Misc/Internal

Make eager upgrade additional dependencies optional (#30811)
Upgrade to pip 23.1.1 (#30808)
Remove protobuf limitation from eager upgrade (#30182)
Remove protobuf limitation from eager upgrade (#30182)
Deprecate skip_exit_code in BashOperator (#30734)
Remove gauge scheduler.tasks.running (#30374)
Bump json5 to 1.0.2 and eslint-plugin-import to 2.27.5 in /airflow/www (#30568)
Add tests to PythonOperator (#30362)
Add asgiref as a core dependency (#30527)
Discovery safe mode toggle comment clarification (#30459)
Upgrade moment-timezone package to fix Tehran tz (#30455)
Bump loader-utils from 2.0.0 to 2.0.4 in /airflow/www (#30319)
Bump babel-loader from 8.1.0 to 9.1.0 in /airflow/www (#30316)
DagBag: Use dag.fileloc instead of dag.full_filepath in exception message (#30610)
Change log level of serialization information (#30239)
Minor DagRun helper method cleanup (#30092)
Improve type hinting in stats.py (#30024)
Limit importlib-metadata backport to < 5.0.0 (#29924)
Align cncf provider file names with AIP-21 (#29905)
Upgrade FAB to 4.3.0 (#29766)
Clear ExecutorLoader cache in tests (#29849)
Lazy load Task Instance logs in UI (#29827)
added warning log for max page limit exceeding api calls (#29788)
Aggressively cache entry points in process (#29625)
Don't use importlib.metadata to get Version for speed (#29723)
Upgrade Mypy to 1.0 (#29468)
Rename db export-cleaned to db export-archived (#29450)
listener: simplify API by replacing SQLAlchemy event-listening by direct calls (#29289)
No multi-line log entry for bash env vars (#28881)
Switch to ruff for faster static checks (#28893)
Remove horizontal lines in TI logs (#28876)
Make allowed_deserialization_classes more intuitive (#28829)
Propagate logs to stdout when in k8s executor pod (#28440, #30860)
Fix code readability, add docstrings to json_client (#28619)
AIP-51 - Misc. Compatibility Checks (#28375)
Fix is_local for LocalKubernetesExecutor (#28288)
Move Hive macros to the provider (#28538)
Rerun flaky PinotDB integration test (#28562)
Add pre-commit hook to check session default value (#28007)
Refactor get_mapped_group_summaries for web UI (#28374)
Add support for k8s 1.26 (#28320)
Replace freezegun with time-machine (#28193)
Completed D400 for airflow/kubernetes/* (#28212)
Completed D400 for multiple folders (#27969)
Drop k8s 1.21 and 1.22 support (#28168)
Remove unused task_queue attr from k8s scheduler class (#28049)
Completed D400 for multiple folders (#27767, #27768)

Doc only changes

Add instructions on how to avoid accidental airflow upgrade/downgrade (#30813)
Add explicit information about how to write task logs (#30732)
Better explanation on how to log from tasks (#30746)
Use correct import path for Dataset (#30617)
Create audit_logs.rst (#30405)
Adding taskflow API example for sensors (#30344)
Add clarification about timezone aware dags (#30467)
Clarity params documentation (#30345)
Fix unit for task duration metric (#30273)
Update dag-run.rst for dead links of cli commands (#30254)
Add Write efficient Python code section to Reducing DAG complexity (#30158)
Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters (#29580)
Add Documentation for notification feature extension (#29191)
Clarify that executor interface is public but instances are not (#29200)
Add Public Interface description to Airflow documentation (#28300)
Add documentation for task group mapping (#28001)
Some fixes to metrics doc (#30290)

airflow - Apache Airflow Helm Chart 1.9.0

Published by jedcunningham over 1 year ago

Significant Changes

Default PgBouncer and PgBouncer Exporter images have been updated (#29919)

The PgBouncer and PgBouncer Exporter images are based on newer software/os. They are also multi-platform AMD/ARM images:

pgbouncer: 1.16.1 based on alpine 3.14 (airflow-pgbouncer-2023.02.24-1.16.1)
pgbouncer-exporter: 0.14.0 based on alpine 3.17 (apache/airflow:airflow-pgbouncer-exporter-2023.02.21-0.14.0)

Default Airflow image is updated to `2.5.3` (#30411)

The default Airflow image that is used with the Chart is now 2.5.3, previously it was 2.5.1.

New Features

Add support for hostAliases for Airflow webserver and scheduler (#30051)
Add support for annotations on StatsD Deployment and cleanup CronJob (#30126)
Add support for annotations in logs PVC (#29270)
Add support for annotations in extra ConfigMap and Secrets (#30303)
Add support for pod annotations to PgBouncer (#30168)
Add support for ttlSecondsAfterFinished on migrateDatabaseJob and createUserJob (#29314)
Add support for using SHA digest of Docker images (#30214)

Improvements

Template extra volumes in Helm Chart (#29357)
Make Liveness/Readiness Probe timeouts configurable for PgBouncer Exporter (#29752)
Enable individual trigger logging (#29482)

Bug Fixes

Add config.kubernetes_executor to values (#29818)
Block extra properties in image config (#30217)
Remove replicas if KEDA is enabled (#29838)
Mount kerberos.keytab to worker when enabled (#29526)
Fix adding annotations for dag persistence PVC (#29622)
Fix bitnami/postgresql default username and password (#29478)
Add global volumes in pod template file (#29295)
Add log groomer sidecar to triggerer service (#29392)
Helm deployment fails when postgresql.nameOverride is used (#29214)

Doc only changes

Add gitSync optional env description (#29378)
Add webserver NodePort example (#29460)
Include Rancher in Helm chart install instructions (#28416)
Change RSA SSH host key to reflect update from Github (#30286)

Misc

Update Airflow version to 2.5.3 (#30411)
Switch to newer versions of PgBouncer and PgBouncer Exporter in chart (#29919)
Reformat chart templates (#29917)
Reformat chart templates part 2 (#29941)
Reformat chart templates part 3 (#30312)
Replace deprecated k8s registry references (#29938)
Fix airflow_dags_mount formatting (#29296)
Fix webserver.service.ports formatting (#29297)

airflow - Apache Airflow 2.5.3

Published by pierrejeambrun over 1 year ago

Significant Changes

No significant changes.

Bug Fixes

Fix DagProcessorJob integration for standalone dag-processor (#30278)
Fix proper termination of gunicorn when it hangs (#30188)
Fix XCom.get_one exactly one exception text (#30183)
Correct the VARCHAR size to 250. (#30178)
Revert fix for on_failure_callback when task receives a SIGTERM (#30165)
Move read only property to DagState to fix generated docs (#30149)
Ensure that dag.partial_subset doesn't mutate task group properties (#30129)
Fix inconsistent returned value of airflow dags next-execution cli command (#30117)
Fix www/utils.dag_run_link redirection (#30098)
Fix TriggerRuleDep when the mapped tasks count is 0 (#30084)
Dag processor manager, add retry_db_transaction to _fetch_callbacks (#30079)
Fix db clean command for mysql db (#29999)
Avoid considering EmptyOperator in mini scheduler (#29979)
Fix some long known Graph View UI problems (#29971, #30355, #30360)
Fix dag docs toggle icon initial angle (#29970)
Fix tags selection in DAGs UI (#29944)
Including airflow/example_dags/sql/sample.sql in MANIFEST.in (#29883)
Fixing broken filter in /taskinstance/list view (#29850)
Allow generic param dicts (#29782)
Fix update_mask in patch variable route (#29711)
Strip markup from app_name if instance_name_has_markup = True (#28894)

Misc/Internal

Revert "Also limit importlib on Python 3.9 (#30069)" (#30209)
Add custom_operator_name to @task.sensor tasks (#30131)
Bump webpack from 5.73.0 to 5.76.0 in /airflow/www (#30112)
Formatted config (#30103)
Remove upper bound limit of astroid (#30033)
Remove accidentally merged vendor daemon patch code (#29895)
Fix warning in airflow tasks test command regarding absence of data_interval (#27106)

Doc only changes

Adding more information regarding top level code (#30040)
Update workday example (#30026)
Fix some typos in the DAGs docs (#30015)
Update set-up-database.rst (#29991)
Fix some typos on the kubernetes documentation (#29936)
Fix some punctuation and grammar (#29342)

airflow - Apache Airflow 2.5.2

Published by pierrejeambrun over 1 year ago

Significant Changes

The date-time fields passed as API parameters or Params should be RFC3339-compliant (#29395)

In case of API calls, it was possible that "+" passed as part of the date-time fields were not URL-encoded, and
such date-time fields could pass validation. Such date-time parameters should now be URL-encoded (as %2B).

In case of parameters, we still allow IS8601-compliant date-time (so for example it is possible that
' ' was used instead of T separating date from time and no timezone was specified) but we raise
deprecation warning.

Default for `[webserver] expose_hostname` changed to `False` (#29547)

The default for [webserver] expose_hostname has been set to False, instead of True. This means administrators must opt-in to expose webserver hostnames to end users.

Bug Fixes

Fix validation of date-time field in API and Parameter schemas (#29395)
Fix grid logs for large logs (#29390)
Fix on_failure_callback when task receives a SIGTERM (#29743)
Update min version of python-daemon to fix containerd file limits (#29916)
POST /dagRuns API should 404 if dag not active (#29860)
DAG list sorting lost when switching page (#29756)
Fix Scheduler crash when clear a previous run of a normal task that is now a mapped task (#29645)
Convert moment with timezone to UTC instead of raising an exception (#29606)
Fix clear dag run openapi spec responses by adding additional return type (#29600)
Don't display empty rendered attrs in Task Instance Details page (#29545)
Remove section check from get-value command (#29541)
Do not show version/node in UI traceback for unauthenticated user (#29501)
Make prev_logical_date variable offset-aware (#29454)
Fix nested fields rendering in mapped operators (#29451)
Datasets, next_run_datasets, remove unnecessary timestamp filter (#29441)
Edgemodifier refactoring w/ labels in TaskGroup edge case (#29410)
Fix Rest API update user output (#29409)
Ensure Serialized DAG is deleted (#29407)
Persist DAG and task doc values in TaskFlow API if explicitly set (#29399)
Redirect to the origin page with all the params (#29212)
Fixing Task Duration view in case of manual DAG runs only (#22015) (#29195)
Remove poke method to fall back to parent implementation (#29146)
PR: Introduced fix to run tasks on Windows systems (#29107)
Fix warning in migrations about old config. (#29092)
Emit dagrun failed duration when timeout (#29076)
Handling error on cluster policy itself (#29056)
Fix kerberos authentication for the REST API. (#29054)
Fix leak sensitive field via V1EnvVar on exception (#29016)
Sanitize url_for arguments before they are passed (#29039)
Fix dag run trigger with a note. (#29228)
Write action log to DB when DAG run is triggered via API (#28998)
Resolve all variables in pickled XCom iterator (#28982)
Allow URI without authority and host blocks in airflow connections add (#28922)
Be more selective when adopting pods with KubernetesExecutor (#28899)
KubenetesExecutor sends state even when successful (#28871)
Annotate KubernetesExecutor pods that we don't delete (#28844)
Throttle streaming log reads (#28818)
Introduce dag processor job (#28799)
Fix #28391 manual task trigger from UI fails for k8s executor (#28394)
Logging poke info when external dag is not none and task_id and task_ids are none (#28097)
Fix inconsistencies in checking edit permissions for a DAG (#20346)

Misc/Internal

Add a check for not templateable fields (#29821)
Removed continue for not in (#29791)
Move extra links position in grid view (#29703)
Bump undici from 5.9.1 to 5.19.1 (#29583)
Change expose_hostname default to false (#29547)
Change permissions of config/password files created by airflow (#29495)
Use newer setuptools v67.2.0 (#29465)
Increase max height for grid view elements (#29367)
Clarify description of worker control config (#29247)
Bump ua-parser-js from 0.7.31 to 0.7.33 in /airflow/www (#29172)
Remove upper bound limitation for pytest (#29086)
Check for run_id url param when linking to graph/gantt views (#29066)
Clarify graph view dynamic task labels (#29042)
Fixing import error for dataset (#29007)
Update how PythonSensor returns values from python_callable (#28932)
Add dep context description for better log message (#28875)
Bump swagger-ui-dist from 3.52.0 to 4.1.3 in /airflow/www (#28824)
Limit importlib-metadata backport to < 5.0.0 (#29924, #30069)

Doc Only Changes

Update pipeline.rst - Fix query in merge_data() task (#29158)
Correct argument name of Workday timetable in timetable.rst (#29896)
Update ref anchor for env var link in Connection how-to doc (#29816)
Better description for limit in api (#29773)
Description of dag_processing.last_duration (#29740)
Update docs re: template_fields typing and subclasses (#29725)
Fix formatting of Dataset inlet/outlet note in TaskFlow concepts (#29678)
Specific use-case: adding packages via requirements.txt in compose (#29598)
Detect is 'docker-compose' existing (#29544)
Add Landing Times entry to UI docs (#29511)
Improve health checks in example docker-compose and clarify usage (#29408)
Remove notes param from TriggerDagRunOperator docstring (#29298)
Use schedule param rather than timetable in Timetables docs (#29255)
Add trigger process to Airflow Docker docs (#29203)
Update set-up-database.rst (#29104)
Several improvements to the Params doc (#29062)
Email Config docs more explicit env var examples (#28845)
Listener plugin example added (#27905)

airflow - Apache Airflow Helm Chart 1.8.0

Published by jedcunningham over 1 year ago

Significant Changes

`bitnami/postgresql` subchart updated to `12.1.9` (#29071)

The version of postgresql installed is still version 11.

If you are upgrading an existing helm release with the built-in postgres database, you will either need to delete your release and reinstall fresh, or manually delete these 2 objects:

    kubectl delete secret {RELEASE_NAME}-postgresql
    kubectl delete statefulset {RELEASE_NAME}-postgresql

As a reminder, it is recommended to set up an external database <https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#database>_ in production.

This version of the chart uses different variable names for setting usernames and passwords in the postgres database.

postgresql.auth.enablePostgresUser is used to determine if the "postgres" admin account will be created.
postgresql.auth.postgresPassword sets the password for the "postgres" user.
postgresql.auth.username and postrgesql.auth.password are used to set credentials for a non-admin account if desired.
postgresql.postgresqlUsername and postgresql.postresqlPassword, which were used in the previous version of the chart, are no longer used.

Users will need to make those changes in their values files if they are changing the Postgres configuration.

Previously the subchart version was 10.5.3.

Default `dags.gitSync.wait` reduced to `5` seconds (#27625)

The default for dags.gitSync.wait has been reduced from 60 seconds to 5 seconds to reduce the likelihood of DAGs
becoming inconsistent between Airflow components. This will, however, increase traffic to the remote git repository.

Default Airflow image is updated to `2.5.1` (#29074)

The default Airflow image that is used with the Chart is now 2.5.1, previously it was 2.4.1.

Default git-sync image is updated to `3.6.3` (#27848)

The default git-sync image that is used with the Chart is now 3.6.3, previously it was 3.4.0.

Default redis image is updated to `7-bullseye` (#27443)

The default redis image that is used with the Chart is now 7-bullseye, previously it was 6-bullseye.

New Features

Add annotations on deployments (#28688)
Add global volume & volumeMounts to the chart (#27781)

Improvements

Add support for webserverConfigConfigMapName (#27419)
Enhance chart to allow overriding command-line args to statsd exporter (#28041)
Add support for NodePort in Services (#26945)
Add worker log-groomer-sidecar enable option (#27178)
Add HostAliases to Pod template file (#27544)
Allow PgBouncer replicas to be configurable (#27439)

Bug Fixes

Create scheduler service to serve task logs for LocalKubernetesExecutor (#28828)
Fix NOTES.txt to show correct URL (#28264)
Add worker service account for LocalKubernetesExecutor (#28813)
Remove checks for 1.19 api checks (#28461)
Add airflow_local_settings to all airflow containers (#27779)
Make custom env vars optional for job templates (#27148)
Decrease default gitSync wait (#27625)
Add extraVolumeMounts to sidecars too (#27420)
Fix PgBouncer after PostgreSQL subchart upgrade (#29207)

Doc only changes

Enhance production guide with a few Argo specific guidelines (#29078)
Add doc note about Pod template images (#29032)
Update production guide db section (#28610)
Fix to LoadBalancer snippet (#28014)
Fix gitSync example code (#28083)
Correct repo example for cloning via ssh (#27671)

Misc

Update Airflow version to 2.5.1 (#29074)
Update git-sync to 3.6.3 (#27848)
Upgrade bitnami/postgresql subchart to 12.1.9 (#29071)
Update redis to 7 (#27443)
Replace helm chart icon (#27704)

airflow - Apache Airflow 2.5.1

Published by pierrejeambrun over 1 year ago

Bug Fixes

Fix masking of non-sensitive environment variables (#28802)
Remove swagger-ui extra from connexion and install swagger-ui-dist via npm package (#28788)
Fix UIAlert should_show when AUTH_ROLE_PUBLIC set (#28781)
Only patch single label when adopting pod (#28776)
Update CSRF token to expire with session (#28730)
Fix "airflow tasks render" cli command for mapped task instances (#28698)
Allow XComArgs for external_task_ids of ExternalTaskSensor (#28692)
Row-lock TIs to be removed during mapped task expansion (#28689)
Handle ConnectionReset exception in Executor cleanup (#28685)
Fix description of output redirection for access_log for gunicorn (#28672)
Add back join to zombie query that was dropped in #28198 (#28544)
Fix calendar view for CronTriggerTimeTable dags (#28411)
After running the DAG the employees table is empty. (#28353)
Fix DetachedInstanceError when finding zombies in Dag Parsing process (#28198)
Nest header blocks in divs to fix dagid copy nit on dag.html (#28643)
Fix UI caret direction (#28624)
Guard not-yet-expanded ti in trigger rule dep (#28592)
Move TI setNote endpoints under TaskInstance in OpenAPI (#28566)
Consider previous run in CronTriggerTimetable (#28532)
Ensure correct log dir in file task handler (#28477)
Fix bad pods pickled in executor_config (#28454)
Add ensure_ascii=False in trigger dag run API (#28451)
Add setters to MappedOperator on_*_callbacks (#28313)
Fix ti._try_number for deferred and up_for_reschedule tasks (#26993)
separate callModal from dag.js (#28410)
A manual run can't look like a scheduled one (#28397)
Dont show task/run durations when there is no start_date (#28395)
Maintain manual scroll position in task logs (#28386)
Correctly select a mapped task's "previous" task (#28379)
Trigger gevent monkeypatching via environment variable (#28283)
Fix db clean warnings (#28243)
Make arguments 'offset' and 'length' not required (#28234)
Make live logs reading work for "other" k8s executors (#28213)
Add custom pickling hooks to LazyXComAccess (#28191)
fix next run datasets error (#28165)
Ensure that warnings from @dag decorator are reported in dag file (#28153)
Do not warn when airflow dags tests command is used (#28138)
Ensure the dagbag_size metric decreases when files are deleted (#28135)
Improve run/task grid view actions (#28130)
Make BaseJob.most_recent_job favor "running" jobs (#28119)
Don't emit FutureWarning when code not calling old key (#28109)
Add airflow.api.auth.backend.session to backend sessions in compose (#28094)
Resolve false warning about calling conf.get on moved item (#28075)
Return list of tasks that will be changed (#28066)
Handle bad zip files nicely when parsing DAGs. (#28011)
Prevent double loading of providers from local paths (#27988)
Fix deadlock when chaining multiple empty mapped tasks (#27964)
fix: current_state method on TaskInstance doesn't filter by map_index (#27898)
Don't log CLI actions if db not initialized (#27851)
Make sure we can get out of a faulty scheduler state (#27834)
dagrun, next_dagruns_to_examine, add MySQL index hint (#27821)
Handle DAG disappearing mid-flight when dag verification happens (#27720)
fix: continue checking sla (#26968)
Allow generation of connection URI to work when no conn type (#26765)

Misc/Internal

Remove limit for dnspython after eventlet got fixed (#29004)
Limit dnspython to < 2.3.0 until eventlet incompatibility is solved (#28962)
Add automated version replacement in example dag indexes (#28090)
Cleanup and do housekeeping with plugin examples (#28537)
Limit SQLAlchemy to below 2.0 (#28725)
Bump json5 from 1.0.1 to 1.0.2 in /airflow/www (#28715)
Fix some docs on using sensors with taskflow (#28708)
Change Architecture and OperatingSystem classes into Enums (#28627)
Add doc-strings and small improvement to email util (#28634)
Fix Connection.get_extra type (#28594)
navbar, cap dropdown size, and add scroll bar (#28561)
Emit warnings for conf.get* from the right source location (#28543)
Move MyPY plugins of ours to dev folder (#28498)
Add retry to purge_inactive_dag_warnings (#28481)
Re-enable Plyvel on ARM as it now builds cleanly (#28443)
Add SIGUSR2 handler for LocalTaskJob and workers to aid debugging (#28309)
Convert test_task_command to Pytest and unquarantine tests in it (#28247)
Make invalid characters exception more readable (#28181)
Bump decode-uri-component from 0.2.0 to 0.2.2 in /airflow/www (#28080)
Use asserts instead of exceptions for executor not started (#28019)
Simplify dataset subgraph logic (#27987)
Order TIs by map_index (#27904)
Additional info about Segmentation Fault in LocalTaskJob (#27381)

Doc Only Changes

Mention mapped operator in cluster policy doc (#28885)
Slightly improve description of Dynamic DAG generation preamble (#28650)
Restructure Docs (#27235)
Update scheduler docs about low priority tasks (#28831)
Clarify that versioned constraints are fixed at release time (#28762)
Clarify about docker compose (#28729)
Adding an example dag for dynamic task mapping (#28325)
Use docker compose v2 command (#28605)
Add AIRFLOW_PROJ_DIR to docker-compose example (#28517)
Remove outdated Optional Provider Feature outdated documentation (#28506)
Add documentation for [core] mp_start_method config (#27993)
Documentation for the LocalTaskJob return code counter (#27972)
Note which versions of Python are supported (#27798)

airflow - Apache Airflow 2.5.0

Published by ephraimbuddy almost 2 years ago

New Features

TaskRunner: notify of component start and finish (#27855)
Add DagRun state change to the Listener plugin system(#27113)
Metric for raw task return codes (#27155)
Add logic for XComArg to pull specific map indexes (#27771)
Clear TaskGroup (#26658, #28003)
Add critical section query duration metric (#27700)
Add: #23880 :: Audit log for AirflowModelViews(Variables/Connection) (#24079, #27994, #27923)
Add postgres 15 support (#27444)
Expand tasks in mapped group at run time (#27491)
reset commits, clean submodules (#27560)
scheduler_job, add metric for scheduler loop timer (#27605)
Allow datasets to be used in taskflow (#27540)
Add expanded_ti_count to ti context (#27680)
Add user comment to task instance and dag run (#26457, #27849, #27867)
Enable copying DagRun JSON to clipboard (#27639)
Implement extra controls for SLAs (#27557)
add dag parsed time in DAG view (#27573)
Add max_wait for exponential_backoff in BaseSensor (#27597)
Expand tasks in mapped group at parse time (#27158)
Add disable retry flag on backfill (#23829)
Adding sensor decorator (#22562)
Api endpoint update ti (#26165)
Filtering datasets by recent update events (#26942)
Support Is /not Null filter for value is None on webui (#26584)
Add search to datasets list (#26893)
Split out and handle 'params' in mapped operator (#26100)
Add authoring API for TaskGroup mapping (#26844)
Add one_done trigger rule (#26146)
Create a more efficient airflow dag test command that also has better local logging (#26400)
Support add/remove permissions to roles commands (#26338)
Auto tail file logs in Web UI (#26169)
Add triggerer info to task instance in API (#26249)
Flag to deserialize value on custom XCom backend (#26343)

Improvements

Allow depth-first execution (#27827)
UI: Update offset height if data changes (#27865)
Improve TriggerRuleDep typing and readability (#27810)
Make views requiring session, keyword only args (#27790)
Optimize TI.xcom_pull() with explicit task_ids and map_indexes (#27699)
Allow hyphens in pod id used by k8s executor (#27737)
optimise task instances filtering (#27102)
Use context managers to simplify log serve management (#27756)
Fix formatting leftovers (#27750)
Improve task deadlock messaging (#27734)
Improve "sensor timeout" messaging (#27733)
Replace urlparse with urlsplit (#27389)
Align TaskGroup semantics to AbstractOperator (#27723)
Add new files to parsing queue on every loop of dag processing (#27060)
Make Kubernetes Executor & Scheduler resilient to error during PMH execution (#27611)
Separate dataset deps into individual graphs (#27356)
Use log.exception where more economical than log.error (#27517)
Move validation branch_task_ids into SkipMixin (#27434)
Coerce LazyXComAccess to list when pushed to XCom (#27251)
Update cluster-policies.rst docs (#27362)
Add warning if connection type already registered within the provider (#27520)
Activate debug logging in commands with --verbose option (#27447)
Add classic examples for Python Operators (#27403)
change .first() to .scalar() (#27323)
Improve reset_dag_run description (#26755)
Add examples and howtos about sensors (#27333)
Make grid view widths adjustable (#27273)
Sorting plugins custom menu links by category before name (#27152)
Simplify DagRun.verify_integrity (#26894)
Add mapped task group info to serialization (#27027)
Correct the JSON style used for Run config in Grid View (#27119)
No extra__conn_type__ prefix required for UI behaviors (#26995)
Improve dataset update blurb (#26878)
Rename kubernetes config section to kubernetes_executor (#26873)
decode params for dataset searches (#26941)
Get rid of the DAGRun details page & rely completely on Grid (#26837)
Fix scheduler crashloopbackoff when using hostname_callable (#24999)
Reduce log verbosity in KubernetesExecutor. (#26582)
Don't iterate tis list twice for no reason (#26740)
Clearer code for PodGenerator.deserialize_model_file (#26641)
Don't import kubernetes unless you have a V1Pod (#26496)
Add updated_at column to DagRun and Ti tables (#26252)
Move the deserialization of custom XCom Backend to 2.4.0 (#26392)
Avoid calculating all elements when one item is needed (#26377)
Add __future__.annotations automatically by isort (#26383)
Handle list when serializing expand_kwargs (#26369)
Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290)
Add more weekday operator and sensor examples #26071 (#26098)
Align TaskGroup semantics to AbstractOperator (#27723)

Bug Fixes

Gracefully handle whole config sections being renamed (#28008)
Add allow list for imports during deserialization (#27887)
Soft delete datasets that are no longer referenced in DAG schedules or task outlets (#27828)
Redirect to home view when there are no valid tags in the URL (#25715)
Refresh next run datasets info in dags view (#27839)
Make MappedTaskGroup depend on its expand inputs (#27876)
Make DagRun state updates for paused DAGs faster (#27725)
Don't explicitly set include_examples to False on task run command (#27813)
Fix menu border color (#27789)
Fix backfill queued task getting reset to scheduled state. (#23720)
Fix clearing child dag mapped tasks from parent dag (#27501)
Handle json encoding of V1Pod in task callback (#27609)
Fix ExternalTaskSensor can't check zipped dag (#27056)
Avoid re-fetching DAG run in TriggerDagRunOperator (#27635)
Continue on exception when retrieving metadata (#27665)
External task sensor fail fix (#27190)
Add the default None when pop actions (#27537)
Display parameter values from serialized dag in trigger dag view. (#27482, #27944)
Move TriggerDagRun conf check to execute (#27035)
Resolve trigger assignment race condition (#27072)
Update google_analytics.html (#27226)
Fix some bug in web ui dags list page (auto-refresh & jump search null state) (#27141)
Fixed broken URL for docker-compose.yaml (#26721)
Fix xcom arg.py .zip bug (#26636)
Fix 404 taskInstance errors and split into two tables (#26575)
Fix browser warning of improper thread usage (#26551)
template rendering issue fix (#26390)
Clear autoregistered DAGs if there are any import errors (#26398)
Fix from airflow import version lazy import (#26239)
allow scroll in triggered dag runs modal (#27965)

Misc/Internal

Remove is_mapped attribute (#27881)
Simplify FAB table resetting (#27869)
Fix old-style typing in Base Sensor (#27871)
Switch (back) to late imports (#27730)
Completed D400 for multiple folders (#27748)
simplify notes accordion test (#27757)
completed D400 for airflow/callbacks/* airflow/cli/* (#27721)
Completed D400 for airflow/api_connexion/* directory (#27718)
Completed D400 for airflow/listener/* directory (#27731)
Completed D400 for airflow/lineage/* directory (#27732)
Update API & Python Client versions (#27642)
Completed D400 & D401 for airflow/api/* directory (#27716)
Completed D400 for multiple folders (#27722)
Bump minimatch from 3.0.4 to 3.0.8 in /airflow/www (#27688)
Bump loader-utils from 1.4.1 to 1.4.2 in /airflow/www (#27697)
Disable nested task mapping for now (#27681)
bump alembic minimum version (#27629)
remove unused code.html (#27585)
Enable python string normalization everywhere (#27588)
Upgrade dependencies in order to avoid backtracking (#27531)
Strengthen a bit and clarify importance of triaging issues (#27262)
Deduplicate type hints (#27508)
Add stub 'yield' to BaseTrigger.run (#27416)
Remove upper-bound limit to dask (#27415)
Limit Dask to under 2022.10.1 (#27383)
Update old style typing (#26872)
Enable string normalization for docs (#27269)
Slightly faster up/downgrade tests (#26939)
Deprecate use of core get_kube_client in PodManager (#26848)
Add memray files to gitignore / dockerignore (#27001)
Bump sphinx and sphinx-autoapi (#26743)
Simplify RTIF.delete_old_records() (#26667)
migrate last react files to typescript (#26112)
Work around pyupgrade edge cases (#26384)

Doc only changes

Document dag_file_processor_timeouts metric as deprecated (#27067)
Drop support for PostgreSQL 10 (#27594)
Update index.rst (#27529)
Add note about pushing the lazy XCom proxy to XCom (#27250)
Fix BaseOperator link (#27441)
[docs] best-practices add use variable with template example. (#27316)
docs for custom view using plugin (#27244)
Update graph view and grid view on overview page (#26909)
Documentation fixes (#26819)
make consistency on markup title string level (#26696)
Add documentation to dag test function (#26713)
Fix broken URL for docker-compose.yaml (#26726)
Add a note against use of top level code in timetable (#26649)
Fix example_datasets dag names (#26495)
Update docs: zip-like effect is now possible in task mapping (#26435)
changing to task decorator in docs from classic operator use (#25711)

airflow - Apache Airflow 2.4.3

Published by ephraimbuddy almost 2 years ago

Bug Fixes

Fix double logging with some task logging handler (#27591)
Replace FAB url filtering function with Airflow's (#27576)
Fix mini scheduler expansion of mapped task (#27506)
SLAMiss is nullable and not always given back when pulling task instances (#27423)
Fix behavior of _ when searching for DAGs (#27448)
Fix getting the dag/task ids from BaseExecutor (#27550)
Fix SQLAlchemy primary key black-out error on DDRQ (#27538)
Fix IntegrityError during webserver startup (#27297)
Add case insensitive constraint to username (#27266)
Fix python external template keys (#27256)
Reduce extraneous task log requests (#27233)
Make RotatingFilehandler used in DagProcessor non-caching (#27223)
Listener: Set task on SQLAlchemy TaskInstance object (#27167)
Fix dags list page auto-refresh & jump search null state (#27141)
Set executor.job_id to BackfillJob.id for backfills (#27020)

Misc/Internal

Bump loader-utils from 1.4.0 to 1.4.1 in /airflow/www (#27552)
Reduce log level for k8s TCP_KEEPALIVE etc warnings (#26981)

Doc only changes

Use correct executable in docker compose docs (#27529)
Fix wording in DAG Runs description (#27470)
Document that KubernetesExecutor overwrites container args (#27450)
Fix BaseOperator links (#27441)
Correct timer units to seconds from milliseconds. (#27360)
Add missed import in the Trigger Rules example (#27309)
Update SLA wording to reflect it is relative to Dag Run start. (#27111)
Add kerberos environment variables to the docs (#27028)

airflow - Apache Airflow 2.4.2

Published by ephraimbuddy almost 2 years ago

Bug Fixes

Make tracebacks opt-in (#27059)
Add missing AUTOINC/SERIAL for FAB tables (#26885)
Add separate error handler for 405(Method not allowed) errors (#26880)
Don't re-patch pods that are already controlled by current worker (#26778)
Handle mapped tasks in task duration chart (#26722)
Fix task duration cumulative chart (#26717)
Avoid 500 on dag redirect (#27064)
Filter dataset dependency data on webserver (#27046)
Remove double collection of dags in airflow dags reserialize (#27030)
Fix auto refresh for graph view (#26926)
Don't overwrite connection extra with invalid json (#27142)
Fix next run dataset modal links (#26897)
Change dag audit log sort by date from asc to desc (#26895)
Bump min version of jinja2 (#26866)
Add missing colors to state_color_mapping jinja global (#26822)
Fix running debuggers inside airflow tasks test (#26806)
Fix warning when using xcomarg dependencies (#26801)
demote Removed state in priority for displaying task summaries (#26789)
Ensure the log messages from operators during parsing go somewhere (#26779)
Add restarting state to TaskState Enum in REST API (#26776)
Allow retrieving error message from data.detail (#26762)
Simplify origin string cleaning (#27143)
Remove DAG parsing from StandardTaskRunner (#26750)
Fix non-hidden cumulative chart on duration view (#26716)
Remove TaskFail duplicates check (#26714)
Fix airflow tasks run --local when dags_folder differs from that of processor (#26509)
Fix yarn warning from d3-color (#27139)
Fix version for a couple configurations (#26491)
Revert "No grid auto-refresh for backfill dag runs (#25042)" (#26463)
Retry on Airflow Schedule DAG Run DB Deadlock (#26347)

Misc/Internal

Clean-ups around task-mapping code (#26879)
Move user-facing string to template (#26815)
add icon legend to datasets graph (#26781)
Bump sphinx and sphinx-autoapi (#26743)
Simplify RTIF.delete_old_records() (#26667)
Bump FAB to 4.1.4 (#26393)

Doc only changes

Fixed triple quotes in task group example (#26829)
Documentation fixes (#26819)
make consistency on markup title string level (#26696)
Add a note against use of top level code in timetable (#26649)
Fix broken URL for docker-compose.yaml (#26726)

airflow - Apache Airflow Helm Chart 1.7.0

Published by jedcunningham about 2 years ago

New Features

Make cleanup job history configurable (#26838)
Added labels to specific Airflow components (#25031)
Add StatsD overrideMappings in Helm chart values (#26598)
Adding podAnnotations to StatsD deployment template (#25732)
Container specific extra environment variables (#24784)
Custom labels for extra Secrets and ConfigMaps (#25283)
Add revisionHistoryLimit to all deployments (#25059)
Adding podAnnotations to Redis StatefulSet (#23708)
Provision Standalone Dag Processor (#23711)
Add configurable scheme for webserver probes (#22815)
Add support for KEDA HPA config to Helm chart (#24220)

Improvements

Add 'executor' label to Airflow scheduler deployment (#25684)
Add default flower_url_prefix in Helm chart values (#26415)
Add liveness probe to Celery workers (#25561)
Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

Fix pod template imagePullPolicy (#26423)
Do not declare a volume for sshKeySecret if dag persistence is enabled (#22913)
Pass worker annotations to generated pod template (#24647)
Fix semver compare number for jobs check command (#24480)
Use --local flag for liveness probes in Airflow 2.5+ (#24999)

Doc only changes

Improve documentation on helm hooks disabling (#26747)
Remove ssh:// prefix from git repo value (#26632)
Fix defaultAirflowRepository comment (#26428)
Baking DAGs into Docker image (#26401)
Reload pods when using the same DAG tag (#24576)
Minor clarifications about result_backend, dag processor, and helm uninstall (#24929)
Add hyperlinks to GitHub PRs for Release Notes (#24532)
Terraform should not use Helm hooks for starting jobs (#26604)
Flux should not use Helm hooks for starting jobs (#24288)
Provide details on how to pull Airflow image from a private repository (#24394)
Helm logo no longer a link (#23977)
Document LocalKubernetesExecutor support in chart (#23876)
Update Production Guide (#23836)

Misc

Default Airflow version to 2.4.1 (#26485)
Vendor in the Bitnami chart (#24395)
Remove kubernetes 1.20 support (#25871)

airflow - Apache Airflow 2.4.1

Published by jedcunningham about 2 years ago

Bug Fixes

When rendering template, unmap task in context (#26702)
Fix scroll overflow for ConfirmDialog (#26681)
Resolve deprecation warning re Table.exists() (#26616)
Fix XComArg zip bug (#26636)
Use COALESCE when ordering runs to handle NULL (#26626)
Check user is active (#26635)
No missing user warning for public admin (#26611)
Allow MapXComArg to resolve after serialization (#26591)
Resolve warning about DISTINCT ON query on dags view (#26608)
Log warning when secret backend kwargs is invalid (#26580)
Fix grid view log try numbers (#26556)
Template rendering issue in passing templates_dict to task decorator (#26390)
Fix Deferrable stuck as scheduled during backfill (#26205)
Suppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617)
Correctly set json_provider_class on Flask app so it uses our encoder (#26554)
Fix WSGI root app (#26549)
Fix deadlock when mapped task with removed upstream is rerun (#26518)
ExecutorConfigType should be cacheable (#26498)
Fix proper joining of the path for logs retrieved from celery workers (#26493)
DAG Deps extends base_template (#26439)
Don't update backfill run from the scheduler (#26342)

Doc only changes

Clarify owner links document (#26515)
Fix invalid RST in dataset concepts doc (#26434)
Document the non-sensitive-only option for expose_config (#26507)
Fix example_datasets dag names (#26495)
Zip-like effect is now possible in task mapping (#26435)
Use task decorator in docs instead of classic operators (#25711)

airflow - Apache Airflow 2.4.0

Published by ephraimbuddy about 2 years ago

New Features

Add Data-aware Scheduling (https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-48+milestone%3A%22Airflow+2.4.0%22)
Add @task.short_circuit TaskFlow decorator (#25752)
Make execution_date_or_run_id optional in tasks test command (#26114)
Automatically register DAGs that are used in a context manager (#23592, #26398)
Add option of sending DAG parser logs to stdout. (#25754)
Support multiple DagProcessors parsing files from different locations. (#25935)
Implement ExternalPythonOperator (#25780)
Make execution_date optional for command dags test (#26111)
Implement expand_kwargs() against a literal list (#25925)
Add trigger rule tooltip (#26043)
Add conf parameter to CLI for airflow dags test (#25900)
Include scheduled slots in pools view (#26006)
Add output property to MappedOperator (#25604)
Add roles delete command to cli (#25854)
Add Airflow specific warning classes (#25799)
Add support for TaskGroup in ExternalTaskSensor (#24902)
Add @task.kubernetes taskflow decorator (#25663)
Add a way to import Airflow without side-effects (#25832)
Let timetables control generated run_ids. (#25795)
Allow per-timetable ordering override in grid view (#25633)
Grid logs for mapped instances (#25610, #25621, #25611)
Consolidate to one schedule param (#25410)
DAG regex flag in backfill command (#23870)
Adding support for owner links in the Dags view UI (#25280)
Ability to clear a specific DAG Run's task instances via REST API (#23516)
Possibility to document DAG with a separate markdown file (#25509)
Add parsing context to DAG Parsing (#25161)
Implement CronTriggerTimetable (#23662)
Add option to mask sensitive data in UI configuration page (#25346)
Create new databases from the ORM (#24156)
Implement XComArg.zip(*xcom_args) (#25176)
Introduce sla_miss metric (#23402)
Implement map() semantic (#25085)
Add override method to TaskGroupDecorator (#25160)
Implement expand_kwargs() (#24989)
Add parameter to turn off SQL query logging (#24570)
Add DagWarning model, and a check for missing pools (#23317)
Add Task Logs to Grid details panel (#24249)
Added small health check server and endpoint in scheduler(#23905)
Add built-in External Link for ExternalTaskMarker operator (#23964)
Add default task retry delay config (#23861)
Add clear DagRun endpoint. (#23451)
Add support for timezone as string in cron interval timetable (#23279)
Add auto-refresh to dags home page (#22900, #24770)

Improvements

Add more weekday operator and sensor examples #26071 (#26098)
Add subdir parameter to dags reserialize command (#26170)
Update zombie message to be more descriptive (#26141)
Only send an SlaCallbackRequest if the DAG is scheduled (#26089)
Promote Operator.output more (#25617)
Upgrade API files to typescript (#25098)
Less hacky double-rendering prevention in mapped task (#25924)
Improve Audit log (#25856)
Remove mapped operator validation code (#25870)
More DAG(schedule=...) improvements (#25648)
Reduce operator_name dupe in serialized JSON (#25819)
Make grid view group/mapped summary UI more consistent (#25723)
Remove useless statement in task_group_to_grid (#25654)
Add optional data interval to CronTriggerTimetable (#25503)
Remove unused code in /grid endpoint (#25481)
Add and document description fields (#25370)
Improve Airflow logging for operator Jinja template processing (#25452)
Update core example DAGs to use @task.branch decorator (#25242)
Update DAG audit_log route (#25415)
Change stdout and stderr access mode to append in commands (#25253)
Remove getTasks from Grid view (#25359)
Improve taskflow type hints with ParamSpec (#25173)
Use tables in grid details panes (#25258)
Explicitly list @dag arguments (#25044)
More typing in SchedulerJob and TaskInstance (#24912)
Patch getfqdn with more resilient version (#24981)
Replace all NBSP characters by whitespaces (#24797)
Re-serialize all DAGs on airflow db upgrade (#24518)
Rework contract of try_adopt_task_instances method (#23188)
Make expand() error vague so it's not misleading (#24018)
Add enum validation for [webserver]analytics_tool (#24032)
Add dttm searchable field in audit log (#23794)
Allow more parameters to be piped through via execute_in_subprocess (#23286)
Use func.count to count rows (#23657)
Remove stale serialized dags (#22917)
AIP45 Remove dag parsing in airflow run local (#21877)
Add support for queued state in DagRun update endpoint. (#23481)
Add fields to dagrun endpoint (#23440)
Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

Have consistent types between the ORM and the migration files (#24044, #25869)
Disallow any dag tags longer than 100 char (#25196)
Add the dag_id to AirflowDagCycleException message (#26204)
Properly build URL to retrieve logs independently from system (#26337)
For worker log servers only bind to IPV6 when dual stack is available (#26222)
Fix TaskInstance.task not defined before handle_failure (#26040)
Undo secrets backend config caching (#26223)
Fix faulty executor config serialization logic (#26191)
Show DAGs and Datasets menu links based on role permission (#26183)
Allow setting TaskGroup tooltip via function docstring (#26028)
Fix RecursionError on graph view of a DAG with many tasks (#26175)
Fix backfill occasional deadlocking (#26161)
Fix DagRun.start_date not set during backfill with --reset-dagruns True (#26135)
Use label instead of id for dynamic task labels in graph (#26108)
Don't fail DagRun when leaf mapped_task is SKIPPED (#25995)
Add group prefix to decorated mapped task (#26081)
Fix UI flash when triggering with dup logical date (#26094)
Fix Make items nullable for TaskInstance related endpoints to avoid API errors (#26076)
Fix BranchDateTimeOperator to be timezone-awreness-insensitive (#25944)
Fix legacy timetable schedule interval params (#25999)
Fix response schema for list-mapped-task-instance (#25965)
Properly check the existence of missing mapped TIs (#25788)
Fix broken auto-refresh on grid view (#25950)
Use per-timetable ordering in grid UI (#25880)
Rewrite recursion when parsing DAG into iteration (#25898)
Find cross-group tasks in iter_mapped_dependants (#25793)
Fail task if mapping upstream fails (#25757)
Support / in variable get endpoint (#25774)
Use cfg default_wrap value for grid logs (#25731)
Add origin request args when triggering a run (#25729)
Operator name separate from class (#22834)
Fix incorrect data interval alignment due to assumption on input time alignment (#22658)
Return None if an XComArg fails to resolve (#25661)
Correct json arg help in airflow variables set command (#25726)
Added MySQL index hint to use ti_state on find_zombies query (#25725)
Only excluded actually expanded fields from render (#25599)
Grid, fix toast for axios errors (#25703)
Fix UI redirect (#26409)
Require dag_id arg for dags list-runs (#26357)
Check for queued states for dags auto-refresh (#25695)
Fix upgrade code for the dag_owner_attributes table (#25579)
Add map index to task logs api (#25568)
Ensure that zombie tasks for dags with errors get cleaned up (#25550)
Make extra link work in UI (#25500)
Sync up plugin API schema and definition (#25524)
First/last names can be empty (#25476)
Refactor DAG pages to be consistent (#25402)
Check expand_kwargs() input type before unmapping (#25355)
Filter XCOM by key when calculating map lengths (#24530)
Fix ExternalTaskSensor not working with dynamic task (#25215)
Added exception catching to send default email if template file raises any exception (#24943)
Bring MappedOperator members in sync with BaseOperator (#24034)

Misc/Internal

Add automatically generated ERD schema for the MetaData DB (#26217)
Mark serialization functions as internal (#26193)
Remove remaining deprecated classes and replace them with PEP562 (#26167)
Move dag_edges and task_group_to_dict to corresponding util modules (#26212)
Lazily import many modules to improve import speed (#24486, #26239)
FIX Incorrect typing information (#26077)
Add missing contrib classes to deprecated dictionaries (#26179)
Re-configure/connect the ORM after forking to run a DAG processor (#26216)
Remove cattrs from lineage processing. (#26134)
Removed deprecated contrib files and replace them with PEP-562 getattr (#26153)
Make BaseSerialization.serialize "public" to other classes. (#26142)
Change the template to use human readable task_instance description (#25960)
Bump moment-timezone from 0.5.34 to 0.5.35 in /airflow/www (#26080)
Fix Flask deprecation warning (#25753)
Add CamelCase to generated operations types (#25887)
Fix migration issues and tighten the CI upgrade/downgrade test (#25869)
Fix type annotations in SkipMixin (#25864)
Workaround setuptools editable packages path issue (#25848)
Bump undici from 5.8.0 to 5.9.1 in /airflow/www (#25801)
Add custom_operator_name attr to _BranchPythonDecoratedOperator (#25783)
Clarify filename_template deprecation message (#25749)
Use ParamSpec to replace ... in Callable (#25658)
Remove deprecated modules (#25543)
Documentation on task mapping additions (#24489)
Remove Smart Sensors (#25507)
Fix elasticsearch test config to avoid warning on deprecated template (#25520)
Bump terser from 4.8.0 to 4.8.1 in /airflow/ui (#25178)
Generate typescript types from rest API docs (#25123)
Upgrade utils files to typescript (#25089)
Upgrade remaining context file to typescript. (#25096)
Migrate files to ts (#25267)
Upgrade grid Table component to ts. (#25074)
Skip mapping against mapped ti if it returns None (#25047)
Refactor js file structure (#25003)
Move mapped kwargs introspection to separate type (#24971)
Only assert stuff for mypy when type checking (#24937)
Bump moment from 2.29.3 to 2.29.4 in /airflow/www (#24885)
Remove "bad characters" from our codebase (#24841)
Remove xcom_push flag from BashOperator (#24824)
Move Flask hook registration to end of file (#24776)
Upgrade more javascript files to typescript (#24715)
Clean up task decorator type hints and docstrings (#24667)
Preserve original order of providers' connection extra fields in UI (#24425)
Rename charts.css to chart.css (#24531)
Rename grid.css to chart.css (#24529)
Misc: create new process group by set_new_process_group utility (#24371)
Airflow UI fix Prototype Pollution (#24201)
Bump moto version (#24222)
Remove unused [github_enterprise] from ref docs (#24033)
Clean up f-strings in logging calls (#23597)
Add limit for JPype1 (#23847)
Simply json responses (#25518)
Add min attrs version (#26408)

Doc only changes

Add url prefix setting for Celery Flower (#25986)
Updating deprecated configuration in examples (#26037)
Fix wrong link for taskflow tutorial (#26007)
Reorganize tutorials into a section (#25890)
Fix concept doc for dynamic task map (#26002)
Update code examples from "classic" operators to taskflow (#25845, #25657)
Add instructions on manually fixing MySQL Charset problems (#25938)
Prefer the local Quick Start in docs (#25888)
Fix broken link to Trigger Rules (#25840)
Improve docker documentation (#25735)
Correctly link to Dag parsing context in docs (#25722)
Add note on task_instance_mutation_hook usage (#25607)
Note that TaskFlow API automatically passes data between tasks (#25577)
Update DAG run to clarify when a DAG actually runs (#25290)
Update tutorial docs to include a definition of operators (#25012)
Rewrite the Airflow documentation home page (#24795)
Fix task-generated mapping example (#23424)
Add note on subtle logical date change in 2.2.0 (#24413)
Add missing import in best-practices code example (#25391)

airflow - Apache Airflow 2.3.4

Published by ephraimbuddy about 2 years ago

Bug Fixes

Disable attrs state management on MappedOperator (#24772)
Serialize pod_override to JSON before pickling executor_config (#24356)
Fix pid check (#24636)
Rotate session id during login (#25771)
Fix mapped sensor with reschedule mode (#25594)
Cache the custom secrets backend so the same instance gets re-used (#25556)
Add right padding (#25554)
Fix reducing mapped length of a mapped task at runtime after a clear (#25531)
Fix airflow db reset when dangling tables exist (#25441)
Change disable_verify_ssl behaviour (#25023)
Set default task group in dag.add_task method (#25000)
Removed interfering force of index. (#25404)
Remove useless logging line (#25347)
Adding mysql index hint to use index on task_instance.state in critical section query (#25673)
Configurable umask to all daemonized processes. (#25664)
Fix the errors raised when None is passed to template filters (#25593)
Allow wildcarded CORS origins (#25553)
Fix "This Session's transaction has been rolled back" (#25532)
Fix Serialization error in TaskCallbackRequest (#25471)
fix - resolve bash by absolute path (#25331)
Add __repr__ to ParamsDict class (#25305)
Only load distribution of a name once (#25296)
convert TimeSensorAsync target_time to utc on call time (#25221)
call updateNodeLabels after expandGroup (#25217)
Stop SLA callbacks gazumping other callbacks and DOS'ing the DagProcessorManager queue (#25147)
Fix invalidateQueries call (#25097)
airflow/www/package.json: Add name, version fields. (#25065)
No grid auto-refresh for backfill dag runs (#25042)
Fix tag link on dag detail page (#24918)
Fix zombie task handling with multiple schedulers (#24906)
Bind log server on worker to IPv6 address (#24755) (#24846)
Add %z for %(asctime)s to fix timezone for logs on UI (#24811)
TriggerDagRunOperator.operator_extra_links is attr (#24676)
Send DAG timeout callbacks to processor outside of prohibit_commit (#24366)
Don't rely on current ORM structure for db clean command (#23574)
Clear next method when clearing TIs (#23929)
Two typing fixes (#25690)

Doc only changes

Update set-up-database.rst (#24983)
Fix syntax in mysql setup documentation (#24893 (#24939)
Note how DAG policy works with default_args (#24804)
Update PythonVirtualenvOperator Howto (#24782)
Doc: Add hyperlinks to Github PRs for Release Notes (#24532)

Misc/Internal

Remove depreciation warning when use default remote tasks logging handlers (#25764)
clearer method name in scheduler_job.py (#23702)
Bump cattrs version (#25689)
Include missing mention of external_executor_id in sql_engine_collation_for_ids docs (#25197)
Refactor DR.task_instance_scheduling_decisions (#24774)
Sort operator extra links (#24992)
Extends resolve_xcom_backend function level documentation (#24965)
Upgrade FAB to 4.1.3 (#24884)
Limit Flask to <2.3 in the wake of 2.2 breaking our tests (#25511)
Limit astroid version to < 2.12 (#24982)
Move javascript compilation to host (#25169)
Bump typing-extensions and mypy for ParamSpec (#25088)

airflow - Apache Airflow 2.3.3

Published by ephraimbuddy over 2 years ago

Bug Fixes

Fix exception in mini task scheduler (#24865)
Fix cycle bug with attaching label to task group (#24847)
Fix timestamp defaults for sensorinstance (#24638)
Move fallible ti.task.dag assignment back inside try/except block (#24533) (#24592)
Add missing types to FSHook (#24470)
Mask secrets in stdout for airflow tasks test (#24362)
DebugExecutor use ti.run() instead of ti._run_raw_task (#24357)
Fix bugs in URI constructor for MySQL connection (#24320)
Missing scheduleinterval nullable true added in openapi (#24253)
Unify return_code interface for task runner (#24093)
Handle occasional deadlocks in trigger with retries (#24071)
Remove special serde logic for mapped op_kwargs (#23860)
ExternalTaskSensor respects soft_fail if the external task enters a failed_state (#23647)
Fix StatD timing metric units (#21106)
Add cache_ok flag to sqlalchemy TypeDecorators. (#24499)
Allow for LOGGING_LEVEL=DEBUG (#23360)
Fix grid date ticks (#24738, #24849)
Debounce status highlighting in Grid view (#24710)
Fix Grid vertical scrolling (#24684)
don't try to render child rows for closed groups (#24637)
Do not calculate grid root instances (#24528)
Maintain grid view selection on filtering upstream (#23779)
Speed up grid_data endpoint by 10x (#24284)
Apply per-run log templates to log handlers (#24153)
Don't crash scheduler if exec config has old k8s objects (#24117)
TI.log_url fix for map_index (#24335)
Fix migration 0080_2_0_2 - Replace null values before setting column not null (#24585)
Patch sql_alchemy_conn if old Postgres schemes used (#24569)
Seed log_template table (#24511)
Fix deprecated log_id_template value (#24506)
Fix toast messages (#24505)
Add indexes for CASCADE deletes for task_instance (#24488)
Return empty dict if Pod JSON encoding fails (#24478)
Improve grid rendering performance with a custom tooltip (#24417, #24449)
Check for run_id for grid group summaries (#24327)
Optimize calendar view for cron scheduled DAGs (#24262)
Use get_hostname instead of socket.getfqdn (#24260)
Check that edge nodes actually exist (#24166)
Fix useTasks crash on error (#24152)
Do not fail re-queued TIs (#23846)
Reduce grid view API calls (#24083)
Rename Permissions to Permission Pairs. (#24065)
Replace use_task_execution_date with use_task_logical_date (#23983)
Grid fix details button truncated and small UI tweaks (#23934)
Add TaskInstance State REMOVED to finished states and success states (#23797)
Fix mapped task immutability after clear (#23667)
Fix permission issue for dag that has dot in name (#23510)
Fix closing connection dbapi.get_pandas_df (#23452)
Check bag DAG schedule_interval match timetable (#23113)
Parse error for task added to multiple groups (#23071)
Fix flaky order of returned dag runs (#24405)
Migrate jsx files that affect run/task selection to tsx (#24509)
Fix links to sources for examples (#24386)
Set proper Content-Type and chartset on grid_data endpoint (#24375)

Doc only changes

Fix doc description of [core] parallelism config setting (#23768)
Update templates doc to mention extras and format Airflow Vars / Conns (#24735)
Document built in Timetables (#23099)
Alphabetizes two tables (#23923)
Clarify that users should not use Maria DB (#24556)
Add imports to deferring code samples (#24544)
Add note about image regeneration in June 2022 (#24524)
Small cleanup of get_current_context() chapter (#24482)
Fix default 2.2.5 log_id_template (#24455)
Update description of installing providers separately from core (#24454)
Mention context variables and logging (#24304)

Misc/Internal

Do not support MSSQL less than v2017 in code (#24095)
Drop Python 3.6 compatibility objects/modules (#24048)
Remove upper-binding for SQLAlchemy (#24819)
Remove internet explorer support (#24495)
Removing magic status code numbers from api_connexion (#24050)
Upgrade FAB to 4.1.2 (#24619)
Switch Markdown engine to markdown-it-py (#19702)
Update rich to latest version across the board. (#24186)
Get rid of TimedJSONWebSignatureSerializer (#24519)
Update flask-appbuilder authlib/ oauth dependency (#24516)
Upgrade to webpack 5 (#24485)
Add typescript (#24337)
The JWT claims in the request to retrieve logs have been standardized: we use nbf and aud claims for
maturity and audience of the requests. Also "filename" payload field is used to keep log name. (#24519)
Address all yarn test warnings (#24722)
Upgrade to react 18 and chakra 2 (#24430)
Refactor DagRun.verify_integrity (#24114)
Upgrade FAB to 4.1.1 (#24399)
We now need at least Flask-WTF 0.15 (#24621)

airflow - Apache Airflow 2.3.2

Published by ephraimbuddy over 2 years ago

Bug Fixes

Run the check_migration loop at least once
Fix grid view for mapped tasks (#24059)
Icons in grid view for different DAG run types (#23970)
Faster grid view (#23951)
Disallow calling expand with no arguments (#23463)
Add missing is_mapped field to Task response. (#23319)
DagFileProcessorManager: Start a new process group only if current process not a session leader (#23872)
Mask sensitive values for not-yet-running TIs (#23807)
Add cascade to dag_tag to dag foreign key (#23444)
Use --subdir argument value for standalone dag processor. (#23864)
Highlight task states by hovering on legend row (#23678)
Fix and speed up grid view (#23947)
Prevent UI from crashing if grid task instances are null (#23939)
Remove redundant register exit signals in dag-processor command (#23886)
Add __wrapped__ property to _TaskDecorator (#23830)
Fix UnboundLocalError when sql is empty list in DbApiHook (#23816)
Enable clicking on DAG owner in autocomplete dropdown (#23804)
Simplify flash message for _airflow_moved tables (#23635)
Exclude missing tasks from the gantt view (#23627)

Doc only changes

Add column names for DB Migration Reference (#23853)

Misc/Internal

Remove pinning for xmltodict (#23992)

airflow - Apache Airflow 2.3.1

Published by ephraimbuddy over 2 years ago

Bug Fixes

Automatically reschedule stalled queued tasks in CeleryExecutor (#23690)
Fix expand/collapse all buttons (#23590)
Grid view status filters (#23392)
Expand/collapse all groups (#23487)
Fix retrieval of deprecated non-config values (#23723)
Fix secrets rendered in UI when task is not executed. (#22754)
Fix provider import error matching (#23825)
Fix regression in ignoring symlinks (#23535)
Fix dag-processor fetch metadata database config (#23575)
Fix auto upstream dep when expanding non-templated field (#23771)
Fix task log is not captured (#23684)
Add reschedule to the serialized fields for the BaseSensorOperator (#23674)
Modify db clean to also catch the ProgrammingError exception (#23699)
Remove titles from link buttons (#23736)
Fix grid details header text overlap (#23728)
Ensure execution_timeout as timedelta (#23655)
Don't run pre-migration checks for downgrade (#23634)
Add index for event column in log table (#23625)
Implement send_callback method for CeleryKubernetesExecutor and LocalKubernetesExecutor (#23617)
Fix PythonVirtualenvOperator templated_fields (#23559)
Apply specific ID collation to root_dag_id too (#23536)
Prevent KubernetesJobWatcher getting stuck on resource too old (#23521)
Fix scheduler crash when expanding with mapped task that returned none (#23486)
Fix broken dagrun links when many runs start at the same time (#23462)
Fix: Exception when parsing log #20966 (#23301)
Handle invalid date parsing in webserver views. (#23161)
Pools with negative open slots should not block other pools (#23143)
Move around overflow, position and padding (#23044)
Change approach to finding bad rows to LEFT OUTER JOIN. (#23528)
Only count bad refs when moved table exists (#23491)
Visually distinguish task group summary (#23488)
Remove color change for highly nested groups (#23482)
Optimize 2.3.0 pre-upgrade check queries (#23458)
Add backward compatibility for core__sql_alchemy_conn__cmd (#23441)
Fix literal cross product expansion (#23434)
Fix broken task instance link in xcom list (#23367)
Fix connection test button (#23345)
fix cli airflow dags show for mapped operator (#23339)
Hide some task instance attributes (#23338)
Don't show grid actions if server would reject with permission denied (#23332)
Use run_id for ti.mark_success_url (#23330)
Fix update user auth stats (#23314)
Use <Time /> in Mapped Instance table (#23313)
Fix duplicated Kubernetes DeprecationWarnings (#23302)
Store grid view selection in url params (#23290)
Remove custom signal handling in Triggerer (#23274)
Override pool for TaskInstance when pool is passed from cli. (#23258)
Show warning if '/' is used in a DAG run ID (#23106)
Use kubernetes queue in kubernetes hybrid executors (#23048)
Add tags inside try block. (#21784)

Doc only changes

Move dag_processing.processor_timeouts to counters section (#23393)
Clarify that bundle extras should not be used for PyPi installs (#23697)
Synchronize support for Postgres and K8S in docs (#23673)
Replace DummyOperator references in docs (#23502)
Add doc notes for keyword-only args for expand() and partial() (#23373)
Document fix for broken elasticsearch logs with 2.3.0+ upgrade (#23821)

Misc/Internal

Add typing for airflow/configuration.py (#23716)
Disable Flower by default from docker-compose (#23685)
Added postgres 14 to support versions(including breeze) (#23506)
add K8S 1.24 support (#23637)
Refactor code references from tree to grid (#23254)

Package Rankings

Top 18.39% on Anaconda.org

Top 0.29% on Pypi.org

Top 8.25% on Conda-forge.org

Badges

Extracted from project README

Related Projects

devops-resources

DevOps resources - Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes...

07 Jan 2019 8,538

terraform-aws-ecs-codepipeline

Terraform Module for CI/CD with AWS Code Pipeline and Code Build for ECS https://cloudposse.com/

07 Jun 2018 154

airflow-ubuntu-ansible-terraform

Demo deployment of Apache Airflow on EC2/Ubuntu with Docker-compose, Terraform, and Ansible

09 Sep 2017 11

apisix

The Cloud-Native API Gateway

10 Apr 2019 13,769

airflow

Bug Fixes

Misc/Internal

Doc only changes

Significant Changes

Default Airflow image is updated to 2.6.2 (#31979)

New Features

Improvements

Bug Fixes

Misc

Bug Fixes

Misc/Internal

Doc only changes

Significant Changes

Clarifications of the external Health Check mechanism and using Job classes (#31277).

Bug Fixes

Misc/Internal

Doc only changes

Significant Changes

Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506).

SLA callbacks no longer add files to the dag processor manager's queue (#30076)

The cleanup() method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152).

The gauge scheduler.tasks.running no longer exist (#30374)

Consolidate handling of tasks stuck in queued under new task_queued_timeout config (#30375)

Improvement Changes

Display only the running configuration in configurations view (#28892)

Explicit skipped states list for ExternalTaskSensor (#29933)

Miscellaneous Changes

Handle OverflowError on exponential backoff in next_run_calculation (#28172)

Move Hive macros to the provider (#28538)

New Features

Improvements

Bug Fixes

Misc/Internal

Doc only changes

Significant Changes

Default PgBouncer and PgBouncer Exporter images have been updated (#29919)

Default Airflow image is updated to 2.5.3 (#30411)

New Features

Improvements

Bug Fixes

Doc only changes

Misc

Significant Changes

Bug Fixes

Misc/Internal

Doc only changes

Significant Changes

The date-time fields passed as API parameters or Params should be RFC3339-compliant (#29395)

Default for [webserver] expose_hostname changed to False (#29547)

Bug Fixes

Misc/Internal

Doc Only Changes

Significant Changes

bitnami/postgresql subchart updated to 12.1.9 (#29071)

Default dags.gitSync.wait reduced to 5 seconds (#27625)

Default Airflow image is updated to 2.5.1 (#29074)

Default git-sync image is updated to 3.6.3 (#27848)

Default redis image is updated to 7-bullseye (#27443)

New Features

Improvements

Bug Fixes

Doc only changes

Misc

Bug Fixes

Misc/Internal

Doc Only Changes

New Features

Improvements

Bug Fixes

Misc/Internal

Doc only changes

Bug Fixes

Misc/Internal

Doc only changes

Bug Fixes

Misc/Internal

Doc only changes

New Features

Improvements

Default Airflow image is updated to `2.6.2` (#31979)

Clarifications of the external Health Check mechanism and using `Job` classes (#31277).

The `cleanup()` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152).

The gauge `scheduler.tasks.running` no longer exist (#30374)

Consolidate handling of tasks stuck in queued under new `task_queued_timeout` config (#30375)

Default Airflow image is updated to `2.5.3` (#30411)

Default for `[webserver] expose_hostname` changed to `False` (#29547)

`bitnami/postgresql` subchart updated to `12.1.9` (#29071)

Default `dags.gitSync.wait` reduced to `5` seconds (#27625)

Default Airflow image is updated to `2.5.1` (#29074)

Default git-sync image is updated to `3.6.3` (#27848)

Default redis image is updated to `7-bullseye` (#27443)