airflow | Docker Ecosystem Directory

Bot releases are visible (Hide)

airflow -

Published by ashb over 3 years ago

New Features
""""""""""""

Add PythonVirtualenvDecorator to Taskflow API (#14761)
Add Taskgroup decorator (#15034)
Create a DAG Calendar View (#15423)
Create cross-DAG dependencies view (#13199)
Add rest API to query for providers (#13394)
Mask passwords and sensitive info in task logs and UI (#15599)
Add SubprocessHook for running commands from operators (#13423)
Add DAG Timeout in UI page "DAG Details" (#14165)
Add WeekDayBranchOperator (#13997)
Add JSON linter to DAG Trigger UI (#13551)
Add DAG Description Doc to Trigger UI Page (#13365)
Add airflow webserver URL into SLA miss email. (#13249)
Add read only REST API endpoints for users (#14735)
Add files to generate Airflow's Python SDK (#14739)
Add dynamic fields to snowflake connection (#14724)
Add read only REST API endpoint for roles and permissions (#14664)
Add new datetime branch operator (#11964)
Add Google leveldb hook and operator (#13109) (#14105)
Add plugins endpoint to the REST API (#14280)
Add worker_pod_pending_timeout support (#15263)
Add support for labeling DAG edges (#15142)
Add CUD REST API endpoints for Roles (#14840)
Import connections from a file (#15177)
A bunch of template_fields_renderers additions (#15130)
Add REST API query sort and order to some endpoints (#14895)
Add timezone context in new ui (#15096)
Add query mutations to new UI (#15068)
Add different modes to sort dag files for parsing (#15046)
Auto refresh on Tree View (#15474)
BashOperator to raise AirflowSkipException on exit code 99 (by default, configurable) (#13421) (#14963)
Clear tasks by task ids in REST API (#14500)
Support jinja2 native Python types (#14603)
Allow celery workers without gossip or mingle modes (#13880)
Add airflow jobs check CLI command to check health of jobs (Scheduler etc) (#14519)
Rename DateTimeBranchOperator to BranchDateTimeOperator (#14720)

Improvements
""""""""""""

Add optional result handler callback to DbApiHook (#15581)
Update Flask App Builder limit to recently released 3.3 (#15792)
Prevent creating flask sessions on REST API requests (#15295)
Sync DAG specific permissions when parsing (#15311)
Increase maximum length of pool name on Tasks to 256 characters (#15203)
Enforce READ COMMITTED isolation when using mysql (#15714)
Auto-apply apply_default to subclasses of BaseOperator (#15667)
Emit error on duplicated DAG ID (#15302)
Update KubernetesExecutor pod templates to allow access to IAM permissions (#15669)
More verbose logs when running airflow db check-migrations (#15662)
When one_success mark task as failed if no success (#15467)
Add an option to trigger a dag w/o changing conf (#15591)
Add Airflow UI instance_name configuration option (#10162)
Add a decorator to retry functions with DB transactions (#14109)
Add return to PythonVirtualenvOperator's execute method (#14061)
Add verify_ssl config for kubernetes (#13516)
Add description about secret_key when Webserver > 1 (#15546)
Add Traceback in LogRecord in JSONFormatter (#15414)
Add support for arbitrary json in conn uri format (#15100)
Adds description field in variable (#12413) (#15194)
Add logs to show last modified in SFTP, FTP and Filesystem sensor (#15134)
Execute on_failure_callback when SIGTERM is received (#15172)
Allow hiding of all edges when highlighting states (#15281)
Display explicit error in case UID has no actual username (#15212)
Serve logs with Scheduler when using Local or Sequential Executor (#15557)
Deactivate trigger, refresh, and delete controls on dag detail view. (#14144)
Turn off autocomplete for connection forms (#15073)
Increase default worker_refresh_interval to 6000 seconds (#14970)
Only show User's local timezone if it's not UTC (#13904)
Suppress LOG/WARNING for a few tasks CLI for better CLI experience (#14567)
Configurable API response (CORS) headers (#13620)
Allow viewers to see all docs links (#14197)
Update Tree View date ticks (#14141)
Make the tooltip to Pause / Unpause a DAG clearer (#13642)
Warn about precedence of env var when getting variables (#13501)
Move [celery] default_queue config to [operators] default_queue to re-use between executors (#14699)

Bug Fixes
"""""""""

Fix 500 error from updateTaskInstancesState API endpoint when dry_run not passed (#15889)
Ensure that task preceding a PythonVirtualenvOperator doesn't fail (#15822)
Prevent mixed case env vars from crashing processes like worker (#14380)
Fixed type annotations in DAG decorator (#15778)
Fix on_failure_callback when task receive SIGKILL (#15537)
Fix dags table overflow (#15660)
Fix changing the parent dag state on subdag clear (#15562)
Fix reading from zip package to default to text (#13962)
Fix wrong parameter for drawDagStatsForDag in dags.html (#13884)
Fix QueuedLocalWorker crashing with EOFError (#13215)
Fix typo in NotPreviouslySkippedDep (#13933)
Fix parallelism after KubeExecutor pod adoption (#15555)
Fix kube client on mac with keepalive enabled (#15551)
Fixes wrong limit for dask for python>3.7 (should be <3.7) (#15545)
Fix Task Adoption in KubernetesExecutor (#14795)
Fix timeout when using XCom with KubernetesPodOperator (#15388)
Fix deprecated provider aliases in "extras" not working (#15465)
Fixed default XCom deserialization. (#14827)
Fix used_group_ids in dag.partial_subset (#13700) (#15308)
Further fix trimmed pod_id for KubernetesPodOperator (#15445)
Bugfix: Invalid name when trimmed pod_id ends with hyphen in KubernetesPodOperator (#15443)
Fix incorrect slots stats when TI pool_slots > 1 (#15426)
Fix DAG last run link (#15327)
Fix sync-perm to work correctly when update_fab_perms = False (#14847)
Fixes limits on Arrow for plexus test (#14781)
Fix UI bugs in tree view (#14566)
Fix AzureDataFactoryHook failing to instantiate its connection (#14565)
Fix permission error on non-POSIX filesystem (#13121)
Fix spelling in "ignorable" (#14348)
Fix get_context_data doctest import (#14288)
Correct typo in GCSObjectsWtihPrefixExistenceSensor (#14179)
Fix order of failed deps (#14036)
Fix critical CeleryKubernetesExecutor bug (#13247)
Fix four bugs in StackdriverTaskHandler (#13784)
func.sum may return Decimal that break rest APIs (#15585)
Persist tags params in pagination (#15411)
API: Raise AlreadyExists exception when the execution_date is same (#15174)
Remove duplicate call to sync_metadata inside DagFileProcessorManager (#15121)
Extra docker-py update to resolve docker op issues (#15731)
Ensure executors end method is called (#14085)
Remove user_id from API schema (#15117)
Prevent clickable bad links on disabled pagination (#15074)
Acquire lock on db for the time of migration (#10151)
Skip SLA check only if SLA is None (#14064)
Print right version in airflow info command (#14560)
Make airflow info work with pipes (#14528)
Rework client-side script for connection form. (#14052)
API: Add CollectionInfo in all Collections that have total_entries (#14366)
Fix task_instance_mutation_hook when importing airflow.models.dagrun (#15851)

Doc only changes
""""""""""""""""

Fix docstring of SqlSensor (#15466)
Small changes on "DAGs and Tasks documentation" (#14853)
Add note on changes to configuration options (#15696)
Add docs to the markdownlint and yamllint config files (#15682)
Rename old "Experimental" API to deprecated in the docs. (#15653)
Fix documentation error in git_sync_template.yaml (#13197)
Fix doc link permission name (#14972)
Fix link to Helm chart docs (#14652)
Fix docstrings for Kubernetes code (#14605)
docs: Capitalize & minor fixes (#14283) (#14534)
Fixed reading from zip package to default to text. (#13984)
An initial rework of the "Concepts" docs (#15444)
Improve docstrings for various modules (#15047)
Add documentation on database connection URI (#14124)
Add Helm Chart logo to docs index (#14762)
Create a new documentation package for Helm Chart (#14643)
Add docs about supported logging levels (#14507)
Update docs about tableau and salesforce provider (#14495)
Replace deprecated doc links to the correct one (#14429)
Refactor redundant doc url logic to use utility (#14080)
docs: NOTICE: Updated 2016-2019 to 2016-now (#14248)
Skip DAG perm sync during parsing if possible (#15464)
Add picture and examples for Edge Labels (#15310)
Add example DAG & how-to guide for sqlite (#13196)
Add links to new modules for deprecated modules (#15316)
Add note in Updating.md about FAB data model change (#14478)

Misc/Internal
"""""""""""""

Fix logging.exception redundancy (#14823)
Bump stylelint to remove vulnerable sub-dependency (#15784)
Add resolution to force dependencies to use patched version of lodash (#15777)
Update croniter to 1.0.x series (#15769)
Get rid of Airflow 1.10 in Breeze (#15712)
Run helm chart tests in parallel (#15706)
Bump ssri from 6.0.1 to 6.0.2 in /airflow/www (#15437)
Remove the limit on Gunicorn dependency (#15611)
Better "dependency already registered" warning message for tasks #14613 (#14860)
Pin pandas-gbq to <0.15.0 (#15114)
Use Pip 21.* to install airflow officially (#15513)
Bump mysqlclient to support the 1.4.x and 2.x series (#14978)
Finish refactor of DAG resource name helper (#15511)
Refactor/Cleanup Presentation of Graph Task and Path Highlighting (#15257)
Standardize default fab perms (#14946)
Remove datepicker for task instance detail view (#15284)
Turn provider's import warnings into debug logs (#14903)
Remove left-over fields from required in provider_info schema. (#14119)
Deprecate tableau extra (#13595)
Use built-in cached_property on Python 3.8 where possible (#14606)
Clean-up JS code in UI templates (#14019)
Bump elliptic from 6.5.3 to 6.5.4 in /airflow/www (#14668)
Switch to f-strings using flynt. (#13732)
use jquery ready instead of vanilla js (#15258)
Migrate task instance log (ti_log) js (#15309)
Migrate graph js (#15307)
Migrate dags.html javascript (#14692)
Removes unnecessary AzureContainerInstance connection type (#15514)
Separate Kubernetes pod_launcher from core airflow (#15165)
update remaining old import paths of operators (#15127)
Remove broken and undocumented "demo mode" feature (#14601)
Simplify configuration/legibility of Webpack entries (#14551)
remove inline tree js (#14552)
Js linting and inline migration for simple scripts (#14215)
Remove use of repeated constant in AirflowConfigParser (#14023)
Deprecate email credentials from environment variables. (#13601)
Remove unused 'context' variable in task_instance.py (#14049)
Disable suppress_logs_and_warning in cli when debugging (#13180)

airflow -

Published by ashb over 3 years ago

Bug Fixes

Bugfix: TypeError when Serializing & sorting iterable properties of DAGs (#15395)
Fix missing on_load trigger for folder-based plugins (#15208)
kubernetes cleanup-pods subcommand will only clean up Airflow-created Pods (#15204)
Fix password masking in CLI action_logging (#15143)
Fix url generation for TriggerDagRunOperatorLink (#14990)
Restore base lineage backend (#14146)
Unable to trigger backfill or manual jobs with Kubernetes executor. (#14160)
Bugfix: Task docs are not shown in the Task Instance Detail View (#15191)
Bugfix: Fix overriding pod_template_file in KubernetesExecutor (#15197)
Bugfix: resources in executor_config breaks Graph View in UI (#15199)
Fix celery executor bug trying to call len on map (#14883)
Fix bug in airflow.stats timing that broke dogstatsd mode (#15132)
Avoid scheduler/parser manager deadlock by using non-blocking IO (#15112)
Re-introduce dagrun.schedule_delay metric (#15105)
Compare string values, not if strings are the same object in Kube executor(#14942)
Pass queue to BaseExecutor.execute_async like in airflow 1.10 (#14861)
Scheduler: Remove TIs from starved pools from the critical path. (#14476)
Remove extra/needless deprecation warnings from airflow.contrib module (#15065)
Fix support for long dag_id and task_id in KubernetesExecutor (#14703)
Sort lists, sets and tuples in Serialized DAGs (#14909)
Simplify cleaning string passed to origin param (#14738) (#14905)
Fix error when running tasks with Sentry integration enabled. (#13929)
Webserver: Sanitize string passed to origin param (#14738)
Fix losing duration < 1 secs in tree (#13537)
Pin SQLAlchemy to <1.4 due to breakage of sqlalchemy-utils (#14812)
Fix KubernetesExecutor issue with deleted pending pods (#14810)
Default to Celery Task model when backend model does not exist (#14612)
Bugfix: Plugins endpoint was unauthenticated (#14570)
BugFix: fix DAG doc display (especially for TaskFlow DAGs) (#14564)
BugFix: TypeError in airflow.kubernetes.pod_launcher's monitor_pod (#14513)
Bugfix: Fix wrong output of tags and owners in dag detail API endpoint (#14490)
Fix logging error with task error when JSON logging is enabled (#14456)
Fix statsd metrics not sending when using daemon mode (#14454)
Gracefully handle missing start_date and end_date for DagRun (#14452)
BugFix: Serialize max_retry_delay as a timedelta (#14436)
Fix crash when user clicks on "Task Instance Details" caused by start_date being None (#14416)
BugFix: Fix TaskInstance API call fails if a task is removed from running DAG (#14381)
Scheduler should not fail when invalid executor_config is passed (#14323)
Fix bug allowing task instances to survive when dagrun_timeout is exceeded (#14321)
Fix bug where DAG timezone was not always shown correctly in UI tooltips (#14204)
Use Lax for cookie_samesite when empty string is passed (#14183)
[AIRFLOW-6076] fix dag.cli() KeyError (#13647)
Fix running child tasks in a subdag after clearing a successful subdag (#14776)

Improvements

Remove unused JS packages causing false security alerts (#15383)
Change default of [kubernetes] enable_tcp_keepalive for new installs to True (#15338)
Fixed #14270: Add error message in OOM situations (#15207)
Better compatibility/diagnostics for arbitrary UID in docker image (#15162)
Updates 3.6 limits for latest versions of a few libraries (#15209)
Adds Blinker dependency which is missing after recent changes (#15182)
Remove 'conf' from search_columns in DagRun View (#15099)
More proper default value for namespace in K8S cleanup-pods CLI (#15060)
Faster default role syncing during webserver start (#15017)
Speed up webserver start when there are many DAGs (#14993)
Much easier to use and better documented Docker image (#14911)
Use libyaml C library when available. (#14577)
Don't create unittest.cfg when not running in unit test mode (#14420)
Webserver: Allow Filtering TaskInstances by queued_dttm (#14708)
Update Flask-AppBuilder dependency to allow 3.2 (and all 3.x series) (#14665)
Remember expanded task groups in browser local storage (#14661)
Add plain format output to cli tables (#14546)
Make airflow dags show command display TaskGroups (#14269)
Increase maximum size of extra connection field. (#12944)
Speed up clear_task_instances by doing a single sql delete for TaskReschedule (#14048)
Add more flexibility with FAB menu links (#13903)
Add better description and guidance in case of sqlite version mismatch (#14209)

Doc only changes

Add documentation create/update community providers (#15061)
Fix mistake and typos in airflow.utils.timezone docstrings (#15180)
Replace new url for Stable Airflow Docs (#15169)
Docs: Clarify behavior of delete_worker_pods_on_failure (#14958)
Create a documentation package for Docker image (#14846)
Multiple minor doc (OpenAPI) fixes (#14917)
Replace Graph View Screenshot to show Auto-refresh (#14571)

Misc/Internal

Import Connection lazily in hooks to avoid cycles (#15361)
Rename last_scheduler_run into last_parsed_time, and ensure it's updated in DB (#14581)
Make TaskInstance.pool_slots not nullable with a default of 1 (#14406)
Log migrations info in consistent way (#14158)

airflow - Apache Airflow 1.10.15, 2021-03-17

Published by kaxil over 3 years ago

Bug Fixes

Fix airflow db upgrade to upgrade db as intended (#13267)
Moved boto3 limitation to snowflake (#13286)
KubernetesExecutor should accept images from executor_config (#13074)
Scheduler should acknowledge active runs properly (#13803)
Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)
Include airflow/contrib/executors in the dist package
Pin Click version for Python 2.7 users
Ensure all statsd timers use millisecond values. (#10633)
[kubernetes_generate_dag_yaml] - Fix dag yaml generate function (#13816)
Fix airflow tasks clear cli command wirh --yes (#14188)
Fix permission error on non-POSIX filesystem (#13121) (#14383)
Fixed deprecation message for "variables" command (#14457)
BugFix: fix the delete_dag function of json_client (#14441)
Fix merging of secrets and configmaps for KubernetesExecutor (#14090)
Fix webserver exiting when gunicorn master crashes (#13470)
Bump ini from 1.3.5 to 1.3.8 in airflow/www_rbac
Bump datatables.net from 1.10.21 to 1.10.23 in airflow/www_rbac
Webserver: Sanitize string passed to origin param (#14738)
Make rbac_app's db.session use the same timezone with @provide_session (#14025)

Improvements

Adds airflow as viable docker command in official image (#12878)
StreamLogWriter: Provide (no-op) close method (#10885)
Add 'airflow variables list' command for 1.10.x transition version (#14462)

Doc only changes

Update URL for Airflow docs (#13561)
Clarifies version args for installing 1.10 in Docker (#12875)

airflow - Airflow 2.0.1, 2021-02-08

Published by kaxil over 3 years ago

Bug Fixes

Bugfix: Return XCom Value in the XCom Endpoint API (#13684)
Bugfix: Import error when using custom backend and sql_alchemy_conn_secret (#13260)
Allow PID file path to be relative when daemonize a process (scheduler, kerberos, etc) (#13232)
Bugfix: no generic DROP CONSTRAINT in MySQL during airflow db upgrade (#13239)
Bugfix: Sync Access Control defined in DAGs when running sync-perm (#13377)
Stop sending Callback Requests if no callbacks are defined on DAG (#13163)
BugFix: Dag-level Callback Requests were not run (#13651)
Stop creating duplicate Dag File Processors (#13662)
Filter DagRuns with Task Instances in removed State while Scheduling (#13165)
Bump datatables.net from 1.10.21 to 1.10.22 in /airflow/www (#13143)
Bump datatables.net JS to 1.10.23 (#13253)
Bump dompurify from 2.0.12 to 2.2.6 in /airflow/www (#13164)
Update minimum cattrs version (#13223)
Remove inapplicable arg 'output' for CLI pools import/export (#13071)
Webserver: Fix the behavior to deactivate the authentication option and add docs (#13191)
Fix: add support for no-menu plugin views (#11742)
Add python-daemon limit for python 3.8+ to fix daemon crash (#13540)
Change the default celery worker_concurrency to 16 (#13612)
Audit Log records View should not contain link if dag_id is None (#13619)
Fix invalid continue_token for cleanup list pods (#13563)
Switches to latest version of snowflake connector (#13654)
Fix backfill crash on task retry or reschedule (#13712)
Setting max_tis_per_query to 0 now correctly removes the limit (#13512)
Fix race conditions in task callback invocations (#10917)
Fix webserver exiting when gunicorn master crashes (#13518)(#13780)
Fix SQL syntax to check duplicate connections (#13783)
BaseBranchOperator will push to xcom by default (#13704) (#13763)
Fix Deprecation for configuration.getsection (#13804)
Fix TaskNotFound in log endpoint (#13872)
Fix race condition when using Dynamic DAGs (#13893)
Fix: Linux/Chrome window bouncing in Webserver
Fix db shell for sqlite (#13907)
Only compare updated time when Serialized DAG exists (#13899)
Fix dag run type enum query for mysqldb driver (#13278)
Add authentication to lineage endpoint for experimental API (#13870)
Do not add User role perms to custom roles. (#13856)
Do not add Website.can_read access to default roles. (#13923)
Fix invalid value error caused by long Kubernetes pod name (#13299)
Fix DB Migration for SQLite to upgrade to 2.0 (#13921)
Bugfix: Manual DagRun trigger should not skip scheduled runs (#13963)
Stop loading Extra Operator links in Scheduler (#13932)
Added missing return parameter in read function of FileTaskHandler (#14001)
Bugfix: Do not try to create a duplicate Dag Run in Scheduler (#13920)
Make v1/config endpoint respect webserver expose_config setting (#14020)
Disable row level locking for Mariadb and MySQL <8 (#14031)
Bugfix: Fix permissions to triggering only specific DAGs (#13922)
Fix broken SLA Mechanism (#14056)
Bugfix: Scheduler fails if task is removed at runtime (#14057)
Remove permissions to read Configurations for User and Viewer roles (#14067)
Fix DB Migration from 2.0.1rc1

Improvements

Increase the default min_file_process_interval to decrease CPU Usage (#13664)
Dispose connections when running tasks with os.fork & CeleryExecutor (#13265)
Make function purpose clearer in example_kubernetes_executor example dag (#13216)
Remove unused libraries - flask-swagger, funcsigs (#13178)
Display alternative tooltip when a Task has yet to run (no TI) (#13162)
User werkzeug's own type conversion for request args (#13184)
UI: Add queued_by_job_id & external_executor_id Columns to TI View (#13266)
Make json-merge-patch an optional library and unpin it (#13175)
Adds missing LDAP "extra" dependencies to ldap provider. (#13308)
Refactor setup.py to better reflect changes in providers (#13314)
Pin pyjwt and Add integration tests for Apache Pinot (#13195)
Removes provider-imposed requirements from setup.cfg (#13409)
Replace deprecated decorator (#13443)
Streamline & simplify __eq__ methods in models Dag and BaseOperator (#13449)
Additional properties should be allowed in provider schema (#13440)
Remove unused dependency - contextdecorator (#13455)
Remove 'typing' dependency (#13472)
Log migrations info in consistent way (#13458)
Unpin mysql-connector-python to allow 8.0.22 (#13370)
Remove thrift as a core dependency (#13471)
Add NotFound response for DELETE methods in OpenAPI YAML (#13550)
Stop Log Spamming when [core] lazy_load_plugins is False (#13578)
Display message and docs link when no plugins are loaded (#13599)
Unpin restriction for colorlog dependency (#13176)
Add missing Dag Tag for Example DAGs (#13665)
Support tables in DAG docs (#13533)
Add python3-openid dependency (#13714)
Add __repr__ for Executors (#13753)
Add description to hint if conn_type is missing (#13778)
Upgrade azure blob to v12 (#12188)
Add extra field to get_connnection REST endpoint (#13885)
Make Smart Sensors DB Migration idempotent (#13892)
Improve the error when DAG does not exist when running dag pause command (#13900)
Update airflow_local_settings.py to fix an error message (#13927)
Only allow passing JSON Serializable conf to TriggerDagRunOperator (#13964)
Bugfix: Allow getting details of a DAG with null start_date (REST API) (#13959)
Add params to the DAG details endpoint (#13790)
Make the role assigned to anonymous users customizable (#14042)
Retry critical methods in Scheduler loop in case of OperationalError (#14032)

Doc only changes

Add Missing Statsd Metrics in Docs (#13708)
Add Missing Email configs in Configuration doc (#13709)
Add quick start for Airflow on Docker (#13660)
Describe which Python versions are supported (#13259)
Add note block to 2.x migration docs (#13094)
Add documentation about webserver_config.py (#13155)
Add missing version information to recently added configs (#13161)
API: Use generic information in UpdateMask component (#13146)
Add Airflow 2.0.0 to requirements table (#13140)
Avoid confusion in doc for CeleryKubernetesExecutor (#13116)
Update docs link in REST API spec (#13107)
Add link to PyPI Repository to provider docs (#13064)
Fix link to Airflow master branch documentation (#13179)
Minor enhancements to Sensors docs (#13381)
Use 2.0.0 in Airflow docs & Breeze (#13379)
Improves documentation regarding providers and custom connections (#13375)(#13410)
Fix malformed table in production-deployment.rst (#13395)
Update celery.rst to fix broken links (#13400)
Remove reference to scheduler run_duration param in docs (#13346)
Set minimum SQLite version supported (#13412)
Fix installation doc (#13462)
Add docs about mocking variables and connections (#13502)
Add docs about Flask CLI (#13500)
Fix Upgrading to 2 guide to use rbac UI (#13569)
Make docs clear that Auth can not be disabled for Stable API (#13568)
Remove archived links from docs & add link for AIPs (#13580)
Minor fixes in upgrading-to-2.rst (#13583)
Fix Link in Upgrading to 2.0 guide (#13584)
Fix heading for Mocking section in best-practices.rst (#13658)
Add docs on how to use custom operators within plugins folder (#13186)
Update docs to register Operator Extra Links (#13683)
Improvements for database setup docs (#13696)
Replace module path to Class with just Class Name (#13719)
Update DAG Serialization docs (#13722)
Fix link to Apache Airflow docs in webserver (#13250)
Clarifies differences between extras and provider packages (#13810)
Add information about all access methods to the environment (#13940)
Docs: Fix FAQ on scheduler latency (#13969)
Updated taskflow api doc to show dependency with sensor (#13968)
Add deprecated config options to docs (#13883)
Added a FAQ section to the Upgrading to 2 doc (#13979)

airflow - Airflow 2.0.0, 2020-12-17

Published by kaxil almost 4 years ago

The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I’ll simply share some of the major features in 2.0.0 compared to 1.10.14:

A new way of writing dags: the TaskFlow API (AIP-31)

(Known in 2.0.0alphas as Functional DAGs.)

DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use

A quick teaser of what DAGs can now look like:

from airflow.decorators import dag, task
from airflow.utils.dates import days_ago

@dag(default_args={'owner': 'airflow'}, schedule_interval=None, start_date=days_ago(2))
def tutorial_taskflow_api_etl():
   @task
   def extract():
       return {"1001": 301.27, "1002": 433.21, "1003": 502.22}

   @task
   def transform(order_data_dict: dict) -> dict:
       total_order_value = 0

       for value in order_data_dict.values():
           total_order_value += value

       return {"total_order_value": total_order_value}

   @task()
   def load(total_order_value: float):

       print("Total order value is: %.2f" % total_order_value)

   order_data = extract()
   order_summary = transform(order_data)
   load(order_summary["total_order_value"])

tutorial_etl_dag = tutorial_taskflow_api_etl()

Fully specified REST API (AIP-32)

We now have a fully supported, no-longer-experimental API with a comprehensive OpenAPI specification

Massive Scheduler performance improvements

As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker.

Over at Astronomer.io we’ve benchmarked the scheduler—it’s fast (we had to triple check the numbers as we don’t quite believe them at first!)

Scheduler is now HA compatible (AIP-15)

It’s now possible and supported to run more than a single scheduler instance. This is super useful for both resiliency (in case a scheduler goes down) and scheduling performance.

To fully use this feature you need Postgres 9.6+ or MySQL 8+ (MySQL 5, and MariaDB won’t work with more than one scheduler I’m afraid).

There’s no config or other set up required to run more than one scheduler—just start up a scheduler somewhere else (ensuring it has access to the DAG files) and it will cooperate with your existing schedulers through the database.

For more information, read the Scheduler HA documentation.

Task Groups (AIP-34)

SubDAGs were commonly used for grouping tasks in the UI, but they had many drawbacks in their execution behaviour (primarirly that they only executed a single task in parallel!) To improve this experience, we’ve introduced “Task Groups”: a method for organizing tasks which provides the same grouping behaviour as a subdag without any of the execution-time drawbacks.

SubDAGs will still work for now, but we think that any previous use of SubDAGs can now be replaced with task groups. If you find an example where this isn’t the case, please let us know by opening an issue on GitHub

For more information, check out the Task Group documentation.

Refreshed UI

We’ve given the Airflow UI a visual refresh and updated some of the styling. Check out the UI section of the docs for screenshots.

We have also added an option to auto-refresh task states in Graph View so you no longer need to continuously press the refresh button :).

Smart Sensors for reduced load from sensors (AIP-17)

If you make heavy use of sensors in your Airflow cluster, you might find that sensor execution takes up a significant proportion of your cluster even with “reschedule” mode. To improve this, we’ve added a new mode called “Smart Sensors”.

This feature is in “early-access”: it’s been well-tested by AirBnB and is “stable”/usable, but we reserve the right to make backwards-incompatible changes to it in a future release (if we have to. We’ll try very hard not to!)

Simplified KubernetesExecutor

For Airflow 2.0, we have re-architected the KubernetesExecutor in a fashion that is simultaneously faster, easier to understand, and more flexible for Airflow users. Users will now be able to access the full Kubernetes API to create a .yaml pod_template_file instead of specifying parameters in their airflow.cfg.

We have also replaced the executor_config dictionary with the pod_override parameter, which takes a Kubernetes V1Pod object for a 1:1 setting override. These changes have removed over three thousand lines of code from the KubernetesExecutor, which makes it run faster and creates fewer potential errors.

Airflow core and providers: Splitting Airflow into 60+ packages

Airflow 2.0 is not a monolithic “one to rule them all” package. We’ve split Airflow into core and 61 (for now) provider packages. Each provider package is for either a particular external service (Google, Amazon, Microsoft, Snowflake), a database (Postgres, MySQL), or a protocol (HTTP/FTP). Now you can create a custom Airflow installation from “building” blocks and choose only what you need, plus add whatever other requirements you might have. Some of the common providers are installed automatically (ftp, http, imap, sqlite) as they are commonly used. Other providers are automatically installed when you choose appropriate extras when installing Airflow.

The provider architecture should make it much easier to get a fully customized, yet consistent runtime with the right set of Python dependencies.

But that’s not all: you can write your own custom providers and add things like custom connection types, customizations of the Connection Forms, and extra links to your operators in a manageable way. You can build your own provider and install it as a Python package and have your customizations visible right in the Airflow UI.

Security

As part of Airflow 2.0 effort, there has been a conscious focus on Security and reducing areas of exposure. This is represented across different functional areas in different forms. For example, in the new REST API, all operations now require authorization. Similarly, in the configuration settings, the Fernet key is now required to be specified.

Configuration

Configuration in the form of the airflow.cfg file has been rationalized further in distinct sections, specifically around “core”. Additionally, a significant amount of configuration options have been deprecated or moved to individual component-specific configuration files, such as the pod-template-file for Kubernetes execution-related configuration.

We’ve tried to make as few breaking changes as possible and to provide deprecation path in the code, especially in the case of anything called in the DAG. That said, please read through UPDATING.md to check what might affect you. For example: We re-organized the layout of operators (they now all live under airflow.providers.*) but the old names should continue to work - you’ll just notice a lot of DeprecationWarnings that need to be fixed up.

airflow - Airflow 1.10.14, 2020-12-10

Published by kaxil almost 4 years ago

Bug Fixes

BugFix: Tasks with depends_on_past or task_concurrency are stuck (#12663)
Fix issue with empty Resources in executor_config (#12633)
Fix: Deprecated config force_log_out_after was not used (#12661)
Fix empty asctime field in JSON formatted logs (#10515)
[AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY (#3651)
[AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac (#3729)
[AIRFLOW-2886] Generate random Flask SECRET_KEY in default config (#3738)
Add missing comma in setup.py (#12790)
Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)
Fix setup.py missing comma in setup_requires (#12880)
Don't emit first_task_scheduling_delay metric for only-once dags (#12835)

Improvements

Update setup.py to get non-conflicting set of dependencies (#12636)
Rename [scheduler] max_threads to [scheduler] parsing_processes (#12605)
Add metric for scheduling delay between first run task & expected start time (#9544)
Add new-style 2.0 command names for Airflow 1.10.x (#12725)
Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
Don't let webserver run with dangerous config (#12747)
Replace pkg_resources with importlib.metadata to avoid VersionConflict errors (#12694)

Doc only changes

Clarified information about supported Databases

airflow - Airflow 2.0.0b2

Published by ashb almost 4 years ago

Update versions in UPDATING.md for 2.0.0b1 release (#12244)
Remove Unnecessary comprehension (#12221)
Sign release files with an apache.org key by default (#12241)
Add back missing api_connextion/init.py file (#12240)
Fixes automated provider installation with extras (#12233)
Enable markdownlint rule - MD031/blanks-around-fences (#12238)
Fix spelling in Python files (#12230)
Add docs about security on GCP (#12187)
Enable Markdownlint rule - MD022/blanks-around-headings (#12225)
CSS Changes to adjust content width as per screen size and responsive table with multiline td. (#12227)

airflow - apache-airflow-upgrade-check 1.0.0

Published by ashb almost 4 years ago

The first release of apache-airflow-upgrade-check module.

Release based off the v1-10-stable branch.

📦 https://pypi.org/project/apache-airflow-upgrade-check/

airflow - Airflow 1.10.13, 2020-11-25

Published by kaxil almost 4 years ago

New Features

Add "already checked" to failed pods in K8sPodOperator (#11368)
Pass SQLAlchemy engine options to FAB based UI (#11395)
[AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
Add permission "extra_links" for Viewer role and above (#10719)
Add generate_yaml command to easily test KubernetesExecutor before deploying pods (#10677)
Add Secrets backend for Microsoft Azure Key Vault (#10898)

Bug Fixes

SkipMixin: Handle empty branches (#11120)
[AIRFLOW-5274] dag loading duration metric name too long (#5890)
Handle no Dagrun in DagrunIdDep (#8389) (#11343)
Fix Kubernetes Executor logs for long dag names (#10942)
Add on_kill support for the KubernetesPodOperator (#10666)
KubernetesPodOperator template fix (#10963)
Fix displaying of add serialized_dag table migration
Fix Start Date tooltip on DAGs page (#10637)
URL encode execution date in the Last Run link (#10595)
Fixes issue with affinity backcompat in Airflow 1.10
Fix KubernetesExecutor import in views.py
Fix issues with Gantt View (#12419)
Fix Entrypoint and _CMD config variables (#12411)
Fix operator field update for SerializedBaseOperator (#10924)
Limited cryptography to < 3.2 for python 2.7
Install cattr on Python 3.7 - Fix docs build on RTD (#12045)
Limit version of marshmallow-sqlalchemy
Pin kubernetes to a max version of 11.0.0 (#11974)
Use snakebite-py3 for HDFS dependency for Python3 (#12340)
Removes snakebite kerberos dependency (#10865)
Fix failing dependencies for FAB and Celery (#10828)
Fix pod_mutation_hook for 1.10.13 (#10850)
Fix formatting of Host information
Fix Logout Google Auth issue in Non-RBAC UI (#11890)
Add missing imports to app.py (#10650)
Show Generic Error for Charts & Query View in old UI (#12495)
TimeSensor should respect the default_timezone config (#9699)
TimeSensor should respect DAG timezone (#9882)
Unify user session lifetime configuration (#11970)
Handle outdated webserver session timeout gracefully. (#12332)

Improvements

Add XCom.deserialize_value to Airflow 1.10.13 (#12328)
Mount airflow.cfg to pod_template_file (#12311)
All k8s object must comply with JSON Schema (#12003)
Validate airflow chart values.yaml & values.schema.json (#11990)
Pod template file uses custom custom env variable (#11480)
Bump attrs and cattrs dependencies (#11969)
Bump attrs to > 20.0 (#11799)
[AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (#4751)
Rename task with duplicate task_id
Manage Flask AppBuilder Tables using Alembic Migrations (#12352)
airflow test only works for tasks in 1.10, not whole dags (#11191)
Improve warning messaging for duplicate task_ids in a DAG (#11126)
Pins moto to 1.3.14 (#10986)
DbApiHook: Support kwargs in get_pandas_df (#9730)
Make grace_period_seconds option on K8sPodOperator (#10727)
Fix syntax error in Dockerfile 'maintainer' Label (#10899)
The entrypoints in Docker Image should be owned by Airflow (#10853)
Make dockerfiles Google Shell Guide Compliant (#10734)
clean-logs script for Dockerfile: trim logs before sleep (#10685)
When sending tasks to celery from a sub-process, reset signal handlers (#11278)
SkipMixin: Add missing session.commit() and test (#10421)
Webserver: Further Sanitize values passed to origin param (#12459)
Security upgrade lodash from 4.17.19 to 4.17.20 (#11095)
Log instead of raise an Error for unregistered OperatorLinks (#11959)
Mask Password in Log table when using the CLI (#11468)
[AIRFLOW-3607] Optimize dep checking when depends on past set and concurrency limit
Execute job cancel HTTPRequest in Dataproc Hook (#10361)
Use rst lexer to format airflow upgrade check output (#11259)
Remove deprecation warning from contrib/kubernetes/pod.py
adding body as templated field for CloudSqlImportOperator (#10510)
Change log level for User's session to DEBUG (#12414)

Deprecations

Deprecate importing Hooks from plugin-created module (#12133)
Deprecate adding Operators and Sensors via plugins (#12069)

Doc only changes

[Doc] Correct description for macro task_instance_key_str (#11062)
Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
Revise "Project Focus" copy (#12011)
Move Project focus and Principles higher in the README (#11973)
Remove archived link from README.md (#11945)
Update download url for Airflow Version (#11800)
Add Project URLs for PyPI page (#11801)
Move Backport Providers docs to our docsite (#11136)
Refactor rebase copy (#11030)
Add missing images for kubernetes executor docs (#11083)
Fix identation in executor_config example (#10467)
Enhanced the Kubernetes Executor doc (#10433)
Refactor content to a markdown table (#10863)
Rename "Beyond the Horizon" section and refactor content (#10802)
Refactor official source section to use bullets (#10801)
Add section for official source code (#10678)
Add redbubble link to Airflow merchandise (#10359)
README Doc: Link to Airflow directory in ASF Directory (#11137)
Fix the default value for VaultBackend's config_path (#12518)

airflow - Airflow 2.0.0b3

Published by ashb almost 4 years ago

Major features since beta2:

New features:

Add description field to connection (#10873)
Use different deserialization method in XCom init_on_load (#12327)
Make nav fully accessible y keyboard, fix mobile nav menus (#12351)
Add metric for scheduling delay between first run task & expected start time (#9544)
Create DAG-level cluster policy (#12184)

Improvements:

UI: Improve the layout of TI modal when browser at narrower widths (#12456)
PR to add 'files' to template-fields in EmailOperator class (#12428)
JSON Response is returned for invalid API requests (#12305)
Make K8sPodOperator backwards compatible (#12384)
Don't display DAG docs when None (#12415)
Simplify using XComArg in jinja template string (#12405)
Remove inapplicable configuration section [ldap] (since 2.0.0) (#12386)
Add info log message about duration taken to load plugins (#12308)
Manage Flask AppBuilder Tables using Alembic Migrations (#12352)
UI: Improve presentation of DAG Docs (#12330)
Wait option for dagrun operator (#12126)
Use default view in TriggerDagRunLink (#11778)
Handle naive datetimes in REST APIi (#12248)
Unify user session lifetime configuration for webserver (#11970)

Bug fixes:

Fix typo in migrations: RESOURCE_DAGS to RESOURCE_DAG. (#12460)
UI: Fix bug in server timezone indicator (#12447)
Fix issues with Gantt View (#12419)
Proper exit status for failed CLI requests (#12375)
Check for TaskGroup in _PythonDecoratedOperator (#12312)
Reject 'connections add' CLI request if URI provided is invalid (#12370)
Reorder Database Migrations (#12362)
Refactor root logger handling in task run (#12342)
Bugfix: REST API Variables update endpoint returns 204 No Content (#12321)
UI: Fix/Enhancement: Disable forms and communicate to user when no DAG Runs (#12320)
UI: Get all "tags" parameters not just one (#12324)
Make dag_id, task_id, and execution_date nullable in event log schema (#12287)
Don't wrap warrning messages when stderr is not a TTY (#12285)
Don't treat warning message as rich formatting codes. (#12283)
UI: Fix pause/unpause toggle to display failed state when unsuccessful (#12267)
UI: Conditionally update button URL only when it is present (#12268)

Docs:

Cleanup requirements in README a bit (#12446)
Synchronization of supported K8S version (#12443)
Show all Providers in Docs (#12363)
Restructure documentation for releasing Airflow/Providers (#12350)
Fix Sample CLI commands for upgrading to 2.0 (#12349)
Docs installation improvements (#12304)
Remove providers imports from core examples (#12252)

Misc/Internal:

Move setup properties out of setup.py in to setup.cfg (#12417)
Change log level for User's session to DEBUG (#12414)
Adds mechanism for provider package discovery. (#12383)
Add success/failed sets to State class (#12359)
Enable protection for v1-10-stable branch (#12343)
Fix prod image build (#12314)
Remove deprecated Elasticsearch Configs (#12296)
Remove deprecated BashTaskRunner (#12295)
Added k9s as integrated tool to help with kubernetes testing (#12163)
Providers in extras are properly configured and verified (#12265)

airflow - Airflow 1.10.12, 2020-08-25

Published by kaxil about 4 years ago

New Features

Add DateTimeSensor (#9697)
Add ClusterPolicyViolation support to airflow local settings (#10282)
Get Airflow configs with sensitive data from Secret Backends (#9645)
[AIRFLOW-4734] Upsert functionality for PostgresHook.insert_rows() (#8625)
Allow defining custom XCom class (#8560)

Bug Fixes

Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067)
Fixes PodMutationHook for backwards compatibility (#9903)
Fix bug in executor_config when defining resources (#9935)
Respect DAG Serialization setting when running sync_perm (#10321)
Show correct duration on graph view for running task (#8311) (#8675)
Fix regression in SQLThresholdCheckOperator (#9312)
[AIRFLOW-6931] Fixed migrations to find all dependencies for MSSQL (#9891)
Avoid sharing session with RenderedTaskInstanceFields write and delete (#9993)
Fix clear future recursive when ExternalTaskMarker is used (#9515)
Handle IntegrityError while creating TIs (#10136)
Fix airflow-webserver startup errors when using Kerberos Auth (#10047)
Fixes treatment of open slots in scheduler (#9316) (#9505)
Fix KubernetesPodOperator reattachment (#10230)
Fix more PodMutationHook issues for backwards compatibility (#10084)
[AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276)
Fix task_instance_mutation_hook (#9910)
Fixes failing formatting of DAG file containing {} in docstring (#9779)
Fix is_terminal_support_colors function (#9734)
Fix PythonVirtualenvOperator when using provide_context=True (#8256)
Fix issue with mounting volumes from secrets (#10366)
BugFix: K8s Executor Multinamespace mode is evaluated to true by default (#10410)
Make KubernetesExecutor recognize kubernetes_labels (#10412)
Fix broken Kubernetes PodRuntimeInfoEnv (#10478)

Improvements

Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
Update Serialized DAGs in Webserver when DAGs are Updated (#9851)
Do not Update Serialized DAGs in DB if DAG did not change (#9850)
Add repr to SerializedDagModel (#9862)
Update JS packages to latest versions (#9811) (#9921)
UI Graph View: Focus upstream / downstream task dependencies on mouseover (#9303)
Allow image in KubernetesPodOperator to be templated (#10068)
[AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod (#7523)
Improve process terminating in scheduler_job (#8064)
Replace deprecated base classes used in bigquery_check_operator (#10272)
[AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
Limit all google-cloud api to <2.0.0 (#10317)
[AIRFLOW-6706] Lazy load operator extra links (#7327) (#10318)
Add Snowflake support to SQL operator and sensor (#9843)
Makes multi-namespace mode optional (#9570)
Pin Pyarrow < 1.0
Pin pymongo version to <3.11.0
Pin google-cloud-container to <2 (#9901)
Dockerfile: Remove package.json and yarn.lock from the prod image (#9814)
Dockerfile: The group of embedded DAGs should be root to be OpenShift compatible (#9794)
Update upper limit of flask-swagger, gunicorn & jinja2 (#9684)
Webserver: Sanitize values passed to origin param (#10334)
Sort connection type list in add/edit page alphabetically (#8692)

Doc only changes

Add new committers: Ry Walker & Leah Cole to project.rst (#9892)
Add Qingping Hou to committers list (#9725)
Updated link to official documentation (#9629)
Create a short-link for Airflow Slack Invites (#10034)
Fix docstrings in BigQueryGetDataOperator (#10042)
Set language on code-block on docs/howto/email-config.rst (#10238)
Remove duplicate line from 1.10.10 CHANGELOG (#10289)
Improve heading on Email Configuration page (#10175)
Fix link for the Jinja Project in docs/tutorial.rst (#10245)
Create separate section for Cron Presets (#10247)
Add Syntax Highlights to code-blocks in docs/best-practices.rst (#10258)
Fix docstrings in BigQueryGetDataOperator (#10042)
Fix typo in Task Lifecycle section (#9867)
Make Secret Backend docs clearer about Variable & Connection View (#8913)

airflow - Airflow 1.10.11, 2020-07-10

Published by kaxil over 4 years ago

New Features

Add task instance mutation hook (#8852)
Allow changing Task States Colors (#9520)
Add support for AWS Secrets Manager as Secrets Backend (#8186)
Add airflow info command to the CLI (#8704)
Add Local Filesystem Secret Backend (#8596)
Add Airflow config CLI command (#8694)
Add Support for Python 3.8 (#8836)(#8823)
Allow K8S worker pod to be configured from JSON/YAML file (#6230)
Add quarterly to crontab presets (#6873)
Add support for ephemeral storage on KubernetesPodOperator (#6337)
Add AirflowFailException to fail without any retry (#7133)
Add SQL Branch Operator (#8942)

Bug Fixes

Use NULL as dag.description default value (#7593)
BugFix: DAG trigger via UI error in RBAC UI (#8411)
Fix logging issue when running tasks (#9363)
Fix JSON encoding error in DockerOperator (#8287)
Fix alembic crash due to typing import (#6547)
Correctly restore upstream_task_ids when deserializing Operators (#8775)
Correctly store non-default Nones in serialized tasks/dags (#8772)
Correctly deserialize dagrun_timeout field on DAGs (#8735)
Fix tree view if config contains " (#9250)
Fix Dag Run UI execution date with timezone cannot be saved issue (#8902)
Fix Migration for MSSQL (#8385)
RBAC ui: Fix missing Y-axis labels with units in plots (#8252)
RBAC ui: Fix missing task runs being rendered as circles instead (#8253)
Fix: DagRuns page renders the state column with artifacts in old UI (#9612)
Fix task and dag stats on home page (#8865)
Fix the trigger_dag api in the case of nested subdags (#8081)
UX Fix: Prevent undesired text selection with DAG title selection in Chrome (#8912)
Fix connection add/edit for spark (#8685)
Fix retries causing constraint violation on MySQL with DAG Serialization (#9336)
[AIRFLOW-4472] Use json.dumps/loads for templating lineage data (#5253)
Restrict google-cloud-texttospeach to <v2 (#9137)
Fix pickling failure when spawning processes (#8671)
Pin Version of Azure Cosmos to <4 (#8956)
Azure storage 0.37.0 is not installable any more (#8833)
Fix modal_backdrop z-index in the UI (#7313)
Fix Extra Links in Gannt View (#8308)
Bug fix for EmrAddStepOperator init with cluster_name error (#9235)
Fix KubernetesPodOperator pod name length validation (#8829)
Fix non updating DAG code by checking against last modification time (#8266)
BugFix: Unpausing a DAG with catchup=False creates an extra DAG run (#8776)

Improvements

Improve add_dag_code_table migration (#8176)
Persistent display/filtering of DAG status (#8106)
Set unique logger names (#7330)
Update the version of cattrs from 0.9 to 1.0 to support Python 3.8 (#7100)
Reduce response payload size of /dag_stats and /task_stats (#8655)
Add authenticator parameter to snowflake_hook (#8642)
Show "Task Reschedule" table in Airflow Webserver (#9521)
Change worker_refresh_interval fallback to default of 30 (#9588)
Use pfromat instead of str to render arguments in WebUI (#9587)
Simplify DagFileProcessorManager (#7521)
Reload gunicorn when plugins has beeen changed (#8997)
Fix the default value for store_dag_code (#9554)
Add support for fetching logs from running pods (#8626)
Persist start/end date and duration for DummyOperator Task Instance (#8663)
Ensure "started"/"ended" in tooltips are not shown if job not started (#8667)
Add context to execution_date_fn in ExternalTaskSensor (#8702)
Avoid color info in response of /dag_stats & /task_stats (#8742)
Make loading plugins from entrypoint fault-tolerant (#8732)
Refactor Kubernetes worker config (#7114)
Add default conf parameter to Spark JDBC Hook (#8787)
Allow passing backend_kwargs to AWS SSM client (#8802)
Filter dags by clicking on tag (#8897)
Support k8s auth method in Vault Secrets provider (#8640)
Monitor pods by labels instead of names (#6377)
Optimize count query on /home (#8729)
Fix json string escape in tree view (#8551)
Add TaskInstance state to TI Tooltip to be colour-blind friendlier (#8910)
Add a tip to trigger DAG screen (#9049)
Use Markup for htmlcontent for landing_times (#9242)
Pinning max pandas version to 2.0 (lesser than) to allow pandas 1.0 (#7954)
Update example webserver_config.py to show correct CSRF config (#8944)
Fix displaying Executor Class Name in "Base Job" table (#8679)
Use existing DagBag for 'dag_details' & 'trigger' Endpoints (#8501)
Flush pending Sentry exceptions before exiting (#7232)
Display DAG run conf in the list view (#6794)
Fix performance degradation when updating dagrun state (#8435)
Don't use the |safe filter in code, it's risky (#9180)
Validate only task commands are run by executors (#9178)
Show Deprecation warning on duplicate Task ids (#8728)
Move DAG._schedule_interval logic out of DAG.__init__ (#8225)
Make retrieving Paused Dag ids a separate method (#7587)
Bulk fetch paused_dag_ids (#7476)
Add a configurable DAGs volume mount path for Kubernetes (#8147)
Add schedulername option for KubernetesPodOperator (#6088)
Support running git sync container as root (#6312)
Add extra options for Slack Webhook operator and Slack hook (#9409)
Monkey patch greenlet celery pools (#8559)
Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)
Prevent clickable sorting on non sortable columns in TI view (#8681)
Make hive macros py3 compatible (#8598)
Fix SVG tooltip positioning with custom scripting (#8269)
Avoid unnecessary sleep to maintain local task job heart rate (#6553)
Include some missing RBAC roles on User and Viewer roles (#9133)
Show Dag's Markdown docs on Tree View (#9448)
Improved compatibility with Python 3.5+ - Convert signal.SIGTERM to int (#9207)
Add 'main' param to template_fields in DataprocSubmitPySparkJobOperator (#9154)
Make it possible to silence warnings from Airflow (#9208)
Remove redundant count query in BaseOperator.clear() (#9362)
Fix DB migration message (#8988)
Fix awkward log info in dbapi_hook (#8482)
Fix Celery default to no longer allow pickle (#7205)
Further validation that only task commands are run by executors (#9240)
Remove vendored nvd3 and slugify libraries (#9136)
Enable configurable git sync depth (#9094)
Reduce the required resources for the Kubernetes's sidecar (#6062)
Refactor K8S codebase with k8s API models (#5481)
Move k8sexecutor out of contrib to closer match master (#8904)
Allow filtering using "event" and "owner" in "Log" view (#4881)
Add Yandex.Cloud custom connection to 1.10 (#8791)
Add table-hover css class to DAGs table (#5033)
Show un/pause errors in dags view. (#7669)
Restructure database queries on /home (#4872)
Add Cross Site Scripting defence (#6913)
Make Gantt tooltip the same as Tree and Graph view (#8220)
Add config to only delete worker pod on task failure (#7507)(#8312)
Remove duplicate error message on chart connection failure (#8476)
Remove default value spark_binary (#8508)
Expose Airflow Webserver Port in Production Docker Image (#8228)
Commit after each alembic migration (#4797)
KubernetesPodOperator fixes and test (#6524)
Docker Image: Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)
Docker Image: Add ADDITIONAL_PYTHON_DEPS (#9031)
Remove httplib2 from Google requirements (#9194)
Merging multiple sql operators (#9124)
Adds hive as extra in pyhive dependency (#9075)
Change default auth for experimental backend to deny_all (#9611)
Restrict changing XCom values from the Webserver (#9614)
Add repr for DagTag so tags display properly in /dagmodel/show (#8719)
Functionality to shuffle HMS connections used by HiveMetastoreHook facilitating load balancing (#9280)
Expose SQLAlchemy's connect_args and make it configurable (#6478)

Doc only changes

Add docs on using DAGRun.conf (#9578)
Enforce code-block directives in doc (#9443)
Carefully parse warning messages when building documentation (#8693)
Make KubernetesPodOperator clear in docs (#8444)
Improve language in Pod Mutation Hook docs (#8445)
Fix formatting of Pool docs in concepts.rst (#8443)
Make doc clearer about Airflow Variables using Environment Variables (#8427)
Fix pools doc for LocalExecutor (#7643)
Add Local and Sequential Executors to Doc (#8084)
Add documentation for CLI command airflow dags test (#8251)
Fix typo in DAG Serialization documentation (#8317)
Add scheduler in production section (#7351)
Add a structural dag validation example (#6727)
Adding Task re-run documentation (#6295)
Fix outdated doc on settings.policy (#7532)
Add docs about reload_on_plugin_change opiton (#9575)
Add copy button to Code Blocks in Airflow Docs (#9450)
Update commands in docs for v1.10+ (#9585)
Add more info on dry-run CLI option (#9582)
Document default timeout value for SSHOperator (#8744)
Fix docs on creating CustomOperator (#8678)
Enhanced documentation around Cluster Policy (#8661)
Use sphinx syntax in concepts.rst (#7729)
Update README to remove Py 3.8 limitation for Master (#9451)
Add note about using dag_run.conf in BashOperator (#9143)
Improve tutorial - Include all imports statements (#8670)
Added more precise Python requirements to README.md (#8455)
Fix Airflow Stable version in README.md (#9360)
Update AWS connection example to show how to set from env var (#9191)
Fix list formatting of plugins doc. (#8873)
Add 'Version Added' on Secrets Backend docs (#8264)
Simplify language re roll-your-own secrets backend (#8257)
Add installation description for repeatable PyPi installation (#8513)
Add note extra links only render on when using RBAC webserver (#8788)
Remove unused airflow import from docs (#9274)
Add PR/issue note in Contribution Workflow Example (#9177)
Don't use the term "whitelist" - language matters (#9174)
Add docs to change Colors on the Webserver (#9607)
Change 'initiate' to 'initialize' in installation.rst (#9619)
Replace old Variables View Screenshot with new (#9620)
Replace old SubDag zoom screenshot with new (#9621)
Update docs about the change to default auth for experimental API (#9617)

airflow - Airflow 1.10.10, 2020-04-09

Published by kaxil over 4 years ago

New Features

[AIRFLOW-7048] Allow user to chose timezone to use in UI (#8046)
Add Production Docker image support (#7832)
Get Airflow Variables from Environment Variables (#7923)
Get Airflow Variables from Hashicorp Vault (#7944)
Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)
Get Airflow Variables from GCP Secrets Manager (#7946)
[AIRFLOW-5705] Add secrets backend and support for AWS SSM / Get Airflow Connections from AWS Parameter Store(#6376)
[AIRFLOW-7104] Add Secret backend for GCP Secrets Manager / Get Airflow Connections from GCP Secrets Manager (#7795)
[AIRFLOW-7076] Add support for HashiCorp Vault as Secrets Backend / Get Airflow Connections from Hashicorp Vault (#7741)
[AIRFLOW-6685] Add ThresholdCheckOperator (#7353)
[AIRFLOW-7080] Add API endpoint to return a DAG's paused state (#7737)

Improvements

Use same tooltip for Graph and Tree views for TaskInstances (#8043)
Allow DateTimePicker in Webserver to actually pick times too (#8034)
[AIRFLOW-5590] Add run_id to trigger DAG run API response (#6256)
[AIRFLOW-6695] Can now pass dagrun conf when triggering dags via UI (#7312)
[AIRFLOW-5336] Add ability to make updating FAB perms on webserver in it optional (#5940)
[AIRFLOW-1467] Allow tasks to use more than one pool slot (#7160)
[AIRFLOW-6987] Avoid creating default connections (#7629)
[AIRFLOW-4175] S3Hook load_file should support ACL policy parameter (#7733)
[AIRFLOW-4438] Add Gzip compression to S3_hook (#7680)
Allow setting Airflow Variable values to empty string (#8021)
Dont schedule dummy tasks (#7880)
Prevent sequential scan of task instance table (#8014)
[AIRFLOW-7017] Respect default dag view in trigger dag origin (#7667)
[AIRFLOW-6837] Limit description length of a Dag on HomePage (#7457)
[AIRFLOW-6989] Display Rendered template_fields without accessing Dag files (#7633)
[AIRFLOW-5944] Rendering templated_fields without accessing DAG files (#6788)
[AIRFLOW-5946] DAG Serialization: Store source code in db (#7217)
[AIRFLOW-7079] Remove redundant code for storing template_fields (#7750)
[AIRFLOW-7024] Add the verbose parameter support to SparkSqlOperator (#7676)
[AIRFLOW-6733] Extend not replace template (#7366)
[AIRFLOW-7001] Further fix for the MySQL 5.7 UtcDateTime (#7655)
[AIRFLOW-6014] Handle pods which are preempted & deleted by kubernetes but not restarted (#6606)
[AIRFLOW-6950] Remove refresh_executor_config from ti.refresh_from_db (#7577)
[AIRFLOW-7016] Sort dag tags in the UI (#7661)
[AIRFLOW-6762] Fix link to "Suggest changes on this page" (#7387)
[AIRFLOW-6948] Remove ASCII Airflow from version command (#7572)
[AIRFLOW-6767] Correct name for default Athena workgroup (#7394)
[AIRFLOW-6905] Update pin.svg with new pinwheel (#7524)
[AIRFLOW-6801] Make use of ImportError.timestamp (#7425)
[AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator (#7451)
[AIRFLOW-6630] Resolve handlebars advisory (#7284)
[AIRFLOW-6945] MySQL 5.7 is used in v1-10-test as an option
[AIRFLOW-6871] Optimize tree view for large DAGs (#7492)
[AIRFLOW-7063] Fix dag.clear() slowness caused by count (#7723)
[AIRFLOW-7023] Remove duplicated package definitions in setup.py (#7675)
[AIRFLOW-7001] Time zone removed from MySQL TIMSTAMP field inserts
[AIRFLOW-7105] Unify Secrets Backend method interfaces (#7830)
Make BaseSecretsBackend.build_path generic (#7948)
Allow hvac pakage installation using 'hashicorp' extra (#7915)
Standardize SecretBackend class names (#7846)
[AIRFLOW-5705] Make AwsSsmSecretsBackend consistent with VaultBackend (#7753)
[AIRFLOW-7045] Update SQL query to delete RenderedTaskInstanceFields (#8051)
Handle DST better in Task Instance tool tips (#8104)

Bug Fixes

BugFix: Show task_id in the Graph View tooltip (#7859)
[AIRFLOW-6730] Use total_seconds instead of seconds (#7363)
[AIRFLOW-6167] Escape column name in create table in hive (#6741)
[AIRFLOW-6628] DAG auto-complete now suggests from all acessible DAGs (#7251)
[AIRFLOW-7113] Fix gantt render error (#7913)
[AIRFLOW-6399] Add _access control to validate deserialized DAGs (#7896)
[AIRFLOW-6399] Serialization: DAG access_control field should be decorated field in DAG serialization (#7879)
[AIRFLOW-4453] Make behavior of none_failed consistent with documentation (#7464)
[AIRFLOW-4363] Fix JSON encoding error (#7628)
[AIRFLOW-6683] Run REST API tests when DAGs are serialized (#7352)
[AIRFLOW-6704] Copy common TaskInstance attributes from Task (#7324)
[AIRFLOW-6734] Use configured base_template instead of hard-coding (#7367)
[AIRFLOW-7098] Simple salesforce release 1.0.0 breaks the build (#7775)
[AIRFLOW-6062] Executor would only delete workers in its own namespace (#7123)
[AIRFLOW-7074] Add Permissions to view SubDAGs (#7752)
[AIRFLOW-7025] Fix SparkSqlHook.run_query to handle its parameter properly (#7677)
[AIRFLOW-6855] Escape project_dataset_table in SQL query in gcs to bq operator (#7475)
[AIRFLOW-6949] Respect explicit conf to SparkSubmitOperator (#7575)
[AIRFLOW-6588] write_stdout and json_format are boolean (#7199)
[AIRFLOW-3439] Decode logs with 'utf-8' (#4474)
[AIRFLOW-6878] Fix misconfigured default value for kube_client_request_args
[AIRFLOW-5167] Update dependencies for GCP packages (#7116)
[AIRFLOW-6821] Success callback not called when task marked as success from UI (#7447)
[AIRFLOW-6740] Remove Undocumented, deprecated, dysfunctional PROXY_FIX_NUM_PROXIES (#7359)
[AIRFLOW-6728] Change various DAG info methods to POST (#7364)
[AIRFLOW-6997] Make sure worker pods initcontainers obtain env vars from config (#7663)
[AIRFLOW-7062] Fix pydruid release breaking the build (#7720)
[AIRFLOW-6040] ReadTimoutError in KubernetesExecutor should not raise exception (#7616)
[AIRFLOW-6943] Fix utf-8 encodeed description in DAG in Python 2 (#7567)
[AIRFLOW-6892] Fix broken non-wheel releases (#7514)
[AIRFLOW-6789] BugFix: Fix Default Worker concurrency (#7494)
[AIRFLOW-6840] Bump up version of future (#7471)
[AIRFLOW-5705] Fix bugs in AWS SSM Secrets Backend (#7745)
[AIRFLOW-5705] Fix bug in Secrets Backend (#7742)
Fix CloudSecretsManagerBackend invalid connections_prefix (#7861)
[AIRFLOW-7045] BugFix: DebugExecutor fails to change task state. (#8073)
BugFix: Datetimepicker is stuck on the UI (#8092)
[AIRFLOW-5277] Gantt chart respects per-user the Timezone UI setting (#8096)
Fix timezones displayed in Task Instance tooltip (#8103)
BugFix: Fix writing & deleting Dag Code for Serialized DAGs (#8151)
Make the default TI pool slots '1' (#8153)
Fix 500 error in Security screens (#8165)
Fix Viewing Dag Code for Stateless Webserver (#8159)
Fix issue with SQLAlchemy 1.3.16 (#8230)

Misc/Internal

Fix Flaky TriggerDAG UI test (#8022)
Remove unnecessary messages in CI (#7951)
Fixes too high parallelism in CI (#7947)
Install version is not persistent in breeze (#7914)
Fixed automated check for image rebuild (#7912)
Move Dockerfile to Dockerfile.ci (#7829)
Generate requirements are now sorted (#8040)
Change name of the common environment initialization function (#7805)
Requirements now depend on python version (#7841)
Bring back reset db explicitly called at CI entry (#7798)
Fixes unclean installation of Airflow 1.10 (#7796)
Change name of the common environment initialization function (#7805)
[AIRFLOW-7029] Use separate docker image for running license check (#7678)
[AIRFLOW-5842] Swtch to Debian buster image as a base (#7647)
[AIRFLOW-5828] Move build logic out from hooks/build (#7618)
[AIRFLOW-6839] Even more mypy speed improvements (#7460)
[AIRFLOW-6820] split breeze into functions (#7433)
[AIRFLOW-7097] Install gcloud beta componensts in CI image (#7772)
[AIRFLOW-7018] fixing travis's job name escaping problem (#7668)
[AIRFLOW-7054] Breeze has an option now to reset db at entry (#7710)
[AIRFLOW-7005] Added exec command to Breeze (#7649)
[AIRFLOW-7015] Detect Dockerhub repo/user when building on Dockerhub (#7673)
[AIRFLOW-6727] Fix minor bugs in Release Management scripts (#7355)
[AIRFLOW-7013] Automated check if Breeze image needs to be pulled (#7656)
[AIRFLOW-7010] Skip in-container checks for Dockerhub builds (#7652)
[AIRFLOW-7011] Pin JPype release to allow to build 1.10 images
[AIRFLOW-7006] Fix missing +e in Breeze script (#7648)
[AIRFLOW-6979] Fix breeze test-target specific test param issue (#7614)
[AIRFLOW-6932] Add restart-environment command to Breeze
[AIRFLOW-6919] Make Breeze DAG-test friedly (#7539)
[AIRFLOW-6838] Introduce real subcommands for Breeze (#7515)
[AIRFLOW-6763] Make systems tests ready for backport tests (#7389)
[AIRFLOW-6866] Fix wrong export for Mac on Breeze (#7485)
[AIRFLOW-6842] Skip fixing ownership on Mac (#7469)
[AIRFLOW-6841] Fixed unbounded variable on Mac (#7465)
[AIRFLOW-7067] Pinned version of Apache Airflow (#7730)
[AIRFLOW-7058] Add support for different DB versions (#7717)
[AIRFLOW-7002] Get rid of yaml "parser" in bash (#7646)
[AIRFLOW-6972] Shorter frequently used commands in Breeze (#7608)

Doc only changes

Fix typo for store_serialized_dags config (#7952)
Fix broken link in README.md (#7893)
Separate supported Postgres versions with comma (#7892)
Fix grammar in setup.py (#7877)
Add Jiajie Zhong to committers list (#8047)
Update Security doc for 1.10.* for auth backends (#8072)
Fix Example in config_templates for Secrets Backend (#8074)
Add backticks in IMAGES.rst command description (#8075)
Change version_added for store_dag_code config (#8076)
[AIRFLOW-XXXX] Remove duplicate docs
[AIRFLOW-XXXX] Remove the defunct limitation of Dag Serialization (#7716)
[AIRFLOW-XXXX] Add prerequisite tasks for all GCP operators guide (#6049)
[AIRFLOW-XXXX] Simplify AWS/Azure/Databricks operators listing (#6047)
[AIRFLOW-XXXX] Add external reference to all GCP operator guide (#6048)
[AIRFLOW-XXXX] Simplify GCP operators listing
[AIRFLOW-XXXX] Simplify Qubole operators listing
[AIRFLOW-XXXX] Add autogenerated TOC (#6038)
[AIRFLOW-XXXX] Create "Using the CLI" page (#5823)
[AIRFLOW-XXXX] Group references in one section (#5776)
[AIRFLOW-XXXX] Fix analytics doc (#5885)
[AIRFLOW-XXXX] Add S3 Logging section (#6039)
[AIRFLOW-XXXX] Move Azure Logging section above operators (#6040)
[AIRFLOW-XXXX] Update temp link to a fixed link (#7715)
[AIRFLOW-XXXX] Add Updating.md section for 1.10.9 (#7385)
[AIRFLOW-XXXX] Remove duplication in BaseOperator docstring (#7321)
[AIRFLOW-XXXX] Update tests info in CONTRIBUTING.rst (#7466)
[AIRFLOW-XXXX] Small BREEZE.rst update (#7487)
[AIRFLOW-XXXX] Add instructions for logging to localstack S3 (#7461)
[AIRFLOW-XXXX] Remove travis config warnings (#7467)
[AIRFLOW-XXXX] Add communication chapter to contributing (#7204)
[AIRFLOW-XXXX] Add known issue - example_dags/init.py (#7444)
[AIRFLOW-XXXX] Fix breeze build-docs (#7445)
[AIRFLOW-XXXX] Less verbose docker builds
[AIRFLOW-XXXX] Speed up mypy runs (#7421)
[AIRFLOW-XXXX] Fix location of kubernetes tests (#7373)
[AIRFLOW-XXXX] Remove quotes from domains in Google Oauth (#4226)
[AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs (#7318)
[AIRFLOW-XXXX] Fix typo commiter => committer (#7392)
[AIRFLOW-XXXX] Remove duplicated paragraph in docs (#7662)
Fix reference to KubernetesPodOperator (#8100)

airflow - Airflow 1.10.9, 2020-02-07

Published by kaxil over 4 years ago

Bug Fixes

[AIRFLOW-6751] Pin Werkzeug (dependency of a number of our dependencies) to < 1.0.0 (#7377)

airflow - Airflow 1.10.8, 2020-02-07

Published by kaxil over 4 years ago

New Features

[AIRFLOW-4026] Add filter by DAG tags (#6489)
[AIRFLOW-6613] Center dag on graph view load (#7238)
[AIRFLOW-5843] Add conf option to Add DAG Run view (#7281)
[AIRFLOW-4495] Allow externally triggered dags to run for future exec dates (#7038)

Improvements

[AIRFLOW-6438] Filter DAGs returned by blocked (#7019)
[AIRFLOW-6666] Resolve js-yaml advisory (#7283)
[AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280)
[AIRFLOW-6667] Resolve serialize-javascript advisory (#7282)
[AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable (#7134)
[AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner (#7090)
[AIRFLOW-6319] Add support for AWS Athena workgroups (#6871)
[AIRFLOW-6677] Remove deprecation warning from SQLAlchmey (#7289)
[AIRFLOW-6428] Fix import path for airflow.utils.dates.days_ago in Example DAGs (#7007)
[AIRFLOW-6595] Use TaskNotFound exception instead of AirflowException (#7210)
[AIRFLOW-6620] Mock celery in worker cli test (#7243)
[AIRFLOW-6608] Change logging level for Bash & PyOperator Env exports
[AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker (#6633)
[AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978)
[AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909)
[AIRFLOW-6316] Use exampleinclude directives in tutorial.rst (#6868)
[AIRFLOW-6519] Make TI logs constants in Webserver configurable (#7113)
[AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886)
[AIRFLOW-6261] flower_basic_auth eligible to _cmd (#6825)
[AIRFLOW-6238] Filter dags returned by dag_stats
[AIRFLOW-5616] Switch PrestoHook from pyhive to prestosql-client to s
[AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg (#7236)
[AIRFLOW-6557] Add test for newly added fields in BaseOperator (#7162)
[AIRFLOW-6584] Pin cassandra driver (#7194)
[AIRFLOW-6537] Fix backticks in rst files (#7140)
[AIRFLOW-4428] Error if exec_date before default_args.start_date in trigger_dag (#6948)
[AIRFLOW-6330] Show cli help when param blank or typo (#6883)
[AIRFLOW-4113] Unpin boto3 (#6884)
[AIRFLOW-6181] Add DebugExecutor (#6740)
[AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting (#7097)
[AIRFLOW-6436] Cleanup for Airflow configs doc generator code (#7036)
[AIRFLOW-6436] Add x_frame_enabled config in config.yml (#7024)
[AIRFLOW-6436] Create & Automate docs on Airflow Configs (#7015)
[AIRFLOW-6527] Make send_task_to_executor timeout configurable (#7143)
[AIRFLOW-6272] Switch from npm to yarnpkg for managing front-end dependencies (#6844)
[AIRFLOW-6350] Security - spark submit operator logging+exceptions should mask passwords
[AIRFLOW-6358] Log details of failed task (#6908)
[AIRFLOW-5149] Skip SLA checks config (#6923)
[AIRFLOW-6057] Update template_fields of the PythonSensor (#6656)
[AIRFLOW-4445] Mushroom cloud errors too verbose (#6952)
[AIRFLOW-6394] Simplify github PR template (#6955)
[AIRFLOW-5385] spark hook does not work on spark 2.3/2.4 (#6976)

Bug Fixes

[AIRFLOW-6345] Ensure arguments to ProxyFix are integers (#6901)
[AIRFLOW-6576] Fix scheduler crash caused by deleted task with sla misses (#7187)
[AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298)
[AIRFLOW-6683] REST API respects store_serialized_dag setting (#7296)
[AIRFLOW-6553] Add upstream_failed in instance state filter to WebUI (#7159)
[AIRFLOW-6357] Highlight nodes in Graph UI if task id contains dots (#6904)
[AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter (#7329)
[AIRFLOW-6627] Email with incorrect DAG not delivered (#7250)
[AIRFLOW-6637] Fix airflow test command in 1.10.x
[AIRFLOW-6636] Avoid exceptions when printing task instance
[AIRFLOW-6522] Clear task log file before starting to fix duplication in S3TaskHandler (#7120)
[AIRFLOW-5501] Make default in_cluster value in KubernetesPodOperator respect config (#6124)
[AIRFLOW-6514] Use RUNNING_DEPS to check run from UI (#6367)
[AIRFLOW-6381] Remove styling based on DAG id from DAGs page (#6985)
[AIRFLOW-6434] Add return statement back to DockerOperator.execute (#7013)
[AIRFLOW-2516] Fix mysql deadlocks (#6988)
[AIRFLOW-6528] Disable flake8 W503 line break before binary operator (#7124)
[AIRFLOW-6517] Make merge_dicts function recursive (#7111)
[AIRFLOW-5621] Failure callback is not triggered when marked Failed on UI (#7025)
[AIRFLOW-6353] Security - ui - add click jacking defence (#6995)
[AIRFLOW-6348] Security - cli.py is currently printing logs with password (#6915)
[AIRFLOW-6323] Remove non-ascii letters from default config (#6878)
[AIRFLOW-6506] Fix do_xcom_push defaulting to True in KubernetesPodOperator (#7122)
[AIRFLOW-6516] BugFix: airflow.cfg does not exist in Volume Mounts (#7109)
[AIRFLOW-6427] Fix broken example_qubole_operator dag (#7005)
[AIRFLOW-6385] BugFix: SlackAPIPostOperator fails when blocks not set (#7022)
[AIRFLOW-6347] BugFix: Can't get task logs when serialization is enabled (#7092)
[AIRFLOW-XXXX] Fix downgrade of db migration 0e2a74e0fc9f (#6859)
[AIRFLOW-6366] Fix migrations for MS SQL Server (#6920)
[AIRFLOW-5406] Allow spark without kubernetes (#6921)
[AIRFLOW-6229] SparkSubmitOperator polls forever if status json can't… (#6918)
[AIRFLOW-6352] Security - ui - add login timeout (#6912)
[AIRFLOW-6397] Ensure sub_process attribute exists before trying to kill it (#6958)
[AIRFLOW-6400] Fix pytest not working on Windows (#6964)
[AIRFLOW-6418] Remove SystemTest.skip decorator (#6991)
[AIRFLOW-6425] Serialization: Add missing DAG parameters to Json Schema (#7002)

Misc/Internal

[AIRFLOW-6467] Use self.dag i/o creating a new one (#7067)
[AIRFLOW-6490] Improve time delta comparison in local task job tests (#7083)
[AIRFLOW-5814] Implementing Presto hook tests (#6491)
[AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516)
[AIRFLOW-XXXX] Move airflow-config-yaml pre-commit before pylint (#7108)
[AIRFLOW-XXXX] Improve clarity of confirm message (#7110)
[AIRFLOW-6662] install dumb init (#7300)
[AIRFLOW-6705] One less chatty message at breeze initialisation (#7326)
[AIRFLOW-6705] Less chatty integration/backend checks (#7325)
[AIRFLOW-6662] Switch to --init docker flag for signal propagation (#7278)
[AIRFLOW-6661] Fail after 50 failing tests (#7277)
[AIRFLOW-6607] Get rid of old local scripts for Breeze (#7225)
[AIRFLOW-6589] BAT tests run in pre-commit on bash script changes (#7203)
[AIRFLOW-6592] Doc build is moved to test phase (#7208)
[AIRFLOW-6641] Better diagnostics for kubernetes flaky tests (#7261)
[AIRFLOW-6642] Make local task job test less flaky (#7262)
[AIRFLOW-6643] Fix flakiness of kerberos tests
[AIRFLOW-6638] Remove flakiness test from test_serialized_db remove
[AIRFLOW-6701] Rat is downloaded from stable backup/mirrors (#7323)
[AIRFLOW-6702] Dumping kind logs to file.io. (#7319)
[AIRFLOW-6491] Improve handling of Breeze parameters (#7084)
[AIRFLOW-6470] Avoid pipe to file when do curl (#7063)
[AIRFLOW-6471] Add pytest-instafail plugin (#7064)
[AIRFLOW-6462] Limit exported variables in Dockerfile/Breeze (#7057)
[AIRFLOW-6465] Add bash autocomplete for airflow in Breeze (#7060)
[AIRFLOW-6464] Add cloud providers CLI tools in Breeze (#7059)
[AIRFLOW-6461] Remove silent flags in Dockerfile (#7052)
[AIRFLOW-6459] Increase verbosity of pytest (#7049)
[AIRFLOW-6370] Skip Cassandra tests if cluster is not up (#6926)
[AIRFLOW-6511] Remove BATS docker containers (#7103)
[AIRFLOW-6475] Remove duplication of volume mount specs in Breeze.. (#7065)
[AIRFLOW-6489] Add BATS support for Bash unit testing (#7081)
[AIRFLOW-6387] print details of success/skipped task (#6956)
[AIRFLOW-6568] Add Emacs related files to .gitignore (#7175)
[AIRFLOW-6575] Entropy source for CI tests is changed to unblocking (#7185)
[AIRFLOW-6496] Separate integrations in tests (#7091)
[AIRFLOW-6634] Set PYTHONPATH in interactive Breeze
[AIRFLOW-6564] Additional diagnostics information on CI check failure (#7172)
[AIRFLOW-6383] Add no trailing-whitespace pre-commit hook (#6941)

Doc only changes

[AIRFLOW-XXXX] Consistency fixes in new documentation (#7207)
[AIRFLOW-XXXX] Improve grammar and structure in FAQ doc (#7291)
[AIRFLOW-XXXX] Fix email configuration link in CONTRIBUTING.rst (#7311)
[AIRFLOW-XXXX] Update docs with new BranchPythonOperator behaviour (#4682)
[AIRFLOW-XXXX] Fix Typo in scripts/ci/ci_run_airflow_testing.sh (#7235)
[AIRFLOW-XXXX] Screenshot showing disk space configuration for OSX (#7226)
[AIRFLOW-XXXX] Add mentoring information to contributing docs (#7202)
[AIRFLOW-XXXX] Add rebase info to contributing (#7201)
[AIRFLOW-XXXX] Increase verbosity of static checks in CI (#7200)
[AIRFLOW-XXXX] Adds branching strategy to documentation (#7193)
[AIRFLOW-XXXX] Move email configuration from the concept page (#7189)
[AIRFLOW-XXXX] Update task lifecycle diagram (#7161)
[AIRFLOW-XXXX] Fix reference in concepts doc (#7135)
[AIRFLOW-XXXX] Clear debug docs (#7104)
[AIRFLOW-XXXX] Fix typos and broken links in development docs (#7086)
[AIRFLOW-XXXX] Clarify wait_for_downstream and execution_date (#6999)
[AIRFLOW-XXXX] Add airflow dags show command guide (#7014)
[AIRFLOW-XXXX] Update operation chaining documentation (#7018)
[AIRFLOW-XXXX] Add .autoenv_leave.zsh to .gitignore (#6986)
[AIRFLOW-XXXX] Fix development packages installation instructions (#6942)
[AIRFLOW-XXXX] Update committers list (#7212)
[AIRFLOW-XXXX] Move UPDATING changes into correct versions (#7166)
[AIRFLOW-XXXX] Add Documentation for check_slas flag (#6974)
[AIRFLOW-XXXX] Fix gcp keyfile_dict typo (#6962)
[AIRFLOW-XXXX] Add tips for writing a note in UPDATIND.md (#6960)
[AIRFLOW-XXXX] Add note warning that bash>4.0 is required for docs build script (#6947)
[AIRFLOW-XXXX] Add autoenv to gitignore (#6946)
[AIRFLOW-XXXX] Fix GCSTaskHandler Comment Typo (#6928)
[AIRFLOW-XXXX] Fix broken DAG Serialization Link (#6891)
[AIRFLOW-XXXX] Add versions_added field to configs

airflow - Airflow 1.10.7, 2019-12-24

Published by kaxil almost 5 years ago

New Features

[AIRFLOW-5088] [AIP-24] Persisting serialized DAG in DB for webserver scalability (#5992)
[AIRFLOW-6083] Adding ability to pass custom configuration to AWS Lambda client. (#6678)
[AIRFLOW-5117] Automatically refresh EKS API tokens when needed (#5731)
[AIRFLOW-5118] Add ability to specify optional components in DataprocClusterCreateOperator (#5821)
[AIRFLOW-5681] Allow specification of a tag or hash for the git_sync init container (#6350)
[AIRFLOW-6025] Add label to uniquely identify creator of Pod (#6621)
[AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489)
[AIRFLOW-5751] add get_uri method to Connection (#6426)
[AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id (#6655)
[AIRFLOW-2694] Declare permissions in DAG definition (#4642)
[AIRFLOW-4940] Add DynamoDB to S3 operator (#5663)
[AIRFLOW-4161] BigQuery to Mysql Operator (#5711)
[AIRFLOW-6041] Add user agent to the Discovery API client (#6636)
[AIRFLOW-6089] Reorder setup.py dependencies and add ci (#6681)
[AIRFLOW-5921] Add bulk_load_custom to MySqlHook (#6575)
[AIRFLOW-5854] Add support for tty parameter in Docker related operators (#6542)
[AIRFLOW-4758] Add GcsToGDriveOperator operator (#5822)

Improvements

[AIRFLOW-3656] Show doc link for the current installed version (#6690)
[AIRFLOW-5665] Add path_exists method to SFTPHook (#6344)
[AIRFLOW-5729] Make InputDataConfig optional in Sagemaker's training config (#6398)
[AIRFLOW-5045] Add ability to create Google Dataproc cluster with custom image from a different project (#5752)
[AIRFLOW-6132] Allow to pass in tags for the AzureContainerInstancesOperator (#6694)
[AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization (#6715)
[AIRFLOW-5947] Make the json backend pluggable for DAG Serialization (#6630)
[AIRFLOW-6239] Filter dags return by last_dagruns (to only select visible dags, not all dags) (#6804)
[AIRFLOW-6095] Filter dags returned by task_stats (to only select visible dags, not all dags) (#6684)
[AIRFLOW-4482] Add execution_date to "trigger DagRun" API response (#5260)
[AIRFLOW-1076] Add get method for template variable accessor (#6793)
[AIRFLOW-5194] Add error handler to action log (#5883)
[AIRFLOW-5936] Allow explicit get_pty in SSHOperator (#6586)
[AIRFLOW-5474] Add Basic auth to Druid hook (#6095)
[AIRFLOW-5726] Allow custom filename in RedshiftToS3Transfer (#6396)
[AIRFLOW-5834] Option to skip serve_logs process with airflow worker (#6709)
[AIRFLOW-5583] Extend the 'DAG Details' page to display the start_date / end_date (#6235)
[AIRFLOW-6250] Ensure on_failure_callback always has a populated context (#6812)
[AIRFLOW-6222] http hook logs response body for any failure (#6779)
[AIRFLOW-6260] Drive _cmd config option by env var (AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD for example) (#6801)
[AIRFLOW-6168] Allow proxy_fix middleware of webserver to be configurable (#6723)
[AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. (#6627)
[AIRFLOW-4145] Allow RBAC roles permissions, ViewMenu to be over-ridable (#4960)
[AIRFLOW-5928] Hive hooks load_file short circuit (#6582)
[AIRFLOW-5313] Add params support for awsbatch_operator (#5900)
[AIRFLOW-2227] Add delete method to Variable class (#4963)
[AIRFLOW-5082] Add subject in AwsSnsHook (#5694)
[AIRFLOW-5715] Make email, owner context available (#6385)
[AIRFLOW-5345] Allow SqlSensor's hook to be customized by subclasses (#5946)
[AIRFLOW-5417] Fix DB disconnects during webserver startup (#6023)
[AIRFLOW-5730] Enable get_pandas_df on PinotDbApiHook (#6399)
[AIRFLOW-3235] Add list function in AzureDataLakeHook (#4070)
[AIRFLOW-5442] implementing get_pandas_df method for druid broker hook (#6057)
[AIRFLOW-5883] Improve count() queries in a few places (#6532)
[AIRFLOW-5811] Add metric for externally killed task count (#6466)
[AIRFLOW-5758] Support the custom cursor classes for the PostgreSQL hook (#6432)
[AIRFLOW-5766] Use httpbin.org in http_default (#6438)
[AIRFLOW-5798] Set default ExternalTaskSensor.external_task_id (#6431)
[AIRFLOW-5643] Reduce duplicated logic in S3Hook (#6313)
[AIRFLOW-5562] Skip grant single DAG permissions for Admin role. (#6199)
[AIRFLOW-6192] Stop creating Hook from SFTPSensor.init (#6748)
[AIRFLOW-5749][AIRFLOW-4162] Support the "blocks" component for the Slack operators (#6418)
[AIRFLOW-5693] Support the "blocks" component for the Slack messages (#6364)
[AIRFLOW-5714] Collect SLA miss emails only from tasks missed SLA (#6384)
[AIRFLOW-5049] Add validation for src_fmt_configs in bigquery hook (#5671)
[AIRFLOW-6177] Log DAG processors timeout event at error level, not info (#6731)
[AIRFLOW-6180] Improve kerberos init in pytest conftest (#6735)
[AIRFLOW-6159] Change logging level of the heartbeat message to DEBUG (#6716)
[AIRFLOW-6144] Improve the log message of airflow scheduler (#6710)
[AIRFLOW-6099] Add host name to task runner log (#6688)
[AIRFLOW-6045] Error on failed execution of compile_assets (#6640)
[AIRFLOW-5144] Add confirmation on delete button click (#6745)
[AIRFLOW-6099] Add host name to task runner log (#6688)
[AIRFLOW-5915] Add support for the new documentation theme (#6563)
[AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
[AIRFLOW-5888] Use psycopg2-binary for postgres operations (#6533)
[AIRFLOW-5870] Allow -1 for pool size and optimise pool query (#6520)

Bug Fixes

[AIRFLOW-XXX] Bump Jira version to fix issue with async
[AIRFLOW-XXX] Add encoding to fix Cyrillic output when reading back task logs (#6631)
[AIRFLOW-5304] Fix extra links in BigQueryOperator with multiple queries (#5906)
[AIRFLOW-6268] Prevent (expensive) ajax calls on home page when no dags visible (#6839)
[AIRFLOW-6259] Reset page to 1 with each new search for dags (#6828)
[AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema (#6754)
[AIRFLOW-3632] Only replace microseconds if execution_date is None in trigger_dag REST API (#6380)
[AIRFLOW-5458] Bump Flask-AppBuilder to 2.2.0 (for Python >= 3.6) (#6607)
[AIRFLOW-5072] gcs_hook should download files once (#5685)
[AIRFLOW-5744] Environment variables not correctly set in Spark submit operator (#6796)
[AIRFLOW-3189] Remove schema from DbHook.get_uri response if None (#6833)
[AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI (#6758)
[AIRFLOW-5889] Make polling for AWS Batch job status more resillient (#6765)
[AIRFLOW-6043] Fix bug in UI when "filtering by root" to display section of dag (#6638)
[AIRFLOW-6033] Fix UI Crash at "Landing Times" when task_id is changed (#6635)
[AIRFLOW-3745] Fix viewer not able to view dag details (#4569)
[AIRFLOW-6175] Fixes bug when tasks get stuck in "scheduled" state (#6732)
[AIRFLOW-5463] Make Variable.set when replacing an atomic operation (#6807)
[AIRFLOW-5582] Add get_autocommit to JdbcHook (#6232)
[AIRFLOW-5867] Fix webserver unit_test_mode data type (#6517)
[AIRFLOW-5819] Update AWSBatchOperator default value (#6473)
[AIRFLOW-5709] Fix regression in setting custom operator resources. (#6331)
[AIRFLOW-5658] Fix broken navigation links (#6374)
[AIRFLOW-5727] SqoopHook: Build --connect parameter only if port/schema are defined (#6397)
[AIRFLOW-5695] use RUNNING_DEPS to check run from UI (#6367)
[AIRFLOW-6254] obscure conn extra in logs (#6817)
[AIRFLOW-4824] Add charset handling for SqlAlchemy engine for MySqlHook (#6816)
[AIRFLOW-6091] Add flushing in execute method for BigQueryCursor (#6683)
[AIRFLOW-6256] Ensure Jobs table is cleared when resetting DB (#6818)
[AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery (#6297)
[AIRFLOW-5179] Remove top level init.py (#5818)
[AIRFLOW-5660] Attempt to find the task in DB from Kubernetes pod labels (#6340)
[AIRFLOW-6241] Fix typo in airflow/gcp/operator/dataflow.py (#6806)
[AIRFLOW-6171] Apply .airflowignore to correct subdirectories (#6784)
[AIRFLOW-6018] Display task instance in table during backfilling (#6612)
[AIRFLOW-6189] Reduce the maximum test duration to 8 minutes (#6744)
[AIRFLOW-6141] Remove ReadyToRescheduleDep if sensor mode == poke (#6704)
[AIRFLOW-6054] Add a command that starts the database consoles (#6653)
[AIRFLOW-6047] Simplify the logging configuration template (#6644)
[AIRFLOW-6017] Exclude PULL_REQUEST_TEMPLATE.md from RAT check (#6611)
[AIRFLOW-4560] Fix Tez queue parameter name in mapred_queue (#5315)
[AIRFLOW-2143] Fix TaskTries graph counts off-by-1 (#6526)
[AIRFLOW-5873] KubernetesPodOperator fixes and test (#6523)
[AIRFLOW-5869] BugFix: Some Deserialized tasks have no start_date (#6519)
[AIRFLOW-4020] Remove DAG edit permissions from Viewer role (#4845)
[AIRFLOW-6263] Fix broken WinRM integration (#6832)
[AIRFLOW-5836] Pin azure-storage-blob version to <12 (#6486)
[AIRFLOW-4488] Fix typo for non-RBAC UI in max_active_runs_per_dag (#6778)
[AIRFLOW-5942] Pin PyMSSQL to <3.0 (#6592)
[AIRFLOW-5451] SparkSubmitHook don't set default namespace (#6072)
[AIRFLOW-6271] Printing log files read during load_test_config (#6842)
[AIRFLOW-6308] Unpin Kombu for Python 3

Misc/Internal

[AIRFLOW-6009] Switch off travis_wait for regular tests (#6600)
[AIRFLOW-6226] Always reset warnings in tests
[AIRFLOW-XXX] Remove cyclic imports and pylint hacks in Serialization (#6601)
[AIRFLOW-XXX] Bump npm from 6.4.1 to 6.13.4 in /airflow/www (#6815)
[AIRFLOW-XXX] Remove executable permission from file
[AIRFLOW-XXX] Group AWS & Azure dependencies (old [emr] etc. extra still work)
[AIRFLOW-5487] Fix unused warning var (#6111)
[AIRFLOW-5925] Relax funcsigs and psutil version requirements (#6580)
[AIRFLOW-5740] Fix Transient failure in Slack test (#6407)
[AIRFLOW-6058] Running tests with pytest (#6472)
[AIRFLOW-6066] Added pre-commit checks for accidental debug stmts (#6662)
[AIRFLOW-6060] Improve conf_vars context manager (#6658)
[AIRFLOW-6044] Standardize the Code Structure in kube_pod_operator.py (#6639)
[AIRFLOW-4940] Simplify tests of DynamoDBToS3Operator (#6836)
[AIRFLOW-XXX] Update airflow-jira release management script (#6772)
[AIRFLOW-XXX] Add simple guidelines to unit test writing (#6846)
[AIRFLOW-6309] Fix stable build on Travis

Doc only changes

[AIRFLOW-6211] Doc how to use conda for local virtualenv (#6766)
[AIRFLOW-5855] Fix broken reference in custom operator doc (#6508)
[AIRFLOW-5875] Fix typo in example_qubole_operator.py (#6525)
[AIRFLOW-5702] Fix common docstring issues (#6372)
[AIRFLOW-5640] Document and test email parameters of BaseOperator (#6315)
[AIRFLOW-XXX] Improve description OpenFaaS Hook (#6187)
[AIRFLOW-XXX] GSoD: How to make DAGs production ready (#6515)
[AIRFLOW-XXX] Use full command in examples (#5973)
[AIRFLOW-XXX] Update docs to accurately describe the precedence of remote and local logs (#5607)
[AIRFLOW-XXX] Fix example "extras" field in mysql connect doc (#5285)
[AIRFLOW-XXX] Fix wrong inline code highlighting in docs (#5309)
[AIRFLOW-XXX] Group executors in one section (#5834)
[AIRFLOW-XXX] Add task lifecycle diagram to documentation (#6762)
[AIRFLOW-XXX] Highlight code blocks (#6243)
[AIRFLOW-XXX] Documents about task_concurrency and pool (#5262)
[AIRFLOW-XXX] Fix incorrect docstring parameter (#6649)
[AIRFLOW-XXX] Add link to XCom section in concepts.rst (#6791)
[AIRFLOW-XXX] Update kubernetes doc with correct path (#6774)
[AIRFLOW-XXX] Add information how to configure pytest runner (#6736)
[AIRFLOW-XXX] more GSOD improvements (#6585)
[AIRFLOW-XXX] Clarified a grammatically incorrect sentence (#6667)
[AIRFLOW-XXX] Add notice for Mesos Executor deprecation in docs (#6712)
[AIRFLOW-XXX] Update list of pre-commits (#6603)
[AIRFLOW-XXX] Updates to Breeze documentation from GSOD (#6285)
[AIRFLOW-XXX] Clarify daylight savings time behavior (#6324)
[AIRFLOW-XXX] GSoD: Adding 'Create a custom operator' doc (#6348)
[AIRFLOW-XXX] Add resources & links to CONTRIBUTING.rst (#6405)
[AIRFLOW-XXX] Update chat channel details from gitter to slack (#4149)
[AIRFLOW-XXX] Add logo info to readme (#6349)
[AIRFLOW-XXX] Fixed case problem with CONTRIBUTING.rst (#6329)
[AIRFLOW-XXX] Google Season of Docs updates to CONTRIBUTING doc (#6283)

airflow - Airflow 1.10.6rc1, 2019-10-18

Published by ashb about 5 years ago

New Features

[AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546)
[AIRFLOW-4741] Optionally report task errors to Sentry (#5407)
[AIRFLOW-4939] Add default_task_retries config (#5570)
[AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130)
[AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789)
[AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743)

Improvements

[AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738)
[AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737)
[AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736)
[AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330)
[AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034)
[AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194)
[AIRFLOW-5657] Update the upper bound for dill dependency (#6334)
[AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891)
[AIRFLOW-4939] Simplify Code for Default Task Retries (#6233)
[AIRFLOW-5126] Read aws_session_token in extra_config of the aws hook (#6303)
[AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302)
[AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304)
[AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309)
[AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153)
[AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) (#6163)
[AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012)
[AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062)
[AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059)
[AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948)
[AIRFLOW-3888] HA for Hive metastore connection (#4708)
[AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873)
[AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768)
[AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214)
[AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696)
[AIRFLOW-2891] allow configurable docker_operator container name (#5689)
[AIRFLOW-4285] Update task dependency context definition and usage (#5079)
[AIRFLOW-5142] Fixed flaky Cassandra test (#5758)
[AIRFLOW-5218] Less polling of AWS Batch job status (#5825)
[AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589)
[AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339)
[AIRFLOW-5560] Allow no confirmation on reset dags in airflow backfill command (#6195)
[AIRFLOW-5280] conn: Remove aws_default's default region name (#5879)
[AIRFLOW-5528] end_of_log_mark should not be a log record (#6159)
[AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154)
[AIRFLOW-4835] Refactor operator render_template (#5461)

Bug Fixes

[AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078)
[Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635)
[AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747)
[AIRFLOW-5497] Update docstring in airflow/utils/dag_processing.py (#6314)
Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance (#5908)
[AIRFLOW-5634] Don't allow editing of DagModelView (#6308)
[AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102)
[AIRFLOW-5387] Fix "show paused" pagination bug (#6100)
[AIRFLOW-5489] Remove unneeded assignment of variable (#6106)
[AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108)
[AIRFLOW-5492] added missing docstrings (#6107)
[AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125)
[AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101)
[AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678)
[AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909)
[AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185)
[AIRFLOW-5306] Fix the display of links when they contain special characters (#5904)
[AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841)
[AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237)
[AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307)
[AIRFLOW-4833] Allow to set Jinja env options in DAG declaration (#5943)
[AIRFLOW-5408] Fix env variable name in Kubernetes template (#6016)
[AIRFLOW-5102] Worker jobs should terminate themselves if they can't heartbeat (#6284)
[AIRFLOW-5572] Clear task reschedules when clearing task instances (#6217)
[AIRFLOW-5543] Fix tooltip disappears in tree and graph view (RBAC UI) (#6174)
[AIRFLOW-5444] Fix action_logging so that request.form for POST is logged (#6064)
[AIRFLOW-5484] fix PigCliHook has incorrect named parameter (#6112)
[AIRFLOW-5342] Fix MSSQL breaking task_instance db migration (#6014)
[AIRFLOW-5556] Add separate config for timeout from scheduler dag processing (#6186)
[AIRFLOW-4858] Deprecate "Historical convenience functions" in airflow.configuration (#5495) (#6144)
[AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics (#6157)
[AIRFLOW-5419] Use sudo to kill cleared tasks when running with impersonation (#6026) (#6176)
[AIRFLOW-5537] Yamllint is not needed as dependency on host
[AIRFLOW-5536] Better handling of temporary output files
[AIRFLOW-5535] Fix name of VERBOSE parameter
[AIRFLOW-5519] Fix sql_to_gcs operator missing multi-level default args by adding apply_defaults decorator (#6146)
[AIRFLOW-5210] Make finding template files more efficient (#5815)
[AIRFLOW-5447] Scheduler stalls because second watcher thread in default args (#6129)

Doc-only changes

[AIRFLOW-5574] Fix Google Analytics script loading (#6218)
[AIRFLOW-5588] Add Celery's architecture diagram (#6247)
[AIRFLOW-5521] Fix link to GCP documentation (#6150)
[AIRFLOW-5398] Update contrib example DAGs to context manager (#5998)
[AIRFLOW-5268] Apply same DAG naming conventions as in literature (#5874)
[AIRFLOW-5101] Fix inconsistent owner value in examples (#5712)
[AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319)
[AIRFLOW-XXX] Fix Documentation for adding extra Operator Links (#6301)
[AIRFLOW-XXX] Add section on task lifecycle & correct casing in docs (#4681)
[AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally omitted from UPDATING.md (#6240)
[AIRFLOW-XXX] Improve format in code-block directives (#6242)
[AIRFLOW-XXX] Format Sendgrid docs (#6245)
[AIRFLOW-XXX] Update to new logo (#6066)
[AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291)
[AIRFLOW-XXX] Add message about breaking change in DAG#get_task_instances in 1.10.4 (#6226)
[AIRFLOW-XXX] Fix incorrect units in docs for metrics using Timers (#6152)
[AIRFLOW-XXX] Fix backtick issues in .rst files & Add Precommit hook (#6162)
[AIRFLOW-XXX] Update documentation about variables forcing answer (#6158)
[AIRFLOW-XXX] Add a third way to configure authorization (#6134)
[AIRFLOW-XXX] Add example of running pre-commit hooks on single file (#6143)
[AIRFLOW-XXX] Add information about default pool to docs (#6019)
[AIRFLOW-XXX] Make Breeze The default integration test environment (#6001)

Misc/Internal

[AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline (#6358) (#6361)
[AIRFLOW-5533] Fixed failing CRON build (#6167)
[AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant from library (#5744)
[AIRFLOW-5369] Adds interactivity to pre-commits (#5976)
[AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165)
[AIRFLOW-4686] Make dags Pylint compatible (#5753)
[AIRFLOW-4864] Remove calls to load_test_config (#5502)
[AIRFLOW-XXX] Pin version of mypy so we are stable over time (#6198)
[AIRFLOW-XXX] Add tests that got missed from #5127
[AIRFLOW-4928] Move config parses to class properties inside DagBag (#5557)
[AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627)
[AIRFLOW-5580] Add base class for system test (#6229)

airflow - Airflow 1.10.5, 2019-08-30

Published by kaxil about 5 years ago

New Features

[AIRFLOW-1498] Add feature for users to add Google Analytics to Airflow UI (#5850)
[AIRFLOW-4074] Add option to add labels to Dataproc jobs (#5606)
[AIRFLOW-4846] Allow specification of an existing secret containing git credentials for init containers (#5475)

Improvements

[AIRFLOW-5335] Update GCSHook methods so they need min IAM perms (#5939)
[AIRFLOW-2692] Allow AWS Batch Operator to use templates in job_name parameter (#3557)
[AIRFLOW-4768] Add Timeout parameter in example_gcp_video_intelligence (#5862)
[AIRFLOW-5165] Make Dataproc highly available (#5781)
[AIRFLOW-5139] Allow custom ES configs (#5760)
[AIRFLOW-5340] Fix GCP DLP example (#594)
[AIRFLOW-5211] Add pass_value to template_fields BigQueryValueCheckOperator (#5816)
[AIRFLOW-5113] Support icon url in slack web hook (#5724)
[AIRFLOW-4230] bigquery schema update options should be a list (#5766)
[AIRFLOW-1523] Clicking on Graph View should display related DAG run (#5866)
[AIRFLOW-5027] Generalized CloudWatch log grabbing for ECS and SageMaker operators (#5645)
[AIRFLOW-5244] Add all possible themes to default_webserver_config.py (#5849)
[AIRFLOW-5245] Add more metrics around the scheduler (#5853)
[AIRFLOW-5048] Improve display of Kubernetes resources (#5665)
[AIRFLOW-5284] Replace deprecated log.warn by log.warning (#5881)
[AIRFLOW-5276] Remove unused helpers from airflow.utils.helpers (#5878)
[AIRFLOW-4316] Support setting kubernetes_environment_variables config section from env var (#5668)

Bug fixes

[AIRFLOW-5168] Fix Dataproc operators that failed in 1.10.4 (#5928)
[AIRFLOW-5136] Fix Bug with Incorrect template_fields in DataProc{*} Operators (#5751)
[AIRFLOW-5169] Pass GCP Project ID explicitly to StorageClient in GCSHook (#5783)
[AIRFLOW-5302] Fix bug in none_skipped Trigger Rule (#5902)
[AIRFLOW-5350] Fix bug in the num_retires field in BigQueryHook (#5955)
[AIRFLOW-5145] Fix rbac ui presents false choice to encrypt or not encrypt variable values (#5761)
[AIRFLOW-5104] Set default schedule for GCP Transfer operators (#5726)
[AIRFLOW-4462] Use datetime2 column types when using MSSQL backend (#5707)
[AIRFLOW-5282] Add default timeout on kubeclient & catch HTTPError (#5880)
[AIRFLOW-5315] TaskInstance not updating from DB when user changes executor_config (#5926)
[AIRFLOW-4013] Mark success/failed is picking all execution date (#5616)
[AIRFLOW-5152] Fix autodetect default value in GoogleCloudStorageToBigQueryOperator(#5771)
[AIRFLOW-5100] Airflow scheduler does not respect safe mode setting (#5757)
[AIRFLOW-4763] Allow list in DockerOperator.command (#5408)
[AIRFLOW-5260] Allow empty uri arguments in connection strings (#5855)
[AIRFLOW-5257] Fix ElasticSearch log handler errors when attemping to close logs (#5863)
[AIRFLOW-1772] Google Updated Sensor doesnt work with CRON expressions (#5730)
[AIRFLOW-5085] When you run kubernetes git-sync test from TAG, it fails (#5699)
[AIRFLOW-5258] ElasticSearch log handler, has 2 times of hours (%H and %I) in _clean_execution_dat (#5864)
[AIRFLOW-5348] Escape Label in deprecated chart view when set via JS (#5952)
[AIRFLOW-5357] Fix Content-Type for exported variables.json file (#5963)
[AIRFLOW-5109] Fix process races when killing processes (#5721)
[AIRFLOW-5240] Latest version of Kombu is breaking airflow for py2

Misc/Internal

[AIRFLOW-5111] Remove apt-get upgrade from the Dockerfile (#5722)
[AIRFLOW-5209] Fix Documentation build (#5814)
[AIRFLOW-5083] Check licence image building can be faster and moved to before-install (#5695)
[AIRFLOW-5119] Cron job should always rebuild everything from scratch (#5733)
[AIRFLOW-5108] In the CI local environment long-running kerberos might fail sometimes (#5719)
[AIRFLOW-5092] Latest python image should be pulled locally in force_pull_and_build (#5705)
[AIRFLOW-5225] Consistent licences can be added automatically for all JS files (#5827)
[AIRFLOW-5229] Add licence to all other file types (#5831)
[AIRFLOW-5227] Consistent licences for all .sql files (#5829)
[AIRFLOW-5161] Add pre-commit hooks to run static checks for only changed files (#5777)
[AIRFLOW-5159] Optimise checklicence image build (do not build if not needed) (#5774)
[AIRFLOW-5263] Show diff on failure of pre-commit checks (#5869)
[AIRFLOW-5204] Shell files should be checked with shellcheck and have identical licence (#5807)
[AIRFLOW-5233] Check for consistency in whitespace (tabs/eols) and common problems (#5835)
[AIRFLOW-5247] Getting all dependencies from NPM can be moved up in Dockerfile (#5870)
[AIRFLOW-5143] Corrupted rat.jar became part of the Docker image (#5759)
[AIRFLOW-5226] Consistent licences for all html JINJA templates (#5828)
[AIRFLOW-5051] Coverage is not properly reported in the new CI system (#5732)
[AIRFLOW-5239] Small typo and incorrect tests in CONTRIBUTING.md (#5844)
[AIRFLOW-5287] Checklicence base image is not pulled (#5886)
[AIRFLOW-5301] Some not-yet-available files from breeze are committed to master (#5901)
[AIRFLOW-5285] Pre-commit pylint runs over todo files (#5884)
[AIRFLOW-5288] Temporary container for static checks should be auto-removed (#5887)
[AIRFLOW-5326] Fix teething proglems for Airflow breeze (#5933)
[AIRFLOW-5206] All .md files should have all common licence, TOC (where applicable) (#5809)
[AIRFLOW-5329] Easy way to add local files to docker (#5933)
[AIRFLOW-4027] Make experimental api tests more stateless (#4854)

Doc-only changes

[AIRFLOW-XXX] Fixed Azkaban link (#5865)
[AIRFLOW-XXX] Remove duplicate lines from CONTRIBUTING.md (#5830)
[AIRFLOW-XXX] Fix incorrect docstring parameter in SchedulerJob (#5729)

airflow -

Published by ashb over 5 years ago

New Feature

[AIRFLOW-4232] Add none_skipped trigger rule (#5032)
[AIRFLOW-3971] Add Google Cloud Natural Language operators (#4980)
[AIRFLOW-4069] Add Opsgenie Alert Hook and Operator (#4903)
[AIRFLOW-3552] Fix encoding issue in ImapAttachmentToS3Operator (#5040)
[AIRFLOW-3552] Add ImapAttachmentToS3Operator (#4476)
[AIRFLOW-1526] Add dingding hook and operator (#4895)
[AIRFLOW-3490] Add BigQueryHook's Ability to Patch Table/View (#4299)
[AIRFLOW-3918] Add ssh private-key support to git-sync for KubernetesExecutor (#4777)
[AIRFLOW-3659] Create Google Cloud Transfer Service Operators (#4792)
[AIRFLOW-3939] Add Google Cloud Translate operator (#4755)
[AIRFLOW-3541] Add Avro logical type conversion to bigquery hook (#4553)
[AIRFLOW-4106] instrument staving tasks in pool (#4927)
[AIRFLOW-2568] Azure Container Instances operator (#4121)
[AIRFLOW-4107] instrument executor (#4928)
[AIRFLOW-4033] record stats of task duration (#4858)
[AIRFLOW-3892] Create Redis pub sub sensor (#4712)
[AIRFLOW-4124] add get_table and get_table_location in aws_glue_hook and tests (#4942)
[AIRFLOW-1262] Adds missing docs for email configuration (#4557)
[AIRFLOW-3701] Add Google Cloud Vision Product Search operators (#4665)
[AIRFLOW-3766] Add support for kubernetes annotations (#4589)
[AIRFLOW-3741] Add extra config to Oracle hook (#4584)
[AIRFLOW-1262] Allow configuration of email alert subject and body (#2338)
[AIRFLOW-2985] Operators for S3 object copying/deleting (#3823)
[AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators (#3828)
[AIRFLOW-3799] Add compose method to GoogleCloudStorageHook (#4641)
[AIRFLOW-3218] add support for poking a whole DAG (#4058)
[AIRFLOW-3315] Add ImapAttachmentSensor (#4161)
[AIRFLOW-3556] Add cross join set dependency function (#4356)

Improvement

[AIRFLOW-3823] Exclude branch's downstream tasks from the tasks to skip (#4666)
[AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes (#4648)
[AIRFLOW-4247] Template Region on the DataprocOperators (#5046)
[AIRFLOW-4008] add envFrom for Kubernetes Executor (#4952)
[AIRFLOW-3947] Flash msg for no DAG-level access error (#4767)
[AIRFLOW-3287] Moving database clean-up code into the CoreTest.tearDown() (#4122)
[AIRFLOW-4058] Name models test file to get automatically picked up (#4901)
[AIRFLOW-3830] Remove DagBag from /dag_details (#4831)
[AIRFLOW-3596] Clean up undefined template variables. (#4401)
[AIRFLOW-3573] Remove DagStat table (#4378)
[AIRFLOW-3623] Fix bugs in Download task logs (#5005)
[AIRFLOW-4173] Improve SchedulerJob.process_file() (#4993)
[AIRFLOW-3540] Warn if old airflow.cfg file is found (#5006)
[AIRFLOW-4000] Return response when no file (#4822)
[AIRFLOW-3383] Rotate fernet keys. (#4225)
[AIRFLOW-3003] Pull the krb5 image instead of building (#3844)
[AIRFLOW-3862] Check types with mypy. (#4685)
[AIRFLOW-251] Add option SQL_ALCHEMY_SCHEMA parameter to specify schema for metadata (#4199)
[AIRFLOW-1814] Temple PythonOperator {op_args,op_kwargs} fields (#4691)
[AIRFLOW-3730] Standarization use of logs mechanisms (#4556)
[AIRFLOW-3770] Validation of documentation on CI] (#4593)
[AIRFLOW-3866] Run docker-compose pull silently in CI (#4688)
[AIRFLOW-3685] Move licence header check (#4497)
[AIRFLOW-3670] Add stages to Travis build (#4477)
[AIRFLOW-3937] KubernetesPodOperator support for envFrom configMapRef and secretRef (#4772)
[AIRFLOW-3408] Remove outdated info from Systemd Instructions (#4269)
[AIRFLOW-3202] add missing documentation for AWS hooks/operator (#4048)
[AIRFLOW-3908] Add more Google Cloud Vision operators (#4791)
[AIRFLOW-2915] Add example DAG for GoogleCloudStorageToBigQueryOperator (#3763)
[AIRFLOW-3062] Add Qubole in integration docs (#3946)
[AIRFLOW-3288] Add SNS integration (#4123)
[AIRFLOW-3148] Remove unnecessary arg "parameters" in RedshiftToS3Transfer (#3995)
[AIRFLOW-3049] Add extra operations for Mongo hook (#3890)
[AIRFLOW-3559] Add missing options to DatadogHook. (#4362)
[AIRFLOW-1191] Simplify override of spark submit command. (#4360)
[AIRFLOW-3155] Add ability to filter by a last modified time in GCS Operator (#4008)
[AIRFLOW-2864] Fix docstrings for SubDagOperator (#3712)
[AIRFLOW-4062] Improve docs on install extra package commands (#4966)
[AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME (#4705)
[AIRFLOW-4002] Option to open debugger on errors in airflow test. (#4828)
[AIRFLOW-3997] Extend Variable.get so it can return None when var not found (#4819)
[AIRFLOW-4009] Fix docstring issue in GCSToBQOperator (#4836)
[AIRFLOW-3980] Unify logger (#4804)
[AIRFLOW-4076] Correct port type of beeline_default in init_db (#4908)
[AIRFLOW-4046] Add validations for poke_interval & timeout for Sensor (#4878)
[AIRFLOW-3744] Abandon the use of obsolete aliases of methods (#4568)
[AIRFLOW-3865] Add API endpoint to get python code of dag by id (#4687)
[AIRFLOW-3516] Support to create k8 worker pods in batches (#4434)
[AIRFLOW-2843] Add flag in ExternalTaskSensor to check if external DAG/task exists (#4547)
[AIRFLOW-2224] Add support CSV files in MySqlToGoogleCloudStorageOperator (#4738)
[AIRFLOW-3895] GoogleCloudStorageHook/Op create_bucket takes optional resource params (#4717)
[AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view (#4774)
[AIRFLOW-4006] Make better use of Set in AirflowSecurityManager (#4833)
[AIRFLOW-3917] Specify alternate kube config file/context when running out of cluster (#4859)
[AIRFLOW-3911] Change Harvesting DAG parsing results to DEBUG log level (#4729)
[AIRFLOW-3584] Use ORM DAGs for index view. (#4390)
[AIRFLOW-2821] Refine Doc "Plugins" (#3664)
[AIRFLOW-3561] Improve queries (#4368)
[AIRFLOW-3600] Remove dagbag from trigger (#4407)
[AIRFLOW-3713] Updated documentation for GCP optional project_id (#4541)
[AIRFLOW-2767] Upgrade gunicorn to 19.5.0 to avoid moderate-severity CVE (#4795)
[AIRFLOW-3795] provide_context param is now used (#4735)
[AIRFLOW-4012] Upgrade tabulate to 0.8.3 (#4838)
[AIRFLOW-3623] Support download logs by attempts from UI (#4425)
[AIRFLOW-2715] Use region setting when launching Dataflow templates (#4139)
[AIRFLOW-3932] Update unit tests and documentation for safe mode flag. (#4760)
[AIRFLOW-3932] Optionally skip dag discovery heuristic. (#4746)
[AIRFLOW-3258] K8S executor environment variables section. (#4627)
[AIRFLOW-3931] set network, subnetwork when launching dataflow template (#4744)
[AIRFLOW-4095] Add template_fields for S3CopyObjectOperator & S3DeleteObjectsOperator (#4920)
[AIRFLOW-2798] Remove needless code from models.py
[AIRFLOW-3731] Constrain mysqlclient to <1.4 (#4558)
[AIRFLOW-3139] include parameters into log.info in SQL operators, if any (#3986)
[AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks (#4043)
[AIRFLOW-3933] Fix various typos (#4747)
[AIRFLOW-3905] Allow using "parameters" in SqlSensor (#4723)
[AIRFLOW-2761] Parallelize enqueue in celery executor (#4234)
[AIRFLOW-3540] Respect environment config when looking up config file. (#4340)
[AIRFLOW-2156] Parallelize Celery Executor task state fetching (#3830)
[AIRFLOW-3702] Add backfill option to run backwards (#4676)
[AIRFLOW-3821] Add replicas logic to GCP SQL example DAG (#4662)
[AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator (#4347)
[AIRFLOW-3647] Add archives config option to SparkSubmitOperator (#4467)
[AIRFLOW-3802] Updated documentation for HiveServer2Hook (#4647)
[AIRFLOW-3817] Corrected task ids returned by BranchPythonOperator to match the dummy operator ids (#4659)
[AIRFLOW-3782] Clarify docs around celery worker_autoscale in default_airflow.cfg (#4609)
[AIRFLOW-1945] Add Autoscale config for Celery workers (#3989)
[AIRFLOW-3590] Change log message of executor exit status (#4616)
[AIRFLOW-3591] Fix start date, end date, duration for rescheduled tasks (#4502)
[AIRFLOW-3709] Validate allowed_states for ExternalTaskSensor (#4536)
[AIRFLOW-3522] Add support for sending Slack attachments (#4332)
[AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www only) (#4373)
[AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www_rbac only) (#4373)
[AIRFLOW-3044] Dataflow operators accept templated job_name param (#3887)
[AIRFLOW-3023] Fix docstring datatypes
[AIRFLOW-2928] Use uuid4 instead of uuid1 (#3779)
[AIRFLOW-2988] Run specifically python2 for dataflow (#3826)
[AIRFLOW-3697] Vendorize nvd3 and slugify (#4513)
[AIRFLOW-3692] Remove ENV variables to avoid GPL (#4506)
[AIRFLOW-3907] Upgrade flask and set cookie security flags. (#4725)
[AIRFLOW-3698] Add documentation for AWS Connection (#4514)
[AIRFLOW-3616][AIRFLOW-1215] Add aliases for schema with underscore (#4523)
[AIRFLOW-3375] Support returning multiple tasks with BranchPythonOperator (#4215)
[AIRFLOW-3742] Fix handling of "fallback" for AirflowConfigParsxer.getint/boolean (#4674)
[AIRFLOW-3742] Respect the fallback arg in airflow.configuration.get (#4567)
[AIRFLOW-3789] Fix flake8 3.7 errors. (#4617)
[AIRFLOW-3602] Improve ImapHook handling of retrieving no attachments (#4475)
[AIRFLOW-3631] Update flake8 and fix lint. (#4436)

Bug fixes

[AIRFLOW-4248] Fix 'FileExistsError' makedirs race in file_processor_handler (#5047)
[AIRFLOW-4240] State-changing actions should be POST requests (#5039)
[AIRFLOW-4246] Flask-Oauthlib needs downstream dependencies pinning due to breaking changes (#5045)
[AIRFLOW-3887] Downgrade dagre-d3 to 0.4.18 (#4713)
[AIRFLOW-3419] Fix S3Hook.select_key on Python3 (#4970)
[AIRFLOW-4127] Correct AzureContainerInstanceHook._get_instance_view's return (#4945)
[AIRFLOW-4172] Fix changes for driver class path option in Spark Submit (#4992)
[AIRFLOW-3615] Preserve case of UNIX socket paths in Connections (#4591)
[AIRFLOW-3417] ECSOperator: pass platformVersion only for FARGATE launch type (#4256)
[AIRFLOW-3884] Fixing doc checker, no warnings allowed anymore and fixed the current… (#4702)
[AIRFLOW-2652] implement / enhance baseOperator deepcopy
[AIRFLOW-4001] Update docs about how to run tests (#4826)
[AIRFLOW-3699] Speed up Flake8 (#4515)
[AIRFLOW-4160] Fix redirecting of 'Trigger Dag' Button in DAG Page (#4982)
[AIRFLOW-3650] Skip running on mysql for the flaky test (#4457)
[AIRFLOW-3423] Fix mongo hook to work with anonymous access (#4258)
[AIRFLOW-3982] Fix race condition in CI test (#4968)
[AIRFLOW-3982] Update DagRun state based on its own tasks (#4808)
[AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names (#4636)
[AIRFLOW-3945] Stop inserting row when permission views unchanged (#4764)
[AIRFLOW-4123] Add Exception handling for _change_state method in K8 Executor (#4941)
[AIRFLOW-3771] Minor refactor securityManager (#4594)
[AIRFLOW-987] pass kerberos cli args keytab and principal to kerberos.run() (#4238)
[AIRFLOW-3736] Allow int value in SqoopOperator.extra_import_options(#4906)
[AIRFLOW-4063] Fix exception string in BigQueryHook [2/2] (#4902)
[AIRFLOW-4063] Fix exception string in BigQueryHook (#4899)
[AIRFLOW-4037] Log response in SimpleHttpOperator even if the response check fails
[AIRFLOW-4044] The documentation of query_params in BigQueryOperator is wrong. (#4876)
[AIRFLOW-4015] Make missing API endpoints available in classic mode
[AIRFLOW-3153] Send DAG processing stats to statsd (#4748)
[AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#4209)
[AIRFLOW-4129] Escape HTML in generated tooltips (#4950)
[AIRFLOW-4070] AirflowException -> log.warning for duplicate task dependencies (#4904)
[AIRFLOW-4054] Fix assertEqualIgnoreMultipleSpaces util & add tests (#4886)
[AIRFLOW-3239] Fix test recovery further (#4074)
[AIRFLOW-4053] Fix KubePodOperator Xcom on Kube 1.13.0 (#4883)
[AIRFLOW-2961] Refactor tests.BackfillJobTest.test_backfill_examples test (#3811)
[AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was broken (#4415)
[AIRFLOW-3543] Fix deletion of DAG with rescheduled tasks (#4646)
[AIRFLOW-2548] Output plugin import errors to web UI (#3930)
[AIRFLOW-4019] Fix AWS Athena Sensor object has no attribute 'mode' (#4844)
[AIRFLOW-3758] Fix circular import in WasbTaskHandler (#4601)
[AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files (#4947)
[AIRFLOW-4100] Correctly JSON escape data for tree/graph views (#4921)
[AIRFLOW-3636] Fix a test introduced in #4425 (#4446)
[AIRFLOW-3977] Add examples of trigger rules in doc (#4805)
[AIRFLOW-2511] Fix improper failed session commit handling causing deadlocks (#4769)
[AIRFLOW-3962] Added graceful handling for creation of dag_run of a dag which doesn't have any task (#4781)
[AIRFLOW-3881] Correct to_csv row number (#4699)
[AIRFLOW-3875] Simplify SlackWebhookHook code and change docstring (#4696)
[AIRFLOW-3733] Don't raise NameError in HQL hook to_csv when no rows returned (#4560)
[AIRFLOW-3734] Fix hql not run when partition is None (#4561)
[AIRFLOW-3767] Correct bulk insert function (#4773)
[AIRFLOW-4087] remove sudo in basetaskrunner on_finish (#4916)
[AIRFLOW-3768] Escape search parameter in pagination controls (#4911)
[AIRFLOW-4045] Fix hard-coded URLs in FAB-based UI (#4914)
[AIRFLOW-3123] Use a stack for DAG context management (#3956)
[AIRFLOW-3060] DAG context manager fails to exit properly in certain circumstances
[AIRFLOW-3924] Fix try number in alert emails (#4741)
[AIRFLOW-4083] Add tests for link generation utils (#4912)
[AIRFLOW-2190] Send correct HTTP status for base_url not found (#4910)
[AIRFLOW-4015] Add get_dag_runs GET endpoint to "classic" API (#4884)
[AIRFLOW-3239] Enable existing CI tests (#4131)
[AIRFLOW-1390] Update Alembic to 0.9 (#3935)
[AIRFLOW-3885] Fix race condition in scheduler test (#4737)
[AIRFLOW-3885] ~10x speed-up of SchedulerJobTest suite (#4730)
[AIRFLOW-3780] Fix some incorrect when base_url is used (#4643)
[AIRFLOW-3807] Fix Graph View Highlighting of Tasks (#4653)
[AIRFLOW-3009] Import Hashable from collection.abc to fix Python 3.7 deprecation warning (#3849)
[AIRFLOW-2231] Fix relativedelta DAG schedule_interval (#3174)
[AIRFLOW-2641] Fix MySqlToHiveTransfer to handle MySQL DECIMAL correctly
[AIRFLOW-3751] Option to allow malformed schemas for LDAP authentication (#4574)
[AIRFLOW-2888] Add deprecation path for task_runner config change (#4851)
[AIRFLOW-2930] Fix celery excecutor scheduler crash (#3784)
[AIRFLOW-2888] Remove shell=True and bash from task launch (#3740)
[AIRFLOW-3885] ~2.5x speed-up for backfill tests (#4731)
[AIRFLOW-3885] ~20x speed-up of slowest unit test (#4726)
[AIRFLOW-2508] Handle non string types in Operators templatized fields (#4292)
[AIRFLOW-3792] Fix validation in BQ for useLegacySQL & queryParameters (#4626)
[AIRFLOW-3749] Fix Edit Dag Run page when using RBAC (#4613)
[AIRFLOW-3801] Fix DagBag collect dags invocation to prevent examples to be loaded (#4677)
[AIRFLOW-3774] Register blueprints with RBAC web app (#4598)
[AIRFLOW-3719] Handle StopIteration in CloudWatch logs retrieval (#4516)
[AIRFLOW-3108] Define get_autocommit method for MsSqlHook (#4525)
[AIRFLOW-3074] Add relevant ECS options to ECS operator. (#3908)
[AIRFLOW-3353] Upgrade Redis client (#4834)
[AIRFLOW-3250] Fix for Redis Hook for not authorised connection calls (#4090)
[AIRFLOW-2009] Fix dataflow hook connection-id (#4563)
[AIRFLOW-2190] Fix TypeError when returning 404 (#4596)
[AIRFLOW-2876] Update Tenacity to 4.12 (#3723)
[AIRFLOW-3923] Update flask-admin dependency to 1.5.3 to resolve security vulnerabilities from safety (#4739)
[AIRFLOW-3683] Fix formatting of error message for invalid TriggerRule (#4490)
[AIRFLOW-2787] Allow is_backfill to handle NULL DagRun.run_id (#3629)
[AIRFLOW-3780] Fix some incorrect when base_url is used (#4643)
[AIRFLOW-3639] Fix request creation in Jenkins Operator (#4450)
[AIRFLOW-3779] Don't install enum34 backport when not needed (#4620)
[AIRFLOW-3079] Improve migration scripts to support MSSQL Server (#3964)
[AIRFLOW-2735] Use equality, not identity, check for detecting AWS Batch failures[]
[AIRFLOW-2706] AWS Batch Operator should use top-level job state to determine status
[AIRFLOW-XXX] Fix typo in http_operator.py
[AIRFLOW-XXX] Solve lodash security warning (#4820)
[AIRFLOW-XXX] Pin version of tornado pulled in by Celery. (#4815)
[AIRFLOW-XXX] Upgrade FAB to 1.12.3 (#4694)
[AIRFLOW-XXX] Pin pinodb dependency (#4704)
[AIRFLOW-XXX] Pin version of Pip in tests to work around pypa/pip#6163 (#4576)
[AIRFLOW-XXX] Fix spark submit hook KeyError (#4578)
[AIRFLOW-XXX] Pin psycopg2 due to breaking change (#5036)
[AIRFLOW-XXX] Pin Sendgrid dep. (#5031)
[AIRFLOW-XXX] Fix flaky test - test_execution_unlimited_parallelism (#4988)

Misc/Internal

[AIRFLOW-4144] add description of is_delete_operator_pod (#4943)
[AIRFLOW-3476][AIRFLOW-3477] Move Kube classes out of models.py (#4443)
[AIRFLOW-3464] Move SkipMixin out of models.py (#4386)
[AIRFLOW-3463] Move Log out of models.py (#4639)
[AIRFLOW-3458] Move connection tests (#4680)
[AIRFLOW-3461] Move TaskFail out of models.py (#4630)
[AIRFLOW-3462] Move TaskReschedule out of models.py (#4618)
[AIRFLOW-3474] Move SlaMiss out of models.py (#4608)
[AIRFLOW-3475] Move ImportError out of models.py (#4383)
[AIRFLOW-3459] Move DagPickle to separate file (#4374)
[AIRFLOW-3925] Don't pull docker-images on pretest (#4740)
[AIRFLOW-4154] Correct string formatting in jobs.py (#4972)
[AIRFLOW-3458] Deprecation path for moving models.Connection
[AIRFLOW-3458] Move models.Connection into separate file (#4335)
[AIRFLOW-XXX] Remove old/non-test files that nose ignores (#4930)

Doc-only changes

[AIRFLOW-3996] Add view source link to included fragments
[AIRFLOW-3811] automatic generation of API Reference in docs (#4788)
[AIRFLOW-3810] Remove duplicate autoclass directive (#4656)
[AIRFLOW-XXX] Mention that statsd must be installed to gather metrics (#5038)
[AIRFLOW-XXX] Add contents to cli (#4825)
[AIRFLOW-XXX] fix check docs failure on CI (#4998)
[AIRFLOW-XXX] Fix syntax docs errors (#4789)
[AIRFLOW-XXX] Docs rendering improvement (#4684)
[AIRFLOW-XXX] Automatically link Jira/GH on doc's changelog page (#4587)
[AIRFLOW-XXX] Mention Oracle in the Extra Packages documentation (#4987)
[AIRFLOW-XXX] Drop deprecated sudo option; use default docker compose on Travis. (#4732)
[AIRFLOW-XXX] Update kubernetes.rst docs (#3875)
[AIRFLOW-XXX] Improvements to formatted content in documentation (#4835)
[AIRFLOW-XXX] Add Daniel to committer list (#4961)
[AIRFLOW-XXX] Add Xiaodong Deng to committers list
[AIRFLOW-XXX] Add history become ASF top level project (#4757)
[AIRFLOW-XXX] Move out the examples from integration.rst (#4672)
[AIRFLOW-XXX] Extract reverse proxy info to a separate file (#4657)
[AIRFLOW-XXX] Reduction of the number of warnings in the documentation (#4585)
[AIRFLOW-XXX] Fix GCS Operator docstrings (#4054)
[AIRFLOW-XXX] Fix Docstrings in Hooks, Sensors & Operators (#4137)
[AIRFLOW-XXX] Split guide for operators to multiple files (#4814)
[AIRFLOW-XXX] Split connection guide to multiple files (#4824)
[AIRFLOW-XXX] Remove almost all warnings from building docs (#4588)
[AIRFLOW-XXX] Add backreference in docs between operator and integration (#4671)
[AIRFLOW-XXX] Improve linking to classes (#4655)
[AIRFLOW-XXX] Mock optional modules when building docs (#4586)
[AIRFLOW-XXX] Update plugin macros documentation (#4971)
[AIRFLOW-XXX] Add missing docstring for 'autodetect' in GCS to BQ Operator (#4979)
[AIRFLOW-XXX] Add missing GCP operators to Docs (#4260)
[AIRFLOW-XXX] Fixing the issue in Documentation (#3756)
[AIRFLOW-XXX] Add Hint at user defined macros (#4885)
[AIRFLOW-XXX] Correct schedule_interval in Scheduler docs (#4157)
[AIRFLOW-XXX] Improve airflow-jira script to make RelManager's life easier (#4857)
[AIRFLOW-XXX] Add missing class references to docs (#4644)
[AIRFLOW-XXX] Fix typo (#4564)
[AIRFLOW-XXX] Add a doc about fab security (#4595)
[AIRFLOW-XXX] Speed up DagBagTest cases (#3974)
[AIRFLOW-XXX] Reduction of the number of warnings in the documentation (#4585)