kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

APACHE-2.0 License

Downloads
228.4K
Stars
7.2K
View Code on GitHub Visit Website View on X
Ecosystems: Java

Bot releases are visible (Hide)

kestra - v0.9.7

Published by github-actions[bot] over 1 year ago

Bug Fixes

  • jdbc: DateTimeFormatter can be reused in the JdbcMapper (Loïc Mathieu)

Chores

  • version: update to version 'v0.9.7'. (YannC)

Commits

  • fix: Updated count variable (YannC)
kestra - v0.10.0

Published by github-actions[bot] over 1 year ago

Kestra Release Note - 0.10.0

This new release brings important new core features such as Blueprints, new Script plugins, integration of basic authentication, and a secret function making the Open Source Edition even more robust.

Features

Blueprints

Blueprints are a curated, organized, and searchable catalog of ready-to-use examples designed to help you kick-start your workflow.

It’s available as a new section on the left bar menu and directly within the editor view**. All blueprints are validated and documented so that they just work.

You can easily customize and integrate them into your new or existing flows with a single click on the “Use” button.

Check out the documentation.

Untitled

Untitled2

Enterprise Edition users could write their own organization blueprints so they can share and grow a library of internal blueprints within teams.
Untitled3

Improved Support for Scripting

In this release, we revamped scripting tasks to bring more flexibility and control.
Each script task is now, by default, running in its own dedicated container.You can attach any Docker image you want if you need a specific one.
New tasks are available for Shell, Python, R, and Node.js:

  • Script: to write and run ad-hoc scripts. It comes with a beforeCommands property to execute any instruction needed before running the main script (installing dependencies for example)
    id: script
    namespace: release
    
    tasks:
      - id: run_python
        type: io.kestra.plugin.scripts.python.Script
        beforeCommands:
          - pip install requests
        warningOnStdErr: false
        script: | 
          import requests
          import json
    
          response = requests.get("https://api.github.com")
          data = response.json()
          print(data)
  • Command: to run arbitrary commands in a single task configuration. This task can be very powerful and associated with the WorkingDirectory task. For example, to clone a Git repository and then execute the corresponding scripts.
    id: command
    namespace: release
    
    tasks:
      - id: working
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
    
          - id: clone_repo
            type: io.kestra.plugin.git.Clone
            url: https://github.com/your_repository
    
          - id: run_python
            type: io.kestra.plugin.scripts.python.Command
            beforeCommands:
              - pip install -r requirements.txt
        dockerOptions:
        	    image: ghcr.io/kestra-io/pydata:latest
            commands:
              - python main.py

  • LocalFile: to create files in the local filesystem or to send files from the local filesystem to the internal storage. This task allows to uncouple inline scripts to their execution.

        id: "local-file"
    namespace: release
    tasks:
      - id: workingDir
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
        - id: inputFiles
          type: io.kestra.core.tasks.storages.LocalFiles
          inputs:
            hello.txt: "Hello World"
            address.json: "{{ outputs.myTaskId.uri }}"
        - id: bash
          type: io.kestra.plugin.scripts.shell.Command
          commands:
            - cat hello.txt
    	- sed -n 's/.*"country":"\([^"]*\)".*/\1/p' address.json

It also allows to expose outputs to internal storage. In the following example, we create two files with Bash commands and expose those into Kestra internal storage with the outputs property LocalFiles

    id: "local-files"
    namespace: release
    
    tasks:
      - id: workingDir
        type: io.kestra.core.tasks.flows.WorkingDirectory
        tasks:
        - id: bash
          type: io.kestra.plugin.scripts.shell.Command
          commands:
            - mkdir -p sub/dir
            - echo "Hello from Bash" >> sub/dir/bash1.txt
            - echo "Hello from Bash" >> sub/dir/bash2.txt
        - id: outputFiles
          type: io.kestra.core.tasks.storages.LocalFiles
          outputs:
            - sub/**

Those tasks run by default on DOCKER but you can use the runner: PROCESS property to run it as a process on the Kestra host.

Note: the old scripting tasks w ill be deprecated, removed from the core and being retro compatible within the new plugins.

DAG task

Creating directed acyclic graphs, a common pattern in data orchestration, was already possible in Kestra by through Flow dependencies. With a brand new DAG task, it’s now even easier to do; directly between tasks at the Flow level.

id: magicDAG
namespace: dev
tasks:
  - id: dag
    type: io.kestra.core.tasks.flows.Dag
    tasks:
      - task:
          id: customers
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_customers.csv
      - task:
          id: orders
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_orders.csv
      - task:
          id: payments
          type: io.kestra.plugin.fs.http.Download
          uri: https://raw.githubusercontent.com/dbt-labs/jaffle_shop/main/seeds/raw_payments.csv
      - task:
          id: transform
          type: io.kestra.core.tasks.log.Log
          message: |
            Transforming data from: 
              {{outputs.customers.uri}}
              {{outputs.orders.uri}}
              {{outputs.payments.uri}}
        dependsOn:
          - customers
          - orders
          - payments

image

Label on executions

So far, it was only possible to add labels on a flow level by adjusting the workflow code. This release adds the ability to set custom labels for specific Executions.

Also, the labels added on a flow level will be automatically propagated to Execution labels.

For example you can add labels for “experiment” executions so you can retrieve them in the UI easily.
Untitled4

Basic Authentication

The community was asking about authentication in the open source to secure their instance in production. We heard you ! You can now add basic authentication to your Kestra instance with username/password values in the Kestra configuration.

server:
  basic-auth:
    enabled: true
    username: admin
    password: *****

Secret Function

We introduced a secret() function allowing to read base64 secret value from environment variables.

id: secret
namespace: release
tasks:
  - id: get_secret
    type: io.kestra.core.tasks.debugs.Return
    format: '{{ secret("my_secret")}}'

Worker Group

This release introduce worker groups, a ****set of workers that can be targeted specifically for a task execution or a polling trigger evaluation. For this, the task or the polling trigger must define the workerGroup.key property with the key of the worker group to target. A default worker group can also be configured at the namespace level.

Here are common use cases in which Worker Groups can be beneficial:

  • Execute tasks and polling triggers on specific compute instances (e.g., a VM with a GPU and preconfigured CUDA drivers).
  • Execute tasks and polling triggers on a worker with a specific Operating System (e.g., a Windows server).
  • Restrict backend access to a set of workers (firewall rules, private networks, ...).
  • Execute tasks and polling triggers close to a remote backend (region selection).

Here is how you can ensure that a task is executed only by specific worker instances:

id: gpuTask
namespace: dev
tasks:
  - id: hello
    type: io.kestra.core.tasks.log.Log
    message: |
      This task will be executed on a specific remote worker that has access to a GPU
    workerGroup:
    key: gpuWorkerGroupKey

Polling Triggers

We have made significant improvements to the Polling trigger in the latest version of Kestra to enhance performance, strengthen security measures, simplify maintenance, and clarify the system architecture.
In previous versions, the trigger evaluation process was directly handled by the Scheduler. This meant that both the Worker and the Scheduler potentially accessed external systems, leading to manage the security and scalability of two services.

From now on, The Scheduler delegates the evaluation of polling triggers to the Worker. This architectural change has several advantages. Firstly, it simplifies the overall system architecture by ensuring that only the Worker needs to interact with external systems. This separation of responsibilities streamlines the flow of data and improves the overall system performance.
By reducing the direct access of the Scheduler to external systems, we also minimize potential security vulnerabilities and fortify the protection of sensitive data.

Plugins

All Changes

Bug Fixes

Documentation

Code Refactoring

Tests

Chores

Commits

  • 57569bf: chore(github) : add a blueprint issue template (#1539) (Anna Geller)
  • 57a67d4: Merge branch 'release' into develop (YannC)
  • 9367ef3: feat(ui+webserver): Blueprint tags filtered by query on blueprints (#1667) (brian-mulier-p) #1667
  • 97fbc93: feat/editor-validation-feedback-rework (#1662) (brian-mulier-p) #1662
  • ed80eb5: fix/non-b64-secrets-prevent-crash (#1711) (brian-mulier-p) #1711
kestra - v0.9.5

Published by github-actions[bot] over 1 year ago

Bug Fixes

  • ui: fix query issue in home component (YannC)

Chores

  • version: update snapshot version 'v0.9.5-SNAPSHOT'. (Ludovic DEHON)
  • version: update to version 'v0.9.5'. (YannC)
kestra - v0.9.4

Published by github-actions[bot] over 1 year ago

Bug Fixes

Chores

kestra - v0.9.2

Published by github-actions[bot] over 1 year ago

Features

Bug Fixes

  • ui: don't load task type and use flow instead (Ludovic DEHON)
  • core: TestUtils.mockTrigger() must call runContext.forScheduler() (Loïc Mathieu)
  • ui: Flow with dependencies can now be deleted (YannC)
  • core: multiple potential deadlocks and races on the AbstractScheduler (Loïc Mathieu)
  • core: redundant synchronization on the FlowListeners (Loïc Mathieu)
  • ui: validation of trigger on lowcode failed (#1440) (Ludovic DEHON)
  • tests: falling test depending on the backend (Ludovic DEHON)

Chores

kestra - v0.9.1

Published by github-actions[bot] over 1 year ago

Features

Bug Fixes

Chores

kestra - v0.9.0

Published by github-actions[bot] over 1 year ago

Kestra Release Note - 0.9.0

This new release focused on user experience improvements and bugs resolving.

New Features

  • ⭐ Flow Metrics Dashboard : since previous versions, any flows execution exposed Metrics. However there were no useful UI to explore and see those metrics. In this new release we created a new "Metrics tab" accessible at the Flow level. It may be useful to track specific metrics (count rows processed, explore third-party integration billing, see time of processes, etc.) and improve the observability of users tasks.
  • Save Flow as drafts: users are now able to save Flows as draft, even in the presence of errors in the Flow declaration. Useful when drafting flows and allows users to conveniently return to their work at a later time.

User Experiences Improvements

  • Manage editor windows size: editor can shows both source and topology or source and documentation. We added the ability to resize those two windows with a slider.
  • Ability to re-execute a Flow with the last inputs.
  • Task required properties are now listed first in the documentation pages.
  • Several improvements and fixes : improved flow and inputs validation, improved page redirections, fixed gantt display, improved error messages, fixed theme discrepancies etc.

Plugins

Documentation

Enterprise Edition

  • New License system.
  • Inherited variables are now available even if the user doesn’t have access to the namespace.
  • Namespaces now have a proper dashboard.
  • Fix stability issue when multiples conditions generate too many logs.
  • Improve login and authentication.
  • Fix bug on Gantt when using Each like tasks.

All Changes

Features

Bug Fixes

  • docs: fix invalid links on docker memory (Ludovic DEHON)
  • cli: failed test PluginDocCommandTest (#1173) #1173 (Loïc Mathieu)
  • docs: make docs index link relative (Ludovic DEHON)
  • core: Serialize null as null and not "null" in the ION file #1183 (Loïc Mathieu)
  • ui: allow topo edition when creating (#1193) #1193 (YannC)
  • ui: fix welcome redirection (#1200) #1200 (YannC)
  • core: wrong warning block in core task documentation #1212 (Loïc Mathieu)
  • ui: invalid link on taskrun list (Ludovic DEHON)
  • ui: display welcome always (Ludovic DEHON)
  • core: Flow trigger documentation #1233 (Loïc Mathieu)
  • don't display task base properties by default #1229 (Loïc Mathieu)
  • loadFlow with revision (#1259) #1259 (YannC)
  • ui: can't open object modal on task form edit (#1266) #1266 (YannC)
  • ui: prevent welcome redirection (#1265) #1265 (YannC)
  • ui: theme on editor was not applied (#1268) #1268 (YannC)
  • ui: Gantt can failed on low duration (#1256) #1256 (brian-mulier-p)
  • ui: better contrast in validation error tooltip (#1269) #1269 (YannC)
  • Display select column correctly in flows and executions list (#1253) #1253 (YannC)
  • ui: typo on replay confirm translation (brian.mulier)
  • core: Title does not disappear anymore on flow creation page (#1250) #1250 (YannC)
  • core: fix text error in flow metrics page (#1287) (YannC)
  • ui: correctly render markdown alert block (#1231) #1231 (Loïc Mathieu)
  • jdbc: properly select on namespace and sub-namespace (#1239) #1239 (Loïc Mathieu)
  • ui: change language don't refresh left menu (Ludovic DEHON)
  • improve error message for invalid task property #1291 (Loïc Mathieu)
  • ui: Fix an issue where topo form had missing properties (#1294) #1294 (YannC)
  • core: multiple condition log should not be sent on the database (Ludovic DEHON)
  • ui: fix plugin definition conflict (#1296) #1296 (YannC)
  • ui: Fix bad redirection on homepage (#1295) #1295 (YannC)
  • ui: topology orientation is now portrait by default on editor (#1298) #1298 (brian-mulier-p)
  • core: if tasks validation exception when the then property is null (#1314) #1314 (Loïc Mathieu)
  • docs: add note about prod setup without root user (#1317) #1317 (Anna Geller)
  • core: allow CREATED & PAUSED to be killed (#1313) #1313 (Loïc Mathieu)
  • core: correctly display additional properties in documentation (#1319) #1319 (Loïc Mathieu)
  • core: missing task and metric on the metric browser (#1312) #1312 (YannC)
  • core: allow usage of Pebble raw tag in our recursive renderer (#1316) #1316 (Loïc Mathieu)
  • ui: view format button (#1323) #1323 (YannC)
  • ui: doc now display correctly even with cursor at end of task. (#1322) #1322 (YannC)
  • core: use a storage prefix with executionId for triggers #1277 (Loïc Mathieu)
  • ui: save button is now purple on flow edit before change #1335 (brian.mulier)
  • ui: Remove usage of "this" in templates #1330 (YannC)
  • ui: Remove flow validation from graph generation #1327 (YannC)
  • ui: tooltip now correctly disappear on charts (#1336) #1336 (YannC)
  • ui: empty routeName on tabs don't works (Ludovic DEHON)
  • ui: error on disabled wrong type for task form (Ludovic DEHON)
  • ui: Fix clear filter in execution outputs #1346 (YannC)
  • core: remove input FILE validation #1359 (Loïc Mathieu)
  • ui: vue-flow last version break the fit on view (#1362) #1362 (YannC)
  • ui: Template CTRL-S is not working (#1364) #1364 (Loïc Mathieu)
  • ui: Fix an issue where if input is an Integer, it can't be display nor edited from the form #1360 (YannC)
  • ui: Fix 422 error in form validation #1360 (YannC)
  • ui: JS error when exporting flows or templates from the settings #1363 (Loïc Mathieu)
  • ui: Fix editor theme when theme is light and editor is dark #1350 (YannC)
  • ui: Limit size reduction of windows in editor #1371 (YannC)
  • ui: no progress bar on flow valiation (#1370) #1370 (YannC)
  • ui: metrics tab from logs view is now working (#1367) #1367 (brian-mulier-p)
  • ui: remove useless api call on editor (#1374) #1374 (YannC)
  • core: shortcircuit killing flowable tasks #1368 (Loïc Mathieu)
  • ui: tabs are loaded after config is loaded (#1385) #1385 (brian-mulier-p)
  • ui: File inputs displayed properly #1387 (brian.mulier)
  • ui: namespace and flow id have now newlines before dots when it lacks space #1377 (brian-mulier-p)
  • ui: Keep highlight in menu when select a plugin #1376 (YannC)
  • ui: use a gist instead of the original URL for the tour data file #1390 (Loïc Mathieu)
  • ui: load plugin list early (#1389) #1389 (YannC)
  • core: use the trigger id from the run context in the Schedule trigger #1391 (Loïc Mathieu)

Chores

Commits

  • ui: welcome page redirection (#1192) #1192 (YannC)
  • a005826: feat(docker) introduce couchbase plugin (brian.mulier)
  • 76386d5: Revert "chore(deps): bump org.apache.httpcomponents:httpclient" (dependabot[bot])
  • 64446f2: feat(docker) introduce NATS plugin (brian-mulier-p)
  • bd968c3: feat(doc) Documentation now generated with required props first & NotNull on interface methods displayed (#1211) (brian-mulier-p) #1211
  • 43b73a9: fix(ui) Flow validation is now triggered on every view (#1230) (brian-mulier-p) #1230
  • added a slider between editor/(doc||topo) view to adjust windows size (#1195) #1195 (YannC)
  • 419bf58: Merge branch 'release' into develop (Ludovic DEHON)
  • c389500: feat:(core): introduce flow tabs metrics (#1226) (YannC) #1226
kestra - v0.8.1

Published by github-actions[bot] over 1 year ago

Chores

kestra - v0.8.0

Published by github-actions[bot] over 1 year ago

Features

Bug Fixes

Documentation

Chores

Commits

  • 5bc65bb: feat(core) : add an If task (#1104) (Loïc Mathieu) #1104
  • 33e04f0: Revert "fix(jdbc): fix postgres migration with non default schema (#1111)" (#1113) (Loïc Mathieu) #1113
kestra - v0.7.1

Published by github-actions[bot] over 1 year ago

Chores

kestra - v0.7.0

Published by github-actions[bot] over 1 year ago

Features

Bug Fixes

Documentation

  • core: document bash plugin outputs & metrics (#898) #898 (yuri)

Code Refactoring

  • repository-elasticsearch: findDistinct abstraction (Ludovic DEHON)
  • core: allow listing trigger without multiple conditions (#992) #992 (Loïc Mathieu)

Tests

Chores

Commits

  • FlowWithSource: Copy JacksonMapper to avoid overwriting property of global JacksonMapper (Yann C)
  • core: add missing @PluginProperty on PollingTriggerInterface.getInteval() (#961) #961 (Loïc Mathieu)
  • core: refactoring flow graph (#968) #968 (YannC)
kestra - v0.6.0

Published by github-actions[bot] almost 2 years ago

Features

Bug Fixes

Chores

Commits

  • 505aa03: Merge branch 'master' into develop (Ludovic DEHON)
kestra - v0.5.3

Published by github-actions[bot] almost 2 years ago

Features

Bug Fixes

Chores

kestra - v0.5.2

Published by github-actions[bot] almost 2 years ago

Features

Bug Fixes

Chores

kestra - v0.5.1

Published by github-actions[bot] about 2 years ago

Features

Bug Fixes

  • jdbc: missing filter on daily statistics for namespace and flow (Ludovic DEHON)
  • jdbc: exectorstate is clean to early that lead to duplicate execution (Ludovic DEHON)
  • core: deleting an execution will still blocked the scheduler (Ludovic DEHON)
  • jdbc: don't send execution on queue from executor (Ludovic DEHON)
  • jdbc: jdbc runner don't purge schedule trigger when you update the flow (Ludovic DEHON)
  • ui: wrong import with @ (Ludovic DEHON)
  • ui: dailyGroupByFlowStatistics is not filtered correctly (Ludovic DEHON)
  • ui: multiple select with too many values are not removable (Ludovic DEHON)
  • core: missing template freeze the executor (Ludovic DEHON)
  • kafka-runner: remove a plugins can lead to stop of executor #745 (Ludovic DEHON)
  • jdbc: remove a plugins can lead to stop of executor #745 (Ludovic DEHON)

Chores

kestra - v0.5.0

Published by github-actions[bot] about 2 years ago

Features

Bug Fixes

  • repository-elasticsearch: use a mapping for labels compatible with opensearch (Ludovic DEHON)
  • docker: wrong kestra port again on compose (Ludovic DEHON)
  • jdbc: cast properly state (Ludovic DEHON)
  • ui: execute on source is not visible (Ludovic DEHON)
  • kafka-runner: restart & change status of an execution don't launch depending flows (Ludovic DEHON)
  • jdbc: fix template repository (Ludovic DEHON)
  • docker: docker-compose error role "root" does not exist (#629) #629 (Yuri Lima)
  • ui: translate ui revisions diff type (#631) #631 (aliczin)
  • jdbc: fix namespace & flow filter on logs (Ludovic DEHON)
  • core: PurgeExecution tasks failed if there is no storage usage (Ludovic DEHON)
  • core: plugins validation failed with micronaut update (Ludovic DEHON)
  • core: each can be blocked if there is a warning state (Ludovic DEHON)
  • ui: fix hotreload and warning (Ludovic DEHON)
  • ui: fix extra / on theme link (#661) #661 (Meng Zhuo)
  • core: restarted & retry execution don't display log in realtime (Ludovic DEHON)
  • core: autodetect unix socket path for the docker runner (Ludovic DEHON)
  • core: unique task id is validated on nested flowable (Ludovic DEHON)
  • jdbc: deduplicate worker tasks execution (Ludovic DEHON)
  • jdbc: better handle deadlock (Ludovic DEHON)
  • core: flow can be null, and we can know if it can be purged (Ludovic DEHON)
  • jdbc: restore metric on jdbc queries (Ludovic DEHON)
  • jdbc: missing quoted on deleted queues (Ludovic DEHON)
  • repository-elasticsearch: healthcheck failed on AWS OpenSearch (Ludovic DEHON)
  • ui: colors on disabled state on v-select (Ludovic DEHON)

Chores

kestra - v0.5.0-BETA

Published by github-actions[bot] over 2 years ago

Features

Bug Fixes

Chores

kestra - v0.4.7

Published by github-actions[bot] over 2 years ago

Features

  • core: add a lateMaximumDelay on Schedule to skip late executions (Ludovic DEHON)
  • task: allow to use system-site-packages for python venv creation (Ludovic DEHON)
  • core: support LocalDate & LocalDateTime on pebble date (Ludovic DEHON)

Bug Fixes

  • docker: missing venv package for python on full image (Ludovic DEHON)
  • core: end event in sse controller is never send (Ludovic DEHON)
  • ui: search with no result on flow raise an error (Ludovic DEHON)

Chores

kestra - v0.4.6

Published by github-actions[bot] over 2 years ago

Bug Fixes

Chores

kestra - v0.4.5

Published by github-actions[bot] over 2 years ago

Features

Bug Fixes

  • core: don't setup globals in configuration (Ludovic DEHON)
  • docker: missing python-is-python3 (Ludovic DEHON)
  • core: too many metrics are created with task_id that overflow metrics server (Ludovic DEHON)
  • ui: show tasks logs on topology execution can open 2 side popup (Ludovic DEHON)
  • core: too many metrics are created with trigger_id that overflow metrics server (Ludovic DEHON)
  • kafka-runner: wrong logger for KafkaStreamService (Ludovic DEHON)

Chores

Package Rankings
Top 22.7% on Repo1.maven.org
Top 11.19% on Pypi.org
Top 7.48% on Proxy.golang.org
Badges
Extracted from project README
Star the Repo