orchest

We are very proud of this release and we call it: The Big One 🔥.

This release introduces a massive new feature to Orchest: full URL deep linking support. Easy back/forward browser navigation to open the right views in your browser. Reloading the URL and directly opening the right view.

Furthermore, environment build dependent actions (e.g. running a job, or interactive pipeline editing) are more automated: no more hard choices to make when builds complete.

In addition, we now support running an exhaustive debug dump using the simple orchest debug command. This will make troubleshooting much easier for both the users & developers.

Many tests were added to the orchest-api test suite for much improved reliability thanks mainly to @fruttasecca.

Finally, many smaller bug fixes and performance improvements were added. Have fun building in Orchest!

This release introduces the concept of Environment variables 📇 to Orchest! With its introduction we're deprecating and removing Data sources.

We've been testing Orchest and listening to feedback 📣. As a result we discovered everybody handles data sources slightly differently in their workflow. We want Orchest to be helpful, but allow for flexibility in for example the choice of the data source connector (e.g. Python offers various Postgres database connectors).

Environment variables allow for simple and effective secret passing (e.g. hostnames, passwords, usernames) to make it easy to connect and configure your favorite data sources.

In addition, this release includes many UI/UX improvements. Such as a dedicated project settings pane, and a cleaned up ✨ job pane.

This release brings some great quality of life 💎 features!

For those of you that prefer configuring your IDEs to your liking (🕶️ dark mode anyone?) - with this release we make JupyterLab configuration persistent. In addition to user config (settings you modify in the JupyterLab Settings pane) we also make client side JupyterLab extensions persistent. Support for server side extensions will be added later.

@howie6879 pointed out that it would be nice to have pipeline parameters. It would alleviate the need to repeat yourself in pipeline steps for global pipeline level parameterization. Hence, you can now configure pipeline parameters 🥳. Thanks @howie6879! Pipeline parameters can also be overridden for scheduled jobs.

In addition, some minor logging, performance, and UI improvements were made.

Hotfix missing import in orchest-webserver.

This release introduces a massive new feature: cron job scheduling ⏱.

From now on you can create jobs that run regularly specified by a cron schedule.

Gain maximal use out of your pipelines by batch scheduling them in your local 💻 or cloud ☁️ environment.

Breaking changes

• Renames "Experiments" to "Jobs"

Bug fixes and other changes

• Updates CLI to show more useful messages on failure
• Updates SDK to support Python3.6
• Makes the pipeline definitions have manageable diffs

This release mostly includes some minor bug fixes in the application web server (client & server side).

In addition we include some quality of life improvements like deletion of steps & connections in the pipeline editor through Delete and Backspace. Credits to @obulat!

Lastly, this release includes improved logging so that we can better analyze Orchest's runtime behavior which should speed up future development.

• Set environment variable default for web-updater

• Rework of permissions across containers for the userdir/ - massive increase in performance (no more daemon 🎉).
• Individual SQLite databases merged into central Postgres service - improved concurrency across services.
• Introduced migrations for schema changes to improve updating experience.
• Improved Orchest SDK robustness in error handling.
• Various small fixes and changes.

• Passing data between pipeline steps now supports named data passing

Heads up: this requires updating your old code. In short: orchest.output(data) becomes orchest.output(data, name).

• File viewer now contains a refresh button (see your Notebook fill up as it runs on the scheduler)
• File viewer now allows you to easily navigate to the parent and child pipeline steps
• Log output / build output to Xterm cleaned up to support loading style line rollback (\r)

• Increment JupyterLab version to rc13 (and drop hotfix patch in orchest extensions).

• Fix SDK bug for data passing
• Fix mounting issue for non-project-root pipelines

• Fix in the build script for tagging build images with the version

• Support for Julia language
• Improved data passing (among other things we bumped pyarrow to 2.0)
• Squashed bugs in pipeline scheduler

• Fix web updater (respect orchest.environment)

• Fetch latest tags on update

• Allow resizing of the pipeline step details pane (logs / step properties)
• Increment EG kernel version of GPU base-kernel-py to 2.3.0
• Fix issue in log streamer
• Fix user management form buttons/forms
• Fix memory server allocation

• Fix experiment snapshot directory copy
• Fix experiment list project-based filtering