Flyte v1.0.0

Flyte 1.0 represents the first major release of the platform. Flyte APIs, and tools can now be considered [stable]. While it's impossible to assert so, any workflow/task written against flytekit v1.0.0 should be expected to run against all v1.x.x versions.

Quick Stats:

ChangeLog

FlyteKit

• File ignore in fast register
  Many thanks to @bimtauer for adding this much requested feature! When packaging local files for fast-registration, flytekit will now respect .gitignore and .dockerignore. Please see the PR for more information.
• pyflyte run
  The pyflyte run command has been slightly updated to not need a : when selecting the workflow.

  $ pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
  

• Script mode: register and run workflows all in one command using a pre-defined base image
• Flyte remote GA: register workflows and interact with Flyte execution artifacts programmmatically
• Configuration overhaul: use the same config across flytekit and flytectl
• Fast register without having AWS/GCP or other cloud credentials on your laptop, all you need is Flyte access

Core Platform

• Improved Garbage Collector
  Garbage collection logic has been revamped to reduce load on KubeAPI and ensure terminated workflows are cleaned up in a timely fashion.
  [Action Required] Due to the change in the computation of how to clean up terminated workflows, users are advised to clean up old workflows once by running the following command per namespace:

  kubectl delete fly -l termination-status=terminated --all-namespaces --cascade='background' --wait=false --force --grace-period=0
  

• Single binary: deploy the entire Flyte back-end as a single binary. This speeds up sandbox and improves the local contributor experience. Coming soon: faster deployment for small-scale use-cases
• Improved map task subtask handling
  • Cache status reporting
  • Individual log links mapped to subtasks
  • Interruptible failure handling for spot instances
  • Secret injection
• Improved performance for fetching and rendering dynamic nodes
• Set raw output data config at create execution time
• Execution overrides at the project level for
  • Kubernetes service account
  • AssumableIAMRole
  • OutputLocationPrefix

Console

• Project dashboard page with recent executions overview along with a config and other settings summary
• Dynamic workflow rendering
• Map task UI and UX improvements: see logs, retry attempts and more at the subtask level

Flyte v1.0.0-b1 Changelog

Flyte v0.19.4 Changelog

Core Platform

• Single binary: deploy the entire Flyte back-end as a single binary. This speeds up sandbox and improves the local contributor experience. Coming soon: faster deployment for small-scale use-cases
• Improved map task subtask handling
  • Cache status reporting
  • Individual log links mapped to subtasks
  • Interruptible failure handling for spot instances
  • Secret injection
• Improved performance for fetching and rendering dynamic nodes
• Set raw output data config at create execution time
• Execution overrides at the project level for
  • Kubernetes service account
  • AssumableIAMRole
  • OutputLocationPrefix

SDK

• Script mode: register and run workflows all in one command using a pre-defined base image
• Flyte remote GA: register workflows and interact with Flyte execution artifacts programmmatically
• Configuration overhaul: use the same config across flytekit and flytectl
• Fast register without having AWS or other cloud credentials on your laptop, all you need is Flyte access

Console

• Project dashboard page with recent executions overview along with a config and other settings summary
• Dynamic workflow rendering
• Map task UI and UX improvements: see logs, retry attempts and more at the subtask level

Flyte v0.19.3 Changelog

Updates

Core Platform

• Support for Type Annotations complete.
• BigQuery plugin
• Support for hash-based caching of schema and structured dataset offloaded types.
• Configurable gRPC message size.
• Cleanup of database connection handling in Admin.
• Bug and performance fixes around

SDK

• New current_task_metadata field in StructuredDatasetDecoder interface.
• Authentication configs removed (auto-detected from Admin)
• AWS Batch Task
• Callable fetched entities

See the complete notes for flytekit v0.31.0. Keep in mind we'll be releasing v0.32.0 in a couple days. At that time the StructuredDataset type will be the default Flyte type for dataframes!

UI

• Fix node IDs for certain nested scenarios where node IDs are shared across nested workflows.

Component Versions

Admin v0.6.92 to v0.6.112
Propeller v0.16.19 to v0.16.32
Console v0.41.0 to v0.43.0

0.19.3-b3 Change Log

System

• Properly control Athena token length in plugin
• Relaxed StructuredDataset column compatibility

0.19.3-b2 Change Log

System

• Allow clusterName to be passed to flytepropeller through the Values.yaml file.

0.19.3-b1 Change Log

System

• Allow extraArgs to be passed to FlytePropeller, DataCatalog and FlyteAdmin deployments.

0.19.2 Change Log

UX

• Added support for Archive/Unarchive Workflow Executions
• Added Support for viewing node metadata on unexecuted nodes
• Minor bug fixes

FlyteKit

• Introducing the new StructuredDataset type. Turn it on by setting the env var FLYTE_SDK_USE_STRUCTURED_DATASET=TRUE. It'll be turned on by default in v1.0. Docs
• This release also removes the legacy API.

Review the full changelog here.

System

• Archive (issue).
  Expose the ability to archive/unarchive executions to remove clutter from the UI.
• Security Updates:
  • Access token validation supports multiple-audience tokens (issue)
  • flytectl supports caching access tokens for multiple endpoints at the same time (issue)
• You can use flytectl to update projects' display names and descriptions (issue)

  flytectl update project  --id data --name datadata --description "Team that manage Data Platform" --labels "team=data,managedby=flyte"
  

0.19.2-b5 Release ChangeLog

System

1. Mount admin auth creds for cluster resource sync run in standalone mode.

0.19.2-b4 Release ChangeLog

System

1. Additions to flyte-core helm chart to allow deploying cluster resource sync as a standalone process with the latest image version.

0.19.2-b3 Release ChangeLog

System

1. Update flyte-core helm chart to allow deploying cluster resource sync as a standalone process.

Flyte v0.19.2-b1

Resolving the minio port conflict.

0.19.1 Release ChangeLog

FlyteKit

No new flytekit releases as part of this release

UX

1. Rendering of a new 'terminating' phase while an aborted execution is processed
2. Misc. bug fixes

System

1. Upgrading flyteadmin migrate due to an issue in gorm's postgres driver. This release fixes the migration path to update id column database types.

Documentation

1. Updated docsearch version
2. Refreshed ToC cards

0.19.0 - Eagle Release ChangeLog

v0.19.0 codenamed Eagle, for Flyte marks the first quarterly release for Flyte in 2022. To remind, starting October 2021, the Flyte community has decided to use patch version for monthly releases and minor versions for Quarterly releases.

The release is the last in set of the 3 releases. The prior 2 can be found at

• v0.18.1 changelog
• v0.18.2 changelog

The focus on this release and the previous 2 can be divided into 2 parts

• UX (UI and Flytekit)
• System

As part of the UX, we have been focusing a lot on getting feature breadth in the UI for Flyte (flyteconsole). This is following our current UX design philosophy. We plan to complete the full design in the next 3 months and then create a fully re-designed UX (Stay tuned for more on this.)

The design for Flytekit has been wildly appreciated and we have been heavily working on improving it further and allowing users to express their various requirements much more easily. We are pretty certain on the design of flytekit and we promise that all our users will not experience a breaking change when we release a v1.0.0.

The focus on the system always is improving reliability and performance. This quarter we were able to drastically improve the size of workflows that can be executed and improvements in transition performance between nodes.

The Eagle release sets a great stage for v1.0.0 - Phoenix release slated for April 2022. We think, the Phoenix release will be a major milestone for the platform and we will be focusing on UX improvements.

Updates specific to v0.19.0

Closed Issues

UX

• New search UX for workflows
• Support for "security context" in launch workflow form
• Support for Google Analytics
• Minor fixes:
  • Removed DISABLE_AUTH env var
  • Issue with launch plan metadata not showing on execution view

FlyteKit

Please find the full changelog here.

Changes

• Support for delayed annotations by @bethebunny in https://github.com/flyteorg/flytekit/pull/760
  Users can now add from __future__ import annotations to the top of their code to stop using "" in type hints.
• Add cache_serialize parameter to tasks by @hamersaw in https://github.com/flyteorg/flytekit/pull/673
  Turn this on to avoid two cached tasks from running at the same time. Make sure you're at least at propeller v0.16.1 or later and 0.3.17 or later in datacatalog.
• Use functools.wraps basically within the task and workflow decorators, @bstadlbauer in https://github.com/flyteorg/flytekit/pull/780

System

• Auto-horizontal-scale for FlytePropeller Docs
• flytectl sandbox start will add sandbox cluster context to the local kubeconfig. No more fiddling with env vars to get your KUBECONFIG right!
• Egress Events Docs
• Full reference of Admin and Propeller configs.

0.18.2 Release ChangeLog

Closed Issues

UX

• Added advanced options to launch form
• Added support for all tasks-types (task execution view)
• Replaced execution id's with node id's on execution list-view
• Fixed bug with some properties not being repopulated on relaunch
• minor fixes

FlyteKit

See the flytekit 0.25.0 release notes for the full list of changes. Here are some of the highlights:

• Improved support for tasks that run shell scripts
• Support for more types in dataclasses:
  • enums
  • FlyteFile and FlyteSchema
• flyteremote improvements, including:
  • Access to raw inputs and outputs
  • Ability to serialize tasks containing arbitrary images
  • Improved UX for navigation of subworkflows and launchplans
  • Support for FlytePickle

System

• Various stability fixes.

• New docker image tags!

  In addition to component-specific versions released from each of the flyte repositories (e.g. flytepropeller:v0.16.5), new images will be re-tagged and pushed that match the flyte release version (e.g. the upcoming flytepropeller-release:v0.18.2). This makes it easier to make sure all your deployments are on the same version to ensure best compatibility.

• Helm changes

  • flyte-core helm chart has reached release preview and can be leveraged to install your cloud(AWS/GCP) deployments of flyte.
  • Going forward flyte-core will install flyte native scheduler, For AWS backword compatibility you need to define workflow_schedule.type to aws. (https://github.com/flyteorg/flyte/pull/1896)
  • flyte helm chart has been refactored to depend on flyte-core helm chart and install additional dependencies to continue to provide a sandboxed installation of flyte.

  Migration Notes

  As part of this move, flyte helm chart is becoming the canonical sandbox cluster. It comes with all external resources needed to fully standup a Flyte cluster. If you have previously been using this chart to deploy flyte on your cloud providers, there will be changes you need to do to migrate:

  • If you have your own myvalues.yaml, you will need to add another nesting level under flyte: for the sections that are now managed through flyte-core. For example:

    configmaps:
      ...
    flyteadmin:
      ...
    minio:
      ...
    countour:
      ...
    

    to:

    flyte:
      configmaps:
        ...
      flyteadmin:
        ...
    minio:
      ...
    countour:
      ...    
    

  • Alternatively, if you do not have any dependency on external flyte depdencies, you can keep your myvalues.yaml and switch to using flyte-core helm chart directly with no changes.

0.18.1 Release ChangeLog

Closed Issues

UX

FlyteKit

See the flytekit 0.24.0 release for the full list of changes. Here are some of the highlights:

1. We've added the following models to the top-level of the flytekit package. The classes will not move from their original locations, but importing from the top level will always be safer as we promise not to break these.

   - Annotations, AuthRole, Labels
   - WorkflowExecutionPhase
   - Literal, LiteralType, Scalar, BlobType, Blob, BlobMetadata
   

   Instead of from flytekit.models.common import Labels, Annotations, please now do

   from flytekit import Labels, Annotations
   

2. Support for python pickle. Starting in this release, flytekit is going to pickle inputs and outputs for types which it doesn't have a specific transformer for. This brings a lot of more freedom in porting over code to flyte's model, since it won't force users to write a type transformer in order to use existing code. Keep in mind that all the caveats around pickling code apply in this case.

3. We added a cookiecutter template, simplifying the Getting started docs and also unlocking the path to cookiecutter templates for specific use-cases, e.g. pytorch-enabled samples, etc.

4. Faster installation in Apple M1 Macs. We're now requiring pyarrow 6.0, which contains prebuilt wheels for the M1.

FlyteConsole

1. Added new UI for Workflow details including execution bar chart.
2. Added new bar chart user-selected filter for workflow executions.
3. Added new launch form controls ("Advanced Options")
4. Minor bug fixes

System

1. Performance improvements for executions.
   1. Smaller workflow CRDs,
   2. Better handling of partial failures in large fanout scenarios,
2. All flyte containers now run as non-root users. Docs (Thanks @frsann)
3. Stability and bug fixes

Documentation

1. Plugin Setup Docs (e.g. MPI, Tensorflow, Spark Operators, AWS Batch & Athena, Snowflake and Google BigQuery)

Sandbox

1. Fixed race condition in minio pod initialization.

Flyte v0.18.0

Platform

1. Flyte Native scheduler
2. Support for Snowflake including backend plugin and flytekit tasks
3. Expose default MaxParallelism in flyteadmin configuration
4. Support custom resource cleanup policy in backend plugins
5. Improved error message in the case of images with invalid names
6. Added default buckets in sandbox helm
7. Fixed bug in helm chart propeller config. logs url will not update correctly
8. Added cluster-admin read only roles for kubernetes dashboard
9. Updated helm chart to include minio & kubernetes Dashboard nodePort

flytekit

1. Continued changes to the FlyteRemote interface, including:
   • remote.sync now operates on the object in-place.
   • A sync_nodes argument has been added to the remote.sync call - by default it's True but set it to False if you want to only render inputs/outputs at the top level (rather than at every node within the execution).
   • The sync call on FlyteWorkflowExecution objects was removed (it was already deprecated and a noop but if you were calling it, you'll need to remove it.)
   • gRPC credentials to FlyteRemote
2. Improved typing errors when scanning user code. Added better type inference to some of the transformers.
3. Plugin tests have been moved into each individual plugin's test folder.
4. Snowflake task has been added
5. Fixes to SQLAlchemy task secrets handling
6. Modin schema transformer

Please see the flytekit release for the full list and more details.

UI

1. Additional information when a Task in a non-terminal state
2. Support for workflow versions

flytectl

1. Sandbox docker images can now be provided as a parameter
2. Bug fixes:
   • panics in calls to get execution details and launchplans.
   • datetime format generated in execFile are now valid

Flyte v0.17.3

Platform

1. Flyte Native scheduler
2. Support for Snowflake including backend plugin and flytekit tasks
3. Expose default MaxParallelism in flyteadmin configuration
4. Support custom resource cleanup policy in backend plugins
5. Improved error message in the case of images with invalid names

flytekit

1. Continued changes to the FlyteRemote interface, including:
   • remote.sync now operates on the object in-place.
   • A sync_nodes argument has been added to the remote.sync call - by default it's True but set it to False if you want to only render inputs/outputs at the top level (rather than at every node within the execution).
   • The sync call on FlyteWorkflowExecution objects was removed (it was already deprecated and a noop but if you were calling it, you'll need to remove it.)
   • gRPC credentials to FlyteRemote
2. Improved typing errors when scanning user code. Added better type inference to some of the transformers.
3. Plugin tests have been moved into each individual plugin's test folder.
4. Snowflake task has been added
5. Fixes to SQLAlchemy task secrets handling
6. Modin schema transformer

Please see the flytekit release for the full list and more details.

UI

1. Additional information when a Task in a non-terminal state
2. Support for workflow versions

flytectl

1. Sandbox docker images can now be provided as a parameter
2. Bug fixes:
   • panics in calls to get execution details and launchplans.
   • datetime format generated in execFile are now valid

Flyte v0.17.3-b4

Final test before releasing 0.17.3.

Flyte v0.17.3-b3

Change Admin event version to 2