dagster

@channel

The first dot release! up-to-date versions of dagster and dagit 0.2.2. (I just skipped 0.2.1 of dagster so that dagit and dagster are in sync. I won’t get into why pypi is dumb and made me do that)

There are virtually no changes to the python API. This update was for the CLI interface to make it so that you can use it without the repository.yml file and without installed modules.

You can now use dagit (or dagster) like:

dagit -f step_one.py -r define_pipeline

to load the pipeline straightaway from a function rather than having to go through repositories and yaml files.

This is the first "major" release of dagster meant for consumption. The public APIs in this release will be supported for some time.

New things in this release:

• Solids do not specify their dependencies anymore. They are more easily reusable between pipelines. Dependencies now specified at the pipeline level.
• Solids support multiple outputs and branching
• Solids can take config, in addition to inputs and outputs.
• Sources and materializations have been eliminated as formal abstractions. Solids accepting configs enabled this.
• New configuration system with full type system instead of argument dictionary. Configs can be arbitrarily nested and support composite types.
• New result api
• New execution engine. this now does a compiler-esque pass where a new logical execution graph of nodes is generated from the logical definition files and config.
• Python 2.7, 3.5 and 3.6 now supported
• RepositoryDefinition has been added. pipelines.yml is gone
• Full documentation of all public APIs
• Multi-part tutorial that introduces all concepts.
• @solid now must take info object, which has config and context members. @lambda_solid is for simple cases that do not require config and context.
• ... Much more

Just starting to use the tag feature to mark releases for the first time.

This is the last version I will be release before the major breaking change coming in 0.2.0 that will change the way that dependencies are configured and eliminate sources and materializations as formalized abstractions.