An orchestration platform for the development, production, and observation of data assets.
APACHE-2.0 License
Bot releases are visible (Hide)
Community Contributions
Bugfixes
mher/flower:0.9.5
for the Flower pod.New
dagster schedule up
or press the Reconcile button before turning on a new schedule for the first timeCommunity Contributions
Bugfixes
Experimental
New
Bugfixes
@pipeline
decorated functions with -> None typing no longer cause unexpected problems.Breaking Changes
CliApiRunLauncher
and GrpcRunLauncher
have been combined into DefaultRunLauncher
.dagster.yaml
, replace it with DefaultRunLauncher
run_launcher:
section entirely.New
Bugfixes
dagster-k8s/config
) will now bedagster-k8s
and dagster-celery-k8s
run launchers.Experimental
QueuedRunCoordinator
enables limiting the number of concurrent runs.DefaultRunCoordinator
launches jobs directly from Dagit, preserving existing behavior.New
Community contributions
Bug fixes
PipelineDefinition
's that do not meet resource requirements for its types will now fail at definition timeDeprecated
@pipeline
. This return value actually had no impact at all and was ignored, but we are making changes that will use that value in the future. By changing your code to not return anything now you will avoid any breaking changes with zero user-visible impact.Breaking Changes
DagsterKubernetesPodOperator
in dagster-airflow
.execute_plan
mutation from dagster-graphql
.ModeDefinition
, PartitionSetDefinition
, PresetDefinition
, @repository
, @pipeline
, and ScheduleDefinition
names must pass the regular expression r"^[A-Za-z0-9_]+$"
and not be python keywords or disallowed names. See DISALLOWED_NAMES
in dagster.core.definitions.utils
for exhaustive list of illegal names.dagster-slack
is now upgraded to use slackclient 2.x - this means that this resource will only support Python 3.6 and above.dagster api grpc-health-check
cli command present in Dagster 0.9.16
and later.New
K8sRunLauncher
, in place of the CeleryK8sRunLauncher
.Community Contributions
Bugfixes
Published by hellendag about 4 years ago
Breaking Changes
executeRunInProcess
was removed from dagster-graphql
New
Community Contributions
Bugfixes
Experimental
Documentation
New
Community Contributions
Bugfixes
Experimental
Published by gibsondan about 4 years ago
Breaking Changes
Community Contributions
New
Bugfixes
Documentation
Breaking Changes
compute
option from Dask DataFrame materialization configs for all output types. Setting this option to False
(default True
) would result in a future that is never computed, leading to missing materializationsCommunity Contributions
New
$DAGSTER_HOME
if it is not set or improperly setup when starting up a Dagster instancedagster debug export
- a new CLI entry added for exporting a run by id to a filedagit-debug
- a new CLI added for loading dagit with a run to debugdagit
now has a button to download the debug file for a run via the action menu on the runs pagedagster api grpc
command now defaults to the current working directory if none is specifiedBugfixes
dagit --empty-workspace
None
when an iterable is expected, causing errors in the celery execution loop.Experimental
Published by chenbobby about 4 years ago
New
Bugfixes
Experimental
Documentation
Published by yuhan about 4 years ago
New
reexecute_pipeline
now takes step_selection
, which accepts queries like *solid_a.compute++
(i.e., solid_a.compute
, all of its ancestors, its immediate descendants, and their immediate descendants). steps_to_execute
is deprecated and will be removed in 0.10.0.Community contributions
Bugfixes
Dagit
Schedules
clickable and link to View All page in the schedule section.Experimental
house update --module package.module —assets my_asset*
Documentation
Published by catherinewu about 4 years ago
Bugfixes
Published by catherinewu about 4 years ago
New
Community contributions
Bugfixes
Experimental
New
Bugfixes
Docs
Published by helloworld about 4 years ago
Breaking Changes
configured
API on a solid or composite solid, a new solid name must be provided.New
dagster.yaml
:opt_in:
local_servers: true
PipelineDefinition
or @pipeline
, e.g. @pipeline(hook_defs={hook_a})
. It will apply the hooks on every single solid instance within the pipeline.New
Bugfixes
Experimental
tags = {
'dagster-k8s/config': {
'container_config': {
'resources': {
'requests': { 'cpu': '250m', 'memory': '64Mi' },
'limits': { 'cpu': '500m', 'memory': '2560Mi' },
}
},
'pod_template_spec_metadata': {
'annotations': { "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"}
},
'pod_spec_config': {
'affinity': {
'nodeAffinity': {
'requiredDuringSchedulingIgnoredDuringExecution': {
'nodeSelectorTerms': [{
'matchExpressions': [{
'key': 'beta.kubernetes.io/os', 'operator': 'In', 'values': ['windows', 'linux'],
}]
}]
}
}
}
},
},
},
)
def my_solid(context):
context.log.info('running')```