An orchestration platform for the development, production, and observation of data assets.
APACHE-2.0 License
Bot releases are hidden (Show)
Published by gibsondan about 2 years ago
multi_asset_sensor
(experimental) now has improved capabilities to monitor asset partitions via a latest_materialization_records_by_partition
method.dagstermill.get_context
to be ignoredk8s_job_op
docker_executor
list_objects
method to S3FakeSession
!AirbyteResource
, allowing users to disble forwarding of Airbyte logs to the compute log, which can be expensive for long-running syncs.Published by johannkm about 2 years ago
cron_schedule
argument to TimeWindowPartitionsDefinition
, you can now supply arbitrary cron expressions to define time window-based partition sets.AssetsDefinition.from_graph(my_graph, can_subset=True)
.RunsFilter
is now exported in the public API.dagster-user-deployments.deployments[].schedulerName
Helm value for specifying custom Kubernetes schedulers will now also apply to run and step workers launched for the given user deployment. Previously it would only apply to the grpc server.grpcio
in dagster to address an issue with the recent 0.48.1 grpcio release that was sometimes causing Dagster code servers to hang.cron_schedule
now also accepts a sequence of cron strings. If a sequence is provided, the schedule will run for the union of all execution times for the provided cron strings, e.g., ['45 23 * * 6', '30 9 * * 0]
for a schedule that runs at 11:45 PM every Saturday and 9:30 AM every Sunday. Thanks @erinov1!install_default_libraries
to databricks_pyspark_step_launcher
. It allows to run Databricks jobs without installing the default Dagster libraries .Thanks @nvinhphuc!container_config
, pod_template_spec_metadata
, pod_spec_config
, job_metadata
, and job_spec_config
) to the experimental k8s_job_op
that can be used to add additional configuration to the Kubernetes pod that is launched within the op.Published by prha about 2 years ago
ingress.labels
is now available in the Helm chart. Any provided labels are appended to the default labels on each object (helm.sh/chart
, app.kubernetes.io/version
, and app.kubernetes.io/managed-by
).GraphDefinition
manually, InputMapping and OutputMapping objects should be directly constructed.with_resources
alongside source assets.InputMapping
and OutputMapping
constructors have changed.Published by Ramshackle-Jamathon about 2 years ago
multi_asset_sensor
(experimental).config
argument of define_asset_job
now works if the job contains partitioned assets.RunRequests
from sensors, you can now optionally supply an asset_selection
argument, which accepts a list of AssetKey
s to materialize from the larger job.load_assets_from_dbt_project
and load_assets_from_dbt_manifest
now support the exlude
parameter, allowing you to more precisely which resources to load from your dbt project (thanks @flvndh!)select
parameter on your dbt_cli_resource
, this would not get passed into the corresponding invocations of certain context.resources.dbt.x()
commands. This has been fixed.Published by sryza about 2 years ago
output_required=False
. If this is set and no result is yielded from the asset, Dagster will not create an asset materialization event, the I/O manager will not be invoked, downstream assets will not be materialized, and asset sensors monitoring the asset will not trigger.JobDefinition.run_request_for_partition
can now be used inside sensors that target multiple jobs (Thanks Metin Senturk!)DAGSTER_GRPC_TIMEOUT_SECONDS
now allows for overriding the default timeout for communications between host processes like dagit and the daemon and user code servers.dagster
module has been reduced, by approximately 50% in initial measurements.AssetIn
now accepts a dagster_type
argument, for specifying runtime checks on asset input values.execute_query
method of the snowflake resource now accepts a use_pandas_result
argument, which fetches the result of the query as a Pandas dataframe. (Thanks @swotai!)load_assets_from_dbt_project
and load_assets_from_dbt_manifest
now support the exclude
parameter. (Thanks @flvndh!)select
parameter was configured on the dbt_cli_resource
, it would not be passed into invocations of context.resources.dbt.run()
(and other similar commands). This has been fixed.dagster_ge_validation_factory
and dagster 1.0 has been fixed.DagstermillExecutionContext
were not exposed. This has since been fixed.Published by clairelin135 about 2 years ago
Failure
now has an allow_retries
argument, allowing a means to manually bypass retry policies.dagstermill.get_context
and dagstermill.DagstermillExecutionContext
have been updated to reflect stable dagster-1.0 APIs. pipeline
/solid
referencing arguments / properties will be removed in the next major version bump of dagstermill
.TimeWindowPartitionsDefinition
now exposes a get_cron_schedule
method.key
argument to GCSFileManager methods to set the GCS blob key, thank you!Published by gibsondan about 2 years ago
Field.__init__
has been typed, which resolves an error that pylance would raise about default_value
dagster_type_materializer
and dagster_type_loader
expected functions to take a context argument from an internal dagster import. We’ve added DagsterTypeMaterializerContext
and DagsterTypeLoaderContext
so that functions annotated with these decorators can annotate their arguments properly.dagster_slack
documentation examples. Thanks @ssingh13-rms!Published by gibsondan about 2 years ago
Published by gibsondan about 2 years ago
dagster project
CLI contains commands for bootstrapping new Dagster projects and repositories
dagster project scaffold
creates a folder structure with a single Dagster repository and other files such as workspace.yaml. This CLI enables you to quickly start building a new Dagster project with everything set up.dagster project from-example
downloads one of the Dagster examples. This CLI helps you to quickly bootstrap your project with an officially maintained example. You can find the available examples via dagster project list-examples
.default_executor_def
argument has been added to the @repository
decorator. If specified, this will be used for any jobs (asset or op) which do not explicitly set an executor_def
.default_logger_defs
argument has been added to the @repository
decorator, which works in the same way as default_executor_def
.execute_job
function presents a Python API for kicking off runs of your jobs.RunRequests
, allowing you to kick off a job in response to the status of another job.input_manager_key
argument to AssetIn and In.In 1.0.0, a large number of previously-deprecated APIs have been fully removed. A full list of breaking changes and deprecations, alongside instructions on how to migrate older code, can be found in MIGRATION.md. At a high level:
solid
and pipeline
APIs have been removed, along with references to them in extension libraries, arguments, and the CLI (deprecated in 0.13.0)
.AssetGroup
and build_asset_job
APIs, and a host of deprecated arguments to asset-related functions, have been removed (deprecated in 0.15.0
).EventMetadata
and EventMetadataEntryData
APIs have been removed (deprecated in 0.15.0
).dagster_type_materializer
and DagsterTypeMaterializer
have been marked experimental and will likely be removed within a 1.x release. Instead, use an IOManager
.FileManager
and FileHandle
have been marked experimental and will likely be removed within a 1.x release.configured
in Dagit. Up through this point, values provided to configured
have not been sent anywhere outside the process where they were used. This change will mean that, like other places you can supply configuration, configured
is not a good place to put secrets: You should not include any values in configuration that you don't want to be stored in the Dagster database and displayed inside Dagit.
fs_io_manager
, s3_pickle_io_manager
, and gcs_pickle_io_manager
, and adls_pickle_io_manager
no longer write out a file or object when handling an output with the None
or Nothing
type.custom_path_fs_io_manager
has been removed, as its functionality is entirely subsumed by the fs_io_manager
, where a custom path can be specified via config.typing_type
of a DagsterType
is now typing.Any
instead of None
.databricks_pyspark_step_launcher
the events sent back to the host process are now compressed before sending, resulting in significantly better performance for steps which produce a large number of events.load_assets_from_dbt_project
while loading your repository, the error message in Dagit will now display additional context from the dbt logs, instead of just DagsterDbtCliFatalRuntimeError
.group_name
argument to AssetsDefinition.from_graph
when a key_prefix
argument is also present.None
return type annotation are now given the Nothing
type instead of the Any
type.AssetsDefinition.from_graph
and from_op
to fail when invoked on a configured
op.materialize
function, which is not experimental, no longer emits an experimental warning.databricks_pyspark_step_launcher
, and the external step failed to start, a RESOURCE_DOES_NOT_EXIST
error would be surfaced, without helpful context. Now, in most cases, the root error causing the step to fail will be surfaced instead.Published by gibsondan about 2 years ago
dict
s.OpDefinition
constructor now accept ins
and outs
arguments, to make direct construction easier.define_dagstermill_op
accepts ins
and outs
in order to make direct construction easier.RunRequests
returned from run_status_sensors
caused the sensor to error.define_asset_job
, an error would occur when selecting most asset subsets. This has been fixed.500
http status codes for GraphQL requests that encountered an unexpected server error.day_offset
, hour_offset
, or minute_offset
parameters were set on the asset’s partitions definition.dagster-k8s/config
didn’t actually override the command.acryl-datahub
to avoid build error.JobDefinition
objects now accept a config argument, and the preset_defs
argument has been removed.DagsterPipelineRunMetadataValue
has been renamed to DagsterRunMetadataValue
. DagsterPipelineRunMetadataValue
will be removed in 1.0.MetadataEntry
static methods are now marked as deprecated in the docs.PartitionMapping
s are now included in the API reference.Published by rexledesma about 2 years ago
DagsterRun
now has a job_name
property, which should be used instead of pipeline_name
.TimeWindowPartitionsDefinition
now has a get_partition_keys_in_range
method which returns a sequence of all the partition keys between two partition keys.OpExecutionContext
now has asset_partitions_def_for_output
and asset_partitions_def_for_input
methods.AssetsDefinition
objects with the same key are provided to the same repository.build_output_context
now accepts a partition_key
argument that can be used when testing the handle_output
method of an IO manager.asset_key
set.@asset
and @multi_asset
decorators no longer have a top-level “assets” entry in their config schema. This entry was unused.load_assets_from_dbt_manifest
with a select
parameter that included sources would result in an error. This has been fixed.@repository
decorated function expects parameters. Thanks @roeij!examples/hacker_news/
, has been removed, because it hadn’t received updates in a long time and had drifted from best practices. The asset version is still there and has an updated README. Check it out here
https://github.com/dagster-io/dagster/compare/0.15.6...0.15.7
Correct check for pickle s3 io manager (#8834)
by @ripplekhera[dagit] Always show “Materialize” instead of “Rematerialize” based on status (#8711)
by @bengotow[dagster-dbt] make group configurable for load_assets_from_dbt (#8863)
by @OwenKephartasset_partitions_def on InputContext and OutputContext (#8858)
by @sryza[dagster-dbt] refactor the dbt asset integration (#8793)
by @OwenKephartPartitionMappings when non-partitioned assets depend on partitioned assets (#8866)
by @sryza[dagster-dbt] seeds and snapshots are assets when using dbt build (#8794)
by @OwenKephartDocument valid names for asset keys (#8765)
by @jamiedemaria[docs] Dagster + dbt guide (#8714)
by @OwenKephart0.15.6 changelog (#8876)
by @yuhanprovide description for MAP ConfigType (#8824)
by @JiafiRetrieve minimal set of asset records for assetsLatestInfo (#8835)
by @bengotowerror when duplicate asset keys on a repository (#8874)
by @sryza[docs] - Add Airflow Operator to Op Docs (#8875)
by @clairelin135dagster-datahub Rest and Kafka Emitter integration (#8764)
by @JiafiAutomation: versioned docs for 0.15.6
by @elementl-devtools[1.0] move solid to dagster.legacy (#8843)
by @dpeng817Extract ECS task overrides (#8842)
by @jmsanders[graphql] tolerate empty runConfigData (#8886)
by @alangenfeld[dagit] Fix edge case where “ “ launchpad config is not coerced to an empty object (#8895)
by @bengotowFix ScheduleRootQuery typo (#8903)
by @johannkmunloadable shit (#8887)
by @prhaChange base image for OSS release builds (#8902)
by @gibsondanchange deprecation warnings to 1.0 (#8892)
by @dpeng817update README for hacker news assets example (#8904)
by @sryza[hacker news] add missing key prefix to activity analytics python assets (#8817)
by @sryza[buildkite] Specify internal branch used for compatibility pipeline (#8881)
by @smackesey[dagit] Remove “mode” label and description in explorer sidebar (#8907)
by @bengotowremove the non-asset version of the hacker news example (#8883)
by @sryzaError when @repository-decorated function has arguments (#8913)
by @roeij[docs] - Correct snippets for dbt (#8923)
by @erinkcochran87[bug] fix input loading regression (#8885)
by @OwenKephart[docs] - graph backed assets doc fix (#8927)
by @jamiedemariasilence system-originated experimental warning for PartitionMapping (#8931)
by @sryzaAdd partition key to build_output_context, add documentation for partition_key on build_op_context (#8774)
by @dpeng817fix dimensions of screenshot on connecting ops tutorial page (#8908)
by @sryzattv: remove undocumented/legacy example - user in loop (#8934)
by @yuhan[docs] - Clean up graph-backed asset example, put under test (#8893)
by @dpeng817[docs] - Fix garbled sentence in ops.mdx (#8935)
by @schrocknenable getting asset partitions def from op context (#8924)
by @sryzaIncrease test_docker_monitoring timeout (#8906)
by @johannkmPartitionsDefinition.get_partition_keys_in_range (#8933)
by @sryzaMove pipeline to dagster.legacy (#8888)
by @dpeng817[dagit] Rename search open event (#8954)
by @hellendag[dagit] Adjust shift-selection behavior in asset graphs (#8950)
by @bengotow[dagit] Clean up code around the graph sidebar (#8914)
by @bengotowadd a job_name property to PipelineRun (#8928)
by @sryzaremove partition entries from asset op config schema (#8951)
by @sryzaavoid pipelines in run status sensor doc snippets (#8929)
by @sryza[bug] Fix issue where 'invalid' asset metadata resulted in an error (#8947)
by @OwenKephart[dagster-dbt] fix tagged source asset loading (#8943)
by @OwenKephartfix black in run status sensor docs example (#8974)
by @sryza[known state] fix build_for_reexecution bug (#8975)
by @alangenfeldDocument deprecation of MetadataEntry static constructors (#8984)
by @smackeseychangelog (#8986)
by @jamiedemaria0.15.7
by @elementl-devtoolsPublished by Ramshackle-Jamathon over 2 years ago
default_logger_defs
argument has been added to the @repository
decorator. Check out the docs on specifying default loggers to learn more.AssetsDefinition.from_graph
and AssetsDefinition.from_op
now both accept a partition_mappings
argument.AssetsDefinition.from_graph
and AssetsDefinition.from_op
now both accept a metadata_by_output_name
argument.define_asset_job
now accepts an executor_def
argument.gql
in dagster-graphql
.@multi_asset
decorator, either by supplying a group_name
argument (which will apply to all of the output assets), or by setting the group_name
argument on individual AssetOut
s.InputContext
and OutputContext
now each have an asset_partitions_def
property, which returns the PartitionsDefinition
of the asset that’s being loaded or stored.build_schedule_from_partitioned_job
now raises a more informative error when provided a non-partitioned asset jobPartitionMapping
, IdentityPartitionMapping
, AllPartitionMapping
, and LastPartitionMapping
are exposed at the top-level dagster
package. They're currently marked experimental.PartitionMapping
.PartitionMappings
on AssetIn
.frame-ancestors
restriction.load_assets_from_dbt_project
and load_assets_from_dbt_manifest
now support a node_info_to_group_name_fn
parameter, allowing you to customize which group Dagster will assign each dbt asset to.runtime_metadata_fn
when loading dbt assets, this metadata is added to the default metadata that dagster-dbt generates, rather than replacing it entirely.use_build_command=True
, seeds and snapshots will now be represented as Dagster assets. Previously, only models would be loaded as assets.DockerRunLauncher
would sometimes use Dagit’s Python environment as the entrypoint to launch the run, even if that environment did not exist in the container.Output
object is used to type annotate the return of an op, an Output object must be returned or an error will result.PickledObjectS3IOManager
in run logs and Asset view. Thanks @danielgafnihttps://github.com/dagster-io/dagster/compare/0.15.5...0.15.6
Fold asset_defs submodule into main dagster structure (#8446)
by @smackeseyOp that runs a kubernetes job (#8161)
by @gibsondanadd validation for graph backed asset graphs (#8754)
by @OwenKephartchore: mark snapshots as generated (#8758)
by @rexledesma[op] fix empty list output bug (#8763)
by @alangenfeld[dagster-dbt] Allow SDAs generated with load_assets_from_dbt* to be partitioned (#8725)
by @OwenKephartdocs: update config schema descriptions for default executors (#8757)
by @rexledesmaRestore sensor daemon yielding when evaluating sensors synchronously (#8756)
by @prhabypass bucketed queries for mysql versions that do not support it (#8753)
by @prha[dagster-dbt] Fix error that occurs when generating events for tests that depend on sources (#8775)
by @OwenKephartSpecifying executor docs examples (#8530)
by @dpeng817prevent multiple sensor evaluations from multithreaded race conditions (#8720)
by @prhaFix config case for default executor (#8777)
by @dpeng817Ensure graph inputs/outputs are included in all_dagster_types (#8736)
by @smackeseyimprove error for build_schedule_from_partitioned_job with non-partitioned asset job (#8776)
by @sryza[dagit] Bring back the global asset graph as an “all asset groups” view (#8709)
by @bengotowfix source asset regression (#8784)
by @smackeseyfix issue with repos and partitioned scheduled asset jobs (#8779)
by @sryza0.15.5 Changelog (#8781)
by @prhachangelog (#8788)
by @prhaOption to hide daemon heartbeat timestamp in Dagit (#8785)
by @johannkmFix bug with how resources are applied in materialize (#8790)
by @dpeng817Add default_logger_defs arg to repository (#8512)
by @dpeng817update hackernews tests to use asset invocation and materialize_to_memory (#8592)
by @dpeng817Add MetaDataEntry.path to PickledObjectS3IOManager (#8732)
by @danielgafniAutomation: versioned docs for 0.15.5
by @elementl-devtoolsadd define_asset_job to __all__ (#8791)
by @sryzaeliminate incorrect SDA warnings (#8769)
by @sryzaupdate hacker news assets example for post-0.15.0 APIs (#7904)
by @sryzapartition mappings on graph-backed assets (#8768)
by @sryzaSnowflake IO Manager handles pandas timestamps (#8760)
by @jamiedemariaAdd Python 3.10 testing to BK and other image adjustments (#7700)
by @smackeseyOption to skip daemon heartbeats with no errors (#8670)
by @johannkmAssorted type annotations (#8356)
by @smackeseyBump urllib3 (#8808)
by @dependabot[bot]Bump rsa (#8807)
by @dependabot[bot]Bump pyyaml (#8806)
by @dependabot[bot]Change default local grpc behavior to send back "dagster" as the entry point to use, rather than dagit's python environment (#8571)
by @gibsondanmove Metadata and Tags concept page under jobs section (#8813)
by @sryzaupdates to multi-assets docs page (#8814)
by @sryzaUpdate asset ID to contain repository location and name (#8762)
by @clairelin135unpin gql (#8822)
by @prharemove unused old partitions ui (#8796)
by @prhafix broken asset deps in hacker news example (#8809)
by @sryza[dagit] Bump TS version (#8704)
by @hellendagfix contributing docs (#8789)
by @prhaBetter support for nested causes in dagit (#8823)
by @gibsondan[easy] Fix docs link for RetryPolicy (#8830)
by @gibsondanRemove the "cronjobs" permission from the helm chart (#8827)
by @gibsondanfix gql resolver for graph-backed assets resources (#8825)
by @smackeseyrefactor run storage to enable backfill status queries (#8695)
by @prharefactor backfill / partition pages to stop run fetching (#8696)
by @prhaadd multi_or_in_process_executor to __all__ (#8831)
by @smackeseyavoid apollo cache collision for partition/backfill status (#8841)
by @prhadistinguish between [] and None for asset queries (#8838)
by @prhaoverride batch loader to use asset records instead of legacy event materialization method (#8839)
by @prha[dagit] Add analytics.group (#8832)
by @hellendag[dagster-io/ui] Fix disabled Button (#8844)
by @hellendagFix issue where partitioned assets with resources fail materialization in dagit (#8837)
by @smackesey[dagit] Tweak analytics function sigs (#8851)
by @hellendag[asset-defs] allow multi assets to have group names (#8847)
by @OwenKephartRefactor op return checking code (#8755)
by @dpeng817[dagit] Remove frame-ancestors restriction (#8850)
by @hellendagadjust error messages (#8853)
by @dpeng817[dagit] Update CRA, simplify CSP (#8854)
by @hellendagkey_prefix for AssetsDefinition from_graph and from_op (#8859)
by @jamiedemariaeasy: fix dagster pandas link (#8862)
by @yuhanexecutor_definition on define_asset_job (#8856)
by @sryzainclude airflow_operator_to_op in apidoc (#8860)
by @sryzaadd metdata_by_output_name (#8861)
by @OwenKephart[dagster-dbt] make group configurable for load_assets_from_dbt (#8863)
by @OwenKephartasset_partitions_def on InputContext and OutputContext (#8858)
by @sryza[dagster-dbt] refactor the dbt asset integration (#8793)
by @OwenKephart[dagster-dbt] seeds and snapshots are assets when using dbt build (#8794)
by @OwenKephartPartitionMappings when non-partitioned assets depend on partitioned assets (#8866)
by @sryza[docs] Dagster + dbt guide (#8714)
by @OwenKephart0.15.6
by @elementl-devtoolsPublished by jmsanders over 2 years ago
materialize
that would cause required resources to not be applied correctly.build_schedule_from_partitioned_job
and define_asset_job
were used together.FROM_FAILURE
strategySTARTING
and SUCCESS
status.default_executor_def
argument on repository where asset jobs that defined executor config would result in errors.databricks_pyspark_step_launcher
now allows you to configure the following (Thanks @Phazure!):
aws_attributes
of the cluster that will be spun up for the step.[dagster-k8s] Added k8s_job_op
to launch a Kubernetes Job with an arbitrary image and CLI command. This is in contrast with the k8s_job_executor
, which runs each Dagster op in a Dagster job in its own k8s job. This op may be useful when you need to orchestrate a command that isn't a Dagster op (or isn't written in Python). Usage:
from dagster_k8s import k8s_job_op
my_k8s_op = k8s_job_op.configured({
"image": "busybox",
"command": ["/bin/sh", "-c"],
"args": ["echo HELLO"],
},
name="my_k8s_op",
)
[dagster-dbt] The dbt asset-loading functions now support partitions_def
and partition_key_to_vars_fn
parameters, adding preliminary support for partitioned dbt assets. To learn more, check out the Github issue!
https://github.com/dagster-io/dagster/compare/0.15.4...0.15.5
Remove unused check_heartbeats arg (#8673)
by @johannkmdocs: use dagster brand colors for README (#8660)
by @rexledesmafix: use absolute url for README images (#8676)
by @rexledesmaExpose TimeWindow in dagster package (#8643)
by @nvinhphuchold shift to force asset config modal (#8668)
by @smackeseyadd helm values for configuring instance sensor config (#8657)
by @prha[dagit] If you have no jobs, prefer routing to asset group over asset catalog (#8613)
by @bengotow[dagit] Add control for graph depth on the Asset Lineage page, default to 5 (#8531)
by @bengotow[assets] Fix issue with graph backed assets + partitions (#8682)
by @OwenKephart[dagster-dbt] rework dagster dbt logging, cleanup (#8681)
by @OwenKephartYield run requests from run status sensors (#8635)
by @jamiedemariafix docs build (#8688)
by @OwenKephartapidoc for AssetSelection and AssetsDefinition (#8618)
by @sryzaMaterialize has resources arg, materialize_to_memory sets mem_io_manager for all io managers (#8659)
by @dpeng817Load asset launchpad for assets with configurable resources (#8677)
by @smackeseydocs: update external links for github issues (#8661)
by @rexledesmadocs: convert github issue templates to forms (#8663)
by @rexledesmaswitch unloadable states to dedupe by selector id (#8656)
by @prha0.15.3 changelog (#8690)
by @dpeng817Automation: versioned docs for 0.15.3
by @elementl-devtoolsdocs: remove extra cruft from issue form (#8701)
by @rexledesmadocs: improve left nav for items with children DREL-359 (#8693)
by @yuhandocs: fix prev/next pagination (#8697)
by @yuhandocs: docs test should capture Next Image broken links and fix broken links (#8702)
by @yuhan[dagit] Display last asset key component on DAG rather than truncated full path (#8692)
by @bengotow[dagit] Collapsed DAG rendering of multiple edges between the same ops (#8479)
by @bengotow[dagit] Add Analytics context (#8674)
by @hellendagorder the backfill partitions before creating (#8703)
by @prhaskip threadpool sensor daemon tests (#8717)
by @prhaskip threaded sensor tests in py36 (#8726)
by @prhaAllow run status sensors to support unresolved asset jobs (#8689)
by @smackesey[dagster_databricks] - support configuration of job / cluster permissions (#8683)
by @Phazure[dagster_databricks] support aws_attributes (#8684)
by @Phazure[Job Log perf] Use rAF to call throttleSetNodes (#8735)
by @salazarm[dagster_databricks] support arbitrary env variables (#8685)
by @Phazureminor changes to make dev_install (#8745)
by @smackeseyRemove deprecated daemon health-check cli (#8751)
by @johannkmAdd reexecution strategy to auto run retries (#8718)
by @johannkmFix for blocks_fn option (#8448)
by @clairelin135Automation: versioned docs for 0.15.4
by @elementl-devtoolsImprove race condition in run monitor (#8729)
by @brad-alexanderAdd get_daemon_statuses instance method (#8752)
by @johannkm0.15.4 changelog here (#8766)
by @prhaadd retention helm values (#8724)
by @prhaOp that runs a kubernetes job (#8161)
by @gibsondan[op] fix empty list output bug (#8763)
by @alangenfeld[dagster-dbt] Allow SDAs generated with load_assets_from_dbt* to be partitioned (#8725)
by @OwenKephartadd validation for graph backed asset graphs (#8754)
by @OwenKephartdocs: update config schema descriptions for default executors (#8757)
by @rexledesma[dagster-dbt] Fix error that occurs when generating events for tests that depend on sources (#8775)
by @OwenKephartRestore sensor daemon yielding when evaluating sensors synchronously (#8756)
by @prhabypass bucketed queries for mysql versions that do not support it (#8753)
by @prhaprevent multiple sensor evaluations from multithreaded race conditions (#8720)
by @prhaFold asset_defs submodule into main dagster structure (#8446)
by @smackeseyFix config case for default executor (#8777)
by @dpeng817Ensure graph inputs/outputs are included in all_dagster_types (#8736)
by @smackesey[dagit] Bring back the global asset graph as an “all asset groups” view (#8709)
by @bengotowfix source asset regression (#8784)
by @smackeseyimprove error for build_schedule_from_partitioned_job with non-partitioned asset job (#8776)
by @sryzafix issue with repos and partitioned scheduled asset jobs (#8779)
by @sryza0.15.5 Changelog (#8781)
by @prhaFix bug with how resources are applied in materialize (#8790)
by @dpeng8170.15.5
by @elementl-devtoolsPublished by prha over 2 years ago
https://github.com/dagster-io/dagster/compare/0.15.3...0.15.4
Automation: versioned docs for 0.15.3
by @elementl-devtoolsRevert "bump timeout for large sensor test (#8671)"
by @johannkmRevert "Add a threadpool to the sensor daemon (#8642)"
by @johannkm0.15.4
by @elementl-devtoolsPublished by rexledesma over 2 years ago
input_manager_key
argument to AssetIn
and In
dagster.yaml
that enables the automatic removal of sensor/schedule ticks after a certain number of days.dagster.yaml
that enables sensor evaluations to happen in a thread pool to increase throughput.materialize_to_memory
and materialize now both have the partition_key argument.Output
and DynamicOutput
objects now work with deep equality checks:Output(value=5, name="foo") == Output(value=5, name="foo") # evaluates to True
RunRequest
can now be returned from run status sensorsresource_defs
argument to AssetsDefinition.from_graph
. Allows for specifying resources required by constituent ops directly on the asset.dbt-core
installed. dbt-core
is now also added as a dependency to the library.0.15.1
that caused Dagit to interpret datetime strings as datetime objects and octal strings as integers.prior_attempts_count
field when using the databricks_pyspark_step_launcher
.databricks_pyspark_step_launcher
is now emitted at the DEBUG
level instead of INFO
.fs_io_manager
(or other similar io managers).--log-format
json flag. In some cases, this may cause dbt to report incorrect or misleading error messages. As a workaround, it is now possible to disable this behavior by setting the json_log_format
configuration option on the dbt_cli_resource
to False
.materialize_to_memory
erroneously allowed non-in-memory io managers to be used. Now, providing io managers to materialize_to_memory
will result in an error, and mem_io_manager
will be provided to all io manager keys.https://github.com/dagster-io/dagster/compare/0.15.2...0.15.3
Published by Ramshackle-Jamathon over 2 years ago
https://github.com/dagster-io/dagster/compare/0.15.1...0.15.2
Automation: versioned docs for 0.15.1
by @elementl-devtoolsfix asset dep resolution bug when two assets in group have same name (#8609)
by @sryza0.15.2
by @elementl-devtoolsPublished by benpankow over 2 years ago
AssetsDefinition.from_graph()
now accepts a group_name
parameter. All assets created by from_graph are assigned to this group.AssetsDefinition.from_op
. Dagster will infer asset inputs and outputs from the ins/outs defined on the @op
in the same way as @graphs
.default_executor_def
argument. The default executor definition will be used for all op/asset jobs that don’t explicitly define their own executor.JobDefinition.run_request_for_partition
now accepts a tags
argument (Thanks @jburnich!)@multi_asset
now accepts a resource_defs argument. The provided resources can be either used on the context, or satisfy the io manager requirements of the outs on the asset.define_asset_job
and to_job
now can now accept a partitions_def
argument and a config
argument at the same time, as long as the value for the config
argument is a hardcoded config dictionary (not a PartitionedConfig
or ConfigMapping
)k8s_job_executor
and docker_executor
would sometimes return the same event lines twice in the command-line output for the step.@op
decorator (Thanks Milos Tomic!)UnresolvedAssetJobDefinition
now supports the run_request_for_partition
method.AssetOut
and AssetIn
https://github.com/dagster-io/dagster/compare/0.15.0...0.15.1
[docs] - Asset grouping [CON-295] (#8375)
by @erinkcochran87Run retries docs (#8367)
by @johannkm[dagit] Update the linking between asset groups + jobs (#8377)
by @bengotow[docs] - New Jobs & Graphs structure [CON-289] (#8035)
by @erinkcochran87remove old asset lineage from docs (#8382)
by @sryzafix docs lint (#8387)
by @gibsondanPatch op definition resolution for assets in GQL (#8384)
by @smackeseyRemove broken/unused examples from dbt guide (#8383)
by @erinkcochran87chore(graphql): remove description linter (#8376)
by @rexledesmaMake asset invocation error if resources conflict (#8390)
by @dpeng817materialize_in_process -> materialize_to_memory (#8391)
by @dpeng817apidoc for AssetOut and AssetIn (#8388)
by @sryzaFix problems from "remove old asset lineage from docs (#8382)" (#8389)
by @sryzaChangelog 0.15.0 (#8395)
by @clairelin135[docs] - Update config docs with assets (#8370)
by @smackesey[dagit] Fix launch parameters when an asset is partitioned and a graph (#8393)
by @bengotowAutomation: versioned docs for 0.15.0
by @elementl-devtoolsMigration guide 0.15.0 (#8392)
by @smackeseyfix type error (#8379)
by @salazarmadd tags in run_request_for_partition (#8344)
by @jburnichAdd example asset groups (#8371)
by @clairelin135[docs] - Update types to contain assets (#8341)
by @clairelin135[dagit] Change compute log op when selecting in Gantt (#8397)
by @hellendagdocs: default to master version when vercel env is not prod (#8396)
by @yuhandocs: when gh discussion dont use Link and open in a new tab (#8402)
by @yuhanAdd redirect (#8412)
by @erinkcochran87[ui] Add optional joined prop for buttons (#7491)
by @benpankowFix deserialization of RunsFilter (#8415)
by @johannkmdocs: fix slack hooks example (#8403)
by @rexledesmaHandle errors with an empty stack trace in dagit error viewer (#8427)
by @gibsondan[dagster-io/ui] Move ConfigEditor to ui (#8413)
by @hellendag[dagster-io/ui] Release v1.0.5 (#8433)
by @hellendag[ui] Ensure JoinedButtons works with nested Buttons (#8419)
by @benpankowbump target deprecation versions (#8386)
by @sryza[docs] - Add SDA upgrade guide to sidenav (#8437)
by @erinkcochran87swap string run config parser from JSON to YAML (#8436)
by @gibsondan[dagit] Send YAML insead of parsed JSON (#8443)
by @hellendagfix some grammar (#8435)
by @prhaadd types-proto pin (#8447)
by @gibsondan[docs] - Clarify the relationship between ops and assets on op concept page (#8409)
by @sryza[dagit] Repair loading state on Instance Overview (#8450)
by @hellendagfix typo (#8451)
by @clairelin135Update runtime type checks in resource definition invocation (#8432)
by @gibsondanremove dagster io repo mention (#8459)
by @yuhanUpdate docs/content/concepts/configuration/configured.mdx (#8464)
by @wenhoujxfix(dagster/decorators): op_decorator documentation update (#8179)
by @mict0[dagit] “Canvas” dots on the DAGs 🎨 (#8455)
by @bengotow[dbt-python-assets] demo repository (#8460)
by @OwenKephart[dagit] Fix UI issues around graph backed assets (#8458)
by @bengotow[dagit] Repair empty config case in Launchpad (#8483)
by @hellendag[dagster-io/ui] MaybeTooltip (#8445)
by @hellendagadd backfills for jobs (#8496)
by @prhafix lint error in types concepts page (#8495)
by @sryzapost-0.15.0 docs fixups (#8470)
by @sryza[hopefully easy] annotations of resource_defs become mapping (#8501)
by @dpeng817remove unsupported python version for python dbt example (#8503)
by @sryzaAccept a group_name in AssetsDefinition.from_graph() (#8406)
by @shalabhcClarify points in direct operation section (#8497)
by @erinkcochran87[dagit] Remove code for a disabled layout feature (#8478)
by @bengotow[dagit] Loosen the state machine expectations around run steps (#8480)
by @bengotowAdd constant for default io manager key (#8506)
by @dpeng817annotations of logger_defs become mapping (#8502)
by @dpeng817[dagit] Add timezone and 24h format to humanCronString (#8520)
by @hellendagFix module structure on deploying docs examples (#8511)
by @dpeng817Make default_executor_def on repo work on jobs (#8490)
by @dpeng817refactor stephandler API to support the same yield+write pattern as other executors (#8357)
by @gibsondangraphql test case for a memoization job (#8442)
by @gibsondan[dagit] Collapse assets in Run header if many of them (#8533)
by @hellendagFallback to a placeholder event on a new event type that we don't recognize instead of raising an exception (#8542)
by @gibsondansolid to op in graph error (#8539)
by @sryzanew typed events for step process start/end and resource init/teardown (#8368)
by @gibsondanUse status column for reading runs from sql storage (#8528)
by @johannkm[ui] Add ability to supply TokenizingField with raw tokens (#8444)
by @benpankowresources_init mypy (#8536)
by @dpeng817fix executor test (#8566)
by @gibsondanAssetsDefinition.from_op (#8508)
by @clairelin135Add resource_defs arg to multi_assets (#8529)
by @dpeng817Add run_request_for_partition to UnresolvedAssetJobDefinition (#8441)
by @smackesey[docs] - Update screenshots in tutorials (#8494)
by @erinkcochran87group-based asset dependency resolution (#8517)
by @sryzaenable setting partitions_def and config at the same time (#8581)
by @sryzachangelog for 0.15.1 (#8582)
by @sryza0.15.1
by @elementl-devtoolsPublished by clairelin135 over 2 years ago
from dagster import asset
@asset(config_schema={"iterations": int})
def my_asset(context):
for i in range(context.op_config["iterations"]):
...
AssetsDefinition.from_graph
:
@graph(out={"asset_one": GraphOut(), "asset_two": GraphOut()})
def my_graph(input_asset):
...
graph_asset = AssetsDefinition.from_graph(my_graph)
execute_in_process
and GraphDefinition.to_job
now both accept an input_values
argument, so you can pass arbitrary Python objects to the root inputs of your graphs and jobs.from dagster import Output, op
@op
def my_op() -> Output[int]:
return Output(5, metadata={"a": "b"})
This release marks the official transition of software-defined assets from experimental to stable. We made some final changes to incorporate feedback and make the APIs as consistent as possible:
fs_asset_io_manager
in favor of merging its functionality with fs_io_manager
. fs_io_manager
is now the default IO manager for asset jobs, and will store asset outputs in a directory named with the asset key. Similarly, removed adls2_pickle_asset_io_manager
, gcs_pickle_asset_io_manager
, and s3_pickle_asset_io_manager
. Instead, adls2_pickle_io_manager
, gcs_pickle_io_manager
, and s3_pickle_io_manager
now support software-defined assets.@asset
decorator and AssetIn has been deprecated. Users should use key_prefix instead.define_assets_job
(replacing AssetGroup.build_job
), and arbitrary sets of assets can be materialized using the standalone function materialize (replacing AssetGroup.materialize
).outs
property of the previously-experimental @multi_asset
decorator now prefers a dictionary whose values are AssetOut
objects instead of a dictionary whose values are Out
objects. The latter still works, but is deprecated.OpExecutionContext
called output_asset_partition_key
is now deprecated in favor of asset_partition_key_for_output
get_event_records
method on DagsterInstance now requires a non-None argument event_records_filter
. Passing a None
value for the event_records_filter
argument will now raise an exception where previously it generated a deprecation warning.events_for_asset_key
and get_asset_events
, which have been deprecated since 0.12.0.dbt_assets = load_assets_from_dbt_project(..., node_info_to_asset_key=lambda node_info: AssetKey(node_info["name"])
.prior_attempts_count
parameter is now removed from step-launching APIs. This parameter was not being used, as the information it held was stored elsewhere in all cases. It can safely be removed from invocations without changing behavior.FileCache
class has been removed.A new define_asset_job
function allows you to define a selection of assets that should be executed together. The selection can be a simple string, or an AssetSelection object. This selection will be resolved into a set of assets once placed on the repository.
from dagster import repository, define_asset_job, AssetSelection
string_selection_job = define_asset_job(
name="foo_job", selection="*foo"
)
object_selection_job = define_asset_job(
name="bar_job", selection=AssetSelection.groups("some_group")
)
@repository
def my_repo():
return [
*my_list_of_assets,
string_selection_job,
object_selection_job,
]
[dagster-dbt] Assets loaded with load_assets_from_dbt_project
and load_assets_from_dbt_manifest
will now be sorted into groups based on the subdirectory of the project that each model resides in.
@asset
and @multi_asset
are no longer considered experimental.
Adds new utility methods load_assets_from_modules
, assets_from_current_module
, assets_from_package_module
, and assets_from_package_name
to fetch and return a list of assets from within the specified python modules.
Resources and io managers can now be provided directly on assets and source assets.
from dagster import asset, SourceAsset, resource, io_manager
@resource
def foo_resource():
pass
@asset(resource_defs={"foo": foo_resource})
def the_resource(context):
foo = context.resources.foo
@io_manager
def the_manager():
...
@asset(io_manager_def=the_manager)
def the_asset():
...
Note that assets provided to a job must not have conflicting resource for the same key. For a given job, all resource definitions must match by reference equality for a given key.
A materialize_to_memory
method which will load the materializations of a provided list of assets into memory:
from dagster import asset, materialize_to_memory
@asset
def the_asset():
return 5
result = materialize_to_memory([the_asset])
output = result.output_for_node("the_asset")
A with_resources
method, which allows resources to be added to multiple assets / source assets at once:
from dagster import asset, with_resources, resource
@asset(required_resource_keys={"foo"})
def requires_foo(context):
...
@asset(required_resource_keys={"foo"})
def also_requires_foo(context):
...
@resource
def foo_resource():
...
requires_foo, also_requires_foo = with_resources(
[requires_foo, also_requires_foo],
{"foo": foo_resource},
)
You can now include asset definitions directly on repositories. A default_executor_def
property has been added to the repository, which will be used on any materializations of assets provided directly to the repository.
from dagster import asset, repository, multiprocess_executor
@asset
def my_asset():
...
@repository(default_executor_def=multiprocess_executor)
def repo():
return [my_asset]
The run_storage
, event_log_storage
, and schedule_storage
configuration sections of the dagster.yaml
can now be replaced by a unified storage
configuration section. This should avoid duplicate configuration blocks with your dagster.yaml
. For example, instead of:
# dagster.yaml
run_storage:
module: dagster_postgres.run_storage
class: PostgresRunStorage
config:
postgres_url: { PG_DB_CONN_STRING }
event_log_storage:
module: dagster_postgres.event_log
class: PostgresEventLogStorage
config:
postgres_url: { PG_DB_CONN_STRING }
schedule_storage:
module: dagster_postgres.schedule_storage
class: PostgresScheduleStorage
config:
postgres_url: { PG_DB_CONN_STRING }
You can now write:
storage:
postgres:
postgres_url: { PG_DB_CONN_STRING }
All assets where a group_name
is not provided are now part of a group called default
.
The group_name parameter value for @asset
is now restricted to only allow letters, numbers and underscore.
You can now set policies to automatically retry Job runs. This is analogous to op-level retries, except at the job level. By default the retries pick up from failure, meaning only failed ops and their dependents are executed.
[dagit] The new repository-grouped left navigation is fully launched, and is no longer behind a feature flag.
[dagit] The left navigation can now be collapsed even when the viewport window is wide. Previously, the navigation was collapsible only for small viewports, but kept in a fixed, visible state for wide viewports. This visible/collapsed state for wide viewports is now tracked in localStorage, so your preference will persist across sessions.
[dagit] Queued runs can now be terminated from the Run page.
[dagit] The log filter on a Run page now shows counts for each filter type, and the filters have higher contrast and a switch to indicate when they are on or off.
[dagit] The partitions and backfill pages have been redesigned to focus on easily viewing the last run state by partition. These redesigned pages were previously gated behind a feature flag — they are now loaded by default.
[dagster-k8s] Overriding labels in the K8sRunLauncher will now apply to both the Kubernetes job and the Kubernetes pod created for each run, instead of just the Kubernetes pod.
docker_executor
to run each step of a job in a different Docker container.https://github.com/dagster-io/dagster/compare/0.14.0...0.15.0
add fallback error state for snapshot types (#6671)
by @prhaGetting Started and Basic Tutorial fixes (#6673)
by @jmsandersrun timeline blank state (#6674)
by @prha[multiproc] default to spawn (#6676)
by @alangenfeldAsset Observations Doc (#6630)
by @clairelin135Restore missing repo.py file (#6677)
by @gibsondan[dagster-aws] Fix bug when logging records from pyspark_step_launcher (#6680)
by @OwenKephartAutomation: versioned docs for 0.14.0
by @elementl-devtoolsChangelog (#6667)
by @clairelin135Fix formatting on schedules feature in changelog (#6686)
by @gibsondanPin grpcio-health-checking (#6685)
by @johannkmfix changelog page in docs (#6694)
by @yuhanredirect ecs page (#6695)
by @yuhanRemove crons from our example dockerfiles (#6645)
by @gibsondanpin markupsafe (#6713)
by @alangenfeldadd cursor label to tick timeline (#6712)
by @prhawhitelist_for_serdes WorkerStatus enum (#6716)
by @johannkmdocstrings for AssetGroup and AssetGroup.build_job (#6696)
by @dpeng8170.14.1 changelog (#6720)
by @alangenfeldRemove bad invariant from back when sensor status was a fixed status, not a default status (#6711)
by @gibsondanAutomation: versioned docs for 0.14.1
by @elementl-devtoolsAdd a check in run execution that the run actually exists, just like in step execution (#6689)
by @gibsondanadd back mutation tests for sensor/schedules with default status (#6721)
by @prhaExport build_reconstructable_target to match cragified docs (#6317)
by @prhaadd build_run_status_sensor_context (#6543)
by @jamiedemariaRename metadata (remove "Event") in graphql/dagit (#6656)
by @smackeseySpeed up get_assets_for_run_id when there are a lot of events (#6735)
by @gibsondanundefined (#6742)
by @alangenfeldFix invariant check in databricks job/pipeline path (#6718)
by @iswariyammake Content Gap an issue label instead of part of title (#6745)
by @sryza0.14.0 migration.md (#6715)
by @clairelin135Move bollinger example to AssetGroup and add README (#6749)
by @sryzamove software-defined assets toy to asset collection (#6643)
by @sryzarename conditional branching example to nyt and add README (#6757)
by @sryza[docs infra] remove old versions from versioned_navigation (#6704)
by @yuhanfix typo in ops rst (#6758)
by @yuhanupdate error messages for ExternalAssetDependency/ExternalAssetDependedBy (#6748)
by @OwenKephart[Docs] Upgrade Next.js to 12.1.0 (#6722)
by @keyz[docs infra] fix changelog's On This Page (#6706)
by @yuhanfix(helm): remove registry field (#6744)
by @rexledesmadocs(helm): update chart metadata (#6778)
by @rexledesmarespect default_value in loadable input check (#6771)
by @alangenfelddisplay non-sda assets in asset catalog (#6779)
by @prhareplace EventMetadata in docs (#6774)
by @smackeseyMigrate user event docs to use log_event, various docs cleanups (#6772)
by @dpeng817[dagster-dbt] fix issues with dbt asset integration (#6777)
by @OwenKephartFilter custom events from stats_for_run query (#6781)
by @johannkm[Dagit] Fix missing op name in the asset sidebar (#6765)
by @bengotowadd validate_run_config to api docs (#6782)
by @prhaadd api doc for op_def, graph_def (#6776)
by @prha[examples] dbt -> dbt-core in dbt_example (#6786)
by @OwenKephart[dagit] Show full asset paths on details pages, tweak filter interactions (#6764)
by @bengotowAdd small toggles to Asset Details graphs, consolidate localStorage-backed state (#6762)
by @bengotow[dagster-shell] Fix deadlock with large output (#6435) (#6490)
by @kbd[dagster-dbt] pin some stuff (#6791)
by @OwenKephartdeflake test_retry_policy (#6644)
by @alangenfeld0.14.2 changelog (#6789)
by @sryzaadd run_request_for_partition for job definition (#6790)
by @prhaAdd a "log framework error" hook to execute_step invocations, similar to the one in execute_run (#6690)
by @gibsondanfix up reexecution api example docs (#6787)
by @prhareorganize dagster.check module (#6750)
by @smackeseyAdjustments/condensing of dagster.check (#6751)
by @smackeseyAutomation: versioned docs for 0.14.2
by @elementl-devtoolsfix: use run status to determine success for an in-process result (#6784)
by @rexledesmanamedtuple to NamedTuple (1st batch) (#6752)
by @smackeseycorrected export message from "~/dagster_home" to ~"/dagster_home" (#6327)
by @Proteusiqdocs: remove duplicate entries under integrations (#6724)
by @kahnwongnamedtuple to NamedTuple (2nd batch) (#6753)
by @smackeseynamedtuple to NamedTuple (3rd batch) (#6767)
by @smackeseynamedtuple to NamedTuple (4th batch) (#6785)
by @smackesey[docs infra] speed up vercel build with ISR (#6773)
by @yuhan[docs infra] no s3 when pulling versioned content (#6705)
by @yuhanStill write cursors even when a sensor tick fails (#6807)
by @gibsondanDirect invocation of assets for unit testing (#6761)
by @jamiedemariaPartition docs updates (#6814)
by @jamiedemariafix pandera link on integrations page (#6813)
by @smackesey[docs] regenerate yarn.lock (#6815)
by @alangenfeld[events] remove message field (#6769)
by @alangenfeld[dagster-postgres] improve connection retry (#6831)
by @alangenfeldset ErrorSource when retry request limit exceeded (#6806)
by @alangenfeldUpdate black, isort (#6714)
by @smackeseyExclude materializing run from materialization warning run count (#6692)
by @clairelin135rename PipelineRunsFilter => RunsFilter (#6811)
by @prhaassorted function type annotations (batch 1) (#6800)
by @smackeseyrestore old metadata name deprecation warnings (#6688)
by @smackeseyAlso override cpu/memory in containerOverrides (#6836)
by @jmsandersmove toys from build_assets_job to AssetGroup (#6801)
by @sryzaSimplify IWorkspace API (#6821)
by @gibsondan[dagster-aws] Add secretsmanager resources (#6802)
by @benpankowchore(daemon): retrieve daemon heartbeats in batch (#6843)
by @rexledesmaRevert isort Makefile config to use git ls-files (#6845)
by @smackeseyfix: query heartbeats db once (#6854)
by @rexledesma[examples] Modern Data Stack + SDA Example (#6862)
by @OwenKephartadd table api doc page (#6840)
by @smackesey[dagit] Avoid copying definition-time tags into launchpad state (#6851)
by @bengotow[dagit] Replace black tag tooltips with as-needed, selectable ones (#6844)
by @bengotowdefault to fs_asset_io_manager in AssetGroup (#6859)
by @sryzamove slack apidoc example to op/job (#6855)
by @sryzaRemove @abstractproperty (#6867)
by @johannkmgate bucketing queries on sqlalchemy version (#6841)
by @prhaDon't carry forward all container definitions (#6850)
by @jmsanderswhen multiple repos have the same asset, have graphql return the non-source one (#6856)
by @sryza[dagit] Add a helper for building instance/runs paths, resolved #6847 (#6848)
by @bengotow[dagster-fivetranAdd Fivetran Resync Op (#6868)
by @dwallace0723[dagit] Use a custom hook to manage query polling (#6805)
by @bengotowenable cross-job asset partitions (#6865)
by @sryza[dagit] Add a gradient behind the Gantt chart step selector now
by @souterjkremove gitignore setting from isort (#6872)
by @smackeseyRandom formatting fixes (#6888)
by @dpeng817AssetGroup.from_modules (#6884)
by @sryzaisort dagster-aws conftest (#6892)
by @sryzafix expansion so its shell-neutral (#6894)
by @smackeseybuild_assets_job -> AssetGroup in SDA guide (#6879)
by @sryzaChangelog (#6908)
by @dpeng817add entries to 0.14.3 changelog (#6909)
by @sryzaRename build_run_status_sensor_context in changelog (#6904)
by @gibsondanmake EventLogEntry.message change safe to downgrade (#6912)
by @alangenfeldFor those whome detest wordwrap in vscode (#6887)
by @dpeng817Solid error message migration (#6883)
by @dpeng817Error when amultiple asset groups are used in one repo, when a job is passed in with the asset group reserved name (#6891)
by @dpeng817assorted sensor and schedule fixes (#6759)
by @jamiedemaria[examples] For modern_data_stack_assets, more detailed README, helper script (#6914)
by @OwenKephartfix bug with subset selection (#6921)
by @OwenKephartenable vixie-style cron strings (#6810)
by @prhaAutomation: versioned docs for 0.14.3
by @elementl-devtoolsIn run status sensors, snapshot the status in a PipelineRunReaction rather than pulling it from the PipelineRun object (#6931)
by @gibsondan[dagster-airbyte][examples] fix bug with empty prefix (#6932)
by @OwenKephartRFC: resolve k8s job and ops tag entirely using k8s_model_from_dict (#6205)
by @gibsondanrevert stray dagster[lint] for BK mypy (#6936)
by @smackeseyMigrate reference to solid in error message (#6935)
by @dpeng817dagit-debug fix in-mem daemon heartbeats (#6943)
by @alangenfeld[docs] details on mapping dynamic outputs (#6140)
by @alangenfeld[docs] op retries concepts section (#6818)
by @alangenfeldfix makefile issue with check_black/check_isort (#6944)
by @smackeseychange query to be compatible with sqlalchemy 1.3 (#6930)
by @prhaadd a blank state message for the asset table (#6792)
by @prhafix black
by @alangenfeldRemove
update_doc_snapshot from root Makefile (#6837)
by @smackeseynamedtuple to NamedTuple (5th batch) (#6917)
by @smackeseyfunction type annotations (batch 2) (#6933)
by @smackeseywhen building AssetGroups from modules, gather lists of assets (#6967)
by @sryzaremove legacy version of hacker news schedule (#6797)
by @sryzaungate run batching (#6966)
by @prhaUse AssetGroup in hacker news demo (#6965)
by @sryzaAssetGroup.from_current_module (#6959)
by @sryza[Dagit] Nits and polish based on user feedback (#6799)
by @bengotowadd support for filtering ticks by status (#6919)
by @prha[dagit] Only warn about the backfill daemon when backfills are present (#6846)
by @bengotowswitch arg name for metadata value class (#6675)
by @smackesey[dagit] Use a virtualized list in <SuggestWIP /> to fix launchpad partition picker perf (#6852)
by @bengotowFix missing partition bars in large partition data-sets (#6877)
by @souterjkQuote selection to avoid grabbing downstreams with the same name prefix when viewing upstream graph, fix react table unhappy log (#6907)
by @souterjkmake all optional params explicit (#6934)
by @smackeseyAdds pez popover as requested in #6470 (#6871)
by @souterjkfix pylint for modern data stack example (#6975)
by @smackeseydev_install tweaks (#6970)
by @smackeseypartitioned assets toy (#6960)
by @sryzaUpdate create-new-project.mdx (#6898)
by @frcodeAdd short-circuit in run start time if run failed to start (#6991)
by @gibsondanAllow PathMetadataValue to accept os.PathLike input (#6920)
by @abkfenrisRaise run_task errors (#6993)
by @jmsanders[dagit] React.FunctionComponent -> React.FC (#7002)
by @hellendagFix partitioned asset jobs with double downstream non-partitioned (#6986)
by @sryzaadd ipython_genutils as dagit, dagstermill dep (#7016)
by @smackeseySet up graphql mutation for setting a sensor cursor value (#6922)
by @prha[dagit] Fix warning icon tests, add one for present-but-stopped schedule (#7026)
by @bengotowStop building duplicate downstream asset dependencies (#7001) (#7012)
by @bengotow[dagit] Add error handling to Asset "Rematerialize Partitions" modal (#7011)
by @bengotow[dagit] Fix tooltips in Gantt chart, use monospace font and middle truncation (#7025)
by @bengotow[dagit] Fix right Asset Graph panel missing if saved "explorer" state is absent (#7013)
by @bengotowFix dagster k8s typo (#7028)
by @johannkmAdd sequence and mapping check functions (#7006)
by @smackeseychange backcompat pin to nbconvert (#7027)
by @smackesey[dagit] Hide the asset group job on the instance overview (#7019)
by @bengotow[dagit] When materializing assets, link to run instead of showing it (#7014)
by @bengotowAssorted typing additions and adjustments (#7007)
by @smackeseyremove unneeded deps from dagit tox file (#7022)
by @gibsondanSwitch sensor/schedule view to show tick list instead of run list (#6861)
by @prhaupdate AssetGraph dedup logic (#7021)
by @smackeseyMake parse_run_config_input more defensive (#7003)
by @gibsondanExpose metadata on AssetNode (and source assets) directly (#6900)
by @smackeseyadds source asset metadata to dagit (#7015)
by @smackeseyAdllow telemetry URL to be overridden with env var (#6946)
by @dpeng817Remove space from dbt example notebook path (#7038)
by @smackeseyadd tick batching to schedule storage (#6995)
by @prhaadd batch tick loader for repository-scoped schedule/sensor ticks (#6997)
by @prhaK8s executor filter down to dagster events (#7000)
by @johannkmAllow the use of environment variables in k8s_job_executor (#7031)
by @johannkm[bug] hacky fix for step launcher behavior (#6866)
by @OwenKephartmake resource config available when loading from assets outside job (#7029)
by @sryzaRename step jobs 'dagster-step' in k8s and docker (#6982)
by @johannkmminor improvements to check (#7036)
by @smackeseytest raising exceptions in AssetGroup in deterministic order (#6945)
by @jamiedemariaSDA docs fixes (#6905)
by @jamiedemariaskip test_giant_external_repository_streaming_grpc (#7049)
by @alangenfeldFault tolerance docs (#5432)
by @johannkmSilence experimental warnings triggered by internal code (#6873)
by @jamiedemariaK8s docs ignore tag and show executor (#7053)
by @johannkmAssorted fixes to the hackernews assets demo (#7030)
by @sryzaSupport the same Secrets config syntax as ECS (#6985)
by @jmsandersfix the sqlite gating logic (#7058)
by @prha[dagster-airbyte] [dagster-fivetran] cleanup + tests (#7056)
by @OwenKephartReuse ECS task definitions (#7009)
by @jmsandersUpdate last_materialization_timestamp to also update on observation (#6885)
by @clairelin135Optimize Latest Runs Query (#7066)
by @clairelin135[dagit] Remove default InstanceWarning mocks injecting randomness (#7054)
by @bengotowfix tag batching with tag-based filters (#7055)
by @prhaFix #6512 (#6969)
by @petehuntK8s check run health test (#7070)
by @johannkmhn local repo that uses duckdb as warehouse (#6968)
by @sryzafix bug in fetch_assets.get_asset_nodes_by_asset_key (#7063)
by @sryzaClear out 'dependsOn' when creating a default ECS task definition (#6929)
by @gibsondan[dagit] Render status page navigation when factory floor loading (#7079)
by @benpankowrun mypy on dagster-graphql (#7075)
by @prhafix logic for turning schedule ticks into hour boundaries when there's an offset (#7071)
by @gibsondanFixes #7082 (#7086)
by @smackeseyFix failing check.path test on windows (#7077)
by @smackesey[dagit] refactor graphql execute methods (#7065)
by @alangenfelddagster-snowflake drop 3.6 support (#7092)
by @alangenfeld[dagit] Fix Gantt chart layout hang on certain graph structures #6942 (#7085)
by @bengotowLog failed compute log to event log rather than buried in a logging output (#7093)
by @gibsondan[dagit] Disallow use of gql from graphql.macro (#7042)
by @bengotow[dagit] Add sensors/schedules to the asset details page (#7041)
by @bengotow[dagit] Keep user-entered tags when changing presets / partitions (#7068)
by @bengotowAdd gitpod.yml (#7088)
by @petehunt[dagit] Lower factory floor timeline fidelity to seconds from ms (#7091)
by @benpankow[dagit] Show job failures on asset graph, fix in-progress runs on global graph (#7067)
by @bengotowOptionally use sidecars (#7087)
by @jmsandersFix kwarg case with invocation (#7095)
by @dpeng817Return an empty secrets list for task definitions (#7096)
by @jmsandersFixup ECS secrets docs (#7097)
by @jmsanders0.14.4 changelog (#7099)
by @yuhanrefetch the backfill before updating, to avoid clobbering cancels (#7094)
by @prhaAutomation: versioned docs for 0.14.4
by @elementl-devtoolsfix perms in dagit (#7115)
by @prhaRemove hard failure if missing inputs (#7116)
by @dpeng817Fix internal code metadata warnings (#7112)
by @smackesey0.14.5 Changelog (#7119)
by @jmsandersAutomation: versioned docs for 0.14.5
by @elementl-devtoolsbatch run fetches in sensor daemon (#6950)
by @prhagcs asset io manager (#7081)
by @jamiedemariaDon't give up if the terminate() call raises an exception during force-termination (#7110)
by @gibsondan[dagit] Add optional close button to Alert component (#7107)
by @benpankowAdd dagit test to make sure Permission flags stay in sync (#7117)
by @prhaexpose compute_kind tag value on gql AssetNode (#7120)
by @smackeseyreplace internal uses of
MetadataEntry static api (#7126)
by @smackeseydynamic out bug fix (#6780)
by @alangenfeldCreate oss-issues.yml (#7132)
by @yuhanfix docs search: js_render shouldnt be true (#7146)
by @yuhanmypy version bump (#7149)
by @smackeseydocs(sda): fix typos and edit wording (#7136)
by @rexledesmaadd permissions guard for schedule mutations (#7133)
by @prhaenable saving run key for ticks that skip runs because of run key idempotence (#7130)
by @prhaadd tick tooltip info for skipped ticks due to run key idempotence (#7131)
by @prhaRename the InstanceMigratedError to be less pushy about migrating (#7141)
by @gibsondan[postgres] share db engine with watcher_thread (#7143)
by @alangenfeldAdd Map to config type API docs (#7154)
by @gibsondanPass on PythonError handling that should be using PythonErrorFragment (#7152)
by @gibsondan[oss telemetry] Get rid of telemetry version (#6760)
by @dpeng817only show warning when force termination checkbox is checked (#7157)
by @prharename several internal submodules (#7134)
by @smackeseyRemove another cron from example dockerfile (#7155)
by @gibsondanBe more willing to return serialized errors in schedule/sensor/grpc partition methods (#7164)
by @gibsondandocs: fix spelling (#7166)
by @antquinonez[multiprocess] tweak preload modules logic (#7142)
by @alangenfeldfeat(dagster-k8s): add run id label to run/step workers (#7167)
by @rexledesmadocs(logging): remove example about capturing from the root logger (#7168)
by @rexledesmadocs: fix typo in job/graph concepts page (#7172)
by @antquinonez[docs] fix dynamic graph examples (#7174)
by @alangenfeldFix inclusion of tests in our published packages (#7169)
by @smackeseyRFC: add partition tags to partitioned config (#7111)
by @prhaadd gcs pickle asset manager to api docs (#7177)
by @jamiedemaria[dagster-databricks] better retry support for step launchers (#7171)
by @OwenKephartfix lint/mypy failures (#7183)
by @prhaAutomation: versioned docs for 0.14.6
by @elementl-devtools0.14.6 changelog (#7184)
by @jamiedemariaupdate docs on Dict type (#7179)
by @jamiedemariadocs: use word "op" instead of "solid" (#7195)
by @joe-hdairefactor: rename is_failure to is_failure_or_canceled (#7200)
by @rexledesmaTime window partitions with custom minute, hour, day offsets (#7125)
by @jamiedemariaFixup ECS docs (#7193)
by @jmsandersUpdate version placeholder "dev" to "0+dev" (#7175)
by @smackeseyfix: prevent runs filter from accepting an empty list of run ids (#7217)
by @rexledesmarefactor: remove screenshots (#7225)
by @rexledesmarefactor: remove scala modules (#7226)
by @rexledesmaget MetadataValue.dagster_run to show up in apidoc (#7231)
by @sryzaAzure asset IO Manager (#7201)
by @jamiedemariaPartitionedConfig testing helpers (#7216)
by @jamiedemaria[postgres] dont dispose engine in watch thread (#7241)
by @alangenfeldfeat(helm): add configuration for Dagster telemetry (#7190)
by @rexledesmaadd some type annotations for code loading (#7244)
by @sryzafeat: add alert failure event (#7238)
by @rexledesma[dagit] Remove the Asset Node context menu (#7151)
by @bengotowOnly consider repos defining the asset with a corresponding op for multiple repo notice (#7186)
by @bengotow[dagit] Fix Gantt chart layout hang on certain graphs, round 2 #6942 (#7188)
by @bengotow[dagit] If re-executing from failure fails, default to from failure again #5019 (#7189)
by @bengotow[dagit] Ignore source assets for “Materialize All” button (#7196)
by @bengotow[dagit] Load “Launch Asset partitions” modal when it’s opened + every time (#7207)
by @bengotow[dagit] Explicitly add the dagster/step_selection tag to asset runs (#7213)
by @bengotowRemove redundant k8s launcher event (#7250)
by @johannkmAllow asset observations in IO Managers (#6653)
by @clairelin135update docs for adls2 asset io manager (#7252)
by @jamiedemaria0.14.7 changelog (#7256)
by @OwenKephartUpdate CHANGES.md
by @prhaAutomation: versioned docs for 0.14.7
by @elementl-devtoolsfix typecheck for In.dagster_type (#7242)
by @smackesey[dagit] RunTimeline: Fix overlap with ongoing run (#7245)
by @hellendag[dagster-io/ui] Remove WIP suffixes (#7229)
by @hellendag[dagit] Modify "Open in Launchpad" from Run (#7263)
by @hellendagRFC: Add register_run_asset event (#7098)
by @clairelin135Add table to keep track of schedule data migrations 1/5 (#7182)
by @prhaadd schema for instigators table, keyed by selector 2/5 (#7185)
by @prhadouble-write selector_id to ticks, jobs, instigators 3/5 (#7191)
by @prhaadd tick selector index migration (#7198)
by @prhaRead instigator state / ticks off of selector_id instead of origin id 5/5 (#7268)
by @prha[dagster-io/ui] Declaration files for TS, Rollup for built JS (#7220)
by @hellendagEnsure telemetry logging dir is created before using logger (#7192)
by @dpeng817cross-repo assets toy (#7210)
by @sryzaupdate apidoc for non_argument_deps (#7271)
by @clairelin135enable loaded_input metadata entries to display in Dagit (#7255)
by @clairelin135[@dagster-io/ui] Cleanup for publish (#7270)
by @hellendagRemove empty k8s_job_task file (#7286)
by @johannkmAdd MenuLink shim for internal compatibility (#7297)
by @hellendagAvoid mentioning pipelines in Dagit empty state (#7288)
by @sryzafix invariant in batch tick loader (#7280)
by @prha[dagster-airbyte] fix airbyte materializations without streamStats (#7281)
by @OwenKephartpoint top-level workspace to latest toys (#6876)
by @sryza[dagster-io/ui] Eliminate RR dependency (#7285)
by @hellendagfix selector injection for k8s-celery test (#7307)
by @prharemove experimental warning for mysql storage (#7290)
by @prhamake some edits to changelog (#7264)
by @prha[@dagster-io/ui] Clean up deps (#7302)
by @hellendag[dagit] Fix config refresh when reloading repo (#7325)
by @hellendag[@dagster-io/ui] Prepare for 1.0.3 (#7326)
by @hellendagAdd run_id as execute_in_process arg (#7317)
by @dpeng817Fixes to root input manager memoization and no context provided (#7316)
by @dpeng817add apple m1 instructions to contributor guide (#7279)
by @jamiedemariaFix celery docker executor failing on None metadata (#7330)
by @jamiedemariareplace old celery docs site (#7331)
by @jamiedemaria[dagster-dbt] fix issue w/ conflicting dbt op names for assets (#7311)
by @OwenKephartdocs infra: remove github/slack bot from docs site (#7315)
by @yuhan[dagit] Fix CodeMirror show-hint error (#7334)
by @hellendag0.14.8 Release Notes (#7336)
by @clairelin135add email icon (#7346)
by @salazarm[dagit] Don't show instigation switch for assets (#7347)
by @hellendag[dagit] Add "Copy config" button to "View tags and config" dialog (#7348)
by @hellendag[dagit] Allow toggling repos in bulk (#7344)
by @hellendag[dagit] Store whitespace state in localStorage (#7350)
by @hellendag[dagit] Adjust details in Run header (#7345)
by @hellendag[dagit] Add feature flag to collapse the left Jobs navigation by default (#7204)
by @bengotowAutomation: versioned docs for 0.14.8
by @elementl-devtoolsupdate README - include github discussion (#7339)
by @yuhan[@dagster-io/eslint-config] Create shared config (#7328)
by @hellendag[@dagster-io/eslint-config] Fix dependencies (#7354)
by @hellendagAdd Slack icon (#7355)
by @salazarmsupport container kwargs for celery-docker (#7335)
by @jamiedemaria[dagster-dbt] Add /runs method to DbtCloudResourceV2 (#7258)
by @kstennettlull[dagster-dbt] add "build" to DBT_RUN_RESULTS_COMMANDS (#7362)
by @OwenKephart[dagit] Show timestamps for “failed to start” runs, other metadata tweaks (#7359)
by @bengotowFix contention bug in windows (#7353)
by @dpeng817fix dagster-ge library (#7369)
by @prhahave fs io manager record path metadata entry (#7261)
by @sryzaAssetGroup.materialize_in_process (#7260)
by @sryzaAdd authenticator field to conn_args (#6983)
by @swotaiMake shell ops inherit env vars from outside environment (#6513)
by @kbdAllow jobs to be passed in lazily to repository definitions (#7382)
by @gibsondan[dagit] Make the “Latest Run” tag on the Job page auto-refresh (#7360)
by @bengotow[dagit] Move data fetching for the asset graph into hooks (#7363)
by @bengotowautodiscover assets at module scope (#7247)
by @sryza[hacker news] pin duckdb (#7390)
by @alangenfeldLet InProcessRepositoryLocationOrigin take in multiple repos (#7386)
by @gibsondan[dagster-fivetran] Fix format strings (#6436)
by @mgasner[dagster-airbyte] Handle 204 reponse in Airbyte Library (#7209)
by @HAMZA310run materializations (#7394)
by @prha[dagit] Fix default Dagit page in single repo, multiple jobs case (#7368)
by @bengotowAdd grpc server timeout config to instance settings (#7387)
by @gibsondaninfer resource descriptions from docstrings (#7309)
by @sryza[dagit] Async layout of large asset graphs (#7366)
by @bengotow[dagit] Align terminology and data structures in asset / op graph layout (#7372)
by @bengotow[dagit] Move asset-graph out of workspace dir, non-graph code to /assets (#7373)
by @bengotowUpdate AssetKeyTable.last_run_id upon materialization planned (#7319)
by @clairelin135asset tutorial (#7269)
by @sryzaAdd a field to the now-somewhat-poorly-named RepositoryPythonOrigin that can add 'deployment target' information within a repo location (#6925)
by @gibsondanAdd EcsContainerContext (just with secrets for now) (#7176)
by @gibsondanAdd K8sContainerContext for configuring k8s pods from the repository location (#7203)
by @gibsondanFixed typo in deploying-with-helm.mdx (#7416)
by @LeoHuckvale[dagit] Use [email protected] (#7393)
by @hellendag[dagit] Add arrow key and double click interactions to asset graph #6407 (#7374)
by @bengotow[dagit] Allow colon in run tag value (#7409)
by @hellendagRe-execute run instance method (#7417)
by @johannkmskip adls test (#7420)
by @jamiedemariaadd open launch pad to run details page (#7304)
by @salazarmFix prerequisite typos in docs (#7424)
by @johannkm[@dagster-io/eslint-config] Enable object-shorthand (#7423)
by @hellendagFilter out asset_materialization_planned event logs client-side (#7397)
by @clairelin135Make user repo filters per deployment (#7376)
by @salazarm[postgres] urlquote usernames as well (#7430)
by @alangenfeld[toys] squelch warnings (#7419)
by @alangenfelddocs for gRPC sever timeout flag (#7400)
by @gibsondanFix master and add DagsterEventType import (#7435)
by @clairelin135avoid extra experimental warning when asset has partitions def (#7431)
by @sryzaAdd option to disable max_concurrent_runs (#7327)
by @johannkmRFC: Fetch Cross-Repo Asset Deps (#7259)
by @clairelin135make k8s container context volumes and volume mounts use snake case, not camel case (#7437)
by @gibsondanmove expensive tests out of core_tests (#7436)
by @gibsondanallow AssetGroups to have assets with different partitions defs (#7388)
by @sryzaUpdate key for saving data in useExecutionSessionStorage (#7333)
by @salazarmasset partitioned io manager (#7413)
by @sryzaUnbreak stopping unloadable schedules/sensors (#7446)
by @gibsondan[dagit] Check for asset group job prefix to support multiple groups per repo (#7418)
by @bengotowchangelog for 0.14.9 (#7450)
by @sryzaupdate isort comment formatting (#7448)
by @smackeseybump black to 22.3.0 (#7449)
by @smackeseyAutomation: versioned docs for 0.14.9
by @elementl-devtoolsTypo in the create a new project doc (#7441)
by @ascrookesFix issue with blank telemetry instance id file (#7433)
by @dpeng817Show repo selector in unloadable schedules/sensors instead of repo location metadata (#7444)
by @gibsondanSplit up screenshot specs (#7460)
by @sryzaresource dependencies for root input managers (#7459)
by @sryzaavoid framework code triggering metadata_entries warning (#7412)
by @sryzaallow combining memoization with step selection (#6431)
by @sryzarefactor: remove local storage file that was committed (#7440)
by @rexledesma[dagster-graphql] remove gevent (#7402)
by @alangenfeld[dagit] Apply middle truncation to long asset nodes, hide run info (#7187)
by @bengotow[dagit] Use CodeMirror for all config displays (#7467)
by @hellendagAdd container_context to gRPC server and Helm chart (#7236)
by @gibsondanAdd asset metadata to one of the dagster_test asset groups (#7462)
by @bengotowFix back-compat for new helm chart param for container context (#7476)
by @gibsondan[graph] use the correct input definition for type checks (#7453)
by @alangenfeld[dagit] Repair async graph layout for path-prefix (#7480)
by @hellendagAdd docs to k8s docs about includeConfigInLaunchedRuns (#7477)
by @gibsondanAcquire leases when using ADLS (#7410)
by @jamiedemariaUpdate Materialization Warning (#7265)
by @clairelin135[dagit] Fix Suggest (#7489)
by @hellendagMake the scheduler/sensor daemons key the schedules that it inspects by selector ID, not origin ID (#7492)
by @gibsondanAdd right of search bar prop to top nav (#7500)
by @salazarmadd flag argument to backfill all partitions in launch backfill graphql mutation (#7499)
by @prha[dagit] Fix job header “View assets” link, add missing graphQL types (#7493)
by @bengotowfix sqlite gating logic for batch tick queries (#7505)
by @prha[dagster-census] Census Integration Library (#7249)
by @dehume[dagit] Show a better error on invalid run filters (#7494)
by @hellendagAdd log message to grpc health check (#7506)
by @johannkmrefactor: remove duplicate solids in event log test (#7511)
by @johannkmcorrect workspaces icon (#7504)
by @salazarm[dagit] More cleanup of query callsites performing background polling (#7508)
by @bengotow[dagit] Avoid resetting hidden repos on each page load #7519 (#7520)
by @bengotow[dagit] Allow disabling keyboard shortcuts (#7515)
by @hellendagUpdate pylint and associated config (#6517)
by @smackesey0.14.10 Changelog (#7521)
by @johannkmfix asset partitions runtime check (#7525)
by @smackesey[dagit] Tiny fix for “asset graph search, dropdown stops showing when ">" character is used” (#7514)
by @bengotow[dagit] Move yarn analyze to packages/app (#7523)
by @hellendag[dagit] Clean up lodash imports (#7526)
by @hellendag[dagit] Display repositories in a stable order (#7518)
by @bengotowAutomation: versioned docs for 0.14.10
by @elementl-devtools[hooks] fix resource reqs in nested graphs (#7529)
by @alangenfeld[instigator] dont throw in start/stop if already in desired state (#7483)
by @alangenfeld[dagit] Fix InstigationTick "Skipped" message if zero runs (#7527)
by @hellendagGet rid of mode in JobDefinition constructor (#7497)
by @dpeng817Fix issue where we were calling the partition function n times, where n is the number of partitions returned by the function (#7539)
by @gibsondan[bugfix] Check that oldData is defined before trying to migrate (#7537)
by @salazarmAssorted tox updates (#6971)
by @smackesey0.14.11 changelog (#7543)
by @gibsondanAutomation: versioned docs for 0.14.11
by @elementl-devtools Reexecute run mutation using just parent run (#7510)
by @johannkmUpdated changelog for 0.14.12 (#7545)
by @gibsondanRevert "[bugfix] Check that oldData is defined before trying to migrate (#7537)" and "Update key for saving data in useExecutionSessionStorage (#7333)" (#7551)
by @gibsondan[graphql] subscription handle_graphql_errors (#7482)
by @alangenfeldAutomation: versioned docs for 0.14.12
by @elementl-devtools[dagit] Add bulk re-execution dialog to Runs page (#7451)
by @hellendagDowngrade tox pip for Python 3.6 (#7547)
by @smackeseyannotate dagster.builtins.Nothing with correct type (#7546)
by @smackeseyPylint: Add encoding to open() (#7548)
by @johannkmasset op tags (#7472)
by @sryza[dagit] Show asset “compute kind” in the global asset graph (#7516)
by @bengotow[dagit] Hide __asset_group partition sets from search bar (#7556)
by @bengotowAdd typing-extensions >=3.10 as dependency. (#7555)
by @smackeseyvalidate tags in partition sets (to turn dicts into strings) (#7554)
by @gibsondan[dagit] Feature flag for disabling WebSocket usage (#7557)
by @hellendag[dagit] Add opNames field to AssetNode (#7565)
by @hellendagadd DEFAULT_PYTHON_VERSIONS env var for BK (#7553)
by @smackeseyadd DEFAULT_PYTHON_VERSIONS env var for BK (#7553)
by @smackesey[1/n] Interop Stack: Use AssetJobInfo at runtime instead of OutputDefinition (#7473)
by @OwenKephart[2/n] Interop Stack: Use AssetJobInfo to construct ExternalAssetNodes (#7475)
by @OwenKephart[dagit] Introduce sticky table headers (#7568)
by @hellendag[dagit] Truncate global search result list (#7569)
by @hellendagTest test_create_reexecuted_run_from_failure (#7566)
by @johannkmTag behavior option for reexecute method (#7574)
by @johannkmadd setuptools as dep (#7576)
by @smackeseyfix errors (#7577)
by @smackeseyRegister a new task definition for IAM changes (#7564)
by @jmsandersImprove typing of schedule/sensor and hooks (#7560)
by @smackeseyAll steps option for rexecute method (#7575)
by @johannkm[docs] fix multi asset sensor example (#7583)
by @alangenfeldCreate runs during event log tests (#7502)
by @johannkm[dagit] Refactor ExecutionSessionStorage (#7559)
by @hellendagfix(helm): support numeric type image tags (#7579)
by @jrouly[dagit] Add bulk re-execution of all steps (#7591)
by @hellendagResourceDefinition.none_resource instead of no_step_launcher (#7381)
by @sryzaRemove unused imports from event log tests (#7592)
by @johannkm[dagit] Fix asset graph navigation edge cases (#6955, #7208) (#7532)
by @bengotowfullbuild in branch name for full BK build (#7595)
by @smackeseydocument partitioned assets (#7466)
by @sryzafix secretsmanager example doc (#7593)
by @prhaAdd ConfigSchema type alias (#7596)
by @smackeseylint dagster_tests (#7594)
by @alangenfeldadd dialect-specific update logic (#7572)
by @prhafix volumes example for docker run launcher (#7607)
by @gibsondan[tests] set platform on docker build (#7603)
by @alangenfeldUse explicitly set env vars rather than autoenvvar prefixes for the grpc command (#7611)
by @gibsondanFixes for running the user-code-deployment helm chart in a different namespace than the system namespace (#7597)
by @gibsondanstandardize additional_message kwarg in check methods (#7605)
by @smackesey[3/n] Interop Stack: Remove asset key from inputs/outputs (#7550)
by @OwenKephartfix(helm): allow max concurrent runs to be 0 (#7618)
by @rexledesma[4/n] Interop Stack: node_def on AssetsDefinition can be graph (#7573)
by @OwenKephartAsset key to node handle mapping (#7599)
by @clairelin135[dagster-aws] update emr pyspark step launcher (#7604)
by @OwenKephartMake dagster-user-deployments service account set up a role and role binding by default, like the main dagster chart service account (#7622)
by @gibsondanAdd resources to K8sContainerContext (#7619)
by @gibsondan[dagit] Add "shared key path" outlines, experimental asset graph flag (#7608)
by @bengotow[@dagster-io/ui] Separate tsconfig for build (#7584)
by @hellendagFixup XXXSource docs (#7629)
by @gibsondanChangelog 0.14.13 (#7631)
by @dpeng817AssetGroup.prefixed (#7395)
by @sryzaAssetsDefinition.from_graph (#7620)
by @clairelin135[Testing] Support for resolvers in ApolloTestProvider (#7612)
by @salazarm[dagit] Show default value for config (#7598)
by @hellendag[docker] improve test time (#7589)
by @alangenfeldUpdate mypy config and bump version (#7625)
by @smackesey[dagster-ge] drop 3.6 (#7638)
by @alangenfeldAutomation: versioned docs for 0.14.13
by @elementl-devtoolsUnify all alembic scripts into single directory, creating single history (#7411)
by @prhaenable mypy checking on dagit (#7626)
by @smackesey[dagit] Add dummy graphName field to AssetNode (#7637)
by @hellendag[RFC] lighter weight pull request template (#7538)
by @alangenfeld[dagit] AssetView: Don't show historical view message if it's not (#7639)
by @hellendagallow flexible types for SourceAsset key arg (#7633)
by @sryzaadd graphql fields to quickly query partition status (#7614)
by @prhaadd option to GraphQueryInput to apply changes without hitting enter (#7635)
by @prhaRefactors the PartitionGraph component to be divorced from the actual graphql query (#7616)
by @prhaAdds a new partition page behind a feature flag (#7617)
by @prhaenable adding asset groups together (#7634)
by @sryzaFix mypy config for dagster main package (#7644)
by @smackeseyallow multiple asset groups on a repository (#7649)
by @sryzarefactor: remove airline demo (#7653)
by @rexledesmato_source_assets (#7643)
by @sryzaadding a Terminate Run method for in-progress execution to the Python GraphQL client (#7443)
by @Javier162380fix apidoc for fs io manager (#7671)
by @sryzaFix celery docker jobs that involve writing directly to command-line output (#7665)
by @gibsondanFix docs lib links (#7680)
by @johannkmFix configmaps that need to be manually set when running the user deployments helm chart in a different namespace (#7660)
by @gibsondanAdd all repos to hacker_news __init__.py (#7677)
by @gibsondanremove primitive types from apidoc (#7674)
by @sryzaAdd py.typed to dagster and all extension libs (#7561)
by @smackesey[eslint-config] Bump dependencies (#7681)
by @hellendag[dagster-airbyte] terminate airbyte sync w python process (#7687)
by @OwenKephart[dagster-dbt] remove annoying color formatting strings from log output (#7688)
by @OwenKephartPull out PostgresEventWatcher into its own module and parameterize it more (#7666)
by @gibsondangraph and job apidoc fixes (#7673)
by @sryzadocs: fix example for multiprocess run config (#7693)
by @rexledesmafinish dagster-fivetran types (#7563)
by @smackeseyfinish dagster-airbyte types (#7562)
by @smackeseychore(graphql): add linter to ensure docstrings on graphene graphql objects (#7669)
by @rexledesmadocs(graphql): add docstrings to mutations.py (#7691)
by @rexledesmaMove EventSpecificData out of TYPE_CHECKING (#7697)
by @smackeseyAdd ability to set mapping key on op/solid invocation context (#7364)
by @dpeng817Make Output generic (#7202)
by @dpeng817update fake adls resource to work with leases (#7587)
by @jamiedemariaadd core partition status storage query (#7662)
by @prhadocs: specify the correct mysql event log module (#7703)
by @rexledesmaRevert "Make Output generic (#7202)" (#7715)
by @dpeng817Add bool metadata type (#7694)
by @clairelin135Add platform arg and other improvements to dagster-docker CLI (#7698)
by @smackeseyAdd a guard around dagit's JSON parsing of incoming requests (#7714)
by @gibsondanSwitch new partition page to use more efficient partition status query (#7663)
by @prha[docs] retry policy invocation example (#7716)
by @alangenfelddocs: fix example with lazy loading repository (#7704)
by @rexledesmaupdate slack api call in op hook docs (#7712)
by @jamiedemariaImprove Buildkite test version specification API (#7699)
by @smackeseyuse ">" for multi-component selection in AssetGroup.build_job (#7661)
by @sryzafinish dagster-pagerduty types (#7668)
by @smackeseyenable providing asset_key to build_output_context (#7696)
by @sryza[dagit] Support special cron strings (#7717)
by @hellendagsolid -> op in dagster-slack (#7684)
by @sryzaTyping additions (#7039)
by @smackesey[instance] rm get_addresses_for_step_output_versions (#7601)
by @alangenfeldAdd run tags for repository/location names (#6893)
by @prhaSurface retried error in Dagit (#7692)
by @johannkmRFC: Keep firing hooks even if a framework exception is raised mid-execution (#7652)
by @gibsondancombine validate_tags and check_tags to JSONify nested tags (#7720)
by @gibsondantest dynamic_output output for node (#7718)
by @clairelin135[postgres] dont use urlquote_plus (#7723)
by @alangenfeld[docs] fix jobs from graphs example (#7722)
by @alangenfeldMake output generic (#7719)
by @dpeng817Missed changes for Dagit surface retry error (#7731)
by @johannkmChange head_bucket to list_objects. (#7485)
by @trevenrawrRFC: include schema in default dbt asset keys (#7645)
by @sryzapopulate op names and graph name for ExternalAssetNode (#7721)
by @OwenKephartadd extra sensor state fetch to minimize chance of clobber state (#7738)
by @prhaWorkaround for mypy cache bug (#7732)
by @smackeseyfix run list for new partitions view (#7742)
by @prha0.14.14 changelog (#7743)
by @prhaMake Output and DynamicOutput no longer be tuple types (#7740)
by @dpeng817[dagit] Use a CSP header instead of meta tag (#7727)
by @hellendagAdd a test with various wait/act use cases (#7728)
by @hellendagAdd job/op equivalents to execution context (#7734)
by @johannkmRename ReexecutionPolicy -> ReexecutionStrategy (#7746)
by @johannkm[dagit] Update tsconfig to es2022, update browserslist (#7749)
by @hellendag[dagit] Use Array flat (#7754)
by @hellendagremove usable_as_dagster_type from integrations (#7682)
by @sryzaAutomation: versioned docs for 0.14.14
by @elementl-devtoolsenable returning lists of run requests from sensor/schedule evaluation functions (#7755)
by @prharemove usable_as_dagster_type from types concept page (#7758)
by @sryza[easy] fix missing space in error message for util function (#7761)
by @gibsondan[dagit] Add more security headers (#7764)
by @hellendagAdditional asset graph improvements (#7707)
by @bengotow[dagit] Add Reload All button to the asset catalog, remove repo filter (#7708)
by @bengotow[dagit] Better indentation guide in our CodeMirrors (#7765)
by @hellendagAllow python objects to be passed as top-level inputs to job (#7032)
by @dpeng817Add a static method on DefaultRunLauncher that launches a run without a workspace (#7777)
by @gibsondanRevert "Allow python objects to be passed as top-level inputs to job (#7032)" (#7779)
by @dpeng817Allow repositories to contain asset definitions and source assets for the same asset key (#7781)
by @sryzafix error message when can't find upstream asset for dep (#7648)
by @sryzaAllow TextInput to have a type of number (#7778)
by @salazarmfix error with inferring description from empty docstring (#7788)
by @sryza[dagit] Add hover actions to run Tags (#7751)
by @hellendagFixes to the apidocs page (#7729)
by @sryza[dagit] Show asset nodes on Asset Details (#7689)
by @hellendag[dagit] Make log filter input clearable (#7805)
by @hellendag[eslint-config] Add recommended jest lint (#7807)
by @hellendagpin xmltodict to fix aws test failures, until moto 3.1.9 (#7806)
by @prhaupdate s3 sensor docs to use context.cursor instead of context.last_run_key (#7748)
by @prhaAdd namespaces / prefixes to the hacker news assets demo (#7782)
by @sryzatoposort frozenset workaround (#7793)
by @smackeseyIncrease executor event log poll interval (#7803)
by @johannkmAdd a few __init__.py in test packages (#7809)
by @smackeseyremove TableSchema experimental warnings (#7799)
by @smackeseysnowflake io manager (#7726)
by @sryzaChange the fake repo that simulates an import error while loading dagster code (#7816)
by @gibsondanremove jinja / nbconvert pin (#7819)
by @prha[dagit] Use
/instead of
> for displaying asset paths (#7818)
by @bengotowfix snowflake io manager tests on python 3.7 (#7824)
by @sryzalink to examples in concepts pages (#7753)
by @jamiedemaria[docs] job execution (#7776)
by @alangenfeld[dagit] Clean up my todos (#7810)
by @hellendag[dagit] Fix global search for repeated item keys (#7827)
by @hellendagsys.meta_path import mapping layer (#7040)
by @smackeseyAllow passing posargs to tox commands (#7822)
by @smackeseyRename check > _check (#7808)
by @smackeseycontext.partition_time_window (#7795)
by @sryzatake out airline demo from make dev_install (#7839)
by @sryza[dagit] Make schedule/sensor tags more prominent in RunTags (#7834)
by @hellendagadd BoolMetadataValue to __all__ in dagster/__init__.py (#7838)
by @sryza[dagit server] guard on async websocket send (#7833)
by @alangenfeld[top-level inputs 1/2] Move InvalidSubsetError try-catch up a level (#7780)
by @dpeng817docs design update (#7531)
by @yuhanRemove test-connection pod (#7842)
by @gibsondanremove validate_asset_key_string (#7811)
by @sryzapartitioned assets from graphs (#7837)
by @sryzafallback logic for resolvers (#7766)
by @OwenKephartmigrate concept docs code snippet paths to graph/job/op (#6775)
by @sryza[rfc] generic + returnable dynamic outputs (#7744)
by @dpeng817[dagster-dbt] dbt build + AssetObservations from tests (#7783)
by @OwenKephart[dagit] New “folder grid” asset view behind experimental flag (#7767)
by @bengotowHandle backcompat asset observation (#7831)
by @clairelin135[dagit] Remove search item key (#7830)
by @hellendaguse generic Output type annotations in hacker news (#7600)
by @sryzachangelog 0.14.15 (#7850)
by @yuhan[dagit] add log-level flag (#7853)
by @alangenfeld[changelog][skip ci] dagit line and autofmt (#7857)
by @alangenfeld[dagit] tweak websocket disable (#7860)
by @alangenfeldLimit pods per run with k8s_job_executor (#7846)
by @johannkmadditions to the 0.14.15 changelog (#7856)
by @sryzafix asset key prefixes in MDS assets example (#7855)
by @sryza[helm] dagit.logLevel (#7862)
by @alangenfeldwarn for instance.get_event_records calls without an event type filter (#7848)
by @prha[top-level inputs 2/2] top level inputs on job and execute_in_process (#7786)
by @dpeng817Automation: versioned docs for 0.14.15
by @elementl-devtoolsrm duplicate changelog entry from navigation (#7873)
by @gibsondanFix error message assertions in Helm tests (#7858)
by @johannkmToxfile adjustments (adding mypy) (#7867)
by @smackesey[resources on repos 1/n] Reorganize job/pipeline/graph duplication error logic in repository (#7817)
by @dpeng817docs: fix vercel build (#7876)
by @yuhanenable @asset-decorated functions to accept kwargs (#7871)
by @sryzafix type error (#7887)
by @smackeseydagster-dbt types (#7878)
by @smackeseyAdd health check for ECS tasks (#7695)
by @jamiedemaria[cloud dagit] Add optional nav tabs parameter for settings root page (#7892)
by @benpankowfeat(helm): support custom configmap for workspace (#7882)
by @johannkmadd data migration for run repo tags (#7815)
by @prhamake docker_compose_cm a contextmanager rather than a fixture (#7905)
by @gibsondan[@dagster-io/eslint-config] v1.0.4 (#7913)
by @hellendagdocs: fix flash white box (#7877)
by @yuhanInputContext.upstream_output is missing key when source asset (#7919)
by @sryzaremove handle_schema_errors that wrap db errors with schema outdated exceptions (#7886)
by @prhaEnable asset partitioning support for @multi_asset defined assets (#7908)
by @aroigupdate multiple dynamic outs example (#7917)
by @jamiedemariadagster-databricks mypy (#7879)
by @smackesey[1/n] Subsetting Stack: use subset_selector for AssetGoup.build_job (#7796)
by @OwenKephartDon't call it a 'framework error' when a run worker unexpectedly restarts (#7885)
by @gibsondanchore(helm): remove local from compute log manager options (#7924)
by @rexledesmaExecutor and dagster-docker typing (#7881)
by @smackeseyremove markupsafe pin (#7898)
by @bollwyvl[2/n] Subsetting Stack: AssetsDefinition subsetting (#7797)
by @OwenKephartfix docstring (#7948)
by @OwenKephartRFC: Better error handling when an ExternalRepository grpc call fails (#7929)
by @gibsondanUse correct configured link in resources (#7949)
by @gibsondanadd CLI command to print the storage schema version (#7910)
by @prha[dagit] Present location reload errors after “Workspace > Reload All” (#7907)
by @bengotow[dagit] Asset details: Graph/op links (#7655)
by @hellendagFix detection of parent process death (#7914)
by @aroigMerge identical get_location and get_repository_location methods (#7952)
by @gibsondanadd missing check to AssetsDefinition.from_graph (#7960)
by @sryzaFix issue where azure IO manager was sometimes failing on recursive deletes (#7956)
by @gibsondanmake input defs deterministic (#7957)
by @clairelin135[events] can_load mitigation (#7955)
by @alangenfeldfix slack image link on Getting Started (#7964)
by @yuhanTyping for automation package (#7812)
by @smackeseymake StepLauncher and friends public (#7945)
by @smackeseyfeat(helm): bump minimum supported kubernetes version to 0.19 (#7925)
by @rexledesmaAsset Subselection (#7835)
by @clairelin135various fixes for new partition page (#7950)
by @prha[dagster-airbyte] don't cancel completed syncs (#7888)
by @OwenKephart[dagster-dbt] small tweaks (#7967)
by @OwenKephartError rather than warn when an event log query comes in that tries to use an int cursor on an event log DB that needs a RunShardedEventsCursor (#7970)
by @gibsondanStill make cursor editable if it is not set (#7969)
by @gibsondan[daemon] tweak per-thread instance/workspace setup (#7965)
by @alangenfeldRevert "feat(helm): bump minimum supported kubernetes version to 0.19 (#7925)" (#7968)
by @rexledesma0.14.16 Changelog (#7976)
by @johannkmfix reexecution (#7980)
by @clairelin135Automation: versioned docs for 0.14.16
by @elementl-devtools[dagit] Fix SearchBootstrapQuery to be lazy (#7981)
by @hellendagsplit graphql query for the backfill page (#7986)
by @prhaRemove can_terminate from run launchers and run coordinators, use a status check instead in graphql (#7983)
by @gibsondanstyle(helm): remove unnecessary pylint suppressions (#7923)
by @rexledesmaprovide job_timeout_in_seconds in dataproc resource config (#7941)
by @3cham[dagit] Replace opName with opNames everywhere (#7977)
by @hellendagSet default hour of day and minute of hour in build_schedule_from_partitioned_job based on the offsets of the parittioned job (#7954)
by @gibsondanWIP: Don't fetch every single run in the InstanceBackfillsQuery (#7985)
by @gibsondan[dagit] Show assets on run pages in more scenarios (#7874)
by @bengotow[dagit] Do not display the partition_set, step_selection tags on asset group runs (#7953)
by @bengotow[dagit] WebSocketProvider: Time out and fall back to disabled (#7978)
by @hellendagCoerce asset_key on AssetIn (#8009)
by @aroigFix issue where we added the run ID repeatedly to the runs for the tick during scheduler failure recovery (#8003)
by @gibsondanLoad fewer backfills on each page (#7998)
by @gibsondanadd loading indicator for job backfills (#8004)
by @prhaRemove duplicate count_resume_run_attempts method (#7915)
by @johannkmMake asset related checks more robust (#8008)
by @aroigFix relative path for pytest fixture in backcompat tests (#7999)
by @dpeng817feat(helm): allow name override for Dagit deployment (#8005)
by @rexledesmaAdd a __main__.py to dagster/daemon (#8012)
by @gibsondandagster-buildkite refactor (#7813)
by @smackeseyFix unsightly scrollbar in custom alert (#8018)
by @gibsondancreate get_records_for_run for event log storage, using an opaque string cursor (#7973)
by @prhaFix error when dagit passes in a non-dict string to run config valiation (#8020)
by @gibsondan[dagster-io/ui] Fix Dialog header/footer alignment (#7975)
by @hellendag[dagit] Disable unloadables if no permissions (#8021)
by @hellendaguse mkdir -p in migration guide
by @johannkm[dagit] Sectioned left nav (#8017)
by @hellendagHandle InvalidSubsetError in launch run results (#8014)
by @gibsondanUse dagster-postgres in alembic migration guide (#7992)
by @johannkmGenerate buildkite headers inside dagster-buildkite (#7987)
by @smackeseybug fix: should allow re-executing all dynamic steps generated from mapper step (#7979)
by @yuhanrefactor: has_index for migrations (#8034)
by @johannkmIn-progress indicator for graph-backed assets (#8015)
by @clairelin135resource defs on asset def (#7918)
by @dpeng817Load all code artifacts on grpc server startup, not just pipelines/jobs (#8040)
by @gibsondanAccept string values for non_argument_deps (#8023)
by @shalabhcadd example Python incompatibilities on Py 3.6 (#8036)
by @smackeseyAdd missing deps for docs-snippets and skip 36 in BK (#8048)
by @smackeseydocs: update readme with social links (#8046)
by @rexledesmaOpExecutionContext.output_asset_key (#7961)
by @sryzaAdd missing method to SnowflakePandasTypeHandler (#8051)
by @gibsondan[dagit] Flip section arrow in left nav (#8056)
by @hellendag[dagster-io/ui] Make disabled+checked Switch state more obvious (#8033)
by @hellendag[dagit] Enable sectioned left nav by default (#8039)
by @hellendag[3/n] Subsetting Stack: dbt assets can be subset (#7798)
by @OwenKephartRaise error upon incomplete graph-backed asset subset (#8041)
by @clairelin135throw error for empty asset key (#8069)
by @smackesey[dagit] Use dialog for multiple schedules/sensors in left nav (#8065)
by @hellendagremove dynamic mapping and collect section from jobs concepts page (#7844)
by @sryzagrammar (#7984)
by @dwinstonPass in engine rather than connection in SnowflakePandasTypeHandler (#8070)
by @gibsondan[dagit] Launchpad: Show disabled "Scaffold" and "Remove" buttons instead of hiding them (#8066)
by @hellendagchore(buildkite): mention user in private channel (#8071)
by @rexledesma[dagster-dbt] allow for static manifest.json-based selection (#8047)
by @OwenKephart[asset-resources 2/n][rfc] io manager defs directly on asset defs (#7920)
by @dpeng817adding a FromSourceAsset StepInputSource (#7942)
by @OwenKephartAdd protobuf pin to dagster (#8078)
by @gibsondanSkip flaky snowflake+pandas tests (#8081)
by @gibsondandocs: revamp the README (#8052)
by @rexledesma0.14.17 changelog (#8083)
by @smackeseyFix upstream context handling in fs_asset_io_manager (#8007)
by @aroigKnownState.ready_outputs (#8016)
by @alangenfeldAutomation: versioned docs for 0.14.17
by @elementl-devtoolschore: mark FileCache for deprecation in 0.15.0 (#7922)
by @rexledesmaremove step stats query (#8089)
by @clairelin135Revert "throw error for empty asset key (#8069)" (#8093)
by @gibsondanmigrate from deprecated sqlalchemy methods (#7864)
by @alangenfeldfix typos (#8090)
by @OwenKephartAdd skip for flaky grpc server test (#8097)
by @gibsondanchore(buildkite): ignore notifications on canceled builds (#8099)
by @rexledesma[dagit] Break apart LaunchpadSessionContainer (#8101)
by @hellendag[dagit] Small changes to the asset catalog (#7993)
by @bengotow[dagit] Track partition set sort order for Launchpad (#8104)
by @hellendagLint rule to make sure queries requiring QueryVariables have them (#8102)
by @salazarmMake dagster-images build work when it is called outside of a git repo (#8088)
by @gibsondanFix configuration schema for k8s executor (#8107)
by @fahadkh[dagit] Clean up some yarn peer deps (#8108)
by @hellendagAdd latestRun resolver to AssetsLatestInfo (#8072)
by @clairelin135Handle op outputs in default asset IO manager (#8074)
by @clairelin135[dagit] Add last materialization, latest run columns to the asset table (#7996)
by @bengotowfix(mypy): refine types (#8129)
by @rexledesma[dagit] Remove the global asset graph in preparation for asset group graphs (#8125)
by @bengotow[dagit] Rename isAssetGroup => isHiddenAssetGroupJob for clarity (#8124)
by @bengotowfix(helm): allow numeric quoted strings as image tags (#8120)
by @rexledesmadocs: add a button to copy/paste code snippets (#8106)
by @yuhan[asset-resources 4/n][rfc] Refactor resource requirement checking code (#7947)
by @dpeng817fix(helm): use templated comment (#8137)
by @rexledesmadocs(helm): add doc hint for dagster-user-deployments.imagePullSecrets (#8112)
by @ceefourAdded group_name to asset (#8110)
by @shalabhcFix invocation on ops that use generic dynamic outputs (#8133)
by @dpeng817[dagit] Show more information for last run on Schedules/Sensors (#8130)
by @hellendag[dagit] Remove asset graph bundling based on path prefixes, experimental flag (#8127)
by @bengotowKnownState parent run info (#8030)
by @alangenfeldadd metadata to jobs (#7849)
by @jamiedemariafix dagit-debug (#8148)
by @alangenfeldadd kwargs to pipeline snapshot from storage (#8149)
by @jamiedemariaFix label sanitization for strings that end in a period (#8151)
by @gibsondan[dagit] Show counts next to log filter tags (#8141)
by @hellendagincrease test_ping timeout (#8150)
by @alangenfeld[dagit] Fix duplicate styled-components (#8144)
by @hellendagBasic asset config (#7590)
by @smackeseyAvoid launching ECS runs with large overrides (#8152)
by @jmsandersOptionally tag images as latest (#8132)
by @jmsanderschore(helm): christen the code server service port name as grpc (#8142)
by @calebfornarifeat(helm): allow the postgresql scheme to be configurable (#8126)
by @peay[dagit] Remove clear-site-data header (#8134)
by @hellendag[dagit] Allow run termination on queued run (#8157)
by @hellendag[dagit] Add active state to left nav items (#8147)
by @hellendag[asset-resources 5/n] io manager defs on source assets (#8105)
by @dpeng817[dagster-io/eslint-config] v1.0.5 (#8121)
by @hellendagfix(helm): use pre-2022 bitnami repository (#8166)
by @rexledesmaAsset config gql resolver (#8163)
by @smackeseyAllow environment variables in
dagster-mlflow schema (#7997)
by @chasleslrchanges (#8169)
by @OwenKephart[dagit] asset Config (#8154)
by @smackeseyclean up create_pg_connection (#8165)
by @alangenfeld[dagit] Add a Lineage tab to the Asset Details page (#8143)
by @bengotow[easy] 0.14.18 => 0.14.19 in changelog (#8180)
by @gibsondanrevert "Scaffold Config" always enabled (#8181)
by @smackeseyAdd asset groups to graphql (#8140)
by @shalabhc[0.15.0] remove attempt count from step launching APIs (#8068)
by @alangenfeld[dagit] Make the left nav stay open/closed (#8173)
by @hellendagstep launcher fix up (#8186)
by @alangenfeldExplicit asset key args to assetsLatestInfo (#8178)
by @clairelin135Improve documentation around Output objects and op output annotations. (#8139)
by @dpeng817Automation: versioned docs for 0.14.19
by @elementl-devtoolsRevert "Automation: versioned docs for 0.14.19" (#8190)
by @gibsondanAutomation: versioned docs for 0.14.19
by @elementl-devtools[docs] - RFC: add a Learning More section to the asset tutorial [CON-17] (#8031)
by @sryzaMerge Asset IO functionality with Library IO managers (#8189)
by @clairelin135let tag inherit pointer style (#8208)
by @salazarmextract event connection to top-level graphql query (#8077)
by @prhaFixes for asset graphql tests to make them easier to generalize and call with other graphql context fixtures (#8199)
by @gibsondan[instance] rm events_for_asset_key and get_asset_events (#7602)
by @alangenfeldRestrict group names to VALID_NAME_REGEX_STR (#8214)
by @shalabhc[asset-resources 6/n] with_resources method (#8019)
by @dpeng817fix encoding for windows test (#8227)
by @prhaMigration: add columns action_type and selector_id to bulk_actions (#7995)
by @johannkmCreate unified storage configuration for
dagster.yaml (#7283)
by @prhaAdd image pull secrets to k8s container context (#8221)
by @gibsondan[dagit] Update MenuLink to support a disabled state (#8200)
by @bengotow[docs] - Document partitioned IO managers [CON-37] (#8191)
by @smackeseyungate new partitions backfill (#8224)
by @prha[dagit] Add top level Asset Group pages, Asset Groups in left nav (#8203)
by @bengotowFix upgrade story for new scheme field in postgres (#8220)
by @gibsondanfix workspace.yaml formatting (#8182)
by @smackeseydocs: improve search 1/ (#8229)
by @yuhan[dagit] Add asset group filter to the Asset Catalog (#8204)
by @bengotow methods [1/2] (#8225)
by @clairelin135Graph-backed asset IO Fix (#8171)
by @clairelin135bring back source asset metadata (#8195)
by @sryza[dagit] Asset graph GraphQL query audit + cleanup (#8205)
by @bengotow[dagit] Remove flat left nav (#8231)
by @hellendag[assets] cycle resolution (#8222)
by @OwenKephart[docs] add multi_assets docs page [CON-33] (#8192)
by @OwenKephartAssets now have a default group name (#8226)
by @shalabhcMove config editor to core so we can reuse in admin portal (#8237)
by @salazarmRenamed the prefix for auto created jobs from '__ASSET_GROUP' to '__ASSET_JOB'. (#8235)
by @shalabhcDerive origin from pipeline run instead of the arg to ExecuteRunArgs (#8156)
by @gibsondanunique name for assets in adls2 tests (#8232)
by @jamiedemariadocs: improve search 2/ (#8244)
by @yuhanAssetSelection (#8202)
by @smackesey[for 0.15.0] default includeConfigInLaunchedRuns in helm chart to true (#7488)
by @gibsondan[dagit] Don't block scroll when mouse hits custom tooltip (#8242)
by @hellendag[dagit] Switch asset group pages to /list and /lineage instead of ?tab= (#8250)
by @bengotow[docs] Asset Job Schedules/Sensors [CON-28, CON-29] (#8155)
by @clairelin135[docs] - Document asset metadata [CON-31] (#8084)
by @erinkcochran87[docs] - Upstream changes for assets [CON-36] (#8175)
by @erinkcochran87add data migration to bulk actions table for backfill jobs (#8153)
by @prha[docs] - Asset config [CON-89] (#8119)
by @erinkcochran87[docs] - Update Ops & Asset pages for release [CON-21] (#8158)
by @erinkcochran87Override resource defs when invoking assets (#8217)
by @dpeng817Add resource defs to source asset, handle transitive dependencies (#8223)
by @dpeng817asset defs directly on repository (#8197)
by @sryza[docs] - Graph-backed assets [CON-34] (#8174)
by @erinkcochran87 methods [2/2] (#8236)
by @clairelin135[assets] UnresolvedAssetJobDefinition (#8207)
by @OwenKephart[dagster-dbt] update dbt keys (#8228)
by @OwenKephartwith_resources top level export (#8264)
by @dpeng817chore(buildkite): separate docs only changes (#8261)
by @rexledesma[docs] Asset storage description in filesystem IO Manager docs (#8240)
by @clairelin135[dagit] Add Materialize button to the Asset Catalog (#8206)
by @bengotowAdded the default group name for assets defined using Out(...) (#8259)
by @shalabhc[assets] Allow graph_name to be None when not specified (#8247)
by @bengotow[dagit] Asset group styling adjustments, polish (#8246)
by @bengotowadd backfill blank state (#8267)
by @prhaFix descriptor that only references solid (#8159)
by @dpeng817add an "environment" key to EcsRunLauncher / EcsContainerContext that sets env vars (#8243)
by @gibsondan[dagit] Add a Materialize All button to Asset Details > Lineage (#8248)
by @bengotow[assets] Update modern-data-stack assets (#8271)
by @OwenKephart[assets] fix tests (#8275)
by @OwenKephartdocs: improve search 3/ handle GH discussion entries in search results (#8252)
by @yuhan[docs] - Cross-repository assets [CON-32] (#8075)
by @erinkcochran87Fix hackernews after change to AssetGroup behavior (#8269)
by @dpeng817[docs] - Document non-argument deps for assets [CON-16] (#7962)
by @erinkcochran87assets_from_package_module -> load_assets_from_package_module (#8280)
by @dpeng817validate resources for assets passed directly to a repository (#8270)
by @dpeng817Standalone materialize method (#8268)
by @dpeng817[0.15.0] move from warnings to errors for sensor/schedule target duplication (#7861)
by @dpeng817refactor: remove FileCache (#7701)
by @rexledesma[assets] add partition_def and config to define_asset_job (#8282)
by @OwenKephartAdd docs for top level job inputs (#8212)
by @dpeng817default executor on repo (#8272)
by @dpeng817Deprecate asset namespace (#8274)
by @smackeseyFix black (#8288)
by @johannkmAdd kvs table to oss (#8213)
by @johannkm[dagit] Hold the lineage graph zoom level constant as you navigate (#8251)
by @bengotowAdd max_retries to user editable tags (#8285)
by @johannkmfix is_subclass bug where issubclass(list[str], DagsterType) throws a surprising exception (#8287)
by @gibsondantake experimental decorators off of asset APIs (#8260)
by @sryzafix group_name in to_source_assets (#8279)
by @sryzaKVS storage methods (#8249)
by @johannkm[dagster-dbt] Fix bug where a dbt invocation that did not successfully start could emit materialization events. (#8293)
by @OwenKephartFix typo in multi_asset docstring (#8292)
by @johannkmMake dagster/reexecution_strategy tag editable (#8286)
by @johannkmasset selection tweaks (#8290)
by @smackeseyRevert "KVS storage methods (#8249)"
by @johannkmonly return config field for solids (#8278)
by @clairelin135docs: fix white box flashing when navigating between pages (#8281)
by @yuhan[dagit] Fix JS error on repo reload (#8297)
by @hellendag[docs] - Move assets into intro tutorial + up in sidenav [CON-18] (#8241)
by @erinkcochran87un-asset-group-ify SDA guide (#8283)
by @dpeng817asset concepts page without AssetGroups (#7901)
by @sryza0.14.20 Changelog (#8301)
by @johannkmsolid -> op in message for omitted outputs (#7903)
by @sryza[docs] - Resources in assets [CON-27] (#8168)
by @erinkcochran87[docs] - SDA guide for existing Dagster users [CON-30] (#8188)
by @sryzafix subclass test on py36 (#8302)
by @gibsondan[dagit] Repair issue where last ten runs are missing for job (#8305)
by @hellendagDeprecate AssetGroup (#8276)
by @smackeseyremove experimental materialization tags (#6650)
by @prhaAutomation: versioned docs for 0.14.20
by @elementl-devtools[docs] - Remove advanced tutorials [CON-290] (#8298)
by @erinkcochran87RFC: asset tutorial without AssetGroups (#7909)
by @sryza[dagit] Don’t let description overflow on asset details page (#8309)
by @bengotow[dagit] Small left-nav design adjustments (#8307)
by @bengotow[dagit] Make the warning icon optional in AppTopNav (#8294)
by @hellendagAdd docker executor example to deploy_docker (#8219)
by @gibsondanAssetSelection.assets (#8316)
by @sryzaupload backcompat artifacts to bk (#8184)
by @jamiedemaria[dagster-io/ui] Expose some text input styles (#8318)
by @hellendagoutput_asset_partition_key -> asset_partition_key_for_output (#8327)
by @sryza[dagit] Fix text overflow behavior in the left nav, keep open if one repo (#8324)
by @bengotow[dagit] Run timeline: make popover scrollable (#8319)
by @hellendagdocs: make sidenav sticky [DREL-329] (#8310)
by @yuhanuse DisplayableEvent in dagit log viewer (#8323)
by @gibsondanuniform prefix parsing (#8332)
by @OwenKephartdocs: fix link from Kubernetes example (#8335)
by @rexledesma[dagster-dbt] group names on dbt assets (#8303)
by @OwenKephartAttempt to speed up query for existing run keys for a given sensor (#8329)
by @gibsondan[docs] - Getting Started/quick start updates [CON-39] (#8187)
by @erinkcochran87in load_assets_from_... functions, make key_prefix docstring consistent with group (#8333)
by @sryzareorder concept super-sections in left nav (#8315)
by @sryzadocs: improve search 4/ house cleaning - remove unused config/cmd (#8273)
by @yuhan[dagit] Repair font-sizes on Safari (#8351)
by @hellendag[assets] fix issues with job selection (#8340)
by @OwenKephartAssetsDefinition.asset_keys -> keys and similar (#8325)
by @sryzarequire get_event_records to have filter arg (#8284)
by @prhaRevert "Revert "KVS storage methods (#8249)"" (#8296)
by @johannkmAllow number type for id in ButtonGroup (#8352)
by @salazarm[dagit] Resolve redundant tag buttons on Launchpad (#8353)
by @hellendagtest(graphql): ensure type names do not leak graphene (#8345)
by @rexledesmadocs(graphql): ensure descriptions for all mutation fields (#8346)
by @rexledesmadocs(graphql): ensure descriptions for all query fields (#8347)
by @rexledesmadocs(graphql): ensure descriptions for all subscription fields (#8348)
by @rexledesmadocs(graphql): ensure descriptions for all types with opt-in enforcement (#8349)
by @rexledesmafix 0.14.20 versioned_navigation entry (#8360)
by @yuhanEvent log methods for event log consumer (#8253)
by @johannkm[docs] - Clarify explanation of graph-backed assets (#8328)
by @sryzaAssetOut (#8359)
by @sryza[docs] - Update IO Manager docs to contain Asset IO management (#8337)
by @clairelin135Auto run reexecution daemon for oss (#8254)
by @johannkmToggle to enable auto run reexecution daemon (#8277)
by @johannkm[with_resources changes 1/n] with_resources docstring, config argument (#8322)
by @dpeng817Update k8s docs to reflect new includeConfigInLaunchedRuns default (#8361)
by @gibsondanSimplify execute_in_process result (#8365)
by @gibsondan[docs] - Graph-backed asset examples (#8339)
by @clairelin135Run retries helm config (#8369)
by @johannkm[with-resources changes 2/n] If resources collide when using with_resources, error. (#8330)
by @dpeng817AssetsDefinition property name changes (#8317)
by @sryzapass on partitions page for assets (#8355)
by @sryzaType annotations for IOManager (#8308)
by @smackeseymaterialize_in_process method (#8364)
by @dpeng817[assets] assorted bugfixes (#8372)
by @OwenKephart[docs] - Update repository concept doc with asset info (#8373)
by @smackeseyasset apiref improvements (#8374)
by @sryzaadd custom k8s labels to dagster k8s jobs, not just dagster k8s pods (#8381)
by @gibsondanasset apiref improvements (#8374)
by @sryzaMerge branch 'master' of https://github.com/dagster-io/dagster into release-0.15.0
by @clairelin135[dagit] Update the linking between asset groups + jobs (#8377)
by @bengotow[docs] - Asset grouping [CON-295] (#8375)
by @erinkcochran87[docs] - New Jobs & Graphs structure [CON-289] (#8035)
by @erinkcochran87fix docs lint (#8387)
by @gibsondanPatch op definition resolution for assets in GQL (#8384)
by @smackeseyRun retries docs (#8367)
by @johannkmMake asset invocation error if resources conflict (#8390)
by @dpeng817materialize_in_process -> materialize_to_memory (#8391)
by @dpeng817remove old asset lineage from docs (#8382)
by @sryzaFix problems from "remove old asset lineage from docs (#8382)" (#8389)
by @sryzaapidoc for AssetOut and AssetIn (#8388)
by @sryza0.15.0
by @elementl-devtoolsPublished by gibsondan over 2 years ago
env_vars
field to the EcsRunLauncher that allows you to configure environment variables in the ECS task for launched runs.env_vars
field on K8sRunLauncher
and k8s_job_executor
can now except input of the form ENV_VAR_NAME=ENV_VAR_VALUE, and will set the value of ENV_VAR_NAME to ENV_VAR_VALUE. Previously, it only accepted input of the form ENV_VAR_NAME, and the environment variable had to be available in the pod launching the job.IOManager.load_input
was called to load an asset that was not being materialized as part of the run, the provided context would not include the metadata for that asset. context.upstream_output.metadata
now correctly returns the metadata on the upstream asset.list[str]
) as the type of an input would raise an exception.https://github.com/dagster-io/dagster/compare/0.14.19...0.14.20
0.14.20rc0
by @elementl-devtoolsFixes for asset graphql tests to make them easier to generalize and call with other graphql context fixtures (#8199)
by @gibsondanbring back source asset metadata (#8195)
by @sryzafix is_subclass bug where issubclass(list[str], DagsterType) throws a surprising exception (#8287)
by @gibsondanAdd max_retries to user editable tags (#8285)
by @johannkmDerive origin from pipeline run instead of the arg to ExecuteRunArgs (#8156)
by @gibsondanadd an "environment" key to EcsRunLauncher / EcsContainerContext that sets env vars (#8243)
by @gibsondanFix upgrade story for new scheme field in postgres (#8220)
by @gibsondanAdd image pull secrets to k8s container context (#8221)
by @gibsondanextract event connection to top-level graphql query (#8077)
by @prhaCreate unified storage configuration for
dagster.yaml (#7283)
by @prha[dagster-dbt] Fix bug where a dbt invocation that did not successfully start could emit materialization events. (#8293)
by @OwenKephartMove config editor to core so we can reuse in admin portal (#8237)
by @salazarmMake dagster/reexecution_strategy tag editable (#8286)
by @johannkmonly return config field for solids (#8278)
by @clairelin135clean up create_pg_connection (#8165)
by @alangenfeld0.14.20 Changelog (#8301)
by @johannkm0.14.20
by @elementl-devtoolsPublished by gibsondan over 2 years ago
metadata
parameter) and viewed in dagit. You can use it to track code owners, link to docs, or add other useful information.config_schema
. If you attempt to materialize an asset with a config schema in Dagit, you'll be able to enter the required config via a modal.k8s_job_executor
where ops with long names sometimes failed to create a pod due to a validation error with the label names automatically generated by Dagster.fs_asset_io_manager
has been removed in favor of merging its functionality with fs_io_manager
. fs_io_manager
is now the default IO manager for asset jobs, and will store asset outputs in a directory named with the asset key.k8s_job_executor
’s max_conccurent
configuration. Thanks @fahadkh!fs_io_manager
to incorrectly handle assets associated with upstream assets. Thanks @aroig!https://github.com/dagster-io/dagster/compare/0.14.17...0.14.19
Automation: versioned docs for 0.14.17
by @elementl-devtoolsRevert "throw error for empty asset key (#8069)" (#8093)
by @gibsondan[dagit] Launchpad: Show disabled "Scaffold" and "Remove" buttons instead of hiding them (#8066)
by @hellendagchore(buildkite): mention user in private channel (#8071)
by @rexledesma[asset-resources 2/n][rfc] io manager defs directly on asset defs (#7920)
by @dpeng817docs: revamp the README (#8052)
by @rexledesma0.14.17 changelog (#8083)
by @smackeseyFix upstream context handling in fs_asset_io_manager (#8007)
by @aroigKnownState.ready_outputs (#8016)
by @alangenfeldchore: mark FileCache for deprecation in 0.15.0 (#7922)
by @rexledesmaremove step stats query (#8089)
by @clairelin135migrate from deprecated sqlalchemy methods (#7864)
by @alangenfeldfix typos (#8090)
by @OwenKephartAdd skip for flaky grpc server test (#8097)
by @gibsondanchore(buildkite): ignore notifications on canceled builds (#8099)
by @rexledesma[dagit] Break apart LaunchpadSessionContainer (#8101)
by @hellendag[dagit] Small changes to the asset catalog (#7993)
by @bengotow[dagit] Track partition set sort order for Launchpad (#8104)
by @hellendagLint rule to make sure queries requiring QueryVariables have them (#8102)
by @salazarmMake dagster-images build work when it is called outside of a git repo (#8088)
by @gibsondanFix configuration schema for k8s executor (#8107)
by @fahadkh[dagit] Clean up some yarn peer deps (#8108)
by @hellendagAdd latestRun resolver to AssetsLatestInfo (#8072)
by @clairelin135Handle op outputs in default asset IO manager (#8074)
by @clairelin135[dagit] Add last materialization, latest run columns to the asset table (#7996)
by @bengotowfix(mypy): refine types (#8129)
by @rexledesmafix(helm): allow numeric quoted strings as image tags (#8120)
by @rexledesmadocs: add a button to copy/paste code snippets (#8106)
by @yuhan[asset-resources 4/n][rfc] Refactor resource requirement checking code (#7947)
by @dpeng817fix(helm): use templated comment (#8137)
by @rexledesmadocs(helm): add doc hint for dagster-user-deployments.imagePullSecrets (#8112)
by @ceefourAdded group_name to asset (#8110)
by @shalabhcFix invocation on ops that use generic dynamic outputs (#8133)
by @dpeng817[dagit] Show more information for last run on Schedules/Sensors (#8130)
by @hellendagKnownState parent run info (#8030)
by @alangenfeldadd metadata to jobs (#7849)
by @jamiedemariafix dagit-debug (#8148)
by @alangenfeldadd kwargs to pipeline snapshot from storage (#8149)
by @jamiedemariaFix label sanitization for strings that end in a period (#8151)
by @gibsondan[dagit] Fix duplicate styled-components (#8144)
by @hellendagBasic asset config (#7590)
by @smackeseychore(helm): christen the code server service port name as grpc (#8142)
by @calebfornarifeat(helm): allow the postgresql scheme to be configurable (#8126)
by @peayAvoid launching ECS runs with large overrides (#8152)
by @jmsanders[dagit] Remove clear-site-data header (#8134)
by @hellendagfix(helm): use pre-2022 bitnami repository (#8166)
by @rexledesmaAsset config gql resolver (#8163)
by @smackeseyMerge branch 'release-0.14.18' of https://github.com/dagster-io/dagster into release-0.14.18
by @OwenKephartchanges (#8169)
by @OwenKephart[dagit] asset Config (#8154)
by @smackesey0.14.18
by @elementl-devtools0.14.18rc5
by @elementl-devtools0.14.18rc6
by @elementl-devtools[easy] 0.14.18 => 0.14.19 in changelog (#8180)
by @gibsondan0.14.19
by @elementl-devtools