dagster

Bugfixes

• [dagster-webserver] Fixed an issue that broke loading static files on Windows.

Bugfixes

• Fixed an issue that caused some missing icons in the UI.

New

• A new @partitioned_config decorator has been added for defined configuration for partitioned jobs. Thanks @danielgafni!
• [dagster-aws] The ConfigurablePickledObjectS3IOManager has been renamed S3PickleIOManager for simplicity. The ConfigurablePickledObjecS3IOManager will continue to be available but is considered deprecated in favor of S3PickleIOManager. There is no change in the functionality of the I/O manager.
• [dagster-azure] The ConfigurablePickledObjectADLS2IOManager has been renamed ADLS2PickleIOManager for simplicity. The ConfigurablePickledObjectADLS2IOManager will continue to be available but is considered deprecated in favor of ADLS2PickleIOManager. There is no change in the functionality of the I/O manager.
• [dagster-dbt] When an exception is raised when invoking a dbt command using DbtCliResource, the exception message now includes a link to the dbt.log produced. This log file can be inspected for debugging.
• [dagster-gcp] The ConfigurablePickledObjectGCSIOManager has been renamed GCSPickleIOManager for simplicity. The ConfigurablePickledObjecGCSIOManager will continue to be available but is considered deprecated in favor of GCSPickleIOManager. There is no change in the functionality of the I/O manager.

Bugfixes

• Fixed a bug that caused a DagsterInvariantViolationError when executing a multi-asset where both assets have self-dependencies on earlier partitions.
• Fixed an asset backfill issue where some runs continue to be submitted after a backfill is requested for cancellation.
• [dagster-dbt] Fixed an issue where using the --debug flag raised an exception in the Dagster framework.
• [ui] “Launched run” and “Launched backfill” toasts in the Dagster UI behave the same way. To open in a new tab, hold the cmd/ctrl key when clicking “View”
• [ui] When opening step compute logs, the view defaults to stderr which aligns with Python’s logging defaults.
• [ui] When viewing a global asset graph with more than 100 assets, the “choose a subset to display” prompt is correctly aligned to the query input.

Community Contributions

• Fix for loading assets with a BackfillPolicy, thanks @ruizh22!

Experimental

• [dagster-graphql] The Dagster GraphQL Python client now includes a default timeout of 300 seconds for each query, to ensure that GraphQL requests don’t hang and never return a response. If you are running a query that is expected to take longer than 300 seconds, you can set the timeout argument when constructing a DagsterGraphQLClient.
• [ui] We are continuing to improve the new horizontal rendering of the asset graph, which you can enable in Settings. This release increases spacing between nodes and improves the traceability of arrows on the graph.

Documentation

• Several Pythonic resources and I/O managers now have API docs entries.
• Updated the tutorial’s example project and content to be more explicit about resources.
• [dagster-dbt] Added API docs examples for DbtCliResource and DbtCliResource.cli(...).
• Some code samples in API docs for InputContext and OutputContext have been fixed. Thanks @Sergey Mezentsev!

Dagster Cloud

• When setting up a new organization by importing a dbt project, using GitLab is now supported.

Experimental

• Added a respect_materialization_data_versions option to auto materialization. It can enabled in dagster.yaml with

  auto_materialize:
      respect_materialization_data_versions: True
  

  This flag may be changed or removed in the near future.

New

• ops or assets with multiple outputs that are all required and return type None/ Nothing will interpret an explicitly or implicitly returned value None to indicate that all outputs were successful.
• The skip_reason argument to the constructor of SensorResult now accepts a string in addition to a SkipReason.
• [dagster-k8s] Added a step_k8s_config field to k8s_job_executor that allows you to customize the raw Kubernetes config for each step in a job. See the docs for more information.
• [dagster-k8s] Launched run pods now have an additional code location label.
• [dagster-ui] The runs table now lets you toggle which tags are always visible.
• [dagster-dbt] dagster-dbt project scaffold now creates the scaffold in multiple files:
  • constants.py contains a reference to your manifest and dbt project directory
  • assets.py contains your initial dbt assets definitions
  • definitions.py contains the code to load your asset definitions into the Dagster UI
  • schedules.py contains an optional schedule to add for your dbt assets
• [dagster-dbt] Added new methods get_auto_materialize_policy and get_freshness_policy to DagsterDbtTranslator.
• [dagster-fivertran] Sync options can now be passed to load_assets_from_fivetran_instance.
• [dagster-wandb] W&B IO Manager now handles partitions natively. (Thanks @chrishiste!)

Bugfixes

• Previously, canceling large asset backfills would cause the daemon to time out and display a “not running” error. This has been fixed.
• [dagster-ssh] Previously the SSHResource would warn when allow_host_key_change was set. Now known hosts are always loaded from the system hosts file, and the allow_host_key_change parameter is ignored.
• Previously, when using AutoMaterializePolicies, partitioned assets downstream of partitioned observable source assets could be materialized before their parent partitions were observed. This has been fixed.

Documentation

• @graph_multi_asset now has an API docs entry.
• The GCSComputeLogManager example in the Dagster Instance reference is now correct.
• Several outdated K8s documentation links have been removed from the Customizing your Kubernetes deployment guide.
• Added callouts to the GitHub and GitLab Branch Deployment guides specifying that some steps are optional for Serverless users.
• The “Graphs” page under the “Concepts” section has been renamed to “Op Graphs” and moved inside under the “Ops” heading.
• [dagster-dbt] Added API examples for @dbt_assets for the following use-cases:
  • Running dbt commands with flags
  • Running dbt commands with --vars
  • Running multiple dbt commands
  • Retrieving dbt artifacts after running a dbt command
  • Invoking other Dagster resouces alongside dbt
  • Defining and accessing Dagster config alongside dbt

Dagster Cloud

• The viewer role now has permission to edit their own user tokens.

New

• @graph_asset now takes a config parameter equivalent to the parameter on @graph.
• Added an optional dynamic_partitions_store argument to DynamicPartitionsDefinition for multi-partition run properly with dynamic partitions (Thanks @elzzz!).
• [dagster-grpahql] Added partitionsByAssets`` to backfillParams`` for ranged partition backfill (Thanks @ruizh22!).
• [dagster-dbt] Support for dbt-core==1.6 has been added.
• [dagster-dbt] DbtCliResource now supports configuring profiles_dir.
• [dagster-k8s] Allow specifying restart_policy on k8s_job_op (Thanks @Taadas!).
• [dagster-snowflake] Added authenticator to SnowflakePandasIOManager, which allows specifying the authentication mechanism to use (Thanks @pengw0048!).

Bugfixes

• In some situations, multiple materializations of the same asset could be kicked off when using a lazy AutoMaterializePolicy with assets that had at least one source asset parent and at least one non-source asset parent. This has been fixed.
• After applying an eager AutoMaterializePolicy to a time-partitioned asset downstream of an unpartitioned asset, the latest partition would only ever be materialized a single time, rather than updating in response to any parent updates. This has been fixed.
• Fixed an issue that would cause the creation of a StaticPartitionsDefinition containing many thousands of partitions could take a significant amount of time.
• The run coordinator daemon now uses a fresh request context on each iteration, fixing an issue where stale grpc server references could be used in certain high volume conditions.
• Automatically generated data versions for partitioned assets now correctly reflect the data versions of upstream partitions. Previously, they were computed using the data versions from the most recent materializations of upstream assets regardless of partition.
• [dagster-airbyte] Previously, attempting to load assets from an Airbyte instance in which some of the tables had hyphens in their name would result in an error. This has been fixed.
• [dagster-dbt] Previously, attempting to load assets from a dbt project in which some of the models had hyphens in their name would result in an error. This has been fixed.
• [dagstermill] Fixed a bug where known state for executing dagstermill ops was not correctly passed in (Thanks @motuzov!).

Documentation

• Added the starter project’s template for Dagster University.
• Fixed an incorrect method name in DagsterDbtTranslator Docs (Thanks @akan72!).

Dagster Cloud

• When importing a dbt project on the Dagster Cloud setup page, an Unexpected exception error would be raised when scaffolding a pull request on a repository with no profiles.yml. This behavior has been updated to raise a more descriptive error message on the repo selection page.
• The running multiple agents guide has been revamped to discuss running agent replicas and zero-downtime deployment of the agent.
• The agentReplicas config setting on the helm chart has been renamed to isolatedAgents. In order to use this config setting, your user code dagster version needs to be 1.4.3 or greater.

New

• [ui] When viewing a run for auto-materialized assets, show a tag with information about the assets that were materialized.
• [ui] In the Auto-materialize History view, when one or more of an asset’s parents have been updated, the set of updated parents will be viewable.
• [ui] Link to the auto-materialized history for an asset from the asset DAG view.
• [ui] For runs that were the result of auto-observation, show a tag for this in the Runs list view.
• Added warnings for storage incompatibility with the experimental global op concurrency.

Bugfixes

• [dagster-dbt] Fixed an issue where dagster-dbt project scaffold didn’t create a project directory with all the scaffolded files.
• Fixed an issue which could cause errors when using the SpecificPartitionsPartitionMapping with auto-materialization.

Breaking Change

• Previously, it was possible to set max_materializations_per_minute on an AutoMaterializePolicy to a non-positive number. This will now result in an error.

Community Contributions

• Fix for loading multipartitions paths in upath_io_manager from @harrylojames; thank you!
• Docs typo fix from @C0DK; thank you!

Documentation

• Revamped the dagster-dbt tutorial to take advantage of dagster project scaffold and the new dagster-dbt APIs.

New

• [dagster-dbt] When invokingdagster-dbt project scaffold on a dbt project directory, if a profiles.yml exists in the root of the directory, its contents are used to add dbt adapter packages to the scaffolded setup.py.
• The default sentinel value for the multiprocessing executor’s max_concurrent field has been changed from 0 to None to more clearly signal its intent. A value of 0 is still interpreted as the sentinel value which dynamically allocates max_concurrent based on detected CPU count.

Bugfixes

• IO managers defined on jobs will now be properly merged with resources defined in Definitions, so that jobs are able to override the IO manager used.
• [dagster-fivetran] Fixed an issue where EnvVars in a FivetranResource would not be evaluated when loading assets from the Fivetran instance.
• [dagster-airbyte] Fixed an issue where EnvVars in an AirbyteResource would not be evaluated when loading assets from the Airbyte resource.

Documentation

• [dagster-dbt] Added API docs for DbtCliResource, DbtCliInvocation, @dbt_assets, DagsterDbtTranslator, dagster-dbt project scaffold
• [dagster-dbt] Expanded references for new APIs:
  • Added documentation to customize asset definition attributes for dbt assets
  • Added documentation to define upstream and downstream dependencies to dbt assets
  • Added documentation to define schedules for dbt assets

Bugfixes

• Fixes a bug in dagster-dbt that was preventing it from correctly materializing subselections of dbt asset.

Bugfixes

• Fixes a bug in dagster-dbt that was preventing it efficiently loading dbt projects from a manifest.

Major Changes since 1.3.0 (core) / 0.19.0 (libraries)

Core

• Auto-materialize history – We’ve added a UI that tracks why assets were or were not materialized according to theirAutoMaterializePolicy. It’s located under Assets → Select an asset with an AutoMaterializePolicy → Auto-materialize history tab.
• Auto-materialize performance – We’ve made significant performance improvements to the Asset Daemon, allowing it to keep up with asset graphs containing thousands of assets and assets with a large history of previously-materialized partitions.
• Asset backfill cancellation — Asset backfills can now be canceled, bring them to parity with job backfills. When an asset backfill is requested for cancellation, the daemon cancels runs until all runs are terminated, then marks the backfill as “canceled”.
• non_argument_deps → deps – We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.
• Group-level asset status UI – the new Assets Overview dashboard, located underneath the Activity tab of the Overview page, shows the status all the assets in your deployment, rolled up by group.
• Op concurrency (experimental) — We’ve added a feature that allows limiting the number of concurrently executing ops across runs. [docs]
• DynamicPartitionsDefinition and SensorResult are no longer marked experimental.
• Automatically observe source assets, without defining jobs (experimental) – The @observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval. Downstream assets with eager auto-materialize policies will automatically run if the observation function indicates that the source asset has changed. [docs]
• Dagit → Dagster UI – To reduce the number of Dagster-specific terms that new users need to learn when learning Dagster, “Dagit” has been renamed to the “The Dagster UI”. The dagit package is deprecated in favor of the dagster-webserver package.
• Default config in the Launchpad - When you open the launchpad to kick off a job or asset materialization, Dagster will now automatically populate the default values for each field.

dagster-dbt

• The new @dbt_assets decorator allows much more control over how Dagster runs your dbt project. [docs]
• The new dagster-dbt project scaffold command line interface makes it easy to create files and directories for a Dagster project that wraps an existing dbt project.
• Improved APIs for defining asset dependencies – The new get_asset_key_for_model and get_asset_key_for_source utilities make it easy to specify dependencies between upstream dbt assets and downstream non-dbt assets. And you can now more easily specify dependencies between dbt models and upstream non-dbt assets by specifying Dagster asset keys in the dbt metadata for dbt sources.

Since 1.3.14 (core) / 0.19.14 (libraries)

New

• The published Dagster Docker images now use Python 3.10, instead of 3.7.
• We’ve deprecated the non_argument_deps parameter of @asset and @multi_asset in favor of a new deps parameter. The new parameter makes it clear that this is a first-class way of defining dependencies, makes code more concise, and accepts AssetsDefinition and SourceAsset objects, in addition to the strs and AssetKeys that the previous parameter accepted.
• The UPathIOManager can now be extended to load multiple partitions asynchronously (Thanks Daniel Gafni!).
• By default, Dagster will now automatically load default config values into the launchpad. This behavior can be disabled in the user settings page.
• [dagster-k8s] The Helm chart now sets readiness probes on user code deployment servers by default. These can be disabled with dagster-user-deployments.deployments.[...].readinessProbe.enabled=false.
• [dagster-airbyte] In line with the deprecation of non_argument_deps in favor of deps, build_airbyte_assets now accepts a deps parameter.
• [dagstermill] In line with the deprecation of non_argument_deps in favor of deps, define_dagstermill_asset now accepts a deps parameter.

Bugfixes

• Duplicate partition keys passed to StaticPartitionsDefinition will now raise an error.
• Fixed a bug that caused lazy AutoMaterializePolicy's to not materialize missing assets.
• [ui] Fixed an issue where global search and large DAGs were broken when using --path-prefix.
• Schedule and sensor run submissions are now kept up to date with the current workspace, fixing an issue where a stale reference to a server would be used in some conditions.

Breaking Changes

• Support for Python 3.7 has been dropped.
• build_asset_reconciliation_sensor (Experimental) has been removed. It was deprecated in 1.3 in favor of AutoMaterializePolicy.
• asset_key(s) properties on AssetIn and AssetDefinition have been removed in favor of key(s). These APIs were deprecated in 1.0.
• root_input_manager and RootInputManagerDefinition have been removed in favor of input_manager and InputManagerDefinition. These APIs were deprecated in 1.0.
• [dagster-pandas] The event_metadata_fn parameter on create_dagster_pandas_dataframe_type has been removed in favor of metadata_fn.
• [dagster-dbt] The library has been substantially revamped to support the new @dbt_assets and DbtCliResource. See the migration guide for details.
  • Group names for dbt assets are now taken from a dbt model's group. Before, group names were determined using the model's subdirectory path.
  • Support for dbt-rpc has been removed.
  • The class alias DbtCloudResourceV2 has been removed.
  • DbtCli has been renamed to DbtCliResource. Previously, DbtCliResource was a class alias for DbtCliClientResource.
  • load_assets_from_dbt_project and load_assets_from_dbt_manifest now default to use_build=True.
  • The default assignment of groups to dbt models loaded from load_assets_from_dbt_project and load_assets_from_dbt_manifest has changed. Rather than assigning a group name using the model’s subdirectory, a group name will be assigned using the dbt model’s dbt group.
  • The argument node_info_to_definition_metadata_fn for load_assets_from_dbt_project and load_assets_from_dbt_manifest now overrides metadata instead of adding to it.
  • The arguments for load_assets_from_dbt_project and load_assets_from_dbt_manifest now must be specified using keyword arguments.
  • When using the new DbtCliResource with load_assets_from_dbt_project and load_assets_from_dbt_manifest, stdout logs from the dbt process will now appear in the compute logs instead of the event logs.

Deprecations

• The dagit python package is deprecated and will be removed in 2.0 in favor of dagster-webserver. See the migration guide for details.
• The following fields containing “dagit” in the Dagster helm chart schema have been deprecated in favor of “dagsterWebserver” equivalents (see migration guide for details):
  • dagit → dagsterWebserver
  • ingress.dagit → ingress.dagsterWebserver
  • ingress.readOnlyDagit → ingress.readOnlyDagsterWebserver
• [Dagster Cloud ECS Agent] We've introduced performance improvements that rely on the AWS Resource Groups Tagging API. To enable, grant your agent's IAM policy permission to tag:DescribeResources. Without this policy, the ECS Agent will log a deprecation warning and fall back to its old behavior (listing all ECS services in the cluster and then listing each service's tags).
• DbtCliClientResource, dbt_cli_resource and DbtCliOutput are now being deprecated in favor of DbtCliResource.
• A number of arguments on load_assets_from_dbt_project and load_assets_from_dbt_manifest are now deprecated in favor of other options. See the migration for details.

Community Contributions

• Docs typo fix from @chodera, thank you!
• Run request docstring fix from @Jinior, thank you!

Documentation

• All public methods in the Dagster API now have docstrings.
• The entirety of the documentation has been updated to now refer to the “Dagster webserver” or “Dagster UI” where “Dagit” was previously used for both entities.

New

• DynamicPartitionsDefinition and SensorResult are no longer marked experimental
• DagsterInstance now has a get_status_by_partition method, which returns the status of each partition for a given asset. Thanks renzhe-brian!
• DagsterInstance now has a get_latest_materialization_code_versions method, which returns the code version of the latest materialization for each of the provided (non-partitioned) assets.
• The error message for when an asset illegally depends on itself is now more informative.
• Further performance improvements for the Asset Daemon.
• Performance improvements in the asset graph view for large asset graphs.
• Pandas 2.x is now supported in all dagster packages.
• build_asset_context has been added as an asset focused replacement for build_op_context.
• build_op_context now accepts a partition_key_range parameter.
• New AssetSelection.upstream_source_assets method allows selecting source assets upstream of the current selection.
• AssetSelection.key_prefixes and AssetSelection.groups now accept an optional include_sources parameter.
• The AutoMaterialize evaluations UI now provides more details about partitions and waiting on upstream assets.
• [dbt] The DbtCli resource is no longer marked experimental.
• [dbt] The global_config parameter of the DbtCli resource has been renamed to global_config_flags
• [dbt] load_assets_from_dbt_project and load_assets_from_dbt_manifest now work with the DbtCli resource.
• [dbt] The manifest argument of the @dbt_assets decorator now additionally can accept a Path argument representing a path to the manifest file or dictionary argument representing the raw manifest blob.
• [dbt] When invoking DbtCli.cli from inside a @dbt_assets-decorated function, you no longer need to supply the manifest argument as long as you provide the context argument.
• [dbt] The DbtManifest object can now generate schedules using dbt selection syntax.

dbt_manifest.build_schedule(
      job_name="materialize_dbt_models",
      cron_schedule="0 0 * * *",
      dbt_select="fqn:*"
  )

• [dbt] When invoking DbtCli.cli and the underlying command fails, an exception will now be raised. To suppress the exception, run the DbtCli.cli(..., raise_on_error=False).
• [ui] You can now alphabetically sort your partitions on the asset partitions page
• [ui] A button in the “Run is materializing this asset” and “Run failed to materialize this asset” banners provides direct access to the relevant run logs

Bugfixes

• Fixed a bug that caused asset metadata to not be available available on the OutputContext when using with_attributes or AssetsDefinition.from_graph.
• Previously, if a partitioned asset at the root of the graph had more missing partitions than its AutoMaterializePolicy’s max_materializations_per_minute parameter, those older partitions would not be properly discarded from consideration on subsequent ticks. This has been fixed.
• Fixed a bug that caused AutoMaterializePolicy.lazy() to not materialize missing assets that were downstream of assets without an AutoMaterializePolicy.
• In rare cases, the AssetDaemon could hit an exception when using a combination of freshness policies and observable source assets. This has been fixed.
• Previously, string type annotations (most commonly via modules containing from __future__ import annotations) would cause errors in most cases when used with Dagster definitions. This has been fixed for the vast majority of cases.
• AssetExecutionContext has returned to being a type alias for OpExecutionContext.
• [ui] Date filtering on the runs page now takes your timezone into consideration
• [ui] Fixed a bug where selecting partitions in the launchpad dialog cleared out your configuration
• [ui] In the run Gantt chart, executed steps that follow skipped steps no longer render off the far right of the visualization.
• [ui] Cancelling a running backfill no longer makes canceled partitions un-selectable on the job partitions page and backfill modal, and cancellation is shown in gray instead of red.

Breaking Changes

• [experimental] The internal time_window_partition_scope_minutes parameter of the AutoMaterializePolicy class has been removed. Instead, max_materializations_per_minute should be used to limit the number of runs that may be kicked off for a partitioned asset.

Deprecations

• [dbt] DbtCliResource has been deprecated in favor of DbtCli.
• The python package dagit has been deprecated in favor of a new package dagster-webserver.
• OpExecutionContext.asset_partition_key_range has been deprecated in favor of partition_key_range.

Community Contributions

• The databricks_pyspark_step_launcher will no longer error when executing steps that target a single partition of a DynamicPartitionsDefinition (thanks @weberdavid!).
• Increased timeout on readinessProbe for example user code images, which prevents breakages in certain scenarios (thanks @leehuwuj)!
• Avoid creation of erroneous local directories by GCS IO manager (thanks @peterjclaw)!
• Fixed typo in intro docs (thanks @adeboyed)!
• Fix typo in bigquery docs (thanks @nigelainscoe)!
• Fix typing on run tag validation (thanks @yuvalgimmunai)!
• Allow passing repositoryCredentials arn as config to ecs run launcher (thanks @armandobelardo)!

Experimental

• The @observable_source_asset decorator now accepts an auto_observe_interval_minutes parameter. If the asset daemon is turned on, then the observation function will automatically be run at this interval.
• [dbt] DbtCliTask has been renamed to DbtCliInvocation
• [dbt] The get_asset_key_by_output_name and get_node_info_by_output_name methods of DbtManifest have been renamed toget_asset_key_for_output_name and get_node_info_for_output_name, respectively.
• [ui] A new feature flag allows you to switch Asset DAG rendering to a tighter horizontal layout, which may be preferable in some scenarios

Documentation

• Many public methods that were missing in the API docs are now documented. Updated classes include DagsterInstance, *MetadataValue, DagsterType, and others.
• dagster-pandera now has an API docs page.
• Deprecated methods in the API docs now are marked with a special badge.

Bugfixes

• Fixes a bug in dagster project from-example that was preventing it from downloading examples correctly.

New

• The --name argument is now optional when running dagster project from-example.
• An asset key can now be directly specified via the asset decorator: @asset(key=...).
• AssetKey now has a with_prefix method.
• Significant performance improvements when using AutoMaterializePolicys with large numbers of partitions.
• dagster instance migrate now prints information about changes to the instance database schema.
• The dagster-cloud-agent helm chart now supports setting K8s labels on the agent deployment.
• [ui] Step compute logs are shown under “Last Materialization” in the asset sidebar.
• [ui] Truncated asset names now show a tooltip when hovered in the asset graph.
• [ui] The “Propagate changes” button has been removed and replaced with “Materialize Stale and Missing” (which was the “Propagate changes” predecessor).

Bugfixes

• [ui] Fixed an issue that prevented filtering by date on the job-specific runs tab.

• [ui] “F” key with modifiers (alt, ctrl, cmd, shift) no longer toggles the filter menu on pages that support filtering.

• [ui] Fix empty states on Runs table view for individual jobs, to provide links to materialize an asset or launch a run for the specific job, instead of linking to global pages.

• [ui] When a run is launched from the Launchpad editor while an editor hint popover is open, the popover remained on the page even after navigation. This has been fixed.

• [ui] Fixed an issue where clicking on the zoom controls on a DAG view would close the right detail panel for selected nodes.

• [ui] Fixed an issue shift-selecting assets with multi-component asset keys.

• [ui] Fixed an issue with the truncation of the asset stale causes popover.

• When using a TimeWindowPartitionMapping with a start_offset or end_offset specified, requesting the downstream partitions of a given upstream partition would yield incorrect results. This has been fixed.

• When using AutoMaterializePolicys with observable source assets, in rare cases, a second run could be launched in response to the same version being observed twice. This has been fixed.

• When passing in hook_defs to define_asset_job, if any of those hooks had required resource keys, a missing resource error would surface when the hook was executed. This has been fixed.

• Fixed a typo in a documentation URL in dagster-duckdb-polars tests. The URL now works correctly.

Experimental

• [dagster-dbt] Added methods to DbtManifest to fetch asset keys of sources and models: DbtManifest.get_asset_key_for_model, DbtManifest.get_asset_key_for_source. These methods are utilities for defining python assets as dependencies of dbt assets via @asset(key=manifest.get_asset_key_for_model(...).
• [dagster-dbt] The use of the state_path parameter with DbtManifestAssetSelection has been deprecated, and will be removed in the next minor release.
• Added experimental support for limiting global op/asset concurrency across runs.

Dependencies

• Upper bound on the grpcio package (for dagster) has been removed.

Breaking Changes

• Legacy methods of PartitionMapping have been removed. Defining custom partition mappings has been unsupported since 1.1.7.

Community Contributions

• [dagster-airbyte] Added the ability to specify asset groups to build_airbyte_assets. Thanks @guy-rvvup!

Documentation

• For Dagster Cloud Serverless users, we’ve added our static IP addresses to the Serverless docs.

New

• Assets with lazy auto-materialize policies are no longer auto-materialized if they are missing but don’t need to be materialized in order to help downstream assets meet their freshness policies.
• [ui] The descriptions of auto-materialize policies in the UI now include their skip conditions along with their materialization conditions.
• [dagster-dbt] Customized asset keys can now be specified for nodes in the dbt project, using meta.dagster.asset_key. This field takes in a list of strings that are used as the components of the generated AssetKey.

version: 2

models:
  - name: users
    config:
      meta:
        dagster:
          asset_key: ["my", "custom", "asset_key"]

• [dagster-dbt] Customized groups can now be specified for models in the dbt project, using meta.dagster.group. This field takes in a string that is used as the Dagster group for the generated software-defined asset corresponding to the dbt model.

version: 2

models:
  - name: users
    config:
      meta:
        dagster:
          group: "my_group"

Bugfixes

• Fixed an issue where the dagster-msteams and dagster-mlflow packages could be installed with incompatible versions of the dagster package due to a missing pin.
• Fixed an issue where the dagster-daemon run command sometimes kept code server subprocesses open longer than it needed to, making the process use more memory.
• Previously, when using @observable_source_assets with AutoMaterializePolicies, it was possible for downstream assets to get “stuck”, not getting materialized when other upstream assets changed, or for multiple down materializations to be kicked off in response to the same version being observed multiple times. This has been fixed.
• Fixed a case where the materialization count for partitioned assets could be wrong.
• Fixed an error which arose when trying to request resources within run failure sensors.
• [dagster-wandb] Fixed handling for multi-dimensional partitions. Thanks @chrishiste

Experimental

• [dagster-dbt] improvements to @dbt_assets
  • project_dir and target_path in DbtCliTask are converted from type str to type pathlib.Path.
  • In the case that dbt logs are not emitted as json, the log will still be redirected to be printed in the Dagster compute logs, under stdout.

Documentation

• Fixed a typo in dagster_aws S3 resources. Thanks @akan72
• Fixed a typo in link on the Dagster Instance page. Thanks @PeterJCLaw

New

• [dagster-dbt] By default, freshness policies and auto materialize policies on dbt assets can now be specified using the dagster field under +meta configuration. The following are equivalent:

Before:

version: 2

models:
  - name: users
    config:
      dagster_freshness_policy:
        maximum_lag_minutes: 60
        cron_schedule: '0 9 * * *'
      dagster_auto_materialize_policy:
        type: 'lazy'

After:

version: 2

models:
  - name: users
    config:
      meta:
        dagster:
          freshness_policy:
            maximum_lag_minutes: 60
            cron_schedule: '0 9 * * *'
          auto_materialize_policy:
            type: 'lazy'

• Added support for Pythonic Config classes to the @configured API, which makes reusing op and asset definitions easier:

  class GreetingConfig(Config):
      message: str
  
  @op
  def greeting_op(config: GreetingConfig):
      print(config.message)
  
  class HelloConfig(Config):
      name: str
  
  @configured(greeting_op)
  def hello_op(config: HelloConfig):
      return GreetingConfig(message=f"Hello, {config.name}!")
  

• Added AssetExecutionContext to replace OpExecutionContext as the context object passed in to @asset functions.

• TimeWindowPartitionMapping now contains an allow_nonexistent_upstream_partitions argument that, when set to True, allows a downstream partition subset to have nonexistent upstream parents.

• Unpinned the alembic dependency in the dagster package.

• [ui] A new “Assets” tab is available from the Overview page.

• [ui] The Backfills table now includes links to the assets that were targeted by the backfill.

Bugfixes

• Dagster is now compatible with a breaking change introduced in croniter==1.4.0. Users of earlier versions of Dagster can pin croniter<1.4.
• Fixed an issue introduced in 1.3.8 which prevented resources from being bound to sensors when the specified job required late-bound resources.
• Fixed an issue which prevented specifying resource requirements on a @run_failure_sensor.
• Fixed an issue where the asset reconciliation sensor failed with a “invalid upstream partitions” error when evaluating time partitions definitions with different start times.
• [dagster-k8s] Fixed an issue where annotations are not included in the Dagster Helm chart for the pod that is created when configuring the Helm chart to run database migrations.
• [ui] Fixed an issue with filtering runs by created date on the Runs page.
• [ui] The “upstream partitions missing” warning no longer appears in the asset backfill dialog if the upstream partitioned asset is a source asset.
• [dagster-dbt] Fixed an issue where asset dependencies for dbt models with ephemeral models in between them would sometimes be improperly rendered.

Community Contributions

• Added support for setting resources in asset and multi_asset sensors. Thanks @plaflamme!
• Fixed an issue where py.typed was missing in the dagster-graphql package. Thanks @Tanguy-LeFloch!

Experimental

• Evaluation history for AutoMaterializePolicys will now be cleared after 1 week.
• [dagster-dbt] Several improvements to @dbt_assets:
  • profile and target can now be customized on the DbtCli resource.
  • If a partial_parse.msgpack is detected in the target directory of your dbt project, it is now copied into the target directories created by DbtCli to take advantage of partial parsing.
  • The metadata of assets generated by @dbt_assets can now be customized by overriding DbtManifest.node_info_to_metadata.
  • Execution duration of dbt models is now added as default metadata to AssetMaterializations.

Documentation

• Added a new tutorial section about using resources.

Dagster Cloud

• Fixed an issue where overriding the container name of a code server pod using serverK8sConfig.containerConfig.name did not actually change the container name.

Dagster Cloud

• Fixed an issue in the 1.3.8 release where the Dagster Cloud agent would sometimes fail to start up with an import error.

New

• Multipartitioned assets with one time dimension can now depend on earlier partitions of themselves.
• define_asset_job now accepts a hooks argument.
• Added support for sqlalchemy==2.x
• [ui] The Runs page has been revamped with better filtering support
• [ui] The automaterialize policy page for SDA’s using the experimental AutomaterializePolicy feature now indicates time periods where no materializations happened due to no materialization conditions being met
• [dagster-k8s] The Dagster Helm chart now includes an additionalInstanceConfig key that allows you to supply additional configuration to the Dagster instance.
• [dagster-aws] The EcsRunLauncher now uses a different task definition family for each job, instead of registering a new task definition revision each time a different job is launched.
• [dagster-aws] The EcsRunLauncher now includes a run_ecs_tags config key that lets you configure tags on the launched ECS task for each run.

Bugfixes

• When a sensor had yield statement and also returned a SkipReason, the SkipReason would be ignored. This has been fixed.
• [dagster-cloud] Fixed a bug in the docker user code launcher that was preventing code location containers from being properly cleaned up.
• Fixed an issue where the Dagster UI would sometimes raise a ``RuntimeError: dictionary changed size during iteration` exception while code servers were being reloaded.
• Fixed an issue where the Dagster daemon reloaded your code server every 60 seconds when using the new experimental dagster code-server start CLI, instead of only reloading your code when you initiate a reload from the Dagster UI.
• Fixed a GraphQL error which would occur when loading the default config for jobs without config.
• [dagster-dbt] Fixed an error which would arise when trying to load assets from a DBT Cloud instance using the Pythonic-style resource.

Community Contributions

• Added the ability to specify metadata on asset jobs, by adding the metadata parameter to define_asset_job (Thanks Elliot2718!)
• [dagster-databricks] Connected databricks stdout to local stdout, to be handled by the compute log manager (Thanks loerinczy!)
• [dagster-census] Fixed poll_sync_run to handle the “preparing” status from the Census API (Thanks ldnicolasmay!)

Experimental

• @observable_source_asset-decorated functions can now return a DataVersionsByPartition to record versions for partitions.
• @dbt_assets
  • DbtCliTask's created by invoking DbtCli.cli(...) now have a method .is_successful(), which returns a boolean representing whether the underlying CLI process executed the dbt command successfully.
  • Descriptions of assets generated by @dbt_assets can now be customized by overriding DbtManifest.node_info_to_description.
  • IO Managers can now be configured on @dbt_assets.

Documentation

• New guide on using Dagster to manage machine learning pipelines

Dagster Cloud

• Added support for streaming upload of compute logs to Dagster Cloud
• The ECS agent now supports setting server_ecs_tags and run_ecs_tags that apply to each service or task created by the agent. See the docs for more information.
• Fixed run filtering for calls to instance.get_run_partition_data in Dagster Cloud.

New

• Adding a .env file in the working directory when running dagster dev can now be used for Dagster system variables like DAGSTER_HOME or environment variables referenced in your dagster.yaml file using an env: key. Previously, setting a .env file only worked for environment variables referenced in your Dagster code.
• When using the GraphQL Python client, submit_job_execution can now take in a RunConfig object. Previously, it could only take a Python dictionary with the run configuration.
• Asset backfills can now be canceled via a button in the UI. This will terminate unfinished runs and prevent new runs from being queued.
• Introduced a new user setting which allows automatically expanding default config for jobs in the launchpad.
• [dagit] Dagit now supports displaying a Polars tag on the asset graph.

Bugfixes

• Fixed an issue where setting a resource in an op didn’t work if the Dagster job was only referenced within a schedule or sensor and wasn’t included in the jobs argument to Definitions.
• [dagster-slack][dagster-pagerduty][dagster-msteams][dagster-airflow] Fixed issue where pre-built sensors and hooks which created urls to the runs page in the UI would use the old /instance/runs path instead of the new /runs.

Community Contributions

• [dagster-databricks] Added a configurable resource key to create_databricks_run_now_op, thanks @srggrs!

New

• Added an experimental dagster code-server start command that can be used to launch a code server, much like dagster api grpc. Unlike dagster api grpc, however, dagster code-server start runs the code in a subprocess, so it can reload code from the Dagster UI without needing to restart the command. This can be useful for jobs that load code from some external source and may want to reload job definitions without restarting the process.
• Added a new sensors.num_submit_workers key to dagster.yaml that can be used to decrease latency when a sensor emits multiple run requests within a single tick. See the docs for more information.
• [dagster-k8s] The k8s_job_executor can now be used to launch each step of a job in its own Kubernetes, pod, even if the Dagster deployment is not using the K8sRunLauncher to launch each run in its own Kubernetes pod.
• [ui] When viewing a list of schedules or sensors in Overview or on individual code locations, it is now possible to filter by running state.
• [ui] A new experimental asset overview page is available via user settings.

Bugfixes

• Fixed issue where asset jobs featuring a subset of a larger multi_asset could be rendered as an op-based job.
• Fixed an issue where Pythonic IO managers could not be passed to the io_manager_def param on an asset.
• Fixed an issue where late-binding an executor to a job, such as providing one to Definitions, would not correctly update the config schema.
• [dagster-k8s] Fixed an issue where setting maxResumeRunAttempts to null in the helm chart would cause it to be set to a default value of 3 instead of disabling run retries.
• [dagster-k8s] Fixed an issue where the k8s_job_executor would sometimes fail with a 409 Conflict error after retrying the creation of a Kubernetes pod for a step, due to the job having already been created during a previous attempt despite raising an error.
• [dagster-dbt] Fixed an issue where dbt logs were not being captured when invoking dbt commands using the resource.
• [dagster-dbt] Fixed an issue where if op_name was passed to load_assets_from_dbt_manifest, and a select parameter was specified, a suffix would be appended to the desired op name.
• [ui] Fixed an issue where using a path prefix for dagit would lead to JavaScript bundle loading errors.
• [ui] Resizing the viewport while viewing the Run timeline now correctly resizes the timeline and its contents.
• [ui] Ctrl-scroll to zoom the DAG view, and shift-scroll to pan horizontally now work on all platforms and an instructional tooltip explains the zoom interaction.

Experimental

• [dagster-dbt] Added a new implementation of the dbt Resource, DbtCli, in dagster_dbt.cli. This new resource only support dbt-core>=1.4.0.
• [dagster-dbt] Added a new decorator @dbt_assets in dagster_dbt.asset_decorator that allows you to specify a compute function for a selected set of dbt assets that loaded as an AssetsDefinition.

Documentation

• [dagster-duckdb] New guide and API reference page for the DuckDB I/O managers and resource
• [rbac] - Added documentation for the new Dagster Cloud Teams feature for role-based access control (RBAC). We’ve also revamped the documentation for Dagster Cloud user roles and permissions - check it out here.
• Fixed a typo in the "Using Environment Variables and Secrets" guide (thanks snikch!)

Dagster Cloud

• Fixed a bug in the multi-asset sensor where using context methods to fetch materializations by partition would cause a timeout.
• The ECS agent can now configure sidecars to be included with the tasks that the agent launches. See the docs for more information.