An orchestration platform for the development, production, and observation of data assets.
APACHE-2.0 License
Bot releases are hidden (Show)
Published by elementl-devtools 6 months ago
@asset_check
functions can now be invoked directly for unit testing.dagster-embedded-elt
dlt resource DagsterDltResource
can now be used from @op
definitions in addition to assets.UPathIOManager.load_partitions
has been added to assist with helping UpathIOManager
subclasses deal with serialization formats which support partitioning. Thanks @danielgafni
!PolarsDeltaIOManager
now supports MultiPartitionsDefinition
with DeltaLake
native partitioning. Metadata value "partition_by": {"dim_1": "col_1", "dim_2": "col_2"}
should be specified to enable this feature. Thanks @danielgafni
!load_assets_from_airbyte_instance
and load_assets_from_airbyte_project
will now be properly propagated to the created assets.build_time_partition_freshness_checks
where it would incorrectly intuit that an asset was not fresh in certain cases.PolarsDeltaIOManager
no longer supports loading natively partitioned DeltaLake tables as dictionaries. They should be loaded as a single pl.DataFrame
/pl.LazyFrame
instead.Dagster Cloud
to Dagster+
all over the docs.before_storage_id
/ after_storage_id
values in the AssetRecordsFilter
class were ignored. This has been fixed.dagster-cloud deployment alert-policies list
to match the format of sync
.Published by elementl-devtools 6 months ago
dagster-dbt project prepare-for-deployment
has been added in conjunction with DbtProject
for managing the behavior of rebuilding the manifest during development and preparing a pre-built one for production.build_last_update_freshness_checks
and build_time_partition_freshness_checks
has been fixed where multi_asset checks passed in would not be executable.DbtArtifacts
class has been replaced by DbtProject
.AssetObservation
to the Software-defined Assets API referencedagster-cloud-deploy.yml
get_asset_key
API documentation (thanks @aksestok!)Published by elementl-devtools 7 months ago
tags
argument on @asset
, AssetSpec
, and AssetOut
. Tags are meant to be used for organizing, filtering, and searching for assets.blocking
, which causes downstream assets in the same run to be skipped if the check fails with ERROR-level severity.@multi_asset_check
decorator enables defining a single op that executes multiple asset checks.build_last_updated_freshness_checks
and build_time_partition_freshness_checks
APIs allow defining asset checks that error or warn when an asset is overdue for an update. Refer to the Freshness checks guide for more info.build_column_schema_change_checks
API allows defining asset checks that warn when an asset’s columns have changed since its latest materialization.None
or MaterializeResult
return type annotations won't use I/O managers; dependencies for these assets can be set using the deps
parameter in the @asset
decorator.fetch_last_updated_timestamps
API, which makes it straightforward to collect data freshness information in source asset observation functions.OutputContext.output_metadata
.DAGSTER_EVENT_BATCH_SIZE
environment variable in a code server to an integer (25 recommended, 50 max). It is only currently supported in Dagster Cloud and OSS deployments with a postgres backend..
, which can occur when checks are ingested from dbt tests.DBT_INDIRECT_SELECTION
will no longer be set to empty
when executing dbt tests as asset checks, unless specific asset checks are excluded. dagster-dbt
will no longer explicitly select all dbt tests with the dbt cli, which had caused argument length issues.{{
config(
meta={
'dagster': {
'ref': {
'name': <ref_name>,
'package': ... # Optional, if included in the ref.
'version': ... # Optional, if included in the ref.
},
}
}
)
}}
...
build_schedule_from_dbt_selection
. Thanks @dragos-pop!dagster_polars
. Thanks @danielgafni!datadog.api
module to the DatadogClient
resource, enabling direct access to API methods. Thanks @shivgupta!metadata
argument on multi_asset
and AssetSpec
to Mapping[str, Any]
.s3_io_manager
.dagster-polars
tests by excluding Decimal
types. Thanks @ion-elgreco!ScheduleDefinition
passed into the Definitions
object. Thanks @2Ryan09!run_id
was previously private and only used for testing. It will now raise an exception.get_partition_keys_in_range
on a MultiPartitionsDefinition
would erroneously return partition keys that were within the one-dimensional range of alphabetically-sorted partition keys for the definition. Now, this method returns the cartesian product of partition keys within each dimension’s range. Thanks, @mst!AssetCheckExecutionContext
to replace AssetExecutionContext
as the type of the context
param passed in to @asset_check
functions. @asset_check
was an experimental decorator.@classmethod
decorators have been removed from dagster-embedded-slt.sling DagsterSlingTranslator
@classmethod
decorators have been removed from DagsterDbtTranslator
.AssetSelection.keys()
has been deprecated. Instead, you can now supply asset key arguments to AssetSelection.assets()
.AssetExecutionContext
has been simplified. Op-related methods and methods with existing access paths have been marked deprecated. For a full list of deprecated methods see this GitHub Discussion.metadata
property on InputContext
and OutputContext
has been deprecated and renamed to definition_metadata
.FreshnessPolicy
is now deprecated. For monitoring freshness, use freshness checks instead. If you are using AutoMaterializePolicy.lazy()
, FreshnessPolicy
is still recommended, and will continue to be supported until an alternative is provided.dagster-polars
documentation. Thanks @danielgafni!Published by elementl-devtools 7 months ago
Published by elementl-devtools 7 months ago
Published by elementl-devtools 7 months ago
AssetCheckResult
now has a text description
property. Check evaluation descriptions are shown in the Checks tab on the asset details page.TimestampMetadataValue
. Timestamp metadata values are represented internally as seconds since the Unix epoch. They can be constructed using MetadataValue.timestamp
. In the UI, they’re rendered in the local timezone, like other timestamps in the UI.AssetSelection.checks
can now accept AssetCheckKeys
as well as AssetChecksDefinition
.add_output_metadata
or by passing to Output
) is now available on HookContext
under the op_output_metadata
property. Thanks @JYoussouf!@asset
, AssetSpec
, and AssetOut
now accept a tags
property. Tags are key-value pairs meant to be used for organizing asset definitions. If "__dagster_no_value"
is set as the value, only the key will be rendered in the UI. AssetSelection.tag
allows selecting assets that have a particular tag.dagster asset materialize --select tag:department=marketing
DagsterDbtTranslator.get_tags
. By default, we take the dbt tags configured on your dbt models, seeds, and snapshots.DatabricksJobRunner
that led to an inability to use dagster-databricks with Databricks instance pools. Thanks @smats0n!AssetKey
. Thanks @maxfirman!load_assets_from_package_module
that would cause a crash when any submodule had the same directory name as a dependency. Thanks @CSRessel!"columns"
to "dagster/column_schema"
. Materializations using the old metadata key will no longer result in the Columns section of the tab being filled out.Published by elementl-devtools 7 months ago
dagster dev
or the Dagster UI would display an error when loading jobs created with op or asset selections.Published by elementl-devtools 7 months ago
@multi_asset
s are in use has been drastically improved.the_asset.with_attributes
is called without providing a group name. Previously, the existing group name was erroneously dropped. Thanks @ion-elgreco!dbt source freshness
.MetadataValue
in Essentials course. Thanks @stianthaulow!tag_keys
argument on DagsterInstance.get_run_tags
is no longer optional. This has been done to remove an easy way of accidentally executing an extremely expensive database operation.Published by elementl-devtools 8 months ago
AssetSelection.
dagster-snowflake
now requires snowflake-connector-python>=3.4.0
@sling_assets
accepts an optional name parameter for the underlying opdagster-openai
library is now available.DagsterDbtTranslatorSettings
called enable_duplicate_source_asset_keys
that allows users to set duplicate asset keys for their dbt sources. Thanks @hello-world-bfree!ConfigurableResource
.SlingConnectionResource
where a blank keyword argument would be emitted as an environment variabledbt source freshness
would cause an error.DbtArtifacts
has been added for managing the behavior of rebuilding the manifest during development but expecting a pre-built one in production.send sample alert
button now exists on both the alert policies page and in the alert policies editor to make it easier to debug and configure alerts without having to wait for an event to kick them off.Published by elementl-devtools 8 months ago
SlingConnectionResource
that raised an error when connecting to a database.graph_multi_assets
with check_specs
now support subsetting.Published by elementl-devtools 8 months ago
run_retries.retry_on_op_or_asset_failures
setting that can be set to false to make run retries only occur when there is an unexpected failure that crashes the run, allowing run-level retries to co-exist more naturally with op or asset retries. See the docs for more information.dagster dev
now sets the environment variable DAGSTER_IS_DEV_CLI
allowing subprocesses to know that they were launched in a development context.@multi_observable_source_asset
decorator enables defining a set of assets that can be observed together with the same function.@sling_assets
and Resource SlingConnectionResource
have been added for the [dagster-embedded-elt.sling](http://dagster-embedded-elt.sling)
package. Deprecated build_sling_asset
, SlingSourceConnection
and SlingTargetConnection
.QueuedRunCoordinator
.Published by elementl-devtools 8 months ago
dagster-polars
has been added as an integration. Thanks @danielgafni!@dbt_assets
now supports loading projects with semantic models.@dbt_assets
now supports loading projects with model versions.get_asset_key_for_model
now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!@asset
decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!security_context
field on the k8s_job_executor
didn't correctly set the security context on the launched step pods. Thanks @krgn!ObserveResult
s with no data_version
.FreshnessPolicy
s on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.Published by elementl-devtools 8 months ago
DbtCliResource
now include metadata from the dbt adapter response. This includes fields like rows_affected
, query_id
from the Snowflake adapter, or bytes_processed
from the BigQuery adapter.k8s_job_executor
.context.log.exception
in a sensor or schedule did not properly capture exception information.@observable_source_asset
-decorated functions can now return an ObserveResult
. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron
class allows you to construct AutoMaterializePolicys
which wait for all parents to be updated after the latest tick of a given cron schedule.Published by elementl-devtools 8 months ago
build_schedule_from_partitioned_job
now supports creating a schedule from a static-partitioned job (Thanks @craustin
!)PipesK8sClient
will now autodetect the namespace when using in-cluster config. (Thanks @aignas
!)PipesK8sClient
can now inject the context in to multiple containers. (Thanks @aignas
!)write_pandas
method to load Pandas DataFrames in Snowflake. To support this change, the database connector was switched from SqlDbConnection
to SnowflakeConnection
.@ion-elgreco
!)@dbt_assets
and multiple dbt resources produce the same AssetKey
, we now display an exception message that highlights the file paths of the misconfigured dbt resources in your dbt project.@jblawatt
!)automountServiceAccountToken: false
to the default service account used by the Helm chart, in order to better comply with security policies. (Thanks @MattyKuzyk
!)@ruizh22
!)PipesK8sClient
has improved handling for init containers and additional containers. (Thanks @aignas
!)last_sensor_start_time
property of the SensorEvaluationContext
, which would get cleared on ticks after the first tick after the sensor starts.dagster instance migrate --bigint-migration
, which caused some operational errors on mysql storages.AssetExecutionContext
have been marked deprecated, with their suggested replacements in parenthesis:
context.op_config
(context.op_execution_context.op_config
)context.node_handle
(context.op_execution_context.node_handle
)context.op_handle
(context.op_execution_context.op_handle
)context.op
(context.op_execution_context.op
)context.get_mapping_key
(context.op_execution_context.get_mapping_key
)context.selected_output_names
(context.op_execution_context.selected_output_names
)context.dagster_run
(context.run
)context.run_id
(context.run.run_id
)context.run_config
(context.run.run_config
)context.run_tags
(context.run.tags
)context.has_tag
(key in context.run.tags
)context.get_tag
(context.run.tags.get(key)
)context.get_op_execution_context
(context.op_execution_context
)context.asset_partition_key_for_output
(context.partition_key
)context.asset_partition_keys_for_output
(context.partition_keys
)context.asset_partitions_time_window_for_output
(context.partition_time_window
)context.asset_partition_key_range_for_output
(context.partition_key_range
)@asset_check
now has a blocking
parameter. When this is enabled, if the check fails with severity ERROR
then any downstream assets in the same run won’t execute.MaterializeResult
Edit
button next to an agent token on the Tokens
tab in Org Settings
to configure permissions for a particular token. You must be an Organization Admin to edit agent token permissions.Published by elementl-devtools 9 months ago
pendulum
library, for Python versions 3.9 and higher.AllPartitionMapping
now supports mapping to downstream partitions, enabling asset backfills with these dependencies. Thanks @craustin!@asset_check
has new fields additional_deps
and additional_ins
to allow dependencies on assets other than the asset being checked.dbt-core==1.4.*
is now removed because the version has reached end-of-life.get_partition_keys_not_in_subset
on a BaseTimeWindowPartitionsSubset
that targeted a partitions definition with no partitions (e.g. a future start date) would raise an error. Now, it returns an empty list.LocalComputeLogManager
on Windows could result in errors relating to invalid paths. This has been resolved. Thanks @hainenber!DatabricksJobRunner
now ensures the correctdatabricks-sdk
is installed. Thanks @zyd14!config.meta.description
. Thanks @CapitanHeMo!PipesK8sClient
can now correctly handle load_incluster_config
. Thanks @aignas!agent_queue
key in a dagster_cloud.yaml
file incorrectly failed to validate when using the dagster-cloud ci init
or dagster-cloud ci check
commands during CI/CD.Published by elementl-devtools 9 months ago
warn_if
/ error_if
severities1.6.0
where run status sensors did not cursor correctly when deployed on Dagster Cloud.Published by elementl-devtools 9 months ago
DAGSTER_REDACT_USER_CODE_ERRORS
environment variable to 1
.@dbt_assets
now accepts a required_resource_keys
argument.Published by elementl-devtools 9 months ago
MaterializeResult
, AssetSpec
, and AssetDep
now marked stable – These APIs, introduced in Dagster 1.5, were previously marked experimental. They offer a more straightforward way of defining assets when you don’t want to use I/O managers.PipesLambdaClient
[guide].report_custom_message
and get_custom_messages
.concurrency
> default_op_concurrency_limit
configuration in your dagster.yaml
(OSS) or Deployment Settings page (Dagster Cloud). In OSS, this feature first requires a storage migration (e.g. dagster instance migrate
).dagster instance migrate
).Nothing
-typed output is returned from an asset
or op
, the handle_output
function of the I/O manager will no longer be called. Users of most Dagster-maintained I/O managers will see no behavioral changes, but users of the In-Memory I/O manager, or custom I/O managers that store Nothing
-typed outputs should reference the migration guide for more information.dagster asset materialize
will now return a non-zero exit code upon failure.dagster<1.5.1
, you may see the first tick of your run status sensor skip runs that completed but were not yet registered by the sensor during your upgrade. This should not be common, but to avoid any chance of that, you may consider an interim upgrade to dagster>=1.5.1,<1.6.0
first.Published by elementl-devtools 10 months ago
graph_multi_asset
s can now accept inputs as kwargs
.resource_defs
parameter of @asset_check
to not be respectedPipesFileMessageReader
that could cause a crash on Windows.context.log
in different threads within a single op could result in some of those log messages being dropped. This has been fixed (thanks @quantum-byte!)file_manager
property on OpExecutionContext
and AssetExecutionContext
has been removed. This is an ancient property that was deprecated prior to Dagster 1.0, and since then had been raising a NotImplementedError
whenever invoked.forward_termination
argument is available for opting out.Published by elementl-devtools 10 months ago
SensorEvaluationContext
object has two new properties: last_sensor_start_time
and is_first_tick_since_sensor_start
. This enables sensor evaluation functions to vary behavior on the first tick vs subsequent ticks after the sensor has started.asset_selection
argument to @sensor
and SensorDefinition
now accepts sequence of AssetsDefinitions
, a sequences of strings, or a sequence of AssetKey
s, in addition to AssetSelection
s.dbt-core==1.3.*
has been removed.show_url_only
parameter to GCSComputeLogManager
that allows you to configure the compute log manager so that it displays a link to the GCS console rather than loading the logs from GCS, which can be useful if giving Dagster access to GCS credentials is undesirable.BranchingIOManager
hashlib.md5
as not used for security purposes (Thanks @jlloyd-widen!)execute_k8s_job
to be aware of run-termination and op failure by deleting the executing k8s job (Thanks @Taadas!).plotly
graphs) when viewing dagstermill notebooks (Thanks @aebrahim!).DbtCliResource
API docs (Thanks @akan72!)report_custom_message
) to the orchestration process (get_custom_messages
).