An orchestration platform for the development, production, and observation of data assets.
APACHE-2.0 License
Bot releases are visible (Hide)
Published by elementl-devtools over 1 year ago
max_materializations_per_minute
parameter (with a default of 1) to AutoMaterializationPolicy.eager()
and AutoMaterializationPolicy.lazy()
allows you to set bounds on the volume of work that may be automatically kicked off for each asset. To restore the previous behavior, you can explicitly set this limit to None
.DailyPartitionsDefinition
, HourlyPartitionsDefinition
, WeeklyPartitionsDefinition
, and MonthlyPartitionsDefinition
now support and end_date
attribute.AutoMaterializePolicy
s or build_asset_reconciliation_sensor
, a single new data version from an observable source asset could trigger multiple runs of the downstream assets. This has been fixed.EnvVar
and IntEnvVar
within raw run config - although they just returned the name of the env var rather than retrieve its value. This has been fixed to error directly.*
was being interpreted as glob pattern, rather than as a dbt selector argument. We now explicitly set the default selection pattern as fqn:*
.dagster-cloud serverless deploy
did not create a unique image tag if the --image
tag was not specified.op_name
on load_assets_from_dbt_project
and load_assets_from_dbt_manifest
(thanks @wkeifenheim!)Published by elementl-devtools over 1 year ago
TimeWindowPartitionMapping
objects are now current-time aware. Subsequently, only upstream/downstream partitions existent at the current time are returned.ExecuteJobResult
was renamed to JobExecutionResult
(ExecuteJobResult
remains a deprecated alias)AssetSelection.key_prefixes
method allows matching asset keys starting with a provided prefix.ConfigurablePickledObjectADLS2IOManager
that uses pythonic configDataProcResource
follows the Pythonic resource system. The existing dataproc_resource
remains supported.SnowflakeIOManager
now supports private_key
s that have been base64
encoded to avoid issues with newlines in the private key. Non-base64 encoded keys are still supported. See the SnowflakeIOManager
documentation for more information on base64
encoded private keys.kvs
, instance_info
, and daemon_hearbeats
table for existing Postgres storage instances that were migrating from before 1.2.2
. This should unblock users relying on the existence of a primary key constraint for replication.SensorResult
evaluation where multipartitioned run requests containing a dynamic partition added in a dynamic partitions request object would raise an invalid partition key error.TimeWindowPartitionMapping
.ExecuteJobResult
is now a deprecated alias for the new name, JobExecutionResult
.airbyte_resource
to load_assets_from_connections
, you may now provide an instance of the AirbyteResource
class, rather than just airbyte_resource.configured(...)
(thanks @joel-olazagasti!)
dagster-celery
CLI accepted an inconsistent configuration format - it now matches the same format as the celery_executor
. Thanks @boenshao!ecs_timeout
in your ECS user code launcher config to extend how long the ECS agent polls for new code servers to start. Extending this timeout is useful if your code server takes an unusually long time to start up - for example, because it uses a very large image.Published by elementl-devtools over 1 year ago
load_assets_from_package_module
and the other core load_assets_from_
methods now accept a source_key_prefix
argument, which allows applying a key prefix to all the source assets that are loaded.
OpExecutionContext
now has an asset_partitions_time_window_for_input
method.
RunFailureSensorContext
now has a get_step_failure_events
method.
The Pythonic resource system now supports a set of lifecycle hooks which can be used to manage setup and teardown:
class MyAPIClientResource(ConfigurableResource):
api_key: str
_internal_client: MyAPIClient = PrivateAttr()
def setup_for_execution(self, context):
self._internal_client = MyAPIClient(self.api_key)
def get_all_items(self):
return self._internal_client.items.get()
Added support for specifying input and output config on ConfigurableIOManager
.
QueuedRunCoordinator
and SubmitRunContext
are now exposed as public dagster exports.
[ui] Downstream cross-location dependencies of all source assets are now visible on the asset graph. Previously these dependencies were only displayed if the source asset was defined as a regular asset.
[ui] A new filtering experience is available on the Runs page after enabling feature flag “Experimental Runs table view with filtering”.
[dagster-aws] Allow the S3 compute log manager to specify a show_url_only: true
config option, which will display a URL to the S3 file in dagit, instead of the contents of the log file.
[dagster-aws] PickledObjectS3IOManager
now fully supports loading partitioned inputs.
[dagster-azure] PickedObjectADLS2IOManager
now fully supports loading partitioned inputs.
[dagster-gcp] New GCSResource
and ConfigurablePickledObjectGCSIOManager
follow the Pythonic resource system. The existing gcs_resource
and gcs_pickle_io_manager
remain supported.
[dagster-gcp] New BigQueryResource
follows the Pythonic resource system. The existing bigquery_resource
remains supported.
[dagster-gcp] PickledObjectGCSIOManager
now fully supports loading partitioned inputs.
[dagster-postgres] The event watching implementation has been moved from listen/notify based to the polling watcher used by MySQL and SQLite.
[dagster-slack] Add monitor_all_repositories
to make_slack_on_run_failure_sensor
, thanks @danielgafni!
[dagster-snowflake] New SnowflakeResource
follows the Pythonic resource system. The existing snowflake_resource
remains supported.
AssetMetadataValue.value
that would cause an infinite recursion error.volumes
and volumeMounts
values have been added to the agent helm chart.load_assets_from_airbyte_instance
and load_assets_from_airbyte_project
now take a connection_to_auto_materialize_policy_fn
for setting AutoMaterializePolicy
s on Airbyte assetsPublished by elementl-devtools over 1 year ago
NumReplicas
parameter on the agent template in CloudFormation, or the dagsterCloudAgent.replicas
field in Helm.enableZeroDowntimeDeploys
parameter to true in the CloudFormation stack for your agent.AssetsDefinition.from_graph
, as well as the@graph_asset
and @graph_multi_asset
decorators now support specifying AutoMaterializePolicy
s.async def
ops/assets no longer prematurely finalize async generators during execution.build_asset_reconciliation_sensor
) could incorrectly launch new runs for partitions that already had an in-progress run. This has been fixed.run_request_for_partition
now throws an error. Instead, users should yield directly instantiated run requests via RunRequest(partition_key=...)
.graph_asset
and graph_multi_asset
now support specifying resource_defs
directly (thanks @kmontag42)!node_info_to_auto_materialize_policy_fn
param added to load_assets_from_dbt_*
functions. (thanks @askvinni)!partition_key
field to RunStatusSensorContext
(thanks @pdstrnadJC)!latest_materialization_records_by_key
method.allPartitions
argument is provided.Published by elementl-devtools over 1 year ago
Published by elementl-devtools over 1 year ago
build_asset_reconciliation_sensor
s , AutoMaterializePolicy
s work across code locations, as well as allow you to customize the conditions under which each asset is auto-materialized. [docs]FreshnessPolicy
s, Dagster’s UI now marks them as “overdue” instead of “late”.AthenaClientResource
, ECRPublicResource
, RedshiftClientResource
, S3Resource
, S3FileManagerResource
, ConfigurablePickledObjectS3IOManager
, SecretsManagerResource
follow Pythonic resource system. The existing APIs remain supported.DatadogResource
follows Pythonic resource system. The existing datadog_resource
remains supported.GEContextResource
follows Pythonic resource system. The existing ge_context_resource
remains supported.GithubResource
follows Pythonic resource system. The existing github_resource
remains supported.MSTeamsResource
follows Pythonic resource system. The existing msteams_resource
remains supported.SlackResource
follows Pythonic resource system. The existing slack_resource
remains supported.pdb.set_trace
no longer worked when running Dagster locally using dagster dev
or dagit
.@asset
or Out
caused an error to be thrown.pipelines
or jobs
could lead to blank pages.Definitions
are now automatically bound to jobs. This will only result in a change in behavior if you a) have a job with no "io_manager" defined in its resource_defs
and b) have supplied an IOManager
with key "io_manager" to the resource_defs
argument of your Definitions
. Prior to 1.3.0, this would result in the job using the default filesystem-based IOManager
for the key "io_manager". In 1.3.0, this will result in the "io_manager" supplied to your Definitions
being used instead. The BindResourcesToJobs
wrapper, introduced in 1.2 to simulate this behavior, no longer has any effect.CeleryK8sRunLauncher
is now the same namespace as the Helm chart, instead of the default
namespace. To restore the previous behavior, you can set the celeryK8sRunLauncher.jobNamespace
field to the string default
.SnowflakePandasIOManager
has historically converted all timestamp data to strings before storing it in Snowflake. Now, it will instead ensure that timestamp data has a timezone, and if not, attach the UTC timezone. This allows the timestamp data to be stored as timestamps in Snowflake. If you have been storing timestamp data using the SnowflakePandasIOManager
you can set the store_timestamps_as_strings=True
configuration to continue storing timestamps as strings. For more information, and instructions for migrating Snowflake tables to use timestamp types, see the Migration Guide.Changes to experimental APIs
RunConfig
to many APIs which previously would only accept a config dictionary.ResourceDefinition
.execution
config when constructing a RunConfig
object.minutes_late
and previous_minutes_late
properties on the experimental FreshnesPolicySensorContext
have been renamed to minutes_overdue
and previous_minutes_overdue
, respectively.Removal of deprecated APIs
metadata_entries
arguments to event constructors have been removed. While MetadataEntry
still exists and will only be removed in 2.0, it is no longer passable to any Dagster public API — users should always pass a dictionary of metadata values instead.latest_materialization_records_by_key
function.BigQueryPandasIOManager
and BigQueryPySparkIOManager
.SnowflakePandasIOManager
and SnowflakePySparkIOManager
.Published by elementl-devtools over 1 year ago
required_resource_keys
and Pythonic resources) are now supported in observable source assets.EcsRunLauncher
now allows you to customize volumes and mount points for the launched ECS task. See the API docs for more information.DuckDBPandasIOManager
and DuckDBPySparkIOManager
follow Pythonic resource system. The existing duckdb_pandas_io_manager
and duckdb_pyspark_io_manager
remain supported.BigQueryPandasIOManager
and BigQueryPySparkIOManager
follow Pythonic resource system. The existing bigquery_pandas_io_manager
and bigquery_pyspark_io_manager
remain supported.gcp_crentials
, a temporary file to store the credentials will be created and the GOOGLE_APPLICATION_CREDENTIALS
environment variable will be set to the temporary file. When the BigQuery resource is garbage collected, the environment variable will be unset and the temporary file deleted.SnowflakePandasIOManager
and SnowflakePySparkIOManager
follow Pythonic resource system. The existing snowflake_pandas_io_manager
and snowflake_pyspark_io_manager
remain supported.dagster dev
would periodically emit a harmless but annoying warning every few minutes about a gRPC server being shut down.RunRequest(partition_key=...)
object.PartitionsDefinition
.run_request_for_partition
would incorrectly raise an error for a job with a DynamicPartitionsDefinition
that was defined with a function.define_asset_job
would raise an error.__ASSET_JOB has no op named ...
errors when using automatic run retries.EcsRunLauncher
would fail to launch runs if the use_current_ecs_task_config
field was set to False
but no task_definition
field was set.EcsRunLauncher
now allows you to set a capacity provider strategy and customize the ephemeral storage used for launched ECS tasks. See the docs for details. Thanks AranVinkItility!AssetsDefinition.from_op
. Thanks @tghanken for the fix!minimum_interval_seconds
parameter to enable customizing the evaluation interval on the slack run failure sensor, thanks @ldnicolasmay!Resource
annotation for Pythonic resource inputs has been renamed to ResourceParam
in preparation for the release of the feature in 1.3.Published by elementl-devtools over 1 year ago
materialize
and materialize_to_memory
now both accept a selection
argument that allows specifying a subset of assets to materialize.MultiPartitionsDefinition
is no longer marked experimental.MultiPartitionsDefinition
s with a time dimension.load_assets_from_package_module
and similar methods now accept a freshness_policy
, which will be applied to all loaded assets.build_sensor_context
and build_multi_asset_sensor_context
can now take a Definitions
object in place of a RepositoryDefinition
s3_resource
now accepts use_ssl
and verify
configurations.selection
argument on define_asset_job
.MultiPartitionsDefinition
.MultiPartitionsMapping
(experimental) is loaded.TimeWindowPartitionMapping
s to not be bubbled up to the UI.pdb
stopped working when using dagster dev.AssetMaterialization
objects with a null AssetKey
.TimeWindowPartitionsDefinition
with a non-standard cron schedule, and also provided a minute_of_hour
or similar argument in build_schedule_from_partitioned_job
. Dagster would silently create the wrong cron expression. It now raises an error.PartitionsDefinition
. These partitions are now ignored.PersistentAirflowDatabase
where versions of airflow from 2.0.0 till 2.3.0 would not use the correct connection environment variable name.poll_sync_run
function ofdagster-census
that prevented polling from working correctly (thanks @ldincolasmay!)run_request_for_partition
method on JobDefinition
and UnresolvedAssetJobDefinition
is now deprecated and will be removed in 2.0.0. Instead, directly instantiate a run request with a partition key via RunRequest(partition_key=...)
.Published by elementl-devtools over 1 year ago
AssetsDefinition.to_source_assets
to method convert a set of assets to SourceAsset
objects.MultiPartitionsDefinitions
.mlflow
pin from the dagster-mlflow
package.When using build_asset_reconciliation_sensor
, in some cases duplicate runs could be produced for the same partition of an asset. This has been fixed.
When using Pythonic configuration for resources, aliased field names would cause an error. This has been fixed.
Fixed an issue where context.asset_partitions_time_window_for_output
threw an error when an asset was directly invoked with build_op_context
.
[dagster-dbt] In some cases, use of ephemeral dbt models could cause the dagster representation of the dbt dependency graph to become incorrect. This has been fixed.
[celery-k8s] Fixed a bug that caused JSON deserialization errors when an Op or Asset emitted JSON that doesn't represent a DagsterEvent
.
Fixed an issue where launching a large backfill while running dagster dev
would sometimes fail with a connection error after running for a few minutes.
Fixed an issue where dagster dev
would sometimes hang when running Dagster code that attempted to read in input via stdin.
Fixed an issue where runs that take a long time to import code would sometimes continue running even after they were stopped by run monitoring for taking too long to start.
Fixed an issue where AssetSelection.groups()
would simultaneously select both source and regular assets and consequently raise an error.
Fixed an issue where BindResourcesToJobs
would raise errors encapsulating jobs which had config specified at definition-time.
Fixed Pythonic config objects erroring when omitting optional values rather than specifying None
.
Fixed Pythonic config and resources not supporting Enum values.
DagsterInstance.local_temp
and DagsterInstance.ephemeral
now use object instance scoped local artifact storage temporary directories instead of a shared process scoped one, removing a class of thread safety errors that could manifest on initialization.
Improved direct invocation behavior for ops and assets which specify resource dependencies as parameters, for instance:
class MyResource(ConfigurableResource):
pass
@op
def my_op(x: int, y: int, my_resource: MyResource) -> int:
return x + y
my_op(4, 5, my_resource=MyResource())
[dagster-azure] Fixed an issue with an AttributeError being thrown when using the async DefaultAzureCredential
(thanks @mpicard)
[ui] Fixed an issue introduced in 1.2.3 in which no log levels were selected by default when viewing Run logs, which made it appear as if there were no logs at all.
environment_vars
argument to ScheduleDefinition
is deprecated (the argument is currently non-functional; environment variables no longer need to be whitelisted for schedules)[CHANGES.md](http://CHANGES.md)
(thanks @fridiculous)--path-prefix
can now be supplied via Helm chart (thanks @mpicard)https://github.com/dagster-io/dagster/compare/1.2.3...1.2.4
[refactor] delete Materialization (#13030)
by @smackeseyAdd in progress subsets to the partition cache (#13045)
by @johannkmAdd multipartitioned assets with dynamic dimension to toys (#13061)
by @clairelin135add docs example for multi-asset conditional materialization (#13054)
by @sryzaAdd docs for source asset observation jobs/schedules (#13062)
by @smackeseyRevert "Add in progress subsets to the partition cache (#13045)"
by @johannkm[asset-reconciliation][bug] Fix issue where overly-aggressive runs would be kicked off. (#13069)
by @OwenKepharttweaks to cross-repo-assets toy (#12973)
by @sryzaShow unauthorized error graphql error message (#13064)
by @salazarm[dagster-io/ui] Make Suggest component a bit more flexible (#13056)
by @hellendagFix disabled state for launchpad button submenu (#13078)
by @salazarmRe-enable in progress subsets in the partition cache (#13082)
by @johannkmAdd dynamic partitions name resolver to dimension type (#13070)
by @clairelin135asset sensor test docs (#13065)
by @prhaFix issue where backfill fails when gRPC server is replaced mid-backfill (#13085)
by @gibsondanUse instance from sensor/schedule context to instantiate resources, delay until accessed (#13041)
by @benpankowAdd materializing subset to asset gql (#13046)
by @johannkmfeat(helm): add path-prefix to dagit command (#13080)
by @mpicardUse dynamic partition definition name for dimension of multipartition definition (#13090)
by @salazarm[typing/static] Fix @repository decorator typing (#12295)
by @smackesey[instance] make local artifact directory scheme thread safe (#13043)
by @alangenfeld1.2.3 changelog (#13094)
by @jamiedemariaEnsure pyright venvs use statically legible editable installs (#13089)
by @smackesey[docs] - Remove finished code from dbt tutorial template (#13091)
by @erinkcochran87[ui] Upgrade react-markdown (#13092)
by @hellendagFix submitting backfills synchronously from graphql (#13093)
by @gibsondanTest get and set serialized_in_progress_partition_subset (#13063)
by @johannkmExitStack.pop_all -> close (#13050)
by @alangenfeldAutomation: versioned docs for 1.2.3
by @elementl-devtoolsFraser/rework readme (#12565)
by @frasermarlow[ui] Use DefaultLogLevels when there is no level state stored (#13109)
by @hellendagSet stdin to DEVNULL when opening dagster subprocesses (#13099)
by @gibsondanadd a vercel github action to build docs/storybook previews (#13052)
by @prha[refactor] Remove
frozen{list,dict,tags} classes (#12293)
by @smackeseyDocs for setting up Gitlab CI, branch deployment guide (#12998)
by @prhaAdd assets def to op context (#13088)
by @clairelin135make AssetsDefinition.to_source_assets public (#13073)
by @sryza[docs] New ML pipeline guide PR (#13100)
by @odette-elementl[freshness-refactor][3/n] Update methods on the CachingDataTimeResolver to work with scalar data time (#12906)
by @OwenKephartFixing refs to images in the README (#13126)
by @tacastillorestrict vercel builds based on paths (#13129)
by @prhafix missing snapshots (#13134)
by @OwenKephartTelemetry for dynamic partitions (#12605)
by @clairelin135fix missing snapshots (again) (#13136)
by @OwenKephart[freshness-refactor][4/n] Simplify scheduling algorithm (#13019)
by @OwenKephartDeprecate environment_vars argument to
ScheduleDefinition,
@schedule (#13044)
by @smackesey[refactor] delete hourly/daily/weekly/monthly schedule decorators, PartitionScheduleDefinition, build_schedule_from_partition (#13006)
by @smackesey[refactor] simplify dependency dict typing (#12521)
by @smackeseycelery-k8s executor: handle stdout that's valid json but not a dagster event (#13143)
by @johannkmfix typo in CHANGES.md (#13140)
by @fridiculousFix branch deployment docs (#13131)
by @dpeng817Update multiple agents docs (#13135)
by @dpeng817add toy for eager asset reconciliation (#13066)
by @sryzatrying again by moving the images to the same directory as the readme (#13127)
by @tacastillo[tech][templates] moving the scaffold project's asset loader outside of the defs (#13103)
by @tacastillo[docs][tutorial-revamp] Adding a section for scheduling to the tutorial (#13101)
by @tacastilloreplacing corrupt .png image (#13157)
by @frasermarlow[dagster-dbt] Fix bug when calculating transitive dependencies (#13128)
by @OwenKephartdeploy storybook to prod when landing pushes on master (#13159)
by @prha[dagster-azure] fix: AttributeError: 'coroutine' object has no attribute 'token' (#13110)
by @mpicardAdd support for more asset tags (#13153)
by @braunjj[caching-refactor] Remove use of get_and_update_asset_status_cache. (#13151)
by @OwenKephart[asset-reconciliation][perf] Cache common properties on the TimeWindowPartitionsDefinition (#12981)
by @OwenKephartMultiPartitionMapping (#12950)
by @clairelin135[asset-reconciliation] Fix issue with duplicate runs for partitions with in-progress materializations (#13130)
by @OwenKephartupdate telemetry documentation links (#13176)
by @Abbe98[structured config] respect field aliases for resources (#13177)
by @jamiedemaria[refactor] Remove PartitionSetDefinition (#13145)
by @smackeseyTableMetadataValue toy (#13124)
by @smackeseyfix repository decorator typing (#13180)
by @smackeseyadd storybook for @core in addition to @ui (#13184)
by @prhafeat(dbt): follow dbt Core's version support constraints (#13189)
by @rexledesmaFix case where a pipeline load takes long enough that run monitoring kills it (#13156)
by @gibsondanAllow newlines in error messages in Dagit (#13193)
by @gibsondanMake sure we always wait for all grpc server processes to spin down in tests (#13146)
by @gibsondanunpin mlflow from dagster-mlflow (#13194)
by @gibsondanGreatly reduce the number of gRPC calls when doing large asset backfills (#13086)
by @gibsondandon't run mlflow tests on python 3.11 (#13197)
by @gibsondan[pythonic config] Treat Optional as both not-required and Noneable (#12975)
by @benpankow[pythonic config] Add support for enums (#12979)
by @benpankow[pythonic resources] Make direct invocation of ops/assets w/ resources easier (#13002)
by @benpankow[bugfix] make AssetSelection.groups() resolve to only regular assets (#13196)
by @smackesey[fix] Fix behavior of BindResourcesToJobs when jobs have config specified (#13057)
by @benpankow[serdes] remove extraneous @cached_method use (#13206)
by @alangenfeld1.2.4 changelog (#13226)
by @smackesey1.2.4
by @elementl-devtools[refactor] delete Materialization (#13030)
by @smackeseyAdd in progress subsets to the partition cache (#13045)
by @johannkmAdd multipartitioned assets with dynamic dimension to toys (#13061)
by @clairelin135add docs example for multi-asset conditional materialization (#13054)
by @sryzaAdd docs for source asset observation jobs/schedules (#13062)
by @smackeseyRevert "Add in progress subsets to the partition cache (#13045)"
by @johannkm[asset-reconciliation][bug] Fix issue where overly-aggressive runs would be kicked off. (#13069)
by @OwenKepharttweaks to cross-repo-assets toy (#12973)
by @sryzaShow unauthorized error graphql error message (#13064)
by @salazarm[dagster-io/ui] Make Suggest component a bit more flexible (#13056)
by @hellendagFix disabled state for launchpad button submenu (#13078)
by @salazarmRe-enable in progress subsets in the partition cache (#13082)
by @johannkmAdd dynamic partitions name resolver to dimension type (#13070)
by @clairelin135asset sensor test docs (#13065)
by @prhaFix issue where backfill fails when gRPC server is replaced mid-backfill (#13085)
by @gibsondanUse instance from sensor/schedule context to instantiate resources, delay until accessed (#13041)
by @benpankowAdd materializing subset to asset gql (#13046)
by @johannkmfeat(helm): add path-prefix to dagit command (#13080)
by @mpicardUse dynamic partition definition name for dimension of multipartition definition (#13090)
by @salazarm[typing/static] Fix @repository decorator typing (#12295)
by @smackesey[instance] make local artifact directory scheme thread safe (#13043)
by @alangenfeld1.2.3 changelog (#13094)
by @jamiedemariaEnsure pyright venvs use statically legible editable installs (#13089)
by @smackesey[docs] - Remove finished code from dbt tutorial template (#13091)
by @erinkcochran87[ui] Upgrade react-markdown (#13092)
by @hellendagFix submitting backfills synchronously from graphql (#13093)
by @gibsondanTest get and set serialized_in_progress_partition_subset (#13063)
by @johannkmExitStack.pop_all -> close (#13050)
by @alangenfeldAutomation: versioned docs for 1.2.3
by @elementl-devtoolsFraser/rework readme (#12565)
by @frasermarlow[ui] Use DefaultLogLevels when there is no level state stored (#13109)
by @hellendagSet stdin to DEVNULL when opening dagster subprocesses (#13099)
by @gibsondanadd a vercel github action to build docs/storybook previews (#13052)
by @prha[refactor] Remove
frozen{list,dict,tags} classes (#12293)
by @smackeseyDocs for setting up Gitlab CI, branch deployment guide (#12998)
by @prhaAdd assets def to op context (#13088)
by @clairelin135make AssetsDefinition.to_source_assets public (#13073)
by @sryza[docs] New ML pipeline guide PR (#13100)
by @odette-elementl[freshness-refactor][3/n] Update methods on the CachingDataTimeResolver to work with scalar data time (#12906)
by @OwenKephartFixing refs to images in the README (#13126)
by @tacastillorestrict vercel builds based on paths (#13129)
by @prhafix missing snapshots (#13134)
by @OwenKephartTelemetry for dynamic partitions (#12605)
by @clairelin135fix missing snapshots (again) (#13136)
by @OwenKephart[freshness-refactor][4/n] Simplify scheduling algorithm (#13019)
by @OwenKephartDeprecate environment_vars argument to
ScheduleDefinition,
@schedule (#13044)
by @smackesey[refactor] delete hourly/daily/weekly/monthly schedule decorators, PartitionScheduleDefinition, build_schedule_from_partition (#13006)
by @smackesey[refactor] simplify dependency dict typing (#12521)
by @smackeseycelery-k8s executor: handle stdout that's valid json but not a dagster event (#13143)
by @johannkmfix typo in CHANGES.md (#13140)
by @fridiculousFix branch deployment docs (#13131)
by @dpeng817Update multiple agents docs (#13135)
by @dpeng817add toy for eager asset reconciliation (#13066)
by @sryzatrying again by moving the images to the same directory as the readme (#13127)
by @tacastillo[tech][templates] moving the scaffold project's asset loader outside of the defs (#13103)
by @tacastillo[docs][tutorial-revamp] Adding a section for scheduling to the tutorial (#13101)
by @tacastilloreplacing corrupt .png image (#13157)
by @frasermarlow[dagster-dbt] Fix bug when calculating transitive dependencies (#13128)
by @OwenKephartdeploy storybook to prod when landing pushes on master (#13159)
by @prha[dagster-azure] fix: AttributeError: 'coroutine' object has no attribute 'token' (#13110)
by @mpicardAdd support for more asset tags (#13153)
by @braunjj[caching-refactor] Remove use of get_and_update_asset_status_cache. (#13151)
by @OwenKephart[asset-reconciliation][perf] Cache common properties on the TimeWindowPartitionsDefinition (#12981)
by @OwenKephartMultiPartitionMapping (#12950)
by @clairelin135[asset-reconciliation] Fix issue with duplicate runs for partitions with in-progress materializations (#13130)
by @OwenKephartupdate telemetry documentation links (#13176)
by @Abbe98[structured config] respect field aliases for resources (#13177)
by @jamiedemaria[refactor] Remove PartitionSetDefinition (#13145)
by @smackeseyTableMetadataValue toy (#13124)
by @smackeseyfix repository decorator typing (#13180)
by @smackeseyadd storybook for @core in addition to @ui (#13184)
by @prhafeat(dbt): follow dbt Core's version support constraints (#13189)
by @rexledesmaFix case where a pipeline load takes long enough that run monitoring kills it (#13156)
by @gibsondanAllow newlines in error messages in Dagit (#13193)
by @gibsondanMake sure we always wait for all grpc server processes to spin down in tests (#13146)
by @gibsondanunpin mlflow from dagster-mlflow (#13194)
by @gibsondanGreatly reduce the number of gRPC calls when doing large asset backfills (#13086)
by @gibsondandon't run mlflow tests on python 3.11 (#13197)
by @gibsondan[pythonic config] Treat Optional as both not-required and Noneable (#12975)
by @benpankow[pythonic config] Add support for enums (#12979)
by @benpankow[pythonic resources] Make direct invocation of ops/assets w/ resources easier (#13002)
by @benpankow[bugfix] make AssetSelection.groups() resolve to only regular assets (#13196)
by @smackesey[fix] Fix behavior of BindResourcesToJobs when jobs have config specified (#13057)
by @benpankow[serdes] remove extraneous @cached_method use (#13206)
by @alangenfeld1.2.4 changelog (#13226)
by @smackesey1.2.4
by @elementl-devtoolsPublished by elementl-devtools over 1 year ago
define_asset_job
now auto-infer their partitions definitions if not explicitly defined.define_asset_job
. This allows putting them on a schedule/sensor.instance
property to the HookContext
object that is passed into Op Hook functions, which can be used to access the current DagsterInstance
object for the hook.create_table_schema_metadata_from_dataframe
function to generate a TableSchemaMetadataValue
from a Pandas DataFrame. Thanks @AndyBys!dag_run
configuration on the integration’s database resources.dagster asset wipe-partitions-status-cache
to see the partitions displayed.1.1.18
, users with a gRPC server that could not access the Dagster instance on user code deployments would see an error when launching backfills as the instance could not instantiate. This has been fixed.build_asset_reconciliation_sensor
from kicking off runs of downstream assets. This has been fixed.build_asset_reconciliation_sensor
is now much more performant in cases where unpartitioned assets are upstream or downstream of static-partitioned assets with a large number of partitions.jobNamespace
field in the Dagster Helm chart when using the CeleryK8sRunLauncher
.en-US
browser locale. This has been fixed.Published by elementl-devtools over 1 year ago
Dagster is now tested on Python 3.11.
Users can now opt in to have resources provided to Definitions
bind to their jobs. Opt in by wrapping your job definitions in BindResourcesToJobs
. This will become the default behavior in the future.
@op(required_resource_keys={"foo")
def my_op(context)
print(context.foo)
@job
def my_job():
my_op()
defs = Definitions(
jobs=BindResourcesToJobs([my_job])
resources={"foo": foo_resource}
Added dagster asset list
and dagster asset materialize
commands to Dagster’s command line interface, for listing and materializing software-defined assets.
build_schedule_from_partitioned_job
now accepts jobs partitioned with a MultiPartitionsDefinition
that have a time-partitioned dimension.
Added SpecificPartitionsPartitionMapping
, which allows an asset, or all partitions of an asset, to depend on a specific subset of the partitions in an upstream asset.
load_asset_value
now supports SourceAsset
s.
[ui] Ctrl+K has been added as a keyboard shortcut to open global search.
[ui] In the run logs table, the timestamp column has been moved to the far left, which will hopefully allow for better visual alignment with op names and tags.
[dagster-dbt] A new node_info_to_definition_metadata_fn
to load_assets_from_dbt_project
and load_assets_from_dbt_manifest
allows custom metadata to be attached to the asset definitions generated from these methods.
[dagster-celery-k8s] The Kubernetes namespace that runs using the CeleryK8sRunLauncher
are launched in can now be configured by setting the jobNamespace
field in the Dagster Helm chart under celeryK8sRunLauncherConfig
.
[dagster-gcp] The BigQuery I/O manager now accepts timeout
configuration. Currently, this configuration will only be applied when working with Pandas DataFrames, and will set the number of seconds to wait for a request before using a retry.
[dagster-gcp] [dagster-snowflake] [dagster-duckdb] The BigQuery, Snowflake, and DuckDB I/O managers now support self-dependent assets. When a partitioned asset depends on a prior partition of itself, the I/O managers will now load that partition as a DataFrame. For the first partition in the dependency sequence, an empty DataFrame will be returned.
[dagster-k8s] k8s_job_op
now supports running Kubernetes jobs with more than one pod (Thanks @Taadas).
AssetValueLoader.load_asset_value()
didn’t load transitive resource dependencies correctly.RunConfig
object with optional config arguments would lead to an error.ScheduleEvaluationContext.scheduled_execution_time
to not be Optional
.OpExecutionContext.partition_time_window
****(thanks @elben10).InputContext.upstream_output.log
is no longer None
when loading a source asset.async def
ops are now invoked with asyncio.run
.TimeWindowPartitionDefinition
now throws an error at definition time when passed an invalid cron schedule instead of at runtime.cron_schedule_timezone
inside of the config for a dbt model would not result in that property being set on the generated FreshnessPolicy
. This has been fixed.GCSComputeLogManager
when the session does not have permissions to generate signed urls.validate_run_config
no longer accepts pipeline_def
or mode
arguments. These arguments refer to legacy concepts that were removed in Dagster 1.0, and since then there have been no valid values for them.Added experimental support for resource requirements in sensors and schedules. Resources can be specified using required_resource_keys
and accessed through the context or specified as parameters:
@sensor(job=my_job, required_resource_keys={"my_resource"})
def my_sensor(context):
files_to_process = context.my_resource.get_files()
...
@sensor(job=my_job)
def my_sensor(context, my_resource: MyResource):
files_to_process = my_resource.get_files()
...
https://github.com/dagster-io/dagster/compare/1.2.1...1.2.2
[dagster-gcp-pandas] add timeout config (#12637)
by @jamiedemaria[docs][tutorial-revamp] Basic tutorial revamp parts 1 through 4 (#12509)
by @tacastillofeat(dbt): add support for
--debug (#12722)
by @rexledesmaDefault useful Dagster helm chart features to on (#12737)
by @gibsondan[For 1.2] Allow both protobuf 3 and 4 in dagster (#12466)
by @gibsondan[toys repo] Export partitioned assets toys (#12733)
by @salazarm[pythonic config] Allow using 'resource_defs' with resource args in assets (#12679)
by @benpankow[refactor] Remove symbols deprecated until 1.2 (#12360)
by @smackesey[dagster-snowflake] fix inconsistencies in snowflake resource (#12633)
by @jamiedemaria[ui] Update permissions for launching jobs and materializing assets (#12681)
by @hellendag[For 1.2] Change default run monitoring settings (#11512)
by @gibsondan[For 1.2] Don't include run/job tags in k8s_job_ops k8s config computations (#12345)
by @gibsondanRemove Partitioned Schedules from docs, fix dagit staleStatusCauses mocks (#12742)
by @smackesey[Sensor Testing] Add "Test again" button (#12735)
by @salazarm[1.2.0] [refactor] Delete DagsterTypeMaterializer (#12516)
by @smackeseychange DynamicPartitionsDefinition.__repr__ (#12754)
by @sryzaRecommended Project Structure Guide (#12656)
by @odette-elementl[pythonic config] Add test showcasing use of Pydantic validators (#12536)
by @benpankowfix serialization of TableRecord (#10731)
by @sryza[dagster-gcp-pandas] revert flaky test (#12748)
by @jamiedemaria[typing/static] PartitionsDefinition covariant type var (#12284)
by @smackesey[rename] LogicalVersion -> DataVersion (#12500)
by @smackesey[rename] LogicalVersion -> DataVersion in gql/dagit (#12501)
by @smackesey[rename] Rename LogicalVersion -> DataVersion in docs (#12503)
by @smackesey[dagster-airflow] add
make_persistent_airflow_db_resource (#12305)
by @Ramshackle-Jamathon[ui] Remove usePermissionsDEPRECATED (#12771)
by @hellendag[Asset Details Page] Auto-select partition in materialization dialog. (#12734)
by @salazarmfeat(databricks)!: remove
create_databricks_job_op (#12600)
by @rexledesmaMake permissions_for_location require keyword args (#12774)
by @gibsondanRemove DynamicPartitionsDefinitions API methods (#12744)
by @clairelin135Set default memory/CPU for new ECS tasks based on runtime platform (#12767)
by @gibsondancron_schedule validation for time window partitions (#12761)
by @prhaupdate some guide titles and descriptions (#12775)
by @sryzaadd primary keys to all run tables (#12711)
by @prhadocument PartitionMappings in partitions concepts page (#12768)
by @sryzamake sure kv/daemon_heartbeat queries explicitly enumerate columns (#12789)
by @prhaupdate API docs for freshness policies (#12788)
by @OwenKephart[apidoc] asset_selection repository -> Definitions (#11303)
by @yuhanFix typing on mem_io_manager (#12791)
by @dpeng817[docs] add note about biquery temp tables (#12747)
by @jamiedemariaRemove deprecated MetadataEntry constructors, update
entry_data internal refs (#12724)
by @smackesey[dagster-airflow] persistent db docs (#12485)
by @Ramshackle-Jamathon[pythonic resources][fix] Fix inheriting attributes when extending Pythonic resources (#12781)
by @benpankowMetadataEntryUnion (#12725)
by @smackesey[docs] - Update RBAC docs for Cloud (#12752)
by @erinkcochran87[dagit] Update the asset partitions / events view after run failures (#12798)
by @bengotow[dagit] Asset node rendering tweaks (#12764, #12765) (#12794)
by @bengotowReflect dynamically-added partitions in asset sidebar immediately (#12778)
by @salazarm[Create Partition Dialog] Autofocus input (#12777)
by @salazarmRemove PartitionMetadataEntry (#12726)
by @smackeseyMaintain a separate cursor in the asset partition cache pointing back to the earliest in progress run (#12782)
by @johannkm[ui] Run logs: make Timestamp the first column (#12805)
by @hellendag[snowflake-pyspark] refactor spark session creation in tests (#12738)
by @jamiedemaria[db io managers] refactor tests to test table creation (#12739)
by @jamiedemaria[dagit] Fix yaml editor help context when indented on a new line (#12537) (#12804)
by @bengotowmake context.upstream_output.log work for source assets (#12787)
by @sryza[typing/runtime] serdes (#12523)
by @smackesey[dagit] Show “Observe” on source asset details pages, not “Materialize" (#12803)
by @bengotow[ui] Disable "Open in launchpad" based on permissions (#12817)
by @hellendagAdd a way to cause partition health data to refetch without passing around refetch callbacks (#12822)
by @salazarm[ui] Fix space oddities on Schedules overview table (#12810)
by @hellendag[ESLint] Update missing graphql variable type rule to check mutations and subscriptions too (#12779)
by @salazarmTest get and set new asset cache fields (#12820)
by @johannkm[duckdb-polars] fix readme (#12499)
by @jamiedemaria[freshness-refactor][1/n] Separate part of CachingInstanceQueryer into CachingDataTimeResolver (#12809)
by @OwenKephart[ui] Affordance for copying asset key (#12829)
by @hellendag[fix][dagit] Allow resources page to scroll (#12827)
by @benpankow[async op] use asyncio.run (#12785)
by @alangenfeld[dagster-airflow] remove double encoding of timezones (#12811)
by @Ramshackle-Jamathon[ui] Support Ctrl+K to trigger search (#12840)
by @hellendagVirtualize TagSelector component (#12841)
by @salazarm[Launchpad] Allow adding new dynamic partition from launchpad UI (#12757)
by @salazarm[tox] download latest pip (#12821)
by @alangenfeld[dagster-snowflake] fix permissions issue when creating schemas (#12802)
by @jamiedemaria[pythonic config] Special rendering for env vars in UI (#12446)
by @benpankownothing input toy (#12849)
by @sryzalatestRunForPartition gql (#12845)
by @johannkm[dagster-dbt] add support for node_info_to_definition_metadata_fn (#12831)
by @OwenKephartRemove nothing types from asset graph (#12848)
by @salazarm[TagSelector] Allow filtering tags in dropdown (#12844)
by @salazarmAdd a tooltip on asset nodes in the graph (#12853)
by @salazarm[pythonic resources] Keep track of utilized env vars in external repo data (#12557)
by @benpankowChangelog 1.2.0 (#12851)
by @dpeng817add a local download url fallback for GCS (#12815)
by @prha[freshness-refactor][2/n] Reorganize/Rename/Remove/Document CachingInstanceQueryer methods (#12813)
by @OwenKephartfix type annotation for partition_time_window (#12850)
by @sryzaRevert "fix type annotation for partition_time_window" (#12860)
by @sryzaAutomation: versioned docs for 1.2.0
by @elementl-devtoolsAdd volatility property to CachingStaleStatusResolver (#12855)
by @smackeseyUpdate CHANGES.md (#12864)
by @Ramshackle-Jamathon[docs] - Add doc checklist to PR template (#12861)
by @erinkcochran87Fix partition_time_window type error (#12838)
by @elben10Add 1.2.0 changes to migration.md (#12866)
by @clairelin135Fix config w/ dynamic partitions error (#12710)
by @clairelin135[1.2] make ScheduleEvaluationContext.scheduled_execution_time non-optional (#12623)
by @sryzaasset materialize CLI (#12691)
by @sryza[refactor] serdes pack/deserialize APIs (#12524)
by @smackesey[refactor] MayHaveInstanceWeakRef error on access non-existent _instance (#12553)
by @smackeseyasset list command (#12790)
by @sryzafixups to config_schedule.py (#12874)
by @sryzaFix some upcoming lint errors in docstrings (#12704)
by @charliermarsh[db io managers] support self dependent assets in db io managers (#12700)
by @jamiedemariaFix markdown links in changelog (#12877)
by @johannkmFix dagster-k8s job execution when a job has more than 1 pod (#12731)
by @Taadastake grpcio-health-checking pins out of backcompat suite (#12876)
by @gibsondanFix ruff (#12879)
by @gibsondanneed to explicitly return a known existing column for updates (#12883)
by @prhaFix docs build (#12884)
by @smackeseyadd 1.2.1 changelog (#12887)
by @prhaadd SpecificPartitionsPartitionMapping (#12878)
by @sryzaAutomation: versioned docs for 1.2.1
by @elementl-devtools[docs] - Apply style guide to Asset versioning and caching guide (#12895)
by @erinkcochran87[dagster-ui] Sync the search filter to the query string on most pages (#12899)
by @bengotowon asset backfills, propagate backfill tags to runs (#12886)
by @sryzaSupport SourceAssets in load_asset_value (#12859)
by @sryzaFix transitive resource deps for asset value loader (#12872)
by @dpeng817[typing/static] telemetry (#12714)
by @smackeseyadd docs page on asset selection syntax (#12871)
by @sryzaAdd dropdown for materializing using launchpad (#12852)
by @salazarm[ui] Refactor permissions objects (#12903)
by @hellendagavoid custom PartitionMapping on IO manager concept page (#12902)
by @sryza[bk] Add changelog check step (#12865)
by @benpankowtutorial intro tweaks (#12780)
by @sryzaMove docs style guidelines from pull request template to contribution guide (#12875)
by @sryzaremove support for legacy APIs from validate_run_config (#12881)
by @sryzamake reconciliation perf test names easier to scan, and turn them all on (#12908)
by @sryzaMake fetch_flattened_time_window_ranges support arbitrary statuses (#12863)
by @johannkmBuild schedule from multipartitioned job (#12907)
by @clairelin135[ui] Clean up temporary permission code (#12923)
by @hellendag[op] error on multiple values for argument (#12473)
by @alangenfeldtest assets in dynamic re-execution (#12651)
by @alangenfeldSupport requiring resources in sensors, experimentally (#12401)
by @benpankowSupport requiring resources in schedules, experimentally (#12697)
by @benpankow[pythonic config] support Pydantic constrained types (#12703)
by @benpankow[docs] Add env var docs to Pythonic config experimental docs (#12740)
by @benpankow[pythonic resources] Surface utilized env vars in GraphQL (#12558)
by @benpankow[dagit] Allow the lineage tab to appear before the asset definition has loaded (#12191) (#12814)
by @bengotowSupport defaulting to selected partitions for Multi-partitioned Assets (#12856)
by @salazarmfix execution plan step output collisions (#12799)
by @alangenfeld[pythonic config] Fix using RunConfig with optional config fields (#12933)
by @benpankow[Partitions] Use OrdinalPartitionSelector for STATIC partitioned assets (#12919)
by @salazarmfix ruff (#12936)
by @alangenfeldUnpin grpcio on dagster for python 3.11 (#12917)
by @gibsondanAllow setting the job_namespace for the celery k8s run launcher (#12911)
by @gibsondan[dagster-dbt] add cron_schedule_timezone to set of properties parsed from dbt config (#12904)
by @OwenKephartRedesign partition backfill modal (#12928)
by @salazarmFix error reporter infinite loop (#12941)
by @salazarmAdd Python 3.11 to dagster buildkite (#12723)
by @gibsondanBind top-level resources to jobs at Definitions-time (#12430)
by @benpankow[asset-reconciliation] Do not reconcile partitions involved in active asset backfills (#12926)
by @OwenKephart[asset-reconciliation] Fix behavior that could cause overly-aggressive updates (#12940)
by @OwenKephartimprove execution plan snapshot use in execute_run (#12660)
by @alangenfeldopen backfill options section by default (#12957)
by @salazarm[Partition Backfill Dialog] Update backfill options spacing (#12959)
by @salazarmResolve transitive resource deps for schedules and sensors (#12946)
by @benpankowchangelog 1.2.2 (#12960)
by @yuhanFix pandera tests (#12947)
by @smackesey[dagit] Use pure-asset backfills instead of hidden asset job backfills (#12948)
by @bengotow1.2.2
by @elementl-devtoolsPublished by elementl-devtools over 1 year ago
dagster instance migrate
after upgrading to 1.2.0
.Published by Ramshackle-Jamathon over 1 year ago
dagster dev
command that can be used to run both Dagit and the Dagster daemon in the same process during local development. [docs]DynamicPartitionsDefinition
allows partitioning assets dynamically - you can add and remove partitions without reloading your definitions (experimental). [docs]TimeWindowPartitionMapping
now accepts start_offset
and end_offset
arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets [docs].dagster-wandb
with Weights & Biases allows you to orchestrate your MLOps pipelines and maintain ML assets with Dagster. [docs]dagster-snowflake-pyspark
allows you to store and load PySpark DataFrames as Snowflake tables using the snowflake_pyspark_io_manager
. [docs]dagster-gcp-pandas
and dagster-gcp-pyspark
allow you to store and load Pandas and PySpark DataFrames as BigQuery tables using the bigquery_pandas_io_manager
and bigquery_pyspark_io_manager
. [docs]dagster-airflow
integration library was bumped to 1.x.x, with that major bump the library has been refocused on enabling migration from Airflow to Dagster. Refer to the docs for an in-depth migration guide.create_databricks_run_now_op
), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op
).create_databricks_job_op
op factory is now deprecated.Stay tuned, as this is only the first part of the overhaul. We’ll be adding more chapters - including automating materializations, using resources, using I/O managers, and more - in the next few weeks.
@graph_asset
and @graph_multi_asset
.project_fully_featured
example now uses the built in DuckDB and Snowflake I/O managers.make_persistent_airflow_db_resource
this enables support for Airflow features like pools and cross-dagrun state sharing. In particular retry-from-failure now works for jobs generated from Airflow DAGs.BigQueryPandasTypeHandler
now uses google.bigquery.Client
methods load_table_from_dataframe
and query
rather than the pandas_gbq
library to store and fetch DataFrames.args
instead of both command
and args
for user code deployments, allowing to include a custom ENTRYPOINT in your the Dockerfile that loads your code.protobuf<4
pin in Dagster has been removed. Installing either protobuf 3 or protobuf 4 will both work with Dagster.@graph_asset
and @graph_multi_asset
now support passing metadata (thanks @askvinni)!@graph_asset
and @graph_multi_asset
to be ignored.TableRecord
.@multi_asset
and other functions would register as type errors for mypy and other static analyzers.start_date
would not be converted correctly causing runs to fail.dagster instance migrate
:
id
column to the kvs
, daemon_heartbeats
and instance_info
tables, enforcing that all tables have a primary key.The minimum grpcio
version supported by Dagster has been increased to 1.44.0 so that Dagster can support both protobuf
3 and protobuf
4. Similarly, the minimum protobuf
version supported by Dagster has been increased to 3.20.0. We are working closely with the gRPC team on resolving the upstream issues keeping the upper-bound grpcio
pin in place in Dagster, and hope to be able to remove it very soon.
Prior to 0.9.19, asset keys were serialized in a legacy format. This release removes support for querying asset events serialized with this legacy format. Contact #dagster-support for tooling to migrate legacy events to the supported version. Users who began using assets after 0.9.19 will not be affected by this change.
[dagster-snowflake] The execute_query
and execute_queries
methods of the SnowflakeResource
now have consistent behavior based on the values of the fetch_results
and use_pandas_result
parameters. If fetch_results
is True, the standard Snowflake result will be returned. If fetch_results
and use_pandas_result
are True, a pandas DataFrame will be returned. If fetch_results
is False and use_pandas_result
is True, an error will be raised. If both are False, no result will be returned.
[dagster-snowflake] The execute_queries
command now returns a list of DataFrames when use_pandas_result
is True, rather than appending the results of each query to a single DataFrame.
[dagster-shell] The default behavior of the execute
and execute_shell_command
functions is now to include any environment variables in the calling op. To restore the previous behavior, you can pass in env={}
to these functions.
[dagster-k8s] Several Dagster features that were previously disabled by default in the Dagster Helm chart are now enabled by default. These features are:
dagster/max_retries
tag set. You can configure a global number of retries in the Helm chart by setting run_retries.max_retries
to a value greater than the default of 0.Each of these features can be disabled in the Helm chart to restore the previous behavior.
[dagster-k8s] The experimental k8s_job_op
op and execute_k8s_job
functions no longer automatically include configuration from a dagster-k8s/config
tag on the Dagster job in the launched Kubernetes job. To include raw Kubernetes configuration in a k8s_job_op
, you can set the container_config
, pod_template_spec_metadata
, pod_spec_config
, or job_metadata
config fields on the k8s_job_op
(or arguments to the execute_k8s_job
function).
[dagster-databricks] The integration has now been refactored to support the official Databricks API.
create_databricks_job_op
is now deprecated. To submit one-off runs of Databricks tasks, you must now use the create_databricks_submit_run_op
.databricks_client
resource must now begin with https://
.LogicalVersion
has been renamed to DataVersion
and LogicalVersionProvenance
has been renamed to DataProvenance
.DynamicPartitionsDefinition
to add, remove, and check for existence of partitions have been removed. Refer to documentation for updated API methods.MetadataEntry
have been removed.DagsterTypeMaterializer
, DagsterTypeMaterializerContext
, and @dagster_type_materializer
have been removed.PartitionScheduleDefinition
has been removed.RunRecord.pipeline_run
has been removed (use RunRecord.dagster_run
).DependencyDefinition.solid
has been removed (use DependencyDefinition.node
).pipeline_run
argument to build_resources
has been removed (use dagster_run
)iteritems
usage was removed and changed to the recommended items
within dagster-snowflake-pandas
(thanks @sethkimmel3)!@asset_graph
decorator (thanks @simonvanderveldt)!DataVersions
can now be returned on Output
OpExecutionContext.get_asset_provenance
Picking a github issue
was not correctly rendering, this has been fixed (thanks @Sedosa)!Published by elementl-devtools over 1 year ago
build_asset_reconciliation_sensor
.TimeWindowPartitionMapping
s with offsets. (Thanks Sean Han!)runtimePlatform
field for the task definitions of the runs that it launches, allowing it to launch runs using Windows Docker images.create_databricks_run_now_op
), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op
). See the new Databricks guide for more details.DuckDBPolarsTypeHandler
for use with build_duckdb_io_manager
, which allows loading / storing Polars DataFrames from/to DuckDB. (Thanks Pezhman Zarabadi-Poor!)bigquery_pyspark_io_manager
.DataFrame
.AssetSelection
which matched no assets was passed into define_asset_job
, the resulting job would target all assets in the repository. This has been fixed.build_schedule_from_partitioned_job
.fs_io_manager
to store multipartitioned materializations in directory levels by dimension. This resolves a bug on windows where multipartitioned materializations could not be stored with the fs_io_manager
.context.partition_key
would raise an error when executing on a partition range within a single run via Dagit.Definitions reloaded
messages on fresh page loads.location
information.ProgrammingError
was imported from the wrong library, this has been fixed. Thanks @herbert-allium!Output
objects rather than using Dagster’s auto-generated versions.get_asset_provenance
method on OpExecutionContext
allows fetching logical version provenance for an arbitrary asset key.examples/assets_dynamic_partitions
folderhttps://github.com/dagster-io/dagster/compare/1.1.20...1.1.21
dagster-census api docs (#12413)
by @yuhangraph_asset and graph_multi_asset decorators (#10152)
by @sryza[dagster-snowflake-pyspark] fix bug loading partitions (#12472)
by @jamiedemariaadd graphql fields for querying run tags (#12409)
by @prhaAdd stale status causes (#11953)
by @smackeseyfix (#12477)
by @salazarmUpdate Contributing doc with instructions for ruff/pyright (#12481)
by @smackesey[bigquery] mark bigquery io manager experimental (#12479)
by @jamiedemariaadd partial tag autocomplete for run filter input (#12410)
by @prhaSupport env valueFrom in Helm chart (#12425)
by @johannkmUpdate GQL to expose StaleStatus and StaleStatusCause (#11952)
by @smackeseyremove timestamp comparisons of code location entries to reduce OSS dagit replica spam (#12407)
by @prhaFix state status logical version test (#12484)
by @smackeseyfix ruff (#12486)
by @alangenfelduse opt_nullable_mapping for dagster library versions (#12487)
by @alangenfeldclarify error when op is missing argument for In (#12456)
by @sryzaRemove existing RunConfig class (#12488)
by @benpankow[pythonic resources] Clean up initialization of env vars, treat resource objects as immutable (#12445)
by @benpankow[structured config] Add support for Selectors w/ pydantic discriminated unions (#11280)
by @benpankowAllow setting logical version inside op (#12189)
by @smackeseyReplace usages of
nslookupwith
nc for user deployments (#11033)
by @michaeljguarinoRevert "Replace usages of
nslookupwith
nc for user deployments (#11033)"
by @johannkm[pythonic resources] Last set of class renames (#12490)
by @benpankow[dagster-azure] Add support for DefaultAzureCredential for adls2_resource (#11309)
by @mpicardAdd example of customizing task role and execution role arn to the ECS agent docs (#12491)
by @gibsondanadd dagster-duckdb-polars library (#12197)
by @pzarabadip[draft][pythonic config][docs] Introduce intro to Resources doc utilizing Pythonic resources (#12260)
by @benpankow[pythonic config] Add structured RunConfig object for specifying runtime, job config (#11965)
by @benpankow[draft][pythonic config][docs] Introduce intro to Config doc utilizing Pythonic config (#12349)
by @benpankow1.1.20 changelog (#12506)
by @benpankowrefactor(databricks): lift polling methods up to the client (#12382)
by @rexledesma[fix] fix sphinx airflow version parsing (#12507)
by @benpankowAutomation: versioned docs for 1.1.20
by @elementl-devtools[dagit] Expose range-based asset health, use it for partition status rendering (#12302)
by @bengotow[dagit] Use range-based asset health for asset partitions / job partitions pages (#12434)
by @bengotowFix 1.1.20 changelog codeblock (#12525)
by @johannkm[dagit] Delete generated GraphQL types before regenerating (#12518)
by @hellendag[refactor] Delete build_solid_context (#12513)
by @smackesey[typing/static] serdes (#12522)
by @smackesey[refactor] make ResolvedRunConfig.to_dict use "ops" (#12514)
by @smackeseyImprove multipartition performance for
get_partition (#12431)
by @clairelin135feat(databricks): override user agent in resource (#12526)
by @rexledesmaBump typing-extensions dep to >=4.4.0 (#12529)
by @smackesey[docs] Snowflake reference page fixes (#12455)
by @jamiedemaria[bugfix] Make assets downstream of partitions never stale in dagit (#12528)
by @smackeseyEnable internal testing for writing asset cached status data (#12497)
by @clairelin135[typing] fix typing in daemon tests (#12475)
by @dpeng817dynamic partitions toy (#12533)
by @sryzapare down multi-partition runtime type checking in upath IO manager (#12508)
by @sryza[dagit] Upgrade to Jest 29, allow more time for coverage collection (#12534)
by @bengotowAdd description to invariant check (#12496)
by @CodeMySky[fix] Correctly resolve asset jobs with empty selections (#12531)
by @OwenKephart[dagit] Support backfills on partition-mapped asset selections (#12458)
by @bengotow[dagit] Remove unnecessary usage of <TestProvider> (#12519)
by @bengotowFix multipartitions w/ fs_io_manager on windows (#12414)
by @clairelin135Fix master (#12550)
by @clairelin135Booleans that check permissions for specific objects and automatically incorporate per-location checks (#12548)
by @gibsondandocs(dagster-dbt): add example to retrieve asset keys from dbt selection (#12323)
by @rexledesmaFix IO manager doc snippets (#12545)
by @clairelin135fix setup.cfg in create package (#12559)
by @jamiedemariaFetch multipartition key from context methods (#12512)
by @clairelin135Include partition in asset materialization planned event (#12333)
by @johannkmhandle multiple non-time-window partitions in db io managers (#12517)
by @sryzaAdd runtime_platform to EcsRunLauncher (#12566)
by @gibsondanTag Selector Component (#12564)
by @salazarmStorybook for LaunchAssetChoosePartitionsDialog (#12567)
by @salazarmUpdate setting-up-alerts.mdx (#12582)
by @johannkmMove useViewport to
uiform
core (#12578)
by @salazarm[ui] Always show "Reload all" button (#12588)
by @hellendagBump minimist from 1.2.5 to 1.2.8 in /js_modules/dagit (#12595)
by @dependabot[bot]Expose partition definition name (#12585)
by @salazarmadd dynamic partitions example (#12494)
by @sryza[dagit] Do not capitalize tag keys in the Dagit op metadata panel (#12591)
by @bengotow[graphql] empty numPartitions and partitionNames for cross-partitioning backfills (#12462)
by @sryza[dagit] Use last path component of linked assets on the asset graph + add tooltips (#12590)
by @bengotow[dagster-snowflake] Fix ProgrammingError import in snowflake_io_manager (#12576)
by @herbert-alliumremove dagster-cloud dep from dynamic partitions example (#12603)
by @sryzafix future ticks for partitioned schedules (#12601)
by @sryzafix loading partitioned source assets in non-partitioned non-asset jobs (#12586)
by @sryzaUpdate stale reasons to remove UNKNOWN, add MISSING (#12584)
by @smackeseyGQL Add dynamic partition mutation (#12562)
by @clairelin135Add asset provenance access to OpExecutionContext (#12467)
by @smackeseyfeat(databricks): add basic op implementations (#12492)
by @rexledesmadocs(databricks): add guide for Databricks integration (#12326)
by @rexledesmamigrate some tests away from AssetGroup (#12606)
by @sryzadivest permissions for editing sensor from updating sensor cursor (#12589)
by @dpeng817[dagit] Fix tests pinned to February (#12616)
by @bengotowMove tests, fixtures, and storybooks into folders (#12614)
by @salazarmAdd root stale causes for assets (#12619)
by @smackeseydocs for graphs that depend on assets (#12597)
by @sryza[UI] Dynamic partition creation and selection (#12615)
by @salazarm[asset-reconciliation][perf] Do not get/set known_used_data in asset reconciliation loop (#12433)
by @OwenKephart[dagster-gcp-pyspark] Add BigQuery PySpark type handler (#12398)
by @jamiedemariaAutomating pipeline guide (#12547)
by @odette-elementlCapture db timeout errors for tag key queries (#12596)
by @prhaGet partitions with planned but not completed materializations, the event log query (#12404)
by @johannkmAdd failures to partition status cache (#12599)
by @johannkm[bigquery-pandas] pipe gcp location through to type handler (#12587)
by @jamiedemaria[docs, bigquery] tutorial and reference guide (#12452)
by @jamiedemariamake freshness policies work with graph_asset and graph_multi_asset (#12630)
by @sryzaRevert "Add failures to partition status cache (#12599)"
by @johannkmRevert "Get partitions with planned but not completed materializations, the event log query (#12404)"
by @johannkmModify
get_first_partition_window to account for offset (#12504)
by @clairelin135[dagit] Switch to staleStatus for stale tags, display causes in tooltips (#12611)
by @bengotowChange format of staleness root causes and remove old resolvers (#12628)
by @smackeseyHelm: Support valueFrom in user deployments (#12644)
by @johannkmUpdate asset versioning guide (#12659)
by @smackeseyupdate CHANGES.md for 1.1.21 (#12662)
by @sryzadocs(databricks): update guide based on feedback (#12669)
by @rexledesma1.1.21
by @elementl-devtoolsPublished by benpankow over 1 year ago
The new @graph_asset
and @graph_multi_asset
decorators make it more ergonomic to define graph-backed assets.
Dagster will auto-infer dependency relationships between single-dimensionally partitioned assets and multipartitioned assets, when the single-dimensional partitions definition is a dimension of the MultiPartitionsDefinition
.
A new Test sensor
/ Test schedule
button that allows you to perform a dry-run of your sensor / schedule. Check out the docs on this functionality here for sensors and here for schedules.
[dagit] Added (back) tag autocompletion in the runs filter, now with improved query performance.
[dagit] The Dagster libraries and their versions that were used when loading definitions can now be viewed in the actions menu for each code location.
New bigquery_pandas_io_manager
can store and load Pandas dataframes in BigQuery.
[dagster-snowflake, dagster-duckdb] SnowflakeIOManagers and DuckDBIOManagers can now default to loading inputs as a specified type if a type annotation does not exist for the input.
[dagster-dbt] Added the ability to use the “state:” selector
[dagster-k8s] The Helm chart now supports the full kubernetes env var spec for Dagit and the Daemon. E.g.
dagit:
env:
- name: “FOO”
valueFrom:
fieldRef:
fieldPath: metadata.uid
ruff
, pyright
).https://github.com/dagster-io/dagster/compare/1.1.19...1.1.20
disable check_same_thread on in-memory sqlite storage (#12229)
by @alangenfeld[direct invoke] yield implicit Nothing Output (#12309)
by @alangenfeldFix multipartitions run length encoding error (#12329)
by @clairelin135[freshness-policies] Allow setting freshness policies when using graph-backed assets (#12357)
by @OwenKephartAdd skip reason to asset sensor (#12343)
by @OwenKephartFix partitions backfill deserialization error (#12238)
by @clairelin135Move CachingRepositoryData.from_list and from_dict into standalone function (#12321)
by @schrocknrefactor(databricks): divest from databricks_api in favor of databricks-cli (#12153)
by @rexledesma nullsafe array index access (#12362)
by @salazarmChange schedule button text (#12361)
by @dpeng817[dagit] add full serialized error to graphql errors (#12228)
by @alangenfeld[dagster-pandas][dagster-pandera] assign a typing_type for generated pandas dataframe DagsterTypes (#12363)
by @OwenKephart[typing/static] Execution API types (#12330)
by @smackesey[refactor] DependencyDefinition renames (#12338)
by @smackesey[refactor] execute_step renames (#12354)
by @smackesey[typing/runtime] Standardize StepInputSource.load_input_object (#12342)
by @smackesey[refactor] Delete NodeInput.solid_name (#12339)
by @smackesey[refactor] NodeDefiniton.iterate_solid_defs -> iterate_op_defs (#12336)
by @smackesey[refactor] GraphDefinition method renames (#12335)
by @smackesey[2/n][structured config] Enable struct config resources, IO managers to depend on other resources (#11645)
by @benpankow[refactor] local var/private arg solid -> node (#12337)
by @smackeseyDagsterLibraryRegistry (#12266)
by @alangenfeld[refactor] misc core solid -> node renames (#12368)
by @smackesey[refactor] dagster._core.definitions.solid_container -> node_container (#12369)
by @smackesey[refactor] Assorted local var solid -> node (#12370)
by @smackeseyFix intermittent dynamic partitions table SQLite concurrency error (#12367)
by @clairelin135change storage signature for run tags (#12348)
by @prha1.1.19 Changelog (#12378)
by @OwenKephartguide to how assets relate to ops and graphs (#12204)
by @sryza[pythonic config] Rename pythonic config classes (#12235)
by @benpankowupdates tests to handle new kubernetes resources field (#12395)
by @alangenfeld[structured config] Migrate resources from project-fully-featured to struct config resources (#11785)
by @benpankowrefactor(databricks): add types to databricks.py (#12364)
by @rexledesmarefactor(databricks): consolidate types (#12366)
by @rexledesmaadd dagster_libraries to ListRepositoriesResponse (#12267)
by @alangenfeld[graphql] add RepositoryLocation.dagsterLibraryVersions (#12268)
by @alangenfeld[dagit] add dagster libraries menu to code location row (#12315)
by @alangenfeld1.1.19 changelog: reorder code block (#12402)
by @yuhanrefactor(databricks): use databricks_cli's raw api client (#12377)
by @rexledesma[docs] [snowflake] Add partitions to snowflake guide (#12231)
by @jamiedemariaAdd valid start time check to materialized time partitions subsets (#12403)
by @clairelin135Add api docs for some PartitionsDefinition and PartitionMapping classes (#12365)
by @sryzaAdd text to timestamp dropdown (#12379)
by @dpeng817[refactor] IExecutionStep.solid_handle -> node_handle (#12371)
by @smackesey[asset-reconciliation] Factor in more run statuses (#12412)
by @OwenKephart[refactor] *ExecutionContext.solid_config -> op_config (#12372)
by @smackesey[refactor] ResolvedRunConfig.solids -> ops (#12373)
by @smackeseyAutomation: versioned docs for 1.1.19
by @elementl-devtools[refactor] assorted Dagstermill renames (#12380)
by @smackesey[refactor] Context
solid renames (#12374)
by @smackeseylambda_solid -> solid (#10816)
by @smackesey[refactor] Assorted pipeline_run -> dagster_run (#12383)
by @smackeseyMake a script to template out new dagster packages (#12389)
by @jamiedemaria[library template] add registry call (#12418)
by @alangenfeld[db io managers] connection refactor (#12258)
by @jamiedemariaCode location alerting docs (#12411)
by @dpeng817[refactor] remove @solid decorator (#10952)
by @smackesey[refactor] Delete PipelineRunsFilter (#12384)
by @smackesey[db io managers] add default_load_type (#12356)
by @jamiedemaria[refactor] Delete RunRecord.pipeline_run (#12385)
by @smackesey[refactor] pipeline_run_from_storage -> dagster_run_from_storage (#12386)
by @smackesey[Docs RFC] Dynamic Partitions (#12227)
by @clairelin135Auto infer multipartition <-> single dimension mapping (#12400)
by @clairelin135[refactor] execution pipeline_run -> dagster_run (#12388)
by @smackesey[test-api-update] execution_tests/dynamic_tests (#12427)
by @smackeseyConsider the run worker unhealthy is the job has no active pods but the run is in a non-terminal state (#11510)
by @gibsondanfix: only inspect schema when we may create tables (#12269)
by @plaflammeMore helpful asset key mismatch errors (#12008)
by @benpankowAdd docs for testing schedules/sensors via UI (#12381)
by @dpeng817document missing breaking change in 1.1.19 changelog (#12424)
by @sryzaBigQuery IO manager (#11425)
by @jamiedemaria[dagster-gcp-pandas] API docs fix (#12450)
by @jamiedemaria[graphql] launch backfills over assets with different partitionings, if all roots have same partitioning (#11827)
by @sryzaFix resolution error with input manager key and custom dagster type (#12449)
by @clairelin135[dagster-dbt] Add ability to use the "state:" selector (#12432)
by @OwenKephartRevert "More helpful asset key mismatch errors (#12008)" (#12459)
by @benpankow[dagit] Add missing React keys to prevent new warning toasts (#12210)
by @bengotow[CustomConfirmationDialog] Allow overriding the button text (#12444)
by @salazarmadd another todo to create_dagster_package (#12453)
by @jamiedemaria[dagster-gcp-pandas] register library in init (#12469)
by @jamiedemaria[bugfix] fix projected logical version resolution for asset downstream of self-dep (#12443)
by @smackeseyfix: resolve correct legacy arguments for emr pyspark step launcher (#12419)
by @rexledesma[dagster-dbt] Add missing files for tests (#12471)
by @OwenKephartAdd task_role_arn and execution_role_arn to EcsContainerContext (#12358)
by @gibsondandagster-census api docs (#12413)
by @yuhangraph_asset and graph_multi_asset decorators (#10152)
by @sryza[dagster-snowflake-pyspark] fix bug loading partitions (#12472)
by @jamiedemariaAdd stale status causes (#11953)
by @smackeseyUpdate Contributing doc with instructions for ruff/pyright (#12481)
by @smackesey[bigquery] mark bigquery io manager experimental (#12479)
by @jamiedemariafix (#12477)
by @salazarmadd graphql fields for querying run tags (#12409)
by @prhaadd partial tag autocomplete for run filter input (#12410)
by @prhaUpdate GQL to expose StaleStatus and StaleStatusCause (#11952)
by @smackeseySupport env valueFrom in Helm chart (#12425)
by @johannkmFix state status logical version test (#12484)
by @smackeseyuse opt_nullable_mapping for dagster library versions (#12487)
by @alangenfeldfix ruff (#12486)
by @alangenfeld[pythonic resources] Clean up initialization of env vars, treat resource objects as immutable (#12445)
by @benpankow[structured config] Add support for Selectors w/ pydantic discriminated unions (#11280)
by @benpankow[pythonic resources] Last set of class renames (#12490)
by @benpankowAllow setting logical version inside op (#12189)
by @smackeseyAdd example of customizing task role and execution role arn to the ECS agent docs (#12491)
by @gibsondanRemove existing RunConfig class (#12488)
by @benpankow[pythonic config] Add structured RunConfig object for specifying runtime, job config (#11965)
by @benpankow[draft][pythonic config][docs] Introduce intro to Resources doc utilizing Pythonic resources (#12260)
by @benpankow[draft][pythonic config][docs] Introduce intro to Config doc utilizing Pythonic config (#12349)
by @benpankow1.1.20 changelog (#12506)
by @benpankow[fix] fix sphinx airflow version parsing (#12507)
by @benpankow1.1.20
by @elementl-devtoolsPublished by OwenKephart over 1 year ago
FreshnessPolicy
object now supports a cron_schedule_timezone
argument.AssetsDefinition.from_graph
now supports a freshness_policies_by_output_name
parameter.@asset_sensor
will now display an informative SkipReason
when no new materializations have been created since the last sensor tick.AssetsDefinition
now has a to_source_asset
method, which returns a representation of this asset as a SourceAsset
.from dagster import asset, job, op
@asset
def emails_to_send():
...
@op
def send_emails(emails) -> None:
...
@job
def send_emails_job():
send_emails(emails_to_send.to_source_asset())
--dagit-host/-h
argument to the dagster dev
command to allow customization of the host where Dagit runs.input_manager_key
to an asset’s input, incorrect resource config could be used when loading that input. This has been fixed.runK8sConfig
in the Dagster Helm chart would not pass configuration through to pods launched using the k8s_job_executor
.execute_k8s_job
op downstream of a dynamic output would result in k8s jobs with duplicate names being created. This has been fixed.asset_key
, asset_partitions
, and asset_partitions_defs
arguments on Out
.@multi_asset
no longer accepts Out
values in the dictionary passed to its outs
argument. This was experimental and deprecated. Instead, use AssetOut
.top_level_resources
argument to the repository
decorator has been renamed to _top_level_resources
to emphasize that it should not be set manually.load_asset_values
now accepts resource configuration (thanks @Nintorac!)UPathIOManager
, paths with the "."
character in them would be incorrectly truncated, which could result in multiple distinct objects being written to the same path. This has been fixed. (Thanks @spenczar!)https://github.com/dagster-io/dagster/compare/1.1.18...1.1.19
feat(dagster-dbt): support dbt-core 1.4.x (#11902)
by @rexledesmaFixes/improvements for pyright script (#12175)
by @smackeseynux examples in oss 1/: add dagster_cloud.yaml to prep cloud nux onboarding (#12172)
by @yuhan[dagster-dbt] [1/2] Add streaming entrypoint for dbt cli execution (#12086)
by @OwenKephartFix import (#12185)
by @OwenKephart(tick-testing 3/6) type annotations to schedules gql (#12057)
by @dpeng817(tick-testing 4/6) mutation to dry-run sensor (#11616)
by @dpeng817[db io managers] support static partitions (#12129)
by @jamiedemariaPass correct config to io manager when using input_manager_key (#12053)
by @jamiedemaria[dagster-dbt] [2/2] Add ability to stream events while executing dbt assets (#12100)
by @OwenKephartAgent downtime alert docs (#12186)
by @johannkm(tick-testing 5/6) Add SensorType (#12021)
by @dpeng817(tick-testing 6/6) Add dry run mutations for schedules (#11869)
by @dpeng817[dagit] Add some icons for Cloud (#12187)
by @hellendag[1/n][structured config] Add ability to runtime-configure struct-config resources (#11773)
by @benpankowfix pyright (#12193)
by @jamiedemaria[dagit] Storybooks for asset partition and event details + dayJS fix (#12178)
by @bengotow[dagster-wandb] Integration with Weights & Biases (#10470)
by @chrishisteStore the instance ref on the grpc server class if it's present, so that grpc api calls can use it if it's there (#12194)
by @gibsondanW&B integration follow up 2/ update example to defs (#12059)
by @yuhannux examples in oss 2/: add required env vars in README's front matter (#12154)
by @yuhanSensor testing UI (#12148)
by @salazarmvalidate time partition keys when adding to a TimeWindowPartitionsSubset (#12195)
by @OwenKephartMake Partition GRPC calls with
instance_ref args optional (#12196)
by @clairelin135Add public api doc for create_repository_using_definitions_args, update typehints, and cleanup Definitions docstring (#12176)
by @schrockn[dagster-airflow] [docs] migration guide updates/considerations (#12198)
by @Ramshackle-Jamathon[dagster-airflow] refactor airflow_db resources (#12202)
by @Ramshackle-JamathonFix alert docs (#12208)
by @johannkmTemporarily disable pyright in BK (#12212)
by @smackeseyRename "Loading multiple repositories" section in workspace files docs (#12201)
by @gibsondanMake top_level_resources argument _top_level_resources (#12211)
by @schrocknRevert "nux examples in oss 2/: add required env vars in README's front matter" (#12199)
by @yuhanPyright config/script fixes and improvements (#12206)
by @smackeseyAdd guide for observable source assets and versioning (#12118)
by @schrockn1.1.18 Changelog (#12217)
by @smackeseyFix dagster-wandb placeholder (#12222)
by @smackeseyAdd "--dagit-host" arg to dagster dev (#12220)
by @gibsondandiff against origin/master in quick_pyright (#12200)
by @sryzaRe-enable pyright in BK (#12214)
by @smackeseyRemove restart-on-failure from local docker agent guide (#12224)
by @gibsondan[dagit] Storybooks for MetadataEntry rendering (#12177)
by @bengotowAdd airbyte guide to experimental guide index (#12207)
by @smackeseyAutomation: versioned docs for 1.1.18
by @elementl-devtoolsSchedules Testing UI (#12160)
by @salazarmKeep dots in paths in UPathIOManager (#12174)
by @spenczarremove Out.asset_key (#12221)
by @sryza[structured config] Fix env vars not working with direct Resource instantiation (#11468)
by @benpankowallow using SourceAssets to satisfy node inputs in non-asset jobs (#12091)
by @sryzaAdd observe function (source asset analogue to materialize) (#11996)
by @smackeseytreat pyright warnings as errors for buildkite purposes (#12239)
by @gibsondanUpdate serverless docs with new run isolation default (#12225)
by @johannkmdocs(dbt-cloud): add instructions to cache dbt Cloud compilation (#11793)
by @rexledesmaAllow
dagster api grpc --max-workers (#12246)
by @smackeseyAllow Ursula to test dynamic partition writes (#12215)
by @clairelin135[docs] fix dbt tutorial error (#12255)
by @jamiedemaria[docs] - Correct name of freshness sensor context object (#12250)
by @erinkcochran87document system tags on metadata/tags page (#12256)
by @sryza[dagster-airflow] remove ref to cli (#12262)
by @Ramshackle-JamathonRevamp intro of partitions concepts page (#12257)
by @sryzaRemove @experimental from build_assets_job and build_source_asset_observation_job (#12252)
by @smackesey[fix] Surface the correct description for multi-assets when a description is provided for the op (#12271)
by @OwenKephartremove legacy APIs from dagster-k8s tests (#12020)
by @sryzaUse Definitions in the body of materialize functions (and other public-facing execution functions) (#12038)
by @schrockn[typing/static] assets (#12003)
by @smackesey[typing/runtime] Make GraphDefinition.create_adjacency_lists internally public (#12291)
by @smackesey[typing/runtime] add AssetsDefinition.partition_mappings (#12290)
by @smackeseyin docs navigation, make partitions independent from schedules (#12264)
by @sryza[typing/static] storage (#12283)
by @smackesey[api docs] Show PartitionsDefinition doc strings on docs site (#12265)
by @jamiedemaria[db io manager] Support MultiPartitions (#12165)
by @jamiedemariaload asset value config (#10991)
by @Nintoracremove some handling for solid in composition.py (#12278)
by @sryzaAdd shortcuts for '--dagit-host' and '--dagit-port' to dagster dev (#12301)
by @gibsondan[typing/runtime] [gql] eliminate resolver **kwargs (2) (#12241)
by @smackesey[typing/runtime] [gql] eliminate resolver **kwargs (4) (#12243)
by @smackesey[typing/runtime] [gql] eliminate resolver **kwargs (5) (#12244)
by @smackesey[Runs Table] Open link in new tab (#12307)
by @salazarm[typing/runtime] [gql] eliminate resolver **kwargs (3) (#12242)
by @smackeseyadd types and break up big functions in dagstermill impl (#12274)
by @sryza[typing/runtime] [gql] eliminate resolver **kwargs (1) (#11724)
by @smackeseyadd cloud to scaffold (#12306)
by @slopp[typing/runtime] Remove unnecessary isinstance checks (#12287)
by @smackesey[typing/runtime] Make AssetKey.to_string always return AssetKey (#12285)
by @smackesey[typing/runtime] Refactor cached_method (#12004)
by @smackeseyRemove mypy from tox files (#12112)
by @smackesey[typing/runtime] Asset graph comparisons (#12286)
by @smackeseyrefactor(databricks): remove legacy Dagster definitions (#12150)
by @rexledesmadocs: update scaffold to use dagster dev (#12297)
by @rexledesma[db io managers] create schema if not exist (#11764)
by @jamiedemariaAssetsDefinition.to_source_asset (#12203)
by @sryzain tags doc, fix schedule tag and add sensor tag (#12313)
by @sryzaSimplify buildkite-build-test-project-image (#12029)
by @gibsondanMove repository_definition into subpackage (#12318)
by @schrocknBreak up repository_definition subpackage into multiple files (#12319)
by @schrockn[docs] - Build on local Dagster guide (#12168)
by @erinkcochran87[freshness-policies] add a cron_schedule_timezone argument to the FreshnessPolicy class (#12263)
by @OwenKephart[typing/runtime] Refactor subset selection tree (#12289)
by @smackeseyFix unimported symbol BK (#12327)
by @smackeseynux examples in oss 3/: sync dagster-io/quickstart-*/setup.py to example/* (#12234)
by @yuhan[dagster-k8s] In the execute_k8s_job op, use step key to generate the k8s job name (#12344)
by @OwenKephartInclude instance-level / code-location-level runK8sConfig in step pods (#12308)
by @gibsondanremove unused lazy-repository docs snippet (#12277)
by @sryza[asset-reconciliation][perf] Perf regression tests (#12230)
by @OwenKepharts/_CacheingDefinitionIndex/CacheingDefinitionIndex/g (#12320)
by @schrockn[typing/runtime] Massage set flattening (#12288)
by @smackesey[asset-reconciliation][perf] Better caching of most recent materializations (#12237)
by @OwenKephart[db io managers] Dynamic Partition tests (#12216)
by @jamiedemariabump limit (#12355)
by @OwenKephartFix multipartitions run length encoding error (#12329)
by @clairelin135[freshness-policies] Allow setting freshness policies when using graph-backed assets (#12357)
by @OwenKephart nullsafe array index access (#12362)
by @salazarmChange schedule button text (#12361)
by @dpeng817[dagster-pandas][dagster-pandera] assign a typing_type for generated pandas dataframe DagsterTypes (#12363)
by @OwenKephartFix partitions backfill deserialization error (#12238)
by @clairelin1351.1.19 Changelog (#12378)
by @OwenKephartguide to how assets relate to ops and graphs (#12204)
by @sryzaupdates tests to handle new kubernetes resources field (#12395)
by @alangenfeld1.1.19 changelog: reorder code block (#12402)
by @yuhan1.1.19
by @elementl-devtoolsPublished by jmsanders over 1 year ago
PartitionsDefinition
s (e.g. HourlyPartitionsDefinition
, DailyPartitionsDefinition
) may now have a FreshnessPolicy
.load_assets_from_dbt_project
or load_assets_from_dbt_manifest
with dbt-core>=1.4
, AssetMaterialization
events will be emitted as the dbt command executes, rather than waiting for dbt to complete before emitting events.make_ephemeral_airflow_db_resource
which returns a ResourceDefinition
for a local only airflow database for use in migrated airflow DAGsdagster instance migrate
.observable_source_asset
incorrectly required its function to have a context
argument.dagster-airbyte
dagster-dbt
integration was incompatible with dbt-core>=1.4
. This has been fixed.load_assets_from_dbt_cloud_job
will now avoid unnecessarily generating docs when compiling a manifest for the job. Compile runs will no longer be kicked off for jobs not managed by this integration.context.asset_partition_key
returned a string instead of a MultiPartitionKey
. This has been fixed.k8s_job_executor
would sometimes unexpectedly fail due to transient 401 errors in certain kubernetes clusters.load_assets_from_airflow_dag
no longer creates airflow db resource definitions, as a user you will need to provide them on Definitions
directlypartitions_fn
argument of the DynamicPartitionsDefinition
class is now deprecated and will be removed in 2.0.0.asset
(rather than using @asset
in decorator form) to create an asset. Thanks @ns-finkelstein!https://github.com/dagster-io/dagster/compare/1.1.17...1.1.18
Add google analytics tracking query param to slack link from OSS (#12024)
by @salazarmAdd --working-directory argument to dagster dev (#12026)
by @gibsondan[dagster-fivetran] Add option to force-create materializations for tables not in API response (#11972)
by @benpankowdocs: removed typo from install.mdx (#12022)
by @clayheaton[dagit] Fix use of fragments causing Apollo caching error in partition health (#12030)
by @bengotow[dagster-airflow] use full timestamp for partition name (#12034)
by @Ramshackle-Jamathon[dagster-airflow] airflow retry support (#11954)
by @Ramshackle-JamathonChange endpoints to the ones that are used by airbyte UI (#12012)
by @emilija-omnisendlint fix (#12044)
by @benpankowfeat(dbt-cloud): compile run only if job has environment variable cache (#12042)
by @rexledesmafix(dbt-cloud): inherit generate docs settings for compile run (#12043)
by @rexledesmaskip import test on windows (#12048)
by @alangenfeld1.1.15 changelog (#12051)
by @jamiedemaria[pyright] [scripts] misc (#11923)
by @smackeseyaccept UndefinedAssetJob in run status sensors (#12054)
by @jamiedemariaEnable use of arguments when using the
asset function directly instead of as a decorator (#11903)
by @nsfinkelsteinAutomation: versioned docs for 1.1.15
by @elementl-devtoolsBump Postgres to 14.6 (#12015)
by @DustyShapmake black (#12078)
by @alangenfeldConstruct required resources correctly when materializing partial asset graphs (#12052)
by @jamiedemariaFix context.asset_partition_key for multidimensional partitions (#12035)
by @clairelin135Install dagster-managed-elements when appropriate (#12046)
by @jmsandersBump http-cache-semantics from 4.1.0 to 4.1.1 in /js_modules/dagit (#12080)
by @dependabot[bot][docs] [dagster-snowflake] update docs with private key auth info (#11746)
by @jamiedemariain comment, clarify that scraping asset info off of In/Out is legacy (#12089)
by @sryzaLazy load ExternalAssetGraph for CachingStaleStatusResolver (#12090)
by @smackesey[telemetry] fix setting __TELEMETRY_ENABLED__ flag on Dagit (#12092)
by @benpankow[pyright] misc (#12102)
by @smackeseyAllow observable source assets to have no context argument (#11981)
by @schrockn[dagster-airflow] remove api's for 1.x.x release (#12023)
by @Ramshackle-Jamathon[dagster-airflow] extract utils into seperate file (#12065)
by @Ramshackle-Jamathon[dagster-airflow] move schedule and asset functions into their own files (#12066)
by @Ramshackle-Jamathon[dagster-airflow] remove vended airflow code (#12121)
by @Ramshackle-JamathonFix pre-release core => library translation (#12109)
by @gibsondanadd run job index (#12033)
by @prhaAdd Databricks compute kind tag (#12117)
by @braunjj[schedules] fix nth weekday of month cron handling (#12130)
by @alangenfeld[dagit] Add tests for unpartitioned case, fix for console.error in partition health (#12085)
by @bengotow[js] fix lint (#12131)
by @alangenfeld[pythonic config] Add disabled tests for various more complex config schemas (#12105)
by @benpankow[docs] Clairfy dagster-pagerduty docs (#12128)
by @benpankowTest with a 64 bit after_cursor (#12093)
by @jmsanders[dagit] Memoize some Intl behavior (#12125)
by @hellendag[dagit] Add storybook coverage of Asset Table states (#12103)
by @bengotow[dagit] Add Storybook coverage of PartitionHealthSummary rendering (#12116)
by @bengotow[dagit] In development, render toasts for unhandled promise exceptions (#12082)
by @bengotowInclude logs in failure message for ECS monitoring failures (#12113)
by @gibsondan[dagster-airflow] 1.x.x api changes (#12067)
by @Ramshackle-Jamathon[dagster-airflow] migration limitations (#12124)
by @Ramshackle-Jamathon[dagit] Mocks, storybooks and a test for BackfillTable (#12095)
by @bengotow[pythonic config] Correctly handle pydantic List types in config (#12106)
by @benpankowcomment to explain what define_solid_dictionary_cls does (#12088)
by @sryzaremove protobuf pin in dagster[test] (#11974)
by @yuhan[dagster-airflow] drop unique_id param (#12127)
by @Ramshackle-JamathonTest with a 64 bit after_cursor (#12137)
by @jmsanders[dagit] Show partition status on the asset graph, catalog pages, add tests (#11914)
by @bengotow[pythonic config] Correctly handle pydantic Dict/Mapping types in config (#12107)
by @benpankowrelease 1.1.17 changelog (#12140)
by @Ramshackle-Jamathon[docs] Add PySpark to Snowflake reference guide (#11814)
by @jamiedemariaAdd bugfixes to 1.1.17 changelog (#12141)
by @gibsondanAutomation: versioned docs for 1.1.17
by @elementl-devtoolsRuff 0.0.212 -> 0.0.241 (#12138)
by @smackeseyAdopt pyright for typechecking (#10983)
by @smackesey[1/n] Serialize top-level resources into repository data (#11529)
by @benpankow[2/n] Expose top-level resources via GraphQL (#11553)
by @benpankowFix serdes on FreshnessPolicy object (#12143)
by @benpankowBetter error output for Pyright script (#12145)
by @smackesey[3/n] Display top-level resources in Dagit sidebar (#11554)
by @benpankowFix misc type errors (#12147)
by @smackeseyDelete no-longer-needed type-ignores (#12156)
by @smackesey[dagit] Fix build failure caused by mocks out of sync with query (#12152)
by @bengotowSupport json output from pyright script and ignore
build (#12155)
by @smackeseyAdd pyright_rebuild Makefile target (#12159)
by @smackesey[pyright] --find-links for gprcio wheels (#12158)
by @alangenfeld[docs] - Remove Pagination component from layout (#12135)
by @erinkcochran87Fix instructions for using serverless from the CLI without GitHub (#12161)
by @shalabhcmaking erin the owner of all docs 👑 (#12139)
by @tacastillo[dagit] Better Asset DAG states for source assets (#12142)
by @bengotow[gql] Add tags to AssetEventMixin (#11973)
by @smackeseytype annotations for input_bindings in composition.py (#12087)
by @sryza[Dynamic Partitions 1] Storage changes (#11994)
by @clairelin135[dagit] System tags (code + logical versions) on asset graph + details (#12151)
by @bengotow[Dynamic Partitions 2] Update DynamicPartitionsDefinition to have name param (#12000)
by @clairelin135hide download links until logs are available (#12164)
by @prha(tick-testing 1/6) Rename GrapheneFutureInstigationTick to GrapheneDryRunInstigationTick (#12055)
by @dpeng817[Dynamic Partitions 3] Display dynamic partitions in Dagit (#11900)
by @clairelin135Avoid creating repo on each partitionStats gql call (#12166)
by @clairelin135[Docs] Removed sneaky citation mark in title on docs (#12120)
by @C0DKAdd 401 to the list of API codes that our k8s client retries on (#12074)
by @gibsondan[freshness-policies] Add the ability to calculate the used_data_time of a TimeWindowPartitioned asset (#11607)
by @OwenKephart(tick-testing 2/6) Revamp dry run tick behavior (#12056)
by @dpeng817feat(dagster-dbt): support dbt-core 1.4.x (#11902)
by @rexledesma[dagster-dbt] [1/2] Add streaming entrypoint for dbt cli execution (#12086)
by @OwenKephart[dagster-dbt] [2/2] Add ability to stream events while executing dbt assets (#12100)
by @OwenKephartFix import (#12185)
by @OwenKephartStore the instance ref on the grpc server class if it's present, so that grpc api calls can use it if it's there (#12194)
by @gibsondan[dagster-wandb] Integration with Weights & Biases (#10470)
by @chrishisteW&B integration follow up 2/ update example to defs (#12059)
by @yuhanvalidate time partition keys when adding to a TimeWindowPartitionsSubset (#12195)
by @OwenKephartMake Partition GRPC calls with
instance_ref args optional (#12196)
by @clairelin135[dagster-airflow] [docs] migration guide updates/considerations (#12198)
by @Ramshackle-Jamathon[dagster-airflow] refactor airflow_db resources (#12202)
by @Ramshackle-JamathonMake top_level_resources argument _top_level_resources (#12211)
by @schrocknAdd guide for observable source assets and versioning (#12118)
by @schrockn1.1.18 Changelog (#12217)
by @smackeseyFix dagster-wandb placeholder (#12222)
by @smackesey1.1.18
by @elementl-devtoolsPublished by gibsondan over 1 year ago
dagster-airflow
library as been moved to 1.x.x to denote the stability of its api's going forward.make_schedules_and_jobs_from_airflow_dag_bag
has been added to allow for more fine grained composition of your transformed airflow DAGs into Dagster.retries
and retry_delay
configuration are now converted to op RetryPolicies with all make_dagster_*
apis.0 5 * * mon#1
to execute on a certain day of the week each month executed every week instead.use_airflow_template_context
, mock_xcom
and use_ephemeral_airflow_db
params have been dropped, by default all make_dagster_*
apis now use a run-scoped airflow db, similiar to how use_ephemeral_airflow_db
worked.make_airflow_dag
has been removed.make_airflow_dag_for_operator
has been removed.make_airflow_dag_containerized
has been removed.airflow_operator_to_op
has been removed.make_dagster_repo_from_airflow_dags_path
has been removed.make_dagster_repo_from_airflow_dag_bag
has been removed.make_dagster_repo_from_airflow_example_dags
has been removed.${dag_id}__${task_id}
from airflow_${task_id}_${unique_int}
.${dag_id}
from airflow_${dag_id}
.Published by shalabhc over 1 year ago
@multi_asset_sensor
now accepts a request_assets
parameter, which allows it to directly request that assets be materialized, instead of requesting a run of a job.Definitions
when using large numbers of assets or many asset jobs.build_schedule_from_partitioned_job
no longer needs to have a partitions_def
directly assigned to it. Instead, Dagster will infer from the partitions from the assets it targets.OpExecutionContext.asset_partition_keys_for_output
no longer requires an argument to specify the default output.pyproject.toml
file that were made while Dagit was running. Previously, Dagit needed to be restarted in order for such changes to be shown.get_run_record_by_id
has been added to DagsterInstance
to provide easier access to RunRecord
objects which expose the start_time
and end_time
of the run.docker_container_op
op and execute_docker_container_op
helper function for running ops that launch arbitrary Docker containers. See the docs for more information.python:3.x-slim
base image.build_asset_reconciliation_sensor
could time out when evaluating ticks over large selections of assets, or assets with many partitions. A series of performance improvements should make this much less likely.run_request_for_partition
in a sensor that targeted multiple jobs created via define_asset_job
.dagster
has been reduced.PartitionsDefinition
.k8s_job_op
raised an Exception when running pods with multiple containers.asset_keys
and asset_selection
parameters of the experimental @multi_asset_sensor
decorator have been replaced with a monitored_assets
parameter. This helps disambiguate them from the new request_assets
parameter.StaticPartitionMapping
is now serializable, thanks @AlexanderVR!build_fivetran_assets
now supports group_name
, thanks @toddy86!AzureBlobComputeManager
now supports authentication via DefaultAzureCredential
, thanks @mpicard!load_assets_from_airflow_dag
that creates graph-backed, partitioned, assets based on the provided Airflow DAG.https://github.com/dagster-io/dagster/compare/1.1.14...1.1.15
Make StaticPartitionMapping serializable. Add autodoc. (#11738)
by @AlexanderVRPerformance improvements for large multi assets (#11782)
by @OwenKephartmonitored_assets param for multi-asset sensor (#11567)
by @sryza[fix] Fix Snowflake IO manager tests for project_fully_featured (#11781)
by @benpankowUpdate docs to include wheel workarounds for m1 macs (#11777)
by @gibsondanfix run_request_for_partition for sensors that target multiple unreso… (#11780)
by @sryzaNew "running dagster locally" deployment guide that walks through
dagster dev usage (#11741)
by @gibsondanRestore grpcio pin for python 3.10 (#11784)
by @gibsondanChangelog 1.1.11 (#11789)
by @dpeng817docs: add clarification for helm migration guide (#10454)
by @adam-bloom[dagit] Bump react-scripts to remove codegen plugin (#11769)
by @hellendagfeat(dagster-dbt): use cached run id to fetch artifacts (#11744)
by @rexledesmafeat(dbt-cloud): raise exception if cached compile has not completed (#11792)
by @rexledesmachangelog 1.1.11 pt 2 (#11800)
by @alangenfeldHeartbeat once at the beginning of every interval daemon (#11802)
by @gibsondan1.1.11 changelog: fix link (#11804)
by @yuhan[dagit] Add missing changelog entries for 1.1.11 (#11803)
by @bengotowadd request_assets param to multi_asset_sensor (#11786)
by @sryzaAutomation: versioned docs for 1.1.11
by @elementl-devtoolsLog to daemon output when running the schedule code fails or times out (#11805)
by @gibsondan1.1.12 changelog (#11807)
by @alangenfeld[dagit] When viewing an asset that is a root, do not fetch root used data (#11806)
by @bengotow[dagit] Fix support for long descriptions in the asset catalog table (#11810)
by @bengotowConvert accesses using '__ASSET_JOB' to instead use the supported implicit job methods (#11348)
by @schrockn[dagster-azure] Add support for DefaultAzureCredential for AzureBlobComputeLogManager (#11310)
by @mpicardadd dagster.yaml tests for nux key (#11820)
by @gibsondanFix 'dagster dev' command with workspace.yaml passed in (#11819)
by @gibsondanWhen can't deserialize asset backfill because asset is no longer partitioned, return it as empty (#11812)
by @sryza1.1.13 changelog (#11818)
by @alangenfeldadd after_cursor to get_materialization_counts_by_partition (#11759)
by @OwenKephartAdds group_label to build_fivetran_assets (#11718)
by @toddy86default argument to asset_partition_keys_for_output (#11811)
by @sryzaSpecify credentials masking. (#11813)
by @joel-olazagastiFix multidimensional partition backfills (#11788)
by @clairelin135Automation: versioned docs for 1.1.13
by @yuhanMove pyproject.toml => origin generation to a WorkspaceLoadTarget class so that it can be reloaded in dagit (#11821)
by @gibsondanfix tz in time_windows_for_partition_keys (#11825)
by @sryzadynamic re-execution fixes (#11581)
by @alangenfeld[instance] get_run_record_by_id (#11643)
by @alangenfeld[asset-graph] keep track of source asset partitions defs (#11840)
by @OwenKephart[dagster-airbyte] Fix issue using freshness policy w/ Airbyte + multiprocessing executor (#11837)
by @benpankow[asset-reconciliation] pre-fetch the results of some queries (#11770)
by @OwenKephart[asset-reconciliation] Fix case where the partitions definition does not have partitions for some subset of the past 24 hours (#11842)
by @OwenKephartFix for backoff logic incorrectly storing state (#11848)
by @gibsondanuse an
AssetGraph to resolve asset selections in asset jobs (#11846)
by @sryzafix(dbt-cloud): remove dbt selector when materializing subset (#11843)
by @rexledesmapin sqlalchemy below 2.0.0 (#11871)
by @benpankow[dagit] Allow materializing “All” without requiring large graphs are rendered (#11854)
by @bengotowchore(dagster-dbt): pin dbt-core<1.4.0 (#11870)
by @rexledesmastop using python semver parsing for mysql versions (#11868)
by @prhaFix aws ssm tests (#11886)
by @gibsondanunbreak backcompat tests after sqlalchemy upgrade (#11885)
by @gibsondan[dagit] Fix jumpy code location status spinner (#11801)
by @hellendagAdd apache-airflow test pin to <2.5.1 (#11888)
by @gibsondanFix databricks tests running in release branch (#11887)
by @gibsondanAdd pin for jupyter-client<8 (#11901)
by @gibsondan[pyright] [core] storage (#11363)
by @smackesey[pyright] [core] standardize is_context_provided (#11364)
by @smackesey[pyright] [core] eliminate
funcsigs (#11365)
by @smackesey1.1.14 changelog (#11907)
by @gibsondan[pyright] [core] remove builitins star import (#11366)
by @smackesey[readme] update twitter badge (#11892)
by @alangenfeld[pyright] [gql] DagsterPipelineRunMetadataValue -> DagsterRunMetadataValue (#11893)
by @smackesey[pyright] [core] _core/definitions/asset_reconciliation_sensor (#11717)
by @smackesey[pyright] [gql] implementation/events (#11894)
by @smackesey[pyright] [gql] implementation/fetch_partition_sets (#11895)
by @smackeseyAutomation: versioned docs for 1.1.14
by @elementl-devtools[pyright] examples/tutorial-notebook-assets (#11910)
by @smackesey[pyright] examples/with-airflow (#11911)
by @smackesey[pyright] [dagit] misc (#11727)
by @smackesey[pyright] [dagster-dask] misc (#11944)
by @smackesey[pyright] [dagster-databricks] misc (#11943)
by @smackesey[pyright] [dagster-test] misc (#11940)
by @smackesey[pyright] [dagster-mysql] misc (#11939)
by @smackesey[pyright] [dagster-duckdb-pyspark] misc (#11937)
by @smackesey[pyright] [dagster-pandas] misc (#11931)
by @smackesey[pyright] [dagster-postgres] misc (#11929)
by @smackesey[pyright] [dagster-snowflake] misc (#11926)
by @smackesey[pyright] [examples/assets-dbt-python] misc (#11922)
by @smackesey[pyright] [dagster-snowflake] misc (#11928)
by @smackesey[pyright] [dagster-pandera] misc (#11930)
by @smackesey[pyright] [dagster-aws] misc (#11730)
by @smackesey[pyright] [dagster-airflow] misc (#11729)
by @smackesey[pyright] examples/quickstart-gcp (#11909)
by @smackesey[pyright] [dagster-dbt] misc (#11941)
by @smackesey[pyright] [automation] misc (#11726)
by @smackesey[pyright] [dagster-msteams] misc (#11932)
by @smackesey[pyright] [dagster-snowflake-pandas] misc (#11927)
by @smackesey[pyright] [dagstermill] misc (#11925)
by @smackesey[pyright] [dagster-docker] misc (#11938)
by @smackesey[pyright] [dagster-gcp] misc (#11936)
by @smackeseyremove need to provide
partitions_defto asset job targeted by
build_schedule_from_partitioned_job (#11844)
by @sryzafix GQL snapshot tests on py310 (#11945)
by @smackesey[pyright] [dagster-airbyte] misc (#11728)
by @smackesey[pyright] [dagster-mlflow] misc (#11948)
by @smackeseyFix k8s_job_op with multiple containers (#11916)
by @gibsondan[pyright] [gql] capture_error sig fix (#11722)
by @smackesey[pyright] [dagster-azure] misc (#11946)
by @smackeseyAdd code versions to AssetGraph (#11950)
by @smackesey[pyright] [gql] schema/solids#resolve_required_resources (#11898)
by @smackeseyrefactor GrpcServerProcess (#11960)
by @smackeseyFix s3_resource docstring example (#11757)
by @dpeng817[pyright] [core] create RunGroupInfo type (#11368)
by @smackeseysnowflake connector pin (#11970)
by @alangenfeld[pyright] [gql] schema/roots/mutation (#11897)
by @smackesey[easy] List dagster_world.mp4 in static files (#11961)
by @salazarm[pyright] [gql] AssetKey cleanup (#11721)
by @smackeseyCachingProjectedLogicalVersionResolver -> CachingStaleStatusResolver (#11951)
by @smackeseywhitelist EventLogRecord for serdes (#11978)
by @OwenKephartdocument tags_for_partition_fn in _partitioned_config API doc (#11977)
by @sryza[pyright] [gql] misc (#11720)
by @smackesey[pyright] [gql] misc type errors 2 (#11947)
by @smackesey[pyright] [gql] implementation/fetch_runs (#11896)
by @smackeseyAllow bare executor in Definitions (#11795)
by @schrocknAdd opt_iterable_param (#11796)
by @schrocknUse opt_iterable_param in Definitions (#11797)
by @schrockn[pyright] [gql] compute log manager (#11725)
by @smackesey[dagster-snowflake-pandas] snowflake-sqlalchemy pin (#11984)
by @smackesey[pyright] [gql] add ResolveInfo everywhere (#11723)
by @smackeseyadd .ruff_cache to gitignore (#11987)
by @Ramshackle-Jamathonsmall tweak to airflow integration page (#11985)
by @sryzaRename
_core.decorator_utils.is_context_provided (#11982)
by @smackeseyAdd docker_container_op and execute_docker_container (#11831)
by @gibsondan[perf] perf improvement for TimeWindowPartitionsSubset (#11850)
by @OwenKephart[dagit] Add ErrorBoundary to Dagit to reduce severity of React errors (#11824)
by @bengotow[dagit] Repair markdownToPlaintext test failure (#11995)
by @hellendag[import perf] defer CachingInstanceQueryer imports to avoid storage imports (#11905)
by @alangenfeldUpdate error messaging for DB IO managers (#11815)
by @jamiedemaria[import perf] moves to prevent grpc import (#11906)
by @alangenfeld[import perf] test to prevent regressions (#11969)
by @alangenfeldremove legacy APIs from dagstermill tests (#11999)
by @sryzaremove legacy APIs from dagster_shell_tests (#11998)
by @sryza[dagit] UI support for launching a single asset run with a range of partition keys (#11866)
by @bengotow[dagit] Add more truncation test cases for Firefox, change const (#12001)
by @bengotowFix black pre-commit hook (#11979)
by @smackesey[3/3 partition status cache] Update graphQL partition data (#10822)
by @clairelin135Upgrade base image on Dagster Dockerfiles to latest python version (#11863)
by @gibsondan[pyright] [examples/docs-snippets] misc (#11921)
by @smackeseyAdd
last_materialization_record to AssetEntry. (#11919)
by @OwenKephart[asset-reconciliation][perf] Improve prefetch accuracy (#11991)
by @OwenKephartfix dagstermill tests (#12002)
by @sryzaType annotations for workspace code (#11958)
by @smackeseyConsolidate GrpcServerRegistry/ProcessGrpcServerRegistry (#11990)
by @smackeseyRefactor ProcessRegistryEntry (#11959)
by @smackesey[pyright] examples/project-fully-featured (#11908)
by @smackesey[pyright] [dagster-k8s] misc (#11935)
by @smackesey[pyright] [core] tests (#11369)
by @smackesey[pyright] Type-ignores for various errors related to managed elements (#12014)
by @smackeseycleam up get_asset_events (#11913)
by @alangenfeldPySpark type handler for snowflake io manager (#11542)
by @jamiedemaria[asset-reconciliation] [perf] do not try to fetch materialization records for sources (#12011)
by @OwenKepharttest(dagster-dbt): add testing for python 3.10 (#11912)
by @rexledesmafix: use new GitHub graphql resolvers for issue automation (#11877)
by @rexledesma[dagit] Export GhostDaggy with tooltip for Cloud usage (#11967)
by @hellendag[dagster-airflow]
load_assets_from_airflow_dag (#11876)
by @Ramshackle-JamathonFix broken doc link for Snowflake credential setup (#12017)
by @clayheaton[dagster-fivetran] Add option to force-create materializations for tables not in API response (#11972)
by @benpankow[dagit] Fix use of fragments causing Apollo caching error in partition health (#12030)
by @bengotowChange endpoints to the ones that are used by airbyte UI (#12012)
by @emilija-omnisendlint fix (#12044)
by @benpankowfeat(dbt-cloud): compile run only if job has environment variable cache (#12042)
by @rexledesmafix(dbt-cloud): inherit generate docs settings for compile run (#12043)
by @rexledesma1.1.15 changelog (#12051)
by @jamiedemaria1.1.15
by @elementl-devtoolsPublished by prha over 1 year ago
*
as the asset selection before materializing the assets.sqlalchemy
package to <2.0.0
due to a breaking change in that version.dbt-core
package to <1.4.0
due to breaking changes in that release that affected the Dagster dbt integration. We plan to remove this pin in the next release.jupyter-client
package to <8.0
due to an issue with the most recent release causing hangs while running dagstermill ops.EcsRunLauncher
sometimes waited much longer than intended before retrying after a failure launching a run.