An orchestration platform for the development, production, and observation of data assets.
APACHE-2.0 License
Bot releases are hidden (Show)
Published by alangenfeld over 1 year ago
nux
section of dagster.yaml
config has been fixed.dagster dev
command raised an errorPublished by alangenfeld over 1 year ago
Published by yuhan over 1 year ago
dagster dev
command that can be used to run both Dagit and the Dagster daemon in the same process during local development. See the new Running Dagster Locally guide in the docs for more information.dagster
package on M1 and M2 Macs that avoids installation errors when building the grpcio
package. See the Installing Dagster guide in the docs for more information.create_repository_using_definitions_args
has been added for users to backport their repository definitions to the new Definitions
APInux:
enabled: false
load_assets_from_dbt*
is now configurable, allowing you to pass in specific parameters to the underlying dbt command (such as --full-refresh
). To do so, supply config of the form {"ops": {"run_dbt_<abcde>": {"config": {"full_refresh": True}}}}
when defining your asset job, or in Dagit.build_asset_reconciliation_sensor
will now only materialize partitions from the past day. Previously, if a new asset was added with a large number of missing partitions, a run for each of the missing partitions would be launched immediately.build_asset_reconciliation_sensor
, which should help significantly speed up sensor evaluation in cases where there is a large number of assets or partitions.https://github.com/dagster-io/dagster/compare/1.1.10...1.1.11
telemetry: avoid race condition of dagit and daemon creating same dir (#11652)
by @yuhan[dagit] Link to the failed run, not the last materialzation, on “Failed" nodes (#11593)
by @bengotow[dagit] Stop foreign node links from creating sprawling asset graphs (#11588)
by @bengotowrefactor: remove workspace.yaml in favor of pyproject.toml (#11665)
by @rexledesmadocs: update license copyright (#11666)
by @rexledesmaKill stray tqdms when starting up dagster (#11658)
by @gibsondan[Community Nux] Don't use blueprint dialog (#11660)
by @salazarmfix code location status when a new location is added in a loading state (#11654)
by @prha[dagster-airflow] pass airflow dag timezone to dagster schedule definition (#11663)
by @Ramshackle-Jamathonadd example and compose_fn argument doc to job decorator (#11655)
by @sryzaRevert "test fixes for python 3.7 (#11636)" and "Fall back to a different port when 3000 is in use instead of failing (#11610)" (#11671)
by @gibsondanRemove community nux for now (#11672)
by @salazarmchore(dbt): remove unnecessary dependencies (#11668)
by @rexledesmachore(phabricator): remove arcanist config (#11667)
by @rexledesmafix a link on deployment/overview (#11674)
by @yuhanUnbreak windows tests (and likely the windows multiprocess executor in general) (#11679)
by @gibsondanAdd black hook to precommit (#11678)
by @smackeseychangelog for 1.1.10 (#11682)
by @sryza[dagit] fix clientTime arg (#11670)
by @alangenfeldFix black precommit (#11683)
by @smackeseyAutomation: versioned docs for 1.1.10
by @elementl-devtools[tick-alerting] Add properties to InstigationTick (#11140)
by @dpeng817[tick-alerting] Add docs for tick alerting (#11417)
by @dpeng817fix broken link to Airflow library page (#11685)
by @sryzaRemove run launcher list from run monitoring docs (#11644)
by @gibsondanMake create_repository_using_definitions_args public (#11684)
by @gibsondanBring back dagit socket reuse (#11688)
by @gibsondanStart event log consumer from latest event (#11681)
by @johannkm[dagit] Restore some previous GraphQL codegen behavior (#11675)
by @hellendagone more windows test fix for signals (#11680)
by @gibsondanShut down the daemon faster when it is interrupted (#11657)
by @gibsondanExample of usage of Branching I/O Manager (#11549)
by @schrockninclude docs snippet for how to record metadata for non-output assets (#11686)
by @sryzaFix stale asset schedule (#11708)
by @smackesey[docs][dagit-screenshot] Add CLI command to generate asset SVGs from snippets (#9764)
by @benpankowfeat(dbt-cloud): allow jobs to be retrieved from project id (#11673)
by @rexledesmafix issue with k8s_job_op when combining command and container_config (#11713)
by @gibsondaneliminate SensorExecutionContext, ScheduleExecutionContext (#11038)
by @smackeseyMove string construction out of non-error path in check (#11701)
by @schrockn[pyright] [core] misc (#11362)
by @smackesey[pyright] [core] _utils/error (#11695)
by @smackesey[pyright] [core] _utils/backcompat (#11696)
by @smackesey[pyright] [core] _cli/api (#11697)
by @smackesey[dagit] Rename a few partition data structures in Dagit (#11574)
by @bengotowfeat(dbt-cloud): disable schedule only if it exists (#11690)
by @rexledesma[dagit] Permissions cleanup (#11712)
by @hellendag[dagit] Move User Settings into a Dialog (#11706)
by @hellendag[dagit] Fix Definitions filter dialog (#11711)
by @hellendag[dagit] Clean up a bunch of document titles (#11715)
by @hellendag"dagster dev" command that spins up dagit and the daemon as subprocesses and requires no DAGSTER_HOME env var to be set (#11584)
by @gibsondan[dagit] Empty state on Runs with better CTAs (#11714)
by @hellendag[OSS Nux] only load an iframe for recaptcha after use agrees to submit email (#11743)
by @salazarmBan invalid escape chars (#11443)
by @clairelin135[@dagster-io/ui] Bump eslint-config (#11703)
by @hellendag[dagit] Display source data used in specific asset materializations (#11736)
by @bengotow[dagit] Add the Asset compute kind tag in more places #11185 (#11569)
by @bengotow[dagit] Add backfill warnings and run tags options to the Materialize modal (#11656)
by @bengotow[1/3 partition status cache] Time window subset perf improvements (#11640)
by @clairelin135[dagster-dbt] add config schema for dbt asset op (#11710)
by @OwenKephart[asset-reconciliation] Limit the partitions that can be reconciled with the sensor (#11582)
by @OwenKephart[dagit] Strip down query for Scheduled Runs (#11751)
by @hellendag[asset-reconciliation] Improve performance for never materialized root partitions (#11755)
by @OwenKephart[dagit] Fix Backfills page locking up when displaying large backfills, improve "target" column (#11693)
by @bengotow[airbyte] Consider keys containing credentials a secret (#11620)
by @joel-olazagasti[asset-reconciliation] Perf improvements for reconciliation sensor (#11633)
by @OwenKephart[OSS Nux] Add kill switch + persist Nux dismissal in temp folder (#11758)
by @salazarm[pyright] [core] _core/definitions/scoped_resources_builder (#11698)
by @smackeseyAdd back dagster_world.mp4 video (#11763)
by @salazarmopen grpc servers in their own process group so they don't get interrupted along with whatever process opened them (#11737)
by @gibsondanDon't spew "Aborted!" to the command-line when the Daemon is cleanly interrupted (#11739)
by @gibsondanSwitch the grpcio dependency to use an internally built m1 wheel for m1 macs only (#11760)
by @shalabhcInclude retry count in the job created by k8s_job_op (#11753)
by @gibsondantest: unpin responses (#11768)
by @rexledesmaSelectively show the multiple files/modules help text to the commands where it applies (#11689)
by @gibsondanfeat(dbt-cloud): cache compile run id as env var for job (#11691)
by @rexledesma[2/3 partition status cache] Serialize version, rebuild cache for unsupported version (#11687)
by @clairelin135Fix airflow_op_to_dagster_op test (#11771)
by @clairelin135DBT code version docs (#11692)
by @smackeseyimprove performance of StaticPartitionsDefinition.__eq__ (#11775)
by @sryza[dagster-snowflake] remove password check for private key auth for unencrypted keys (#11745)
by @jamiedemariaRename "base asset job" to "implicit asset job" in RepositoryDefinition methods (#11344)
by @schrocknmark PartitionMappings not experimental in IO manager doc (#11774)
by @sryzadagit => dagster dev in docs (#11740)
by @gibsondanRevert "Switch the grpcio dependency to use an internally built m1 wheel for m1 macs only (#11760)" (#11776)
by @gibsondanPerformance improvements for large multi assets (#11782)
by @OwenKephartUpdate docs to include wheel workarounds for m1 macs (#11777)
by @gibsondanNew "running dagster locally" deployment guide that walks through
dagster dev usage (#11741)
by @gibsondanChangelog 1.1.11 (#11789)
by @dpeng817Restore grpcio pin for python 3.10 (#11784)
by @gibsondanfeat(dagster-dbt): use cached run id to fetch artifacts (#11744)
by @rexledesmafeat(dbt-cloud): raise exception if cached compile has not completed (#11792)
by @rexledesmachangelog 1.1.11 pt 2 (#11800)
by @alangenfeld[dagit] Add missing changelog entries for 1.1.11 (#11803)
by @bengotow1.1.11
by @elementl-devtoolsPublished by gibsondan almost 2 years ago
selection
argument of define_asset_job
now accepts lists of AssetKey
s or AssetsDefinitions
.RunRequest
now takes a stale_assets_only
flag that filters the full set of assets that would be materialized by a job to stale assets only. This can be used in schedules and sensors.dagit
command and the default port 3000 is already in use.grpcio
pin in Dagster to <1.48.1 has been removed for Python versions 3.10 and 3.11. Python 3.7, 3.8, and 3.9 are still pinned to <1.48.1 due to a bug in the grpc library that is causing the process to sometimes hang.stdout
or stderr
tab. The next time you view compute logs, you will see that tab first by default.executor
and loggers
arguments on Definitions
are no longer experimental.json_log_format
is set to False
when using the dbt_cli_resource
, logs will be emitted at the appropriate log level in some situations. Previously, all logs would be emitted at the INFO
level.dagster-azure
/ dagster-aws
/ dagster-gcp
)build_asset_reconciliation_sensor
with asset graphs that contained references to source assets without the associated SourceAsset
objects (which may happen when using load_assets_from_dbt_*
).DbtManifestAssetSelection
which could result in KeyErrors
when selecting from projects with sources defined./instance
and /workspace
path prefixes were removed in previous version, but redirects were left in place. These redirects have now been removed.StaticPartitionMapping
enables explicitly defining the dependencies between partitions in two StaticPartitionsDefinition
s. Thanks Alexander VR!https://github.com/dagster-io/dagster/compare/1.1.9...1.1.10
docs for new tag_concurrency_limits feature on executor (#11499)
by @gibsondan[dagster-airflow] warn on airflow dataset (#11498)
by @Ramshackle-JamathonAdd typehints to snapshot arguments to create_run (#11454)
by @schrocknTypehint and do invariants for root_run_id, parent_run_id, step_keys_to_execute and execution_plan_snapshot (#11455)
by @schrocknTypehint and validate tags and status params to create_run (#11456)
by @schrocknRequire kwargs on create_reexecuted_run (#11460)
by @schrocknMake frozentags inherit from Mapping[str, str] and correctly type _check_execute_external_pipeline_args (#11461)
by @schrocknAdd support for loading cacheable assets from module (#10389)
by @benpankowenable filtering by asset tag when asset tags table is not present (#11509)
by @sryzaonly count materializations within backfill (#11506)
by @sryzatelemetry: add num_assets_in_repo in log_repo_stats metadata (#11513)
by @yuhantelemetry: add location_name in repo-level metadata (#11514)
by @yuhan[dagit] Clean up /instance and /workspace redirects (#11503)
by @hellendagFix getting partition mappings when asset dependencies are resolved (#11483)
by @sryzadocstring examples for AssetSelection.keys (#11477)
by @sryza[dagster-airflow] add
make_dagster_definitions_from_airflow_dags_pathand
make_dagster_definitions_from_airflow_dag_bag apis (#11441)
by @Ramshackle-JamathonFix maxResumeRunAttempts being explicitly enabled even if you set it to 0 (#11511)
by @gibsondandagster-graphql tests for permissions failures (#11486)
by @gibsondanHook for Cloud dagit into PythonErrorInfo (#11467)
by @salazarmChangelog 1.1.8 (#11526)
by @clairelin135Add experimental examples directory (#11505)
by @schrockn[dagster-airbyte] Allow specifying freshness policies on Airbyte assets (#11525)
by @benpankowMigrate airflow guide to definitions (#11518)
by @petehuntAutomation: versioned docs for 1.1.8
by @elementl-devtoolsRemove packaging v22 cap (#11212)
by @smackeseyremove pragmas in docs snippets (#11335)
by @smackeseyupdate black 22.3.0 -> 22.12.0 and enable --preview (#10900)
by @smackeseyAdopt ruff as linter (#10901)
by @smackeseyAdd .git to excludes for ruff in pyproject.toml (#11534)
by @schrocknchore(ruff): add pre-commit config (#11538)
by @rexledesmachore(dagit): remove unnecessary dependencies (#11532)
by @rexledesmadocs: add links to follow LinkedIn page (#11533)
by @rexledesma[dagit] Use cache-and-network as default watchQuery fetch policy (#11523)
by @hellendag[docs] - Add GSC site verification file (#11547)
by @erinkcochran87disable executing airflow_example_http_operator in tests (#11544)
by @Ramshackle-JamathonAdd and apply ruff docstring rules (#10907)
by @smackeseyRuff docstrings D209 (NewLineAfterLastParagraph) (#10993)
by @smackeseyUse Config class instead of base class kwargs in structured config to not break early pydantic versions (#11546)
by @schrockn[fix] Fix builtin type error related to resource annotations (#11551)
by @benpankow[changelog] 1.1.9 (#11556)
by @benpankowadd NullMetadataValue (#10969)
by @smackeseyRuff docstrings D210 (NoSurroundingWhitespace) (#10994)
by @smackesey[changelog] 1.1.9 round 2 (#11560)
by @benpankowMake JsonMetadataValue accept lists (#11550)
by @smackeseyRuff docstrings D212,D214 (MultilineSummaryFirstLine,SectionNotOverIndented) (#10995)
by @smackeseyAutomation: versioned docs for 1.1.9
by @elementl-devtoolsRuff docstring D300 (UsesTripleQuotes) (#10996)
by @smackeseyAdd a graphql field and API for location-scoped permissions (#11097)
by @gibsondan[@dagster-io/eslint-config] Remove GraphQL type parameter rule (#11479)
by @hellendagRuff docstrings D301 (Raw string for backslash in docstring)) (#11329)
by @smackesey[dagit] Update eslint-config dep in dagster-io/ui (#11563)
by @hellendag[dagit] Stop generating docblocks on schema.graphql and codegen (#11564)
by @hellendag[dagit] Track most recent preference for computed log type view (#11527)
by @hellendagFix graphql permisisons checks, fix black (#11566)
by @gibsondan[docs] - Move ops + jobs tutorial to Guides (#11528)
by @erinkcochran87[dynamic] fix how skips cascade (#11561)
by @alangenfeld[dagit] Add GraphQL configuration for VSCode (#11565)
by @hellendag[dagster-dbt] add DbtManifestAssetSelection to API docs (#11562)
by @OwenKephartdefine_asset_job accept AssetKey and AssetsDefinition selections (#11568)
by @sryzaRuff docstrings D403,D405 (FirstLineCapitalized,CapitalizeSectionName) (#10997)
by @smackeseyfix remote captured log manager complete check (#11552)
by @prhamisc docs fixes in tutorial / getting started / assets (#11575)
by @gibsondandocs: fix typo Telementry -> Telemetry (#11571)
by @chriszsfix docstring for static_partitioned_config (#11535)
by @severoAdd tip when the multiprocess executor hits a likely OOM error (#11570)
by @gibsondanRuff docstrings D414 (NonEmptySection) (#10999)
by @smackeseyRuff docstrings D419 (NonEmpty) (#11000)
by @smackesey[@dagster-io/ui] Output cjs (#11594)
by @hellendagShow clearer grpc errors, as well as a custom failure for sensor timeouts in particular since those are so common (#11576)
by @gibsondanAdds OSS email collection dialog (#11480)
by @salazarmRuff docstrings D202 (NoBlankLineAfterFunction) (#10992)
by @smackeseyRuff docstrings D411,D412 (BlankLineBeforeSection,NoBlankLinesBetweenHeaderAndContent) (#10998)
by @smackesey Change exclude to extend-exclude in pyproject.toml settings for ruff (#11591)
by @schrocknrefactor(dbt-cloud): remove v2 suffix (#11586)
by @rexledesmafeat(dbt-cloud): allow env vars to be retrieved and set (#11592)
by @rexledesmaRuff docstrings (misc fixes) (#11002)
by @smackesey[docs] - Remove accidental comments (#11601)
by @erinkcochran87docs: fix dead link to Slack community (#11602)
by @EmilRexFix graphene enum not being instantiated (#11605)
by @gibsondan[dagit] Add location-scoped permissions to Dagit (#11587)
by @hellendag[dagster-dbt] parse log level when json_log_level is not set (#11596)
by @OwenKephartuse Python type annotations in loading-from-config docs examples (#11609)
by @sryza[dagit] Use updated react-scripts fork for codegen babel plugin (#11583)
by @hellendagFall back to a different port when 3000 is in use instead of failing (#11610)
by @gibsondan[dagit] Some Permissions tweaks to support Cloud (#11612)
by @hellendagRuff docstrings D415 (EndsInPunctuation) (#11001)
by @smackesey[dagster-dbt] allow for non-model nodes to be part of the DbtManifestAssetSelection (#11617)
by @OwenKephartUpdate OSS Nux to receive size from iframe via postmessage (#11625)
by @salazarm[freshness-policies] fix get_non_source_roots (#11619)
by @OwenKephartremove reexecute_pipeline_iterator (#11608)
by @alangenfeld[eslint-config] Enable ignoreExternal on no-cycle rule (#11632)
by @hellendag[@dagster-io/ui] Bump eslint-config (#11634)
by @hellendagadd example to TimeWindowPartitionMapping apidoc (#11615)
by @sryzaFix typo in Dagster instance documentation (#11621)
by @GrigoriiKushniruse run tags instead of asset tags to filter materializations within backfill (#11555)
by @sryzatest fixes for python 3.7 (#11636)
by @gibsondanRestrict the grpcio pin to only py3.7/3.8/3.9 (#11635)
by @gibsondan Remove experimental flag on executor and loggers parameters of Definitions and add more detail to docblock (#11626)
by @schrockn[dagster-snowflake-pandas] update doc string to use Definitions (#11606)
by @jamiedemaria[dagster-snowflake] Add private key authentication for Snowflake (#10750)
by @joshuataylorLoad OSS Nux iframe via https (#11638)
by @salazarmtelemetry: add step start/success/failure event + client timestamp (#11624)
by @yuhanAdd StaticPartitionMapping as a built-in PartitionMapping (#11521)
by @AlexanderVR[dagster-airbyte] Enable setting a destination table prefix using managed ingestion (#11057)
by @benpankowtelemetry: add is_known_ci_env (#11642)
by @yuhanstale asset materialization for schedules and sensors (#11516)
by @smackeseytelemetry: avoid race condition of dagit and daemon creating same dir (#11652)
by @yuhan[Community Nux] Don't use blueprint dialog (#11660)
by @salazarmfix code location status when a new location is added in a loading state (#11654)
by @prha[dagster-airflow] pass airflow dag timezone to dagster schedule definition (#11663)
by @Ramshackle-JamathonRevert "test fixes for python 3.7 (#11636)" and "Fall back to a different port when 3000 is in use instead of failing (#11610)" (#11671)
by @gibsondanRemove community nux for now (#11672)
by @salazarmUnbreak windows tests (and likely the windows multiprocess executor in general) (#11679)
by @gibsondan1.1.10
by @elementl-devtoolsPublished by jmsanders almost 2 years ago
tag_concurrency_limits
key that allows you to specify limits on the number of ops with certain tags that can be executing at once within a single run. See the docs for more information.ExecuteInProcessResult
, the type returned by materialize
, materialize_to_memory
, and execute_in_process
, now has an asset_value
method that allows you to fetch output values by asset key.AssetIn
s can now accept Nothing
for their dagster_type
, which allows omitting the input from the parameters of the @asset
- or @multi_asset
- decorated function. This is useful when you want to specify a partition mapping or metadata for a non-managed input.start_offset
and end_offset
arguments of TimeWindowPartitionMapping
now work across TimeWindowPartitionsDefinitions
with different start dates and times.add_output_metadata
is called multiple times within an op, asset, or IO manager handle_output
, the values will now be merged, instead of later dictionaries overwriting earlier ones.materialize
and materialize_to_memory
now both accept a tags
argument.SingleDimensionDependencyMapping
, a PartitionMapping
object that defines a correspondence between an upstream single-dimensional partitions definition and a downstream MultiPartitionsDefinition
.RUN_DEQUEUED
event has been removed from the event log, since it was duplicative with the RUN_STARTING
event.raise Exception() from e
syntax.runK8sConfig
key in the k8sRunLauncher
section. See the docs for more information.securityContext
can now be set in the k8sRunLauncher
section of the Dagster Helm chart.EcsRunLauncher
can now be configured with cpu and memory resources for each launched job. Previously, individual jobs needed to be tagged with CPU and memory resources. See the docs for more information.S3ComputeLogManager
now takes in an argument upload_extra_args
which are passed through as the ExtraArgs
parameter to the file upload call.make_dagster_definitions_from_airflow_dags_path
and make_dagster_definitions_from_airflow_dag_bag
which are passed through as the ExtraArgs
parameter to the file upload call.ExperimentalWarning
s related to LogicalVersions
to appear even when version-based staleness was not in use.load_assets_from_modules
and load_assets_from_package_module
utilities now will also load cacheable assets from the specified modules.dequeue_num_workers
config setting on QueuedRunCoordinator
is now respected.databricks_pyspark_step_launcher
will now cancel the relevant databricks job if the Dagster step execution is interrupted.databricks_pyspark_step_launcher
could exit with an unhelpful error after receiving an HTTPError from databricks with an empty message. This has been fixed.execute_queries
or execute_query
on a snowflake_resource
would raise an error unless the parameters
argument was explicitly set.EcsRunLauncher
when launching many runs in parallel. Previously, each run risked hitting a ClientError
in AWS for registering too many concurrent changes to the same task definition family. Now, the EcsRunLauncher
recovers gracefully from this error by retrying it with backoff.make_dagster_definitions_from_airflow_dags_path
and make_dagster_definitions_from_airflow_dag_bag
for creating Dagster definitions from a given airflow Dag file path or DagBagUPathIOManager
, thanks @danielgafni!FakeS3Session
now includes additional functions and improvements to align with the boto3 S3 client API, thanks @asharov!BranchingIOManager
to model use case where you wish to read upstream assets from production environments and write them into a development environment.create_repository_using_definitions_args
to allow for the creation of named repositories.DbtManifestAssetSelection
, which allows you to define selections of assets loaded from a dbt manifest using dbt selection syntax (e.g. tag:foo,path:marts/finance
).@repository
, replacing them with Definitions
.https://github.com/dagster-io/dagster/compare/1.1.7...1.1.8
Adjust resources guide to be in a Definitions world
by @schrocknadd thread name prefix to run dequeue workers (#11155)
by @alangenfeldmake schedules produced by build_schedule_from_partitioned_job more p… (#11147)
by @sryza[k8s launcher] security context (#9788)
by @alangenfeld[run coordinator] fix threaded tests (#11139)
by @alangenfeld[docs] - [definitions] Update Dagit + tutorial screenshots (#11089)
by @erinkcochran87[docs] - [definitions] Update Partitions concept docs (#11030)
by @erinkcochran87Port asset sensor guide to Definitions
by @schrocknUse buildkite_deps.txt to declare explicit buildkite deps (#11025)
by @jmsandersTrigger builds when .ini files change (#11161)
by @jmsandersRevert "keep track of max timestamps client side for code location up… (#11162)
by @prha[docs] - Fix links (#11163)
by @erinkcochran87Add create_repository_using_definitions_args
by @schrocknChange Graph-backed asset guide code examples to be on Definitions
by @schrocknDelete repository unit testing using load_all_definitions in testing guide
by @schrockn1.1.7 Changelog (#11165)
by @jamiedemariaDo a single pip install in tox suites [OSS] (#11164)
by @gibsondanUpdate declarative scheduling guide to include Definitions
by @schrocknadd thread name prefix to grpc server (#11158)
by @alangenfeldAutomation: versioned docs for 1.1.7
by @elementl-devtoolsFixup rename of requirements.txt -> buildkite_deps.txt (#11160)
by @jmsandersfix precedence ordering when merging dictionaries in container context (#11169)
by @gibsondansupport in and len on PartitionsSubsets (#11172)
by @sryzaDisable breaking azure tests in master (#11183)
by @schrockn[docs] Fix typo in title (#11156)
by @vpicavetFix regression with passing in None to snowflake resource (#11182)
by @gibsondan[dagit] Add tests for partition health data parsing / accessors (#11114)
by @bengotowAllow setting resources on EcsContainerContext and EcsRunLauncher (#11170)
by @gibsondanfix a small bug in UPathIOManager (#11110)
by @danielgafni[dagster-dbt] in tests, pin dbt rpc < 0.3.0 (#11196)
by @OwenKephartMove core_tests/storage_tests to storage_tests (#11180)
by @schrocknskip sqlite env var test on windows (#11195)
by @gibsondanMove old_sqlalchemy_tests to only run on storage_tests (#11181)
by @schrockn[dagster-airflow] airflow terminology mapping (#11015)
by @Ramshackle-JamathonMove core_tests/definitions_tests to definitions_tests (#11184)
by @schrocknfix dequeue_num_workers setting (#11198)
by @alangenfeldMove core_tests/asset_defs_tests to asset_defs_tests (#11186)
by @schrocknupdate timeout in sensor run tests (#11193)
by @jamiedemariaMove core_tests/launcher_tests to launcher_tests (#11187)
by @schrocknadd assets example to AssetSelection apidoc (#11194)
by @sryzaMove various logging tests into logging_tests (#11188)
by @schrocknfeat(dbt-cloud): add Dagster run id to dbt Cloud run (#11005)
by @rexledesma[docs] snowflake reference page (#10985)
by @jamiedemaria[declarative-scheduling] Fix bug with declarative scheduling where repeated calls to get_latest_materialization_record could return incorrect results (#11214)
by @OwenKephartonly reconcile multi-assets if all parents are reconciled (#11190)
by @sryza[dagit] Fix dragging on the DAG clearing your selection / clicking links (#11202)
by @bengotow[dagster-aws] Extend fake S3 resource (#11105)
by @asharov[dagit] Optimizations to backfill UI for large partition key sets (#11201)
by @bengotowdo not compute projected logical versions of partitioned assets (#11204)
by @sryzafactor out asset reconciliation graph traversal into util (#11206)
by @sryza[dagit] Fresh + Stale should not show “Stale” on the Asset Graph (#11234)
by @bengotow[dagit] Round middle truncate calculations for Firefox (#11203)
by @bengotowsupport tags argument on materialize and materialize_to_memory (#11225)
by @sryzadocs: update license to include year and company (#11231)
by @vwbusguy[docs] - [definitions] Update OSS deployment overview (#11199)
by @erinkcochran87split backfill table so partition status is fetched lazily (#11205)
by @prhafix tslint (#11242)
by @alangenfeld[dagit] webpack-bundle-analyzer (#11230)
by @hellendag[dagit] Virtualized Asset Catalog (#11168)
by @hellendagCan remove hardcoded_resource from project_fully_featured because of Definitions (#11243)
by @schrockn[dagit] Replace moment-timezone (#11197)
by @hellendagRemove unused backfillStatus, which loads individual run status (#11246)
by @prhaRetry registering task definitions (#11192)
by @jmsandersRerun dagster_tests with --snapshot-update (#11208)
by @schrocknAdd test case for binding assets before passing to Definitions (#11216)
by @schrockn[dagster-airflow] from airflow to dagster guide updates (#11218)
by @Ramshackle-Jamathon[graphql test] share schema instance (#11236)
by @alangenfeldUse FromSourceAsset instead of FromRootInputManager when loading assets with input managers (#11233)
by @jamiedemariaUse bare objects for the hacker news resources (#11249)
by @schrocknUse bare I/O manager in fully featured (#11250)
by @schrocknDelete unused fixed_s3_pickle_io_manager (#11251)
by @schrocknConsolidate _resolve_bound_config and have logger and resource use the same one (#11209)
by @schrocknDelete op version of _resolve_bound_config and call generic one (#11211)
by @schrocknSkip race condition tests (#11286)
by @jmsandersSkip flaky test (#11292)
by @jmsanders[dagit] Replace remaining moment usage (#11278)
by @hellendagUpdate op-retries.mdx - no solid (#11284)
by @yuhan[docs] airbyte guide repository -> definitions (#11296)
by @yuhan[docs] fivetran guide repository -> definitions (#11297)
by @yuhan[docs] dbt guide repository -> definitions (#11298)
by @yuhan[docs] dbt cloud guide repository -> definitions (#11299)
by @yuhan[declarative-scheduling] Rework constraint passing (#11229)
by @OwenKephart[dagster-airflow] basic airflow migration guide (#11012)
by @Ramshackle-Jamathon[dagstermill] add retries to flaky tests (#11291)
by @jamiedemariaProduct tour component (#11227)
by @salazarm[dagit] Utility for timezone-aware date/time formatting (#11285)
by @hellendag[dagit] Restore some metadata on Code Locations page (#11281)
by @hellendag[docs] - [definitions] Update Loggers Concept docs (#11171)
by @erinkcochran87[docs] - Update ECS deployment guide (#11289)
by @erinkcochran87[docs] - [definitions] Update Dagster instance docs (#11241)
by @erinkcochran87Pass duckdb_path to __init__ rather than relying on context (#11300)
by @schrockn[structured config] Base structured config implementation (#11268)
by @benpankow[structured config] Add support for default values (#11272)
by @benpankowAdd gql pin (#11312)
by @gibsondan[structured config] Add support for class, field descriptions (#11274)
by @benpankowMove env var injection earlier in step command (#11239)
by @gibsondanaccrete metadata with multiple calls to add_output_metadata (#9518)
by @sryza[docs] - [definitions] - Update Executors docs (#11247)
by @erinkcochran87[docs] - [definitions] Update Helm guide (#11320)
by @erinkcochran87Eliminating unnecessary output_context.resource_config check (#11313)
by @schrockn[docs] - Consolidate sections in Resources concept doc (#11316)
by @erinkcochran87[docs] - [definitions] Update SDA Concept docs (#11018)
by @erinkcochran87[docs] - [definitions] Update Run launchers guide (#11200)
by @erinkcochran87[docs] - Re-do Cloud Getting Started guides (#10429)
by @erinkcochran87[docs] - [definitions] Update Docker guide (#11295)
by @erinkcochran87backfill perf: swap backfill requested for num cancelable (#11304)
by @prhaallow upload config to pass through s3 compute log manager (#11317)
by @prha[apidoc] define_asset_job repository -> Definitions (#11302)
by @yuhanIgnore stale timestamps from code location updates (#11173)
by @prha[structured config] Fix usage with Assets (#11327)
by @benpankowInclude parent exceptions in Dagster errors, even if they weren't explicitly raised (#11306)
by @gibsondan[docs] - [definitions] Update Dagster daemon docs (#11226)
by @erinkcochran87Docs release backfill 1.1.7 (#11328)
by @yuhanUse dark mode logo on README.md (#11326)
by @hellendagAssetGraph.from_external_assets().get_required_multi_asset_keys() (#11318)
by @sryza[declarative-scheduling] Update retry logic to attempt to retry failed materializations after some time has passed (#11294)
by @OwenKephartAdd instance property to InputContext (#11331)
by @schrocknchore: add method to strip error stack trace (#11307)
by @rexledesmaHoist schema and database to DbIOManager constructor (#11301)
by @schrocknRename _resolve_bound_config to resolve_bound_config (#11287)
by @schrocknAdd asset_materialization property to EventLogEntry (#11340)
by @schrocknstore repository on external asset graph (#11332)
by @sryzaRevert "Use dark mode logo on README.md (#11326)" (#11345)
by @hellendag[dagster-io/eslint-config] v1.0.6 (#11324)
by @hellendagAdd get_implicit_global_asset_job on Definitions and RepositoryDefinition (#11279)
by @schrocknfeat: add retry number and url to integration api call failure (#11308)
by @rexledesmaFix dagster-graphql circular import (BK broken) (#11339)
by @smackeseySkipping dbt rpc resource tests (#11357)
by @schrocknskip test_threaded_ephemeral_instance (#11359)
by @schrocknAllow specifying resources in Op/Asset params list (#11322)
by @benpankowchange version placeholder 0+dev -> 1!0+dev (#11334)
by @smackesey[structured config] Structured-config-backed Resources (#11321)
by @benpankow[structured config] Add traditional resource wrapper base class (#11337)
by @benpankow[structured config] Fix test importing functools.cached_property on py37 tests (#11361)
by @benpankow[structured config] Structured-config-backed IO managers (#11343)
by @benpankowfix some bugs in ExternalAssetGraph (#11350)
by @sryzamore refactors to reconciliation sensor (#11223)
by @sryza[dagster-io/eslint-config] v1.0.7 (#11347)
by @hellendagAdd Materialize Button hook (#11319)
by @salazarmAllow setting raw k8s config at the run launcher / container context level (#11333)
by @gibsondanMake it more likely that we hit our lock (#11290)
by @jmsandersfix __contains__ of TimeWindowPartitionsSubset (#11380)
by @sryza[docs] - add note on pandas integration to redirect users interested in pandas w/out validation (#11342)
by @sloppfix asset reconciliation bug that ignores earlier partitions (#11336)
by @sryzafix typo on partitions concepts page (#11349)
by @sryzaClicking on top-level concept sections takes you to a page (#10140)
by @sryzaAdd single dimension -> multidimension partition mapping (#10910)
by @clairelin135[RFC] Update cached_status_data column 1/2 (#10821)
by @clairelin135Remove stray console.log (#11410)
by @gibsondanExecuteInProcessResult.asset_value (#11403)
by @sryzaElminate PIPELINE_DEQUEUED event (#11393)
by @gibsondan[dagit] Begin adding new graphql-codegen (#11411)
by @hellendagDocs for new ECS resource options (#11391)
by @gibsondanDocs for new k8s configuration options (#11390)
by @gibsondanFix malformed dagster_cloud.yaml in code location docs (#11416)
by @gibsondan[dagit] Refactor mutations to new GraphQL codegen (#11414)
by @hellendag[dagit] Refactor queries in src/workspace (#11418)
by @hellendag[dagit] dedupeFragments (#11426)
by @hellendagrelax matching criteria for TimeWindowPartitionMappings w/offsets (#11422)
by @sryza[dagster-databricks] handle empty responses (#11430)
by @OwenKephartExport RunRecord in the public API (#11427)
by @gibsondan[dagit] Refactor queries in src/runs (#11419)
by @hellendag[dagit] Refactor queries in Assets (#11423)
by @hellendag[dagit] Refactor queries in instance, instigation, launchpad (#11429)
by @hellendagBranching I/O Manager (#11315)
by @schrocknFix issues with a SerializableErrorInfo being coerced to a GraphenePythonError (#11437)
by @gibsondanexport product tour component (#11314)
by @salazarm[dagit] Refactor queries in remaining src (#11435)
by @hellendagEliminate unused config_or_config_fn arg in copy_for_configured (#11433)
by @schrockn[dagit] Don’t show assets as stale if projectedLogicalVersion is null (#11224)
by @bengotow[dagit] Show observations about the latest materialization on Asset > Partitions (#11288)
by @bengotowasset backfill core logic (#11377)
by @sryzaAllow AssetIn(dagster_type=Nothing) (#11436)
by @sryzaRename and refactor in structured_config.py for clarity (#11372)
by @schrocknConsolidate all logic and trickery to support private cached properties in pydantic in a single class (#11373)
by @schrockndeprecate job-level memoization in docs (#11392)
by @sryzaAdd StructuredIOManagerAdapter (#11383)
by @schrockn[dagit] Fix line height and size of timestamps on the asset graph (#11465)
by @bengotowDefault new-style config mappings to *Source equivalents rather than raw scalars (#11386)
by @schrocknUnexperimentalize LogicalVersion and quell warning messages whenever assets are materialized (#11407)
by @sryzaPassthrough pydantic.ModelField.required to dagster.Field.is_required (#11388)
by @schrocknSupport pydantic aliases in config schema mapping (#11389)
by @schrocknMake conversion to *Source types work on direct annotations of the config parameter (#11469)
by @schrockn[dagster-databricks] handle execution interrupts (#11421)
by @OwenKephartSkip flaky race condition test (#11471)
by @jmsandersremove AssetStoreHandle (#11412)
by @sryzarefactor: sequester job backfill code (#11252)
by @sryzaBe more resilient to user code errors when dequeuing runs (#11406)
by @gibsondanMake all arguments on DagsterInstance.create_run keyword-only (#11446)
by @schrocknElimiinate default value for asset_selection param on DagsterInstance.create_run (#11447)
by @schrocknEliminate default values for solid_selection, external_pipeline_origin, and pipeline_code_origin on create_run (#11448)
by @schrocknrename create_run to create_run_for_test on TestQueuedRunCoordinator to increase greppability of create_run (#11449)
by @schrockn[dagit] Remove old GraphQL Codegen (#11474)
by @hellendag[dagit] Delete apollo CLI dep (#11478)
by @hellendagAdd tag_concurrency_limits config to executors (#11472)
by @gibsondanadd run filter for updated before (#11481)
by @prhadocs(dagster-dbt): clarify that the integration supports arbitrary dbt profiles (#11351)
by @rexledesma[structured config] Add support for Permissive fields (#11275)
by @benpankowhandle pure asset backfills in backfill daemon (#11378)
by @sryzaFix missing callsite of tag_concurrency_limits (#11496)
by @gibsondanfix(docs): format docs (#11501)
by @rexledesma[dagster-dbt] add DbtManifestAssetSelection (#11473)
by @OwenKephartAdd parameter invariants around external_pipeline_origin and pipeline_code_origin arguments (#11450)
by @schrocknsupply solid_selection to fix submitting runs from pure asset backfills (#11502)
by @sryzatelemetry: add num_assets_in_repo in repo-level metadata (#11490)
by @yuhanfix(docs): run mdx-format again (#11504)
by @rexledesmaTighten solids_to_execute and solid_selection invariant (#11451)
by @schrocknBump json5 from 1.0.1 to 1.0.2 in /js_modules/dagit (#11485)
by @dependabot[bot]Add invariant for asset_selection (#11452)
by @schrocknTypehint pipeline_name, run_id, and mode (#11453)
by @schrockngraphql for pure asset backfills (#11379)
by @sryzadocs for new tag_concurrency_limits feature on executor (#11499)
by @gibsondanAdd support for loading cacheable assets from module (#10389)
by @benpankowenable filtering by asset tag when asset tags table is not present (#11509)
by @sryzaonly count materializations within backfill (#11506)
by @sryzatelemetry: add num_assets_in_repo in log_repo_stats metadata (#11513)
by @yuhantelemetry: add location_name in repo-level metadata (#11514)
by @yuhan[dagster-airflow] warn on airflow dataset (#11498)
by @Ramshackle-Jamathon[dagster-airflow] add
make_dagster_definitions_from_airflow_dags_pathand
make_dagster_definitions_from_airflow_dag_bag apis (#11441)
by @Ramshackle-JamathonChangelog 1.1.8 (#11526)
by @clairelin1351.1.8
by @elementl-devtoolsPublished by yuhan almost 2 years ago
Definitions
is no longer marked as experimental and is the preferred API over @repository
for new users of Dagster. Examples, tutorials, and documentation have largely ported to this new API. No migration is needed. Please see GitHub discussion for more details./locations
.TimeWindowPartitionMapping
now accepts start_offset
and end_offset
arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets.dagit
can now accept multiple arguments for the -m
and -f
flags. For each argument a new code location is loaded.build_schedule_from_partitioned_job
now execute more performantly - in constant time, rather than linear in the number of partitions.QueuedRunCoordinator
now supports options dequeue_use_threads
and dequeue_num_workers
options to enable concurrent run dequeue operations for greater throughput.load_assets_from_dbt_project
, load_assets_from_dbt_manifest
, and load_assets_from_dbt_cloud_job
now support applying freshness policies to loaded nodes. To do so, you can apply dagster_freshness_policy
config directly in your dbt project, i.e. config(dagster_freshness_policy={"maximum_lag_minutes": 60})
would result in the corresponding asset being assigned a FreshnessPolicy(maximum_lag_minutes=60)
.DAGSTER_RUN_JOB_NAME
environment variable is now set in containerized environments spun up by our run launchers and executor.make_dagster_repo_from_airflow_dags_path
,make_dagster_job_from_airflow_dag
and make_dagster_repo_from_airflow_dag_bag
have a new connections
parameter which allows for configuring the airflow connections used by migrated dags.Fixed a bug where the log
property was not available on the RunStatusSensorContext
context object provided for run status sensors for sensor logging.
Fixed a bug where the re-execute button on runs of asset jobs would incorrectly show warning icon, indicating that the pipeline code may have changed since you last ran it.
Fixed an issue which would cause metadata supplied to graph-backed assets to not be viewable in the UI.
Fixed an issue where schedules often took up to 5 seconds to start after their tick time.
Fixed an issue where Dagster failed to load a dagster.yaml file that specified the folder to use for sqlite storage in the dagster.yaml
file using an environment variable.
Fixed an issue which would cause the k8s/docker executors to unnecessarily reload CacheableAssetsDefinitions (such as those created when using load_assets_from_dbt_cloud_job
) on each step execution.
[dagster-airbyte] Fixed an issue where Python-defined Airbyte sources and destinations were occasionally recreated unnecessarily.
Fixed an issue with build_asset_reconciliation_sensor
that would cause it to ignore in-progress runs in some cases.
Fixed a bug where GQL errors would be thrown in the asset explorer when a previously materialized asset had its dependencies changed.
[dagster-airbyte] Fixed an error when generating assets for normalization table for connections with non-object streams.
[dagster-dbt] Fixed an error where dbt Cloud jobs with dbt run
and dbt run-operation
were incorrectly validated.
[dagster-airflow] use_ephemeral_airflow_db
now works when running within a PEX deployment artifact.
Definitions
Definitions
. Any content not ported to Definitions
in this release is in the process of being updated.https://github.com/dagster-io/dagster/compare/1.1.6...1.1.7
Non isolated runs docs (#10860)
by @johannkm[dagit] Fix Gantt chart rendering of per-step resource init log messages (#10943)
by @bengotow[dagit] Fix the “Assets” label on large asset runs (#10932)
by @bengotow[dagit] Fix “Job In” label regression in Chrome v109 (#10934)
by @bengotow[dagit] Pass repository tag when loading runs for Partitions page (#10948)
by @bengotow[dagit] Updated asset DAG styles, added additional compute tags (#10931)
by @bengotow[docs] - add a guide for scheduling assets (#10949)
by @sloppcap packaging requirement at 22.0 (#10968)
by @smackesey Execution result typing (#10919)
by @smackeseysolid -> node method renames (#10920)
by @smackesey[dagster-airflow] re-enable airflow 2.5.0 tests (#10966)
by @Ramshackle-Jamathon[dagster-airbyte][docs] Use dagster-airbyte CLI alias in docs (#10955)
by @benpankow[dagster-slack] create slack_on_freshness_policy_sensor (#10960)
by @OwenKephart[docs ] - fix image dimensions in hello-dagster materialize (#10977)
by @slopp1.1.6 Changelog (#10978)
by @OwenKephartAutomation: versioned docs for 1.1.6
by @elementl-devtools[convert-environment-variables-and-secrets-guide-stack-2] Convert env vars and secrets guide from repository to Definitions
by @schrocknunexperimentalize PartitionMapping (#10980)
by @sryzaremove validation in AssetGraph.get_child_partition_keys_of_parent an… (#10981)
by @sryza[convert-development-to-production-1] Move repository to __init__.py
by @schrockn[convert-development-to-production-2] Changing development to production to use snowflake_pandas_io_manager
by @schrockn[convert-deployment-to-production-3] Convert @repository to Definitions
by @schrockn[convert-development-to-production-4] Use base object instead of resource
by @schrockn[convert-development-to-production-6] Use pyproject.toml instead of workspace.yaml
by @schrockn[convert-development-to-production-7] Convert guide to use Definitions
by @schrockn[dagit] New Code Locations table (#10975)
by @hellendagPin graphene to <3.2
by @schrocknfix missing metadata in dagit on graph-backed assets (#10988)
by @OwenKephart[dagit] Export a few Code Location components for Cloud (#11008)
by @hellendagTemporarily disable some Azure test suites (#11007)
by @jmsanders[graphql] fix for graphene 3.2 (#11011)
by @alangenfeld[code-location-selector-stack] Code location sensor tests 1/N. Rename workspace_load_target function to create_workspace_load_target
by @schrockn[code-location-selector-stack] Code location sensor tests 2/N Make instance_with_multiple_repos_with_sensors workspace_load_target parameterizable
by @schrockn[code-location-selector-stack] Code location sensor tests 3/N. Refactor instance_with_multiple_repos_with_sensors to handle multiple code locations
by @schrockn[code-location-selector-stack] Code location sensor tests 4/N Actually add test to test cross code location selector
by @schrockn[code-location-selector-stack Add CodeLocationSelector; Have run_status_sensor accept it
by @schrocknFix test_persistent
by @schrocknchore: auto-assign dependabot pull requests (#10953)
by @rexledesmafix(dbt-cloud): parse command string to find materialization commands (#10989)
by @rexledesma[code-location-selector-stack] Change typehint on make_slack_on_run_failure_sensor to accept CodeLocationSelector
by @schrockn[docs] re-org snowflake integration guide (#10984)
by @jamiedemariaCheck scheduler ticks right after each minute boundary instead of once every 5 seconds (#10886)
by @gibsondan[docs] - [definitions] Update Configured API concept doc (#11020)
by @erinkcochran87Fix bug with re-execution snapshot ids (#10967)
by @OwenKephart[bugfix] UPathIOManger load_input type checking (#11022)
by @danielgafniPathspec typing fix (#11036)
by @smackesey[dagit] /code-locations -> /locations (#11024)
by @hellendagReplace partition ranges with subsets (#10909)
by @clairelin135[dagit] With multiple assets selected, backfill “missing” should include partially materialized partitions (#11027)
by @bengotow[dagit] Add empty value string for invalid tag input in Runs filter (#11044)
by @hellendag[docs] - [definitions] Update Repository page for Definitions (#10986)
by @erinkcochran87[definitions-accessors] Add get_job_def to
Definitions.
by @schrockn1/ definitions in create new project: update dagster project CLI (#10829)
by @yuhan2/ definitions in create new project: update create-new-project docs (#10830)
by @yuhan[docs] - [definitions] - Update dbt tutorial to use Definitions (#10842)
by @erinkcochran872.1/ definitions in in create new project: remove unnecessary asset dir in scaffold + update docs (#10831)
by @yuhanupdate isolated run docs to say that it's enabled by default (#11039)
by @gibsondan[docs] - remove repository from hello dagster as its not needed (#11048)
by @slopp[dagster-fivetran][ez] Add missing docstring entry for build_fivetran_assets (#11049)
by @benpankowAdd start_offset and end_offset to TimeWindowPartitionMapping (#10979)
by @sryzafix subsettable multi asset case (#10878)
by @OwenKephartmove ecs task tagging to run_task call (#11037)
by @prhaAdd custom resource key to
load_assets_from_dbt_project (#10827)
by @dpeng817[dagit] Highlight Assets in top nav for global asset graph (#11052)
by @hellendag[dagster-airbyte][docs][ez] Fix AirbyteConnection apidoc rendering (#11059)
by @benpankowconvert-examples 1.1/ assets_dbt_python defs + pyproject (#11060)
by @yuhanconvert-examples 2.1/ assets_modern_data_stack repo -> defs, pyproject (#11062)
by @yuhanconvert-examples 5/ assets_smoke_test repository -> definitions (#11071)
by @yuhanconvert-examples 6.1/ feature_graph_backed_assets pyproject, @repository -> Definitions (#11072)
by @yuhanconvert-examples 7.1/ quickstart_etl workspace->pyproject, repo->defs (#11074)
by @yuhanconvert-examples 8.1/ with_great_expectations workspace->pyproject, repo->defs (#11077)
by @yuhanconvert-examples 1.2/ assets_dbt_python file renames (#11061)
by @yuhanconvert-examples 6.2/ feature_graph_backed_assets file rename and reorder (#11073)
by @yuhanconvert-examples 2.2/ assets_modern_data_stack file renames (#11063)
by @yuhanRemoves codecov commands from tox (#10635)
by @dpeng817[dagit] Disable Re-execute menu item in run action menu for perms (#11026)
by @hellendagSupport loading multiple modules in single command line invocation
by @schrocknEnable multiple files in CLI tools
by @schrocknAdd get_sensor_def and get_schedule_def to Definitions
by @schrocknconvert-examples 10/ with_pyspark_emr repository.py -> defs.py, update [docs] (#11080)
by @yuhanconvert-examples 9/ with_pyspark repository.py -> defs.py, update [docs] (#11079)
by @yuhanAdd load_asset_value and get_asset_value_loader to Definitions
by @schrockn[dagster-airflow] reorg dagster airflow api docs (#11009)
by @Ramshackle-Jamathonconvert-examples 3.1/ assets_pandas_pyspark definitons, pyproject (#11067)
by @yuhanconvert-examples 8.2/ with_great_expectations file rename (#11078)
by @yuhanconvert-examples 7.2/ quickstart_etl file rename + update README (#11075)
by @yuhanSupport module_name in tool.dagster section of pyproject.toml
by @schrocknMove all examples to use module_name in pyproject.toml
by @schrocknPin tox on windows to < 4 (#11013)
by @jmsanders[docs] - [definitions] Code location concept page (#10843)
by @erinkcochran87Add instance for test to top level API (#10709)
by @dpeng817[example] recomment code in dagstermill tutorial (#11098)
by @jamiedemariaMake JobSelector work on cross code location
by @schrockn[convert-branch-deployments-stack] Convert branch deployments guide to Definitions
by @schrocknEliminate package_name from pyproject.toml loading spec
by @schrockn[docs] - [definitions] Update Enriching with SDAs guide (#10849)
by @erinkcochran87[dagit] Don't show View link on code location toast if already there (#11095)
by @hellendagset DAGSTER_RUN_JOB_NAME env var (#10888)
by @alangenfeldremove parameter tag from dagstermill notebook (#11109)
by @jamiedemariasort task definition config secrets and env vars to ensure consistent ordering (#10982)
by @gibsondan[dagster-airflow] from airflow to dagster guide (#10923)
by @Ramshackle-Jamathon[dagit] Improvements to partition range selection interactions (#11017)
by @bengotowadd tests for legacy storage (#11047)
by @prhakeep track of max timestamps client side for code location updates (#11100)
by @prhain default IO manager, handle PartitionMappings that return 0 partitions (#11065)
by @sryzaFix bug with logical versions for assets with changed deps (#11083)
by @smackeseyAdding apidocs for Definitions
by @schrocknconvert-examples 3.2/ asset_pandas_pyspark re-arrange files + [docs] (#11068)
by @yuhan[dagit] Persist AssetPartitions state to URL, open Materialize panel with same selection (#11042)
by @bengotow[dagit] Render assets with self-dependencies in Dagit (#11043)
by @bengotowrun queue daemon refactor (#11090)
by @alangenfeldAllow assets to depend on earlier partitions of themselves (#11066)
by @sryza[docs] - [definitions] Update Schedules documentation (#11046)
by @erinkcochran87Re-add section for customizing Docker images pre-fast-deploys (#11112)
by @shalabhcAllow setting base_dir as a StringSource in sqlite storage (#11031)
by @gibsondanhandle self-dependencies in reconciliation sensor (#11085)
by @sryza[docs] - [definitions] Update Workspaces concept page (#10944)
by @erinkcochran87Coerce IOManager objects into IOManagerDefinitions in Definitions
by @schrocknRevert "[docs] - [definitions] Update Workspaces concept page (#10944)"
by @alangenfeldAllow overriding the default max_concurrent for the default executor via env var (#11116)
by @gibsondanconvert-examples 4.1/ assets_pandas_type_metadata definitions, pyproject (#11069)
by @yuhan[docs] [definitions] convert snowflake guide to definitions (#11029)
by @jamiedemaria[docs] - Fix workspace link (#11121)
by @erinkcochran87convert-examples 4.2/ assets_pandas_type_metadata repository.py -> __init__.py (new) (#11127)
by @yuhanAdd executor and loggers to Definitions
by @schrockn[docs] - [definitions] Update Sensors documentation (#11050)
by @erinkcochran87threaded run queue daemon (#11113)
by @alangenfeldconvert-examples 11.1/ project_fully_featured defs, pyproject, no with_resources (#11081)
by @yuhan[dagster-airbyte] Fix normalization table generation, materializations for non-object Streams (#10899)
by @benpankowupdate assets_modern_data_stack example readme (#11122)
by @yuhan[CacheableAssets] Fix bug that would cause definitions to be recomputed when using a StepDelegatingExecutor (#11086)
by @OwenKephartconvert-examples 11.2/ project_fully_featured repository.py -> __init__.py (#11082)
by @yuhanupdate assets_dbt_python example readme (#11126)
by @yuhan[dagster-dbt] Enable setting FreshnessPolicies on dbt assets (#11103)
by @OwenKephartMark Definitions as not experimental
by @schrockn[declarative-scheduling] Fix issue where in-progress runs were not properly handled (#11118)
by @OwenKephart[dagster-airbyte] Avoid unnecessarily recreating sources, destinations w/ managed ingestion (#11117)
by @benpankow[dagster-airflow] pex compatibility for ephemeral db (#11115)
by @Ramshackle-Jamathon[dagster-airflow] provide interface for passing connection models directly to dagster (#11006)
by @Ramshackle-Jamathon[dagit] Left nav: show entire code location string for non-dunder repos (#11133)
by @hellendagmove quickstart_aws back to mono repo (#11130)
by @yuhanmove quickstart_gcp back to mono repo (#11131)
by @yuhanmove quickstart_snowflake back to mono repo (#11132)
by @yuhanUpdate Definitions docs
by @schrocknUse wrap_resources_for_execution in Definitions
by @schrocknUse i/o manager coercion in assets_pandas_pyspark example
by @schrocknconvert-examples-repo-to-defs cloud nux quickstart_aws (#11135)
by @yuhanconvert-examples-repo-to-defs cloud nux quickstart_gcp (#11136)
by @yuhanconvert-examples-repo-to-defs cloud nux quickstart_snowflake (#11137)
by @yuhanenable run status sensor logging (#11145)
by @prhaMove jobs concept guide to refer to Definitions
by @schrocknMove I/O Manager Guide to be on Definitions
by @schrocknadd thread name prefix to run dequeue workers (#11155)
by @alangenfeldAdjust resources guide to be in a Definitions world
by @schrocknmake schedules produced by build_schedule_from_partitioned_job more p… (#11147)
by @sryza[docs] - [definitions] Update Dagit + tutorial screenshots (#11089)
by @erinkcochran87[docs] - [definitions] Update Partitions concept docs (#11030)
by @erinkcochran87Port asset sensor guide to Definitions
by @schrocknRevert "keep track of max timestamps client side for code location up… (#11162)
by @prha[docs] - Fix links (#11163)
by @erinkcochran87Merge branch 'release-1.1.7' of github.com:dagster-io/dagster into release-1.1.7
by @jamiedemaria1.1.7 Changelog (#11165)
by @jamiedemaria1.1.7
by @elementl-devtoolsPublished by gibsondan almost 2 years ago
__repository__
is used for a repo, only the code location name will be shown. This change also applies to URL paths.load_asset_value
to error with the default IO manager when a partition_key
argument was provided.context.partition_key
or context.asset_partition_key_for_output
when invoking an asset directly (e.g. in a unit test) would result in an error. This has been fixed.RetryRequested
when using a retry policy.sqlite3.ProgrammingError
error was raised when creating an ephemeral DagsterInstance
, most commonly when build_resources
was called without passing in an instance parameter.duckdb_pyspark_io_manager
helper to automatically create a DuckDB I/O manager that can store and load PySpark DataFrames.8.0.31
would raise an error on some run queries.Definitions
entrypoint for tools and the UI has been added. A single Definitions
object per code location may be instantiated, and accepts typed, named arguments, rather than the heterogenous list of definitions returned from an @repository
-decorated function. To learn more about this feature, and provide feedback, please refer to the Github Discussion.make_slack_on_freshness_policy_status_change_sensor
allows you to create a sensor to alert you when an asset is out of date with respect to its freshness policy (and when it’s back on time!)dagstermill
guide and reference page https://docs.dagster.io/integrations/dagstermill
dagster-snowflake
guide: https://docs.dagster.io/integrations/snowflake
Published by gibsondan almost 2 years ago
Published by gibsondan almost 2 years ago
job_name
parameter to InputContext
.execute_in_process
on a GraphDefinition
(it would use the fs_io_manager
instead of the in-memory io manager)./instance
URL path prefix has been removed. E.g. /instance/runs
can now be found at /runs
./workspace
URL path prefix has been changed to /locations
. E.g. the URL for job my_job
in repository foo@bar
can now be found at /locations/foo@bar/jobs/my_job
.dagstermill.yield_event
.save_on_notebook_failure
parameter.use_ephemeral_airflow_db
which will create a job run scoped airflow db for airflow dags running in dagsterAssetKey
s.1.0.16
for some compute log managers where an exception in the compute log manager setup/teardown would cause runs to fail.prefix
argument to prevent badly constructed paths.tag:
. This resolves an issue where retrieving the available tags could cause significant performance problems. Tags can still be searched with freeform text, and by adding them via click on individual run rows.get_output_asset_key
on the IOManager
handling the output. Previously, this was experimental and undocumented.log
property, which log events that can later be viewed in Dagit. To enable these log views in dagit, navigate to the user settings and enable the Experimental schedule/sensor logging view
option. Log links will now be available for sensor/schedule ticks where logs were emitted. Note: this feature is not available for users using the NoOpComputeLogManager
.Published by rexledesma almost 2 years ago
BETWEEN
was used to determine the section of the table to replace. BETWEEN
included values from the next partition causing the I/O manager to erroneously delete those entries.BETWEEN
was used to determine the section of the table to replace. BETWEEN
included values from the next partition causing the I/O manager to erroneously delete those entries.Published by sryza almost 2 years ago
Published by sryza almost 2 years ago
MultiPartitionsDefinition
API. In Dagit, you can filter and materialize certain partitions by providing ranges per-dimension, and view your materializations by dimension.build_asset_reconciliation_sensor
.FreshnessPolicy
to any of your software-defined assets, to specify how up-to-date you expect that asset to be. You can view the freshness status of each asset in Dagit, alert when assets are missing their targets using the @freshness_policy_sensor
, and use the build_asset_reconciliation_sensor
to make a sensor that automatically kick off runs to materialize assets based on their freshness policies.op_version
s to software-defined assets or observation_fn
s to SourceAsset
s. When a set of assets is versioned in this way, their “Upstream Changed” status will be based on whether upstream versions have changed, rather than on whether upstream assets have been re-materialized. You can launch runs that materialize only stale assets.@multi_asset_sensor
decorator enables defining custom sensors that trigger based on the materializations of multiple assets. The context object supplied to the decorated function has methods to fetch latest materializations by asset key, as well as built-in cursor management to mark specific materializations as “consumed”, so that they won’t be returned in future ticks. It can also fetch materializations by partition and mark individual partitions as consumed.RepositoryDefinition
now exposes a load_asset_value
method, which accepts an asset key and invokes the asset’s I/O manager’s load_input
function to load the asset as a Python object. This can be used in notebooks to do exploratory data analysis on assets.asset_selection
parameter on @sensor
and SensorDefinition
, you can now define a sensor that directly targets a selection of assets, instead of targeting a job.dagit
or dagster-daemon
locally, environment variables included in a .env
file in the form KEY=value
in the same folder as the command will be automatically included in the environment of any Dagster code that runs, allowing you to easily use environment variables during local development.dagster-dbt
now supports generating software-defined assets from your dbt Cloud jobs.dagster-airbyte
and dagster-fivetran
now support automatically generating assets from your ETL connections using load_assets_from_airbyte_instance
and load_assets_from_fivetran_instance
.dagster-duckdb
integration: build_duckdb_io_manager
allows you to build an I/O manager that stores and loads Pandas and PySpark DataFrames in DuckDB.dagster instance migrate
:
define_dagstermill_solid
, a legacy API, has been removed from dagstermill
. Use define_dagstermill_op
or define_dagstermill_asset
instead to create an op
or asset
from a Jupyter notebook, respectively.ComputeLogManager
API is marked as deprecated in favor of an updated interface: CapturedLogManager
. It will be removed in 1.2.0
. This should only affect dagster instances that have implemented a custom compute log manager.dagster-graphql
and dagit
now use version 3 of graphene
UPathIOManager
base class is now a top-level Dagster export. This enables you to write a custom I/O manager that plugs stores data in any filesystem supported by universal-pathlib
and uses different serialization format than pickle
(Thanks Daniel Gafni!).fs_io_manager
now inherits from the UPathIOManager
, which means that its base_dir
can be a path on any filesystem supported by universal-pathlib
(Thanks Daniel Gafni!).build_asset_reconciliation_sensor
now works with support partitioned assets.build_asset_reconciliation_sensor
now launches runs to keep assets in line with their defined FreshnessPolicies.FreshnessPolicy
object is now exported from the top level dagster package.FreshnessPolicy
defined, their current freshness status will be rendered in the asset graph and asset details pages.upload_interval
which specifies in seconds, the interval in which partial logs will be uploaded to the respective cloud storage. This can be used to display compute logs for long-running compute steps.dagit
or dagster-daemon
locally, environment variables included in a .env
file in the form KEY=value
in the same folder as the command will be automatically included in the environment of any Dagster code that runs, allowing you to easily test environment variables during local development.observable_source_asset
decorator creates a SourceAsset
with an associated observation_fn
that should return a LogicalVersion
, a new class that wraps a string expressing a version of an asset’s data value.execute_k8s_job
function that can be called within any op to run an image within a Kubernetes job. The implementation is similar to the build-in k8s_job_op
, but allows additional customization - for example, you can incorporate the output of a previous op into the launched Kubernetes job by passing it into execute_k8s_job
. See the dagster-k8s API docs for more information.define_dagstermill_asset
now supports RetryPolicy
. Thanks @nickvazz!
load_assets_from_airbyte_instance
, users can now optionally customize asset names using connector_to_asset_key_fn
.load_assets_from_fivetran_instance
, users can now alter the IO manager using io_manager_key
or connector_to_io_manager_key_fn
, and customize asset names using connector_to_asset_key_fn
..env
filesPublished by yuhan almost 2 years ago
asset_selection
parameter on @sensor
and SensorDefinition
, you can now define a sensor that directly targets a selection of assets, instead of targeting a job.materialize
and materialize_to_memory
now accept a raise_on_error
argument, which allows you to determine whether to raise an Error if the run hits an error or just return as failed.MultiPartitionsDefinition
object. An optional schema migration enables support for this feature (run via dagster instance migrate
). Users who are not using this feature do not need to run the migration.-db-pool-recycle
cli flag (and dbPoolRecycle helm option) have been added to control how long the pooled connection dagit uses persists before recycle. The default of 1 hour is now respected by postgres (mysql previously already had a hard coded 1hr setting). Thanks @adam-bloom!load_assets_from_airbyte_instance
and load_assets_from_airbyte_project
.dbt_cloud_resource
resource configuration account_id
can now be sourced from the environment. Thanks @sowusu-ba!load_assets_from_fivetran_instance
helper which automatically pulls assets from a Fivetran instance.securityContext
configuration of the Dagit pod in the Helm chart didn’t apply to one of its containers. Thanks @jblawatt!asset_selection
parameter of RunRequest
to not be respected when used inside a schedule.task_definition
field in the EcsRunLauncher
to an environment variable stopped working.load_assets_from_dbt_manifest
. This fixed then error when load_assets_from_dbt_manifest
failed to load from dbt manifest with exposures. Thanks @sowusu-ba!build_asset_reconciliation_sensor
has changed to be more focused on reconciliation. It now materializes assets that have never been materialized before and avoids materializing assets that are “Upstream changed”. The build_asset_reconciliation_sensor
API no longer accepts wait_for_in_progress_runs
and wait_for_all_upstream
arguments.https://github.com/dagster-io/dagster/compare/1.0.16...1.0.17
[jog] host_representation_tests, instance_tests, selector_tests (#10256)
by @dpeng817[jog] resource_tests (#10257)
by @dpeng817[dagstermill] notebook backed assets (#10277)
by @jamiedemarianotebook and assets example project (#10315)
by @jamiedemaria[dagster-airbyte][managed-elements] Explicit handling of reconciling secret values (#10195)
by @benpankow[duckdb] integration improvements (#10114)
by @jamiedemariaFix Helm package skipping (#10313)
by @jmsandersRemove special casing for graphql tests (#10327)
by @jmsandersAdd client ID to dagit (#10316)
by @dpeng817[dagster-airflow] add airflow 2 support to
make_dagster_job_from_airflow_dag + xcom mock option (#10337)
by @Ramshackle-Jamathon1.0.16 changelog (#10340)
by @alangenfelddisentangle asset reconciliation sensor from MultiAssetSensorDefinition (#10258)
by @sryzaMock tqdm in tests to avoid segfaults (#10343)
by @jmsandersAutomation: versioned docs for 1.0.16
by @elementl-devtoolsupdate 1.0.16 changelog (#10346)
by @yuhanBetter fix for tooltips around disabled buttons + gitlab icon (#10338)
by @salazarmBack-compat fix for setting the task definition in the EcsRunLauncher to an env var (#10341)
by @gibsondanMock each callsite of tqdm (#10368)
by @gibsondanBuild integration steps before test project steps (#10367)
by @jmsandersPackages with test changes only shouldn't run deps (#10309)
by @jmsandersRemove test-project-core (#10310)
by @jmsanderschore: add project urls to dagster pypi (#10328)
by @rexledesmafix: add comma (#10373)
by @rexledesmaadd securityContext to dagit init-user-deployment initContainers (#10369)
by @gibsondanJB - Docs Typography Follow-ups (#10344)
by @braunjj[dagit] Add more logos to tags (#10355)
by @jamiedemaria[Multi-dimensional partitions 1/n] Add schema migration for asset event tags table (#10001)
by @clairelin135Revert "Mock each callsite of tqdm" (#10379)
by @jmsanders[Multi-dimensional partitions 2/n] Read/write from asset event tags table (#10056)
by @clairelin135Prevent segfaults (#10372)
by @jmsanders[apidocs] clarify usage of create_databricks_job_op (#10299)
by @sryzaMark celery-k8s-test-suite as flaky (#10204)
by @jmsanders[Multi-dimensional partitions 3/n] Output multi-dimensional materializations (#10082)
by @clairelin135fix duckdb configs in examples (#10378)
by @jamiedemariaRevert "Mark celery-k8s-test-suite as flaky" (#10384)
by @jmsandersUnpin grpc (#10386)
by @gibsondansome refactorings to asset reconciliation sensor (#10233)
by @sryzafix mysql heading level + typo (#10336)
by @domsj[dagster-fivetran] Enable loading of Fivetran connection assets from instance (#10290)
by @benpankowResolver -> AssetGraph (#10401)
by @sryzaEnable accessing base jobs off of repository definitions (#10405)
by @sryzaadd pyrightconfig.json to .gitignore (#10403)
by @sryza[dagster-dbt] add exposures in load_assets_from_dbt_manifest (#10395)
by @sowusu-baoperate on keys instead of strings in AssetSelection and AssetGraph (#10402)
by @sryzaUpdated "Loaded with Errors" link (#10421)
by @gibsondan[dagster-io/ui] Add stroke color to TextArea (#10422)
by @hellendagInstall project-fully-featured (#10419)
by @jmsandersMore process cleanup in grpc tests (#10423)
by @gibsondanAdd methods to AssetGraph for traversing asset partitions graph (#10413)
by @sryza[dagit] Update partner compute_kind tags to latest designs (#10394)
by @bengotow[dagit] Fix materialization log padding when no metadata is present (#10350)
by @bengotow[dagit] Respect backfill permission in asset backfill modal (#10387)
by @bengotow[freshness-policy] add latency freshness policy (#10393)
by @OwenKephartRevert "Unpin grpc (#10386)" (#10428)
by @gibsondansupport RunRequest.asset_selection in schedules (#10400)
by @sryzasupport raise_on_error in materialize and materialize_to_memory (#10397)
by @sryzadocs(cacheable-assets): mark unimplemented methods as abstract (#10424)
by @rexledesma[easy] fix incorrect header in docker guide (#10427)
by @gibsondan[dagit] Fix bug in asset live reloading after step failures, clean up AssetView (#10349)
by @bengotowallow sensors to target asset selections instead of jobs (#10417)
by @sryzaImplement abstract methods (#10434)
by @jmsanders[graphql] handle null int metadata val (#10436)
by @alangenfeld[jog] convert runtime_types_tests (#10317)
by @dpeng817[jog] convert core_test/selector_tests (#10318)
by @dpeng817[easy][dagster-airbyte] Treat SSH key as secret for managed stack (#10390)
by @benpankow[jog] Convert snap tests (#10319)
by @dpeng817Expose sqlalchemy pool recycle option for mysql/postgres (#10416)
by @adam-bloomcached_method decorator (#10398)
by @sryzaEnable action logging in dagit (#10342)
by @dpeng817Temporarily skip celery-k8s tests (#10438)
by @jmsandersRun builds on changes to .txt files (#10433)
by @jmsandersHook for loading secrets in grpc/run/step entry points (#10089)
by @gibsondanRead
account_idfrom an environment variable in
dbt_cloud_resource (#10324)
by @sowusu-baAdd container_name to EcsContainerContext (#10446)
by @gibsondanMake asset reconciliation sensor more reconcile-y (#10435)
by @sryza[dagster-airbyte] Generate src/dst Python classes for managed stacks (#10272)
by @benpankow[docs] - Guides for configuring environment variables (#10034)
by @erinkcochran87Revert "specify schema in pandas to_sql" (#10450)
by @jamiedemariafeat: bind command-K to search (#10449)
by @rexledesma[dagster-airbyte] Add ability to specify output IO managers (#10217)
by @benpankow[dagster-airbyte] When loading managed connections, ignore those which are not managed (#10388)
by @benpankowenable launching a run that targets a range of asset partitions (#10441)
by @sryzachore: remove support for github dark mode (#10329)
by @rexledesma[dagit] Clarify notebook buttons (#10451)
by @jamiedemaria[dagstermill] allow custom io manager key specification (#10448)
by @jamiedemaria[dagster-airbyte][docs] Reorganize Airbyte guide (#10215)
by @benpankowFix check_step_health with delayed op retries (#10458)
by @johannkmchangelog 1.0.17 (#10464)
by @yuhan1.0.17
by @elementl-devtoolsPublished by alangenfeld almost 2 years ago
@asset
and @multi_asset
now accept a retry_policy
argument. (Thanks Adam Bloom!)fs_io_manager
will now return a dictionary that maps partition keys to the stored values for those partitions. (Thanks andrewgryan!).JobDefinition.execute_in_process
now accepts a run_config
argument even when the job is partitioned. If supplied, the run config will be used instead of any config provided by the job’s PartitionedConfig
.run_request_for_partition
method on jobs now accepts a run_config
argument. If supplied, the run config will be used instead of any config provided by the job’s PartitionedConfig
.NotebookMetadataValue
can be used to report the location of executed jupyter notebooks, and Dagit will be able to render the notebook.dagster.yaml
file; check out the docs.in_process
executor, where all steps are executed in the same process, the captured compute logs for all steps in a run will be captured in the same file.make_dagster_job_from_airflow_dag
now supports airflow 2, there is also a new mock_xcom parameter that will mock all calls to made by operators to xcom.PartitionedConfig
), previously run_request_for_partition
would produce a run with no config. Now, the run has the hardcoded dictionary as its config.AssetsDefinition
, through group-based asset dependency resolution, which would later error because of a circular dependency. This has been fixed.https://github.com/dagster-io/dagster/compare/1.0.15...1.0.16
rm all_types from config_context (#10203)
by @alangenfeldAlso skip docs next (#10178)
by @jmsandersMake handleLaunchResult agnostic to the query that returned the data (#10179)
by @salazarmMake PythonPackages.get() more flexible to _/- (#10184)
by @jmsanders[dagit] Invert stored state for expand/collapse in Overview pages (#10212)
by @hellendag[config] memoize ConfigType snap creation (#10210)
by @alangenfeld[config] avoid double init on field cache objects (#10214)
by @alangenfeldOverride default io manager in more places (#10202)
by @johannkmFix black (#10216)
by @johannkmtype annotations on backfill-related code paths (#9402)
by @sryzaFix lint (#10218)
by @johannkmAdd docs for customizing the serverless base image (#9571)
by @petehuntSkip test-project image builds (#10099)
by @jmsanders[dagit] Updated asset details event view (#10143)
by @bengotow[dagit] Fix AssetView.test key warnings (#10226)
by @hellendagAlso trigger builds when tox.ini changes (#10209)
by @jmsanders1.0.15 changelog (#10227)
by @jamiedemariaAutomation: versioned docs for 1.0.15
by @elementl-devtoolsadd materialization property on dagster event (#10230)
by @prhaParse HEAD commit in addition to BUILDKITE_MESSAGE (#10208)
by @jmsandersAllow default io_manager load_input to support partitions of differing frequencies (#10172)
by @andrewgryanInstall the correct dagster-buildkite CLI (#10211)
by @jmsandersType annotations in dagster-graphql (#10005)
by @smackeseyfix black, mypy (#10235)
by @prha[freshness-policy] [1/n] FreshnessPolicy object (#10024)
by @OwenKephartBreak up sensor_definition.py (#10181)
by @sryzaHighlight config entry when being hovered in yaml editor (#10239)
by @salazarmResolve multi-asset deps when they have the same group (#10222)
by @peayupdate apidoc for postgres (#10241)
by @prhafix schedules threading config (#10247)
by @alangenfeld[dagit] Ship Overview/Workspace (#10245)
by @hellendagFix stray references to define_assets_job (#10199)
by @bengotowcaptured log manager (#9429)
by @prhaAllow for longer messages in Teams HeroCard (#10234)
by @jayhale[dagit] Clean up code post-Overview changes (#10249)
by @hellendag[dagit] Move the refresh indicator on asset details pages to avoid flicker (#10253)
by @bengotow[dagit] Fix issues with embedding fonts in downloaded DAG SVGs (#10252)
by @bengotow[dagit] Fix daylight savings issue in humanCronString.test (#10263)
by @hellendag[dagit] Make 50-level colors opaque (#10186)
by @hellendag[dagit] Add query countdown/refresh to Timeline (#10250)
by @hellendagSupport overriding run config for partitions with execute_in_process (#10246)
by @sryzaAdd calendar icon (#10267)
by @salazarmUPathIOManager - filesystem-agnostic IOManager base (#10193)
by @danielgafnirename logKey to fileKey / logFileKey for disambiguation with new API (#9956)
by @prhafix some cases in group-based asset dep resolution (#10266)
by @sryzadagster-io/ui release notes (#10269)
by @salazarmfix UnresolvedAssetJobDefinition.run_request_for_partition when confi… (#10238)
by @sryzaImprove doc for
InputContext.{dagster_type,metadata} (#10242)
by @peayFetch asset materialization planned event from index shard in Sqlite (#10248)
by @clairelin135fix #9193: add retry policy to @asset and @multi_asset (#10150)
by @adam-bloomadds dagster-aws install to dockerfile in docker deployment guide (#10225)
by @jamiedemariakeep storage name for serialized event backcompat (#10280)
by @prhagraphql for captured log subscription (#9957)
by @prhaAdd NotebookMetadataValue (#10278)
by @jamiedemariaAdd volumes and volumeMounts to dagit and daemon in OSS helm chart (#10285)
by @gibsondansupport new captured log API for process-based execution (#9958)
by @prhaadd new capture APIs in frontend queries (#9959)
by @prhaspecify schema in pandas to_sql (#10289)
by @jamiedemaria[dagit] Fix AssetView flakiness (#10293)
by @hellendag[dagit] Middle-truncate asset key path in virtualized list (#10275)
by @hellendagSnowflake IO Manager API docs (#10175)
by @jamiedemariaCompress execute_step args (#10244)
by @johannkmfix black formatting (#10298)
by @prhafix backfill table status (#10295)
by @prharun_config argument on run_request_for_partition (#10279)
by @sryza[dagit] NotebookMetadataValue support (#10287)
by @jamiedemaria[dagit] Fix very tall configs in run config dialog (#10301)
by @hellendagObservable source asset decorator (#9899)
by @smackeseyconvert NoOpComputeLogManager to support captured log API (#9960)
by @prhaconvert S3 compute log manager to support new captured log API (#9961)
by @prhaTest createdBefore run filter (#10270)
by @benpankowfix build; isort; black (#10303)
by @prhaFix example scaffold (#10003)
by @smackeseyAdd version to asset decorator (#10167)
by @smackeseyFix logic to run lints on builds (#10304)
by @jmsandersDon't repeat skip logs (#10307)
by @jmsanders[jog] Add a utility method to execute op inside of graph (#10255)
by @dpeng817[jog] execution_tests,hook_tests (#10254)
by @dpeng817Revert "convert S3 compute log manager to support new captured log API (#9961) (#10311)
by @prha[docs] Automatically toggle tab components to display URL hash/anchor (#10231)
by @benpankow[dagstermill] notebook backed assets (#10277)
by @jamiedemarianotebook and assets example project (#10315)
by @jamiedemaria[dagster-airbyte][managed-elements] Explicit handling of reconciling secret values (#10195)
by @benpankow[dagster-airflow] add airflow 2 support to
make_dagster_job_from_airflow_dag + xcom mock option (#10337)
by @Ramshackle-Jamathon1.0.16 changelog (#10340)
by @alangenfeld1.0.16
by @elementl-devtoolsPublished by rexledesma almost 2 years ago
key_prefix
with methods like load_assets_from_modules
.dagster instance migrate
.-
were not being properly sanitized in some situations. Thanks @peay!load_assets_from_airbyte_project
. Thanks @adam-bloom!ServiceUnavailable
error. Thanks @cavila-evoliq!display_raw_sql
flag to the dbt asset loading functions. If set to False, this will remove the raw sql blobs from the asset descriptions. For large dbt projects, this can significantly reduce the size of the generated workspace snapshots.https://github.com/dagster-io/dagster/compare/1.0.14...1.0.15
[dagit] Add actions to run timeline empty state (#10092)
by @hellendagfix future ticks cursor when no ticks are available (#10095)
by @prhaRevert "Make the default_job_io_manager overridable via env (#9950)"
by @johannkmAdd helm vals for scheduler threading (#10094)
by @dpeng817[dagit] Shift-clicking Materialize should show asset launchpad even if assets are partitioned (#10047)
by @bengotowRemove references to services in docker-compose down command test fixture (#10042)
by @dpeng8171.0.14 changelog (#10106)
by @dpeng8171.0.14 changelog addendum (#10107)
by @gibsondan[dagit] Show all future ticks in the timeline window (#10097)
by @hellendag[dagster-postgres] Use SQLAlchemy engine in pynotify instead of psycopg2 directly (#10090)
by @peayJb/Docs UI Cleanup (#10050)
by @braunjj[mypy] fixes for uvicorn types (#10108)
by @alangenfeldRevert "Revert "Make the default_job_io_manager overridable via env (#9950)""
by @johannkm[dagit] Use gray tags in repo headers (#10078)
by @hellendagRun make mdx-format (#10113)
by @jmsandersUpgrade kind (#9992)
by @jmsandersrebuild event log indices to include id (#10105)
by @prha[dagit] Fix repo bucket sorting in virtualized tables (#10101)
by @hellendagAutomation: versioned docs for 1.0.14
by @elementl-devtoolsdocs(slack-alert-policies): add slack bot name to setup instructions (#10116)
by @rexledesma[dagit] Some typographic and spacing tweaks on virtualized tables (#10118)
by @hellendagDon't gate on elementl emails (#10119)
by @jmsandersSkip docs changes (#10030)
by @jmsanders[docs] - improve partitioned asset examples on partitions concepts page (#10084)
by @sryzaWalk the Python dependency tree (#10096)
by @jmsanders[dagit] Add some right padding to clearable text input (#10129)
by @hellendag[dagster-airbyte] Docs for how to use basic auth in current release (#10131)
by @benpankowCache docstring check (#10127)
by @schrockn[dagster-airbyte] Docs fix using with_resources with cacheable asset (#10132)
by @benpankowRetry on Service Unavailable (#10110)
by @cavila-evoliqMake AssetKey.__eq__ more efficient (#10139)
by @schrockn[docs formatter] Drop mypy ignore lines in docs (#10138)
by @benpankow[refactor] take logic for base asset jobs out of AssetGroup (#10071)
by @sryza[dagit] Subscribe to in progress asset runs, refresh on relevant events (#10028)
by @bengotowdocs: add h1 header to Dagster readme (#10156)
by @rexledesmadocs: update @dagsterio -> @dagster (#10158)
by @rexledesmaUse instance.python_log_level when determining default system logger level (#10073)
by @gibsondanin hacker news demo, fix column filtering with partitions and snowfla… (#10141)
by @sryzaAllow overriding svg with color (#10163)
by @salazarm[dagster-airbyte] Add top-level support for basic auth username & password (#10130)
by @benpankowimprove api doc examples for cloud object store IO managers (#10153)
by @sryzaswitch order of assets and jobs in partitions doc (#10142)
by @sryzaFix AssetsDefinition.with_prefix_or_group to update partition mappings (#10164)
by @peayFix assets with dashes in their path and
io_manager_def (#10087)
by @peayRefactor db io manager to core (#10128)
by @jamiedemariadocs design: fix icons. make icon changes backcompat (#10174)
by @yuhanSupport multi-repo builds (#10135)
by @jmsanders[docs] Out -> AssetOut docs tweak (#10146)
by @benpankowConditionally skip dagit builds (#10165)
by @jmsanders[dagster-managed-elements] Dagster-managed-elements CLI, APIs (#10011)
by @benpankowEventLogRecord.partition_key and EventLogRecord.asset_key (#10180)
by @sryza[dagster-airbyte] Airbyte managed elements impl (#10013)
by @benpankow[dagster-airbyte] allow filtering connections by directory names (#10151)
by @adam-bloommake_airflow_dag airflow 2 compatibility (#10115)
by @Ramshackle-Jamathon[dagit] Bump TypeScript (#10133)
by @hellendag[@dagster-io/ui] Release v1.0.6 (#10188)
by @hellendag[dagster-dbt] allow missing run_results.json (#10187)
by @OwenKephartdocs(dagster-cloud-agent): update IAM role link (#10189)
by @rexledesma[dagit] Expand run config dialog (#10173)
by @hellendagReplace custom in-memory logic with in-memory sqlite connection (#10154)
by @prhaadd cpu and memory to DagsterEcsTaskDefinitionConfig (#10198)
by @gibsondanAllow silencing failures with default io manager override (#10067)
by @johannkm[dagster-airbyte] Managed elements typo (#10196)
by @benpankow[dagit] Consolidate expand/collapse state in Overview tables (#10194)
by @hellendagreduce locking in workspace reload (#10192)
by @alangenfeld[dagster-dbt] add flag to make asset snapshots smaller (#10213)
by @OwenKephartExtend default timeout (#10205)
by @jmsanders[dagster-airbyte] Fix recreating source/destination logic (#10197)
by @benpankowRevert 4938f9beb9 and ec3c5099c2
by @johannkmMake handleLaunchResult agnostic to the query that returned the data (#10179)
by @salazarmrm all_types from config_context (#10203)
by @alangenfeld[config] memoize ConfigType snap creation (#10210)
by @alangenfeld[config] avoid double init on field cache objects (#10214)
by @alangenfeldFix black (#10216)
by @johannkm[dagit] Invert stored state for expand/collapse in Overview pages (#10212)
by @hellendag[dagit] Updated asset details event view (#10143)
by @bengotow1.0.15 changelog (#10227)
by @jamiedemaria1.0.15
by @elementl-devtoolsPublished by benpankow almost 2 years ago
load_assets_from_airbyte_project
now caches the project data generated at repo load time so it does not have to be regenerated in subprocesses.load_assets_from_airbyte_instance
or load_assets_from_airbyte_project
.Shift
while clicking the repository name, and all repository groups will be collapsed or expanded accordingly.EcsRunLauncher
now allows you to pass in a dictionary in the task_definition
config field that specifies configuration for the task definition of the launched run, including role ARNs and a list of sidecar containers to include. Previously, the task definition could only be configured by passing in a task definition ARN or by basing the the task definition off of the task definition of the ECS task launching the run. See the docs for the full set of available config.SkipReason
within a multi-asset sensor (experimental) would raise an error. This has been fixed.define_asset_job
, you would run into a CheckError
when launching the job from Dagit. This has been fixed.https://github.com/dagster-io/dagster/compare/1.0.13...1.0.14
Fix materialization count by partition (#9979)
by @clairelin135[dagit] Turn on timeline run bucketing for everyone (#9993)
by @hellendag[dagit] "Remove all" tab button to clear Launchpad tabs (#9981)
by @hellendag[dagit] Add polling to Overview pages (#9996)
by @hellendagReturn each Python PackageSpec's distribution (#9889)
by @jmsanders[dagster-airbyte] Support union types while generating normalization tables (#9937)
by @benpankow[dagster-airbyte] Add option to specify custom API request params to Airbyte resource (#10000)
by @benpankowView Notebook button opens notebook as link if it's a url (#9894)
by @jamiedemariaIO manager concept doc improvements (#9987)
by @sryza[dagster-airbyte] Add optional connection name filter when generating assets (#9975)
by @benpankow[docs] Update deployment settings reference with SSO default role (#9984)
by @benpankow[docs] - dbt-focused intro tutorial (#9853)
by @jamiedemariaConfigure tokens -> Tokens (#9999)
by @salazarmadd a link to discuss.dagster.io on README (#10002)
by @yuhan1.0.13 changelog (#10010)
by @yuhan[fix] fix docs bk error (#10014)
by @benpankowOnly add buildkite steps for affected changes (#9897)
by @jmsanders[dagit] Add daemon alerts to Overview schedules/sensors pages (#9972)
by @hellendag[dagit] Make asset catalog toolbar sticky (#9974)
by @hellendag[dagit] Update workspace flag label (#9994)
by @hellendagfix code example in create_databricks_job_op (#10012)
by @sryzaFixup setup paths (#10019)
by @jmsandersSkip Python checks if no Python files change (#10016)
by @jmsandersSkip package steps instead of excluding them (#10018)
by @jmsandersDon't raise an exception when changing the EcsRunLauncher's container name to a new name (#10026)
by @gibsondan[event log tests] add origin to run (#10017)
by @alangenfeldmove the big honkin asset graph to latest asset APIs (#10025)
by @sryzaConditionally skip helm steps (#10031)
by @jmsanders[dagit] Fix run tab counts not updating with poll interval (#9929)
by @hellendagAutomation: versioned docs for 1.0.13
by @elementl-devtoolsAlso run helm on merges to main (#10038)
by @jmsanders[dagit] Split asset graphs onto a new “Plots” tab (#9735)
by @bengotow[dagit] Fetch isAssetJob in WorkspaceContext, rm Launchpad tab flicker (#9770)
by @bengotowWorkaround for upstream snowflake-sqlalchemy issue (#10049)
by @gibsondan[dagster-airbyte] Generate schema metadata at load time when loading from project or instance (#9939)
by @benpankow[cacheable-assets] Enable use of with_resources, with_prefix_or_group on cacheable assets (#9978)
by @benpankow[docs] Fix for asset preview image next to code block (#9881)
by @benpankowfix credential helper links and volume path (#10046)
by @gibsondanRefactor build python package skipping (#10037)
by @jmsandersreplace tag joins with subquery over tag intersection queries (#10036)
by @prhaAlso correctly skip dagster_buildkite (#10053)
by @jmsanderspass tags to asset reconciliation sensor (#10032)
by @jamiedemariainfer graph input types from inner input types (#9658)
by @sryzaAdd hook for launch pad root execution button (#9976)
by @salazarmMake the default_job_io_manager overridable via env (#9950)
by @johannkm[dagit] Jest cleanup for missing canvas context (#10035)
by @hellendag[easy] fix incorrect cli link (#10058)
by @gibsondanSkip integration tests (#10051)
by @jmsandersconsistent capitalization in LaunchAssetChoosePartitionsDialog (#10033)
by @sryzaFix package skipping for dagster-test (#10063)
by @jmsandersfix graph type inference with fan-in (#10064)
by @sryzaFix runs yielded error for multi asset sensor (#10059)
by @clairelin135[dagit] Add "Materialize" action item to asset action menu (#10065)
by @hellendag[dagit] Update Backfill table styles (#10070)
by @hellendagRaise if we can't infer distribution (#10066)
by @jmsandersfix Dagit execution of partitioned asset jobs with hardcoded config (#10057)
by @sryzaSkip grapqhl and mysql checks (#10054)
by @jmsanders[windows tests] fix start time == end time issue (#10021)
by @alangenfeld[dagster-io/ui] Calculate available width for MiddleTruncate (#10052)
by @hellendag[dagit] Add Shift+click to expand/collapse all repos in run timeline and tables (#10076)
by @hellendag[dagit] Fix repo row expanded icon (#10077)
by @hellendagAllow passing in task definition config to the EcsRunLauncher instead of just a task definition ARN (#10044)
by @gibsondanbump test_execute_schedule_on_celery_k8s to 3m (#10088)
by @alangenfeldExpose permissions loading state (#10093)
by @salazarmRevert "Make the default_job_io_manager overridable via env (#9950)"
by @johannkmAdd helm vals for scheduler threading (#10094)
by @dpeng817[dagit] Shift-clicking Materialize should show asset launchpad even if assets are partitioned (#10047)
by @bengotow1.0.14 changelog (#10106)
by @dpeng8171.0.14
by @elementl-devtoolsPublished by brad-alexander about 2 years ago
AssetMaterialization
now has a metadata
property, which allows accessing the materialization’s metadata as a dictionary.DagsterInstance
now has a get_latest_materialization_event
method, which allows fetching the most recent materialization event for a particular asset key.RepositoryDefinition.load_asset_value
and AssetValueLoader.load_asset_value
now work with IO managers whose load_input
implementation accesses the op_def
and name
attributes on the InputContext
.RepositoryDefinition.load_asset_value
and AssetValueLoader.load_asset_value
now respect the DAGSTER_HOME
environment variable.InMemoryIOManager
, the IOManager
that backs mem_io_manager
, has been added to the public API.multi_asset_sensor
(experimental) now supports marking individual partitioned materializations as “consumed”. Unconsumed materializations will appear in future calls to partitioned context methods.build_multi_asset_sensor_context
testing method (experimental) now contains a flag to set the cursor to the newest events in the Dagster instance.TableSchema
now has a static constructor that enables building it from a dictionary of column names to column types.dagster run migrate-repository
which lets you migrate the run history for a given job from one repository to another. This is useful to preserve run history for a job when you have renamed a repository, for example.DagsterCloudOperator
and DagsterOperator
now support Airflow 2. Previously, installing the library on Airflow 2 would break due to an import error.execute_in_process
, no error would be raised. Now, a DagsterMaxRetriesExceededError
will be launched off.load_assets_from_...(..., use_build=True)
, AssetObservation events would be emitted for each test. These events would have metadata fields which shared names with the fields added to the AssetMaterialization events, causing confusing historical graphs for fields such as Compilation Time. This has been fixed.load_assets_from_...
was generated in a way which was non-deterministic for dbt projects which pulled in external packages, leading to errors when executing across multiple processes. This has been fixed.dagster_airflow.hooks
. Thanks @bollwyvl!dagster-gcp
now supports google-api-python-client 2.x. Thanks @amarrella!build_asset_reconciliation_sensor
.multi_asset_sensor
and kicking off subsequent partitioned runs.https://github.com/dagster-io/dagster/compare/1.0.12...1.0.13
[dagit] Run timeline: reduce scheduled tick width, reduce chunk min width (#9913)
by @hellendag[dagster-airflow] Add DagsterOperator and associated airflow abstractions (#9780)
by @Ramshackle-Jamathonpartition_key and upstream_output.asset_key in load_asset_value (#9914)
by @sryzaPin dask-kubernetes until we switch KubeCluster (#9918)
by @jmsandersFix executor test (#9919)
by @gibsondanAutomation: versioned docs for 1.0.12
by @elementl-devtoolschange log for 1.0.12 (#9923)
by @sryzarespect DAGSTER_HOME in AssetValueLoader (#9922)
by @sryza[dagit] Create Overview root (#9907)
by @hellendagfix 1.0.12 changelog (#9925)
by @sryza[dagit] Bucketed virtualized tables for Jobs, Schedules, Sensors (#9909)
by @hellendag[Feature] Add possibility to change request timeout value when calling airbyte (#9906)
by @FransDelasset selection diff apidoc (#9917)
by @sryzaadd InMemoryIOManager to public API (#9882)
by @sryzaadd SDAs to fs_io_manager docstring examples (#9872)
by @sryzaExplicitly error when retries are exceeded if raise_on_error is set (#9934)
by @dpeng817add cli command to migrate job runs from one repo to another (#9376)
by @prhaadd empty
init.pyto make
dagster_airflow.hooks,
.links importable (#9932)
by @bollwyvlAdd metadata hooks for making grpc client calls (#9825)
by @gibsondanTableSchema.from_name_type_dict (#9926)
by @sryzalinting fixups for dagster-airflow (#9948)
by @Ramshackle-Jamathonadd name and op_def to asset value load context (#9942)
by @sryza[dagster-io/ui] Middle truncation (#9933)
by @hellendagUnpin google dependencies (#9319)
by @amarrella[dagster-dbt] Fix dbt asset op name (#9963)
by @OwenKephartAdd asset_selection arg to execute_job (#9876)
by @dpeng817Convert definitions tests to use graph/job/op APIs (#9736)
by @dpeng817make it easier to fetch asset materialization metadata (#9951)
by @sryzaData pipeline smoke test example (#9945)
by @sryzaremove dagster-dbt deps on pandas and dagster-pandas (#9953)
by @sryzaAdd skipped events to multi asset sensor context (#9903)
by @clairelin135[dagstermill] update DagsterTranslator to support newer versions of papermill (#9901)
by @tizz98[dagster-io/ui] Test and comments for middle truncation search (#9955)
by @hellendag[dagster-dbt] fix dbt test metadata (#9965)
by @OwenKephartShow noteable logo on noteable backed assets/ops (#9916)
by @jamiedemariaFlag to set cursor to latest materializations on build_multi_asset_sensor_context (#9814)
by @clairelin135[docs] asset reconciliation sensor concept page (#9912)
by @jamiedemariasupport asset keys and asset selection for multi asset sensors (#9954)
by @jamiedemaria[docs] - Partitioned multi asset sensor examples (#9722)
by @clairelin135Update multi asset sensor docstring (#9971)
by @jamiedemariaClarify
with_resources error and update docs (#9784)
by @clairelin135Add threading to scheduler daemon (#9885)
by @dpeng817Ignore warnings sent from the dagster module (#9577)
by @dpeng817[dagster-airflow] remove default param for parent init (#9966)
by @Ramshackle-JamathonFix incorrect secrets_tags docs (#9980)
by @gibsondanDocument cloud Docker credential helpers (#9982)
by @jmsanders[dagster-dbt] boldly ignore type hints (#9989)
by @OwenKephartIncrease timeout in asset sensor tests (#9990)
by @jamiedemaria[dagster-dbt] make dbt cli resource methods public (#9973)
by @OwenKephartduckdb integration library (#9869)
by @jamiedemariaFix materialization count by partition (#9979)
by @clairelin135[dagit] Turn on timeline run bucketing for everyone (#9993)
by @hellendag[dagster-airbyte] Support union types while generating normalization tables (#9937)
by @benpankow[dagster-airbyte] Add option to specify custom API request params to Airbyte resource (#10000)
by @benpankow[dagster-airbyte] Add optional connection name filter when generating assets (#9975)
by @benpankow[docs] Update deployment settings reference with SSO default role (#9984)
by @benpankow[docs] - dbt-focused intro tutorial (#9853)
by @jamiedemaria1.0.13 changelog (#10010)
by @yuhanDon't raise an exception when changing the EcsRunLauncher's container name to a new name (#10026)
by @gibsondan1.0.13
by @elementl-devtoolsPublished by sryza about 2 years ago
multi_asset_sensor
(experimental) now accepts an AssetSelection
of assets to monitor. There are also minor API updates for the multi-asset sensor context.AssetValueLoader
, the type returned by RepositoryDefinition.get_asset_value_loader
is now part of Dagster’s public API.RepositoryDefinition.load_asset_value
and AssetValueLoader.load_asset_value
now support a partition_key
argument.RepositoryDefinition.load_asset_value
and AssetValueLoader.load_asset_value
now work with I/O managers that invoke context.upstream_output.asset_key
.dagster.yaml
as follows:code_servers:
local_startup_timeout: 120
load_assets_from_airbyte_instance
function which automatically generates asset definitions from an Airbyte instance. For more details, see the new Airbyte integration guide.DagsterCloudOperator
and DagsterOperator
, which are airflow operators that enable orchestrating dagster jobs, running on either cloud or OSS dagit instances, from Apache Airflow.port
configuration in the airbyte_resource
was marked as not required, but if it was not supplied, an error would occur. It is now marked as required.load_assets_from_dbt_project
or load_assets_from_manifest_json
. This has been fixed.sqlalchemy.exc.TimeoutError
now retryredshift_resource
no longer accepts a schema
configuration parameter. Previously, this parameter would error whenever used, because Redshift connections do not support this parameter.LazyPysparkResource
that only initializes a spark session once it’s accessed (thank you @zyd14!)build_asset_reconciliation_sensor
function accepts a set of software-defined assets and returns a sensor that automatically materializes those assets after their parents are materialized.Published by clairelin135 about 2 years ago
RepositoryDefinition
now exposes a load_asset_value
method, which accepts an asset key and invokes the asset’s I/O manager’s load_input
function to load the asset as a Python object. This can be used in notebooks to do exploratory data analysis on assets.PartitionKeyRange
now exist on the op execution context and input/output context.@multi_asset
decorator now respects its config_schema
parameter.define_asset_job
is now prefilled in the modal that pops up when you click the Materialize button on an asset job page, so you can quickly adjust the defaults.DagsterDbtCliError
s produced from the dagster-dbt library would contain large serialized objects representing the raw unparsed logs from the relevant cli command. Now, these messages will contain only the parsed version of these messages.deploy_ecs
example didn’t work when built and deployed on an M1 Mac.resync_parameters
configuration on the fivetran_resync_op
is now optional, enabling triggering historical re*syncs for connectors. Thanks @dwallace0723!https://github.com/dagster-io/dagster/compare/1.0.10...1.0.11
[dagster-dbt] fix star issue for asset input names (#9763)
by @OwenKephartapi docs for snowflake resource (#9717)
by @jamiedemaria[dagit] Hide hidden asset group job from Instance Overview (#9742)
by @hellendag[dagit] Workspace Overview page (#9744)
by @hellendag[dagit] Virtualized table for assets (#9759)
by @hellendag1.0.10 Changelog (#9771)
by @OwenKephartadd api doc for run_request_for_partition (#9645)
by @sryza[fixit] Add library versions to docs (#9768)
by @smackeseyAutomation: versioned docs for 1.0.10
by @elementl-devtoolsRemove unused workspace args from schedule wipe command (#9753)
by @gibsondan[dagster-fivetran] Optional Fivetran Historical Resync Parameters (#9774)
by @dwallace0723Helm template tests support raw dict (#9773)
by @johannkm[dagit] Prefill default job config when materializing assets (#9769)
by @bengotow[dagit] Multi-colored run batch backgrounds (#9775)
by @hellendag[dagit] Virtualized graphs table, reuse ops view (#9778)
by @hellendag[dagster-dbt] remove logs and raw output from dbt errors (#9779)
by @OwenKephartstarlette TestClient deps (#9791)
by @alangenfeldExtend ECS stub to include new cpu/mem (#9786)
by @jmsandersGet asset partition keys from IOManager and Op Contexts (#9776)
by @clairelin135[dagit] Fix lint (#9798)
by @hellendag[dagit] Top tooltip for backfill segments (#9783)
by @hellendag[dagit] Add reload button to WorkspaceHeader (#9789)
by @hellendag[dagit] Add top-level polling to virtualized table pages (#9797)
by @hellendagfix 4xx in docs.dagster.io CLOUD-1843 (#9802)
by @yuhan[scheduler] dont clean auto running states for error locations (#9805)
by @alangenfeld[perf] resolve lru_cache usage issues (#9782)
by @alangenfeldA few tweaks to concept docs (#9793)
by @sryzaSpecify platform to build in deploy_ecs example (#9815)
by @gibsondanAdd way to build and deploy the deploy_ecs project locally (#9760)
by @gibsondanrm dead code in GrpcServerRepositoryLocation (#9811)
by @alangenfeldRevert "remove dagster cli api subcommand from docs (#9165)" (#9740)
by @gibsondanload asset values outside of a run (#9792)
by @sryza[dagit] Make asset group row clickable (#9810)
by @hellendag[dagit] Update homepage and top nav for new workspace flag (#9819)
by @hellendag[dagit] Align virtualized table cells to top (#9809)
by @hellendag[dagit] Hover/active state for top nav items (#9816)
by @hellendaghandle None outputs in snowflake IO manager (#9818)
by @sryza[docs] - simplify multi-asset sensor examples and fix formatting (#9754)
by @sryzamark dbt outputs as Nothing (#9822)
by @sryzamake methods of PartitionMapping public (#9747)
by @sryzaMake alembic pin much less restrictive (#9830)
by @gibsondan[dagit] Export the new Workspace grid for Cloud use (#9824)
by @hellendagfix config schema on multi-asset (#9828)
by @sryza1.0.11
by @elementl-devtoolsPublished by alangenfeld about 2 years ago
monitor_all_repositories=True
in the run status sensor decorator.run_key
argument on RunRequest
and run_request_for_partition
is now optional.multi_asset_sensor
where the cursor would fail to update, and materializations would be returned out of order for latest_materialization_records_by_partition
.%Y-%m-%d
.foo == None
to foo is None
(thanks @chrisRedwine!).__ASSET_JOB
name, rather than a comprehensible set of asset keys. This has been fixed.k8s_job_op
would sometimes fail if the Kubernetes job that it creates takes a long time to create a pod.k8s_job_executor
now uses environment variables in place of CLI arguments to avoid limits on argument size with large dynamic jobs.https://github.com/dagster-io/dagster/compare/1.0.9...1.0.10
[dagster-airbyte] Add config to avoid pulling/forwarding Airbyte logs to compute log (#9649)
by @benpankow[dagster-airbyte] Support representing normalization-created tables as Assets (#9655)
by @benpankowisolate backfill partition status from partition status (#9698)
by @prha[dagster-airbyte] Add ability to generate Airbyte assets from YAML files (#9664)
by @benpankow[grpc] deferred snapshot support (#9630)
by @alangenfelddocstring updates for build_op_context (#9690)
by @sryzatags argument on build_schedule_from_partitioned_job (#9699)
by @sryza[dagit] Fix Run Timeline support for multiple jobs with the same name (#9684)
by @bengotow[dagit] Defensively pretty-print JSON in run logs (#9682)
by @bengotowreplace run count with a run status indicator (#9700)
by @prhaAdd asset selection flag to run_request_for_partition (#9679)
by @clairelin135[dagster-dbt] fix missing select when using dbt_cli_resource (#9680)
by @OwenKephart[dagit] remove mode and graphName from workspace context (#9701)
by @alangenfeld[dagster-databricks] fix library install version (#9696)
by @OwenKephart[dagit] Fix run.assets returning duplicate asset keys, tags appearing twice (#9681)
by @bengotowEnable step retries on docker executor (#9675)
by @johannkm[docs] - Clean up Guides sidenav + category page (#9247)
by @erinkcochran87[docs] - Dagster Cloud full deployment docs (#9643)
by @erinkcochran87[grpc] include timeout length in timeout error (#9492)
by @alangenfeld1.0.9 Changelog (#9704)
by @smackeseyAutomation: versioned docs for 1.0.9
by @elementl-devtoolsrun status sensor that can monitor the whole instance (#9666)
by @jamiedemariaimprove dagster debug help text (#9712)
by @alangenfeld[dagit] Use workspace-level repo filtering in virtualized jobs table (#9691)
by @hellendag[dagster-graphql] dont batch if only 1 target (#9710)
by @alangenfeld[dagit] Virtualized schedules table (#9706)
by @hellendagFixes for multi_asset_sensor (#9708)
by @clairelin135Retry while waiting for the k8s_job_op job to have pods (#9709)
by @gibsondan[dagit] Virtualized sensor table (#9718)
by @hellendagstop populating output_description on ExternalAssetNode (#9721)
by @OwenKephart[docs] - Graph-backed asset subsetting docs (#9695)
by @clairelin135fix repeat file import (#9719)
by @alangenfeld[docs] - Fix dbt > dbt Cloud in docs sidenav (#9720)
by @erinkcochran87remove unused modes from factory floor query (#9724)
by @gibsondanmaking it possible to skip debug log within polling loop (#9716)
by @zyd14[dagit] Add unloadable alert to virtualized schedule/sensor pages (#9725)
by @hellendag[dagit] Avoid skipping subscription for compute log panel (#9734)
by @hellendag[dagit] Show ad hoc asset materializations in run timeline (#9726)
by @hellendagConvert singleton comparisons from equality to identity (#9587)
by @chrisRedwinemodernize dynamic_tests (#9732)
by @alangenfeld[dagit] Allow deleting a run from the Run page itself (#9729)
by @hellendag[dagit] Don't link to nowhere on Backfill history table (#9733)
by @hellendagk8s executor pass args via env var (#8564)
by @johannkm[docs] - Fix IS_BRANCH_DEPLOYMENT docs examples (#9730)
by @gibsondan[dagit] Allow collapsing the asset event row that is expanded by default (#9683)
by @bengotow[dagit] Repair some spacing on requested runs dialog (#9746)
by @hellendagmake run keys optional on run requests (#9748)
by @sryzafix time window partition mapping with offset and lossy format (#9751)
by @sryza[dagit] Avoid showing hidden asset jobs on the backfill page (#9737)
by @bengotow[dagster-dbt] fix star issue for asset input names (#9763)
by @OwenKephart1.0.10 Changelog (#9771)
by @OwenKephart[fixit] Add library versions to docs (#9768)
by @smackesey1.0.10
by @elementl-devtools