dvc

What's Changed

• deps: bump dvc-data to 0.25.1 by @efiop in https://github.com/iterative/dvc/pull/8487

Full Changelog: https://github.com/iterative/dvc/compare/2.32.1...2.33.0

What's Changed

• yappi: add --yappi-separate-threads flag by @efiop in https://github.com/iterative/dvc/pull/8485

Full Changelog: https://github.com/iterative/dvc/compare/2.32.0...2.32.1

What's Changed

• Use celery status as the exp show status by @karajan1001 in https://github.com/iterative/dvc/pull/8369
• index: support multiple targets within output in IndexView by @efiop in https://github.com/iterative/dvc/pull/8471
• auto solve corrupted rwlock info by @karajan1001 in https://github.com/iterative/dvc/pull/8469

Full Changelog: https://github.com/iterative/dvc/compare/2.31.0...2.32.0

What's Changed

• exp show :Add --hide-queued and --hide-failed flag by @karajan1001 in https://github.com/iterative/dvc/pull/8318
• build(deps): Bump dvc-render from 0.0.11 to 0.0.12 by @dependabot in https://github.com/iterative/dvc/pull/8401
• Refactor dvc get-url by @rlamy in https://github.com/iterative/dvc/pull/8410
• deps: bump dvc-data to 0.17.1 by @pmrowla in https://github.com/iterative/dvc/pull/8416

Full Changelog: https://github.com/iterative/dvc/compare/2.29.0...2.30.0

What's Changed

• push: use update_pipeline=False for cloud versioning by @efiop in https://github.com/iterative/dvc/pull/8324

Full Changelog: https://github.com/iterative/dvc/compare/2.27.1...2.27.2

What's Changed

• expose dvcfs in dvc.api and add to fsspec's registry by @skshetry in https://github.com/iterative/dvc/pull/8287
• import-url: pass fs_config down from imp_url to get_cloud_fs by @dtrifiro in https://github.com/iterative/dvc/pull/8286
• deps: remove unused mock dep by @dtrifiro in https://github.com/iterative/dvc/pull/8290
• dvcfs: default open to binary mode by @skshetry in https://github.com/iterative/dvc/pull/8295
• worktree push/fetch: support dirs by @pmrowla in https://github.com/iterative/dvc/pull/8273
• schema: add strict schema validation for top-level plots by @skshetry in https://github.com/iterative/dvc/pull/8289

Full Changelog: https://github.com/iterative/dvc/compare/2.25.0...2.26.0

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• deps: bump dvc-data to 0.4.0 by @efiop in https://github.com/iterative/dvc/pull/8219 and https://github.com/iterative/dvc/pull/8213

🚀 New Features and Enhancements

• exp run: Support composing and dumping Hydra config. by @daavoo in https://github.com/iterative/dvc/pull/8093

🐛 Bug Fixes

• data status: fix path for committed changes in Windows by @skshetry in https://github.com/iterative/dvc/pull/8220

Full Changelog: https://github.com/iterative/dvc/compare/2.23.0...2.24.0

What's Changed

• build: skip install fpm and ruby on Windows by @skshetry in https://github.com/iterative/dvc/pull/8169
• import-url/update: add --no-download flag by @dtrifiro in https://github.com/iterative/dvc/pull/8024
• fs: optimize dvc list --recursive by @rlamy in https://github.com/iterative/dvc/pull/8150

New Contributors

• @rlamy made their first contribution in https://github.com/iterative/dvc/pull/8150

Full Changelog: https://github.com/iterative/dvc/compare/2.19.0...2.20.0

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• utils: proper file interface for HashedStreamReader (#5508) @isidentical
• azure: transfer bigger chunks of data (#5504) @isidentical
• azure: migrate AzureFileSystem to fsspec (#5463) @isidentical
• update: forbid updating repo-imports with --to-remote (#5475) @isidentical
• add, import-url: always resolve the output in the local path for --to-remote (#5473) @isidentical
• live integration: turn on tests (#5487) @pared
• gdrive: support service auth with json (#5485) @isidentical

Thanks again to @efiop, @isidentical, @pared, @pmrowla and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

The full changelog is coming soon, stay tuned! 📡

Breaking change

• Changed dvc.lock schema.
  It is still backward compatible in terms of read and
  will be migrated to the new format whenver it's written (#5222).

Deprecations and Removals

• Removed --cwd from dvc repro (#5048).
• Deprecated dvc run command. dvc stage add is now recommended instead.

New Features

• dvc.yaml supports parametrization and looping constructs.
• Support list of commands in cmd field in dvc.yaml file (#4934).
• Added experiments feature and dvc exp command.
• Added support for straight-to-remote feature.
• Added support for live, and corresponding dvc live command.
• Added stage:add command as a replacement for dvc run.
• Added stage:list command to list all the stages in the repo.
• Added --glob for wildcard pattern as target option for add/push/pull (#4864, #5033, #5032)
• diff/params diff/metrics diff got --targets flag (#4938, #5122)
• config: Support listing config through --list and their source with --show-origin (#5080, #5188)
• Added '--show-md' flag for 'dvc metrics show' (#4770) (#5161)

Bug Fixes

• dvcignore: Fix incorrect ignored output computation (#4986)
• fix partial file checkouts in the same directory (#5163)
• diff accounts for deleted workspace data (#5205)
• fix --help output (#5193, #4999)
• config: read merged config by default (#5179)
• aliyun-oss: fix pull error for large file (#5053)
• Run graph checks on collect/find_outs_by_path (#5035)
• follow the symlink when deciding filesystem type on dvc version (#5067)
• Fix get-url failing with AttributeError (#5118)

Enhancements

• .dvc files can store executable permissions (#5061).
• dvc metrics displays as a table (#4706).
• metrics/params diff/show table are now ordered.
• commit: support granularity (#4297) (#5088).
• dvc import can be passed --jobs (#4977).
• diff can detect file renames (#5160).
• Added support for autocompletions of stage names in zsh/bash.
• Add ">" to script during dvc run/repro (#5041)
• Reducing clutter of log messages (#5017, #5236)
• Added dvc repo was initialized when dvc init-ed (#5190)
• make IterStream seekable and peekable (#5084)
• Improve OverlappingOutputsError exception message. (#5015)
• delete .gitignore file if empty (#5148)

Optimizations and Performance

• push: Gradual task allocation for pushing big datasets (#5285)
• Startup time has been improved.
• git: don't list all commits to check for an empty repo (#5075)

Internals and Libraries

• Replace deprecated pyarrow.hdfs.connect (#4973)
• psutil is now a requirement (#5232)
• Increment shtab's version (#5212)

• hdfs: fix typo (https://github.com/iterative/dvc/commit/f7551c33e66a506cfa57f4a6ae72d0186141baeb)
• build: require git >= 1.7.0 (#5467) (#5468)

• run: do not compute hash when no_exec is set (#5380) @Honzys
• ssh: show total-size when uploading/downloading on ssh remote (#5383) @isidentical

• params: support unicode characters (#5363) @isidentical

• dvc: ignore errors on protect/set_exec (#5335) @efiop

• Revert "utils/fs: checking files ownership in 'move' (#4348) (#4832)

• s3: Allow authentication using session token (#5240) @isidentical

🐛 Bug Fixes

• add: handle long filenames (#5201) @efiop

Refer to https://dvc.org/doc/install for installation instructions.

🐛 Bug Fixes

• lock: fix AttributeError in dvc.lock.HardlinkLock (#5152) @pmrowla
• [experimental] fixes params diff error when parameterization is enabled (#5127)

Refer to https://dvc.org/doc/install for installation instructions.

🐛 Bug Fixes

• repro: simplify the logic for frozen stages (#5114) @efiop
• repro: accept multiple targets (#5111) @efiop

Changes

• Remove version restriction from networkx (#5110) @jprafael