dvc

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🚀 New Features and Enhancements

• ui: use global singleton UI object for printing (#5653) @skshetry

🐛 Bug Fixes

• plots: quote and resolve paths properly (#5670) @skshetry

🔨 Maintenance

• build: upgrade pyinstaller to 4.2 (#5682) @efiop

Thanks again to @efiop and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• Using existing logging level if defined in addLoggingLevel (#5632) @asford
• fsspec: skip instance cache on all filesystems (#5668) @isidentical
• fsspec: support raw buckets on path splitting (#5658) @isidentical
• diff: move Repo.diff related test for renames to test_diff.py (#5654) @skshetry
• update: use unresolved path with --to-remote calls (#5631) @isidentical

🐛 Bug Fixes

• diff: clear calculated renamed files from inverted hashmap (#5652) @skshetry

Thanks again to @asford, @efiop, @isidentical and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• gs: process etag when it is available (#5634) @isidentical
• build: add pyinstaller hook for googleapiclient (#5619) @0x2b3bfa0
• setup: remove duplicate 'pytest-cov' dependency (#5617) @nik123
• checkpoints: completely remove checkpoint outs on exp run --reset (#5586) @pmrowla
• import-url: support importing empty directories on object storage services (#5536) @isidentical
• gs: disable consistency checker (#5613) @isidentical
• stage add: unsuppress -c/--checkpoints help text (#5599) @pmrowla
• Add a link to Barrak&Adams citation (#5592) @dmpetrov

🚀 New Features and Enhancements

• plots: support opening plots file directly in the browser (#5584) @skshetry

🏇 Optimizations

• repro/exp run: revert use git status when reminding to git-track files (#5596) @pmrowla

🐛 Bug Fixes

• exp run: fix issue where duplicate workspace runs would incorrectly conflict (#5611) @pmrowla
• exp run: do nothing if result matches parent (HEAD) (#5600) @pmrowla
• http: fix info() if there is no content-length header (#5602) @skshetry

Thanks again to @0x2b3bfa0, @dmpetrov, @efiop, @isidentical, @nik123, @pmrowla and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• scm: add force parameter to BaseGitBackend.checkout() (#5582) @duijf

Thanks again to @duijf and @efiop for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• gdrive: change the temporary directory location to .dvc/tmp (#5583) @isidentical
• pull: proper path join for partial pulls (#5579) @isidentical
• Add Barrak, Eghan & Adam citation (#5573) @dmpetrov
• fs: ssh: pass timeout to paramiko's connect() (#5569) @duijf
• pygit/dulwich: implement stash functionality (#5286) @pmrowla
• state: use pickle v4 (#5571) @efiop
• gdrive: use the normal file backend for saved credentials (#5554) @isidentical
• utils: makedirs: ignore chmod errors (#5570) @efiop
• fs: git: use pygit2 (#5559) @efiop

Thanks again to @dmpetrov, @duijf, @efiop, @isidentical and @pmrowla for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🐛 Bug Fixes

• setup: rename azure.indentity to azure-identity (#5548) @gbmarc1

Thanks again to @efiop and @gbmarc1 for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• dependency: repo: try to fetch before staging (#5549) @efiop

🐛 Bug Fixes

• repro/exp run: use git status when reminding to git-track files (#5544) @pmrowla
• exp diff: fix unresolved experiment names bug (#5543) @pmrowla

Thanks again to @efiop and @pmrowla for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• setup: move google-cloud-storage to test requirements (#5540) @efiop

Thanks again to @efiop for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

New features

Please check the release blog post https://dvc.org/blog/dvc-2-0-release

Changes

See https://github.com/iterative/dvc/pulls for a full list of changes.

💣 Breaking Changes

• Dropped support for external outputs in Google Cloud Storage and changed the default checksum from md5 to etag.
• Dropped support for login with p12 files on service authentication for Google Drive.
• Stages without dependencies will not always run as if changed. Instead, use --always-changed.
• Environment variables inside the cmd of a stage using ${VAR} syntax must be escaped as ${VAR} in 2.0 due to the use of ${} syntax for templating.

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• update: clear error message for updating repo imports (#5520) @isidentical
• live integration: get rid of dvc from live (#5466) @pared
• gs: Use rm() instead of delete() calls (#5535) @isidentical
• gs: use etag for the checksum parameter (#5534) @isidentical

Thanks again to @efiop, @isidentical and @pared for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• gs: Migrate Google Storage to FSSpec (#5526) @isidentical
• plots: ignore binary plot files (#5523) @pared

Thanks again to @efiop, @isidentical and @pared for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• azure: add account key to the config schema (#5522) @isidentical
• exp run: if target is provided, only checkout target and dependencies (#5519) @pmrowla
• azure: support service principal in error messages as login_method (#5514) @isidentical
• state: use diskcache (#5505) @efiop

Thanks again to @efiop, @isidentical and @pmrowla for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• utils: proper file interface for HashedStreamReader (#5508) @isidentical
• azure: transfer bigger chunks of data (#5504) @isidentical
• azure: migrate AzureFileSystem to fsspec (#5463) @isidentical
• update: forbid updating repo-imports with --to-remote (#5475) @isidentical
• add, import-url: always resolve the output in the local path for --to-remote (#5473) @isidentical
• live integration: turn on tests (#5487) @pared
• gdrive: support service auth with json (#5485) @isidentical

Thanks again to @efiop, @isidentical, @pared, @pmrowla and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

The full changelog is coming soon, stay tuned! 📡

Breaking change

• Changed dvc.lock schema.
  It is still backward compatible in terms of read and
  will be migrated to the new format whenver it's written (#5222).

Deprecations and Removals

• Removed --cwd from dvc repro (#5048).
• Deprecated dvc run command. dvc stage add is now recommended instead.

New Features

• dvc.yaml supports parametrization and looping constructs.
• Support list of commands in cmd field in dvc.yaml file (#4934).
• Added experiments feature and dvc exp command.
• Added support for straight-to-remote feature.
• Added support for live, and corresponding dvc live command.
• Added stage:add command as a replacement for dvc run.
• Added stage:list command to list all the stages in the repo.
• Added --glob for wildcard pattern as target option for add/push/pull (#4864, #5033, #5032)
• diff/params diff/metrics diff got --targets flag (#4938, #5122)
• config: Support listing config through --list and their source with --show-origin (#5080, #5188)
• Added '--show-md' flag for 'dvc metrics show' (#4770) (#5161)

Bug Fixes

• dvcignore: Fix incorrect ignored output computation (#4986)
• fix partial file checkouts in the same directory (#5163)
• diff accounts for deleted workspace data (#5205)
• fix --help output (#5193, #4999)
• config: read merged config by default (#5179)
• aliyun-oss: fix pull error for large file (#5053)
• Run graph checks on collect/find_outs_by_path (#5035)
• follow the symlink when deciding filesystem type on dvc version (#5067)
• Fix get-url failing with AttributeError (#5118)

Enhancements

• .dvc files can store executable permissions (#5061).
• dvc metrics displays as a table (#4706).
• metrics/params diff/show table are now ordered.
• commit: support granularity (#4297) (#5088).
• dvc import can be passed --jobs (#4977).
• diff can detect file renames (#5160).
• Added support for autocompletions of stage names in zsh/bash.
• Add ">" to script during dvc run/repro (#5041)
• Reducing clutter of log messages (#5017, #5236)
• Added dvc repo was initialized when dvc init-ed (#5190)
• make IterStream seekable and peekable (#5084)
• Improve OverlappingOutputsError exception message. (#5015)
• delete .gitignore file if empty (#5148)

Optimizations and Performance

• push: Gradual task allocation for pushing big datasets (#5285)
• Startup time has been improved.
• git: don't list all commits to check for an empty repo (#5075)

Internals and Libraries

• Replace deprecated pyarrow.hdfs.connect (#4973)
• psutil is now a requirement (#5232)
• Increment shtab's version (#5212)

• hdfs: fix typo (https://github.com/iterative/dvc/commit/f7551c33e66a506cfa57f4a6ae72d0186141baeb)
• build: require git >= 1.7.0 (#5467) (#5468)

• run: do not compute hash when no_exec is set (#5380) @Honzys
• ssh: show total-size when uploading/downloading on ssh remote (#5383) @isidentical

• params: support unicode characters (#5363) @isidentical

• dvc: ignore errors on protect/set_exec (#5335) @efiop

• Revert "utils/fs: checking files ownership in 'move' (#4348) (#4832)

• s3: Allow authentication using session token (#5240) @isidentical