dvc

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🐛 Bug Fixes

• diff: use relpaths (#7158) @efiop
• fs.hdfs: skip port if not available (#7189) @skshetry
• fs: repo: raise proper FileNotFoundError in metadata (#7156) @pared

🚀 New Features and Enhancements

• exp show: Add parallel coordinates plot. (#6933) @daavoo
• exp show: Alternate row background color (#7142) @daavoo

🔨 Maintenance

• deps: remove requests-kerberos from webdhfs; provide it via webdhfs_kerberos. (#7178) @skshetry
• build(deps): Bump scmrepo from 0.0.4 to 0.0.7 (#7150, #7187) @dependabot
• deps: explicitly specify pyarrow for hdfs (#7179) @skshetry
• build(deps-dev): Bump mypy from 0.920 to 0.921 (#7185) @dependabot
• build(deps-dev): Bump types-requests from 2.26.1 to 2.26.2 (#7170) @dependabot
• build(deps-dev): Bump mypy from 0.910 to 0.920 (#7162) @dependabot
• build(deps-dev): Bump pytest-timeout from 2.0.1 to 2.0.2 (#7144) @dependabot
• build(deps-dev): Bump pytest-xdist from 2.4.0 to 2.5.0 (#7132) @dependabot

Thanks again to @daavoo, @dependabot, @dependabot[bot], @efiop, @karajan1001, @pared, @pmrowla and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🔨 Maintenance

• setup: bump gcsfs (#7117) @pared
• install: fix archives (#7125) @pared
• build(deps-dev): Bump dvclive[image] from 0.4.5 to 0.4.6 (#7112) @dependabot
• build(deps): Bump scmrepo from 0.0.3 to 0.0.4 (#7113) @dependabot

Thanks again to @dependabot, @dependabot[bot], @efiop and @pared for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• webdav: fix unstrip_protocol

Refer to https://dvc.org/doc/install for installation instructions.

Changes

💣 Breaking Changes

• exp diff: Remove --old option. (#6888) @daavoo

🚀 New Features and Enhancements

• introduces exp init command @skshetry
• render: image: make html less ugly (#7075) @pared
• plots: collect: provide callable for data resolution (#6832) @pared
• live: Call webbrowser.open (#6913) @daavoo
• scm: hint to autostage (#6985) @dberenbaum
• exp show: Add --only-changed option (#6867) @daavoo

🐛 Bug Fixes

• exp: respect repo hardlink_lock config for the exp scm lock (#7030) @kwon-young
• dulwich: fallback to default SSH keys in asyncssh vendor (#6983) @pmrowla
• plots: special chars handling for vega plots paths (#6900) @pared
• params: Use ast.literal_eval for python params. (#6954) @daavoo
• push: fix http storage ignoring method setting (#6981) @ditod
• hardlink: fallback to other link methods if "Function not implemented" is raised (#6963) @skshetry
• argparse: dont override cmd (#6958) @pared

🔨 Maintenance

• build(deps): Bump scmrepo from 0.0.1 to 0.0.2 (#7055) @dependabot
• packages: build binary pkgs with Python 3.9 (#7016) @skshetry
• dvc.scm: refactor exceptions (#7017) @skshetry
• build(deps): Bump adlfs from 2021.9.1 to 2021.10.0 (#6734) @dependabot
• build: bump sshfs from 2021.8.1 to 2021.11.2 (#6999) @guludo
• build: bump pyinstaller from 4.6 to 4.7 (#6960) @dependabot
• dvc: get rid of path_info (#6946) @efiop

Thanks again to @daavoo, @dberenbaum, @dependabot, @dependabot[bot], @ditod, @efiop, @guludo, @isidentical, @karajan1001, @kwon-young, @pared, @pmrowla, @pre-commit-ci, @pre-commit-ci[bot], @skshetry and Dimitar Todorov for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🚀 New Features and Enhancements

• machine status: include two new field in the output (#6869) @karajan1001
• exp init: ask for command before paths (#6872) @skshetry
• LiveTask: write output path at start/end (#6815) @daavoo
• transfer: preserve workspace data (#6860) @efiop
• exp run: add experiment name check. (#6848) @karajan1001
• fs: transfer: try reflink (#6853) @efiop

🏇 Optimizations

• output: save: remove redundant changed() call (#6887) @efiop
• fs: transfer: try reflink (#6853) @efiop

🐛 Bug Fixes

• webhdfs: expose kerberos and https options (#6936) @gudmundur-heimisson
• plots: fix resolve template path bug (#6866) @pared
• plots: Use URI for --open. (#6845) @daavoo
• transfer: preserve workspace data (#6860) @efiop

🔨 Maintenance

• build(deps-dev): bump gcsfs from 2021.10.1 to 2021.11.0 (#6938) @dependabot
• deps: bump rich to >=10.13.0 (#6937) @skshetry
• build(deps): bump actions/checkout from 2.3.5 to 2.4.0 (#6921) @dependabot
• build: remove numpy restriction (#6906) @skshetry
• build(deps): bump pyinstaller from 4.5.1 to 4.6 (#6897) @dependabot
• dependencies: remove python 3.10 restriction for pyarrow (#6901) @skshetry
• build(deps): update pyparsing requirement from ==2.4.7 to >=2.4.7 (#6858) @dependabot

Thanks again to @daavoo, @dberenbaum, @dependabot, @dependabot[bot], @efiop, @gudmundur-heimisson, @karajan1001, @pared, @pmrowla, @pre-commit-ci, @pre-commit-ci[bot] and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• git: use asyncssh as dulwich SSH vendor (#6797) @pmrowla
• Dependabot/pip/pylint 2.11.1 (#6843) @karajan1001
• Revert "build(deps): Bump pylint from 2.9.6 to 2.11.1" (#6842) @karajan1001
• dvc machine: add a new command rename to it (#6633) @karajan1001
• Exp show md (#6799) @daavoo
• machine: Add a new command dvc machine status (#6649) @karajan1001
• exp/metrics/params diff: Replaced Old/New with a_rev/b_rev (#6814) @daavoo
• [pre-commit.ci] pre-commit autoupdate (#6829) @pre-commit-ci
• Added test_show_experiments_sort_by (#6798) @daavoo

🚀 New Features and Enhancements

• exp run: throw error if new parameters are added by --set-param (#6521) @mattlbeck
• plots: make generator out of collect (#6793) @pared
• plots: load templates from package (#6550) @pared

🐛 Bug Fixes

• dvcfs: handle new workspace subdirectories in walk (#6811) @eggqq007
• ui: write_json: don't forget json kwargs (#6810) @efiop
• dvc: add 'repos' list to pre-commit config if it does not exist (#6795) @mjlbach

🔨 Maintenance

• build(deps): Bump pylint from 2.9.6 to 2.11.1 (#6636) @dependabot
• move some configs to pyproject.toml (#6834) @skshetry
• setup: bump fsspec deps to 2021.10.1 (#6828) @efiop
• Revert "webdav: limit httpx to fix the CI" (#6827) @skshetry
• build(deps-dev): Bump filelock from 3.3.0 to 3.3.1 (#6825) @dependabot
• build(deps-dev): Bump types-requests from 2.25.9 to 2.25.11 (#6820) @dependabot
• build(deps): Bump actions/checkout from 2.3.4 to 2.3.5 (#6817) @dependabot
• build(deps-dev): Bump types-tabulate from 0.8.2 to 0.8.3 (#6790) @dependabot
• build(deps-dev): Bump types-toml from 0.10.0 to 0.10.1 (#6821) @dependabot
• build(deps-dev): Bump dvclive from 0.4.0 to 0.4.1 (#6786) @dependabot
• build(deps): Update flufl-lock requirement from <4,>=3.2 to >=5 (#6576) @dependabot
• pathspec 0.9.0 quickfix (#6689) @endremborza
• webdav: limit httpx to fix the CI (#6800) @isidentical
• dvc: drop 3.6 support (#6792) @efiop
• use setuptools_scm and build_requires (#6775) @skshetry

Thanks again to @daavoo, @dependabot, @dependabot[bot], @efiop, @eggqq007, @endremborza, @isidentical, @karajan1001, @mattlbeck, @mjlbach, @pared, @pmrowla, @pre-commit-ci, @pre-commit-ci[bot] and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Maintenance

• setup.cfg: use text/x-rst instead of text/rst

Refer to https://dvc.org/doc/install for installation instructions.

🚀 New Features and Enhancements

• Fix exp show --sort-by issue #6675 (#6764) @eggqq007
• webhdfs: migrate to fsspec (#6662) @isidentical
• hdfs: migrate to fsspec (#6604) @isidentical
• command: Should not consume 'print' to output default remote info (#6650) @eggqq007
• command: output prettify json in tty (#6743) @eggqq007
• Run on Python3.10 (#6745) @skshetry
• http: explicitly retry all http-related errors (#6728) @isidentical
• exp init: introduce --with={default,live} flag (#6716) @skshetry
• cli: add short alias for --show-{csv,json,md} (#6711) @skshetry
• command: Show dvc list output in pretty JSON format (#6647) @eggqq007
• exp init: show heading based on the presence of dvc.yaml file (#6701) @skshetry
• exp init: support interactive and explicit options (#6681) @skshetry
• better exception messages for schema errors (#6679) @skshetry
• ui: support pager universally (#6692) @skshetry
• add alias ls for dvc list (#6683) @ap-kulkarni
• remote: add support for Amazon S3 timeout configuration (#6634) @whysage
• merge yaml parsing and schema validation into one, aka strictyaml.load (#6661) @skshetry
• exceptional exceptions (#6654) @skshetry
• Allow user to specify different state and index tmp dir locations (#6419) @nueschs

🐛 Bug Fixes

• Fix exp show --sort-by issue #6675 (#6764) @eggqq007
• gs: use 3-char prefixes (#6759) @isidentical
• Fix dvclive running subdir (#6740) @daavoo
• build(setup): avoid leaking tests package into site-packages (#6585) @pwoolvett
• doctor: fixes unknown cache directory error on some windows mapped network shares (#6364) @anasitomtn
• http: allow reading proxies from the current environment (#6688) @isidentical
• ssh: fix path normalization on windows (#6517) @isidentical
• ssh: bundle bcrypt by default (#6678) @isidentical
• fix dvc[all] (#6657) @casperdcl
• progress: do not block on async filesystems (#6612) @isidentical

🔨 Maintenance

• hdfs: enable mocked hdfs tests by default (#6766) @isidentical
• advertise Python 3.10 support (#6772) @skshetry
• build(deps): Bump filelock from 3.2.0 to 3.3.0 (#6737) @dependabot
• build(deps-dev): Bump types-requests from 2.25.6 to 2.25.9 (#6770) @dependabot
• build(deps-dev): Bump types-paramiko from 0.1.9 to 2.7.1 (#6769) @dependabot
• build(deps-dev): Bump types-toml from 0.1.5 to 0.10.0 (#6768) @dependabot
• build(deps): Bump dvclive from 0.3.0 to 0.4.0 (#6753) @dependabot
• build(deps): Bump jaraco-windows from 5.6.0 to 5.7.0 (#6735) @dependabot
• build(deps): Bump pytest-cov from 2.12.1 to 3.0.0 (#6736) @dependabot
• Bump typing_extensions to >=3.7.4 (#6732) @gcoter
• setup: bump fsspec/s3fs/gcsfs (#6727) @isidentical
• build(deps): Bump google-cloud-storage from 1.42.2 to 1.42.3 (#6722) @dependabot
• build(deps): Bump filelock from 3.1.0 to 3.2.0 (#6710) @dependabot
• build(deps): Bump filelock from 3.0.12 to 3.1.0 (#6698) @dependabot
• build(deps): Bump pytest-xdist from 2.3.0 to 2.4.0 (#6696) @dependabot
• build(deps): Bump google-cloud-storage from 1.42.1 to 1.42.2 (#6635) @dependabot
• system: get rid of speedcopy (#6686) @efiop
• fix dvc[all] (#6657) @casperdcl

Thanks again to @anasitomtn, @ap-kulkarni, @casperdcl, @daavoo, @dependabot, @dependabot[bot], @efiop, @eggqq007, @gcoter, @isidentical, @nueschs, @pared, @pre-commit-ci, @pre-commit-ci[bot], @pwoolvett, @skshetry and @whysage for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🚀 New Features and Enhancements

• Force sorted order in plots rendering (#6601) @daavoo

🐛 Bug Fixes

• gdrive: strip bucket (#6618) @skshetry
• ssh: ask password before setting it (#6615) @isidentical
• dvc cli: fixed error in 'dvc config --list' in non repo dir (#6587) @ap-kulkarni
• output: use remote kwarg instead of odb (#6600) @efiop

🔨 Maintenance

• build(deps): Bump codecov/codecov-action from 2.0.3 to 2.1.0 (#6606) @dependabot
• build(deps): Bump mock-ssh-server from 0.9.0 to 0.9.1 (#6607) @dependabot

Thanks again to @ap-kulkarni, @daavoo, @dependabot, @dependabot[bot], @efiop, @isidentical, @karajan1001, @nik123, @pared and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🚀 New Features and Enhancements

• Added GitAuthError (#6493) @daavoo
• diff: don't forget to use dvcignore and state (#6595) @efiop
• project: formalize extra dependencies (#6560) @isidentical

🏇 Optimizations

• diff: don't forget to use dvcignore and state (#6595) @efiop

🐛 Bug Fixes

• ssh: support passphrases for private keys (#6566) @isidentical

🔨 Maintenance

• build(deps): Bump adlfs from 2021.8.2 to 2021.9.1 (#6597) @dependabot
• objects.transfer: minor refactoring, move lazier taskset inside custom executor (#6591) @skshetry
• build(deps): Bump grandalf from 0.6 to 0.7 (#6575) @dependabot
• build(deps): Bump hdfs from 2.5.8 to 2.6.0 (#6577) @dependabot
• build(deps): Bump google-cloud-storage from 1.42.0 to 1.42.1 (#6568) @dependabot
• project: formalize extra dependencies (#6560) @isidentical

Thanks again to @daavoo, @dependabot, @dependabot[bot], @efiop, @isidentical, @pared and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• Show csv format of experiments (#6468) @karajan1001

🐛 Bug Fixes

• http: pass prepared filesystem arguments (#6565) @isidentical

🔨 Maintenance

• remove tpi.base dependency (#6554) @casperdcl
• bump webdav4 to 0.9.1 (#6557) @skshetry

Thanks again to @casperdcl, @efiop, @isidentical, @karajan1001 and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🚀 New Features and Enhancements

• http: migrate to fsspec & aiohttp (#6525) @isidentical

Thanks again to @efiop, @isidentical, @pmrowla and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🚀 New Features and Enhancements

• Clean up remotes's exps (#6471) @karajan1001
• Add auto-refresh option to html (#6400) @daavoo
• machine: add dvc machine ssh (#6501) @pmrowla
• dvc machine: implement initial terraform support (#6462) @pmrowla
• dvcfile: support remote per output (#6486) @efiop
• plots: introduce support for images (#6431) @pared
• Update README.rst (#6461) @himanshugarg

🏇 Optimizations

• s3: inline bucket region caching (#6406) @isidentical

🐛 Bug Fixes

• ssh: support multiple keys (#6519) @isidentical
• add terraform to tests dependencies (#6502) @skshetry
• hdfs: don't read the whole file into memory (#6498) @efiop
• fix console appearing on daemon in Windows (#6450) @skshetry
• [6478] Fix 401 unauthorized for directories in basic auth protected http remote (#6479) @simonlsk

🔨 Maintenance

• extract transfer related logic from odb to fs.utils.transfer (#6539) @skshetry
• try to fix flaky dvclive test (#6541) @skshetry
• setup: bump fsspec/s3fs/gcsfs/adlfs/sshfs to 08.01 (#6524) @isidentical
• docs: add missing stage name (#6535) @Hongbo-Miao
• machine: move to tpi[ssh] (#6528) @casperdcl
• build(deps): Remove google-crc32c (#6529) @dependabot
• build(deps): Bump pytest from 6.2.4 to 6.2.5 (#6509) @dependabot
• restrict typing_extensions version for py3.10 (#6503) @skshetry
• ref: use fs.checksum instead of get_mtime_and_size (#6495) @efiop
• build(deps): Bump codecov/codecov-action from 2.0.2 to 2.0.3 (#6487) @dependabot

Thanks again to @Hongbo-Miao, @casperdcl, @daavoo, @dependabot, @dependabot[bot], @efiop, @himanshugarg, @isidentical, @karajan1001, @pared, @pmrowla, @pre-commit-ci, @pre-commit-ci[bot], @simonlsk and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• Remove all of the experiments in the workspace (#6394) @karajan1001

🐛 Bug Fixes

• exp run: pass recursive args to checkout (#6458) @skshetry

🔨 Maintenance

• ruamel.yaml: adjust to new DuplicatedKeyError (#6464) @pared

Thanks again to @efiop, @karajan1001, @pared and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🚀 New Features and Enhancements

• analytics: introduce DVC_NO_ANALYTICS (#6453) @efiop
• Update dvc exp branch help (#6439) @daavoo

Thanks again to @daavoo, @efiop and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🐛 Bug Fixes

• fetched: correctly order mixed odbs (#6436) @isidentical

Thanks again to @efiop and @isidentical for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🚀 New Features and Enhancements

• Change ui message for the updater (#6437) @skshetry
• plots: upgrade to latest Vega library versions (#6371) @arjvik
• updater: only notify in dvc doctor (#6247) @efiop

🐛 Bug Fixes

• fs: find(..., detail=True) now casts values properly (#6432) @isidentical

🔨 Maintenance

• add support for Python 3.10 (#6422) @skshetry

Thanks again to @arjvik, @dependabot, @dependabot[bot], @efiop, @isidentical and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

🚀 New Features and Enhancements

• Remove a special queued experiments (#6393) @karajan1001
• Push running checkpoint to remote (#6332) @karajan1001
• cleanup index, and add test for used_objs (#6390) @skshetry
• Show name of experiments in the queue (#6382) @karajan1001
• fs: get_fs_config: raise RemoteNotFoundError (#6375) @efiop
• ssh: migrate to sshfs (#6295) @isidentical
• remote: add excludes for AzureDefaultCredentials (#6359) @meierale
• plots: cleanup data extraction (#6355) @pared
• setup: bump flatten-dict to 0.4.1 (#6357) @isidentical
• fsspec: implement callback mixins (#6322) @isidentical
• do not print in style by default for errors (#6344) @skshetry
• plots/metrics/params/experiments show: stop throwing exceptions (#5984) @pared
• oss: migrate to ossfs (#6307) @karajan1001
• azure: support directory imports with azure (#6245) @isidentical
• exp show: display running/queued state for experiments (#6174) @pmrowla
• add: ignore duplicated targets? (#6286) @skshetry

🏇 Optimizations

• repofs: use underlying fs.download to download files (#6401) @efiop
• ssh: migrate to sshfs (#6295) @isidentical
• odb: use regular upload() when the source filesystem is local (#6365) @isidentical
• output: cache loaded tree objects (#6301) @efiop
• fsspec: use rm_file() (#6358) @isidentical
• setup: bump flatten-dict to 0.4.1 (#6357) @isidentical

🐛 Bug Fixes

• Fix issue while sending analytics report (#6407) @skshetry
• pygit: account for locked dirs on windows during checkout (#6408) @pmrowla
• More granular read-write lock requests in the run() and reproduce() methods of the Stage class (#5815) @maximerischard
• benedict: cleanup code and add tests (#6380) @skshetry
• ssh: handle concurrency issues for raw command execution (#6370) @isidentical
• do not print in style by default for errors (#6344) @skshetry
• objects: fix gdrive CI issue (#6338) @pmrowla
• import-url: use the same chunk size for istextblock() calls (#6303) @isidentical
• plots/metrics/params/experiments show: stop throwing exceptions (#5984) @pared
• import-url: compute the stage hash in --to-remote (#6175) @isidentical
• fs: repo: handle directories with multiple artifacts (#6278) @efiop

🔨 Maintenance

• build(deps): bump google-cloud-storage from 1.41.1 to 1.42.0 (#6415) @dependabot
• build(deps): bump jaraco-windows from 5.5.0 to 5.6.0 (#6396) @dependabot
• gdrive: migrate to pydrive2.fs (#6399) @isidentical
• build(deps): bump wheel from 0.36.2 to 0.37.0 (#6404) @dependabot
• introduce index (#6300) @skshetry
• Unittest to pytest: moved few unittest tests to pytest (#6379) @nik123
• objects: use object IDs and references instead of naive objs in status/transfer (#6360) @pmrowla
• build(deps): bump pylint from 2.8.3 to 2.9.6 (#6369) @dependabot
• Fix merging of params when empty (#6378) @skshetry
• setup: relax aiobotocore requirements (#6367) @efiop
• build(deps): bump codecov/codecov-action from 2.0.1 to 2.0.2 (#6361) @dependabot
• setup: bump flatten-dict to 0.4.1 (#6357) @isidentical
• build(deps): bump google-cloud-storage from 1.41.0 to 1.41.1 (#6341) @dependabot
• update webdav4 to latest fsspec version (#6343) @skshetry
• build(deps): bump codecov/codecov-action from 1.5.2 to 2.0.1 (#6334) @dependabot
• objects: migrate remote push/pull to objects.transfer (#6308) @pmrowla
• setup: upgrade adlfs to 2021.7.1 (#6337) @isidentical
• build(deps): bump google-cloud-storage from 1.40.0 to 1.41.0 (#6311) @dependabot
• setup: pathspec <0.9.0 (#6331) @efiop
• azure: move to default fsspec loop (#6323) @isidentical
• setup: bump ossfs==2021.7.5 (#6326) @isidentical
• setup: upgrade fsspec/s3fs/gcsfs to 2021.7.0 (#6317) @isidentical
• fsspec: create directories on upload/upload_fobj (#6309) @isidentical
• build(deps): bump wsgidav from 3.1.0 to 3.1.1 (#6302) @dependabot
• objects: use separate staging ODB for staging trees (#6195) @pmrowla

Thanks again to @Sayanta66, @cclauss, @dependabot, @dependabot[bot], @efiop, @isidentical, @karajan1001, @maximerischard, @meierale, @nik123, @pared, @pmrowla, @pre-commit-ci, @pre-commit-ci[bot] and @skshetry for the contributions! 🎉

Refer to https://dvc.org/doc/install for installation instructions.

Changes

• setup: relax requirements for importlib-metadata

Refer to https://dvc.org/doc/install for installation instructions.

Changes

🐛 Bug Fixes

• pkg: don't build universal wheels (#6299) @efiop

🔨 Maintenance

• pkg: don't build universal wheels (#6299) @efiop

Thanks again to @efiop for the contributions! 🎉