Warp speed Data Transfer (WDT) is an embeddedable library (and command line tool) aiming to transfer data between 2 systems as fast as possible over multiple TCP paths.
OTHER License
Bot releases are visible (Hide)
Long time since the previous release (I hope you enjoy github master / trunk which we keep stable!) with lots of changes:
Enjoy!
Published by ldemailly over 8 years ago
Published by ldemailly over 8 years ago
All of the goodies in v1.25 plus:
Published by ldemailly over 8 years ago
Long time since the previous release (but there is always github master!) - lot's of changes:
New features/major changes:
ssh dsthost wdt -directory destdir | ssh srchost wdt -directory srcdir -
# Or more complex:
(ssh localhost wdt -fork -start_port 0 -directory /tmp/dst1; ls *.h ) | wdt -manifest - -
Bug fixes:
Misc:
Published by ldemailly almost 9 years ago
Big release !
New features:
The receiver is generating a one time 128bits secret, encoded in the URL. This secret needs to be transmitted securely (at facebook we use secure thrift for this but you can use ssh for instance). Note that we do not try to authenticate and only encrypt.
Also estimate the time it may take for a slow receiver to consume the enqueued bytes to avoid timing out waiting for the ack to the end of transfer
There is a new https://github.com/facebook/wdt/blob/master/Wdt.h simpler API for C++ (which is work in progress but shows the direction we are headed)
file descriptor (in FileInfo api) wasn't used when download resumption was also on, locking , error code summary, ...
Issue #102
Published by ldemailly almost 9 years ago
Important bug fixes (notification change in big refactor was incorrect. issue with long running mode,...)
Changed code layout, moved to <wdt/X.h> from "X.h"
Clang-3.7 warnings
Summarize the error codes for the sender (more specific than ERROR/OK, for instance ABORT is differentiated from CONN_ERROR)
Published by nikunjy almost 9 years ago
New Features
Bugs fixed
Published by ldemailly about 9 years ago
Changed the retry logic to be based on lack of progress instead of being arbitrarily attributed to files retry counts
Fixed serious int overflow in lseek result handling - files between 2 and 4g (and 6 and 8g and 10-12...) ie whose size has bit 31 set and being set using block mode would fail
Removed the "e"xit command on receiver (dated from old times where the receiver would run forever by default)
Download resumption (TransferLogManager), thread history, checkpoints cleanup and important bug fixes
Published by ldemailly about 9 years ago
Really fixed URL backward compatibility (use -url_backward_compatibility=true if you have a wdt v20 talking to v18 or earlier)
new -exit_on_bad_flags=false if you want to continue despite bad command line (useful for forward compatibility)
shorter urls (wdt://host:port?num_port=4) and correct escaping of ipv6 [addr]
wdt receiver will now not overwrite any file by default unless you add -overwrite or use download resumption
misc bug fixes and cleanup
Published by ldemailly about 9 years ago
Changes since previous tag (Sept 16) :
Published by ldemailly about 9 years ago
fixes #73 (major version negotiation issue with URI/URL. v19 command line will work with any previous version but v18 for instance would crash v17 etc...). If using the C++ api directly and checking the errorCode of TransferRequest you need to use v20 and url_backward_compatibility=true option.
reset the connection timeout back to 1s (for high latency links)
Published by ldemailly about 9 years ago
Notable changes from previous version:
WDT can now resume interrupted transfer based solely on file size based an examining the directory tree (as opposed to previously using it's own transfer log)
If you are using wdt on single disk systems use that mode and disable blocks mode, if you are using flash continue to use the transfer log and the block mode which are most effectives when doing simultaneous IOs is not expensive (like flash, unlike single disk systems)
Published by ldemailly about 9 years ago
Notable changes from previous version:
WDT now correctly binds to multiple sockets for hosts with both IPv4 and IPv6 addresses
Bug fixes and more/better tests
Published by ldemailly about 9 years ago
shell script problem in new test fixed in next build v1.16.1508311
Published by ldemailly about 9 years ago
First github release (should have started tagging earlier but let's start now)
No major known bugs
Our versioning scheme is vMAJOR.MINOR.BUILD where BUILD is YYMMDDP = date + build index starting at 0 (first build of the day) up to 9 (10th build of the day)