wdt

Long time since the previous release (I hope you enjoy github master / trunk which we keep stable!) with lots of changes:

• Common "wdt> " prefix in all the logging (helps finding/filtering wdt specific logs when embedded in another service)
• Progress report during file discovery
• Throttler changes working with the resource controller and can be lowered in the middle of a transfer and improved fairness.
• Multiple applications support within one process, Wdt options not forced to be a singleton, apply to a tree of sub object below the application wdt object
• Allow receiver to specify its --hostname
• New --fsync option to fsync after the last block of each file is written.
• Incremental transfers (-enable_download_resumption) improved reporting
• Wdt URI encoding/decoding is now binary safe (transferid can be any binary data)
• Fixed disk mode (files transferred serially and not cut in blocks) >2Gb errors; integer overflow
• Fewer dependencies. Uses std::make_uniq (need cxx14)
• Receiver now drives Sender's download resumption through the URI
• Benchmark generation (bench/) code
• New more efficient Varint encoding and less likely to have memory corruption
• Bug fixes (see commits) like listening sockets staying open after receiver timeout
• Misc changes like: Release/build improvements, Code quality improvements like constructor cleanups (#16)

Enjoy!

• Option to use fadvise(POSIX_FADVISE_DONTNEED) to avoid taking too much cache/buffers on sender/read path
• other fixes/improvements
• Simpler/shorter tests

All of the goodies in v1.25 plus:

• Support for deletion during resumption through the delete_extra_files - if a subsequent snapshot has fewer files than were already sent before, delete those first. This is useful for use cases like rocksdb/myrocks where after compaction some files disappear
• Dynamic ports (available ports) is now the default, you don't need to specify -start_port 0 anymore for the receiver side of wdt, if you want to revert to fixed ports, use -static_ports or set a non 0 start_port explicitly. This should make it easier for wdt transfers using pipe to connect the receiver and sender to always succeed even if another transfer is already ongoing on the same destination

Long time since the previous release (but there is always github master!) - lot's of changes:

New features/major changes:

• Much better command line interface to match the need of the encryption key transmission securely, "-" argument for the sender reads the connection url from stdin, which the receiver emits on stdout. So the 2 can be plugged through a pipe (securely using ssh - which will only be used for secure key exchange, everything else blazing fast and encrypted through WDT's own socket)

ssh dsthost wdt -directory destdir | ssh srchost wdt -directory srcdir -
# Or more complex:
(ssh localhost wdt -fork -start_port 0 -directory /tmp/dst1; ls *.h ) | wdt -manifest - -

• Aes GCM crypto support (requires a recent openssl version) : is now the default - it does both encryption and validates the integrity in 1 pass. We also support this during error handling only resuming at the last block that was successfully validated.
• Socket buffer size can be set explicitly (using -send_buffer_size and -receive_buffer_size)

Bug fixes:

• improved TransferLogManager locking (fail fast when can't be acquired) and detection of deleted containing directory/log file
• ResourceController double create, wdtSend can cancel previous / preexisting transfer
• with opening files during discovery

Misc:

• Consistent use of "Mbytes" throughout WDT for 1024*1024 = 1048576 bytes (aka Mebibyte (MiB)
• Testing/path improvements
• Removing direct use of WdtOptions::get across the code base - now the options can be different for different sender/receivers in the same process.

Big release !

New features:

• WDT now supports encryption (aes128)

The receiver is generating a one time 128bits secret, encoded in the URL. This secret needs to be transmitted securely (at facebook we use secure thrift for this but you can use ssh for instance). Note that we do not try to authenticate and only encrypt.

• Other improvements

Also estimate the time it may take for a slow receiver to consume the enqueued bytes to avoid timing out waiting for the ack to the end of transfer

There is a new https://github.com/facebook/wdt/blob/master/Wdt.h simpler API for C++ (which is work in progress but shows the direction we are headed)

• Bug fixes:

file descriptor (in FileInfo api) wasn't used when download resumption was also on, locking , error code summary, ...

• Known issues:

Issue #102

Important bug fixes (notification change in big refactor was incorrect. issue with long running mode,...)

Changed code layout, moved to <wdt/X.h> from "X.h"

Clang-3.7 warnings

Summarize the error codes for the sender (more specific than ERROR/OK, for instance ABORT is differentiated from CONN_ERROR)

New Features

1. Refactored the code to split Sender and Receiver into clearly identifiable single threaded code and multi-threaded part (SenderThread / ReceiverThread).
2. Wdt now has a thread controller, with thread primitives such as Barrier and Funnel Executors which made the writing of certain features of sender and receiver easier like doing process version mismatch, or sending file chunks in download resumption.
3. wdt_max_send_test.sh has been updated and is more robust. This test can be used to benchmark wdt and detect regressions.

Bugs fixed

1. A sender thread could fail on a connection and have a file in its thread transfer history that has not been acknowledged, however it wouldn't stop other threads to finish their part of the transfer even on getting the checkpoint, since the last source would not return to queue. This would result in the final status of transfers being error.
2. Transfer log manager had a writer thread that would write empty buffer to the transfer log file.
3. gcc4.9 SIOPF fix for one our static strings in GFlags.

Changed the retry logic to be based on lack of progress instead of being arbitrarily attributed to files retry counts

Fixed serious int overflow in lseek result handling - files between 2 and 4g (and 6 and 8g and 10-12...) ie whose size has bit 31 set and being set using block mode would fail

Removed the "e"xit command on receiver (dated from old times where the receiver would run forever by default)

Download resumption (TransferLogManager), thread history, checkpoints cleanup and important bug fixes

Really fixed URL backward compatibility (use -url_backward_compatibility=true if you have a wdt v20 talking to v18 or earlier)

new -exit_on_bad_flags=false if you want to continue despite bad command line (useful for forward compatibility)

shorter urls (wdt://host:port?num_port=4) and correct escaping of ipv6 [addr]

wdt receiver will now not overwrite any file by default unless you add -overwrite or use download resumption

misc bug fixes and cleanup

Changes since previous tag (Sept 16) :

1. Added fd to FileInfo. If fd, is provided, we use that fd and do not open the file. I am currently using pread always. I have to more perf test for pread. Max send is really slow in my new dev server, so need to test for perf in some other machine.
2. If connection url was provided, sender ignored file list. Fixed that and
   added a test for file info list
3. Added an option in e2e simple test to test o_direct mode
4. Simplifying o_direct code. We should never define macros named O_DIRECT.

fixes #73 (major version negotiation issue with URI/URL. v19 command line will work with any previous version but v18 for instance would crash v17 etc...). If using the C++ api directly and checking the errorCode of TransferRequest you need to use v20 and url_backward_compatibility=true option.

reset the connection timeout back to 1s (for high latency links)

Notable changes from previous version:

WDT can now resume interrupted transfer based solely on file size based an examining the directory tree (as opposed to previously using it's own transfer log)

If you are using wdt on single disk systems use that mode and disable blocks mode, if you are using flash continue to use the transfer log and the block mode which are most effectives when doing simultaneous IOs is not expensive (like flash, unlike single disk systems)

Notable changes from previous version:

WDT now correctly binds to multiple sockets for hosts with both IPv4 and IPv6 addresses

Bug fixes and more/better tests

shell script problem in new test fixed in next build v1.16.1508311

First github release (should have started tagging earlier but let's start now)

No major known bugs

Our versioning scheme is vMAJOR.MINOR.BUILD where BUILD is YYMMDDP = date + build index starting at 0 (first build of the day) up to 9 (10th build of the day)