elasticsearch-dump

If there is an error, we now log it to stderr, not stdout

• move the async package to dependancies, as it might be needed for multielasticdump
• note the Apache2 license in package.json

Set custom transport names (both input and output)

• by @TheDeveloper via https://github.com/taskrabbit/elasticsearch-dump/pull/140
• by @evantahler via https://github.com/taskrabbit/elasticsearch-dump/pull/180

Readme updates

• by @arcz

MultiElasticDump

This package also ships with a second binary, multielasticdump. This is a wrapper for the normal elasticdump binary, which provides a limited option set, but will run elasticdump in parallel across many indexes at once. It runs a process which forks into n (default your running host's # of CPUs) subprocesses running elasticdump.

The limited option set includes:

• parallel: os.cpus(),
• match: '^.*$',
• input: null,
• output: null,
• scrollTime: '10m',
• limit: 100,
• offset: 100,

In this mode, --input MUST be a URL for the base location of an ElasticSearch server (http://localhost:9200) and --output MUST be a directory. The new options, --parallel is how many forks should be run simultaneously and --match is used to filter which indexes should be dumped (regex). Each index that does match will have a data, mapping, and analyzer file created.

Update docker base image to Alpine Linux (saves ~400MB)

• by @khanhicetea via https://github.com/taskrabbit/elasticsearch-dump/pull/167

This release updates docket to use v4.x of node.js

Solved a bug introduced in v1.0.0 in which large dumpfiles, when read, would cause the application to crash.

Issue report: https://github.com/taskrabbit/elasticsearch-dump/issues/160
Solve: https://github.com/taskrabbit/elasticsearch-dump/pull/161

Breaking Changes!!!

• Simplified the outputs of both stdio and file to remove the starting/stopping braces and leading comma.
  • The file format is now simply a line-delimited JSON elements.
  • This allows for more versatile handling of dump files with the rest of the unix toolset.
    • For example, it would now be valid to restore a dumpfile from arbitrary lines without appending the starting and stopping brace characters.
• Created one file transport that combines both file and stdio... as in unix-land, they are more-or-less the same thing.
  • We will now also throw an error if a file to write to already exists, and will no longer overwrite existing files (safety first!)
• This tool now requires nodejs v1.0.0 (iojs) or higher to function due to changes in how steams are handled.

Converting old dump files.

To convert an old dump file, in a text-editor of your choice, do the following:

• remove the first line (which should just be [)
• remove the last ling (which should just be ])
• skip the fist line, and then remove the first character of every remaining line (which should just be ,)

https://github.com/taskrabbit/elasticsearch-dump/pull/156

Thanks to @bigsan for both of these PRs!

add --input-file-buffer-size option

• Set the buffer size for file type input to increase reading performance for large size documents.
• Set the buffer size for file type input. If there are large size documents, increase the buffer size to get better performance. (default: null)
• https://github.com/taskrabbit/elasticsearch-dump/pull/150

add --bulk-mode option to enable bulk partial update and bulk delete

• The mode can be index, delete or update. 'index': Add or replace documents on the destination index. 'delete': Delete documents on destination index 'update': Use 'doc_as_upsert' option with bulk update API to do partial update. (default: index)
• https://github.com/taskrabbit/elasticsearch-dump/pull/152

https://github.com/taskrabbit/elasticsearch-dump/pull/148

https://github.com/taskrabbit/elasticsearch-dump/pull/146

• do not suppress the original error message when JSON cannot be parsed
  • via https://github.com/taskrabbit/elasticsearch-dump/commit/d679213b85b40bcbc0fe573beb3d097d2505c702
• Add a docker file and instructions how to use it
  • via https://github.com/taskrabbit/elasticsearch-dump/pull/133

• Add sourceOnly and jsonLines options via https://github.com/taskrabbit/elasticsearch-dump/pull/90
• lib/transport/elasticsearch.js: Also include _routing and _parents fields during data import via https://github.com/taskrabbit/elasticsearch-dump/pull/90
• Added --input-index and --output-index flags via https://github.com/taskrabbit/elasticsearch-dump/pull/128

https://github.com/taskrabbit/elasticsearch-dump/pull/123

https://github.com/taskrabbit/elasticsearch-dump/pull/120

https://github.com/taskrabbit/elasticsearch-dump/pull/115

https://github.com/taskrabbit/elasticsearch-dump/pull/112

https://github.com/taskrabbit/elasticsearch-dump/pull/100

https://github.com/taskrabbit/elasticsearch-dump/pull/93

https://github.com/taskrabbit/elasticsearch-dump/pull/87 and https://github.com/taskrabbit/elasticsearch-dump/issues/83