monstache
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
MIT License
Published by rwynn over 6 years ago
Changes
- Fix for a benign race condition in shutdown, introduced in 3.6.2, that caused a panic
Published by rwynn over 6 years ago
Changes
- Resume usage of upstream elastic client library now that fix for Elasticsearch going down has been merged
- When Elasticsearch goes down the elastic client will now put back pressure on Add and Flush calls. When Elasticsearch comes back up it will resume Adding and Flushing were it left off. Do to the blocking nature of Add and Flush the shutdown function of monstache has been refactored to take this into account. Shutdown will not hang if Elasticsearch is down. It will try to Flush pending documents but if this blocks due to a down server it will still exit after a 5 second deadline.
Published by rwynn over 6 years ago
Changes
- Added more detailed error logging. Each bulk request line that failed will be logged separately with details. This is much more lightweight than having to turn on
verboseto get error details. Verbose is not a recommended setting for production.
Published by rwynn over 6 years ago
Changes
- This release focuses on improvements with regards to handling dropped connections to either Elasticsearch or MongoDB and resuming gracefully when they come back online
Published by rwynn over 6 years ago
Changes
- The previous release safeguards the integrity of inserts and updates with a version number, but neglected deletes. This release adds versions to deletes such that an [insert, delete] sequence that gets sent to Elasticsearch in 2 different requests (due to
elasticsearch-max-conns> 1) cannot actually perform a [delete, insert] instead. In this case the insert would now carry a version number < the delete version number and be rejected.
Published by rwynn over 6 years ago
Changes
- Fix for issue #37 - out of order indexing due to concurrent bulk indexing requests. With
elasticsearch-max-connsset to greater than 1 you may get out of order index requests; however after this fix each document is versioned such that Elasticsearch will not replace a newer version with an older one. The version of the document is the timestamp from the MongoDB oplog of when the change (insert, update) occurred. Out of order indexing typically happens when both an insert and an update are queued for a bulk request at around the same time. In this case, do to the way the bulk processor multiplexes requests onto multiple connections, the document may be received out of order.
Published by rwynn almost 7 years ago
Changes
- Support for sharded MongoDB cluster. See docs for details
- Performance optimizations
- Turn off bulk retries if configured to do so
Published by rwynn almost 7 years ago
Changes
- Allow the stats index name format to be configurable. Continues to default to index per day.
Published by rwynn almost 7 years ago
Changes
- Fix for the javascript mapping functions. An Otto Export does not appear to recurse into arrays. Need to do a recursive Export for this scenario.
Published by rwynn almost 7 years ago
Changes
- Add ability to embed documents during the mapping phase. Javascript plugins get 3 new global functions: findId, findOne, and find. Golang plugins get access to the mgo.Session. See the docs for details.
Published by rwynn almost 7 years ago
Changes
- Improve support for additional indexing metadata.
- Fix issue where indexing metadata was not honored
Published by rwynn almost 7 years ago
Changes
- Added optional http server. Enable with --enable-http-server flag. Listens on :8080 by default. Configure address with --http-server-addr :8000. The server responds to the following endpoints (/started, /healthz, /config, and /stats). The stats endpoint is only enabled if stats are enabled. The /started and /healthz endpoints can be used to check for liveness.
- Upgraded the gtm library with performance improvements
Published by rwynn almost 7 years ago
Changes
- Add systemd support
Published by rwynn about 7 years ago
Changes
- Built with go1.9
- Fix golint warnings
Published by rwynn over 7 years ago
Changes
- timestamp stats indexes by day for easier cleanup using e.g. curator
Published by rwynn over 7 years ago
Changes
- add print-config argument to display the configuration and exit
- add index-stats option to write indexing statistics into Elasticsearch for analysis
Published by rwynn over 7 years ago
Changes
- fix elasticsearch client http scheme for secure connections
Published by rwynn over 7 years ago
Changes
- fix invalid struct field tag
Published by rwynn over 7 years ago
Changes
- add direct-read-batch-size option
- upgrade gtm to accept batch size and to ensure all direct read errors are logged
Published by rwynn over 7 years ago
Changes
- fix slowdown on direct reads for large mongodb collections
