monstache

• Seperate namespace regexes for drop operations vs create/update/delete operations
• Better handling of panics
• A new index-as-update boolean config option that allow merge instead of replace
• Fixes to the find and findOne functions available in scripts

Changes

• add -pprof setting. When enabled and combined with -enable-http-server you can read profiling information. See Profiling for Go for more information.
• add -enable-easy-json setting. When enabled easy-json will be used for serialization to Elasticsearch.
• tweaks to the tailing code in gtm

Changes

• add -pprof setting. When enabled and combined with -enable-http-server you can read profiling information. See Profiling for Go for more information.
• add -enable-easy-json setting. When enabled easy-json will be used for serialization to Elasticsearch.
• tweaks to the tailing code in gtm

Changes

• decrease the fetch channel flush timeout
• clarify version conflicts and invalid json messages as warnings
• remove the /config endpoint for better security (use -print-config instead)
• fix LoadPlugins method (contributed by @YouthLab)
• performance tweaks in gtm

Changes

• decrease the fetch channel flush timeout
• clarify version conflicts and invalid json messages as warnings
• remove the /config endpoint for better security (use -print-config instead)
• fix LoadPlugins method (contributed by @YouthLab)
• performance tweaks in gtm

Changes

• Expose new setting direct-read-split-max which limits the number of times a collection is split for reading during direct-reads and thus the number of go routines and MongoDB connections spawned. The default is 9. Tune this setting to increase/decrease the amount of memory the monstache process will consume.
• Better Docker support due to the contributions of @a-magdy.

Changes

• Expose new setting direct-read-split-max which limits the number of times a collection is split for reading during direct-reads and thus the number of go routines and MongoDB connections spawned. The default is 9. Tune this setting to increase/decrease the amount of memory the monstache process will consume.
• Better Docker support due to the contributions of @a-magdy.

Changes

• Fix for issue #65, year outside of [0,9999]. Invalid time will be removed now with prune-invalid-json turned on
• Fix for issue #62, the number of connections to MongoDB is now limited to a max of 32 per namespace
• Fix for issue #59, unsupported values of +/- Infinity and NaN. These values can now be removed with the prune-invalid-json setting
• Fix for issue $46 and #66, having to do with filtering. Filters now use locks to ensure the javascript environment is used by one at a time.

Changes

• Fix for issue #65, year outside of [0,9999]. Invalid time will be removed now with prune-invalid-json turned on
• Fix for issue #62, the number of connections to MongoDB is now limited to a max of 32 per namespace
• Fix for issue #59, unsupported values of +/- Infinity and NaN. These values can now be removed with the prune-invalid-json setting
• Fix for issue $46 and #66, having to do with filtering. Filters now use locks to ensure the javascript environment is used by one at a time.

Note there is a problem with javascript filters in this release. This will be fixed shortly in 4.6.3, but until then please build monstache v4 using go get github.com/rwynn/monstache. This fixes the javascript filter issue

Changes

• Fix regression in 3.13 series where collections under 50K documents were not synching
• Performing Tuning. The following defaults have changed so please update your config files accordingly.

elasticsearch-max-conns went from 10 -> 4
elasticsearch-max-docs went from 1000 -> do not flush based on count (I suggest not overriding this since document sizes can vary greatly - instead use max-bytes)
elasticsearch-max-bytes went from 5MB -> 8MB

Note when you specify elasticseach-max-bytes the value is in bytes not MB. Sorry, I know that is annoying.

Note there is a problem with javascript filters in this release. This will be fixed shortly in 3.13.3, but until then please build monstache v3 using go get gopkg.in/rwynn/monstache.v3. This fixes the javascript filter issue

Changes

• Fix regression in 3.13 series where collections under 50K documents were not synching
• Performing Tuning. The following defaults have changed so please update your config files accordingly.

elasticsearch-max-conns went from 10 -> 4
elasticsearch-max-docs went from 1000 -> do not flush based on count (I suggest not overriding this since document sizes can vary greatly - instead use max-bytes)
elasticsearch-max-bytes went from 5MB -> 8MB

Note when you specify elasticseach-max-bytes the value is in bytes not MB. Sorry, I know that is annoying.

Note there is a major regression in versions 4.6.0 and 4.6.1. This will be fixed shortly in 4.6.2, but until then please build monstache v4 using go get github.com/rwynn/monstache. This will pick up new version of a dependency to fix the issue

Changes

• Performance and bug fixes in the gtm library

Note there is a major regression in versions 3.13.0 and 3.13.1. This will be fixed shortly in 3.13.2, but until then please build monstache v3 using go get gopkg.in/rwynn/monstache.v3. This will pick up new version of a dependency to fix the issue

Changes

• Performance and bug fixes in the gtm library

Changes

• Performance improvements. Much of the performance gains come from an upgrade of the gtm library. This library now uses split vector failing back to a paginated range queries. Also, some buffering has been removed at the gtm level for certain operations.

Changes

• Performance improvements. Much of the performance gains come from an upgrade of the gtm library. This library now uses split vector failing back to a paginated range queries. Also, some buffering has been removed at the gtm level for certain operations.

Changes

• Adds an option delete-index-pattern to specify an Elasticsearch index pattern to scope stateless deletes. Indexes outside of this
  pattern will not be considered when propogating deletes from MongoDB to Elasticsearch. By default all Elasticsearch indexes are queried.
• Adds the ability to specify a global filter function in Javascript. Previously, a filter function needed to be tied to a MongoDB namespace.
  Now you can leave off the namespace and the filter function will be applied to all namespaces. The filter function will receive the document
  as the first argument and the MongoDB namespace as the second argument.
• Breaking change: direct-read-cursors and direct-read-batch-size have been removed as options. The underlying gtm library of monstache has been
  upgraded and no longer supports parallelCollectionScan which is being removed in future versions of MongoDB. Now gtm will use splitVector to divy
  up collections to read documents concurrently. Also, the batch size will be managed by MongoDB and not set explicitly. See the gtm library docs for more information.
• Adds a boolean configuration option, prune-invalid-json, which defaults to false. Set this to true if your MongoDB data has values such as +Inf,
  -Inf, or NaN which are not supported by the golang JSON parser and cause infinite error loops to occur. With prune-invalid-json set to true Monstache
  will remove these values before indexing into Elasticsearch to avoid these errors.

Changes

• Adds an option delete-index-pattern to specify an Elasticsearch index pattern to scope stateless deletes. Indexes outside of this
  pattern will not be considered when propogating deletes from MongoDB to Elasticsearch. By default all Elasticsearch indexes are queried.
• Adds the ability to specify a global filter function in Javascript. Previously, a filter function needed to be tied to a MongoDB namespace.
  Now you can leave off the namespace and the filter function will be applied to all namespaces. The filter function will receive the document
  as the first argument and the MongoDB namespace as the second argument.
• Breaking change: direct-read-cursors and direct-read-batch-size have been removed as options. The underlying gtm library of monstache has been
  upgraded and no longer supports parallelCollectionScan which is being removed in future versions of MongoDB. Now gtm will use splitVector to divy
  up collections to read documents concurrently. Also, the batch size will be managed by MongoDB and not set explicitly. See the gtm library docs for more information.
• Adds a boolean configuration option, prune-invalid-json, which defaults to false. Set this to true if your MongoDB data has values such as +Inf,
  -Inf, or NaN which are not supported by the golang JSON parser and cause infinite error loops to occur. With prune-invalid-json set to true Monstache
  will remove these values before indexing into Elasticsearch to avoid these errors.

Changes

• This release changes the default deletion strategy to be stateless. This means that, by default, monstache no longer stores routing metadata in the monstache.meta collection. Instead, if custom routing is performed via scripts (changing parent, index, routing, etc. values), monstache will perform a term search by _id across all indexes to find and delete documents if they are deleted from MongoDB. To revert to the previous strategy of saving routing info in monstache.meta, run monstache with the config option delete-strategy set to 1. To use a delete strategy of ignore set delete-strategy to 2. The ignore delete strategy just means that monstache will not attempt to sync individual documents deletes in MongoDB to Elasticsearch.
• The previous deletion strategy, saving state to monstache.meta, was time and space intensive. With the new strategy you will likely see better indexing performance if you have been using scripts to alter the _meta_monstache variable.

Changes

• This release changes the default deletion strategy to be stateless. This means that, by default, monstache no longer stores routing metadata in the monstache.meta collection. Instead, if custom routing is performed via scripts (changing parent, index, routing, etc. values), monstache will perform a term search by _id across all indexes to find and delete documents if they are deleted from MongoDB. To revert to the previous strategy of saving routing info in monstache.meta, run monstache with the config option delete-strategy set to 1. To use a delete strategy of ignore set delete-strategy to 2. The ignore delete strategy just means that monstache will not attempt to sync individual documents deletes in MongoDB to Elasticsearch.
• The previous deletion strategy, saving state to monstache.meta, was time and space intensive. With the new strategy you will likely see better indexing performance if you have been using scripts to alter the _meta_monstache variable.

Changes

• Allow specifying a script without a namespace. In this case documents from all collections will be run through the script. The document object will continue to be the 1st argument to the function and a new 2nd argument will be the namespace of the source document. Fixes #55.
• Fix an issue where a Date object created in Javascript would not be formatted correctly for indexing.
• Build with go 1.10.1