monstache

• Fix a race condition when a related config is used and a golang plugin implements Process.

• Use less CPU resources
• Fix for race conditions

• Use less CPU resources
• Fix for race conditions

• Fix an issue where a paused monstache process would not resume correctly in cluster mode

• Fix an issue where a paused monstache process would not resume correctly in cluster mode

• Fix an issue with workers where only one worker would be used for change documents
• Redact sensitive connection information when logging connection failures

• Fix an issue with workers where only one worker would be used for change documents
• Redact sensitive connection information when logging connection failures

• Fix for connection checker thread exiting early in cluster mode
• Better handling of JSON serialization errors

• Fix for connection checker thread exiting early in cluster mode
• Better handling of JSON serialization errors

• Reliability improvements
• Fix allowing one to use a MongoDB view as a direct-read-namespace
• Addition of the relate config to declare dependencies between collections
• Experimental support for AWS Signing Version 4

• Reliability improvements
• Fix allowing one to use a MongoDB view as a direct-read-namespace
• Addition of the relate config to declare dependencies between collections
• Experimental support for AWS Signing Version 4

• Fixes related to i/o timeout errors
• Default timeout configurations changed to no timeout (0) with the exception of the initial connection to MongoDB which times out after 15 seconds. Values of 0 disable timeouts. All other positive values are in seconds. New defaults shown below. You probably do not need to specify any of these values unless you encounter problems.

[mongo-dial-settings]
timeout=15
read-timeout=0
write-timeout=0

[mongo-session-settings]
socket-timeout=0
sync-timeout=0

• Fixes related to i/o timeout errors
• Default timeout configurations changed to no timeout (0) with the exception of the initial connection to MongoDB which times out after 15 seconds. Values of 0 disable timeouts. All other positive values are in seconds. New defaults shown below. You probably do not need to specify any of these values unless you encounter problems.

[mongo-dial-settings]
timeout=15
read-timeout=0
write-timeout=0

[mongo-session-settings]
socket-timeout=0
sync-timeout=0

• Clean up timeout configurations and increase default timeout values
• New timeout configurations surfaced - read and write timeout. Configure as follows (default values shown):

[mongo-dial-settings]
timeout=10
read-timeout=600
write-timeout=30

[mongo-session-settings]
socket-timeout=600
sync-timeout=600

• Clean up timeout configurations and increase default timeout values
• New timeout configurations surfaced - read and write timeout. Configure as follows (default values shown):

[mongo-dial-settings]
timeout=10
read-timeout=600
write-timeout=30

[mongo-session-settings]
socket-timeout=600
sync-timeout=600

This is a big release with support for change streams and aggregation pipelines!

• Fix for issue #97 where monstache would exit before syncing all documents with -exit-after-direct-reads enabled
• Support added for MongoDB change streams via the change-stream-namespaces option
• New golang plugin functions Process and Pipeline added to the existing Map and Filter functions. The Process function allows one to code complex processing after an event. The Process function has access to the MongoDB session, the Elasticsearch client, the Elasticsearch bulk processor, and information about the change that occurred (insert, update, delete). The Pipeline function allows one to assign MongoDB pipeline stages to both direct reads and change streams. Since the pipeline stages may differ between direct reads and change streams the function is passed a boolean indicating the source of the data. For example, a $match clause on the change stream may need to reference the fullDocument field since the root will be the change event. For direct reads the root will simply be the full document.
• New config option pipeline allows one to create aggregation pipelines in javascript for direct reads and change streams. This can be used instead of the Pipeline function in a golang plugin. The exported function in javascript takes a namespace and a boolean indicating whether or not the source was a change stream. The function should return an array of pipeline stages to apply.
• New config option pipe-allow-disk which when enabled allows large pipelines to use the disk to save intermediate results.
• New global function available in javascript script functions named pipe. The pipe function is simliar to existing find function but takes an array of aggregation pipeline stages as the first argument.

direct-read-namespaces = [test.test]
change-stream-namespaces = [test.test]
[[pipeline]]
script = """
module.exports = function(ns, changeStream) {
  if (changeStream) {
    return [
      { $match: {"fullDocument.foo": 1} }
    ];
  } else {
    return [
      { $match: {"foo": 1} }
    ];
  }
}
"""
[[script]]
namespace = "test.test"
script = """
module.exports = function(doc, ns) {
  doc.extra = pipe([
    { $match: {foo: 1} },
    { $limit: 1 },
    { $project: { _id: 0, foo: 1}}
  ]);
  return doc;
}
"""

This is a big release with support for change streams and aggregation pipelines!

• Fix for issue #97 where monstache would exit before syncing all documents with -exit-after-direct-reads enabled
• Support added for MongoDB change streams via the change-stream-namespaces option
• New golang plugin functions Process and Pipeline added to the existing Map and Filter functions. The Process function allows one to code complex processing after an event. The Process function has access to the MongoDB session, the Elasticsearch client, the Elasticsearch bulk processor, and information about the change that occurred (insert, update, delete). The Pipeline function allows one to assign MongoDB pipeline stages to both direct reads and change streams. Since the pipeline stages may differ between direct reads and change streams the function is passed a boolean indicating the source of the data. For example, a $match clause on the change stream may need to reference the fullDocument field since the root will be the change event. For direct reads the root will simply be the full document.
• New config option pipeline allows one to create aggregation pipelines in javascript for direct reads and change streams. This can be used instead of the Pipeline function in a golang plugin. The exported function in javascript takes a namespace and a boolean indicating whether or not the source was a change stream. The function should return an array of pipeline stages to apply.
• New config option pipe-allow-disk which when enabled allows large pipelines to use the disk to save intermediate results.
• New global function available in javascript script functions named pipe. The pipe function is simliar to existing find function but takes an array of aggregation pipeline stages as the first argument.

direct-read-namespaces = [test.test]
change-stream-namespaces = [test.test]
[[pipeline]]
script = """
module.exports = function(ns, changeStream) {
  if (changeStream) {
    return [
      { $match: {"fullDocument.foo": 1} }
    ];
  } else {
    return [
      { $match: {"foo": 1} }
    ];
  }
}
"""
[[script]]
namespace = "test.test"
script = """
module.exports = function(doc, ns) {
  doc.extra = pipe([
    { $match: {foo: 1} },
    { $limit: 1 },
    { $project: { _id: 0, foo: 1}}
  ]);
  return doc;
}
"""

• Fix to omit version information on deletes when the index-as-update setting is ON (to match the omitted version information at indexing time)
• Fix issue #89 by making the indexed oplog field names and date format configurable

• Fix to omit version information on deletes when the index-as-update setting is ON (to match the omitted version information at indexing time)
• Fix issue #89 by making the indexed oplog field names and date format configurable

• Seperate namespace regexes for drop operations vs create/update/delete operations
• Better handling of panics
• A new index-as-update boolean config option that allow merge instead of replace
• Fixes to the find and findOne functions available in scripts