This project is maintained in the spare time of contributors and so time able to be dedicated to it is extremely limited.
Please file issues only for reproducable problems. Given our limited time, we are not able to provide any help using the river and issues that simply say "things aren't working" will be closed. However, if you're able to diagnose an issue yourself we may be able to help with a fix and we are happy to review pull requests.
To debug issues, try changing the log level to trace as described in the wiki. You may wish to build a custom version of the river with extra logging.
This plugin uses MongoDB or the TokuMX fork of MongoDB as datasource to store data in ElasticSearch. Filtering and transformation are also possible. See the wiki for more details.
In order to install the plugin, simply run: bin/plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.9
Note that you must be using MongoDB replica sets since this river tails the oplog.
MongoDB River Plugin | ElasticSearch | MongoDB | TokuMX |
---|---|---|---|
master | 1.7.3 | 3.0.0 | 1.5.1 |
2.0.9 | 1.4.2 | 3.0.0 | 1.5.1 |
2.0.5 | 1.4.2 | 2.6.6 | 1.5.1 |
2.0.2 | 1.3.5 | 2.6.5 | 1.5.1 |
2.0.1 | 1.2.2 | 2.4.9 -> 2.6.3 | 1.5.0 |
2.0.0 | 1.0.0 -> 1.1.1 | 2.4.9 | |
1.7.4 | 0.90.10 | 2.4.8 | |
1.7.3 | 0.90.7 | 2.4.8 | |
1.7.2 | 0.90.5 | 2.4.8 | |
1.7.1 | 0.90.5 | 2.4.6 | |
1.7.0 | 0.90.3 | 2.4.5 | |
1.6.11 | 0.90.2 | 2.4.5 | |
1.6.9 | 0.90.1 | 2.4.4 | |
1.6.8 | 0.90.0 | 2.4.3 | |
1.6.7 | 0.90.0 | 2.4.3 | |
1.6.6 | 0.90.0 | 2.4.3 |
Initial implementation by aparo.
Modified to get the same structure as the other Elasticsearch rivers (like CouchDB)
The latest version monitors the oplog capped collection and supports attachment (GridFS).
Configure the river using the definition described in the wiki:
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "DATABASE_NAME",
"collection": "COLLECTION",
"gridfs": true
},
"index": {
"name": "ES_INDEX_NAME",
"type": "ES_TYPE_NAME"
}
}'
Example:
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "testmongo",
"collection": "person"
},
"index": {
"name": "mongoindex",
"type": "person"
}
}'
Import data from mongo console:
use testmongo
var p = {firstName: "John", lastName: "Doe"}
db.person.save(p)
Query index:
curl -XGET 'http://localhost:9200/mongoindex/_search?q=firstName:John'
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "testmongo",
"collection": "fs",
"gridfs": true
},
"index": {
"name": "mongoindex",
"type": "files"
}
}'
Import binary content in mongo:
%MONGO_HOME%\bin>mongofiles.exe --host localhost:27017 --db testmongo --collection fs put test-document-2.pdf
connected to: localhost:27017
added file: { _id: ObjectId('4f230588a7da6e94984d88a1'), filename: "test-document-2.pdf", chunkSize: 262144, uploadDate: new Date(1327695240206), md5: "c2f251205576566826f86cd969158f24", length: 173293 }
done!
Query index:
curl -XGET 'http://localhost:9200/files/4f230588a7da6e94984d88a1?pretty=true'
Admin URL: http://localhost:9200/_plugin/river-mongodb/
See more details check the wiki
This software is licensed under the Apache 2 license, quoted below.
Copyright 2009-2012 Shay Banon and ElasticSearch <http://www.elasticsearch.org>
Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
options/import_all_collections
parameter can be used to import all collection of a database (see issue #177)options/store_statistics
parameter can be used to store statistic each time bulk processor is flushed. Data are store in _river/{river.name}index/bulk/concurrent_bulk_requests
has been changed to the number of cores available.options/drop_collection
will also track dropDatabase
(see issue #133)options/skip_initial_import
to skip initial import using collection data. Default value is false.ts
or op
to be used in user collection. (see pr #136)options/exclude_fields
by @ozanozen (see issue #103)options/drop_collection
(see issue #105)options/drop_collection
option (issue #79)options/include_collection
parameter to include the collection name in the document indexed. (see #101)Install Maven (e.g brew install maven
). Be sure mvn
command is available.
Clone the repository:
git clone https://github.com/richardwilly98/elasticsearch-river-mongodb.git
Execute installation script (inside the dir where you cloned the project):
./elasticsearch-river-mongodb/install-local.sh
If your elasticsearch is not installed in /usr/share/elasticsearch
you can set it ES_HOME
. eg:
ES_HOME=/usr/local/Cellar/elasticsearch/1.1.1/ ./elasticsearch-river-mongodb/install-local.sh
Install the m2eclipse plugin:
Install the testng eclipse plugin
Run mvn eclipse:eclipse -DdownloadSources=true
Import the project with File > Import > Maven > Existing Maven Projects