Sample web crawler and search engine written in Node.JS and MongoDb
MIT License
A simple and easy to use search server written in node.js and MongoDb.
Search-Crawler is composed by a Node.Js web application to manage one or more websites and a set of json based REST API that can be used to query crawled pages and integrate the result inside any existing website.
Website crawling is implemented using Christopher Giffard's SimpleCrawler. Pages are stored in a MongoDb and search is powered by a full text mongodb query, see full text index.
To install search-crawler you need the following components:
Get the latest version of search-crawler at github. Install it by running the following command inside the folder where you have downloaded the package:
npm install
These will install in the current folder all the required libraries.
After installation you can execute the node.js application by executing:
npm run server
or
node index.js
The web server is created at port 8181, so you can browse it at http://localhost:8181. Search-Crawler try to connect to a mongo database using the following url:
mongodb://localhost:27017/search-crawler
See Configuration section for more information.
./src/config.js
file contains all the parameters used by search-crawler
.
Here some of the parameters:
// Allowed extension for crawling
config.crawler.allowedUrlPatterns = [
"/[^./]*$" // extension less
,"\\.(html|htm|aspx|php)$" // .html + .htm
];
// List of content types to process
config.crawler.contentTypes = ["text/html"];
// crawler interval
config.crawler.interval = 300;
// crawler maxConcurrency
config.crawler.maxConcurrency = 2;
// mongo host and database (mongodb version => 2.6 required)
config.db.mongo = {};
config.db.mongo.ip = process.env.IP || "localhost";
config.db.mongo.url = "mongodb://" + config.db.mongo.ip + ":27017/search-crawler";
// html "jquery style" selector for the body content (es. "body", "article", "div#text")
// can be override on each site
config.parser.defaultContentSelector = "body";
// nodejs server listening port
config.web.port = process.env.PORT || process.env.WEB_PORT || 8181;
config.web.ip = process.env.IP;
See ./src/config.js
for all available parameters.
To create a custom configuration you can edit config.js
file or you can create
a custom startup file like contoso.index.js
with a content like:
// Here I can modify configuration...
var config = require('./src/config.js');
config.web.port = 8282;
// then run real index.js
require('./index.js');
And then instead of executing index.js you can execute your custom
contoso.index.js
. This method has the advantage that you don't modify any
original file.
Each website has also it's own configuration (stored in mongodb inside each site document):
These configuration can be edited using the web application or through the API.
To configure automatic crawling of website you should set the crawlingCron
configuration with your required frequency specified as a cron expression.
Below a quick CRON guide.
Remember that to apply any changed in the cron scheduled expression you have to reload jobs using the appropriate command or restart the node application.
* * * * * *
= every seconds
0 * * * * *
= every minutes at second 0
0 5 * * * *
= every hours at minute 5
0 0 1 * * *
= every days at 1 AMOther then the user interface the following REST API are available:
/api/sites/:siteName/search?query=:query&limit=:limit
Search for a given expression inside a site. The result is a json with the list of pages that match the query with the following format:
[
{
"_id": "54550c07b242a89d4c862e6e",
"title": "Page Title",
"description": "Page description",
"url": "http://pageurl",
"score": 1.5690104166666665,
"keywords": [
"key1",
"key2"
]
},
{
...
}
]
/api/sites
Get the list of registered sites
/api/sites/:siteName
Get a specific site by its name
/api/sites/:siteName/pages
Get the list of pages of a specific site
/api/sites
Create a new site
/api/sites/:siteName/update-config
Update a site configuration
/api/sites/:siteName
Delete a specific site
/api/sites/:siteName/crawl
Start crawling process of a specific site
/api/sites/:siteName/register-page
Add a specific page to a site
/api/sites/:siteName/remove-pages
Remove all the pages from a site
/api/sites/:siteName/page-count
Get the registered page count of a site
Get all the configured jobs
/api/jobs/load
Load all the available jobs
Unload and stop all the jobs
You can debug Search-Crawler using node inspector with the following command:
npm run server-debug
Run karma unit tests with the following command:
npm run test-unit
Run mocha end to end tests with the following command:
npm run test-e2e
MIT License
Copyright (c) 2014 Davide Icardi
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: