A fast customizable RSS, ATOM, and RDF feed parser and aggregator
MIT License
This module is inspired by Dan Mactough's node-feedparser module which parses RSS, Atom, and RDF feeds using Isaac Schlueter's sax parser.
FeedJett is built on much of the same model for sax parsing as feedparser and shares alot of the same boiler plate code to parse feeds, though it does contain many enhancements and features as listed below
npm install feedjett
The easiest way to use feedjett is to just give it a readable stream. It will then return a readable object stream.
var FeedJett = require('feedjett'),
request = require('request');
var req = request('http://somefeedurl.xml'),
feedjett = FeedJett.createInstance([options]);
req.on('error', function (error) {
// handle any request errors
});
req.on('response', function (res) {
var stream = this;
if (res.statusCode != 200) return this.emit('error', new Error('Bad status code'));
stream.pipe(feedjett);
});
feedjett.on('error', function(error) {
// always handle errors
}).on('meta', function (meta) {
// emits the meta event if parseMeta is true (defaults to true)
console.log(meta);
}).on('readable', function() {
// This is where the action is!
var stream = this, item;
while (item = stream.read()) {
//item will also have a meta object attached to it if addMeta is true (defaults to true)
console.log(item);
}
});
I strongly encourage you to take a look at the demo test case for a more thorough working example.
FeedJett parses exactly what you want and how you want. By default FeedJett defines Arrays of properties for meta that dictate what the result meta or item should look like for all feeds (assuming normalize is set to true). These Arrays of properties are defined in an object called normalizedProps (one for meta, one for item). Each property in the Array will map to a parsing function which is responsible for parsing and normalizing node values into the defined property. This way, we aren't looping through each and ever property in the node, but instead are just parsing what needs to be parsed and nothing more. Default normalized props and parsing functions can be tailored and customized as described in the next section.
When creating a new instance of FeedJett, you can pass in an options object with one or more of the following object properties
NOTE: some of the following options may be similar to FeedParser but syntactically different (camelCased in most cases)
normalize
- (Default: true
) - Set to false
to override FeedJett's default behavior, which is to parse feeds
into an object that contains the generic properties patterned after (although not identical to) the RSS 2.0 format,
regardless of the feed's format. These properties are defined in the next section.
addMeta
- (Default: true
) - Set to false
to override FeedJett's default behavior, which is to add the
feed's meta
information to each item.
feedUrl
- (Default: undefined
) - The url (string) of the feed. FeedJett will use this value to set the xmlBase or
and resolve relative URL's if xml:base
isn't defined. Unlike FeedParser, FeedJett resolves relative URL's only if
xml:base is defined, it does not try to determine the base url from other tags.
resumeOnSaxError
- (Default: true
) - Set to false
to override FeedJett's default behavior, which is to emit
any SAXError
on error
and then automatically resume parsing. As noted by FeedParser, SAXErrors
are not usually
fatal, so this is usually helpful behavior. If you want total control over handling these errors and optionally
aborting parsing the feed, use this option.
strict
- (Default: false
) set to true
for strict mode (npm sax option)
MAX_BUFFER_LENGTH
- (Default: 16K (16 * 1024 * 1024)
- Default buffer length (npm sax option)
itemLimit
- (Default: undefined) set to an integer to enforce a limit on the number of items within a single feed
parseMeta
- (Default: true
) - set to false
to skip parsing AND emitting the meta
object. This can speed up
processing if you know you don't need or want meta
information.
parseRawNode
- (Default: false
) - set to true
if you would like the raw properties (non-normalized) for a
meta
or item
node to be added to the raw
property (meta.raw or item.raw). If set to true, each property
will retain it's original prefix. If a property does not include a prefix, then the feed type will be the prefix
for the raw property. (Default behavior in FeedParser)
Add the following to the options object to further customize the parsing framework for FeedJett. These options and
customizations assume that the normalize
option is not set to false
.
blackList
- {Array} - Define an array of String properties in order to disable parsing for specific normalized
properties. For example if you want all normalized properties to be parsed for a meta or item node except for
description and updatedDate, then you would set these in the blackList array. This would let FeedJett know to NOT
call the parseDescription or parseUpdatedDate parser functions, thus excluding both properties from the result meta
or item object. These values apply to both meta
and item
.
whiteList
- {Array} - Define an array of String properties in order to restrict parsing for ONLY the specified
normalized properties. For example if you only want the result object to contain a title, description, link, and
pubDate, you would set these in the whiteList array. This would let FeedJett know to call ONLY the parser functions
associated with these normalized properties and nothing else. These values apply to both meta
and item
.
Whitelisting only the properties you require can speed up processing.
NOTE: You can only define either blackList
OR whiteList
on options. If both are defined, whiteList
will take
precedence
var options = {
blackList: ['description', 'updatedDate']
};
//OR!! Don't define both!!!
var options = {
whiteList: ['title', 'description', 'link', 'pubDate']
};
var feedjett = FeedJett.createInstance(options); //... other feedjett logic as shown in earlier section
The following object (defined in feedjett.js) maps normalized props to parsing functions defined in the parser.js
file. You will also find detailed parsing information and logic for each parsing function.
var propertyParserMap = {
title: 'parseTitle',
description: 'parseDescription',
link: 'parseLink',
updatedDate: 'parseUpdatedDate',
pubDate: 'parsePubDate',
categories: 'parseCategories',
author: 'parseAuthor',
image: 'parseImage',
xmlUrl: 'parseXmlUrl',
origLink: 'parseOrigLink',
language: 'parseLanguage',
icon: 'parseIcon',
copyright: 'parseCopyright',
generator: 'parseGenerator',
cloud: 'parseCloud',
comments: 'parseComments',
commentRss: 'parseCommentRss',
updateInfo: 'parseUpdateInfo',
enclosures: 'parseEnclosures',
guid: 'parseGuid',
source: 'parseSource',
content: 'parseContent'
};
If the parsing logic for a specific parsing function doesn't suit your needs, then you can easily override the function
with your own. For example, if you would like to parse the normalized description property, you can add the following
function to the options
object, (must match the parsing function name) which will then be used to parse and
normalize the description property.
NOTE: each parsing function can define the following parameters be passed in the following node
node
: {Object} The raw item or meta used to check for other properties in order to normalize descriptionnodeType
: {String} The type of node (meta
or item
)feedType
: {String} The type of feed (rss
, atom
, or rdf
)
var options = {
parseDescription: function (node, nodeType, feedType) {
//add your custom parsing logic here
return 'custom parsed string value!';
}
};
var feedjett = FeedJett.createInstance(options); //... other feedjett logic as shown in earlier section
FeedJett.overridePropNames
FeedParser provides a static function for overriding the names of the default normalized properties. All you have to do is pass in a mapping object and the feed item/meta will have your custom property name instead of the default. The mapping object keys should be the original property name, and the value should be your custom property name
Example..
FeedJett.overridePropNames({
description : 'summary',
link : 'url',
pubDate : 'pub_date',
guid : 'id'
});
var feedjett = FeedJett.createInstance(); //... other feedjett logic as shown in earlier section
Now when you listen to readable for items or the meta event for meta, the result object will include myProperty
which
would have a value that equals the return value of parseMyProperty
. Behind the scenes, addCustomParser will add the
property to the normalizedProps meta or item Arrays, then add the myProperty -> parseMyProperty
mapping to the
propertyParserMap
object, then dynamically invoke the custom function and set the property on the result object.
FeedJett.addCustomParser
FeedJett provides a static function for adding custom properties not avialable by FeedJett as well as providing a
custom parser function to normalize the property. To achieve this you can call the FeedJett.addCustomParser
, passing
in the custom property name, nodeType for the custom property, parser function name, and parser function.
For example, if you wanted to add a custom normalized property to both meta and item objects called myProperty
, you
would write the following code.
FeedJett.addCustomParser('myProperty', ['item', 'meta'], 'parseMyProperty', function (node, nodeType, feedType) {
//custom logic for parsing and normalizing myProperty. For example if you wanted the text value of atom:id to be
//the value of myProperty, you would do the following.. ('#' are for text values, @ for attribute objects)
return node['atom:id'] && node['atom:id']['#'];
});
var feedjett = FeedJett.createInstance(); //... other feedjett logic as shown in earlier section
Now when you listen to readable for items or the meta event for meta, the result object will include myProperty
which
would have a value that equals the return value of parseMyProperty
. Behind the scenes, addCustomParser will add the
property to the normalizedProps meta or item Arrays, then add the myProperty -> parseMyProperty
mapping to the
propertyParserMap
object, then dynamically invoke the custom function and set the property on the result object.
You can also include functions that can hook into the parsing framework. The following functions can be defined on
the options
object to achieve this
afterParseItem
- {Function} - If passed in through options, this function will be invoked AFTER all property
parsing functions have been invoked for an item
and the item
is normalized, and after the meta
object is
attached to the item
. (Assuming both parseMeta
and addMeta
are both true.
afterParseMeta
- {Function} - If passed in through options, this function will be invoked AFTER all property
parsing functions have been invoked for a meta
and the meta
is normalized. (Assuming parseMeta
is true).
This will be invoked BEFORE the meta
event has been emitted.
The following example illustrates how you can hook into the parsing framework AFTER an item or meta is parsed and
define any additional logic. The raw node (item
or meta
) and feedType
(rss, atom or rdf) will be available
as parameters. No return value is needed as you can add properties on the meta or item object.
var options = {
afterParseItem: function (item, feedType) {
//custom logic.. set link to guid if the parsing functions couldn't normalize link
if (!item.link) {
item.link = item.guid;
}
},
afterParseMeta: function (meta, feedType) {
//custom logic.. set updatedDate to pubDate if the parsing functions couldn't normalize updatedDate
if (!meta.updatedDate) {
meta.updatedDate = meta.pubDate;
}
}
};
var feedjett = FeedJett.createInstance(options); //... other feedjett logic as shown in earlier section
isMetaValid
- {Function} - If passed in through options, this function will be invoked AFTER all parsing is
complete for meta (including afterParseMeta
). This is the function that determines whether or not a meta object
is valid and can be emitted from FeedJett. The default isMetaValid
function just returns true, but you can
provide your own implementation to override the default behavior.
isItemValid
- {Function} - If passed in through options, this function will be invoked AFTER all parsing is
complete for an item (including afterParseItem
). This is the function that determines whether or not an item
object is valid and can be pushed (and emitted in readable) from FeedJett. The default isItemValid
function
returns true, but you can provide your own implementation to override the default behavior.
The following example illustrates how you can override the default behavior to determine whether or not an item or
meta is valid. The raw node (item
or meta
) and feedType
(rss, atom or rdf) will be available as parameters.
A boolean value true
or false
must be returned
var options = {
isMetaValid: function (meta, feedType) {
//custom logic.. mark meta valid and emit meta only if feedType is rss..
return feedType === 'rss';
},
isItemValid: function (item, feedType) {
//custom logic.. mark an item as valid only if it contains a pubDate
return !!item.pubDate;
}
};
var feedjett = FeedJett.createInstance(options); //... other feedjett logic as shown in earlier section
FeedJett comes equipped with a variety of utility functions for parsing feeds. To reuse a utility function in your custom parser(s), get an instance of the utils module as shown below
var utils = FeedJett.utils;
var options = {
parseAuthor: function (node, nodeType, feedType) {
return utils.getFirstFoundPropValue(node, ['author', 'dc:creator', 'itunes:author', 'dc:publisher']);
}
};
var feedjett = FeedJett.createInstance(options); //... other feedjett logic as shown in earlier section
See the demo
directory.
Like FeedParser, FeedJett is a transform stream operating in "object mode": XML in -> Javascript objects out. Each readable chunk is an object representing an item (article) in the feed.
meta
- called with feed meta
when it has been parsedreadable
- called when an item has been parsedend
- called when a feed has finished parsingerror
- called with error
whenever there is a FeedJett error of any kind (SAXError, FeedJett error, etc.)FeedJett parses each feed into a meta
(emitted on the meta
event) portion and one or more item
(emitted on the data
event or readable after the readable
is emitted).
Regardless of the format of the feed, the meta
and each item
contain a uniform set of generic properties
patterned after (although not identical to) the RSS 2.0 format, and optionally all of the properties originally
contained in the feed. So, for example, an Atom feed may have a meta.description
property, but it could also have
a meta['atom:subtitle']
property.
The purpose of the generic properties is to provide the user a uniform interface for accessing a feed's information
without needing to know the feed's format (i.e., RSS versus Atom) or having to worry about handling the differences
between the formats. However, the original information is configurable in case you need it. In addition, FeedJett
supports some popular namespace extensions (or portions of them), such as portions of the itunes
, media
,
feedburner
and pheedo
extensions. So, for example, if a feed article contains either an itunes:image
or
media:thumbnail
, the url for that image will be contained in the article's image
property.
Unlike FeedParser, FeedJett does NOT pre-initialize generic properties and keeps result objects as thin as possible. Additionally, result object properties are camelCased. In contrast, the original raw node will contain lowercase property names (which is how sax parses the nodes). So for example if you are defining a custom function to parse the pubDate normalized prop, you look for node.pubdate in the Raw node, but the normalized object will have the property item.pubDate.
The title
and description
properties of meta
and the title
property of each item
have any HTML stripped
if you let FeedJett normalize the output. If you really need the HTML in those elements, there are always
the originals by passing in parseRawNode: true in options and accessing the original property in the meta.raw object.
e.g., meta['atom:subtitle']['#']
.
NOTE:
#ns - {Array} An array of namespace Strings for the feed
@' - {Object} Contains any attributes of the top level tag (feed, rss, etc)
#xml - {Object} The parsed xml tag and/or attributes
#type - {String} The type of feed (rss, atom, or rdf)
raw
property that contains all raw nodes from the feedorigLink
contains the original linkurl
, type
, and/or length
representing a podcast or enclosure.url
, title
; The original source for an article; see the RSS Spec for an explanation of this element)NOTE:
meta
property that includes meta properties.raw
property that contains all raw nodes from the feed