Tokenizer/Lexer (Scanner) for JSDoc.
MIT License
Tokenizer (Scanner) for JSDoc. This project only operate on/with Node.js Buffer and Uint8Array (No conversion to String required).
This package is available in the Node Package Repository and can be easily installed with npm or yarn.
$ npm i jsdoc-tokenizer
# or
$ yarn add jsdoc-tokenizer
const { scan, TOKENS } = require("jsdoc-tokenizer");
const it = scan(Buffer.from("/** @type {String} **/"));
for (const [token, value] of it) {
if (value instanceof Uint8Array) {
console.log(token, String.fromCharCode(...value));
}
else {
const tValue = typeof value === "number" ? String.fromCharCode(value) : value;
console.log(token, tValue);
}
}
Scan (tokenize) JSDoc block. The scanner only take single instance of block (not build to detect start and end). To extract JSDoc block as buffer, please take a look at jsdoc-extractor.
Available tokens are described by the following interface:
interface Tokens {
KEYWORD: Symbol,
IDENTIFIER: Symbol,
SYMBOL: Symbol
}
Tokens are exported in the module.
@
that must be preceded by *\s
.MIT