lmdb-js

V2.2.0 has enabled the overlappingSync functionality by default and improved the scheduling of sync operations to achieve better performance.

This release also introduces support for temporary databases. By simply omitting the path when opening a database, lmdb-js will auto-generate a temporary database (in the system temp directory), and delete it on close (it will also be opened with sync disabled for better performance). This can be useful for testing or creating intermediary tables.

The release also includes support for coordinating with the weak referencing cache to clear "kept" objects for better garbage collection in long-running synchronous jobs.

And v2.2.0 includes better performance for retrieving large binary entries by employing a reduced memory copying technique (affects entries over > 64KB).

The most significant new capability in this release is support for Deno. Deno functionality still has a number of limitations (no async transactions), but core functionality is now available. This release also includes new functionality for supporting asynchronous prefetching for gets, in situations where page faults/disk-I/O are likely. Also includes bug fixes for closing databases. Of particular note is that the close() method is now asynchronous (returns a promise).

Producer-Consumer writes

The most central change in v2.0 is a rewrite of how data is batched and written to LMDB, implementing a true multi-thread producer-consumer protocol such that all writes are directly and immediately serialized to a binary format that the writer thread can consume in real-time. This has multiple implications (and hopefully benefits):

• The writer thread can begin writing pending puts/deletes to LMDB (and async-txns can be queued) immediately after they have been called (in parallel to the main thread), rather than having to wait for a future event to submit the writes and start the transaction. And as more writes occur, they can continue to be queued and consumed by the writer thread as long as it is still processing a transaction. This should provide better write latency and flexibility in batching.
• This implementation improves memory management, allowing pending writes to written and be cleared/collected from memory without having to wait for transactions to complete and events to be queued, which should be more robust for heavy writing situations, preventing overloading of the queue of operations.
• All keys are immediately serialized using JavaScript. The (async) put/delete operations don't need to depend on a C++ for writing keys, so any errors in serializing keys are immediately triggered and since key (de)-serialization happens in JS, lmdb-js can now support custom (user-provided) key encodings (so for example, if you want a custom UUID encoder that can efficiently encode UUIDs in 16 bytes).
• Some additional put flags can now be supported (like "append" puts) (Expose MDB_APPEND #43) in performant way.
• The V8 "fast api" calls can be used for synchronous puts/deletes (used within async-txns).

Overlapping Sync

In lmdb-js, LMDB now supports "overlapping" syncs. This allows transactions to safely commit prior to be flushed to disk, and then the data can be flushed (and database metadata updated) after the transaction is visible and unlocked, allowing most of the disk I/O to happen in parallel with future write transactions, which can potentially provide some significant performance benefits in some heavy-write situations.

Other Performance Improvements

• A new binary key comparison function is used that does (32-bit) word by word comparison, instead of byte by byte comparison, for faster b-tree traversals.
• Compression is now performed in separate thread/tasks from the writes for more parallelism and speed.

Name Change

V.2+ is now published to https://www.npmjs.com/package/lmdb as its canonical NPM package. The 1.x line will continue to be available in the lmdb-store package. The repository has also been renamed to lmdb-js, to reflect the cross-platform goals of this project with Deno support coming soon (and significant internal upgrades in v2 to support this).

Other Upgrades

• The codebase has been upgraded to ESM (and refactored to be more modular), so lmdb-js can be used with native ESM loading (of course, built CJS modules are included too). This is also one of the necessary steps in hopefully supporting Deno (#83) at some point (once their foreign function interface is complete enough to be useable).
• Added functions for explicit batching.
• Improved compatibility with LevelUp.

Legacy Compatibility/Changes

There aren't any intentionally substantive breaking changes to the (documented) API itself, just cleanup and removal of previously deprecated APIs. But the underlying changes are large enough that testing should be done with an update.