xodus

This release brings new features (Bitmaps), API updates, the new database format, bug fixes and performance improvements.

Bitmaps

Two new interfaces has been added to the Environments API: Bitmap and ContextualBitmap. Bitmaps have unique names (like stores do), they let you transactionally set, get, clear and iterate over bits identified by index. Bitmaps are implemented atop of stores, so technically they aren't large arrays, rather they are similar to roaring bitmaps. They don't tend to have good data locality, but compaction is good — in particular for sparse bitmap, i.e. consisting mostly of ones or zeros.

API updates

• To trigger an action in the background cleaner thread just before database GC, Environment.executeBeforeGc(Runnable action) added.
• StoreTransaction.findContaining(..) returns an EntityIterable of entities of specified entity type with string property having a value containing specified substring. Implementation of the EntityIterable is much faster than brute force enumeration of entities followed by filtering by property value. Thanks to Hannes Halenka for the feature request.
• Added ability to set, add, and remove entity links based on an Entity ID.
• SignedFloatBinding and SignedDoubleBinding added.
• For boolean property values, there is no longer a distinction of the value 'false' and no value. This means that setting false as a property value is equal to calling deleteProperty(..). In previous versions, boolean properties could have three values: true, false, and null. This is the only potentially breaking change.

The new database format

As of 2.0.1, Patricia trees (backend trees for all stores created with key prefixing, i.e. with either StoreConfig.WITHOUT_DUPLICATES_WITH_PREFIXING or StoreConfig.WITH_DUPLICATES_WITH_PREFIXING) can be stored using the new format. Trees stored in the v2 format tend to be more compact with faster operations. For further insights, read a note on Xodus Version 2 Database Format and Patricia Tree Improvements.

By default, the v2 database format is turned off. To turn it on, use the EnvironmentConfig.setUseVersion1Format(boolean) method. Once an application configured to use the v2 format is run over a database, it can't be downgraded to a version using the v1 format without restoring the database from backup.

If the v2 format is turned on for databases accessed using the Entity Store API, lots of database refactorings and optimizations are automatically activated. Those include use of bitmaps for built-in indices, more compact in-place blobs and their periodic de-duplication.

Bug fixes

• Environments

  • XD-823 — Get rid of recursion on saving mutable Patricia tree
  • XD-825 — Get rid of recursion on reclaiming Patricia tree nodes by GC
  • XD-830 — On Java SE 9 or higher, FileChannel is interruptible and thus can be closed unexpectedly
  • XD-839 — Insufficient lock file permissions can result in mistaken detection of disk space exhaustion

• Entity Stores

  • XD-805 — Negative values of properties of type Float or Double break sorting, search by value and range search
  • XD-810 — Attempt to write too big property results in data corruption
  • XD-853 — Entity#setBlob() does not invalidate caches correctly

Release Notes
Maven Central

This release brings bug fixes and performance improvements related to database ciphering and memory usage control.

ChaCha20 implementation by JetBrains

The new implementation of ChaCha20, modern stream ciphering algorithm widely used nowadays, is written in Kotlin. Prior to this release, ChaCha20 had the only implementation provided by the Legion of the Bouncy Castle. The new implementation is identical to the Bouncy Castle's one in terms of ciphering, but consumes 13% less CPU:

Benchmark                                      Mode  Cnt    Score   Error   Units
JMHStreamCipherBenchmarks.chaChaCrypt         thrpt   12   97.439 ± 1.069  ops/us
JMHStreamCipherBenchmarks.jbChaChaCrypt       thrpt   12  110.342 ± 0.543  ops/us

The new implementation has been used already for two months in YouTrack InCloud servers, as well as in the main production instance of JetBrains YouTrack. So it's quite safe to just replace the the Bouncy Castle's implementation (cipherId:jetbrains.exodus.crypto.streamciphers.ChaChaStreamCipherProvider) by the new one (cipherId:jetbrains.exodus.crypto.streamciphers.JBChaChaStreamCipherProvider) and have less CPU consumption as a benefit. The Bouncy Castle's implementation won't be removed in future versions.

Log Cache can use Soft References

As of 1.3.232, it is possible to configure shared log cache to use Soft References (see EnvironmentConfig#setLogCacheUseSoftReferences). If this setting is turned on, Java GC would be able to reclaim some heap memory occupied by log cache. As before, log cache can't use more memory than certain memory control settings allow. Such memory-flexible log cache can be extremely useful during load peaks, if and when memory consumption and/or memory traffic increases explosively. The setting should be turned on explicitly, any defaults remain unchanged. Thanks to Martin Häusler for the proposal.

Bug fixes

• Environments

  • XD-793 — GarbageCollector can miss BTree leaf in certain case of data corruption
  • XD-802 — Forced close of Environment can result in deadlock under load

• Entity Stores

  • XD-801 — NumberOfCachingJobs in EntityStoreStatistics is always 0

Release Notes
Maven Central

This release brings bug fixes and minor performance improvements related to Entity Stores.

API clarification.

Starting from version 1.3.124, EntityStore.close() doesn't close underlying explicitly created Environment. Prior to 1.3.124, this contract was undefined.

Bug fixes

• Environments
  • XD-786 — Xodus OutOfMemory during environment.truncateStore() of stores with large (100k+) records in 1.3.x.
  • XD-787 — Xodus cannot open multiple entity stores with the same environment
  • XD-789 — Stuck transaction monitor fails to finish expired transaction created against ContextualEnvironment

Release Notes
Maven Central

This is bug fix update.

Bug fixes

• Environments
  • XD-770 — Update of existing BTree fails with exception if specified BTreeBalancePolicy's maximum page size is 4 times less than used before
  • XD-774 — removeStore() and openStore() in same transaction cause weird behaviour (thanks to Martin Häusler for reporting)
  • XD-778 — WatchingFileDataReader ignores file system events produced by rsync
  • XD-780 — Forced GC run doesn't estimate total utilization

Release Notes
Maven Central

This release offers Service Provider Interface for I/O customization in the package jetbrains.exodus.io. In addition to default disk-based I/O, it lets you configure your application to use an in-memory database, or to access in read-only mode disk-based database opened in another JVM. You can also create your own I/O provider, e.g., for storing data in a remote/cloud file system.

New versioning scheme

Since this release, versioning is changed in order to satisfy development requirements @Jetbrains. Patch numbers no longer will be successive. E.g., next released version definitely won't be 1.3.1. Apart from that, the new versioning scheme respects all requirements of Semantic Versioning.

Updates of dependencies

Kotlin 1.3.10.

Bug fixes

• Environments

  • #18 — File descriptors leaking issue fix (thanks to Valery Vlasov)
  • XD-728 — Log recovery procedure can false negatively diagnose the database is encrypted with unknown cipher parameters in case of recoverable data corruption
  • XD-733 — After an OutOfDiskSpaceException, recovery procedure can require application restart
  • XD-738 — Utilization profile can get invalid even if there were no cold restarts
  • XD-739 — Utilization computed from scratch can be highly inconsistent
  • XD-763 — Xodus corrupts data store upon no space left on device

• Entity Stores

  • XD-730 — FilterLinksIterable doesn't work with multiple links
  • XD-736 — EntityIterable handle for SortIterable ignores stability flag
  • XD-737 — Invalidation of cached FilterLinksIterable is broken if target EntityIterable is constructed as a query using links
  • XD-746 — Reversing iterable with entities may throw an exception
  • XD-749 — findLinks fails on empty EntityIterableBase

Performance improvements

• Environments
  • XD-507 — Detect automatically if utilization profile should be recalculated
  • XD-757 — StringBinding is notably slower than String#toByteArray
  • XD-762 — Xodus environment.close() can take a long time shortly after opening if Envionment.setGcUtilizationFromScratch(true)

Features

• Environments

  • XD-709 — Add SPI for log reader and writer customization

• Lucene Directory

  • XD-754 — Lucene codec compatible with Lucene70Codec with no compression of stored fields

Release Notes
Maven Central

Release 1.2.3 brings bug fixes and minor performance improvements.

As of 1.2.3, Xodus no longer uses NIO by default, i.e. EnvironmentConfig.DEFAULT.getLogCacheUseNio() returns false. This change can slightly affect performance of certain workloads, but it makes applications more stable and robust.

Bug fixes

• Environments
  • XD-629 — Transactions per second could get stuck on wildly incorrect value
  • XD-682 — Indeterminate behavior of Cursor opened against a transaction which is flushed then
  • XD-692 — Bad diagnostics of invalid cipher parameters
  • XD-697 — Illegal reflective access error when running on JRE 9
  • XD-698 — Unable to run an application in Docker under Windows
  • XD-704 — EnvironmentConfig.DEFAULT can be mutated
  • XD-705 — Cursor.close() is supposed to be idempotent but is not
  • XD-706 — Undocumented restriction: Can't create mutable tree in a thread different from the one which transaction was created in

Performance improvements

• Entity Stores
  • XD-680 — Introduce all_links_idx
  • XD-681 — FilterLinksIterable should use Cursors API instead of Store.get()

Release Notes
Maven Central

Release 1.2.2 brings more bug fixes mostly related to database encryption.

Bug fixes

• Environments

  • XD-678 — Scytale: create encrypted database in existing folder if it is empty

• Entity Stores

  • XD-679 — In an encrypted EntityStore, Entity.setBlob(String, java.io.InputStream) can fail if the stream is read before the transaction is flushed

• Lucene Directory

  • XD-677 — False DataCorruptionException from ExodusIndexInput

Release Notes
Maven Central

This is bug fix update. Only database encryption functionality is affected.

Bug fixes

• Environments

  • XD-676 — Blockwise read of StreamCipherInputStream results in corrupted output

• Entity Stores

  • XD-675 — In an encrypted EntityStore, Entity.setBlob(String, java.io.File) fails

Release Notes
Maven Central

This release offers database encryption. For details, refer to the documentation.

Updates of dependencies

Kotlin 1.2.21 is used.

Bug fixes

• Environments

  • XD-667 — Transaction leak in GC as a result of race with switching Environment to read-only mode
  • XD-673 — put returns false in case of overwrite

• Entity Stores

  • XD-669 — Entity.getLinks(linkNames) can return result with duplicates
  • XD-670 — For descending sorting, StableInMemorySortIterator returns entities with the null property value first

Performance improvements

• Virtual File Systems

  • XD-664 — VirtualFileSystem#getFileLength() enumerates all file clusters in case of linear clustering strategy

Release Notes
Maven Central

This release makes Xodus compliant with Java 9. Applications using Xodus would be able to run on JRE 9, though the project itself cannot be built under JDK9 due to the bug in Kotlin dokka.

Updates of dependencies

Kotlin 1.1.51 is used.

API changes

• Entity Stores

  • XD-612 — Implement execute/compute in exclusive transaction for persistent store level
  • XD-615 — Add method to determine whether StoreTransaction is finished

Bug fixes

• Environments

  • XD-614 — Cursor.getPrev() sometimes moves to wrong position in StoreConfig.WITHOUT_DUPLICATES_WITH_PREFIXING setup
  • XD-619 — For enough large Store, Cursor.getNext() and Cursor.getPrev() invalidate previously loaded key/value pair even though cursor position doesn't change
  • XD-625 — GC might delete data that is still used by opened transactions
  • XD-628 — An application can false positively report data corruption
  • XD-660 — Exceptions are logged when running on jre 9

• Entity Stores

  • XD-618 — Invalidation of cached instances of PropertyValueIterable and PropertyRangeIterable can fail for properties of the ComparableSet type
  • XD-626 — Attempt to cache an EntityIterable (queried over entities of a newly created type) result can fail

• Virtual File Systems

  • XD-624 — For enough large file, seek to the end of the file using VfsOutputStream fails

• Utils

  • XD-622 — An application using Xodus won't run on JDK 9 because of changes in java versioning scheme

Performance improvements

• Entity Stores

  • XD-609 — Implement O(M * (log(M) + log(N))) sorting option in case we are sorting an M-sized subset of larger N

Further reduction of memory traffic in implementation of the Environments API is done. Tokyo Cabinet benchmarks results updated. Benchmarks for LMDB JNI and Akiban PersistIt added.

Release Notes
Maven Central

This is bug fix update.

Bug fixes

• Environments

  • XD-606 — Files of a closed Environment are kept mapped by SharedMappedFilesCache
  • XD-608 — Cursor.getPrev() does not move cursor after call to Cursor.getSearchKey(..) with StoreConfig.WITHOUT_DUPLICATES_WITH_PREFIXING

• Entity Stores

  • XD-603 — PersistentEntityStore.registerCustomPropertyType() fails if invoked twice
  • XD-610 — Iterable handle.identity.hashCode() is not thread-safe

Release Notes
Maven Central

Release 1.0.5 brings lots of performance improvements in implementation of the Environments API (related mostly to reducing memory allocations) and in implementaion of the EntityStores API (more compact EntityIterableCache, its partial invalidation and updates are times faster).

Besides, 1.0.5 delivers fixes of several bugs. One of the bugs (XD-601) is pretty critical, thanks to Thorsten Schemm for reporting. The bug is old (it was introduced prior to 1.0.0), so we recommend to update unconditionally to 1.0.5.

Release Notes
Maven Central

This is bug fix update. One of the fixed bugs (XD-566) is pretty critical, it was introduced in 1.0.2.

Bug fixes

• Environments
  • XD-565 — If GC is deferred by the "exodus.gc.startIn" setting, it starts computing utilization from scratch immediately if the "exodus.gc.utilization.fromScratch" setting is on
• Entity Stores
  • XD-566 — Cached result on BinaryOperatorEntityIterable potentially containing entities of different types can be falsely left in the EntityIterableCache

Release Notes
Maven Central

This release brings minor (~5%) performance improvements of random access operations using Environments API. Tokyo Cabinet benchmarks reran, results updated.

Besides that, this version is aimed at current development of YouTrack and Hub.

Release Notes
Maven Central

This release is mostly devoted to performance on large workloads, when physical database size is much more than Java heap size.

Bug fixes

• Environments
  • XD-547 — Environment.clear() doesn't invalidate StoreGetCache and tree nodes cache
  • XD-551 — Race on setting custom job processor for background cleaner results in an exception
• Entity Stores
  • XD-553 — PersistentEntityStoreRefactorings.refactorMakePropTablesConsistent() detects wrong phantom values in indices of properties of the ComparableSet type
  • XD-555 — Custom property types don't persist
  • XD-557 — PersistentLinkedHashMap is inconsistent

What's new

• XD-533 — Benchmark of MVStore Map similar to Tokyo Cabinet benchmark
• XD-537 — Support for compressed negative long and integer values

Performance improvements

• XD-542 — Open files cache should be shared amongst all open Environments
• XD-543 — Map read-only (immutable) files into memory if the OS has enough free physical memory
• XD-545 — For enough large database, background cleaner can be inaccurate under load with lots of writes
• XD-546 — GC should be able to process several files within a single transaction if it is configured to be acquired as exclusive
• XD-549 — On large workloads, utilization profile holds excessively large objects on SoftReferences

...and more

Maven Central

Bug fixes

• Environments
  • XD-532 — File size leak with environment.truncateStore()
  • XD-538 — Under some circumstances GC can stop working with a NPE
  • XD-539 — A file can be created whose size is an integer multiple of required size
• Entity Stores
  • XD-536 — PersistentEntityStore.clear() doesn't invalidate its EntityIterableCache

Performance improvements

• XD-534 — SharedLogCache as probabilistic data structure with primitive type keys
• XODUS-CR-32 — Reducing Java GC pressure measured for BTree random read

Tokyo Cabinet benchmarks reran for Xodus BTree and Patricia. According to the benchmark results, BTree random read is 17% faster, BTree successive read is 34% faster, Patricia random read is 8% faster and Patricia successive read is 7% faster compared to 1.0.0.

Maven Central

Maven Central