nmslib

Note: We unfortunately had deployment issues. As a result we had to delete several versions between 2.0.6 and 2.1.1. If you installed one of these versions, please, delete them and install a more recent version (>=2.1.1).

The current build focuses on:

1. Providing more efficient ("optimized") implementations for spaces: negdotprod , l1, linf.
2. Binaries for ARM 64 (aarch64).

Just providing Python 3.8 binaries.

The main objective of this release to provide binary wheels. For compatibility reasons, we need to stick to basic SSE2 instructions. However, when the Python library is being imported, it prints a message suggesting that a more efficient version can be installed from sources (and tells how to do this).

Furthermore, this release removes a lot of old code, which speeds up compilation by 70%:

1. Non-performing methods
2. Double-indices

This is a step towards more lightweight NMSLIB library.

#398 Fixing memory leak in loadIndex

This is a clean-up release focusing on several important issues:

1. Fixing a bug with knnQuery #370
2. Added a possibility to save/load data efficiently from the Python bindings (and the query server) #356 Python notebooks are updated accordingly
3. We have bit Jaccard space (many thanks @gregfriedland)
4. Upgraded the query server to use a recent Apache Thrift
5. Importantly the documentation is reorganized quite a bit:
   5.1 There is now a single entry point for all the docs
   5.2 Most of the docs are now online and only fairly technical description of search spaces and methods is in the PDF manual.

Hopefully this will fix the Windows build #348

An additional fix for #327

See https://github.com/nmslib/nmslib/issues/324#issuecomment-404524969

Resolving issue #327

1. Improving concurrency in Python (preventing hanging in a certain situation https://github.com/searchivarius/nmslib/issues/291)
2. Improving ParallelFor : passing thread ID and not starting threads in a single-thread mode.

This release mostly focuses on bug fixing and documentation improving.

Here are the list of changes for the version 1.6 (manual isn't updated yet):

We especially thank the following people for the fixes:

• Bileg Naidan (@bileg)
• Bob Poekert (@bobpoekert)
• @orgoro

1. We simplified the build by excluding the code that required 3rd party code from the core library. In other words, the core library does not have any 3rd party dependencies (not even boost). To build the full version of library you have to run cmake as follows: cmake . -DWITH_EXTRAS=1
2. It should now be possible to build on MAC.
3. We improve Python bindings (thanks to @bileg) and their installation process (thanks to @bobpoekert):
   1. We merged our generic and vector bindings into a single module. We upgraded to a more standard installation process via distutils. You can run: python setup.py build and then sudo python setup.py install.
   2. We improved our support for sparse spaces: you can pass data in the form of a numpy sparse array!
   3. There are now batch multi-threaded querying and addition of data.
   4. addDataPoint* functions return a position of an inserted entry. This can be useful if you use function getDataPoint
   5. For examples of using Python API, please, see *.py files in the folder python_bindings.
   6. Note that to execute unit tests you need: python-numpy, python-scipy, and python-pandas.
4. Because we got rid of boost, we, unfortunately, do not support command-line options WITHOUT arguments. Instead, you have pass values 0 or 1.
5. However, the utility experiment (experiment.exe) now accepts the option recallOnly. If this option has argument 1, then the only effectiveness metric computed is recall. This is useful for evaluation of HNSW, because (for efficiency reasons) HNSW does not return proper distance values (e.g., for L2 it's a squared distance, not the original one). This makes it impossible to compute effectiveness metrics other than recall (returning wrong distance values would also lead to experiment terminating with an error message).
6. Additional spaces:
   1. negdotprod_sparse: negative inner (dot) product. This is a sparse space.
   2. querynorm_negdotprod_sparse: query-normalized inner (dot) product, which is the dot product divded by the query norm.
   3. renyi_diverg: Renyi divergence. It has the parameter alpha.
   4. ab_diverg: α-β-divergence. It has two parameters: alpha and beta.
7. Additional search methods:
   1. simple_invindx: A classical inverted index with a document-at-a-time processing (via a prirority queue). It doesn't have parameters, but works only with the sparse space negdotprod_sparse.
   2. falconn: we ported (created a wrapper for) a June 2016's version of FALCONN library.
      1. Unlike the original implementation, our wrapper works directly with sparse vector spaces as well as with dense vector spaces.
      2. However, our wrapper has to duplicate data twice: so this method is useful mostly as a benchmark.
      3. Our wrapper directly supports a data centering trick, which can boost performance sometimes.
      4. Most parameters (hash_family, cross_polytope, hyperplane, storage_hash_table, num_hash_bits, num_hash_tables, num_probes, num_rotations, seed, feature_hashing_dimension) merely map to FALCONN parameters.
      5. Setting additional parameters norm_data and center_data tells us to center and normalize data. Our implementation of the centering (which is done unfortunately before the hashing trick is applied) for sparse data is horribly inefficient, so we wouldn't recommend using it. Besides, it doesn't seem to improve results. Just in case, the number of sprase dimensions used for centering is controlled by the parameter max_sparse_dim_to_center.
      6. Our FALCONN wrapper would normally use the distance provided by NMSLIB, but you can force using FALCONN's distance function implementation by setting: use_falconn_dist to 1.

1. Releasing GIL to enable Python threading
2. A slightly faster VP-tree #52
3. New scalar-product spaces #110

Performance improvement.

This is a bugfix release to address issue #98

1. A new efficient method: a hierarchical (navigable) small-world graph (HNSW), contributed by Yury Malkov (@yurymalkov). Works with g++, Visual Studio, Intel Compiler, but doesn't work with Clang yet.
2. A query server, which can have clients in C++, Java, Python, and other languages supported by Apache Thrift
3. Python bindings for vector and non-vector spaces
4. Improved performance of two core methods SW-graph and NAPP
5. Better handling of the gold standard data in the benchmarking utility experiment
6. Updated API that permits search methods to serialize indices
7. Improved documentation (e.g., we added tuning guidelines for best methods)