vearch

Added

• add go sdk
• add python sdk
• add openapi
• add trace option for document interface

Removed

• Remove some unnecessary logic in document query

Changed

• alloc FieldOperate in stack to reduce new/delete
• merge io operator to raw_vector
• use StorageManager in RawVector
• chore: use unconditional strings.TrimSuffix
• Using reference counting instead of deferred release

Fixed

• check index MetricType is empty
• delete data directionary after engine cloesed
• wait for engine closure before deleting the data directory

New Contributors

• @testwill made their first contribution in https://github.com/vearch/vearch/pull/783

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.0...v3.5.1

Added

• add rocksdb bitmap manager
• add engine search and createtable error msg in response
• add data type stringArray
• ivfflat support brute force search
• Add status to manage function return values
• add alias interface

Removed

• remove useless error code
• remove boost in search request
• remove OnlineLogLevel
• remove MSearch
• remove response field _source
• remove document vector field feature in query response
• remove support for search condition of document_ids
• remove upsert vector field.feature
• remove result field.extra and table.compress_mode
• remove support for indivisible nsubvector of index ivfpq
• remove id_type

Changed

• unify error return format
• reduce memory copy in table Serialize
• return status in CreateVectorTable
• separate query from search method
• Classify the HTTP return values of Vearch as standard HTTP status codes
• use recall_num instead of quick
• rename search size to limit
• reformat query/filters
• Unify return data format of document interface
• Unify return data format of cluster version interface
• Unify the format of meta define interface
• rename index_size to training_threshold
• set default value for index_params to simplify space schema
• update space operation interface
• deal with log error without directly exit

Fixed

• call free only when an error occurs
• fix request auth

Added

• support query next undeleted document with a docid value
• add retry for listen pprof port
• add interface of space/describe
• add vearch_backup tool
• add new version python sdk struct

Removed

• remove retrieval_types

Changed

• switch the router http framework to gin
• simplify error messages
• change FieldType in space to vearchpb.FieldType
• change some error return for document interface
• replace cbjson builder by sonic
• replace ioutil by os

Fixed

• fix deleteByQuery nil result panic
• fix GetMemoryInfo crash when deleting docs

Added

• add go sdk demo

Removed

Changed

• check vector field and document_id for document/upsert

Fixed

• fix multi filter search error, fix field range index boundary error
• fix multiple vector search
• fix override warning
• fix macOS compile error

Added

• add option to show hnsw search information
• add metric "vearch_disk_stat"
• add interface for index/flush and index/forcemerge
• add move_member interface to move members
• support for modifying a single field
• add option to describe index information for interface of index/rebuild
• Add "space_name" to the engine to distinguish logs from different spaces
• add interface of index/rebuild
• add devtoolset-10

Removed

• remove return value of source and set return fields in unified way
• remove field source
• remove storage type "Mmap"
• remove zstd
• remove zfp

Changed

• Change the storage type of the table to RocksDB
• print "meminfo" and "UnaryHandler" only when the LogInfoPrintSwitch switch is turned on
• check store_type and retrival_type for index
• change indexed str limit to 1024 in router
• unified rocksdb version

Fixed

• fix load of ivfpq index
• fix contex leak and reduce debug.Stack() calls
• fix bug when create vector table failed

Added

• add log for panic infomation
• add load_balance
• add option for check_restart
• add option to show detail info or not for cluster/health interface
• add resource check to prevent crash

Removed

Changed

• extend STR_MAX_INDEX_LEN from 255 to 1024

Fixed

• modify field_range_index delete fail log level
• Some index optimizations
• fix context leak
• limit field_range_index_ size to prevent large memory alloc on loading

Added

• Add option of cluster_state in config
• Add node to faulty list on rpc error
• Add interface of document/query
• Add interface of document/upsert
• Add interface of document/search
• Add limit for document interface
• Add config option for replica_num

Removed

Changed

• Redefine data interfaces
• Make rocksdb the default component
• Document/query support to get document on specify partition
• Unify the return format of document/delete interface
• Simplify the return data of document/upsert
• Unify the return format of document/[query|upsert] interface
• Update langchain plugins
• Update etcd version to v3.5.6
• Change etcd InitialClusterToken to cluster name
• Change item_to_docid_ storage to rocksdb

Fixed

• Reformat codes and clear some warnnings
• Fix etcd import address

Added

• Add option for limited db num to one (https://github.com/vearch/vearch/commit/5756df13193d1661a316fb03dcaa46bdf4e0ee76) (https://github.com/vearch/vearch/commit/07f4e91dca4c802ee7ae6b6e8388bad59591e2c2)
• Add docker-image.yml (https://github.com/vearch/vearch/commit/5bb07da9aa2b5c7cc6ca3a6133845d5306c7a778)
• Add Authorization for interface of router (https://github.com/vearch/vearch/commit/7679e61cbd004d9edab10e94ad011f68914f577e)

Removed

Changed

• Surport domain_name for etcd (https://github.com/vearch/vearch/commit/f0c1e2b995d54b192534f4a22bd65875a95e5fa5)
• Update process property array log (https://github.com/vearch/vearch/commit/b017ae3e5d0c7d49b4c8437d93b61bacdd605938)
• Add the _id return value of the get interface (https://github.com/vearch/vearch/commit/e1dcd76ce279e3bbc1cb0acec0d0ba409515d683)

Fixed

Added

• Add option for limited db num to one (https://github.com/vearch/vearch/commit/5756df13193d1661a316fb03dcaa46bdf4e0ee76) (https://github.com/vearch/vearch/commit/07f4e91dca4c802ee7ae6b6e8388bad59591e2c2)
• Add docker-image.yml (https://github.com/vearch/vearch/commit/5bb07da9aa2b5c7cc6ca3a6133845d5306c7a778)
• Add Authorization for interface of router (https://github.com/vearch/vearch/commit/7679e61cbd004d9edab10e94ad011f68914f577e)

Removed

Changed

• Surport domain_name for etcd](https://github.com/vearch/vearch/commit/f0c1e2b995d54b192534f4a22bd65875a95e5fa5)
• Update process property array log (https://github.com/vearch/vearch/commit/b017ae3e5d0c7d49b4c8437d93b61bacdd605938)
• Add the _id return value of the get interface (https://github.com/vearch/vearch/commit/e1dcd76ce279e3bbc1cb0acec0d0ba409515d683)

Fixed

Added

• Add proxy master request in router (https://github.com/vearch/vearch/commit/07f4e91dca4c802ee7ae6b6e8388bad59591e2c2)
• Add GetMemoryInfo (https://github.com/vearch/vearch/commit/bd035ab26d6c138e8cf98c4fe36068effd2dd3ab)
• Add prefix for etcd key (https://github.com/vearch/vearch/commit/c60fdb3cb785953ed9d48126a57202bacf6a6a75)

Removed

• Remove BuildFieldIndex (https://github.com/vearch/vearch/commit/b5b00e005b1408292b81e879881219191ef5ef1d)

Changed

• Filter error items in handlerQueryDocByIdsFeature(https://github.com/vearch/vearch/commit/ce2b3d7941ba9a64c7eebcc5c617f5b5aa5345ff)
• Show handler cost time (https://github.com/vearch/vearch/commit/d611261f7ea178c389533eab81f4e43974241561)

Fixed

• Fix engine close MultiFieldsRangeIndex::FieldOperateWorker crash (https://github.com/vearch/vearch/commit/6dc2dd8ca38034cd9872015be13698e60c3fa4b1)
• Fix etcd version (https://github.com/vearch/vearch/commit/20502adc66bbf5b7c8f03e6988c654e0133090a9)

Added

• Add support for langchain (https://github.com/vearch/vearch/commit/a1b2d02e7613a3c3ebf61d434eea0aa91088d274)

Changed

• Set some default value for GammaVectorInfo in python sdk (https://github.com/vearch/vearch/commit/16792a008ffe643b3c819c5c8fe4715fe5d4e102)

Fixed

• Fix float32 filter error (https://github.com/vearch/vearch/commit/d8a6c261da7137a6243c6f5200c878a08894f67b)
• Fix schema index nil panic (https://github.com/vearch/vearch/commit/176fab90e8dcd8d1e9db2c4f7214f27989dddbc1)
• Fix python sdk compatibility (https://github.com/vearch/vearch/commit/b550395791b141fc984b178d940670429a38d21b)
• Fix HasBoost (https://github.com/vearch/vearch/commit/07b98c4f8bac5d7d13dff9f0af0a1d244bcf0ee8)

Added

• Filter maxStrLen from router (https://github.com/vearch/vearch/commit/e5bc5634e9c0d456f5807b88ca23e2f25d3fa8f3)
• Add interface to get doc through docid in the specified partition (https://github.com/vearch/vearch/commit/f1e3ce235bf5128dd9aff4491c179d18de27147e)

Removed

• Remove gammacb/lib (https://github.com/vearch/vearch/commit/6d13d227f0fb803ea4629ceb45985fc243efb972)

Changed

• Simplify python sdk function "create_table" (https://github.com/vearch/vearch/commit/07a83795c8fdabc049aaffe0220dbcac161fc62b)
• Load PS space meta info from master (https://github.com/vearch/vearch/commit/504e0177e205f75551714542fdafd299c18cb7a1)

Fixed

• Accelerate PS restart speed in place (https://github.com/vearch/vearch/commit/56fb09ebadd70c3fd65e45bfd713581a5450cdc1)
• Fix python sdk load err (https://github.com/vearch/vearch/commit/911c464f3ad91cd1186b81d35abcd36b98f58f3e)
• Check nprobe for ivfflat (https://github.com/vearch/vearch/commit/c47c6a5ca75c1b373567296fbe6ebbe75c43774a)
• Check update/delete id for hnsw (https://github.com/vearch/vearch/commit/553b06c84371a7f5987d1687e6dfb2de67d71180)
• Fix create table error which may cause ps crash (https://github.com/vearch/vearch/commit/952fb14ca375f65b000c51131f5a2647eeb6b618)

Added

• Add python demos
• Add more info in cluster/health and cluster/stats
• Add python sdk
• Add ps faulty list

Removed

• Delete_by_query no longer supports containing vectors

Changed

• Use rocksdb instead of mmap to persist memory raw vector

Fixed

• Fix raft error
• Fix delete_by_query
• Fix possible int overflow when vector dimension is large
• Fix null pointer when the search result is null
• Fix illegal json error when id type is long and the value is out of bound

1.Add option for build script.
2.HNSW now support brute force search.
3.Classification of gamma log.

1. Fixed build script.
2. Fixed continuous integration script.
3. Improved the speed of range index deletion.
4. Added test cases.
5. Integrated gamma into vearch.

Improve the performance of Python sdk;
Add trace and timeout;
Support router and ps resource isolation by resource name;
Update vearch image search demo;
Search replica using round robin instead of random strategy;
Fix bugs of inserting and searching
Compatible with faiss v1.7.1

Overview: Gamma supports three storage types: RocksDB, MemoryOnly, and Mmap. v3.2.7 mainly modifies and optimizes Mmap. The specific modifications are as follows:

1.Use a unified way to store vectors and scalars;
2.Use cache to improve query performance;
3.Reduce scalar memory usage;
4.Supports dynamic modification of the cache size, and flexible storage and retraction of memory;
5.Fix the occasional inaccurate query of scalar index
6.Support raft snapshot

1、Master API move to router，master can replace by etcd server。 need open config ： master_merge_into_router。
2、Add search current limit function
3、Optimize loading logic
4、Replace faiss hnsw by hnswlib
5、Support the multiple retrieval models
6、Compatible with faiss 1.6.4
7、Update raft version
8、Some fix

1. fix compiling error without rocksdb in v3.2.0

1. The new interfaces of gamma engine retrieval model, ivfpq, ivfflat, ivfbinary and hnsw are all changed to the new interfaces.
2. Router, PS reconstruction: json changed to protobuf; Insert and search performance improved; Router added grpc interfaces.
3. Original vector compression, table string compression.
4. Master and etcd are separated, and the Master service supports automatic service discovery, query data node automatically shielded from accessing the failed node.