bleve

Emergency release to try and fix issues caused by the repository rename of one of our dependencies: #1592

Bug fixes:

• Update repository name #1592
• Switch to newer vellum also addressing package rename issue #1596
• Remove redundant code #1582

BROKEN: this release still contains dependency issues

Bug Fixes

• Fix race condition in moss wrapper (https://github.com/blevesearch/bleve/pull/1558)

Enhancements

• Scorch supports new config setting for bolt_timeout (https://github.com/blevesearch/bleve/pull/1553)
• Bleve now uses vellum repository in the blevesearch organization (https://github.com/blevesearch/bleve/pull/1556)

Bug Fixes

• Fix query memory estimation (https://github.com/blevesearch/bleve/pull/1554)

Enhancements

• Export an Analyze(..) API for scorch and upsidedown. While the API signatures are slightly different, they will allow users to analyze a document per the index's mapping (https://github.com/blevesearch/bleve/pull/1540)
• Additional unit tests, formatting fixes and README updates.

This release is a new major version because it contains breaking changes to the package API. For complete details of the contents and reasoning behind these changes, please see: https://github.com/blevesearch/bleve/issues/1495

Highlights

• Remove circular dependency between Bleve and Zap modules
• Make Scorch and Zap v15 the default index/segment type when using the New() method
• New option to disable freq/norm information for a field
• Types corrected for MatchQueryOperatorOr and MatchQueryOperatorAnd (see #1410)

Deprecated features (may be removed in the future)

• upsidedown index format and all key/value adapters
• HTTP sub-package
• bleve command-line tool sub-commands: bulk, create, index, dump
• config sub-package

This version is identical to v1.0.13 except that the go.mod now refers to a tagged release (v1.0.0) of the blevex module.

This was necessary as blevex master will soon be evolving to support the planned release of Bleve 2.0.0.

Enhancements

• Support for zap v15.0.2 a file-format compatible change which improves performance (https://github.com/blevesearch/zap/pull/44)

Bug Fixes

• Fix analyzer lookup during search when the field name contains the . character (https://github.com/blevesearch/bleve/pull/1496)
• Remove duplicated text in the help output of the command-line tool (https://github.com/blevesearch/bleve/pull/1489)
• Fix an inconsistency between an index mapping's default value for DocValuesDynamic and the omitempty JSON struct tag (https://github.com/blevesearch/bleve/pull/1485)

Enhancements

• Support for new zap v15 file format (significant space savings) https://github.com/blevesearch/zap/pull/27
• Make it possible to shutdown the analysis queue freeing goroutines https://github.com/blevesearch/bleve/pull/1414
• New stat reporting number of deleted items and estimate of space used by deleted items https://github.com/blevesearch/bleve/pull/1470

Bug Fixes

• HTML highlighter now performs escaping prior to output (SECURITY) https://github.com/blevesearch/bleve/pull/1465
• Fix crash in ASCII folding character filter https://github.com/blevesearch/bleve/pull/1434
• Fix bleve not closing index when encountering an error after open, but before it is returned https://github.com/blevesearch/bleve/pull/1479
• Fix test issue with Go v1.15 https://github.com/blevesearch/bleve/pull/1466
• Fix a test which was no longer testing the intended behavior https://github.com/blevesearch/bleve/issues/1458

Enhancements

• Geo search compute range code cleanup #1447
• Remove disjunction unadorned avoidance heuristic #1446
• Remove the tooManyClauses limitation when an unadorned disjunction optimization is possible #1426
• Improve performance of scorch internal event handling by using atomic ops instead of mutex #1419
• Improve error message for tooManyClauses to report the field name #1413
• Allow advanced users to alter the sort function implementation (only used in MultiSearch or searchBefore) #1400

Bug Fixes

• Update roaring bitmaps and bbolt, previous versions possibly have "unsafe" issues with newer versions of Go #1422
• Update to latest vellum, fixes performance issue in corner case see https://github.com/couchbase/vellum/issues/32
• Fix ineffectual assignment in merge planner options #1450
• Improve sort mode auto heuristic for detecting numeric terms #1435
• Fix error handling in numeric range searcher #1445
• Fix memory leak when performing unadorned conjunction/disjunction optimization #1438
• Fix error handling in DocIDReader #1443
• Fix file handle leak for corner case of the merger #1417
• Fix accounting of TermFieldReader started/finished when the Reader is reset internally performing Advance backwards #1415

The first usable release of the v1.0.x line, versions v1.0.0 through v1.0.4 were all unusable due to release issues.

For full information about the v1.0.x release, see #1350

Bug Fixes

• Better avoiding panic during highlighting. See #1371
• Improved regexp replace character filter, now supports references. See #1351
• Updated to latest versions of zap moving to our fork of mmap-go (fixing file cleanup issue on Windows). See #1289

Enhancements

• New ForceMerge API available on scorch index. See #1393
• New Builder API introduced to allow write-only index building, into an optimized single segment. See #1282
• Add support for zap v13, addresses an index size regression introduced in zap v12.
• Changed MultiSearch comparison implementation, with possible performance improvement. See #1398

Bug Fixes

• Fix analyzer inheritance issue in Document Mapping. See #1391

Enhancements

• Add support for zap v14, reduced disk usage over previous versions.

Bug Fixes

• Fixed bug in parsing of forceSegmentVersion scorch config when configuration map was parsed from JSON (float64 instead of int). See #1401
• Fixed bug in the Builder which caused it to always use zap v11, regardless of override configuration being used. See #1406

Drafting a new release, primarily so that Go module users get a version of bleve that works with vellum master.

Many bug fixes, and the first official release in which we consider scorch production ready.

The default index type is still upsidedown.

Behavior Changes

• This is the last release of Bleve to support Go 1.5 and Go 1.6

Enhancements

• Pure Go Analyzers added for several languages added: Russian, Danish, Finnish, Hungarian, Dutch, Norwegian,
  Romanian, Swedish, Turkish
• New token filter UniqueTerm which removes duplicate terms from the token stream

Bug Fixes

• Many scorch improvements as it is still under heavy development

Behavior Changes

• Index Format changed to save space in the backindex (MUST REBUILD INDEX!!!)

Enhancements

• New experimental indexing scheme scorch
• Adjustments to Searchers to avoid moving Term Freq Reader backwards
• Numeric range queries filter against the term dictionary now
• Add support for BleveType() alternative for type detection
• Add support for mapping to recognize/use TextMarshaler interface
• Command-line query tool supports -sortby option
• New pure Go Spanish analyzer
• New pure Go German analyzer
• Topn collector switches approach (heap vs slice) based on size+skip
• Add IndexAdvanced() to allow direct indexing of documents without mapping
• Optimmize heap collector Final() for large counts
• New term range query
• New geo bounding box and point distance queries
• New geo point distance sorting
• ForestDB K/V store has been removed
• Reduce garbage created while processing facets
• Experimental index scheme smolder has been removed
• Improve query string compatibility with ES
• New multi phrase query
• Improve performance of regular expression and wildcard queries
• Many garbage and allocation improvements from Steve Yen

Bug Fixes

• Fix tests to properly close/remove temp indexes
• Fix race condition in TestIndexMetadataRaceBug198
• Fix mapping bug where closestDocMapping selecting wrong mapping
• Fix data race in doc id search
• Fix token start/end/position values in camelCase tokenizer
• Fix issue with numeric range queries in query string
• Fix nil ptr panic when using new text marshaler support
• Fix panic in term range search
• Fix geo point distance search
• Fix race condition in incorrectly shared state in MultiSearch
• Fix edge ngram output in some corner cases

NOTE: these release notes are not up to our normal stands due to waiting far too long between releases. We will attempt to release more frequently and more carefully annotate release notes.

Behavior Changes

• DumpAll(), DumpDoc(), DumpFields() methods removed (#429)
• New method to create in-memory indexes (#452)

Enhancements

• New experimental indexing scheme smolder
• All bleve utilities migrated to single bleve cmd (#430)
• Index upside_down performance optimizations
• Searcher performance optimizaitons

Bug Fixes

• Tokenizer regexp no longer produces empty tokens
• Shingle filter now stateless, was producing bogus output (#431)
• Fix panic in collector when skipping more than total hits (#453)

Behavior Changes

• POSSIBLE BREAKING CHANGE - Byte Array Converters removed - SEE #392
• Firestorm indexing scheme removed
• Build tags added around references to persistent storage, making it slightly more possible to use in Google App Engine

Enhancements

• Match query supports changing default operator AND/OR
• Facet speedups
• UpsideDown indexing scheme query perf improvements
• Index API changes to make internal document identifiers opaque
• Ability to sort results by indexed fields
• Support read_only flag for boltdb indexes
• Improved unit test code coverage to 74%
• Removed nex query string lexer, replaced with custom lexer
• New query string lexer supports escaping reserved characters
• Minor text analysis perf tweaks

Bug Fixes

• Corrected Advance() method of regexp, prefix and fuzzy searchers #342
• Indexing of primitives fixed inside map/struct #389
• Moss handing of 0-length values
• Docs updated for date range querying #382

Behavior Changes

• Search result hits with the same score will now be returned sorted by document ID ascending, previously they were sorted by document ID descending. This behavior is not guaranteed, so applications should not rely on it.
• CJK analyzer now uses unicode tokenizer and cjk width filter, this enhances compatibility with Lucene/ES, though it changes behavior in some cases.
• Whitespace tokenizer now faster and behaves more like the one with the same name in Lucene/ES

Enhancements

• Multi-term searches now check the setting of DisjunctionMaxClauseCount earlier in the process, avoiding unnecessary CPU usage
• Simplified date parsing in queries, now always use top-level setting QueryDateTimeParser
• Query Strings now support date range queries
• SearchRequest now has a Validate() method which can report invalid facet requests
• Performance - loading stored fields and highlighting result snippets now only does a single lookup
• Performance - top score collector faster by doing less work for hits with scores that cannot be in the top N
• New token filter to handle camel cased input
• Standardized behavior of index mapping with anonymous struct fields

Bug Fixes

• Fix metrics stats serialization crash on Windows
• Fix JSON marshaling of MatchNone
• Properly initialize stats object for in-memory indexes
• Unregister index from expvar on Close()
• Configuring ngram filter more straightforward and robust