gorse

Features

• Get more details from the recommendation (https://github.com/gorse-io/gorse/pull/796) by @ubrmnsh

Warning This is a pre-release version of the Gorse recommender system. It means:

1. Don't use the pre-release version in production unless you can afford risks.
2. APIs, configurations, and data storage might be changed between pre-release versions.
3. Clear local caches and cache databases after the upgrade.

Features

• Support JSON labels (#647).
• Remove recommendation cache for inactive users (#691).
• Support multiple categories filtering (#704).

BREAK CHANGES

• Local caches are stored in a folder (#640).
• Remove Oracle database support (#641).
• Remove Redis cluster support (#645).
• Remove ClickHouse support (#648).

Fix

• Fix the concurrent access problem of rand.Rand (#694) by @WisperDin.
• Fix concurrent map iteration and map write in the worker (#697).
• Fix index out of range caused by duplicate labels (#700).

Features

• Support log files rotation (#659).

Fix

• Fix Google fonts loading from the mainland (#630).
• Fix authorization failure in gorse-in-one (#634).
• Support redacted config in the dashboard (#635).

Go SDK

• Support modifying users and items (#636) by @DDefiebre.

Features

• Support YAML/JSON configuration file format (#604).
• Encode cursor by base64 (#612).
• Support Docker Windows container images (#618).

Fix

• Fix CORS error with API key by @NMathar (#598).
• Fix worker nil pointer panic in refreshCache (#611).
• Fix batch import feedback (#620).
• Support Windows log file sink (#621).

Features

• Support health checks for liveness and readiness (#594).
• Add cache_table_prefix and data_table_prefix option (#593).
• Support excluding "read" items from popular/latest recommendations (#595).

Fix

• Fix boolean type parsing of ClickHouse driver (#596).

Features

• Support managed mode: schedule tasks manually (#575).
• Support OpenTelemetry tracing (#576).
• Support compressed playground dataset (#584).

Fix

• Fix purge cache on Redis (#571).
• Fix timestamp conditions on loading feedback from MongoDB (#571).
• Apply clock_error on loading feedback(#573).
• Fix "index out of range [-1]" on building IVF index for zero IDF (#585).

Features

• Support displaying scores of users/items neighbors in the dashboard (#557).
• Support deleting users and items in the dashboard (#558).
• Support Redis cluster as cache storage (#564).

Fix

• Fix RESTful API document (#566).

Features

• Support AVX512 SIMD instruments (#533).
• Support purge all data and cache (#549).
• Implement Go SDK in client package (#522).
• Add CORS support to RESTul APIs by @NMathar (#550).

Fix

• Fix updating item visibility and categories (#552).
• Fix the deadlock problem of the scheduler (#554).

Performance

• Remove interface call in SIMD dispatch (#545).

Features

• Support dynamic goroutine allocation for multiple tasks (#513).
• Support database prefixes postgresql:// and rediss:// (#499).
• Support disable/enable model size searching (#502).
• Support prefixes for database table names (#503).
• Build windows arm64 release (#500).
• Support HTTPS for ClickHouse connection: chhttps:// (#508).
• Prune solely user labels and item labels (#512).

Fix

• Fix unexpected database schema migration (#501).
• Fix duplicate measurements (#505).
• Stop the HTTP server and RPC server gracefully (#506).

Features

• Implement a playground based on the all-in-one server (#479).
• Support Oracle as one of the cache stores (#482).
• Support Oracle as one of the data stores (#484).

Fix

• Remove tasks from disconnected worker nodes (#480).
• Show progress for index building (#490).
• Fix the isolation level (READ UNCOMMITTED) in MySQL (#481).
• Fix the positive feedback display in the dashboard (#486, by @bradrisse).

Features

• Support SQLite as one of the data stores (all-in-one server only, #469).
• Support SQLite as one of the cache stores (all-in-one server only, #467).
• Implement the all-in-one server (#470).

Fix

• Fix delete users and items (#471).
• Fix import feedback CSV with quoted field (#472).

Performance

• Remove redundant cache deletion (#475).

Features

• Track the memory usage of the master node and worker nodes (#461).
• Calculate positive feedback rates during load dataset (#462).

Fix

• Report metrics (precision, recall, etc.) once saved models are loaded (#457).
• Redact username and password in DSN (#459).
• Fix migration error with PostgresSQL and ClickHouse (#464).
• Fix insert duplicate users/items/feedback (#465).

Features

• Display binary versions in the dashboard (#448).
• Implement the request ID for RESTful APIs (#449).
• Move measurements from the data store to the cache (#450).
• Support database auto migrate by gorm (#453).
• Reimplement auto similarity as hybrid similarity based on labels and users/items (#455).

Performance

• Set MySQL and Postgres isolation level to READ UNCOMMITTED (#452).

Features

• Support session-based recommendation (#430).
• Invalidate cache after the configuration change (#436).
• Support cache garbage collection (#437).
• Export metrics for new Gorse Grafana dashboard (#445).

Fix

• Fix hidden items and deleted items visibility (#442).

Performance

• Cache popular items in the server node (#438).
• Cache hidden items in the server node (#439).

Features

• Support MongoDB as cache_store (#406).
• Support MySQL as cache_store (#415).
• Support PostgresSQL as cache_store (#414).

Fix

• Fix typos in Swagger spec (#405) by @europaer.
• Unify timezone in ClickHouse (#417).
• Fix inserting duplicate cache rows in Postgres (#417).
• Fix item timestamp modification (#417).

Performance

• Optimize inserting users/items/feedbacks (#411).

BREAK CHANGES

• New config file specification and default values are introduced in v0.4.0 (#418).
• The write-back-delay parameter of GET /api/recommend/{user-id} requires time unit now. For example:

curl -X GET "http://172.18.0.3:8087/api/recommend/zhenghaoz?write-back-type=read&write-back-delay=10m&n=10" \
    -H "accept: application/json" \
    -H "X-API-Key: 19260817"

Features

• Support set check recommend period (#384).
• Support replacement in the recommendation (#385).
• Support DNS SRV record lookup for MongoDB (#386, authorized by @pranav377).
• Support set clock error in the cluster (#397).

Fix

• Import and export items with hidden flags and categories (#383).
• Hide hidden items in the latest items, popular items, and item neighbors (#393).
• Fix inaccurate representation of timestamps in float32 (#394).
• Fix potential Redis memory leak (#395).

Upgrade Guide

• Configuration: Replacement has been supported in the configuration, as well as check_recommend_period and epsilon_time.

[server]
# ...
epsilon_time = 5                # clock error in the cluster in seconds

[recommend]
# ...
# The time period to check recommendation for users (minutes). The default values is 1.
check_recommend_period = 1

# Replace historical items back to recommendations.
enable_replacement = false

# Decay the weights of replaced items from positive feedbacks.
positive_replacement_decay = 0.8

# Decay the weights of replaced items from read feedbacks.
read_replacement_decay = 0.6

• Redis: Remove incompatible stale cache.

redis-cli KEYS "offline_recommend*" | xargs redis-cli DEL

Feature

• Support login authorization in Gorse dashboard (#376).
• Support AVX2 for vector computation (#372).
• Support Arm Neon for vector computation (#374).
• Support feedback type-level "click-through rate" (#379).

Fix

• Fix recommendations to untrained users (#380).
• Create folders for log files and handle ignored errors (#378).
• Return zero recall on empty ground truth (#381).

Upgrade Guide

• Configuration: Set username and password for Gorse dashboard. If both dashboard_user_name and dashboard_password are empty, login authorization will be skipped.

dashboard_user_name = "admin"   # dashboard user name
dashboard_password = "password" # dashboard password

• Redis: Remove incompatible stale cache.

redis-cli KEYS "item_neighbors*" | xargs redis-cli DEL

Feature

• Support modify configuration via environment variables (#359).
  There are 8 environment variables available:

• (Experimental) Support IVF-based neighborhood searching (#363).
• (Experimental) Support HNSW based recommended items searching (#368).

Fix

• Split large Redis writes to batches (#352).
• Allow empty JSON string in relational databases (#355).
• Fix offset overflow in recommend API (#365).
• Optimize dashboard overview page layout (#369).

Upgrade Guide

• Configuration: IVF-based neighborhood searching and HNSW based recommended items searching are experimental features.
  • Use enable_xxx_index to enable experimental features.
  • xxx_index_recall is the expected recall of approximate searching.
  • xxx_index_fit_epoch is the number of epochs to adapt indices.

These approximate vector searching indices trade tolerable accuracy (recall) with larger throughput. The index builder tries to reach xxx_index_recall in xxx_index_fit_epoch. The building process will stop when xxx_index_recall or xxx_index_fit_epoch reached. Index-based searching costs more memory than brute force searching and its building process costs additional time, but they are negligible compared to the benefits.

# Enable approximate item neighbor searching using vector index.
enable_item_neighbor_index = false

# Minimal recall for approximate item neighbor searching.
item_neighbor_index_recall = 0.8

# Maximal number of fit epochs for approximate item neighbor searching vector index.
item_neighbor_index_fit_epoch = 3

# Enable approximate user neighbor searching using vector index.
enable_user_neighbor_index = false

# Minimal recall for approximate user neighbor searching.
user_neighbor_index_recall = 0.8

# Maximal number of fit epochs for approximate user neighbor searching vector index.
user_neighbor_index_fit_epoch = 3

# Enable approximate collaborative filtering recommend using vector index.
enable_collaborative_index = false

# Minimal recall for approximate collaborative filtering recommend.
collaborative_index_recall = 0.9

# Maximal number of fit epochs for approximate collaborative filtering recommend vector index.
collaborative_index_fit_epoch = 3

• Redis: Remove incompatible stale cache.

redis-cli KEYS "item_neighbors*" | xargs redis-cli DEL
redis-cli KEYS "user_neighbors*" | xargs redis-cli DEL