redis-benchmarks-specification

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Include a DECR benchmark covering embstring object @filipecosta90 (#237)
• fix logging level @slice4e (#232)
• Aligned lhash* test-files' names with name inside file @markovamaria (#228)
• Bumping min python version to 3.8.1 and locking deps @filipecosta90 (#231)

What's Changed

• Enabled arch selection on self-contained-coordinator by @filipecosta90 in https://github.com/redis/redis-benchmarks-specification/pull/230

Full Changelog: https://github.com/redis/redis-benchmarks-specification/compare/v0.1.70...v0.1.71

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Fix for TypeError in self-coordinator @markovamaria (#224)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Enabled defaults metrics parsing and merging @filipecosta90 (#226)
• Fixed memory requirements on specific tests that could fail on Sapphire Rappids @slice4e (#225)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Added --maxmemory to client runner. @filipecosta90 (#221)
• Move getting commits by tags to separate function @markovamaria (#220)
• updated description for mutli-exec - for issue 212 @slice4e (#219)
• changed log level from error to info - fixing issue #84 @slice4e (#218)
• Added conversion for str for git timestamps @markovamaria (#216)
• Fixed memory requirements of 4 hash/list tests @slice4e (#215)
• Updated geo benchmarks to fully saturate one primary. Include zset benchmarks with long score @filipecosta90 (#214)
• Update timestamp conversion in client @markovamaria (#211)
• Fix mem reqs @slice4e (#213)
• Client runner: enabled oss cluster benchmarks for string (just set and get commands) benchmarks @filipecosta90 (#200)
• fixed memory requirements for load-list @slice4e (#207)
• fixed the load-stream bechmarks such that the memory footprint does not keep growing. @slice4e (#205)
• added recommendation to use taskset for the redis server when using t… @slice4e (#197)
• fix hincrby data population to correspond to the description. @slice4e (#196)
• added support for hexists @slice4e (#195)
• Memtier without docker @slice4e (#192)
• Fix memory reqs for load stream @slice4e (#202)
• Enabled --cluster-mode in client runner @filipecosta90 (#199)
• Added SADD test variations discussed in redis/pull/11595 @filipecosta90 (#198)
• modified Readme file to remove codenames and details which could chan… @slice4e (#193)
• Bumping version from 0.1.65 to 0.1.66 @filipecosta90 (#191)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Running tox using self hosted runner. maxmemory check fix on client-runner @filipecosta90 (#189)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Fixes tests after memtier password args @filipecosta90 (#187)
• Updated benchmark config to include resource memory usage. Checking valid memory settings upon CLI runner @filipecosta90 (#186)
• Updated dbconfig requests memory limit based uppon client runner checks @filipecosta90 (#188)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Updated the benchmark SPEC to include command priority. Included --dry-run and --tests-priority-upper-limit in redis-benchmarks-spec-client-runner @filipecosta90 (#184)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Added support for ZREVRANK, ZRANK, ZINCRBY and ZREM @slice4e (#182)
• Fixed stats checker @filipecosta90 (#181)
• Added HINCRBY, HMSET, MGET, ZSCORE, EVALSHA, GETBIT, EXISTS, EXPIRE, EXPIREAT, PEXPIRE, SCAN, TOUCH, TTL, and ZRANGEBYSCORE benchmarks @filipecosta90 (#177)
• Added execution and updated installation section to Readme @markovamaria (#175)
• Fix value bug @slice4e (#179)
• Added support for BITFIELD command. @slice4e (#178)
• Added command ZREVRANGEBYSCORE @slice4e (#174)
• Added support for MULTI EXEC @filipecosta90 (#176)
• Added FPADD command testcase. @slice4e (#172)
• Validated SPEC fields uppon PR and commit @filipecosta90 (#173)
• Avoid RDB usage on unit tests @filipecosta90 (#145)
• Fixed runner tests according to test cases @filipecosta90 (#170)
• Enable RESP 3 support in benchmark-tools @ofekshenawa (#169)
• Introduced --tests-regex flag to ease the process of running a subsetof tests @filipecosta90 (#168)
• Fix issue 130 @slice4e (#167)
• Fix bug 87 @slice4e (#142)
• Included ZCARD and HMGET tests @filipecosta90 (#144)
• Formatted all yml test-suite files @filipecosta90 (#143)
• Extended runner functional coverage @filipecosta90 (#138)
• Added GEOSEARCH BYBOX variation @filipecosta90 (#140)
• Enforcing keyspace length check and checking for datasets within scc @filipecosta90 (#136)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Updated README with visuals and making reference to Intel @filipecosta90 (#135)
• Updated README with visuals @filipecosta90 (#134)
• Fixed CLI commit date conversion issue @filipecosta90 (#133)
• Included memtier_benchmark-2keys-lua-eval-hset-expire test case for EVAL perf tracking @filipecosta90 (#127)
• Fixing CI integration @filipecosta90 (#126)

Tooling changes: Benchmark artifacts builder

• Added -g -fno-omit-frame-pointer to gcc:8.5.0-amd64-debian-buster-default @filipecosta90 (#128)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Updated min python version to 3.7 poetry and CI scripts @filipecosta90 (#124)
• Included double and long score ZSET ingestion benchmarks @filipecosta90 (#122)
• Added API healthcheck endpoints @filipecosta90 (#123)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Fixed self contained coordinator circular dependency import @filipecosta90 (#121)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Fixed linting issues @filipecosta90 (#115)

Tooling changes: Benchmark artifacts builder

• Added --docker-air-gap to builder and scc @filipecosta90 (#116)

Tooling changes: Self-contained coordinator

• Added --docker-air-gap to builder and scc @filipecosta90 (#116)

Tooling changes: Setup agnostic cli benchmark runner (solely client)

• Allows the standalone runner to collect perf profiles @slice4e (#111)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Fix preload warning @slice4e (#112)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

Tooling changes: Self-contained coordinator

• Added consumer-id to self contained coordinator. Enables multi consumers @filipecosta90 (#110)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

Tooling changes: Self-contained coordinator

• Fixed self_contained_coordinator r.ping() using 6379 port and not redis_proc_start_port @filipecosta90 (#109)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

Tooling changes: Self-contained coordinator

• Enabled to specify redis_proc_start_port on self contained coordinator; Include merge-base commit tests on API trigger @filipecosta90 (#106)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Fixed dbconfig array version of SPEC @filipecosta90 (#103)
• Included development how-to on the readme @filipecosta90 (#102)

Tooling changes: Setup agnostic cli benchmark runner (solely client)

• Adding --test-time to memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values-pipeline-10.yml @slice4e (#100)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• Fixed 4 test-cases, which were completing too quickly @slice4e (#98)
• Fixed small duration benchmarks for HSET. Included 1KiB HSET variation @filipecosta90 (#99)

Redis performance and observability spec change

Project status:

The following is a high level status report for currently available specs and tools.

• Benchmark platforms specification: Experimental components are released and available for beta testing.
• Benchmark topologies specification: Experimental components are released and available for beta testing.
• Benchmark artifacts builder specification: Experimental components are released and available for beta testing.
• Test-suites specification: Experimental components are released and available for beta testing.
• Spec tool implementation: API: Experimental components are released and available for beta testing.
• Spec tool implementation: Benchmark artifacts builder: Experimental components are released and available for beta testing.
• Spec tool implementation: Self-contained coordinator: Experimental components are released and available for beta testing.
• Spec tool implementation: Large scale coordinator: Draft components are under design, and have not been added to the specification.
• *Spec tool implementation: Setup agnostic cli benchmark runner (solely client): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.

Release changes:

• delete the stopped container by default at tear down @slice4e (#96)