questdb

An open source time-series database for fast ingest and SQL queries

APACHE-2.0 License

Stars
13.4K
Committers
152
View Code on GitHub Visit Website View on X
Ecosystems: Grafana, Java, PostgreSQL, C++

Bot releases are hidden (Show)

questdb - 7.4.0 Latest Release

Published by bluestreak01 7 months ago

๐ŸŒธ QuestDB 7.4 Release Notes

Ahh, Spring! Sunshine, flowers, all that nice stuff.

As the wheels of the seasons turn, so to the wheels of the applications we're all busy building.

Hopefully good things bloom for us all.

QuestDB is pleased to release version 7.4!

As you've come to expect, this release further improves core database performance. Also as per usual, several bugs and rough edges have been addressed. In important addition, there are note-able breaking changes that will provide a smoother overall experience and set us up well for future changes. Please address them prior to upgrade, and reach out to us if you run into any issues.

โš ๏ธ New Features with breaking changes

  • SAMPLE BY queries now use ALIGN TO CALENDAR by default. As such, if you're using SAMPLE BY and expecting ALIGN TO FIRST OBSERVATION, you'll need to update your queries to specify that syntax.

  • Java client: The ILP Sender used in our first-party Java client now requires an explicit selection of Transport, (HTTP or TCP) for enhanced reliability and clarity in data ingestion workflows. You will need to pass this value forward. See the PR for more information. If you do not use the Java client, but use another language client, see your respective language client repository and update if needed.

  • CREATE TABLE AS and INSERT INTO SELECT statements now operate non-atomically and are batched by default. If you require an atomic table, you can now use the very powerful sounding: CREATE ATOMIC TABLE. For more information, checkout the docs.

๐Ÿš€ Performance Improvements

This release improves parallel GROUP BY query performance and also makes SAMPLE BY queries more efficient. In addition, we've reduced thread-local allocator overhead in single-threaded GROUP BY & SAMPLE BY queries. The first/last aggregate functions for string columns have also been optimized.

๐Ÿ› Bug Fixes

  • Resolved issues leading to segmentation faults and incorrect results in certain SQL queries involving GROUP BY and SAMPLE BY clauses.
  • Addressed a bug that could cause a table to be omitted from snapshots after significant DDL changes.
  • Fixed HTTP request handling to prevent rejections when content-length is set to 0.
  • Solved potential TLS handshake stalls on MacOS.
  • Corrected the behavior of altering tables to add indexes that could inadvertently remove deduplication keys.
  • Eliminated a crash during ingestion and various other stability issues.

What's Changed

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3.10...7.4.0

questdb - 7.3.10

Published by bluestreak01 8 months ago

Fresh from QuestDB ๐Ÿฒ

The 2024 Lunar New Year marks the Year of the Dragon. ๐Ÿ‰ When you think of the mighty dragon, what comes to mind? For us, well, we're a little different. Dragons evoke strength, wisdom, and - naturally - wild performance (we're performance obsessed).

Within that theme, the 7.3.10 release offers a 5x-10x speed-up in ASOF and LT JOIN queries, as well as a number of optimizations in parallel filters, GROUP BY, and within the JIT compiler. Further, UUID and LONG256 columns are now supported by parallel GROUP BY for efficient use of all hardware resources. For workloads that feature small, but frequent transactions you'll also see improved performance.

Familiar with InfluxDB? Interested in using InfluxDB Line Protocol (ILP), but with something more robust under-the-hood? This release provides 5x faster ingestion performance when multiple connections are in use, further increasing overall throughput strength compared to InfluxDB, especially in data with high cardinality. Whether you're looking to leverage ILP for high performance streaming, or upgrading an existing InfluxDB workload over to QuestDB, this update makes it easier.

Read more about "drop-in" InfluxDB migration in our recent blog.

Alongside the usual battery of bug fixes, this release also includes a new HyperLogLog-based approx_count_distinct() SQL function that is 3x-5x faster and more memory efficient than the existing count_distinct() function for high cardinality data sets. And finally, Write-Ahead Log (WAL) enabled tables are now the default table type. This means that the benefits of WAL & WAL-required features like deduplication are available to each newly created table.

Whether you're new to QuestDB, working in prod, or migrating existing time series workloads, this release has something for you.

Performance improvements

New features

Fixes and other improvements

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3.9...7.3.10

questdb - 7.3.8b

Published by bluestreak01 10 months ago

Welcome back from the holidays! (If you had one...)

In the latest QuestDB release, we've focused on enhanced stability and introduced two innovative features, currently in beta. Notably, we've improved GROUP BY performance tenfold through parallel SQL executions and introduced InfluxDB Line Protocol (ILP) support over HTTP, which is compatible with existing InfluxDB drivers. Those familiar with ILP will find a clean entryway into QuestDB.

Thanks to @ssharaev and @pompeiifreckles for their first contributions - we appreciate you.

Dive in to explore the release in detail. ๐Ÿคฟ

What's changed

Please note that the following features are in beta - please let us know if you run into any issues.

  • 10x GROUP BY performance improvement via parallel SQL executions. Should you encounter any problems with this implementation, you can safely disabled [cairo.sql.parallel.groupby.enabled=false] it via configuration without downgrading the database instance.
  • InfluxDB Line Protocol (ILP) support over HTTP. This implementation supports existing InfluxDB drivers and eases the transition into QuestDB for existing ILP users. The new client has control over transaction boundaries and will offer helpful feedback on transaction activity. We're also pleased to offer no noticeable performance impact in ingestions performance compared to our TCP implementation.

New features

Fixes and improvements

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3.7...7.3.8b

questdb - 7.3.7

Published by bluestreak01 11 months ago

What's Changed

The hits keep comin'. Newly released QuestDB 7.3.7 adds supporting functions to pair with our recently released window functions. It also offers brings further improvements to SQL and ingestion performance - this has been a pattern of late!

Additional key features include:

  • A 5x speed up of WAL Apply for small transactions
  • New built-in functions for covariance, correlation, cumulative sum, and first value window
  • Enhanced compatibility with Grafana

Thanks to @AjCH1 and @ksankeerth for their first contributions!

New features

Fixes and improvements

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3.5...7.3.7

questdb - 7.3.5

Published by bluestreak01 11 months ago

What's Changed

Following up our recent 7.3.4 release which provided Window Functions, Web Console improvements, and more, we're pleased to announce another significant release. Headlining 7.3.5 are the following key additions:

  • Percentile function: Find an approximate percentile for a given series with approx_percentile(DD).
  • Optimized queries: QuestDB has long stood out for its high performance ingestion. Not be left behind, queries also keep getting faster. In benchmarks, recent performance optimizations have shown 50% improvement to several query types.

New features

Fixes and improvements

questdb - 7.3.4

Published by goodroot 12 months ago

What's Changed

Better. Faster. Stronger. And even more good looking. ๐Ÿ˜Ž

Highlighting our latest release:

  • Window functions. These flexible functions provide moving averages, running totals, or Top-N results within a group, and more via a "window", defined by an OFFSET against a RANGE or ROWS. Unlike aggregate functions, window functions do not cause rows to become grouped into a single output row โ€” the rows retain their separate identities. Additional perks? Performance and an easing of query complexity.

  • More functions. We've added:first(boolean), last(boolean), first(string), last(string), mathematical exponent exp(D) and first_not_null(), last_not_null(), which operate on most nullable data types, except IPv4 and GeoHash.

  • Performance. Our goal is top performance. LIKE/ILIKE queries now perform over 80% faster.

  • Web Console Refresh. The best tools provide the help that you need without getting in your way. We've tuned the Web Console UI to put what you need closer to where you need it. Less used UI elements have had their significance reduced and overall navigation has been streamlined. On top of that, a discrete news feed will send you all the latest, so that you don't miss a beat.

Before:

After:

New features

UI

Improvements and bug fixes

Documentation

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3.3...7.3.4

questdb - 7.3.3

Published by bluestreak01 about 1 year ago

Overview

Thanks for all who contributed to our most recent release.

This release provides a set of stability fixes and usability improvements.

We are excited to highlight the following areas:

  • Improved operations: A trio of features including improved snapshots to replace existing backup functionality, connection count gauge metrics and a new mkdir.mode for detached partitions make QuestDB easier to operate.

  • InfluxDB Line Protocol improvements: The InfluxDB Line Protocol is commonly used for rapid ingest. Weโ€™ve made it better, by introducing more flexible and less error prone timestamps. In addition, it also authenticates faster and disconnects on error when using WAL tables.

  • New aggregate functions: Thanks to community member @charlespnh QuestDB now offers variance(), var_samp() and var_pop(), stddev(), stddev_pop() functions.

What is new

CORE

SQL

ILP

What is fixed

CORE

SQL

ILP

HTTP

PGWire

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3.2...7.3.3

questdb - 7.3.2

Published by bluestreak01 about 1 year ago

Overview

Maintenance release improving stability of the deduplication logic, which we introduced in 7.3. For anyone relying on
deduplication or considering using it, we recommend to upgrade at the earliest convenience.

What is new

SQL

What is fixed

CORE

SQL

PGWire

HTTP

UI

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3.1...7.3.2

questdb - 7.3.1

Published by bluestreak01 about 1 year ago

Overview

Follow-up release for the data deduplication logic and IPv4 type, first released in 7.3.
This version brings the dedup logic for CSV Import alongside minor fixes and additional functions compatible with IPv4.

We also update the Web Console with a new "create table" UI to create new tables interactively. We would love your feedback on this new UI item; let us know your thoughts.

What is new

SQL

UI

What is fixed

CORE

SQL

HTTP

PGWire

UI

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.3...7.3.1

questdb - 7.3

Published by bluestreak01 about 1 year ago

Overview

Data deduplication

This release includes the highly anticipated data deduplication logic, which is now supported across all ingestion protocols. Its primary objective is to simplify the process of restarting data streams. If a data stream is interrupted, clients can easily resolve the issue by re-publishing an arbitrarily long window of data. The server will then handle the task of retaining only the missing data.

Data deduplication functions as an upsert operation by preserving the newer version of a record when duplicates are encountered.

To control data deduplication the following SQL syntax is available:

-- deduplicate on 2 columns
CREATE TABLE no_ts_dups_table (timestamp ts, l long, symbol sym, uuid id, string s) timestamps(ts) 
PARTITION BY DAY WAL 
DEDUPLICATE UPSERT KEYS(ts, id)

-- deduplication on 3 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, id, l);

-- deduplication on 4 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, sym, id, l);

We are thrilled to assure you that data deduplication has a negligible impact on ingestion performance, allowing it to be enabled at all times.

New IPV4 data type

This release introduces a new data type called ipv4. The primary objective behind this addition is to assist users working with IPv4 addresses in alleviating storage constraints and enhancing the overall SQL execution performance. Please see this the original pull request for more details.

What is new

CORE

SQL

What is fixed

CORE

SQL

UI

  • fix incorrect rendering of last few columns in Web Console grid

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.2.1...7.3

questdb - 7.2.1

Published by bluestreak01 over 1 year ago

Overview

This is a maintenance release ahead of the next major release. Consider upgrading if you are affected by the issues fixed below.

What is new

SQL

What is fixed

CORE

SQL

PGWire

ILP

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.2...7.2.1

questdb - 7.2

Published by bluestreak01 over 1 year ago

Overview

In QuestDB 7.2, we address a key issue faced by our users regarding write amplification and performance under heavy loads.

Previously, QuestDB utilized fixed-size time partitions to store data in ascending chronological order. However, this approach resulted in write amplification, particularly towards the end of each time interval. To tackle this problem, we have introduced implicit variable-size time partitions in this release.

Before

image

After

image

The new feature intelligently splits partitions at strategic points, ensuring smaller partition sizes and reducing the surface area for copy-on-write operations. This minimizes write amplification and maintains high write performance, even during heavy load scenarios.

To further optimize the system, we have implemented a sophisticated statistical model to balance the split and squash logic. As a result, partitions statistically unlikely to be modified again are asynchronously squashed, reducing the number of files on disk and alleviating stress on the OS metadata catalog.

Write amplification is reduced by orders of magnitude. Users can now experience sustained write performance, even in demanding environments.

Gotchas

This is a breaking change. If you decide to downgrade from this release, partitions, created by 7.2 may not be understood by earlier versions. To proceed with the downgrade you may need to "squash" partitions on each table. To do that, run the following SQL before the downgrade:

ALTER TABLE <tab> SQUASH PARTITIONS

What is new

SQL

ILP

UI

What is fixed

CORE

SQL

ILP

HTTP

UI

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.1.3...7.2

questdb - 7.1.3

Published by bluestreak01 over 1 year ago

Overview

This release includes several minor hotfixes before the next major release, 7.2.

What we fixed

CORE

SQL

ILP

UI

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.1.2...7.1.3

questdb - 7.1.2

Published by bluestreak01 over 1 year ago

Overview

We ship a new user interface for CSV imports in the Web Console. This stability release collates bug fixes for issues raised by our community. We decided to cut this release ahead of the feature-packed 7.2.

What is new

UI

CORE

SQL

HTTP

What we fixed

CORE

SQL

HTTP

PGRWire

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.1.1...7.1.2

questdb - 7.1.1

Published by bluestreak01 over 1 year ago

Overview

This release was prompted by the library dependency that QuestDB 7.1 introduced in the Web Console. This dependency has affected a lot of users, and we have decided to fix it as soon as possible.

What is new

SQL

What we fixed

CORE

SQL

UI

HTTP

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.1...7.1.1

questdb - 7.1

Published by bluestreak01 over 1 year ago

Overview

This release introduces fixes for bugs found in the WAL storage mechanism (introduced in 7.0.1) as a result of thousands of hours of fuzz testing. We also fixed all of the issues reported on GitHub and by users in our public Slack channel. It also includes an improved UI for the Web Console.

What is new

CORE

SQL

UI
PGWire

What we fixed

CORE

SQL

PGWire

ILP

HTTP

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/7.0.1...7.1

questdb - 7.0.1

Published by bluestreak01 over 1 year ago

Overview

QuestDB 7.0.1 is the production-ready implementation of our new storage type, which makes data ingestion faster and more reliable. In this release, we introduce a Write-Ahead-Log (WAL) to ingest data. Newly created tables will now be WAL tables by default. Existing tables will use legacy storage until explicitly migrated to a new storage type.

๐Ÿš€ WAL Storage new features

Benchmark

Below are our latest benchmark results using the Time Series Benchmarking Suite:

  • Benchmark: CPU-only
  • Hardware: 32CPUs, 64GB RAM

The ingestion rate is measured in rows per second:

ย  QuestDB 7.0 influxDB 1.8.10 TimescaleDB 2.10 (tuned)
4 workers 1.458M 0.181M 0.304M
8 workers 2.874M 0.306M 0.464M
12 workers 4.155M 0.372M 0.517M
16 workers 4.373M 0.375M 0.510M

How to migrate from existing legacy tables to WAL-tables

To create a WAL-based table, the following syntax should be used (note WAL at the end):

create table ABC(x int, t timestamp) timestamp(t) partition by day WAL;

You can also convert your existing table to WAL (followed by an instance restart). The conversion touches table configuration (not data!) and is fully reversible.

To convert an existing table to WAL:

  • alter table ABC set type WAL;
  • restart instance

To convert a WAL table to a non-WAL table:

  • alter table ABC set type bypass WAL;
  • restart instance

What is new

CORE

SQL

HTTP

UI

What we fixed

CORE

SQL

ILP

Full Changelog: https://github.com/questdb/questdb/compare/7.0.0...7.0.1

questdb - 7.0.0

Published by ideoma over 1 year ago

Overview

๐Ÿš€ New Features

This is the Beta release of the Write Ahead Log (WAL) table storage model #2951.

We are always listening to our users, and took to heart your problem of ingestion slowdown with heavy out-of-order data flow. This release provides a comprehensive solution to that.

The reasons why you should switch to WAL are:

  • Consistent ingestion performance, unaffected by out-of-order data processing (2x performance increase on ILP)
  • Concurrent writing via all interfaces. Issues like #1461 are solved for WAL tables.
  • Enables future work on replication. Replication will work with WAL tables only.

The WAL storage model is inactive by default. To create a WAL-based table, the following syntax should be used (note WAL at the end):

create table ABC(x int, t timestamp) timestamp(t) partition by day WAL;

You can also convert your existing table to WAL (followed by an instance restart). The conversion touches table configuration (not data!) and is fully reversible.

To convert an existing table to WAL:

  • alter table ABC set type WAL;
  • restart instance

To convert a WAL table to non-WAL:

  • alter table ABC set type bypass WAL;
  • restart instance

What is new

CORE

SQL

UI

What we fixed

SQL

Full Changelog: https://github.com/questdb/questdb/compare/6.7...7.0.0

questdb - 6.7

Published by bluestreak01 over 1 year ago

Overview

๐Ÿš€ New Features

  • 128-bit UUID column type to make storage and queries more efficient (#2769)
  • explain SQL to show insights into query execution plan (#2680)
  • Trigonometric functions and a performance improvement for SQL queries (#2890)

โ— Breaking changes

  • As of this release, it is strongly discouraged to modify or delete table directories in the db directory or delete table directories while the database instance is still running. It is still safe to do those activities while database is shut down. (https://github.com/questdb/questdb/pull/2752)

  • If multiple database instances share the same db directory, you will now have to declare all but one of them as read-only instances via their configuration. (https://github.com/questdb/questdb/pull/2752)

What is new

CORE

SQL

PGWire

ILP

What we fixed

CORE

SQL

HTTP

New Contributors

Full Changelog: https://github.com/questdb/questdb/compare/6.6.1...6.7

questdb - 6.6.1

Published by bluestreak01 almost 2 years ago

Overview

This is a follow up release for the 6.6. It fixes a serious data consistency issue, introduced specifically by 6.6.

โ— Breaking change

Unless you're upgrading from versions before 6.6, you will be affected by a minor breaking change: commitLag keyword in SQLs is no longer supported in this release. We recommend you remove these keywords and "lag" values and let the system handle things automatically.

What we fixed

CORE

Full Changelog: https://github.com/questdb/questdb/compare/6.6...6.6.1

Package Rankings
Top 8.17% on Proxy.golang.org
Top 5.8% on Repo1.maven.org
Related Projects
changelog.com

Changelog is news and podcast for developers. This is our open source platform.

28 Dec 2015 2,694
antares

A modern, fast and productivity driven SQL client with a focus in UX

30 Apr 2020 1,780
eventstore

Event store using PostgreSQL for persistence

22 Feb 2016 1,009