An open source time-series database for fast ingest and SQL queries
APACHE-2.0 License
Bot releases are visible (Hide)
Ahh, Spring! Sunshine, flowers, all that nice stuff.
As the wheels of the seasons turn, so to the wheels of the applications we're all busy building.
Hopefully good things bloom for us all.
QuestDB is pleased to release version 7.4!
As you've come to expect, this release further improves core database performance. Also as per usual, several bugs and rough edges have been addressed. In important addition, there are note-able breaking changes that will provide a smoother overall experience and set us up well for future changes. Please address them prior to upgrade, and reach out to us if you run into any issues.
SAMPLE BY
queries now use ALIGN TO CALENDAR
by default. As such, if you're using SAMPLE BY
and expecting ALIGN TO FIRST OBSERVATION
, you'll need to update your queries to specify that syntax.
Java client: The ILP Sender used in our first-party Java client now requires an explicit selection of Transport, (HTTP or TCP) for enhanced reliability and clarity in data ingestion workflows. You will need to pass this value forward. See the PR for more information. If you do not use the Java client, but use another language client, see your respective language client repository and update if needed.
CREATE TABLE AS
and INSERT INTO SELECT
statements now operate non-atomically and are batched by default. If you require an atomic table, you can now use the very powerful sounding: CREATE ATOMIC TABLE
. For more information, checkout the docs.
This release improves parallel GROUP BY
query performance and also makes SAMPLE BY
queries more efficient. In addition, we've reduced thread-local allocator overhead in single-threaded GROUP BY
& SAMPLE BY
queries. The first/last aggregate functions for string columns have also been optimized.
sample by
implementation by @bluestreak01 in https://github.com/questdb/questdb/pull/4259
Full Changelog: https://github.com/questdb/questdb/compare/7.3.10...7.4.0
Published by bluestreak01 8 months ago
The 2024 Lunar New Year marks the Year of the Dragon. ๐ When you think of the mighty dragon, what comes to mind? For us, well, we're a little different. Dragons evoke strength, wisdom, and - naturally - wild performance (we're performance obsessed).
Within that theme, the 7.3.10 release offers a 5x-10x speed-up in ASOF
and LT JOIN
queries, as well as a number of optimizations in parallel filters, GROUP BY
, and within the JIT compiler. Further, UUID
and LONG256
columns are now supported by parallel GROUP BY
for efficient use of all hardware resources. For workloads that feature small, but frequent transactions you'll also see improved performance.
Familiar with InfluxDB? Interested in using InfluxDB Line Protocol (ILP), but with something more robust under-the-hood? This release provides 5x faster ingestion performance when multiple connections are in use, further increasing overall throughput strength compared to InfluxDB, especially in data with high cardinality. Whether you're looking to leverage ILP for high performance streaming, or upgrading an existing InfluxDB workload over to QuestDB, this update makes it easier.
Read more about "drop-in" InfluxDB migration in our recent blog.
Alongside the usual battery of bug fixes, this release also includes a new HyperLogLog-based approx_count_distinct()
SQL function that is 3x-5x faster and more memory efficient than the existing count_distinct()
function for high cardinality data sets. And finally, Write-Ahead Log (WAL) enabled tables are now the default table type. This means that the benefits of WAL & WAL-required features like deduplication are available to each newly created table.
Whether you're new to QuestDB, working in prod, or migrating existing time series workloads, this release has something for you.
Full Changelog: https://github.com/questdb/questdb/compare/7.3.9...7.3.10
Published by bluestreak01 10 months ago
Welcome back from the holidays! (If you had one...)
In the latest QuestDB release, we've focused on enhanced stability and introduced two innovative features, currently in beta. Notably, we've improved GROUP BY
performance tenfold through parallel SQL executions and introduced InfluxDB Line Protocol (ILP) support over HTTP, which is compatible with existing InfluxDB drivers. Those familiar with ILP will find a clean entryway into QuestDB.
Thanks to @ssharaev and @pompeiifreckles for their first contributions - we appreciate you.
Dive in to explore the release in detail. ๐คฟ
Please note that the following features are in
beta
- please let us know if you run into any issues.
GROUP BY
performance improvement via parallel SQL executions. Should you encounter any problems with this implementation, you can safely disabled [cairo.sql.parallel.groupby.enabled=false]
it via configuration without downgrading the database instance.nullif(double, double)
SQL function by @ssharaev in https://github.com/questdb/questdb/pull/4063
truncate
& table corruption after restore from snapshot by @bluestreak01 in https://github.com/questdb/questdb/pull/4095
Full Changelog: https://github.com/questdb/questdb/compare/7.3.7...7.3.8b
Published by bluestreak01 11 months ago
The hits keep comin'. Newly released QuestDB 7.3.7 adds supporting functions to pair with our recently released window functions. It also offers brings further improvements to SQL and ingestion performance - this has been a pattern of late!
Additional key features include:
Thanks to @AjCH1 and @ksankeerth for their first contributions!
information_schema.columns()
function to provide list of columns across all tables in the database by @marregui in https://github.com/questdb/questdb/pull/3975
Full Changelog: https://github.com/questdb/questdb/compare/7.3.5...7.3.7
Published by bluestreak01 11 months ago
Following up our recent 7.3.4
release which provided Window Functions, Web Console improvements, and more, we're pleased to announce another significant release. Headlining 7.3.5
are the following key additions:
approx_percentile(DD)
.show server_version
by @nitram509 in https://github.com/questdb/questdb/pull/3901
show parameters
SQL by @bluestreak01 in https://github.com/questdb/questdb/pull/3972
Published by goodroot 12 months ago
Better. Faster. Stronger. And even more good looking. ๐
Highlighting our latest release:
Window functions. These flexible functions provide moving averages, running totals, or Top-N results within a group, and more via a "window", defined by an OFFSET
against a RANGE
or ROWS
. Unlike aggregate functions, window functions do not cause rows to become grouped into a single output row โ the rows retain their separate identities. Additional perks? Performance and an easing of query complexity.
More functions. We've added:first(boolean)
, last(boolean)
, first(string)
, last(string)
, mathematical exponent exp(D)
and first_not_null()
, last_not_null()
, which operate on most nullable data types, except IPv4 and GeoHash.
Performance. Our goal is top performance. LIKE
/ILIKE
queries now perform over 80% faster.
Web Console Refresh. The best tools provide the help that you need without getting in your way. We've tuned the Web Console UI to put what you need closer to where you need it. Less used UI elements have had their significance reduced and overall navigation has been streamlined. On top of that, a discrete news feed will send you all the latest, so that you don't miss a beat.
Before:
After:
short
columns by @bluestreak01 (#3924)name
to table_name
in cursor function tables by @marregui in https://github.com/questdb/questdb/pull/3857
Full Changelog: https://github.com/questdb/questdb/compare/7.3.3...7.3.4
Published by bluestreak01 about 1 year ago
Thanks for all who contributed to our most recent release.
This release provides a set of stability fixes and usability improvements.
We are excited to highlight the following areas:
Improved operations: A trio of features including improved snapshots to replace existing backup functionality, connection count gauge metrics and a new mkdir.mode
for detached partitions make QuestDB easier to operate.
InfluxDB Line Protocol improvements: The InfluxDB Line Protocol is commonly used for rapid ingest. Weโve made it better, by introducing more flexible and less error prone timestamps. In addition, it also authenticates faster and disconnects on error when using WAL tables.
New aggregate functions: Thanks to community member @charlespnh QuestDB now offers variance()
, var_samp()
and var_pop()
, stddev()
, stddev_pop()
functions.
mkdir.mode
permission config for detached partitions by @debanjanc01 in https://github.com/questdb/questdb/pull/3818
Full Changelog: https://github.com/questdb/questdb/compare/7.3.2...7.3.3
Published by bluestreak01 about 1 year ago
Maintenance release improving stability of the deduplication logic, which we introduced in 7.3. For anyone relying on
deduplication or considering using it, we recommend to upgrade at the earliest convenience.
alter table resume wal
not having the effect by @ideoma in https://github.com/questdb/questdb/pull/3715
update
sql execution by @ideoma in https://github.com/questdb/questdb/pull/3703
Full Changelog: https://github.com/questdb/questdb/compare/7.3.1...7.3.2
Published by bluestreak01 about 1 year ago
Follow-up release for the data deduplication
logic and IPv4
type, first released in 7.3.
This version brings the dedup logic for CSV Import alongside minor fixes and additional functions compatible with IPv4
.
We also update the Web Console with a new "create table" UI to create new tables interactively. We would love your feedback on this new UI item; let us know your thoughts.
Full Changelog: https://github.com/questdb/questdb/compare/7.3...7.3.1
Published by bluestreak01 about 1 year ago
This release includes the highly anticipated data deduplication logic, which is now supported across all ingestion protocols. Its primary objective is to simplify the process of restarting data streams. If a data stream is interrupted, clients can easily resolve the issue by re-publishing an arbitrarily long window of data. The server will then handle the task of retaining only the missing data.
Data deduplication functions as an upsert operation by preserving the newer version of a record when duplicates are encountered.
To control data deduplication the following SQL syntax is available:
-- deduplicate on 2 columns
CREATE TABLE no_ts_dups_table (timestamp ts, l long, symbol sym, uuid id, string s) timestamps(ts)
PARTITION BY DAY WAL
DEDUPLICATE UPSERT KEYS(ts, id)
-- deduplication on 3 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, id, l);
-- deduplication on 4 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, sym, id, l);
We are thrilled to assure you that data deduplication has a negligible impact on ingestion performance, allowing it to be enabled at all times.
IPV4
data typeThis release introduces a new data type called ipv4. The primary objective behind this addition is to assist users working with IPv4 addresses in alleviating storage constraints and enhancing the overall SQL execution performance. Please see this the original pull request for more details.
ipv4
as supported column type by @allegraharris in https://github.com/questdb/questdb/pull/3523
string
and char
comparison by @bziobrowski in https://github.com/questdb/questdb/pull/3607
Full Changelog: https://github.com/questdb/questdb/compare/7.2.1...7.3
Published by bluestreak01 over 1 year ago
This is a maintenance release ahead of the next major release. Consider upgrading if you are affected by the issues fixed below.
Full Changelog: https://github.com/questdb/questdb/compare/7.2...7.2.1
Published by bluestreak01 over 1 year ago
In QuestDB 7.2, we address a key issue faced by our users regarding write amplification and performance under heavy loads.
Previously, QuestDB utilized fixed-size time partitions to store data in ascending chronological order. However, this approach resulted in write amplification, particularly towards the end of each time interval. To tackle this problem, we have introduced implicit variable-size time partitions in this release.
The new feature intelligently splits partitions at strategic points, ensuring smaller partition sizes and reducing the surface area for copy-on-write operations. This minimizes write amplification and maintains high write performance, even during heavy load scenarios.
To further optimize the system, we have implemented a sophisticated statistical model to balance the split and squash logic. As a result, partitions statistically unlikely to be modified again are asynchronously squashed, reducing the number of files on disk and alleviating stress on the OS metadata catalog.
Write amplification is reduced by orders of magnitude. Users can now experience sustained write performance, even in demanding environments.
This is a breaking change. If you decide to downgrade from this release, partitions, created by 7.2 may not be understood by earlier versions. To proceed with the downgrade you may need to "squash" partitions on each table. To do that, run the following SQL before the downgrade:
ALTER TABLE <tab> SQUASH PARTITIONS
Full Changelog: https://github.com/questdb/questdb/compare/7.1.3...7.2
Published by bluestreak01 over 1 year ago
This release includes several minor hotfixes before the next major release, 7.2.
Full Changelog: https://github.com/questdb/questdb/compare/7.1.2...7.1.3
Published by bluestreak01 over 1 year ago
We ship a new user interface for CSV imports in the Web Console. This stability release collates bug fixes for issues raised by our community. We decided to cut this release ahead of the feature-packed 7.2.
COPY CANCEL
API by @bluestreak01 in https://github.com/questdb/questdb/pull/3298
nullif(long,long)
SQL function by @javier in https://github.com/questdb/questdb/pull/3188
rank()
analytical SQL function by @puzpuzpuz in https://github.com/questdb/questdb/pull/2929
QuestDB
to string returned by version()
function by @davidcooke2 in https://github.com/questdb/questdb/pull/3239
Full Changelog: https://github.com/questdb/questdb/compare/7.1.1...7.1.2
Published by bluestreak01 over 1 year ago
This release was prompted by the library dependency that QuestDB 7.1 introduced in the Web Console. This dependency has affected a lot of users, and we have decided to fix it as soon as possible.
Full Changelog: https://github.com/questdb/questdb/compare/7.1...7.1.1
Published by bluestreak01 over 1 year ago
This release introduces fixes for bugs found in the WAL storage mechanism (introduced in 7.0.1) as a result of thousands of hours of fuzz testing. We also fixed all of the issues reported on GitHub and by users in our public Slack channel. It also includes an improved UI for the Web Console.
LIKE
and ILIKE
operator by @SiyaoIsHiding in https://github.com/questdb/questdb/pull/3006
Full Changelog: https://github.com/questdb/questdb/compare/7.0.1...7.1
Published by bluestreak01 over 1 year ago
QuestDB 7.0.1 is the production-ready implementation of our new storage type, which makes data ingestion faster and more reliable. In this release, we introduce a Write-Ahead-Log (WAL) to ingest data. Newly created tables will now be WAL tables by default. Existing tables will use legacy storage until explicitly migrated to a new storage type.
Below are our latest benchmark results using the Time Series Benchmarking Suite:
The ingestion rate is measured in rows per second:
ย | QuestDB 7.0 | influxDB 1.8.10 | TimescaleDB 2.10 (tuned) |
---|---|---|---|
4 workers | 1.458M | 0.181M | 0.304M |
8 workers | 2.874M | 0.306M | 0.464M |
12 workers | 4.155M | 0.372M | 0.517M |
16 workers | 4.373M | 0.375M | 0.510M |
To create a WAL-based table, the following syntax should be used (note WAL
at the end):
create table ABC(x int, t timestamp) timestamp(t) partition by day WAL;
You can also convert your existing table to WAL (followed by an instance restart). The conversion touches table configuration (not data!) and is fully reversible.
To convert an existing table to WAL:
alter table ABC set type WAL;
To convert a WAL table to a non-WAL table:
alter table ABC set type bypass WAL;
Full Changelog: https://github.com/questdb/questdb/compare/7.0.0...7.0.1
Published by ideoma over 1 year ago
This is the Beta release of the Write Ahead Log (WAL) table storage model #2951.
We are always listening to our users, and took to heart your problem of ingestion slowdown with heavy out-of-order data flow. This release provides a comprehensive solution to that.
The reasons why you should switch to WAL are:
The WAL storage model is inactive by default. To create a WAL-based table, the following syntax should be used (note WAL
at the end):
create table ABC(x int, t timestamp) timestamp(t) partition by day WAL;
You can also convert your existing table to WAL (followed by an instance restart). The conversion touches table configuration (not data!) and is fully reversible.
To convert an existing table to WAL:
alter table ABC set type WAL;
To convert a WAL table to non-WAL:
alter table ABC set type bypass WAL;
Full Changelog: https://github.com/questdb/questdb/compare/6.7...7.0.0
Published by bluestreak01 over 1 year ago
As of this release, it is strongly discouraged to modify or delete table directories in the db
directory or delete table directories while the database instance is still running. It is still safe to do those activities while database is shut down. (https://github.com/questdb/questdb/pull/2752)
If multiple database instances share the same db
directory, you will now have to declare all but one of them as read-only instances via their configuration. (https://github.com/questdb/questdb/pull/2752)
sin
, cos
, tan
, cot
, asin
, acos
, atan
, atan2
, toRadians
, fromRadians
, pi
by @marregui in https://github.com/questdb/questdb/pull/2890
explain
SQL statement by @bziobrowski in https://github.com/questdb/questdb/pull/2680
group by
, sample by
and other queries by @puzpuzpuz in https://github.com/questdb/questdb/pull/2903
UUID
column type by @jerrinot in https://github.com/questdb/questdb/pull/2769
long256
comparisons on different columns of the same table @jerrinot https://github.com/questdb/questdb/pull/2924
Full Changelog: https://github.com/questdb/questdb/compare/6.6.1...6.7
Published by bluestreak01 almost 2 years ago
This is a follow up release for the 6.6. It fixes a serious data consistency issue, introduced specifically by 6.6.
Unless you're upgrading from versions before 6.6, you will be affected by a minor breaking change: commitLag
keyword in SQLs is no longer supported in this release. We recommend you remove these keywords and "lag" values and let the system handle things automatically.
Full Changelog: https://github.com/questdb/questdb/compare/6.6...6.6.1