questdb

Release highlights

This release focuses on bug fixes and includes changes relating to out-of-order ingestion, InfluxDB line protocol, backups, and filesystem fixes. These fixes enhance the reliability of the system through error prevention and improved stability.

What we fixed

• ILP: check for buffer overflow on writer queue (#1365)
• ILP: fix various edge cases, behaviour of type parsers tcp/udp is closer, allow empty field value as null, support other types (#1312)
• O3: fix wrong timestamp when first o3 row cancelled (#1348)
• O3: fixed o3 crash from ILP usage (#1335)
• SQL: include configuration files and tab index in backups (#1333)
• SQL: ignored LATEST BY column filter if it is indexed (#1327)
• SQL: cast long to GeoHash implementation (#1319)
• SQL: fix lt join for equal timestamps (#1321)
• CORE: we no longer map files to the length of file from the filesystem, we map them to the size received through a transaction "message"
• CORE: we were silently mapping files to filesystem length rather than the required size. We now always map to the required size and report when this is not possible
• CORE: the structure of the "index" column for variable-size columns has changed. It contains N + 1 entries with the last entry being the size of the variable-length column
• CORE: truncated column files to OS page size to avoid undefined behaviour on pages used by readers
• CORE: fixed an issue with truncating symbol.d column on Windows where the order of closing files was incorrect

What is new

• SQL: base64 function to encode binary data (#1355)

Release highlights

Release version 6.0.5 introduces support for geospatial data in QuestDB via the addition of geohash types. Geohashes encode geographic areas as base-32 strings, and native support for this type allows for fast and efficient querying and storage of geodata. For more information on geospatial support in QuestDB, including examples and references, see the QuestDB geohash documentation.

Additional features are new timestamp_floor() and timestamp_ceil() functions, improvements on the performance of first() and last() functions, out-of-order parameter support via REST API, and multiple bug fixes. These fixes and features improve application stability and enhance the system's flexibility of use.

What is new

• SQL: implement "touch" function, useful to preload the data from disk into system page cache (#1305)
• SQL: GeoHash type support
• SQL: within SQL operator - SIMD accelerated GeoHash prefix search with time-slice
• SQL: accelerated first() and last() aggregation functions for scanning index
• SQL: GeoHash literal support for character and bitmask
• SQL: allow comparisons between GeoHash and string, for chars notation (#1310)
• SQL: accelerated make_geohash() function (#1265)
• SQL: timestamp_floor() and timestamp_ceil() implementations (#1300)
• Core: new Row API to store GeoHash values (#1311)
• PGWire: add geohash support for PGWire protocol (#1270)
• ILP: add ingest support for type geohash, when schema exists (#1251)
• HTTP: add parameters commitLag and maxUncommittedRows to /imp (#1203)
• UI: basic syntax highlighting for geohashes in web console (#1318)

What we fixed

• SQL: fixed possible incorrect results in group by queries (#1285)
• SQL: throw invalid date exception on negative time interval (#1277)
• SQL: push timestamp required on asof join on both sides (#1225)
• SQL: support SQL keywords as column names (#1267)
• SQL: make SQL functions case-insensitive (#1219)
• O3: fixed race condition that can occur between O3 copy jobs and purge job (#1232)
• O3: row count on TableWriter is wrong when partition start in lag (#1223)
• ILP: review ILP/UDP value parser and fix corner cases (#1289)
• ILP: fixed missing index update when ingesting data over ILP (#1302)
• HTTP: fix writer left locked after CSV upload (#1218)
• Core: virtual memory refactoring (#1208)
• Misc: questdb.sh to work on FreeBSD (#1264)

Release highlights

Release version 6.0.4 focuses on community-driven topics raised with us recently by our users. The features included are performance improvements, increased parallelization of existing code, and calendar alignment for SAMPLE BY queries. Also included is a framework for exposing Prometheus metrics by Piotr Rżysko. The addition of Prometheus metrics is an exciting feature that we expect will be continuously developed with more types of counters and gauges added as we get feedback on its use.

What is new

• HTTP: Framework for adding Prometheus metrics #905
• HTTP: SQL executed via HTTP can make use of full worker pool on machines with over 16 cores #1127
• SQL: Performance improvements on distinct queries with user-configurable settings #1096
• SQL: Performance improvements on LATEST BY queries with indexed symbol #1084 #1131
• SQL: first() and last() functions optimized within SAMPLE BY #1157
• SQL: SAMPLE BY can be aligned to calendar values #1133
• ILP: Socket read retry is now user-configurable for InfluxDB line protocol ingestion over TCP #905
• Core: Improvements to server configuration file and settings on startup #1195
• Core: Improved error reporting in cases where tables are locked by writers #1129

What we fixed

• SQL: Fetch only last values from cursor on LATEST BY queries #1190
• SQL: Fixed queries which use SAMPLE BY without explicitly selecting timestamp columns #1189
• SQL: Added explicit insertion of null values into columns #592
• SQL: symbol types are implicitly cast to strings in string functions #1056
• SQL: Fixed issue that could cause infinite scan of table #1125
• SQL: SAMPLE BY queries using JOIN do not need explicit designated timestamp in subquery #867
• PGWire: Slow clients no longer causes incorrect flow control over Postgres wire #1160
• PGWire: Fixed corrupt reading of values in PHP/PDO #1059
• ILP: Varying the number of columns handles timestamps correctly #1182
• ILP: Buffer underflow when spaces escaped in InfluxDB line protocol messages #1143
• ILP: Fixed handling of quotes within InfluxDB line protocol field values #1158
• Core: Fixed a crash that could occur when returning readers to the pool #1097
• Core: Scoreboard cleaned on exclusive opens to prevent txn-inflight errors #1082
• Core: Path to executable of Windows service can include spaces #1116
• Core: Classes are now loaded correctly in embedded and OSGi applications #1168
• Core: Force query cache expiration on table meta change #1204

Breaking changes

The addition of special handling for null in #1179 introduces changes that enforce timestamps having only positive values. This means that timestamps cannot predate epoch 0 in UTC, disallowing pre-1970 values.

Release highlights

This release is focused primarily on stability of newly-implemented features. Particular attention is in the recently-implemented ingestion mechanism designed for out-of-order data. This release enhances the reliability of this feature and enhances the flexibility of the system.

What is new

• SQL: unary - (negation) implementation for all numeric types (#1070)
• SQL: optimised SQL aggregate plan when constants or runtime constant functions are selected (#1055)
• SQL: to_timezone() and to_utc() timestamp functions (#1062)
• SQL: using IN with bind variables, i.e. IN ($1) to work as with const literals (#1058)
• DOC: added JavaDoc to io.questdb.cairo.sql package
• UI: notification customization (#1049)

What we fixed

• ILP: fix NPE on processing InfluxDB line protocol (ILP) data (#1068)
• SQL: fix commit lag can be set to invalid value on alter table (#1069)
• O3: fix ability to insert null values in designated timestamp as select (#1066)

Highlights

This is a maintenance release to improve overall system stability. We also added several requested SQL functions.

What is new

• SQL: implementation of 'distinct' algorithm for time series data (#1053)
• SQL: log and sqrt implementations (#1050)
• SQL: bitwise ops implementation (#1046)
• SQL: support Postgres syntax to cast string literal to timestamp and timestamp with timezone (#740)
• SQL: support Postgres literal cast (#1032)
• UI: add copy build version link in footer (#1007)

What we fixed

• SQL: filter order execution could have been incorrect (#1025)
• SQL: fix several errors with count() aggregations
• SQL: fixed isnan() function for SSE2/SSE4 platforms (#1044)
• SQL: fixed issues in latest by implementation to do with incorrectly calculated column cross-indexes (#1038)
• SQL: fixed "invalid column" error, which was caused by incorrect top-down column propagation (#1027)
• SQL: make table column names case insensitive in where (#1045)
• ILP: handle trying to convert ILP strings into non STRING columns (#1029)
• DOCKER: removed volume from dockerfile and unified amd64 and arm64 files (#1023)
• O3: fixed crash when "lag" buffer is overrun
• O3: added debug logging to be able to understand o3 issues better just from logs (#1024)

Highlights

Out-of-order (O3) performance is heavily reliant on correctly configured hysteresis values. In 6.0.0, it is very easy to reset these values to 0 with alter table statements, causing ingest performance degradation. This version address fragility of the settings by fixing bugs in all affected alter table variations.

Additionally we made an effort to make timestamp search more coherent. You will be able to search consistently on timestamp represented by string values.

What we fixed

• PG Wire: Postgres JDBC driver cursor mode. Fixes #806 (#1013)
• REST: allow underscore in CSV imported column names (#1002)
• O3: fix alter table SQL erasing hysteresis values from table metadata (#1014)
• EMBEDDED: java 11 maven artifact name was wrong (#1019)

What is new

• DOCKER: enable logger configuration for docker container (#1018)
• SQL: timestamps as strings in selects (#977)
• SQL: alter table hysteresis (#1015)

The following changes are a delta of version 5.0.6.1 and therefore may include changes announced in 6.0.0 prereleases.

Release Highlights

Version 6.0.0 adds support for out-of-order (O3) record ingestion on-the-fly, and as a result, does not enforce that records appear at QuestDB in strict chronological order. O3 support adds flexibility and efficiency of use to the system and provides compatibility with the Time Series Benchmark Suite (TSBS) which is used to reliably measure and compare the performance of time series databases. Also included with this version is a a massive internal revision of InfluxDB Line Protocol (ILP) ingestion which brings significant performance improvements, alongside multiple UI fixes for the Web Console and SQL features.

What is new

• Arch: ARM64 and Apple Silicon (M1) support (#878)
• O3 support for ingestion of records which are out-of-order (O3) by timestamp (#926)
• REST: fileName parameter can be specified for the /exp endpoint for CSV exports (#830)
• PG wire: PostgreSQL JDBC driver now supports getSQLKeywords method (#836)
• UI: UI improvements for results with only one column (#897)
• UI: Notification element does not obscure returned rows (#896)
• O3: stable sort, reuse memory during sort (#987)
• O3: move o3SortVarColumn0 to C++ (#984)

What we fixed

• PG wire: fixed server "hang" when clients are slow to read data (#954)
• SQL: implicit handling of scientific notation for floating-point literals (ex: .12e-20) (#957)
• SQL: LIMIT -1 returns last row as expected (#831)
• SQL: Epoch timestamps supported in CSV imports (#876)
• SQL: Behavior of ORDER BY query returns correct values on non-cached symbol types (#877)
• SQL: Column names allow minus - underscore _ for compatibility with InfluxDB Line Protocol messages (#859)
• SQL: latest by query could return data in incorrect timestamp order (#986)
• ILP: logging of errno and replaced catch(CairoException) with catch(Throwable) (#983)
• O3: fixed fairness issue in ILP (#981)

JDK versions for embedded applications

Users who wish to run QuestDB 6.0 in an embedded capacity in a Maven project may choose between JDK versions.

JDK11

        <dependency>
            <groupId>org.questdb</groupId>
            <artifactId>questdb</artifactId>
            <version>6.0.0</version>
        </dependency>

JDK8

        <dependency>
            <groupId>org.questdb</groupId>
            <artifactId>questdb-jdk8</artifactId>
            <version>6.0.0</version>
        </dependency>

Release Highlights

Version 6.0.0-beta includes bug fixes relating to ingestion subsystems, SQL execution and type handling. This release adds support for out-of-order (O3) record ingestion on-the-fly and as a result, does not enforce that records appear at QuestDB in strict chronological order. O3 support adds flexibility and efficiency of use to the system and provides compatibility with the Time Series Benchmark Suite (TSBS) which is used to reliably measure and compare the performance of time series databases. Also included with this version is a a massive internal revision of InfluxDB Line Protocol (ILP) ingestion which brings significant performance improvements, alongside multiple UI fixes for the Web Console and SQL features.

What is new

• O3 support for ingestion of records which are out-of-order by timestamp (#926)
• ARM64 and Apple Silicon (M1) support (#878)
• fileName parameter can be specified for the /exp endpoint for CSV exports (#830)
• PostgreSQL JDBC driver now supports getSQLKeywords method (#836)
• UI improvements for results with only one column (#897)
• Notification element does not obscure returned rows (#896)

What we fixed

• PG wire: fixed server "hang" when clients are slow to read data (#954)
• SQL: implicit handling of scientific notation for floating-point literals (ex: .12e-20) (#957)
• SQL: LIMIT -1 returns last row as expected (#831)
• SQL: Epoch timestamps supported in CSV imports (#876)
• SQL: Behavior of ORDER BY query returns correct values on non-cached symbol types (#877)
• SQL: Column names allow minus - underscore _ for compatibility with InfluxDB Line Protocol messages (#859)

What is new

SQL: timestamp operators support time range with modifier (#802) Piotr Rżysko
SQL: coalesce function (#799) Alex Pelagenko - Documentation
SQL: function factory cache automatically derives bool operators from '=', '>' etc. (#796) Piotr Rżysko
Web Console: remove google tag manager (#803)

What we fixed

SQL: make JoinRecordMetadata metadata case insensitive (#800)
SQL: wrong results / exception when latest by with filtering (#808)
SQL: correctly calculate vector frames when partition interval is used (#804)

Release highlights

• Refactoring of PostgreSQL wire support
• Support RFC339Nano for grafana alerts
• Postgres binary protocol support
• build() function with QuestDB build details (version number, commit hash)
• create table if not exists

What is new

• SQL: create table if not exists .. implementation (#760)
• SQL: Add build() function to list QuestDB build details (#739) Fixed #720 Tiago Martins
• SQL: support RFC339Nano, from Grafana alerts (#737)

What we fixed

• PG Wire: SQL compiler can define bind variables and bind variable now support type cast (#732)
• PG Wire: deep refactoring of PostgreSQL wire support (#738)
• PG Wire: Fixed "busy table" error after insert failure (#790)
• PG Wire: Fixed crash when "writer" is re-created between inserts (#790)
• PG Wire: Fixed DATE insert from python (#790)
• PG Wire: Support for timestamps from pycop2 Python driver (#786)
• PG Wire: Fixed insert of timestamp and date from Rust (#784)
• PG Wire: Create table race condition fix (#787)
• PG Wire: fixed issue dealing with TCP fragmentation (#777)
• PG Wire: fixed transaction failure in simple query mode (#768)
• PG Wire: Added support for DotNet types query, to remove friction while establishing a simple DotNet connection (#755)
• PG Wire: Rust client support (#756)
• PG Wire: Support bind variables in Rust. Fixed #662,#639 (#744)
• PG Wire: preserve table id when table is altered (#751)
• PG Wire: metadata schema improvements (#747)
• SQL: Support bind variable in index-based 'where' clause lookups (#771)
• SQL: Fixed timestamp propagation issue for sampleBy. Fixed #749 (#758)
• SQL: resize issue with DirectLongList. Fixed #748 (#759)
• SQL: When forming execution plan group-by key could get ignored in sub-queries (#752)
• SQL: required timestamp was not added to cross-index for some SQLs, which lead to inaccurate query results (#736)
• SQL: union model should not be forced to use columns of the sibling model (fixed #734) (#735)
• SQL: prevent sql generator from assuming latest by key is always a symbol (#791)
• SQL: inverted int-to-boolean cast behaviour. Fixed #616 (#761)
• HTTP: lock table name before creating table on upload (#781)
• HTTP: ignore import parameters when table already exists
• HTTP: clear text loader state to fix concurrent csv imports on different tables (#774)
• HTTP: improved logging (#762)
• HTTP: Fixed importing single-column CSV bug #547 (#729) tonytamwk
• SERVER: fixed NPE when starting questdb with disabled HTTP server (#753)
• SERVER: pre-allocate disk when mapping files to memory to prevent data corruption when disk runs out of space (#743)

Release highights

• Integration with Kafka JDBC connector - Documentation
• SquirrelSQL support
• Implemented JDBC metadata queries

What is new

• CAIRO: block writer implementation
• CAIRO: table id (#683)
• SQL: drop partition based on boolean expression - fix #669 (#671) - Documentation
• SQL: add support for to_lowercase and to_uppercase (fixed #668) (#672) - Tiago Martins - Documentation
• SQL: support "record" column type (#693)
• SQL: various view and functions needed for Kafka metadata queries (#697)
• SQL: || concatenation operator. Fixed #635 Alex Pelagenko
• SQL: row_number() analytic function implementation, bugfixes (#706)
• HTTP: min http server for health monitoring (#688) Documentation

What we fixed

• ILP: Fix logging on disconnect in TCP ILP (#663)
• ILP: Log CairoException when writing rows in ILP TCP (#665)
• ILP: Fix flapping test (#687)
• ILP: Improve testing (#689)
• ILP: log more characters from TCP buffer (#698)
• HTTP: fixed bug where quote symbol was unduly removed from string value in INSERT SQL (#666)
• HTTP: change http status of SQL errors from 400 to 200 (#684)
• HTTP: queue write requests when table writer is busy (#421)
• SQL: fixed issues where columns were incorrectly propagated top down on models (#685)
• SQL: order by propagation caused QueryModels to become inconsistent (#696)
• SQL: column names are no longer case-sensitive
• SQL: fixed table leak out of sample by cursors when table is empty (#702)
• SQL: 'sample by' fix to align sampling timestamp to the data (#709)
• SQL: parser issue (https://github.com/questdb/questdb/issues/691)
• SQL: dropping partition after renaming column fails #678
• SQL: double-quotes in SQL queries #492
• PG Wire: Kafka JDBC connector does not work #609
• PG Wire: added error logging for pg-wire (#686)

Release highlights

• We improved PostgreSQL compatibility even further to include batch inserts from Java, bind variables from NodeJS, C and Python
• We now support Grafana via build-in PostgreSQL plugin
• We added secure authentication for Influx Line Protocol

What is new

• SQL: BETWEEN/AND SQL semantics (#642)
• SQL: haversine distance aggregation function (#597)
• SQL: first/last aggregation functions (#586)
• SQL: support joins of comma-delimited table names. Fixed #630 (#634)
• ILP: optional authentication to influx line protocol over TCP
• SERVER: debug logging level can be switched on from Env variable (#645)
• SERVER: log configuration can be sources from Env (#644)
• SERVER: Enabled override of all config properties from Env

What we fixed

• EMBEDDED: SQL functions are not being picked up when questdb is embedded without modules. Fixed #610 (#631)
• PG Wire: support batch insert, and 'BEGIN'. Fixed #145,#608 (#612)
• PG Wire: fixed inferring string parameter and supporting `yyyy-MM-ddTHH mm s (#627)
• SQL: Fixed column name for simple selects with a semicolon (#629). Fixed #617 Tiago Martins
• SQL: fix issues with bitmap index that lead to errors when the index is used to search contents up to the end of the partition
• SQL: various bugfixes (#643)
• SERVER: logging system's 'errno' from all sites that handle CairoException (#619)
• SERVER: possible data corruption when same thread performs different tasks on the same table.

Release highlights

• We improved PostgreSQL wire protocol to support more clients, such as C, Go, Python and NodeJS
• We improved performance of memory reads across the board. See our blog here

What is new

• SQL: LIKE implementation #476 (#512) Abhishek Pratap Singh*
• SQL: added count(string), count(symbol) and count(long256). Also added rnd_long256(count) (#556)
• SQL: contiguous memory implementation (#538), across the board performance improvement
• SQL: added millisecond sampling ('T') (#553
• SQL: insert statement accepts timestamp as string in both micro- and milli formats.
• SERVER: introduced version of web console and removed -f command line arg (#528
• SERVER: set default shared worker pool size to cpu count - 1 (#521)
• EMBEDDED simplified embedded API (#539)
• PG Wire: implemented protocol correctness. Fixes #450 (#544)
• PG Wire: parameter type inferring from parameter values, support NodeJS bind variables. Fixed (#290)

What we fixed

• SQL: drop table fix. Fixed #559 (#565)
• SQL: as of join failed when timestamp was not in select clause (#560)
• SQL: fixed timestamp propagation to join models (#558)
• SQL: fix memory leak in LimitRecordCursorFactory (#554)
• SQL: fixed issues with long NULL converting to double NULL. Also fixed group-by validation logic to deal with clauses more systematically. Fixed #540 (#545)
• SQL: fixed incorrect column referencing from index-based filter implementation (#542)
• SQL: fixed 'with' clause behaviour in set operations (union, expct, intersect), in recursive 'with' and in sub-queries (#541)
• TCP ILP: change ILP TCP connection defaults to never disconnec (#533)
• SQL: timezone parser could fail with unhandled exception (#536)
• SQL: simplified symbol table access from record cursors (#530)
• SERVER: fixed issue with bad request getting stuck in infinite loop
• SERVER: removed gc logging that was causing issues in macOS
• PACKAGE:fixed name of directory inside of 'no-jre' archive to match name of the archive

Release highlights

• We have introduced TCP receiver for Influx Line Protocol. It allows you to send to QuestDB quickly, easily and reliably.
• We have focused on SQL, added no-op group by clause, fixed numerous SQL bugs and added set operations except and intersect.

What is new?

• TCP ILP: tcp influx line protocol implementation
• HTTP: HTTP/1.0 support to enable server use behind SSH tunnel
• SQL: set operations: 'intersect' and 'except' (#518)
• SQL: add no-op 'group by' clause. Fixed #464 (#517)
• SQL: faster round() impl when scale is zero
• SQL: interrupting non-keyed aggregation
• Server: print web console URLs on startup. Fixed #462
• Server: added highly abstract and anonymous telemetry
• Web Console: improve syntax highlighting + harmonise with v2 documentation

What we fixed

• PostgreSQL: psychopg2 to connect and execute raw SELECT
• PostgreSQL: fixed tag in response to INSERT execution
• Docker: add --add-exports flag to Docker base command - fixed #475
• Web Console: improve query picker styles (#438)
• Web Console: remove columns reset when collapsing table - fixes #448
• SQL: speed up sym != value
• SQL: text Function !~ doesn't work when used with single character. Fixed #505
• SQL: fixed "invalid column" message in where clause after group-by
• SQL: reader was failing to reload correctly after it missed over 2 partition deletes in row
• SQL: fixed SQL error for a union of queries without 'from' clause (#513) Vlad Ilyushchenko* 22/07/2020 08:26
• SQL: memory leak in DirectLongList. Refactored configurations to have "default" class for each configuration element
• SQL: long256 to string conversion
• SQL: bug where 'where' clause in sub-query is dropped. Fixed #484
• SQL: fixed internal error while aggregating on hour with where clause
• SQL: multiple cross join was producing errors
• SQL: fixed exception when selecting distinct value of function
• SQL: fixed inconsistent output of group-by SQLs
• SQL: log web console URLs. Fixed #462
• HTTP: escape ascii 0 character in JSON
• HTTP: NPE fix when URL is null (url abuse)
• HTTP: re-throw flow control exceptions from JsonProcessor, prevent server from not-responding to HTTP client
• HTTP: log fd in StaticContentProcessor
• SERVER: server loads function factories only once per instance, rather than once per thread as before
• SERVER: do not copy config when starting with '-f'
• SERVER: sensible reader/writer idle TTL values
• SERVER: added gc log file
• SERVER: bind error on server startup is no longer displayed as an exception
• BUILD: maven test runner (surefire) was picking up default encoding and thus failing tests

What we fixed

• Web Console: fix Chrome handling MouseLeave events on disabled buttons
• Web Console: improve responsiveness of the QueryPicker component
• Web Console: add missing horizontal scrollbar on the grid
• Web Console: add syntax highlighting for some keywords/functions
• Web Console: remove unused resources to reduce bundle size
• Web Console: add extra checks on TouchEvent to prevent issues on Firefox (fixed #410)
• Web Console: polyfill ResizeObserver for older browsers
• SQL: outer join with AS OF join did not apply filter correctly
• SQL: prevent invalid characters in column name when creating table or adding column. Fixed #412

What is new

• Web Console: add icon for indexed columns
• SQL: SHOW COLUMNS FROM ... shows more information on columns
• SQL: added configurable limits to the max number of pages in VirtualMemory (#411)
• SQL: support rename column. fixed #223
• SQL: alter system lock/unlock writer

What is new ?

Migrated to Java 11 (#272). Fixed #154
SQL: vectorized group by hour(timestamp) (#398)
SQL: cancel active read-only queries on client disconnect; added resource constraints (#396)
SQL: vectorized count() execution (#382)
SQL: vector aggregation for time range searches (#380)
SQL: Add all_tables() and table_columns(...) functions (#337)
SQL: show tables and show columns (#334)
SQL: less than join implementation (lt join)
SQL: implement negated and commutative boolean function factories (#266)
SQL: limiting impact of joins on hardware with configuration
SQL: Limit the number of rows returned in HTTP requests (http.security.max.response.rows) (#347)
SQL: improved accuracy of round() by more consistent use of tolerance. The latter is adjusted to be lower. JsonQueryProcessor now prints doubles up to scale of 12. (#301)
SQL: abs(double) implementation + unbranching (#318)
SQL: round() unbranching (#317)
Web console: introduce new color scheme and new layout
Web console: migrate code editor to React + overhaul query result experience (#376)
Web console: migrate grid/chart container to React (#386)
Web console: migrate Footer container to React (#339)
Web console: add custom layout for small screens (#389)
Web console: disable "Import" section when readOnly mode is on (#357)
Web console: add Saved Queries and support for new config file (#352)
Web console: new Schema component - fetch tables and columns from server + add tooltips (#340)
Web console: turn the logo in top left corner into a link (to questdb.io) (#328)
Web console: open documentation links in a new tab - #299 (#325)
Server: added all properties to release server.conf (#384)
Server: tracking requests and bytes per connection, tests for new configuration values, passing FD to execution context (#361)
Server: questdb can be configured to limit the number of in memory rows (#308)
Server: questdb can be configured as globally read-only

What we fixed ?

SQL: order by on column with function throws exception. #288 (#289)
SQL: order by does not validate columns correctly. #284 (#305)
SQL: order by timestamp descending bug. #287 (#306)
SQL: support single character symbol. #279 (#311)
SQL: asof join error unless timestamp is selected (#310)
SQL: support millis timestamp in import (#362)
SQL: support uppercase 'DROP', 'ADD', 'ALTER' in ALTER TABLE. #275 (#276)
SQL: GORM sends ';' after login, making sure we reply to that. (#388)
SQL: PSQL port in docker container and documentation (8892 -> 8812). (#271)
SQL: where clause outside of join sub-query was discarded (failed to properly move inside of sub-query) (#387)
SQL: alter table add column symbol does not respect cached flag - #374 (#385)
SQL: insert as select without nominating timestamp - #364 #349 (#365)
SQL: binary search scan for timestamps (#371)
SQL: avg() ksum() nsum() sum() handling of infinity #313 (#338)
SQL: microsecond timestamp (#335)
SQL: partial/short date in where clause >=, > ,<, <= #323 #265 (#333)
SQL: timestamp search issue with partial date. Equality fix #323 (#331)
SQL: problem with asof join with where clause. also dangling where statement fix. Fixed #307 (#326)
Web console: disable warning for end of line check (#324)
Web console: remove unused deps/assets which reduces the bundle from 2.1MB to 800KB
Web console: remove unnecessary comma when inserting code in the editor

What is new ?

• SQL: Support for PostgreSQL dialect "truncate table ONLY tab"
• SQL: 20% performance gain for vector aggregations
• SQL: vector min() and max() for TIMESTAMP and DATE types
• SQL: table and database backup
• SQL: added support for quoted literals to comply with PostgreSQL dialect
• SQL: timestamp function dateadd(), datediff()
• SQL: ksum() and nsum() - vector implementation of compensated summation. Kahan and Neumaier respectively
• SQL: stable random generator for generating test data repeatedly
• SQL: copy supports 'with header' now
• Web Console: shows more query timing values
• Web Console: it is possible to specify nominated timestamp & partition type (thanks to @clickingbuttons)

What we fixed ?

• Server may fail to start when OS language setting is not EN.
• SQL: select distinct .... could have ignored the where clause
• Web Console: fixed tooltips on import buttons
• Web Console: build is now using webpack. Refer for instructions to run console here
• Web Console: fixed grid rendering issues during fast scrolling
• SQL: fixed order of where clause in latest by query
• SQL: fixed timestamp less that search
• SQL: symbol table reload did not work properly
• SQL: where clause could produce false error

Clickhouse benchmark

Hardware

c5.metal AWS instance, which has two Intel 8275CL 24-core CPUs and 192GB of memory. QuestDB was running on 16 threads pinned to NUMA zone 0. Clickhouse is using all threads (96 - hyperthreadin on)

With NULLs

Result

Without NULLs

SUMMARY

We have implemented SIMD-based vector execution of queries, such as

select sum(value) from table

This is around 100x faster than non-vector based execution. This is just the beginning. We are vectorising more operations.

Try our first implementation in this release and stay tuned for more features in the upcoming releases!

IMPORTANT

Metadata file format has been changed to include a new flag for columns of type symbol. It is necessary to convert existing tables to new format. Running the following sql: repair table myTable, will update the table metadata.

What is new?

• Java: vectorized sum(), avg(), min(), max() for DOUBLE, LONG, INT
• Java: select distinct symbol optimisation
• FreeBSD support
• Automatically restore data consistency and recover from partial data loss.

What we fixed

• SQL: NPE when parsing SQL text with malformed table name expression , for example ')', or ', blah'
• SQL: parsing 'fill' clause in sub-query context was causing unexpected syntax error (#115)
• SQL: possible internal error when ordering result of group-by or sample-by
• Data Import: Ignore byte order marks (BOM) in table names created from an imported CSV (#114)
• SQL: 'timestamp' propagation thru group-by code had issues. sum() was tripping over null values. Added last() aggregate function. (#113)
• LOG: make service log names consistent on windows (#106)
• SQL: deal with the following syntax 'select * from select ( select a from ....)'
• SQL: allow the following syntax 'case cast(x as int) when 1 then ...'
  fix(griffin): syntax check for "case"-')' overlap, e.g. "a + (case when .. ) end"

IMPORTANT

Metadata file format has been changed to include version and more column flags. It is necessary to convert existing tables to new format. For more information please follow this guide.

What is new?

• Java: TableWriter.addIndex() implementation - adds index to existing table
• SQL: CASE statement implementation
• SQL: CAST implementation (documentation)
• SQL: RENAME TABLE implementation
• SQL: round() multiple implementations (documentation)
• SQL: ALTER TABLE X ALTER COLUMN SYM ADD INDEX - SQL to add index to SYMBOL columns of existing tables (documentation)
• SQL: symbol in (list) function implementation
• SQL: optimiser improvements
• UI: Web Console has simple Charts implementation

select * from trades where ts>'2015-01-02T00:00:00.000Z' and ts<'2015-01-02T10:30:00.000Z' and sym in ('EEM', 'VXX', 'BCLI', ...) order by sym

What we fixed

• SQL: double-to-string implementation parses IEEE number, performance is up 80% on simpler values and 30% on complex ones

• SQL: aggregation without key, e.g. no 'group by' clause are 6x faster than before.

• SQL: 'SAMPLE BY' queries are faster than before

• SQL: 'COPY' is limited to picking up files only from configured location (documentation)

• SQL: improved error reporting

• UI: LONG256 0 is printed as '0x00'

• UI: Double "inifinity" values (division by zero) used to corrupt JSON

Release highlights

• production ready version of Influx Line Protocol Receiver. It works well with existing line protocol publishers and is much faster than Influx's own implementation. Refer to this how-to

• choice of transaction durability

What's new ?

• GRIFFIN: 'asof' join performs much faster
• GRIFFIN: systimestamp() SQL function
• line protocol receiver supports abbreviated Boolean values, such as 'T', 'F' and capitalised TRUE and FALSE
• line protocol receiver 30% performance boost
• line protocol receiver supports all of InfluxDB's timestamp formats
• CAIRO: durable commits, support for NOSYNC, ASYNC and SYNC as documented [here]
• HTTP: improved performance of JSON HTTP end point by 100% - SQL results are serialised and delivered to browser faster.
• Web Console: grid headers can be clicked to help construction of column list in 'select' clause

What's fixed ?

• line protocol receiver could fail on field values that have space
• GRIFFIN: INSERT statement could have incorrectly referenced column names of the target table and fail uncontrollably
• GRIFFIN: column name in timestamp() clause wasn't validate leading to uncontrollable failure
• CAIRO: UTF8 encoding on Windows. Although QuestDB was unaffected, directory names containing UTF8 characters were garbled.
• Web Console: highlighted more SQL keywords
• Web Console: removed unfinished "visualisation" tab
• Web Console: trim SQL statement before sending to server. Cleaner logging on server.
• HTTP: headers are now case insensitive
• HTTP: NPE when 'content-disposition' header is missing
• improved hash function distribution leading to overall performance boost by 10%
• windows docker image is smaller. Reduced the size of JRE package

Download

Via Docker or cross-platform binary below: