What we fixed?

• removed accidental garbage generation via lambdas in PG wire implementation
• reduced size of binaries
• error handling from INSERT statement execution over PG wire
• OS error reporting out of network interface
• optimised ASOF join for non-keyed joins
• cached SQL plan re-evaluation when table structure changes

What's new?

• Influx UDP receiver can either use shared threads or its own busy thread via configuration

What we fixed?

• import single column text file #91
• 'insert' bugfixes
• switched to parallel GC to improve performance across the board. CMS we used to use, although not running explicitly, was slowing down our code.
• NPE in TableReader under some conditions

What is new?

• Influx line-protocol receiver is started by default when server starts
• 'alter table drop partition' implementation
• to_long(TIMESTAMP) implementation (thanks to clickingbuttons)
• 'insert' implementation over PostgreSQL wire

What's new?

• docker files for Windows, Linux and ARM64 Linux

What did we fix?

• compatibility with Java 11
• #90
• print architecture ID as recognised by application
• #84
• ANSI SQL compatible count(*) support
• #88

Welcome to QuestDB 4.0.0 Release Notes

Before you continue, some keywords we used and their meaning:

CAIRO - storage engine
GRIFFIN - SQL engine
STD - QuestDB standard libraries

Whats new?

• PostgresSQL wire server implementation (BETA)

• GRIFFIN: new algo for sample by .. fill none without key. It executes a little faster than generic algo.

• GRIFFIN: order by implementation for cursors without random access to rows.

• GRIFFIN: group by implementation

• GRIFFIN: support multiple columns in latest by clause

• GRIFFIN: renamed to_char for date and timestamp to to_str to make them easier to find

• GRIFFIN: optimised and and or operators to recognise constant conditions better and avoid unnecessary data processing.

• GRIFFIN: = implementation for SYMBOL = CHAR and STRING = CHAR

• GRIFFIN: 'sum' performance optimisation

• Network: ability to set multicast TTL and bind port for unicast

• GRIFFIN: "insert into ... values" implementation

• GRIFFIN: implementation of 'select count() from table' that would use result of base.size() when its available. This speed up exploratory 'count()' queries dramatically

• GRIFFIN: UNION and UNION ALL implementations and tests

• GRIFFIN: "splice join" implementation

• GRIFFIN: alter table drop column SQL implementation

• GRIFFIN: SQL limit implementation

• GRIFFIN: INNER, OUTER, AS OF join implementation

• CAIRO: "remove partition" API on TableWriter

• CAIRO: new LONG256 data type

• STD: ARM64 Linux support and binary

• HTTP server implementation. The server is able to send and receive unlimited amount of data from and to data store.

• STD: Os.currentTimeNanos() system clock in nanosecond resolution (not to be confused with System.nanoTime())

• STD: (network) set/get TCP_NODELAY

• STD: (network) implementation of interrupt for blocking accept()

• STD: (network) bridge to native IP_MULTICAST_LOOP and IP_MULTICAST_IF

• STD: (network) connect() and configureNoLinger() bridges

• STD: (network) SO_REUSEADDR and SO_REUSEPORT bridges

• STD: (log) implemented log writer that can roll files on size and/or date.

• LineProtoSender can now be bound to a specific interface. It also receives IP addresses as inti n order to offload dealing with address parsing errors somewhere else.

• STD: (threading) available() method is made available only on SP- and SC- sequences. available() is new to SPSequence and allows bulk publishing and enables higher queue throughput STD: Worker can have CPU affinity now

Whats changed?

• new top level package name! This is a breaking change. We wanted to make package name consistent with our product web site.

What did we fix?

• logging OS error code whenever close(fd) fails.
• null for symbol value now corresponds to null for int. This removes unnecessary complexity around implementing getInt()
• fixed transactional behaviour of TableWriter.truncate(). TableReader will now process this event correctly.
• CAIRO: TableWriter maintains minTimestamp so that TableReader doesn't have to count directories to determine data interval
• CAIRO: TableReader no longer relies on partition directories to be removed in order to remove logical data chunk. This is necessary for Windows where open files cannot be removed.
• CAIRO: ByteSequence could allow reading beyond the size of the blob.
• get/set lastModified on Windows could set incorrect time
• fixed TextImportProcessor resume function
• Files.exists() does not use Files.getLastModified() anymore
• network send() and recv() on Linux did not handle errors correctly, which sometimes results in broken wire protocols.
• Numbers.parseInt() could overflow without throwing exception
• Removed garbage-generating lambdas.
• CAIRO: SymbolTable reload could produce an exception

Beta release of new storage system dubbed CAIRO and new query system GRIFFIN.

CAIRO improves on now legacy system in several key areas:

1. Performance. It is at least 30% faster in every test.
2. Resilience. Cairo has been tested to withstand file system errors, such as corruption, space issue as well deal hardware stability issues, such as power loss or sudden reboots. Cairo will perform automatic data recovery in all of these circumstances
3. Data management. Cairo implements functions of adding and removing table columns and partition management.
4. Stability. Cairo is production ready. Data structures have been extensively tested to resolve all known defects.

Downsides:

1. Object store and retrieval is not implemented against CAIRO. The system focus is on zero-GC embedded storage.

GRIFFIN - SQL implementation that uses CAIRO as the store.

This system is under active development. For now lacks features of legacy SQL, such as joins, group-by and analytic functions - these are not yet developed. However GRIFFIN provides sample-by implementation with various missing data fill options, including linear interpolation and more extensive language basics, such as select from function. Eventually GRIFFIN will implement all features of legacy SQL and well beyond that.

GRIFFIN also is not yet linked to HTTP server or any network service.

Release notes

• BUGFIX: microsecond timer also works on pre-Windows 8 systems
• BUGFIX: race condition in CAIRO WriterPool
• BUGFIX: microsecond time returned incorrect time
• OS-specific binary is loaded implicitly when required
• CAIRO engine performance improvement

This release introduces new storage system, CAIRO. I am writing it to run in parallel with legacy storage and keep existing functionality completely unaffected. CAIRO is available to try if you look for any of the following:

• easy DDL: add/remove column, create/rename/drop table etc. These operations are either possible or much faster than before.
• dedicated writer and reader implementations. Writer reuses resources very aggressively, caches very little and is double the speed of legacy implementation
• CAIRO is tested for every possible outcome of file system operation, it does not corrupt data under any circumstances, attempts to recover automatically and concisely reports errors
• commits are much faster, there is very little performance difference between committing batches and every record
• i18n support out of box, table names and column names accept UTF8 encoded characters

CAIRO is not yet integrated with neither HTTP server nor SQL. These are the things to come.

Also included in this release CUTLASS: UDP receiver for Influx line protocol. Everything broadcast for Influx can be easily stored in QuestDB. CUTLASS supports on-the-fly table definition and column addition and it stores data in CAIRO.

Stay tuned for more exciting releases ;-)

configuration file bugfix

Bugfix release, stability improvements.

Bugfixes

• fixed windows distribution (questdb.exe)
• incorrect default home directory on Linux
• memory page size used by variance and stddev functions moved to config
• renamed package for text parser

QuestDB initial release.

This release brings technology focused on making your data workflow faster, more productive and more fun.

• install QuestDB in seconds
• import any amount of your data via copy-paste or drag-n-drop via Web browser
• make sense of your data directly from browser via high-performance SQL engine

QuestDB does in seconds what other tools do in minutes, hours, days or don't do at all.

New features

SSL encryption.

Replication can be done over secure channel using industry-standard SSL encryption protocol. Configuring SSL is notoriously complicated, so I tried my best to make it as simple as possible for both JournalServer and JournalClient. I shall let you be the judge though. More...

SSL authentication.

Client Certificate came almost for free after implementing SSL. I did try to make it simple for you though. More...

Shared secret authentication.

It is now possible for client and server to exchange shared secret for authentication and authorization purposes. Shared Secret can be anything you want it to be, there is no limit on amount of data you can send. More...

Kerberos authentication.

Usually Kerberos is a pain, not with NFSdb, not anymore. Kerberos implementation uses Shared Secret exchange mechanism and in addition it handles generation and validation of service tokens, Your application deals with usernames. It is that cool! More...

Kerberos integrated Windows authentication.

If you have JournalClient running on Windows platform you can generate service token for currently logged in user with minimum fuss. User is not prompted for password! More...

JournalClient auto-reconnect

JournalClient can automatically recover after server restarts. If you have multiple clients replicating single server you can restart server without needing to restart clients afterwards. Whats cooler still is that when you shutdown server clients can failover automatically. JournalClient will also maintain transactional consistency during server outages.

Compatibility warning

Replication protocol is incompatible with previous release of NFSdb. I had to change the protocol to accomodate shared secret exchange.

Installation

<dependency>
    <groupId>com.nfsdb</groupId>
    <artifactId>nfsdb-core</artifactId>
    <version>2.0.1</version>
</dependency>

Change log

• #10 Support for sublasses
• #16 Fixed defect in BulkWriter that could cause index data loss
• Reader creates new journal if one does not exist instead of throwing exception.
• #15 Unique key support
• #4 Data increment processing

Unique key support

Unique keys can be either String or int fields. In fact it is just an indexed field that you can search.

Configure int "id" field as follows:

JournalFactory factory = new JournalFactory(new JournalConfigurationBuilder() {{
    $(Order.class)
        .$int("id").index()
        // or .$str("id).index() for String keys
    ;
}}.build(args[0]));

To prepare for search by "id" create parameterisable query stream. It is analogous to PreparedStatement.

Journal<Order> reader = factory.reader(Order.class);
Order order = new Order();
Q q = new QImpl();

StringRef col = new StringRef();
col.value = "id";
IntRef id = new IntRef();

DataSource<Order> ds = q.ds(
        q.forEachPartition(
                q.sourceDesc(reader)
                , q.headEquals(col, id)
        )
        , order
);

Execute query for given "id":

id.value = i;
Order o = ds.$new().head();

Data increment processing

On client have a reader for journal that is being replicated. increment() and incrementBuffered() create iterator over added data.

This is fully featured client that measures transaction latency.

public static void main(String[] args) throws Exception {
    JournalFactory factory = new JournalFactory(args[0]);
    final JournalClient client = new JournalClient(factory);

    final Journal<Price> reader = factory.bulkReader(Price.class, "price-copy");

    client.subscribe(Price.class, null, "price-copy", new TxListener() {
        @Override
        public void onCommit() {
            int count = 0;
            long t = 0;
            for (Price p : reader.incrementBuffered()) {
                if (count == 0) {
                    t = p.getTimestamp();
                }
                count++;
            }
            System.out.println("took: "
                            + (System.currentTimeMillis() - t) 
                            + ", count=" + count);
        }
    });
    client.start();
    System.out.println("Client started");
}

This release introduces two major features:

• network component to NFSdb, enabling journal replication over TCP/IP with automated multicast discovery.
• java based configuration replaced venerable nfsdb.xml

This nfsdb.xml:

<db>
    <journal class="org.nfsdb.examples.model.Quote" defaultPath="quote" timestampColumn="timestamp"
             partitionType="MONTH" recordCountHint="1000000" openPartitionTTL="180" lagHours="24" key="sym">
        <sym name="sym" indexed="true" maxsize="4" hintDistinctCount="15"/>
        <sym name="ex" maxsize="2" hintDistinctCount="1"/>
        <sym name="mode" hintDistinctCount="1"/>
    </journal>

    <journal class="org.nfsdb.examples.model.Price" defaultPath="price" timestampColumn="timestamp"
             partitionType="MONTH" recordCountHint="1000000" openPartitionTTL="180" lagHours="24" key="sym">
        <sym name="sym" indexed="true" maxsize="4" hintDistinctCount="15"/>
    </journal>
</db>

is replaced with this configuration:

    public static final JournalConfigurationBuilder CONFIG = new JournalConfigurationBuilder() {{
        $(Quote.class)
                .partitionBy(PartitionType.MONTH)
                .lag(24, TimeUnit.HOURS)
                .key("sym")
                .$sym("sym").index().size(4).valueCountHint(15)
                .$sym("ex").size(2).valueCountHint(1)
                .$ts()
        ;

        $(Price.class)
                .key("sym")
                .partitionBy(PartitionType.MONTH)
                .$sym("sym").index().size(4).valueCountHint(15)
                .$ts()
        ;
    }};

Changelog:

• added TCP/IP replication
• added java-based configuration
• zero-configuration support for all components to simplify API for light usage.
• API refactoring (simplification) breaks code compatibility with earlier versions. SymbolIndex class is renamed to KVIndex. SymbolIndex.put() is replaced with KVIndex.add()
• Broken binary compatibility with previous versions for Strings. String length now always 4 bytes, which improves performance. Older journals would need to be converted.
• improved index performance
• improved append performance
• revised size calculations to make journals take less space on disk

• thrift support is optional. NFSdb works well with POJOs.
• durable transaction support (JournalWriter.commitDurable())
• Journal.select() to chose active columns. This saves a lot of unnecessary disk reading.
• Queries for invalid symbols return empty result set instead of throwing exceptions
• 5% performance improvement

NFSdb 1.0.2

• bulk writer and reader to enable access to journals that do not fit available RAM
• increased speed of append and read (in tests 160ms read and 500ms write of 1,000,000 records)
• increased speed and reduced memory footprint of appendLag() method.
• increased performance of commit() method

First release of NFSdb to maven central.