druid

Apache Druid: a high performance real-time analytics database.

APACHE-2.0 License

Stars
13.4K
Committers
639
View Code on GitHub Visit Website
Ecosystems: Groovy, Apache Cassandra, Cordova, Maven, Apache Spark

Bot releases are hidden (Show)

druid - druid-0.18.0

Published by jihoonson over 4 years ago

Apache Druid 0.18.0 contains over 200 new features, performance enhancements, bug fixes, and major documentation improvements from 42 contributors. Check out the complete list of changes and everything tagged to the milestone.

# New Features

# Join support

Join is a key operation in data analytics. Prior to 0.18.0, Druid supported some join-related features, such as Lookups or semi-joins in SQL. However, the use cases for those features were pretty limited and, for other join use cases, users had to denormalize their datasources when they ingest data instead of joining them at query time, which could result in exploding data volume and long ingestion time.

Druid 0.18.0 supports real joins for the first time ever in its history. Druid supports INNER, LEFT, and CROSS joins for now. For native queries, the join datasource has been newly introduced to represent a join of two datasources. Currently, only the left-deep join is allowed. That means, only a table or another join datasource is allowed for the left datasource. For the right datasource, lookup, inline, or query datasources are allowed. Note that join of Druid datasources is not supported yet. There should be only one table datasource in the same join query.

Druid SQL also supports joins. Under the covers, SQL join queries are translated into one or several native queries that include join datasources. See Query translation for more details of SQL translation and best practices to write efficient queries.

When a join query is issued, the Broker first evaluates all datasources except for the base datasource which is the only table datasource in the query. The evaluation can include executing subqueries for query datasources. Once the Broker evaluates all non-base datasources, it replaces them with inline datasources and sends the rewritten query to data nodes (see the below "Query inlining in Brokers" section for more details). Data nodes use the hash join to process join queries. They build a hash table for each non-primary leaf datasource unless it already exists. Note that only lookup datasource currently has a pre-built hash table. See Query execution for more details about join query execution.

Joins can affect performance of your queries. In general, any queries including joins can be slower than equivalent queries against a denormalized datasource. The LOOKUP function could perform better than joins with lookup datasources. See Join performance for more details about join query performance and future plans for performance improvement.

https://github.com/apache/druid/pull/8728
https://github.com/apache/druid/pull/9545
https://github.com/apache/druid/pull/9111

# Query inlining in Brokers

Druid is now able to execute a nested query by inlining subqueries. Any type of subquery can be on top of any type of another, such as in the following example:

             topN
               |
       (join datasource)
         /          \
(table datasource)  groupBy

To execute this query, the Broker first evaluates the leaf groupBy subquery; it sends the subquery to data nodes and collects the result. The collected result is materialized in the Broker memory. Once the Broker collects all results for the groupBy query, it rewrites the topN query by replacing the leaf groupBy with an inline datasource which has the result of the groupBy query. Finally, the rewritten query is sent to data nodes to execute the topN query.

# Query laning and prioritization

When you run multiple queries of heterogenous workloads at a time, you may sometimes want to control the resource commitment for a query based on its priority. For example, you would want to limit the resources assigned to less important queries, so that important queries can be executed in time without being disrupted by less important ones.

Query laning allows you to control capacity utilization for heterogeneous query workloads. With laning, the broker examines and classifies a query for the purpose of assigning it to a 'lane'. Lanes have capacity limits, enforced by the Broker, that can be used to ensure sufficient resources are available for other lanes or for interactive queries (with no lane), or to limit overall throughput for queries within the lane.

Automatic query prioritization determines the query priority based on the configured strategy. The threshold-based prioritization strategy has been added; it automatically lowers the priority of queries that cross any of a configurable set of thresholds, such as how far in the past the data is, how large of an interval a query covers, or the number of segments taking part in a query.

See Query prioritization and laning for more details.

https://github.com/apache/druid/issues/6993
https://github.com/apache/druid/pull/9407
https://github.com/apache/druid/pull/9493

New dimension in query metrics

Since a native query containing subqueries can be executed part-by-part, a new subQueryId has been introduced. Each subquery has different subQueryIds but same queryId. The subQueryId is available as a new dimension in query metrics.

New configuration

A new druid.server.http.maxSubqueryRows configuration controls the maximum number of rows materialized in the Broker memory.

Please see Query execution for more details.

https://github.com/apache/druid/pull/9533

# SQL grouping sets

GROUPING SETS is now supported, allowing you to combine multiple GROUP BY clauses into one GROUP BY clause. This GROUPING SETS clause is internally translated into the groupBy query with subtotalsSpec. The LIMIT clause is now applied after subtotalsSpec, rather than applied to each grouping set.

https://github.com/apache/druid/pull/9122

# SQL Dynamic parameters

Druid now supports dynamic parameters for SQL. To use dynamic parameters, replace any literal in the query with a question mark (?) character. These question marks represent the places where the parameters will be bound at execution time. See SQL dynamic parameters for more details.

https://github.com/apache/druid/pull/6974

# Important Changes

# applyLimitPushDownToSegments is disabled by default

applyLimitPushDownToSegments was added in 0.17.0 to push down limit evaluation to queryable nodes, limiting results during segment scan for groupBy v2. This can lead to performance degradation, as reported in https://github.com/apache/druid/issues/9689, if many segments are involved in query processing. This is because “limit push down to segment scan” initializes an aggregation buffer per segment, the overhead for which is not negligible. Enable this configuration only if your query involves a relatively small number of segments per historical or realtime task.

https://github.com/apache/druid/pull/9711

# Roaring bitmaps as default

Druid supports two bitmap types, i.e., Roaring and CONCISE. Since Roaring bitmaps provide a better out-of-box experience (faster query speed in general), the default bitmap type is now switched to Roaring bitmaps. See Segment compression for more details about bitmaps.

https://github.com/apache/druid/pull/9548

# Complex metrics behavior change at ingestion time when SQL-compatible null handling is disabled (default mode)

When SQL-compatible null handling is disabled, the behavior of complex metric aggregation at ingestion time has now changed to be consistent with that at query time. The complex metrics are aggregated to the default 0 values for nulls instead of skipping them during ingestion.

https://github.com/apache/druid/pull/9484

# Array expression syntax change

Druid expression now supports typed constructors for creating arrays. Arrays can be defined with an explicit type. For example, <LONG>[1, 2, null] creates an array of LONG type containing 1, 2, and null. Note that you can still create an array without an explicit type. For example, [1, 2, null] is still a valid syntax to create an equivalent array. In this case, Druid will infer the type of array from its elements. This new syntax applies to empty arrays as well. <STRING>[], <DOUBLE>[], and <LONG>[] will create an empty array of STRING, DOUBLE, and LONG type, respectively.

https://github.com/apache/druid/pull/9367

# Enabling pending segments cleanup by default

The pendingSegments table in the metadata store is used to create unique new segment IDs for appending tasks such as Kafka/Kinesis indexing tasks or batch tasks of appending mode. Automatic pending segments cleanup was introduced in 0.12.0, but has been disabled by default prior to 0.18.0. This configuration is now enabled by default.

https://github.com/apache/druid/pull/9385

# Creating better input splits for native parallel indexing

The Parallel task now can create better splits. Each split can contain multiple input files based on their size. Empty files will be ignored. The split size is controllable with the new split hint spec. See Split hint spec for more details.

https://github.com/apache/druid/pull/9360
https://github.com/apache/druid/pull/9450

# Transform is now an extension point

Transform is an Interface that represents a transformation to be applied to each row at ingestion time. This interface is now an Extension point. Please see Writing your own extensions for how to add your custom Transform.

https://github.com/apache/druid/pull/9319

# chunkPeriod query context is removed

chunkPeriod has been deprecated since 0.14.0 because of its limited usage (it was sometimes useful for only groupBy v1). This query context is now removed in 0.18.0.

https://github.com/apache/druid/pull/9216

# Experimental support for Java 11

Druid now experimentally supports Java 11. You can run the same Druid binary distribution with Java 11 which is compiled with Java 8. Our tests on Travis include:

  • Compiling and running unit tests with Java 11
  • Compiling with Java 8 and running integration tests with Java 11

Performance testing results are not available yet.

Warnings for illegal reflective accesses when running Druid with Java 11

Since Java 9, it issues a warning when it is found that some libraries use reflection to illegally access internal APIs of the JDK. These warnings will be fixed by modifying Druid codes or upgrading library versions in future releases. For now, these warnings can be suppressed by adding JVM options such as --add-opens or --add-exports. See JDK 11 Migration Guide for more details.

Some of the warnings are:

2020-01-22T21:30:08,893 WARN [main] org.apache.druid.java.util.metrics.AllocationMetricCollectors - Cannot initialize org.apache.druid.java.util.metrics.AllocationMetricCollector
java.lang.reflect.InaccessibleObjectException: Unable to make public long[] com.sun.management.internal.HotSpotThreadImpl.getThreadAllocatedBytes(long[]) accessible: module jdk.management does not "exports com.sun.management.internal" to unnamed module @6955cb39

This warning can be suppressed by adding --add-exports jdk.management/com.sun.management.internal=ALL-UNNAMED.

2020-01-22T21:30:08,902 WARN [main] org.apache.druid.java.util.metrics.JvmMonitor - Cannot initialize GC counters. If running JDK11 and above, add --add-exports java.base/jdk.internal.perf=ALL-UNNAMED to the JVM arguments to enable GC counters.

This warning can be suppressed by adding --add-exports java.base/jdk.internal.perf=ALL-UNNAMED.

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

This warning can be suppressed by adding --add-opens java.base/java.lang=ALL-UNNAMED.

https://github.com/apache/druid/pull/7306
https://github.com/apache/druid/pull/9491

# New Extension

# New Pac4j extension

A new extension is added in 0.18.0 to enable OpenID Connect based Authentication for Druid Processes. This can be used with any authentication server that supports same e.g. Okta. This extension should only be used at the router node to enable a group of users in existing authentication server to interact with Druid cluster, using the Web Console.

https://github.com/apache/druid/pull/8992

# Security Issues

# [CVE-2020-1958] Apache Druid LDAP injection vulnerability

CVE-2020-1958 has been reported recently and fixed in 0.18.0 and 0.17.1. When LDAP authentication is enabled, callers of Druid APIs can bypass the credentialsValidator.userSearch filter barrier or retrieve any LDAP attribute values of users that exist on the LDAP server, so long as that information is visible to the Druid server. Please see the description in the link for more details. It is strongly recommended to upgrade to 0.18.0 or 0.17.1 if you are using LDAP authentication with Druid.

https://github.com/apache/druid/pull/9600

# Updating Kafka client to 2.2.2

Kafka client library has been updated to 2.2.2, in which CVE-2019-12399 is fixed.

https://github.com/apache/druid/pull/9259

# Bug fixes

Druid 0.18.0 includes 40 bug fixes. Please see https://github.com/apache/druid/pulls?page=1&q=is%3Apr+milestone%3A0.18.0+is%3Aclosed+label%3ABug for the full list of bug fixes.

# Upgrading to Druid 0.18.0

Be aware of the following changes between 0.17.1 and 0.18.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.17.1, please see the release notes of the relevant intermediate versions.

# S3 extension

The S3 storage extension now supports cleanup of stale task logs and segments. When deploying 0.18.0, please ensure that your extensions directory does not have any older versions of druid-s3-extensions extension.

https://github.com/apache/druid/pull/9459

# Core extension for Azure

The Azure storage extension has been promoted to a core extension. It also supports cleanup of stale task logs and segments now. When deploying 0.18.0, please ensure that your extensions-contrib directory does not have any older versions of druid-azure-extensions extension.

https://github.com/apache/druid/pull/9394
https://github.com/apache/druid/pull/9523

# Google Storage extension

The Google storage extension now supports cleanup of stale task logs and segments. When deploying 0.18.0, please ensure that your extensions directory does not have any older versions of druid-google-extensions extension.

https://github.com/apache/druid/pull/9519

# Hadoop AWS library included in binary distribution

Hadoop AWS library is now included in the binary distribution for better out-of-box experience. When deploying 0.18.0, please ensure that your hadoop-dependencies directory or any other directories in the classpath does not have duplicate libraries.

# PostgreSQL JDBC driver for Lookups included in binary distribution

PostgreSQL JDBC driver for Lookups is now included in the binary distribution for better out-of-box experience. When deploying 0.18.0, please ensure that your extensions/druid-lookups-cached-single directory or any other directories in the classpath does not have duplicate JDBC drivers.

https://github.com/apache/druid/pull/9399

# Known Issues

# Query failure with topN or groupBy on scan with multi-valued columns

Query inlining in Brokers is newly introduced in 0.18.0 but has a bug that queries with topN or groupBy on top of scan fail if the scan query selects multi-valued dimensions. See https://github.com/apache/druid/issues/9697 for more details.

# NullPointerException when using Avro parser with Kafka indexing service.

Avro parser doesn't work with Kafka indexing service because of a wrong null check. See https://github.com/apache/druid/issues/9728 for more details.

# Misleading segment/unavailable/count metric during handoff

This metric is supposed to take the number of segments served by realtime tasks into consideration as well, but it isn't now. As a result, it appears that unavailability spikes up before the new segments are loaded by historicals, even if all segments actually are continuously available on some combination of realtime tasks and historicals.

https://github.com/apache/druid/issues/9677

# Slight difference between the result of explain plan for query and the actual execution plan

The result of explain plan for can be slightly different from what Druid actually executes when the query includes joins or subqueries. The difference can be found in that each part of the query plan would be represented as if it was its own native query in the result of explain plan for. For example, for a join of a datasource d1 and a groupBy subquery on datasource d2, the explain plan for could return a plan like below

     join
    /    \
scan    groupBy
 |        |
d1       d2

whereas the actual query plan Druid would execute is

     join
    /    \
  d1    groupBy
          |
         d2

# Other known issues

For a full list of open issues, please see https://github.com/apache/druid/labels/Bug.

# Credits

Thanks to everyone who contributed to this release!

@a2l007
@abhishekrb19
@aditya-r-m
@AlexanderSaydakov
@als-sdin
@aP0StAl
@asdf2014
@benhopp
@bjozet
@capistrant
@Caroline1000
@ccaominh
@clintropolis
@dampcake
@fjy
@Fokko
@frnidito
@gianm
@himanshug
@JaeGeunBang
@jihoonson
@jon-wei
@JulianJaffePinterest
@kou64yama
@lamber-ken
@leventov
@liutang123
@maytasm
@mcbrewster
@mgill25
@mitchlloyd
@mrsrinivas
@nvolungis
@prabcs
@samarthjain
@sthetland
@suneet-s
@themaric
@vogievetsky
@xvrl
@zachjsh
@zhenxiao

druid - druid-0.17.1

Published by jon-wei over 4 years ago

Apache Druid 0.17.1 is a security bug fix release that addresses the following CVE for LDAP authentication:

druid - druid-0.17.0

Published by jon-wei over 4 years ago

Apache Druid 0.17.0 contains over 250 new features, performance enhancements, bug fixes, and major documentation improvements from 52 contributors. Check out the complete list of changes and everything tagged to the milestone.

Highlights

Batch ingestion improvements

Druid 0.17.0 includes a significant update to the native batch ingestion system. This update adds the internal framework to support non-text binary formats, with initial support for ORC and Parquet. Additionally, native batch tasks can now read data from HDFS.

This rework changes how the ingestion source and data format are specified in the ingestion task. To use the new features, please refer to the documentation on InputSources and InputFormats.

Please see the following documentation for details:
https://druid.apache.org/docs/0.17.0/ingestion/data-formats.html#input-format
https://druid.apache.org/docs/0.17.0/ingestion/native-batch.html#input-sources
https://druid.apache.org/docs/0.17.0/ingestion/native-batch.html#partitionsspec

https://github.com/apache/druid/issues/8812

Single dimension range partitioning for parallel native batch ingestion

The parallel index task now supports the single_dim type partitions spec, which allows for range-based partitioning on a single dimension.

Please see https://druid.apache.org/docs/0.17.0/ingestion/native-batch.html for details.

Compaction changes

Parallel index task split hints

The parallel indexing task now has a new configuration, splitHintSpec, in the tuningConfig to allow for operators to provide hints to control the amount of data that each first phase subtask reads. There is currently one split hint spec type, SegmentsSplitHintSpec, used for re-ingesting Druid segments.

Parallel auto-compaction

Auto-compaction can now use the parallel indexing task, allowing for greater compaction throughput.

To control the level of parallelism, the auto-compactiontuningConfig has new parameters, maxNumConcurrentSubTasks and splitHintSpec.

Please see https://druid.apache.org/docs/0.17.0/configuration/index.html#compaction-dynamic-configuration for details.

https://github.com/apache/incubator-druid/pull/8570

Stateful auto-compaction

Auto-compaction now uses the partitionSpec to track changes made by previous compaction tasks, allowing the coordinator to reduce redundant compaction operations.

Please see https://github.com/apache/druid/issues/8489 for details.

If you have auto-compaction enabled, please see the information under "Stateful auto-compaction changes" in the "Upgrading to Druid 0.17.0" section before upgrading.

Parallel query merging on brokers

The Druid broker can now opportunistically merge query results in parallel using multiple threads.

Please see druid.processing.merge.useParallelMergePool in the Broker section of the configuration reference for details on how to configure this new feature.

Parallel merging is enabled by default (controlled by the druid.processing.merge.useParallelMergePool property), and most users should not have to change any of the advanced configuration properties described in the configuration reference.

Additionally, merge parallelism can be controlled on a per-query basis using the query context. Information about the new query context parameters can be found at https://druid.apache.org/docs/0.17.0/querying/query-context.html.

https://github.com/apache/incubator-druid/pull/8578

SQL-compatible null handling

In 0.17.0, we have added official documentation for Druid's SQL-compatible null handling mode.

Please see https://druid.apache.org/docs/0.17.0/configuration/index.html#sql-compatible-null-handling and https://druid.apache.org/docs/0.17.0/design/segments.html#sql-compatible-null-handling for details.

Several bugs that existed in this previously undocumented mode have been fixed, particularly around null handling in numeric columns. We recommend that users begin to consider transitioning their clusters to this new mode after upgrading to 0.17.0.

The full list of null handling bugs fixed in 0.17.0 can be found at https://github.com/apache/druid/issues?utf8=%E2%9C%93&q=label%3A%22Area+-+Null+Handling%22+milestone%3A0.17.0+

LDAP extension

Druid now supports LDAP authentication. Authorization using LDAP groups is also supported by mapping LDAP groups to Druid roles.

  • LDAP authentication is handled by specifying an LDAP-type credentials validator.
  • Authorization using LDAP is handled by specifying an LDAP-type role provider, and defining LDAP group->Druid role mappings within Druid.

LDAP integration requires the druid-basic-security core extension. Please see https://druid.apache.org/docs/0.17.0/development/extensions-core/druid-basic-security.html for details.

As this is the first release with LDAP support, and there are a large variety of LDAP ecosystems, some LDAP use cases and features may not be supported yet. Please file an issue if you need enhancements to this new functionality.

https://github.com/apache/incubator-druid/pull/6972

Dropwizard emitter

A new Dropwizard metrics emitter has been added as a contrib extension.

The currently supported Dropwizard metrics types are counter, gauge, meter, timer and histogram. These metrics can be emitted using either a Console or JMX reporter.

Please see https://druid.apache.org/docs/0.17.0/design/extensions-contrib/dropwizard.html for details.

https://github.com/apache/incubator-druid/pull/7363

Self-discovery resource

A new pair of endpoints have been added to all Druid services that return information about whether the Druid service has received a confirmation that the service has been added to the cluster, from the central service discovery mechanism (currently ZooKeeper). These endpoints can be useful as health/ready checks.

The new endpoints are:

  • /status/selfDiscovered/status
  • /status/selfDiscovered

Please see the Druid API reference for details.

https://github.com/apache/incubator-druid/pull/6702
https://github.com/apache/incubator-druid/pull/9005

Supervisors system table

Task supervisors (e.g. Kafka or Kinesis supervisors) are now recorded in the system tables in a new sys.supervisors table.

Please see https://druid.apache.org/docs/0.17.0/querying/sql.html#supervisors-table for details.

https://github.com/apache/incubator-druid/pull/8547

Fast historical start with lazy loading

A new boolean configuration property for historicals, druid.segmentCache.lazyLoadOnStart, has been added.

This new property allows historicals to defer loading of a segment until the first time that segment is queried, which can significantly decrease historical startup times for clusters with a large number of segments.

Please see the configuration reference for details.

https://github.com/apache/incubator-druid/pull/6988

Historical segment cache distribution change

A new historical property, druid.segmentCache.locationSelectorStrategy, has been added.

If there are multiple segment storage locations specified in druid.segmentCache.locations, the new locationSelectorStrategy property allows the user to specify what strategy is used to fill the locations. Currently supported options are roundRobin and leastBytesUsed.

Please see the configuration reference for details.

https://github.com/apache/incubator-druid/pull/8038

New readiness endpoints

A new Broker endpoint has been added: /druid/broker/v1/readiness.

A new Historical endpoint has been added: /druid/historical/v1/readiness.

These endpoints are similar to the existing /druid/broker/v1/loadstatus and /druid/historical/v1/loadstatus endpoints.

They differ in that they do not require authentication/authorization checks, and instead of a JSON body they only return a 200 success or 503 HTTP response code.

https://github.com/apache/incubator-druid/pull/8841

Support task assignment based on MiddleManager categories

It is now possible to define a "category" name property for each MiddleManager. New worker select strategies that are category-aware have been added, allowing the user to control how tasks are assigned to MiddleManagers based on the configured categories.

Please see the documentation for druid.worker.category in the configuration reference, and the following links, for more details:
https://druid.apache.org/docs/0.17.0/configuration/index.htmlEqual-Distribution-With-Category-Spec
https://druid.apache.org/docs/0.17.0/configuration/index.html#Fill-Capacity-With-Category-Spec
https://druid.apache.org/docs/0.17.0/configuration/index.html#WorkerCategorySpec

https://github.com/apache/druid/pull/7066

Security vulnerability updates

A large number of dependencies have been updated to newer versions to address security vulnerabilities.

Please see the PRs below for details:

Upgrading to Druid 0.17.0

Select native query has been replaced

The deprecated Select native query type has been removed in 0.17.0.

If you have native queries that use Select, you need to modify them to use Scan instead. See the Scan query documentation (https://druid.apache.org/docs/0.17.0/querying/scan-query.html) for syntax and output format details.

For Druid SQL queries that use Select, no changes are needed; the SQL planner already uses the Scan query type under the covers for Select queries.

https://github.com/apache/incubator-druid/pull/8739

Old consoles have been removed

The legacy coordinator and overlord consoles have been removed, and they have been replaced with the new web console on the coordinator and overlord.

https://github.com/apache/incubator-druid/pull/8838

Calcite 1.21 upgrade, Druid SQL null handling

Druid 0.17.0 updates Calcite to version 1.21. This newer version of Calcite can make additional optimizations that assume SQL-compliant null handling behavior when planning queries.

If you use Druid SQL and rely on null handling behavior, please read the information at https://druid.apache.org/docs/0.17.0/configuration/index.html#sql-compatible-null-handling and ensure that your Druid cluster is running in the SQL-compliant null handling mode before upgrading.

https://github.com/apache/incubator-druid/pull/8566

Logging adjustments

Druid 0.17.0 has tidied up its lifecycle, querying, and ingestion logging.

Please see https://github.com/apache/incubator-druid/pull/8889 for a detailed list of changes. If you relied on specific log messages for external integrations, please review the new logging changes before upgrading.

The full set of log messages can still be seen when logging is set to DEBUG level. Template log4j2 configuration files that show how to enable per-package DEBUG logging are provided in the _common configuration folder in the example clusters under conf/druid.

Stateful auto-compaction changes

The auto-compaction scheduling logic in 0.17.0 tracks additional segment partitioning information in Druid's metadata store that is not present in older versions. This information is used to determine whether a set of segments has already been compacted under the cluster's current auto-compaction configurations.

When this new metadata is not present, a set of segments will always be scheduled for an initial compaction and this new metadata will be created after they are compacted, allowing the scheduler to skip them later if auto-compaction config is unchanged.

Since this additional segment partitioning metadata is not present before 0.17.0, the auto-compaction scheduling logic will re-compact all segments within a datasource once after the upgrade to 0.17.0.

This re-compaction on the entire set of segments for each datasource that has auto-compaction enabled means that:

  • There will be a transition period after the upgrade where more total compaction tasks will be queued than under normal conditions
  • The deep storage usage will increase as the entire segment set is re-compacted (the old set of segments is still kept in deep storage unless explicitly removed).

Users are advised to be aware of the temporary increase in scheduled compaction tasks and the impact on deep storage usage. Documentation on removing old segments is located at https://druid.apache.org/docs/0.17.0/ingestion/data-management.html#deleting-data

targetCompactionSizeBytes property removed

The targetCompactionSizeBytes property has been removed from the compaction task and auto-compaction configuration. For auto-compaction, maxRowsPerSegment is now a mandatory configuration. For non-auto compaction tasks, any partitionsSpec can be used.

https://github.com/apache/incubator-druid/pull/8573

Compaction task tuningConfig

Due to the parallel auto-compaction changes introduced by #8570, any manually submitted compaction task specs need to be updated to use an index_parallel type for the tuningConfig section instead of index. These spec changes should be applied after the cluster is upgraded to 0.17.0.

Existing auto-compaction configs can remain unchanged after the update; the auto-compaction will create non-parallel compaction tasks until the auto-compaction configs are updated to use parallelism post-upgrade.

To control the level of parallelism, the auto-compactiontuningConfig has new parameters, maxNumConcurrentSubTasks and splitHintSpec.

Please see https://druid.apache.org/docs/0.17.0/configuration/index.html#compaction-dynamic-configuration for details.

Compaction task ioConfig

The compaction task now requires an ioConfig in the task spec.

Please see https://druid.apache.org/docs/0.17.0/ingestion/data-management.html#compaction-ioconfig for details.

ioConfig does not have to be added to existing auto-compaction configurations, the coordinator after the upgrade will automatically create task specs with ioConfig sections.

https://github.com/apache/incubator-druid/pull/8571

Renamed partition spec fields

The targetPartitionSize and maxSegmentSize fields in the partition specs have been deprecated. They have been renamed to targetNumRowsPerSegment and maxRowsPerSegment respectively.

https://github.com/apache/incubator-druid/pull/8507

Cache metrics are off by default

Cache metrics are now disabled by default. To enable cache metrics, add "org.apache.druid.client.cache.CacheMonitor" to the druid.monitoring.monitors property.

https://github.com/apache/incubator-druid/pull/8561

Supervisor API has changed to be consistent with task API

Supervisor task specs should now put the dataSchema, tuningConfig, and ioConfig sections as subfields of a spec field. Please see https://github.com/apache/incubator-druid/pull/8810 for examples.

The old format is still accepted in 0.17.0.

Segments API semantics change

The /datasources/{dataSourceName}/segments endpoint on the Coordinator now returns all used segments (including overshadowed) on the specified intervals, rather than only visible ones.

https://github.com/apache/incubator-druid/pull/8564

Password provider for basic authentication of HttpEmitterConfig

The druid.emitter.http.basicAuthentication property now accepts a password provider. We recommend updating your configurations to use a password provider if using the HTTP emitter.

https://github.com/apache/incubator-druid/pull/8618

Multivalue expression transformation change

Reusing multi-valued columns in expressions will no longer result in unnecessary cartesian explosions. Please see the following links for details.

https://github.com/apache/druid/issues/8947
https://github.com/apache/incubator-druid/pull/8957

Kafka/Kinesis ingestion during rolling upgrades

During a rolling upgrade, if there are tasks running 0.17.0 and overlords running older versions, and a task made progress reading data from its stream but rejected all the records it saw (e.g., all were unparseable), you will see NullPointerExceptions on overlords running older versions when the task updates the overlord with its current stream offsets.

Previously, there was a bug in this area (https://github.com/apache/incubator-druid/issues/8765) where such tasks would fail to communicate their current offsets to the overlord. The task/overlord publishing protocol has been updated to fix this, but older overlords do not recognize this protocol change.

This condition should be fairly rare.

Default Hadoop version change

Druid 0.17.0 now bundles Hadoop 2.8.5 libraries instead of 2.8.3.

If you were referencing the bundled 2.8.3 libraries in your configuration via the druid.indexer.task.defaultHadoopCoordinates property, or the hadoopDependencyCoordinates property in your Hadoop ingestion specs, you will need to update these references to point to the bundled 2.8.5 libraries instead.

ParseSpec.verify method removed

If you were maintaining a custom extension that provides an implementation for the ParseSpec interface, the verify method has been removed, and the @Override annotation on the method will need to be removed in any custom implementations.

https://github.com/apache/druid/pull/8744

Known issues

Filtering on long columns in SQL-compatible null handling mode

We are currently aware of a bug with applying certain filter types on null values from long columns when SQL-compatible null handling is enabled (https://github.com/apache/druid/issues/9255).

Please file an issue if you encounter any other null handling problems.

Ingestion spec preview in web console

The preview specs shown for native batch ingestion tasks created in the Data Loader of the web console are not correctly formatted and will fail if you copy them and submit them manually. Submitting these specs through the Data Loader submits a correctly formatted spec, however.

https://github.com/apache/druid/issues/9144

Other known issues

For a full list of open issues, please see https://github.com/apache/druid/labels/Bug

Credits

Thanks to everyone who contributed to this release!

@a2l007
@abhishekrb19
@aditya-r-m
@AlexanderSaydakov
@asdf2014
@capistrant
@ccaominh
@clintropolis
@denever
@elloooooo
@FaxianZhao
@fjy
@Fokko
@fstolba
@gianm
@gkc2104
@glasser
@GraceKoo
@himanshug
@jihoonson
@jnaous
@jon-wei
@karthikbhat13
@legendtkl
@leventov
@maytasm3
@mcbrewster
@mitchlloyd
@mohammadjkhan
@nishantmonu51
@pdeva
@pjain1
@pzhdfy
@QiuMM
@qutang1
@renevan10
@richardstartin
@samarthjain
@SandishKumarHN
@sashidhar
@sekingme
@SEKIRO-J
@suneet-s
@surekhasaharan
@tijoparacka
@vogievetsky
@xingbowu
@xvrl
@yuanlihan
@yunwan
@yurmix
@zhenxiao

druid - druid-0.16.1-incubating

Published by jon-wei almost 5 years ago

Apache Druid 0.16.1-incubating is a bug fix and user experience improvement release that fixes a rolling upgrade issue, improves the startup scripts, and updates licensing information.

Bug Fixes
#8682 implement FiniteFirehoseFactory in InlineFirehose
#8905 Retrying with a backward compatible task type on unknown task type error in parallel indexing

User Experience Improvements
#8792 Use bundled ZooKeeper in tutorials.
#8794 Startup scripts: verify Java 8 (exactly), improve port/java verification messages.
#8942 Improve verify-default-ports to check both INADDR_ANY and 127.0.0.1.
#8798 Fix verify script.

Licensing Update
#8944 Add license for tutorial wiki data
#8968 Add licenses.yaml entry for Wikipedia sample data

Other
#8419 Bump Apache Thrift to 0.10.0

Updating from 0.16.0-incubating and earlier

PR #8905 fixes an issue with rolling upgrades when updating from earlier versions.
Credits
Thanks to everyone who contributed to this release!

@aditya-r-m
@clintropolis
@Fokko
@gianm
@jihoonson
@jon-wei

Apache Druid (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

druid - druid-0.16.0-incubating

Published by clintropolis about 5 years ago

Apache Druid 0.16.0-incubating contains over 350 new features, performance enhancements, bug fixes, and major documentation improvements from 50 contributors. Check out the complete list of changes and everything tagged to the milestone.

Highlights

# Performance

# 'Vectorized' query processing

An experimental 'vectorized' query execution engine is new in 0.16.0, which can provide a speed increase in the range of 1.3-3x for timeseries and group by v2 queries. It operates on the principle of batching operations on rows instead of processing a single row at a time, e.g. iterating bitmaps in batches instead of per row, reading column values in batches, filtering in batches, aggregating values in batches, and so on. This results in significantly fewer method calls, better memory locality, and increased cache efficiency.

This is an experimental feature, but we view it as the path forward for Druid query processing and are excited for feedback as we continue to improve and fill out missing features in upcoming releases.

  • Only timeseries and groupBy have vectorized engines.
  • GroupBy doesn't handle multi-value dimensions or granularity other than "all" yet.
  • Vector cursors cannot handle virtual columns or descending order.
  • Expressions are not supported anywhere: not as inputs to aggregators, in virtual functions, or in filters.
  • Only some aggregators have vectorized implementations: "count", "doubleSum", "floatSum", "longSum", "hyperUnique", and "filtered".
  • Only some filters have vectorized matchers: "selector", "bound", "in", "like", "regex", "search", "and", "or", and "not".
  • Dimension specs other than "default" don't work yet (no extraction functions or filtered dimension specs).

The feature can be enabled by setting "vectorize": true your query context (the default is false). This works both for Druid SQL and for native queries. When set to true, vectorization will be used if possible; otherwise, Druid will fall back to its non-vectorized query engine. You can also set it to "force", which will return an error if the query cannot be fully vectorized. This is helpful for confirming that vectorization is indeed being used.

You can control the block size during execution by setting the vectorSize query context parameter (default is 1000).

https://github.com/apache/incubator-druid/issues/7093
https://github.com/apache/incubator-druid/pull/6794

# GroupBy array-based result rows

groupBy v2 queries now use an array-based representation of result rows, rather than the map-based representation used by prior versions of Druid. This provides faster generation and processing of result sets. Out of the box this change is invisible and backwards-compatible; you will not have to change any configuration to reap the benefits of this more efficient format, and it will have no impact on cached results. Internally this format will always be utilized automatically by the broker in the queries that it issues to historicals. By default the results will be translated back to the existing 'map' based format at the broker before sending them back to the client.

However, if you would like to avoid the overhead of this translation, and get even faster results,resultAsArray may be set on the query context to directly pass through the new array based result row format. The schema is as follows, in order:

  • Timestamp (optional; only if granularity != ALL)
  • Dimensions (in order)
  • Aggregators (in order)
  • Post-aggregators (optional; in order, if present)

https://github.com/apache/incubator-druid/issues/8118
https://github.com/apache/incubator-druid/pull/8196

# Additional performance enhancements

The complete set of pull requests tagged as performance enhancements for 0.16 can be found here.

# "Minor" compaction

Users of the Kafka indexing service and compaction and who get a trickle of late data, can find a huge improvement in the form of a new concept called 'minor' compaction. Enabled by internal changes to how data segments are versioned, minor compaction is based on the idea of 'segment' based locking at indexing time instead of the current Druid locking behavior (which is now referred to as 'time chunk' locking). Segment locking as you might expect allows only the segments which are being compacted to be locked, while still allowing new 'appending' indexing tasks (like Kafka indexing tasks) to continue to run and create new segments, simulataneously. This is a big deal if you get a lot of late data, because the current behavior results in compaction tasks starving as higher priority realtime tasks hog the locks. This prevention of compaction tasks from optimizing the datasources segment sizes results in reduced overall performance.

To enable segment locking, you will need to set forceTimeChunkLock to false in the task context, or set druid.indexer.tasklock.forceTimeChunkLock=false in the Overlord configuration. However, beware, after enabling this feature, due to the changes in segment versioning, there is no rollback path built in, so once you upgrade to 0.16, you cannot downgrade to an older version of Druid. Because of this, we highly recommend confirming that Druid 0.16 is stable in your cluster before enabling this feature.

It has a humble name, but the changes of minor compaction run deep, and it is not possible to adequately describe the mechanisms that drive this in these release notes, so check out the proposal and PR for more details.

https://github.com/apache/incubator-druid/issues/7491
https://github.com/apache/incubator-druid/pull/7547

# Druid "indexer" process

The new Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system and to better enable resource sharing across tasks.

The advantage of the Indexer is that it allows query processing resources, lookups, cached authentication/authorization information, and much more to be shared between all running indexing task threads, giving each individual task access to a larger pool of resources and far fewer redundant actions done than is possible with the Peon model of execution where each task is isolated in its own process.

Using Indexer does come with one downside: the loss of process isolation provided by Peon processes means that a single task can potentially affect all running indexing tasks on that Indexer. The druid.worker.globalIngestionHeapLimitBytes and druid.worker.numConcurrentMerges configurations are meant to help minimize this. Additionally, task logs for indexer processes will be inline with the Indexer process log, and not persisted to deep storage.

You can start using indexing by supplying server indexer as the command-line argument to org.apache.druid.cli.Main when starting the service. To use Indexer in place of a MiddleManager and Peon, you should be able to adapt values from the configuration into the Indexer configuration, lifting druid.indexer.fork.property. configurations directly to the Indexer, and sizing heap and direct memory based on the Peon sizes multiplied by the number of task slots (unlike a MiddleManager, it does not accept the configurations druid.indexer.runner.javaOpts or druid.indexer.runner.javaOptsArray). See the indexer documentation for details.

https://github.com/apache/incubator-druid/pull/8107

# Native parallel batch indexing with shuffle

In 0.16.0, Druid's index_parallel native parallel batch indexing task now supports 'perfect' rollup with the implementation of a 2 stage shuffle process.

Tasks in stage 1 perform a secondary partitioning of rows on top of the standard time based partitioning of segment granularity, creating an intermediary data segment for each partition. Stage 2 tasks are each assigned a set of the partitionings created during stage 1, and will collect and combine the set of intermediary data segments which belong to that partitioning, allowing it to achieve complete rollup when building the final segments. At this time, only hash-based partitioning is supported.

This can be enabled by setting forceGuaranteedRollup to true in the tuningConfig; numShards in partitionsSpec and intervals in granularitySpec must also be set.

The Druid MiddleManager (or the new Indexer) processes have a new responsibility for these indexing tasks, serving the intermediary partition segments output of stage 1 into the stage 2 tasks, so depending on configuration and cluster size, the MiddleManager jvm configuration might need to be adjusted to increase heap allocation and http threads. These numbers are expected to scale with cluster size, as all MiddleManager or Indexer processes involved in a shuffle will need the ability to communicate with each other, but we do not expect the footprint to be significantly larger than it is currently. Optimistically we suggest trying with your existing configurations, and bumping up heap and http thread count only if issues are encountered.

https://github.com/apache/incubator-druid/issues/8061

# Web console

# Data loader

The console data loader, introduced in 0.15, has been expanded in 0.16 to support Kafka, Kinesis, segment reindexing, and even "inline" data which can be pasted in directly.

DataLoaderKafka

https://github.com/apache/incubator-druid/pull/7643
https://github.com/apache/incubator-druid/pull/8181
https://github.com/apache/incubator-druid/pull/8056
https://github.com/apache/incubator-druid/pull/7947

# SQL query view

The query view on the web console has received a major upgrade for 0.16, transitioning into an interactive point-and-click SQL editor. There is a new column picker sidebar and the result output table let you directly manipulate the SQL query without needing to actually type SQL.

rn_sql_query_view

There is also a query history view and a the ability to fully edit the query context.

https://github.com/apache/incubator-druid/pull/7905
https://github.com/apache/incubator-druid/pull/7934
https://github.com/apache/incubator-druid/pull/8251
https://github.com/apache/incubator-druid/pull/7816

# Servers view

The "Data servers" view has been renamed to "Servers", and now displays a list of all Druid processes which are discovered as part of the cluster, using the sys.servers system table.

rn_servers_view

This should make it much more convenient to at a glance ensure that all your Druid servers are up and reporting to the rest of the cluster.

https://github.com/apache/incubator-druid/pull/7770
https://github.com/apache/incubator-druid/pull/7654

# Datasources view

The datasource view adds a new segment timeline visualization, allowing segment size and distribution to be gauged visually.

rn_datasources_view

If you previously have used what is now the legacy coordinator console, do not be alarmed, the timeline is reversed and the newest segments are now on the right side of the visualization!

https://github.com/apache/incubator-druid/pull/8202

# Tasks view

The tasks view has been significantly improved to assist a cluster operators ability to manage Druid supervisors and indexing tasks. The supervisors display now includes status information to allow determining their overall status and displaying error messaging if a supervisor is in error.

https://github.com/apache/incubator-druid/pull/7428
https://github.com/apache/incubator-druid/pull/7799

# SQL and native query enhancements

# SQL and native expression support for multi-value string columns

Druid SQL and native query expression 'virtual columns' can now correctly utilize multi-value string columns, either operating on them as individual VARCHAR values for parity with native Druid query behavior, or as an array like type, with a new set of multi-value string functions. Note that so far this is only supported in query time expressions; use of multi-value expressions at ingestion time via a transformSpec will be available in a future release. The complete list of added functions and behavior changes is far too large to list here, so check out the SQL and expression documentation for more details.

https://github.com/apache/incubator-druid/issues/7525
https://github.com/apache/incubator-druid/pull/7588
https://github.com/apache/incubator-druid/pull/7950
https://github.com/apache/incubator-druid/pull/7973
https://github.com/apache/incubator-druid/pull/7974
https://github.com/apache/incubator-druid/pull/8011

# TIMESTAMPDIFF

To complement TIMESTAMPADD which can modify timestamps by time units, a TIMESTAMPDIFF which can compute the signed number of time units between two timestamps. For syntax information, check out the SQL documentation

https://github.com/apache/incubator-druid/pull/7695

# TIME_CEIL

TIME_CEIL, a time specific, more flexible version of the CEIL function, has also been added to Druid 0.16.0. This function can round a timestamp up by an ISO8601 period, like P3M (quarters) or PT12H (half-days), optionally for a specific timezone. See SQL documentation for additional information.

https://github.com/apache/incubator-druid/pull/8027

# IPv4

Druid 0.16.0 also adds specialized SQL operators and native expressions for dealing with IPv4 internet addresses in dotted-decimal string or integer format. The new operators are IPV4_MATCH(address, subnet), IPV4_PARSE(address), and IPV4_STRINGIFY(address), which can match IP addresses to subnets in CIDR notation, translate dotted-decimal string format to integer format, and translate integer format into dotted-decimal string format, respectively. See SQL documentation for details.

https://github.com/apache/incubator-druid/pull/8223

# NVL

To increase permissiveness and SQL compatibility with users coming to Druid with experience in other databases, an alias for the COALESCE function has been added in the form of NVL, which some SQL dialects use instead. See SQL documentation for details

https://github.com/apache/incubator-druid/pull/7965

# long/double/float sum/min/max aggregator support for string columns

Long, double, and float sum, min, and max aggregators now will permissively work when used with string columns, performing a best-effort parsing to try and translate them to the correct numerical type.

https://github.com/apache/incubator-druid/pull/8243
https://github.com/apache/incubator-druid/pull/8319

# Official Docker image

An official "convenience binary" Docker image will now be offered for every release starting with 0.16, available at https://hub.docker.com/r/apache/incubator-druid.

# Refreshed website documentation

The documentation section of the Druid website has received a major upgrade, transitioning to using Docusaurus, which creates much more beautiful and functional pages than we have currently. Each documentation page now has a left-hand collapsible table of contents showing the outline of the overall docs, and a right-hand table of contents showing the outline of that particular doc page, vastly improving navigability.

Alongside this, the ingestion documentation has been totally refreshed, beginning with a new ingestion/index.md doc that introduces all the key ingestion spec concepts, and describes the most popular ingestion methods. This was a much needed rework of many existing documentation pages, which had grown organically over time and have become difficult to follow, into a simpler set of fewer, larger, more cross-referenced pages. They are also a bit more 'opinionated', pushing new people towards Kafka, Kinesis, Hadoop, and native batch ingestion. They discuss Tranquility but don't present it as something highly recommended since it is effectively in minimal maintenance mode.

Check out these improvements (and many more) here.

https://github.com/apache/incubator-druid/pull/8311

Extensions

# druid-datasketches

The druid-datasketches extension, built on top of Apache Datasketches (incubating), has been expanded with 3 new post aggregators, quantilesDoublesSketchToRank which computes an approximation to the rank of a given value that is the fraction of the distribution less than that value, and quantilesDoublesSketchToCDF which computes an approximation to the Cumulative Distribution Function given an array of split points that define the edges of the bins.

Another post aggregation, thetaSketchToString which will print a summary of sketch has been added to assist in debugging. See Datasketches extension documentation to learn more.

https://github.com/apache/incubator-druid/pull/7550
https://github.com/apache/incubator-druid/pull/7937

The HLLSketch aggregator has been improved with a query-time only round option to support rounding values into whole numbers, to give it feature parity with the built-in cardinality and hyperUnique aggregators.

https://github.com/apache/incubator-druid/pull/8023

Finally, users of HllSketch should also see a performance improvement due to some changes made which allow Druid to precompute an empty sketch and copy that into the aggregation buffers, greatly decreasing time to initialize the aggregator during query processing.

https://github.com/apache/incubator-druid/pull/8194

# druid-stats

The druid-stats core extension has been enhanced with SQL support, exposing VAR_POP and VAR_SAMP to compute variance population and sample with the variance aggregator, as well as STDDEV_POP and STDDEV_SAMPto compute standard deviation population and sample using the standard deviation post aggregator. Additionally, VARIANCE and STDDEV functions are added as aliases for VAR_SAMP and STDDEV_SAMP respectively. See SQL documentation and stats extension documentation for more details.

https://github.com/apache/incubator-druid/pull/7801

# druid-histogram

The 'fixed bucket' histogram aggregator of the druid-histogram extension has a new added property finalizeAsBase64Binary that enables serializing the resulting histogram as a base64 string instead of a human readable JSON summary, making it consistent with the approximate histogram aggregator of the same extension. See the documentation for further information.

https://github.com/apache/incubator-druid/pull/7784

# statsd-emitter

The statsd-emitter extension has a new option, druid.emitter.statsd.dogstatsdServiceAsTag which enables emitting the Druid service name as a 'tag', such as druid_service:druid/broker instead of part of the metric name as in druid.broker.query.time. druid is used as the prefix. so the example of druid.broker.query.time would instead be druid.query.time, allowing consolidation metrics across Druid service types and discriminating by tag instead. druid.emitter.statsd.dogstatsd must be set to true for this setting to take effect. See the documentation for more details.

https://github.com/apache/incubator-druid/pull/8238
https://github.com/apache/incubator-druid/pull/8472

# New extension: druid-tdigestsketch

A new set of approximate sketch aggregators for computing quantiles and the like and based on t-digest has been added in Druid 0.16. T-digest was designed for parallel programming use cases like distributed aggregations or map reduce jobs by making combining two intermediate t-digests easy and efficient. It serves to complement existing algorithms provided by the Apache Datasketches extension and moments sketch extension. See the extension documentation for more details.

https://github.com/apache/incubator-druid/issues/7303
https://github.com/apache/incubator-druid/pull/7331

# New extension: druid-influxdb-emitter

A new Druid emitter extension to allow sending Druid metrics to influxdb over HTTP has also been added in 0.16. Currently this emitter only emits service metric events to InfluxDB (See Druid metrics for a list of metrics). When a metric event is fired it is added to a queue of events. After a configurable amount of time, the events on the queue are transformed to InfluxDB's line protocol and POSTed to the InfluxDB HTTP API. The entire queue is flushed at this point. The queue is also flushed as the emitter is shutdown. See the extension docs for details.

https://github.com/apache/incubator-druid/pull/7717

# Fine tuning your workloads

In addition to the experimental vectorized query engine and new indexer process type, 0.16 also has some additional features available to allow potentially fine tuning indexing and query performance via experimentation.

# Control of indexing intermediary segment compression

First up, the ability to independently control what compression is used (or disable it) when persisting intermediary segments during indexing. This configuration available to the indexSpec property, and can be added to tuningConfig as:

"indexSpecForIntermediatePersists": {
  "dimensionCompression": "uncompressed",
  "metricCompression": "none"
}

for example to disable compression entirely for intermediary segments. One potential reason to consider 'uncompressed' intermediary segments is to ease up on the amount of Java 'direct' memory required to perform the final merge of intermediary segments before they are published and pushed to deep storage, as reading data from uncompressed columns does not require the 64kb direct buffers which are used to decode lz4 and other encoded columns. Of course this is a trade-off of storage space and page cache footprint, so we recommend experimenting with this before settling on a configuration to use for your production workloads.

https://github.com/apache/incubator-druid/pull/7919

# Control Filter Bitmap Index Utilization

Bitmap indexes are usually a huge performance boost for Druid, but under some scenarios can result in slower query speeds, particularly in cases of computationally expensive filters on very high cardinality dimensions. In Druid 0.16, a new mechanism to provide some manual control over when bitmap indexes are utilized, and when a filter will be done as a row scan are now in place, and available on a per filter, per query basis. Most filters will accept a new property, filterTuning, which might look something like this:

"filterTuning": {
  "useBitmapIndex": true
}

useBitmapIndex if set to false will disallow a filter to utilize bitmap indexes. This property is optional, and default behavior if filterTuning is not supplied remains unchanged. Note that this feature is not documented in user facing documentation, considered experimental, and subject to change in any future release.

https://github.com/apache/incubator-druid/pull/8209

# Request logging

If you would have liked to enable Druid request logging, but use Druid SQL and find them a bit too chatty due to all the metadata queries, you are luck with 0.16 due to a new configuration option that allows selectively muting specific types of queries from request logging. The option, druid.request.logging.mutedQueryTypes, accepts a list of "queryType" strings as defined by Druid's native JSON query API, and defaults to an empty list (so nothing is ignored). For example,

druid.request.logging.mutedQueryTypes=["segmentMetadata", "timeBoundary"]

would mute all request logs for segmentMetadata and timeBoundary queries.

https://github.com/apache/incubator-druid/pull/7562

# Upgrading to Druid 0.16.0

# S3 task log storage

After upgrading to 0.16.0, the task logs will require the same S3 permissions as pushing segments to deep storage, which has some additional logic to possibly get the bucket owner (by fetching the bucket ACL) and set the ACL of the segment object to give the bucket owner full access. If you wish to avoid providing these additional permissions, the existing behavior can be retained by setting druid.indexer.logs.disableAcl=true.

https://github.com/apache/incubator-druid/pull/7907

# Druid SQL

Druid SQL is now enabled on the router by default. Note that this may put some additional pressure on broker processes, due to additional metadata queries required to maintain datasource schemas for query planning, and can be disabled to retain the previous behavior by setting druid.sql.enable=false.

https://github.com/apache/incubator-druid/pull/7808

# Druid SQL lookup function

Druid SQL queries that use the LOOKUP function will now take advantage of injective lookups that are a one-to-one transformation, which allows an optimization to be performed that avoids pushing down lookup evaluation to historical processes, instead allowing the transformation to be done later. The drawback of this change is that lookups which are incorrectly defined as being one-to-one, but are not in fact a one-to-one transformation at evaluation time will produce incorrect query results.

https://github.com/apache/incubator-druid/pull/7655

# Coordinator metadata API changes

The metadata API provided by the coordinator has been modified to provide more consistent HTTP responses. The changes are minor, but backwards incompatible if you depend on a specific API.

  • POST /druid/coordinator/v1/datasources/{dataSourceName}
  • DELETE /druid/coordinator/v1/datasources/{dataSourceName}
  • POST /druid/coordinator/v1/datasources/{dataSourceName}/markUnused
  • POST /druid/coordinator/v1/datasources/{dataSourceName}/markUsed

now return a JSON object of the form {"numChangedSegments": N} instead of 204 (empty response) when no segments were changed. On the other
hand, 500 (server error) is not returned instead of 204 (empty response).

  • POST /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}
  • DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}

now return a JSON object of the form {"segmentStateChanged": true/false}.

/druid/coordinator/v1/metadata/datasources?includeDisabled is now /druid/coordinator/v1/metadata/datasources?includeUnused. includeDisabled is still accepted, but a warning is emitted in a log.

https://github.com/apache/incubator-druid/pull/7653

# Compaction tasks

The keepSegmentGranularity option would create compaction tasks that ignored interval boundary of segments before compacting them, but was deprecated because it was not very useful in most cases. If you did find this behavior useful, it can still be achieved by setting segmentGranularity to ALL.

https://github.com/apache/incubator-druid/pull/7747

There is a known issue with all Druid versions that support 'auto' compaction via the Coordinator, where auto compaction can get stuck repeatedly trying the same interval. This issue can now be mitigated in 0.16 by setting targetCompactionSizeBytes, but is still present when using maxRowsPerSegment or maxTotalRows. A future version of Druid will fully resolve this issue.

https://github.com/apache/incubator-druid/issues/8481
https://github.com/apache/incubator-druid/pull/8495

# Indexing spec

All native indexing task types now use PartitonsSpec to define secondary partitioning to be consistent with other indexing task types. The changes are not backwards incompatible, but users should consider migrating maxRowsPerSegment, maxTotalRows, numShards, and partitionDimensions to the partitions spec of native, kafka, and kinesis indexing tasks.

https://github.com/apache/incubator-druid/pull/8141

# Kafka indexing

Incremental publishing of segments for the Apache Kafka indexing service was introduced in Druid 0.12, at which time the existing Kafka indexing service was retained as a 'legacy' format to allow rolling update from older versions of Druid. This 'legacy' codebase has now been removed in Druid 0.16.0, which means that a rolling-update from a Druid version older than 0.12.0 is not supported.

https://github.com/apache/incubator-druid/pull/7735

# Avro extension

The fromPigAvroStorage option has been removed from the Apache Avro extension in Druid 0.16, in order to clean up dependencies. This option provided a special transformation of GenericData.Array, but this column type will now be handled by the default List implementation.

https://github.com/apache/incubator-druid/pull/7810

# Kerberos extension

druid.auth.authenticator.kerberos.excludedPaths superseded by druid.auth.unsecuredPaths which applies across all authenticators/authorizers, so use this configuration property instead.

https://github.com/apache/incubator-druid/pull/7745

# Zookeeper

The setting druid.zk.service.terminateDruidProcessOnConnectFail has been removed and this is now the default behavior - Druid processes will exit when an unhandled Curator client exception occurs instead of continuing to run in a 'zombie' state.

Additionally, a new optional setting druid.zk.service.connectionTimeoutMs, which wires into the Curator client connectionTimeoutMs, setting is now available to customize. By default it will continue to use the Curator default of 15000 milliseconds.

https://github.com/apache/incubator-druid/pull/8458

# Realtime process

The deprecated standalone 'realtime' process has been removed from Druid. Use of this has been discouraged for some time, but now middleManager and the new index processes should process all of your realtime indexing tasks.

https://github.com/apache/incubator-druid/pull/7915
https://github.com/apache/incubator-druid/pull/8020

Credits

Thanks to everyone who contributed to this release!

@a2l007
@abossert
@AlexanderSaydakov
@AlexandreYang
@andrewluotechnologies
@asdf2014
@awelsh93
@blugowski
@capistrant
@ccaominh
@clintropolis
@dclim
@dene14
@dinsaw
@Dylan1312
@esevastyanov
@fjy
@Fokko
@gianm
@gocho1
@himanshug
@ilhanadiyaman
@jennyzzz
@jihoonson
@Jinstarry
@jon-wei
@justinborromeo
@kamaci
@khwj
@legoscia
@leventov
@litao91
@lml2468
@mcbrewster
@nieroda
@nishantmonu51
@pjain1
@pphust
@samarthjain
@SandishKumarHN
@sashidhar
@satybald
@sekingme
@shuqi7
@surekhasaharan
@viongpanzi
@vogievetsky
@xueyumusic
@xvrl
@yurmix

druid - druid-0.15.1-incubating

Published by clintropolis about 5 years ago

Apache Druid 0.15.1-incubating is a bug fix release that includes important fixes for Apache Zookeeper based segment loading, the 'druid-datasketches' extension, and much more.

Bug Fixes

Coordinator

#8137 coordinator throwing exception trying to load segments (fixed by #8140)

Middlemanager

#7886 Middlemanager fails startup due to corrupt task files (fixed by #7917)
#8085 fix forking task runner task shutdown to be more graceful

Queries

#7777 timestamp_ceil function is either wrong or misleading (fixed by #7823)
#7820 subtotalsSpec and filtering returns no results (fixed by #7827)
#8013 Fix ExpressionVirtualColumn capabilities; fix groupBy's improper uses of StorageAdapter#getColumnCapabilities.

API

#6786 apache-druid-0.13.0-incubating router /druid/router/v1/brokers (fixed by #8026)
#8044 SupervisorManager: Add authorization checks to bulk endpoints.

Metrics Emitters

#8204 HttpPostEmitter throw Class cast exception when using emitAndReturnBatch (fixed by #8205)

Extensions
Datasketches

#7666 sketches-core-0.13.4
#8055 force native order when wrapping ByteBuffer

Kinesis Indexing Service

#7830 Kinesis: Fix getPartitionIds, should be checking isHasMoreShards.

Moving Average Query

#7999 Druid moving average query results in circular reference error (fixed by #8192)

Documentation Fixes

#8002 Improve pull-deps reference in extensions page.
#8003 Add missing reference to Materialized-View extension.
#8079 Fix documentation formatting
#8087 fix references to bin/supervise in tutorial docs

Updating from 0.15.0-incubating and earlier

Due to issue #8137, when updating from specifically 0.15.0-incubating to 0.15.1-incubating, it is recommended to update the Coordinator before the Historical servers to prevent segment unavailability during an upgrade (this is typically reversed). Upgrading from any version older than 0.15.0-incubating does not have these concerns and can be done normally.

Known Issues

Building Docker images is currently broken and will be fixed in the next release, see #8054 which is fixed by #8237 for more details.

Credits

Thanks to everyone who contributed to this release!

@AlexanderSaydakov
@ArtyomyuS
@ccl0326
@clintropolis
@gianm
@himanshug
@jihoonson
@legoscia
@leventov
@pjain1
@yurmix
@xueyumusic

Apache Druid (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

druid - druid-0.15.0-incubating

Published by jihoonson over 5 years ago

Apache Druid 0.15.0-incubating contains over 250 new features, performance/stability/documentation improvements, and bug fixes from 39 contributors. Major new features and improvements include:

  • New Data Loader UI
  • Support transactional Kafka topic
  • New Moving Average query
  • Time ordering for Scan query
  • New Moments Sketch aggregator
  • SQL enhancements
  • Light lookup module for routers
  • Core ORC extension
  • Core GCP extension
  • Document improvements

The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+is%3Aclosed+milestone%3A0.15.0

Documentation for this release is at: http://druid.apache.org/docs/0.15.0-incubating/

Highlights

New Data Loader UI (Batch indexing part)

Druid has a new Data Loader UI which is integrated with the Druid Console. The new Data Loader UI shows some sampled data to easily verify the ingestion spec and generates the final ingestion spec automatically. The users are expected to easily issue batch index tasks instead of writing a JSON spec by themselves.

Added by @vogievetsky and @dclim in https://github.com/apache/incubator-druid/pull/7572 and https://github.com/apache/incubator-druid/pull/7531, respectively.

Support Kafka Transactional Topics

The Kafka indexing service now supports Kafka Transactional Topics.

Please note that only Kafka 0.11.0 or later versions are supported after this change.

Added by @surekhasaharan in https://github.com/apache/incubator-druid/pull/6496.

New Moving Average Query

A new query type was introduced to compute moving average.

Please see http://druid.apache.org/docs/0.15.0-incubating/development/extensions-contrib/moving-average-query.html for more details.

Added by @yurmix in https://github.com/apache/incubator-druid/pull/6430.

Time Ordering for Scan Query

The Scan query type now supports time ordering. Please see http://druid.apache.org/docs/0.15.0-incubating/querying/scan-query.html#time-ordering for more details.

Added by @justinborromeo in https://github.com/apache/incubator-druid/pull/7133.

New Moments Sketch Aggregator

The Moments Sketch is a new sketch type for approximate quantile computation. Please see http://druid.apache.org/docs/0.15.0-incubating/development/extensions-contrib/momentsketch-quantiles.html for more details.

Added by @edgan8 in https://github.com/apache/incubator-druid/pull/6581.

SQL enhancements

Druid community has been striving to enhance SQL support and now it's no longer experimental.

New SQL functions

Autocomplete in Druid Console

Druid Console now supports autocomplete for SQL.

Added by @shuqi7 in https://github.com/apache/incubator-druid/pull/7244.

Time-ordered scan support for SQL

Druid SQL supports time-ordered scan query.

Added by @justinborromeo in https://github.com/apache/incubator-druid/pull/7373.

Lookups view added to the web console

You can now configure your lookups from the web console directly.

Added by @shuqi7 in https://github.com/apache/incubator-druid/pull/7259.

Misc web console improvements

"NoSQL" mode : https://github.com/apache/incubator-druid/pull/7493 [@shuqi7]

The web console now has a backup mode that allows it to function as best as it can if DruidSQL is disabled or unavailable.

Added compaction configuration dialog : https://github.com/apache/incubator-druid/pull/7242 [@shuqi7]

You can now configure the auto compaction settings for a data source from the Datasource view.

Auto wrap query with limit : https://github.com/apache/incubator-druid/pull/7449 [@vogievetsky]

The console query view will now (by default) wrap DruidSQL queries with a SELECT * FROM (...) LIMIT 1000 allowing you to enter queries like SELECT * FROM your_table without worrying about the impact to the cluster. You can still send 'raw' queries by selecting the option from the ... menu.

SQL explain query : https://github.com/apache/incubator-druid/pull/7402 [@shuqi7]

You can now click on the ... menu in the query view to get an explanation of the DruidSQL query.

Surface is_overshadowed as a column in the segments table https://github.com/apache/incubator-druid/pull/7555 , https://github.com/apache/incubator-druid/pull/7425 [@shuqi7][@surekhasaharan]

is_overshadowed column represents that this segment is overshadowed by any published segments. It can be useful to see what segments should be loaded by historicals. Please see http://druid.apache.org/docs/0.15.0-incubating/querying/sql.html for more details.

Improved status UI for actions on tasks, supervisors, and datasources : https://github.com/apache/incubator-druid/pull/7528 [shuqi7]

This PR condenses the actions list into a tidy menu and lets you see the detailed status for supervisors and tasks. New actions for datasources around loading and dropping data by interval has also been added.

Light Lookup Module for Routers

Light lookup module was introduced for Routers and they now need only minimum amount of memory. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/basic-cluster-tuning.html#router for basic memory tuning.

Added by @clintropolis in https://github.com/apache/incubator-druid/pull/7222.

Core ORC extension

ORC extension is now promoted to a core extension. Please read the below 'Updating from 0.14.0-incubating and earlier' section if you are using the ORC extension in an earlier version of Druid.

Added by @clintropolis in https://github.com/apache/incubator-druid/pull/7138.

Core GCP extension

GCP extension is now promoted to a core extension. Please read the below 'Updating from 0.14.0-incubating and earlier' section if you are using the GCP extension in an earlier version of Druid.

Added by @drcrallen in https://github.com/apache/incubator-druid/pull/6953.

Document Improvements

Single-machine deployment example configurations and scripts

Several configurations and scripts were added for easy single machine setup. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/single-server.html for details.

Added by @jon-wei in https://github.com/apache/incubator-druid/pull/7590.

Tool for migrating from local deep storage/Derby metadata

A new tool was added for easy migration from single machine to a cluster environment. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/deep-storage-migration.html for details.

Added by @jon-wei in https://github.com/apache/incubator-druid/pull/7598.

Document for basic tuning guide

Documents for basic tuning guide was added. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/basic-cluster-tuning.html for details.

Added by @jon-wei in https://github.com/apache/incubator-druid/pull/7629.

Security Improvement

The Druid system table now requires only mandatory permissions instead of the read permission for the whole sys database. Please see http://druid.apache.org/docs/0.15.0-incubating/development/extensions-core/druid-basic-security.html for details.

Added by @jon-wei in https://github.com/apache/incubator-druid/pull/7579.

Deprecated/removed

Drop support for automatic segment merge

The automatic segment merge by the coordinator is not supported anymore. Please use auto compaction instead.

Added by @jihoonson in #6883.

Drop support for insert-segment-to-db tool

In Druid 0.14.x or earlier, Druid stores segment metadata (descriptor.json file) in deep storage in addition to metadata store. This behavior has changed in 0.15.0 and it doesn't store segment metadata file in deep storage anymore. As a result, insert-segment-to-db tool is no longer supported as well since it works based on descriptor.json files in deep storage. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/insert-segment-db.html for details.

Please note that kill task will fail if you're using HDFS as deep storage and descriptor.json file is missing in 0.14.x or earlier versions.

Added by @jihoonson in https://github.com/apache/incubator-druid/pull/6911.

Removed "useFallback" configuration for SQL

This option was removed since it generates unscalable query plans and doesn't work with some SQL functions.

Added by @gianm in https://github.com/apache/incubator-druid/pull/7567.

Removed a public API in CompressionUtils for extension developers

public static void gunzip(File pulledFile, File outDir) was removed in https://github.com/apache/incubator-druid/pull/6908 by @clintropolis.

Other behavior changes

Coordinator await initialization before finishing startup

A new configuration (druid.coordinator.segment.awaitInitializationOnStart) was added to make Coordinator wait for segment view initialization. This option is enabled by default.

Added by @QiuMM in https://github.com/apache/incubator-druid/pull/6847.

Coordinator API behavior change

The coordinator periodically polls segment metadata information from metadata store and caches them in memory. In Druid 0.14.x or earlier, removing segments via coordinator APIs (/druid/coordinator/v1/datasources/{dataSourceName} and /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}) immediately updates the segment cache in memory as well as metadata store. But this behavior has changed in 0.15.0 and the cache is updated per poll rather than being updated immediately on removal. The below APIs can return removed segments via the above API calls until the cache is updated in the next poll.

  • /druid/coordinator/v1/metadata/datasources/{dataSourceName}
  • /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments
  • /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments/{segmentId}
  • /druid/coordinator/v1/metadata/datasources
  • /druid/coordinator/v1/loadstatus

The below metrics can also contain removed segments via the above API calls until the cache is updated in the next poll.

  • segment/unavailable/count
  • segment/underReplicated/count

This behavior was changed in https://github.com/apache/incubator-druid/pull/7595 by @surekhasaharan.

Listing Lookup API change

The /druid/coordinator/v1/lookups/config API now returns a list of tiers currently active in the cluster in addition to ones known in the dynamic configuration.

Added by @clintropolis in https://github.com/apache/incubator-druid/pull/7647.

Zookeeper loss

With a new configuration (druid.zk.service.terminateDruidProcessOnConnectFail), Druid processes can terminate itself on disconnection to ZooKeeper.

Added by @michael-trelinski in https://github.com/apache/incubator-druid/pull/6740.

Updating from 0.14.0-incubating and earlier

Minimum compatible Kafka version change for Kafka Indexing Service

Kafka 0.11.x or later versions are only supported after https://github.com/apache/incubator-druid/pull/6496. Please consider updating Kafka version if you're using an older one.

ORC extension changes

The ORC extension has been promoted to a core extension. When deploying 0.15.0-incubating, please ensure that your extensions directory does not have any older versions of druid-orc-extensions extension.

Additionally, even though the new core extension can index any data the old contrib extension could, the JSON spec for the ingestion task is incompatible, and will need modified to work with the newer core extension.

To migrate to 0.15.0-incubating:

  • In inputSpec of ioConfig, inputFormat must be changed from "org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat" to
    "org.apache.orc.mapreduce.OrcInputFormat"
  • The contrib extension supported a typeString property, which provided the schema of the
    ORC file, of which was essentially required to have the types correct, but notably not the column names, which facilitated column renaming. In the core extension, column renaming can be achieved with flattenSpec expressions.
  • The contrib extension supported a mapFieldNameFormat property, which provided a way to specify a dimension to flatten OrcMap columns with primitive types. This functionality has also been replaced with flattenSpec expressions.

For more details and examples, please see http://druid.apache.org/docs/0.15.0-incubating/development/extensions-core/orc.html.

GCP extension changes

The GCP extension has been promoted to a core extension. When deploying 0.15.0-incubating, please ensure that your extensions directory does not have any older versions of the druid-google-extensions extension.

Dropped auto segment merge

The coordinator configuration for auto segment merge (druid.coordinator.merge.on) is not supported anymore. Please use auto compaction instead.

Removed descriptor.json metadata file in deep storage

The segment metadata file (descriptor.json) is not stored in deep storage any more. If you are using HDFS as your deep storage and need to roll back to 0.14.x or earlier, then please consider that the kill task could fail because of the missing descriptor.json files.

Credits

Thanks to everyone who contributed to this release!

@a2l007
@asdf2014
@capistrant
@clintropolis
@dampcake
@dclim
@donbowman
@drcrallen
@Dylan1312
@edgan8
@es1220
@esevastyanov
@FaxianZhao
@fjy
@gianm
@glasser
@hpandeycodeit
@jihoonson
@jon-wei
@jorbay-au
@justinborromeo
@kamaci
@KazuhitoT
@leventov
@lxqfy
@michael-trelinski
@peferron
@puneetjaiswal
@QiuMM
@richardstartin
@samarthjain
@scrawfor
@shuqi7
@surekhasaharan
@venkatramanp
@vogievetsky
@xueyumusic
@xvrl
@yurmix

druid -

Published by clintropolis over 5 years ago

Apache Druid 0.14.2-incubating is a bug fix release that includes important fixes for the 'druid-datasketches' extension and the broker 'result' level caching.

Bug Fixes

  • #7607 thetaSketch(with sketches-core-0.13.1) in groupBy always return value no more than 16384
  • #6483 Exception during sketch aggregations while using Result level cache
  • #7621 NPE when both populateResultLevelCache and grandTotal are set

Credits

Thanks to everyone who contributed to this release!

@AlexanderSaydakov
@clintropolis
@jihoonson
@jon-wei

Apache Druid (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

druid -

Published by clintropolis over 5 years ago

Apache Druid 0.14.1-incubating is a small patch release that includes a handful of bug and documentation fixes from 16 contributors.

Important Notice

This release fixes an issue with druid-datasketches extension with quantile sketches, but introduces another one with theta sketches that was confirmed after the release was finalized, caused by #7320 and described in #7607. If you utilize theta sketches, we recommend not upgrading to this release. This will be fixed in the next release of Druid by #7619.

Bug Fixes

  • use latest sketches-core-0.13.1 #7320
  • Adjust BufferAggregator.get() impls to return copies #7464
  • DoublesSketchComplexMetricSerde: Handle empty strings. #7429
  • handle empty sketches #7526
  • Adds backwards-compatible serde for SeekableStreamStartSequenceNumbers. #7512
  • Support Kafka supervisor adopting running tasks between versions #7212
  • Fix time-extraction topN with non-STRING outputType. #7257
  • Fix two issues with Coordinator -> Overlord communication. #7412
  • refactor druid-bloom-filter aggregators #7496
  • Fix encoded taskId check in chatHandlerResource #7520
  • Fix too many dentry cache slab objs#7508. #7509
  • Fix result-level cache for queries #7325
  • Fix flattening Avro Maps with Utf8 keys #7258
  • Write null byte when indexing numeric dimensions with Hadoop #7020
  • Batch hadoop ingestion job doesn't work correctly with custom segments table #7492
  • Fix aggregatorFactory meta merge exception #7504

Documentation Changes

  • Fix broken link due to Typo. #7513
  • Some docs optimization #6890
  • Updated Javascript Affinity config docs #7441
  • fix expressions docs operator table #7420
  • Fix conflicting information in configuration doc #7299
  • Add missing doc link for operations/http-compression.html #7110

Updating from 0.14.0-incubating and earlier

Kafka Ingestion

Updating from version 0.13.0-incubating or earlier directly to 0.14.1-incubating will not require downtime like the migration path to 0.14.0-incubating due to the issue described in #6958, which has been fixed for this release in #7212. Likewise, rolling updates from version 0.13.0-incubating and earlier should also work properly due to #7512.

Native Parallel Ingestion

Updating from 0.13.0-incubating directly to 0.14.1-incubating will not encounter any issues during a rolling update with mixed versions of middle managers due to the fixes in #7520, as could be experienced when updating to 0.14.0-incubating.

Credits

Thanks to everyone who contributed to this release!

@AlexanderSaydakov
@b-slim
@benhopp
@chrishardis
@clintropolis
@ferristseng
@es1220
@gianm
@jihoonson
@jon-wei
@justinborromeo
@kaka11chen
@samarthjain
@surekhasaharan
@zhaojiandong
@zhztheplayer

Apache Druid (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

druid -

Published by jon-wei over 5 years ago

Apache Druid (incubating) 0.14.0-incubating contains over 200 new features, performance/stability/documentation improvements, and bug fixes from 54 contributors. Major new features and improvements include:

  • New web console
  • Amazon Kinesis indexing service
  • Decommissioning mode for Historicals
  • Published segment cache in Broker
  • Bloom filter aggregator and expression
  • Updated Apache Parquet extension
  • Force push down option for nested GroupBy queries
  • Better segment handoff and drop rule handling
  • Automatically kill MapReduce jobs when Apache Hadoop ingestion tasks are killed
  • DogStatsD tag support for statsd emitter
  • New API for retrieving all lookup specs
  • New compaction options
  • More efficient cachingCost segment balancing strategy

The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+is%3Amerged+milestone%3A0.14.0

Documentation for this release is at: http://druid.io/docs/0.14.0-incubating/

Highlights

New web console

new-druid-console

Druid has a new web console that provides functionality that was previously split between the coordinator and overlord consoles.

The new console allows the user to manage datasources, segments, tasks, data processes (Historicals and MiddleManagers), and coordinator dynamic configuration. The user can also run SQL and native Druid queries within the console.

For more details, please see http://druid.io/docs/0.14.0-incubating/operations/management-uis.html

Added by @vogievetsky in #6923.

Kinesis indexing service

Druid now supports ingestion from Kinesis streams, provided by the new druid-kinesis-indexing-service core extension.

Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/kinesis-ingestion.html for details.

Added by @jsun98 in #6431.

Decommissioning mode for Historicals

Historical processes can now be put into a "decommissioning" mode, where the coordinator will no longer consider the Historical process as a target for segment replication. The coordinator will also move segments off the decommissioning Historical.

This is controlled via Coordinator dynamic configuration. For more details, please see http://druid.io/docs/0.14.0-incubating/configuration/index.html#dynamic-configuration.

Added by @egor-ryashin in #6349.

Published segment cache on Broker

The Druid Broker now has the ability to maintain a cache of published segments via polling the Coordinator, which can significantly improve response time for metadata queries on the sys.segments system table.

Please see http://druid.io/docs/0.14.0-incubating/querying/sql.html#retrieving-metadata for details.

Added by @surekhasaharan in #6901

Bloom filter aggregator and expression

A new aggregator for constructing Bloom filters at query time and support for performing Bloom filter checks within Druid expressions have been added to the druid-bloom-filter extension.

Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/bloom-filter.html

Added by @clintropolis in #6904 and #6397

Updated Parquet extension

druid-extensions-parquet has been moved into the core extension set from the contrib extensions and now supports flattening and int96 values.

Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/parquet.html for details.

Added by @clintropolis in #6360

Force push down option for nested GroupBy queries

Outer query execution for nested GroupBy queries can now be pushed down to Historical processes; previously, the outer queries would always be executed on the Broker.

Please see https://github.com/apache/incubator-druid/pull/5471 for details.

Added by @samarthjain in #5471.

Better segment handoff and retention rule handling

Segment handoff will now ignore segments that would be dropped by a datasource's retention rules, avoiding ingestion failures caused by issue #5868.

Period load rules will now include the future by default.

A new "Period Drop Before" rule has been added. Please see http://druid.io/docs/0.14.0-incubating/operations/rule-configuration.html#period-drop-before-rule for details.

Added by @QiuMM in #6676, #6414, and #6415.

Automatically kill MapReduce jobs when Hadoop ingestion tasks are killed

Druid will now automatically terminate MapReduce jobs created by Hadoop batch ingestion tasks when the ingestion task is killed.

Added by @ankit0811 in #6828.

DogStatsD tag support for statsd-emitter

The statsd-emitter extension now supports DogStatsD-style tags. Please see http://druid.io/docs/0.14.0-incubating/development/extensions-contrib/statsd.html

Added by @deiwin in #6605, with support for constant tags added by @glasser in #6791.

New API for retrieving all lookup specs

A new API for retrieving all lookup specs for all tiers has been added. Please see http://druid.io/docs/0.14.0-incubating/querying/lookups.html#get-all-lookups for details.

Added by @jihoonson in #7025.

New compaction options

Auto-compaction now supports the maxRowsPerSegment option. Please see http://druid.io/docs/0.14.0-incubating/design/coordinator.html#compacting-segments for details.

The compaction task now supports a new segmentGranularity option, deprecating the older keepSegmentGranularity option for controlling the segment granularity of compacted segments. Please see the segmentGranularity table in http://druid.io/docs/0.14.0-incubating/ingestion/compaction.html for more information on these properties.

Added by @jihoonson in #6758 and #6780.

More efficient cachingCost segment balancing strategy

The cachingCost Coordinator segment balancing strategy will now only consider Historical processes for balancing decisions. Previously the strategy would unnecessarily consider active worker tasks as well, which are not targets for segment replication.

Added by @QiuMM in #6879.

New metrics:

  • New allocation rate metric jvm/heapAlloc/bytes, added by @egor-ryashin in #6710.
  • New query count metric query/count, added by @QiuMM in #6473.
  • SQL query metrics sqlQuery/bytes and sqlQuery/time, added by @gaodayue in #6302.
  • Apache Kafka ingestion lag metrics ingest/kafka/maxLag and ingest/kafka/avgLag, added by @QiuMM in #6587
  • Task count metrics task/success/count, task/failed/count, task/running/count, task/pending/count, task/waiting/count, added by @QiuMM in #6657

New interfaces for extension developers

RequestLogEvent

It is now possible to control the fields in RequestLogEvent, emitted by EmittingRequestLogger. Please see #6477 for details. Added by @leventov.

Custom TLS certificate checks

An extension point for custom TLS certificate checks has been added. Please see http://druid.io/docs/0.14.0-incubating/operations/tls-support.html#custom-tls-certificate-checks for details. Added by @jon-wei in #6432.

Kafka Indexing Service no longer experimental

The Kafka Indexing Service extension has been moved out of experimental status.

SQL Enhancements

Enhancements to dsql

The dsql command line client now supports CLI history, basic autocomplete, and specifying query timeouts in the query context.

Added in #6929 by @gianm.

Add SQL id, request logs, and metrics

SQL queries now have an ID, and native queries executed as part of a SQL query will have the associated SQL query ID in the native query's request logs. SQL queries will now be logged in the request logs.

Two new metrics, sqlQuery/time and sqlQuery/bytes, are now emitted for SQL queries.

Please see http://druid.io/docs/0.14.0-incubating/configuration/index.html#request-logging and http://druid.io/docs/0.14.0-incubating/querying/sql.html#sql-metrics for details.

Added by @gaodayue in #6302

More SQL aggregator support

The follow aggregators are now supported in SQL:

  • DataSketches HLL sketch
  • DataSketches Theta sketch
  • DataSketches quantiles sketch
  • Fixed bins histogram
  • Bloom filter aggregator

Added by @jon-wei in #6951 and @clintropolis in #6502

Other SQL enhancements

  • SQL: Add support for queries with project-after-semijoin. #6756
  • SQL: Support for selecting multi-value dimensions. #6462
  • SQL: Support AVG on system tables. #601
  • SQL: Add "POSITION" function. #6596
  • SQL: Set INFORMATION_SCHEMA catalog name to "druid". #6595
  • SQL: Fix ordering of sort, sortProject in DruidSemiJoin. #6769

Added by @gianm.

Updating from 0.13.0-incubating and earlier

Kafka ingestion downtime when upgrading

Due to the issue described in #6958, existing Kafka indexing tasks can be terminated unnecessarily during a rolling upgrade of the Overlord. The terminated tasks will be restarted by the Overlord and will function correctly after the initial restart.

Parquet extension changes

The druid-parquet-extensions extension has been moved from contrib to core. When deploying 0.14.0-incubating, please ensure that your extensions-contrib directory does not have any older versions of the Parquet extension.

Additionally, there are now two styles of Parquet parsers in the extension:

  • parquet-avro: Converts Parquet to Avro, and then parses the Avro representation. This was the existing parser prior to 0.14.0-incubating.
  • parquet: A new parser that parses the Parquet format directly. Only this new parser supports int96 values.

Prior to 0.14.0-incubating, a specifying a parquet type parser would have a task use the Avro-converting parser. In 0.14.0-incubating, to continue using the Avro-converting parser, you will need to update your ingestion specs to use parquet-avro instead.

The inputFormat field in the inputSpec for tasks using Parquet input must also match the choice of parser:

  • parquet: org.apache.druid.data.input.parquet.DruidParquetInputFormat
  • parquet-avro: org.apache.druid.data.input.parquet.DruidParquetInputFormat

Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/parquet.html for details.

Running Druid with non-2.8.3 Hadoop

If you plan to use Druid 0.14.0-incubating with Hadoop versions other than 2.8.3, you may need to do the following:

  • Set the Hadoop dependency coordinates to your target version as described in http://druid.io/docs/0.14.0-incubating/operations/other-hadoop.html under Tip #3: Use specific versions of Hadoop libraries.
  • Rebuild Druid with your target version of Hadoop by changing hadoop.compile.version in the main Druid pom.xml and then following the standard build instructions.

Other Behavior changes

Old task cleanup

Old task entries in the metadata storage will now be cleaned up automatically together with their task logs. Please see http:/druid.io/docs/0.14.0-incubating/development/extensions-core/configuration/index.html#task-logging and #6592 for details.

Automatic processing buffer sizing

The druid.processing.buffer.sizeBytes property has new default behavior if it is not set. Druid will now automatically choose a value for the processing buffer size using the following formula:

processingBufferSize = totalDirectMemory / (numMergeBuffers + numProcessingThreads + 1)
processingBufferSize = min(processingBufferSize, 1GB)

Where:

  • totalDirectMemory: The direct memory limit for the JVM specified by -XX:MaxDirectMemorySize
  • numMergeBuffers: The value of druid.processing.numMergeBuffers.
  • numProcessingThreads: The value of druid.processing.numThreads.

At most, Druid will use 1GB for the automatically chosen processing buffer size. The processing buffer size can still be specified manually.

Please see #6588 for details.

Retention rules now include the future by default

Please be aware that new retention rules will now include the future by default. Please see #6414 for details.

Property changes

Segment announcing

The druid.announcer.type property used for choosing between Zookeeper or HTTP-based segment management/discovery has been moved to druid.serverview.type. If you were using http prior to 0.14.0-incubating, you will need to update your configs to use the new druid.serverview.type.

Please see the following for details:

fix missing property in JsonTypeInfo of SegmentWriteOutMediumFactory

The druid.peon.defaultSegmentWriteOutMediumFactory.@type property has been fixed. The property is now druid.peon.defaultSegmentWriteOutMediumFactory.type without the "@".

Please see #6656 for details.

Deprecations

Approximate Histogram aggregator

The ApproximateHistogram aggregator has been deprecated; it is a distribution-dependent algorithm without formal error bounds and has significant accuracy issues.

The DataSketches quantiles aggregator should be used instead for quantile and histogram use cases.

Please see Histogram and Quantiles Aggregators

Cardinality/HyperUnique aggregator

The Cardinality and HyperUnique aggregators have been deprecated in favor of the DataSketches HLL aggregator and Theta Sketch aggregator. These aggregators have better accuracy and performance characteristics.

Please see Count Distinct Aggregators for details.

Query Chunk Period

The chunkPeriod query context configuration is now deprecated, along with the associated query/intervalChunk/time metric. Please see #6591 for details.

keepSegmentGranularity for Compaction

The keepSegmentGranularity option for compaction tasks has been deprecated. Please see #6758 and the segmentGranularity table in http://druid.io/docs/0.14.0-incubating/ingestion/compaction.html for more information on these properties.

Interface changes for extension developers

SegmentId class

Druid now uses a SegmentId class instead of plain Strings to represent segment IDs. Please see #6370 for details.

Added by @leventov.

druid-api, druid-common, java-util moved to druid-core

The druid-api, druid-common, java-util modules have been moved into druid-core. Please update your dependencies accordingly if your project depended on these libraries.

Please see #6443 for details.

Credits

Thanks to everyone who contributed to this release!

@a2l007
@AlexanderSaydakov
@anantmf
@ankit0811
@asdf2014
@awelsh93
@benhopp
@Caroline1000
@clintropolis
@dclim
@deiwin
@DiegoEliasCosta
@drcrallen
@dyf6372
@Dylan1312
@egor-ryashin
@elloooooo
@evans
@FaxianZhao
@gaodayue
@gianm
@glasser
@Guadrado
@hate13
@hoesler
@hpandeycodeit
@janeklb
@jihoonson
@jon-wei
@jorbay-au
@jsun98
@justinborromeo
@kamaci
@leventov
@lxqfy
@mirkojotic
@navkumar
@niketh
@patelh
@pzhdfy
@QiuMM
@rcgarcia74
@richardstartin
@robertervin
@samarthjain
@seoeun25
@Shimi
@surekhasaharan
@taiii
@thomask
@VincentNewkirk
@vogievetsky
@yunwan
@zhaojiandong

druid - druid-0.14.0-incubating-rc3

Published by jon-wei over 5 years ago

druid - druid-0.13.0-incubating

Published by dclim almost 6 years ago

Druid 0.13.0-incubating contains over 400 new features, performance/stability/documentation improvements, and bug fixes from 81 contributors. It is the first release of Druid in the Apache Incubator program. Major new features and improvements include:

  • native parallel batch indexing
  • automatic segment compaction
  • system schema tables
  • improved indexing task status, statistics, and error reporting
  • SQL-compatible null handling
  • result-level broker caching
  • ingestion from RDBMS
  • Bloom filter support
  • additional SQL result formats
  • additional aggregators (stringFirst/stringLast, ArrayOfDoublesSketch, HllSketch)
  • support for multiple grouping specs in groupBy query
  • mutual TLS support
  • HTTP-based worker management
  • broker backpressure
  • maxBytesInMemory ingestion tuning configuration
  • materialized views (community extension)
  • parser for Influx Line Protocol (community extension)
  • OpenTSDB emitter (community extension)

The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+is%3Aclosed+milestone%3A0.13.0

Documentation for this release is at: http://druid.io/docs/0.13.0-incubating/

Highlights

Native parallel batch indexing

Introduces the index_parallel supervisor which manages the parallel batch ingestion of splittable sources without requiring a dependency on Hadoop. See http://druid.io/docs/latest/ingestion/native_tasks.html for more information.

Note: This is the initial single-phase implementation and has limitations on how it expects the input data to be partitioned. Notably, it does not have a shuffle implementation which will be added in the next iteration of this feature. For more details, see the proposal at #5543.

Added by @jihoonson in #5492.

Automatic segment compaction

Previously, compacting small segments into optimally-sized ones to improve query performance required submitting and running compaction or re-indexing tasks. This was often a manual process or required an external scheduler to handle the periodic submission of tasks. This patch implements automatic segment compaction managed by the coordinator service.

Note: This is the initial implementation and has limitations on interoperability with realtime ingestion tasks. Indexing tasks currently require acquisition of a lock on the portion of the timeline they will be modifying to prevent inconsistencies from concurrent operations. This implementation uses low-priority locks to ensure that it never interrupts realtime ingestion, but this also means that compaction may fail to make any progress if the realtime tasks are continually acquiring locks on the time interval being compacted. This will be improved in the next iteration of this feature with finer-grained locking. For more details, see the proposal at #4479.

Documentation for this feature: http://druid.io/docs/0.13.0-incubating/design/coordinator.html#compacting-segments

Added by @jihoonson in #5102.

System schema tables

Adds a system schema to the SQL interface which contains tables exposing information on served and published segments, nodes of the cluster, and information on running and completed indexing tasks.

Note: This implementation contains some known overhead inefficiencies that will be addressed in a future patch.

Documentation for this feature: http://druid.io/docs/0.13.0-incubating/querying/sql.html#system-schema

Added by @surekhasaharan in #6094.

Improved indexing task status, statistics, and error reporting

Improves the performance and detail of the ingestion-related APIs which were previously quite opaque making it difficult to determine the cause of parse exceptions, task failures, and the actual output from a completed task. Also adds improved ingestion metric reporting including moving average throughput statistics.

Added by @surekhasaharan and @jon-wei in #5801, #5418, and #5748.

SQL-compatible null handling

Improves Druid's handling of null values by treating them as missing values instead of being equivalent to empty strings or a zero-value. This makes Druid more SQL compatible and improves integration with external BI tools supporting ODBC/JDBC. See #4349 for proposal.

To enable this feature, you will need to set the system-wide property druid.generic.useDefaultValueForNull=false.

Added by @nishantmonu51 in #5278 and #5958.

Results-level broker caching

Implements result-level caching on brokers which can operate concurrently with the traditional segment-level cache. See #4843 for proposal.

Documentation for this feature: http://druid.io/docs/0.13.0-incubating/configuration/index.html#broker-caching

Added by @a2l007 in #5028.

Ingestion from RDBMS

Introduces a sql firehose which supports data ingestion directly from an RDBMS.

Added by @a2l007 in #5441.

Bloom filter support

Adds support for optimizing Druid queries by applying a Bloom filter generated by an external system such as Apache Hive. In the future, #6397 will support generation of Bloom filters as the result of Druid queries which can then be used to optimize future queries.

Added by @nishantmonu51 in #6222.

Additional SQL result formats

Adds result formats for line-based JSON and CSV and additionally X-Druid-Column-Names and X-Druid-Column-Types response headers containing a list of columns contained in the result.

Added by @gianm in #6191.

'stringLast' and 'stringFirst' aggregators

Introduces two complementary aggregators, stringLast and stringFirst which operate on string columns and return the value with the maximum and minimum timestamp respectively.

Added by @andresgomezfrr in #5789.

ArrayOfDoublesSketch

Adds support for numeric Tuple sketches, which extend the functionality of the count distinct Theta sketches by adding arrays of double values associated with unique keys.

Added by @AlexanderSaydakov in #5148.

HllSketch

Adds a configurable implementation of a count distinct aggregator based on HllSketch from https://github.com/DataSketches. Comparison to Druid's native HyperLogLogCollector shows improved accuracy, efficiency, and speed: https://datasketches.github.io/docs/HLL/HllSketchVsDruidHyperLogLogCollector.html

Added by @AlexanderSaydakov in #5712.

Support for multiple grouping specs in groupBy query

Adds support for the subtotalsSpec groupBy parameter which allows Druid to be efficient by reusing intermediate results at the broker level when running multiple queries that group by subsets of the same set of columns. See proposal in #5179 for more information.

Added by @himanshug in #5280.

Mutual TLS support

Adds support for mutual TLS (server certificate validation + client certificate validation). See: https://en.wikipedia.org/wiki/Mutual_authentication

Added by @jon-wei in #6076.

HTTP based worker management

Adds an HTTP-based indexing task management implementation to replace the previous one based on ZooKeeper. Part of a set of improvements to reduce and eventually eliminate Druid's dependency on ZooKeeper. See #4996 for proposal.

Added by @himanshug in #5104.

Broker backpressure

Allows the broker to exert backpressure on data-serving nodes to prevent the broker from crashing under memory pressure when results are coming in faster than they are being read by clients.

Added by @gianm in #6313.

'maxBytesInMemory' ingestion tuning configuration

Previously, a major tuning parameter for indexing task memory management was the maxRowsInMemory configuration, which determined the threshold for spilling the contents of memory to disk. This was difficult to properly configure since the 'size' of a row varied based on multiple factors. maxBytesInMemory makes this configuration byte-based instead of row-based.

Added by @surekhasaharan in #5583.

Materialized views

Supports the creation of materialized views which can improve query performance in certain situations at the cost of additional storage. See http://druid.io/docs/latest/development/extensions-contrib/materialized-view.html for more information.

Note: This is a community-contributed extension and is not automatically included in the Druid distribution. We welcome feedback for deciding when to promote this to a core extension. For more information, see Community Extensions.

Added by @zhangxinyu1 in #5556.

Parser for Influx Line Protocol

Adds support for ingesting the Influx Line Protocol data format. For more information, see: https://docs.influxdata.com/influxdb/v1.6/write_protocols/line_protocol_tutorial/

Note: This is a community-contributed extension and is not automatically included in the Druid distribution. We welcome feedback for deciding when to promote this to a core extension. For more information, see Community Extensions.

Added by @njhartwell in #5440.

OpenTSDB emitter

Adds support for emitting Druid metrics to OpenTSDB.

Note: This is a community-contributed extension and is not automatically included in the Druid distribution. We welcome feedback for deciding when to promote this to a core extension. For more information, see Community Extensions.

Added by @QiuMM in #5380.

Updating from 0.12.3 and earlier

Please see below for changes between 0.12.3 and 0.13.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.12.3, please see release notes of the relevant intermediate versions for additional notes.

MySQL metadata storage extension no longer includes JDBC driver

The MySQL metadata storage extension is now packaged together with the Druid distribution but without the required MySQL JDBC driver (due to licensing restrictions). To use this extension, the driver will need to be downloaded separately and added to the extension directory.

See http://druid.io/docs/latest/development/extensions-core/mysql.html for more details.

AWS region configuration required for S3 extension

As a result of switching from jets3t to the AWS SDK (#5382), users of the S3 extension are now required to explicitly set the target region. This can be done by setting the JVM system property aws.region or the environment variable AWS_REGION.

As an example, to set the region to 'us-east-1' through system properties:

  • add -Daws.region=us-east-1 to the jvm.config file for all Druid services
  • add -Daws.region=us-east-1 to druid.indexer.runner.javaOpts in middleManager/runtime.properties so that the property will be passed to peon (worker) processes

Ingestion spec changes

As a result of renaming packaging from io.druid to org.apache.druid, ingestion specs that reference classes by their fully-qualified class name will need to be modified accordingly.

As an example, if you are using the Parquet extension with Hadoop indexing, the inputFormat field of the inputSpec will need to change from io.druid.data.input.parquet.DruidParquetInputFormat to org.apache.druid.data.input.parquet.DruidParquetInputFormat.

Metrics changes

New metrics

  • task/action/log/time - Milliseconds taken to log a task action to the audit log (#5714)
  • task/action/run/time - Milliseconds taken to execute a task action (#5714)
  • query/node/backpressure - Nanoseconds the channel is unreadable due to backpressure being applied (#6335) (Note that this is not enabled by default and requires a custom implementation of QueryMetrics to emit)

New dimensions

  • taskId and taskType added to task-related metrics (#5664)

Other

  • HttpPostEmitterMonitor no longer emits maxTime and minTime if no times were recorded (#6418)

Rollback restrictions

64-bit doubles aggregators

64-bit doubles aggregators are now used by default (see #5478). Support for 64-bit floating point columns was release in Druid 0.11.0, so if this is enabled, versions older than 0.11.0 will not be able to read the data segments.

To disable and keep the old format, you will need to set the system-wide property druid.indexing.doubleStorage=float.

Disabling bitmap indexes

0.13.0 adds support for disabling bitmap indexes on a per-column basis, which can save space in cases where bitmap indexes add no value. This is done by setting the 'createBitmapIndex' field in the dimension schema. Segments written with this option will not be backwards compatible with older versions of Druid (#5402).

Behavior changes

Java package name changes

Druid's package names have all moved from io.druid to org.apache.druid. This affects the name of the Java main class that you should run when starting up services, which is now org.apache.druid.cli.Main. It may also affect installation and configuration of extensions and monitors.

ParseSpec is now a required field in ingestion specs

There is no longer a default ParseSpec (previously the DelimitedParseSpec). Ingestion specs now require parseSpec to be specified. If you previously did not provide a parseSpec, you should use one with "format": "tsv" to maintain the existing behavior (#6310).

Change to default maximum rows to return in one JDBC frame

The default value for druid.sql.avatica.maxRowsPerFrame was reduced from 100k to 5k to minimize out of memory errors (#5409).

Router behavior change when routing to brokers dedicated to different time ranges

As a result of #5595, routers may now select an undesired broker in configurations where there are different tiers of brokers that are intended to be dedicated to queries on different time ranges. See #1362 for discussion.

Ruby TimestampSpec no longer ignores milliseconds

Timestamps parsed using a TimestampSpec with format 'ruby' no longer truncates the millisecond component. If you were using this parser and wanted a query granularity of SECOND, ensure that it is configured appropriately in your indexing specs (#6217).

Small increase in size of ZooKeeper task announcements

The datasource name was added to TaskAnnouncement which will result in a small per task increase in the amount of data stored in ZooKeeper (#5511).

Addition of 'name' field to filtered aggregators

Aggregators of type 'filtered' now support a 'name' field. Previously, the filtered aggregator inherited the name of the aggregator it wrapped. If you have provided the 'name' field for both the filtered aggregator and the wrapped aggregator, it will prefer the name of the filtered aggregator. It will use the name of the wrapped aggregator if the name of the filtered aggregator is missing or empty (#6219).

utf8mb4 is now the recommended metadata storage charset

For upgrade instructions, use the ALTER DATABASE and ALTER TABLE instructions as described here: https://dev.mysql.com/doc/refman/5.7/en/charset-unicode-conversion.html.

For motivation and reference, see #5377 and #5411.

Removed configuration properties

  • druid.indexer.runner.tlsStartPort has been removed (#6194).
  • druid.indexer.runner.separateIngestionEndpoint has been removed (#6263).

Interface changes for extension developers

  • Packaging has been renamed from io.druid to org.apache.druid. All third-party extensions will need to rename their META-INF/io.druid.initialization.DruidModule to org.apache.druid.initialization.DruidModule and update their extension's packaging appropriately (#6266).

  • The DataSegmentPuller interface has been removed (#5461).

  • A number of functions under java-util have been removed (#5461).

  • The constructor of the Metadata class has changed (#5613).

  • The 'spark2' Maven profile has been removed (#5581).

API deprecations

Overlord

  • The /druid/indexer/v1/supervisor/{id}/shutdown endpoint has been deprecated in favor of /druid/indexer/v1/supervisor/{id}/terminate (#6272 and #6234).
  • The /druid/indexer/v1/task/{taskId}/segments endpoint has been deprecated (#6368).
  • The status field returned by /druid/indexer/v1/task/ID/status has been deprecated in favor of statusCode (#6334).
  • The reportParseExceptions and ignoreInvalidRows parameters for ingestion tuning configurations have been deprecated in favor of logParseExceptions and maxParseExceptions (#5418).

Broker

  • The /druid/v2/datasources/{dataSourceName}/dimensions endpoint has been deprecated. A segment metadata query or the INFORMATION_SCHEMA SQL table should be used instead (#6361).
  • The /druid/v2/datasources/{dataSourceName}/metrics endpoint has been deprecated. A segment metadata query or the INFORMATION_SCHEMA SQL table should be used instead (#6361).

Credits

Thanks to everyone who contributed to this release!

@a2l007
@adursun
@ak08
@akashdw
@aleksi75
@AlexanderSaydakov
@alperkokmen
@amalakar
@andresgomezfrr
@apollotonkosmo
@asdf2014
@awelsh93
@b-slim
@bolkedebruin
@Caroline1000
@chengchengpei
@clintropolis
@dansuzuki
@dclim
@DiegoEliasCosta
@dragonls
@drcrallen
@dyanarose
@dyf6372
@Dylan1312
@erikdubbelboer
@es1220
@evasomething
@fjy
@Fokko
@gaodayue
@gianm
@hate13
@himanshug
@hoesler
@jaihind213
@jcollado
@jihoonson
@jim-slattery-rs
@jkukul
@jon-wei
@josephglanville
@jsun98
@kaijianding
@KenjiTakahashi
@kevinconaway
@korvit0
@leventov
@lssenthilkumar
@mhshimul
@niketh
@NirajaM
@nishantmonu51
@njhartwell
@palanieppan-m
@pdeva
@pjain1
@QiuMM
@redlion99
@rpless
@samarthjain
@Scorpil
@scrawfor
@shiroari
@shivtools
@siddharths
@SpyderRivera
@spyk
@stuartmclean
@surekhasaharan
@susielu
@varaga
@vineshcpaul
@vvc11
@wysstartgo
@xvrl
@yunwan
@yuppie-flu
@yurmix
@zhangxinyu1
@zhztheplayer

druid - druid-0.12.3

Published by jon-wei about 6 years ago

Druid 0.12.3 contains stability improvements and bug fixes from 6 contributors. Major improvements include:

  • More stable Kafka indexing service
  • Several query bug fixes

The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+milestone%3A0.12.3+is%3Aclosed

Documentation for this release is at: http://druid.io/docs/0.12.3

Highlights

More stable Kafka indexing service

0.12.3 fixes a serious issue where the Kafka indexing service would incorrectly delete published segments in certain situations. Please see #6155, contributed by @gianm, for details.

Other stability improvements include:

  • Fix NPE in KafkaSupervisor.checkpointTaskGroup: #6206, added by @jihoonson
  • Fix NPE for taskGroupId when rolling update: #6168, added by @jihoonson

Query bug fixes

0.12.3 includes a memory allocation adjustment for the GroupBy query that should reduce heap usage. Added by @gaodayue in #6256.

This release also contains several fixes for the TopN and GroupBy queries when numeric dimension output types are used. Added by @gianm in #6220.

SQL fixes

Additionally, 0.12.3 includes the following Druid SQL bug fixes:

Other

0.12.3 fixes an issue with the druid-basic-security extension where non-coordinator nodes would sometimes fail when updating their cached views of the authentication and/or authorization tables from the coordinator. Fixed by @gaodayue in #6270.

Full change list

The full change list can be found here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+milestone%3A0.12.3+is%3Aclosed

Updating from 0.12.2 and earlier

0.12.3 is a minor release and compatible with 0.12.2. If you're updating from an earlier version than 0.12.2, please see release notes of the relevant intermediate versions for additional notes.

Credits

Thanks to everyone who contributed to this release!

@gaodayue
@gianm
@jihoonson
@jon-wei
@QiuMM
@vogievetsky
@wpcnjupt

druid - druid-0.12.2

Published by jihoonson about 6 years ago

Druid 0.12.2 contains stability improvements and bug fixes from 13 contributors. Major improvements include:

  • More stable Kafka indexing service
  • More stable data ingestion
  • More stable segment balancing
  • Bug fixes in querying and result caching

The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+milestone%3A0.12.2+is%3Aclosed

Documentation for this release is at: http://druid.io/docs/0.12.2-rc1

Highlights

More stable Kafka indexing service

We have fixed a bunch of bugs in Kafka indexing service, which are mostly race conditions when incrementally publishing segments.

Added by @jihoonson in #5805.
Added by @surekhasaharan in #5899.
Added by @surekhasaharan in #5900.
Added by @jihoonson in #5905.
Added by @jihoonson in #5907.
Added by @jihoonson in #5996.

More stable data ingestion

We also have fixed some bugs in general data ingestion logics. Especially we have fixed a bug of wrong segment data when you use auto encoded long columns with any compression.

Added by @jihoonson in #5932.
Added by @clintropolis in #6045.

More stable segment balancing

Coordinator is now capable of more stable segment management especially for segment balancing. We have fixed an unexpected segment imbalancing caused by the conflicted decisions of Coordinator rule runner and balancer.

Added by @clintropolis in #5528.
Added by @clintropolis in #5529.
Added by @clintropolis in #5532.
Added by @clintropolis in #5555.
Added by @clintropolis in #5591.
Added by @clintropolis in #5888.
Added by @clintropolis in #5928.

Bug fixes in querying and result caching

We've fixed the wrong lexicographic sort of topN queries and the wrong filter application for the nested queries. The bug of ClassCastException when caching topN queries with Float dimensions has also fixed.

Added by @drcrallen in #5650.
Added by @gianm in #5653.

And much more!

The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+milestone%3A0.12.2+is%3Aclosed

Updating from 0.12.1 and earlier

0.12.2 is a minor release and compatible with 0.12.1. If you're updating from an earlier version than 0.12.1, please see release notes of the relevant intermediate versions for additional notes.

Credits

Thanks to everyone who contributed to this release!

@acdn-ekeddy
@awelsh93
@clintropolis
@drcrallen
@gianm
@jihoonson
@jon-wei
@kaijianding
@leventov
@michas2
@Oooocean
@samarthjain
@surekhasaharan

druid - druid-0.12.1

Published by jihoonson over 6 years ago

Druid 0.12.1 contains stability improvements and bug fixes from 10 contributors. Major improvements include:

  • Large performance improvements for coordinator's loadstatus API
  • More memory limiting for HttpPostEmitter
  • Fix several issues of Kerberos Authentication
  • Fix SQLMetadataSegmentManager to allow successive start and stop
  • Fix default interval handling in SegmentMetadataQuery
  • Support HTTP OPTIONS request
  • Fix a bug of different segments of the same segment id in Kafka indexing

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+milestone%3A0.12.1

Documentation for this release is at: http://druid.io/docs/0.12.1

Highlights

Large performance improvements for coordinator's loadstatus API

The loadstatus API of Coordinators returns the percentage of segments actually loaded in the cluster versus segments that should be loaded in the cluster. The performance of this API has greatly been improved.

Added by @jon-wei in #5632.

More memory limiting for HttpPostEmitter

Druid now can limit the amount of memory used by HttpPostEmitter to 10% of the available JVM heap, thereby avoiding OutOfMemory errors from buffered events.

Added by @jon-wei in #5300.

Fix several issues of Kerberos Authentication

There were some bugs in Kerberos authentication like authentication failure without cookies or broken authentication when router is used. See https://github.com/druid-io/druid/pull/5596, https://github.com/druid-io/druid/pull/5706, and https://github.com/druid-io/druid/pull/5766 for more details.

Added by @nishantmonu51 in #5596.
Added by @b-slim in #5706.
Added by @jon-wei in #5766.

Fix SQLMetadataSegmentManager to allow successive start and stop

Coordinators could be stuck if it loses leadership while starting. This bug has been fixed now.

Added by @jihoonson in #5554.

Fix default interval handling in SegmentMetadataQuery

SegmentMetadataQuery is supposed to use the interval of druid.query.segmentMetadata.defaultHistory if the interval is not specified, but it queried all segments instead which incurs an unexpected performance hit. SegmentMetadataQuery now respects the defaultHistory option again.

Added by @gianm in #5489.

Support HTTP OPTIONS request

Druid now supports the HTTP OPTIONS request by fixing its auth handling.

Added by @jon-wei in #5615.

Fix a bug of different segments of the same segment id in Kafka indexing

Kafka indexing service allowed retrying tasks to overwrite the segments in deep storage written by the previous failed tasks. However, this caused another bug that the same segment ID could have different data on historicals and in deep storage. This bug has been fixed now by using unique segment paths for each Kafka index tasks.

Added by @dclim in #5692.

And much more!

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+milestone%3A0.12.1

Updating from 0.12.0 and earlier

0.12.1 is a minor release and compatible with 0.12.0. If you're updating from an earlier version than 0.12.0, please see release notes of the relevant intermediate versions for additional notes.

Credits

Thanks to everyone who contributed to this release!

@dclim
@gianm
@JeKuOrdina
@jihoonson
@jon-wei
@leventov
@niketh
@nishantmonu51
@pdeva

druid - druid-0.12.0

Published by jon-wei over 6 years ago

Druid 0.12.0 contains over a hundred performance improvements, stability improvements, and bug fixes from almost 40 contributors. This release adds major improvements to the Kafka indexing service.

Other major new features include:

  • Prioritized task locking
  • Improved automatic segment management
  • Test stats post-aggregators
  • Numeric quantiles sketch aggregator
  • Basic auth extension
  • Query request queuing improvements
  • Parse batch support
  • Various performance improvements
  • Various improvements to Druid SQL

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.12.0

Documentation for this release is at: http://druid.io/docs/0.12.0/

Highlights

Kafka indexing incremental handoffs and decoupled partitioning

The Kafka indexing service now supports incremental handoffs, as well as decoupling the number of segments created by a Kafka indexing task from the number of Kafka partitions. Please see https://github.com/druid-io/druid/pull/4815#issuecomment-346155552 for more information.

Added by @pjain1 in #4815.

Prioritized task locking

Druid now supports priorities for indexing task locks. When an indexing task needs to acquire a lock on a datasource + interval, higher-priority tasks can now preempt lower-priority tasks. Please see http://druid.io/docs/0.12.0-rc1/ingestion/tasks.html#task-priority for more information.

Added by @jihoonson in https://github.com/druid-io/druid/pull/4550.

Improved automatic segment management

Automatic pending segments cleanup

Indexing tasks create entries in the "pendingSegments" table in the metadata store; prior to 0.12.0, these temporary entries were not automatically cleaned up, leading to possible cluster performance degradation over time. Druid 0.12.0 allows the coordinator to automatically clean up unused entries in the pending segments table. This feature is enabled by setting druid.coordinator.kill.pendingSegments.on=true in coordinator properties.

Added by @jihoonson in #5149.

Compaction task

Compacting segments (merging a set of segments within a given interval to create a set with larger but fewer segments) is a common Druid batch ingestion use case. Druid 0.12.0 now supports a Compaction Task that merges all segments within a given interval into a single segment. Please see http://druid.io/docs/0.12.0-rc1/ingestion/tasks.html#compaction-task for more details.

Added by @jihoonson in #4985.

Test stats post-aggregators

New z-score and p-value test statistics post-aggregators have been added to the druid-stats extension. Please see http://druid.io/docs/0.12.0-rc1/development/extensions-core/test-stats.html for more details.

Added by @chunghochen in #4532.

Numeric quantiles sketch aggregator

A numeric quantiles sketch aggregator has been added to the druid-datasketches extension.

Added by @AlexanderSaydakov in #5002.

Basic auth extension

Druid 0.12.0 includes a new authentication/authorization extension that provides Basic HTTP authentication and simple role-based access control. Please see http://druid.io/docs/0.12.0-rc1/development/extensions-core/druid-basic-security.html for more information.

Added by @jon-wei in #5099.

Query request queuing improvements

Currently clients can overwhelm a broker inadvertently by sending too many requests which get queued in an unbounded Jetty worker pool queue. Clients typically close the connection after a certain client-side timeout but the broker will continue to process these requests, giving the appearance of being unresponsive. Meanwhile, clients would continue to retry, continuing to add requests to an already overloaded broker..

The newly introduced properties druid.server.http.queueSize and druid.server.http.enableRequestLimit in the broker configuration and historical configuration allow users to configure request rejection to prevent clients from overwhelming brokers and historicals with queries.

Added by @himanshug in #4540.

Parse batch support

For developers of custom ingestion parsing extensions, it is now possible for InputRowParsers to return multiple InputRows from a single input row. This can simplify ingestion pipelines by reducing the need for input transformations outside of Druid.
Added by @pjain1 in #5081.

Performance improvements

SegmentWriteOutMedium

When creating new segments, Druid stores some pre-processed data in temporary buffers. Prior to 0.12.0, these buffers were always kept in temporary files on disk. In 0.12.0, PR #4762 by @leventov allows these temporary buffers to be stored in off-heap memory, thus reducing the number of disk I/O operations during ingestion. To enable using off-heap memory for these buffers, the druid.peon.defaultSegmentWriteOutMediumFactory property needs to be configured accordingly. If using off-heap memory for the temporary buffers, please ensure that -XX:MaxDirectMemorySize is increased to accommodate the higher direct memory usage.

Please see http://druid.io/docs/0.12.0-rc1/configuration/indexing-service.html#SegmentWriteOutMediumFactory for configuration details.

Parallel merging of intermediate GroupBy results

PR #4704 by @jihoonson allows the user to configure a number of processing threads to be used for parallel merging of intermediate GroupBy results that have been spilled to disk. Prior to 0.12.0, this merging step would always take place within a single thread.

Please see http://druid.io/docs/0.12.0-rc1/configuration/querying/groupbyquery.html#parallel-combine for configuration details.

Other performance improvements

  • DataSegment memory optimizations: #5094 by @leventov
  • Remove IndexedInts.iterator(): #4811 by @leventov
  • ExpressionSelectors: Add optimized selectors: #5048 by @gianm

SQL improvements

Various improvements and features have been added to Druid SQL, by @gianm in the following PRs:

  • Improve translation of time floor expressions: #5107
  • Add TIMESTAMPADD: #5079
  • Add rule to prune unused aggregations: #5049
  • Support CASE-style filtered count distinct: #5047
  • Add "array" result format, and document result formats: #5032
  • Fix havingSpec on complex aggregators: #5024
  • Improved behavior when implicitly casting strings to date/time literals: #5023
  • Add Router connection balancers for Avatica queries: #4983
  • Fix incorrect filter simplification: #4945
  • Fix Router handling of SQL queries: #4851

And much more!

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.12.0

Updating from 0.11.0 and earlier

Please see below for changes between 0.11.0 and 0.12.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.11.0, please see release notes of the relevant intermediate versions for additional notes.

Rollback restrictions

Please note that after upgrading to 0.12.0, it is no longer possible to downgrade to a version older than 0.11.0, due to changes made in https://github.com/druid-io/druid/pull/4762. It is still possible to roll back to version 0.11.0.

com.metamx.java-util library migration

The Metamarkets java-util library has been brought into Druid. As a result, the following package references have changed:

com.metamx.common -> io.druid.java.util.common
com.metamx.emitter -> io.druid.java.util.emitter
com.metamx.http -> io.druid.java.util.http
com.metamx.metrics -> io.druid.java.util.metrics

This will affect the druid.monitoring.monitors configuration. References to monitor classes under the old com.metamx.metrics.* package will need to be updated to reference io.druid.java.metrics.* instead, e.g. io.druid.java.util.metrics.JvmMonitor.

If classes under the the com.metamx packages shown above are referenced in other configurations such as log4j2.xml, those references will need to be updated as well.

Extension developers will need to update their code to use the new Druid packages as well.

Caffeine cache extension

The Caffeine cache extension has been moved out of an extension, into core Druid. In addition, the Caffeine cache is now the default cache implementation. Please remove druid-caffeine-cache if present from the extension list when upgrading to 0.12.0. More information can be found at https://github.com/druid-io/druid/pull/4810.

Kafka indexing service changes

earlyMessageRejectPeriod

The semantics of the earlyMessageRejectPeriod configuration have changed. The earlyMessageRejectPeriod will now be added to (task start time + task duration) instead of just (task start time) when determining the bounds of the message window. Please see https://github.com/druid-io/druid/pull/4990 for more information.

Rolling upgrade

In 0.12.0, there are protocol changes between the Kafka supervisor and Kafka Indexing task and also some changes to the metadata formats persisted on disk. Therefore, to support rolling upgrade, all the Middle Managers will need to be upgraded first before the Overlord. Note that this ordering is different from the standard order of upgrade, also note that this ordering is only necessary when using the Kafka Indexing Service. If one is not using Kafka Indexing Service or can handle down time for Kafka Supervisor then one can upgrade in any order.

Until the point in time Overlord is upgraded, all the Kafka Indexing Task will behave in same manner (even if they are upgraded) as earlier which means no decoupling and incremental hand-offs. Once, Overlord is upgraded, the new tasks started by the upgraded Overlord will support the new features.

Please see https://github.com/druid-io/druid/pull/4815 for more info.

Roll back

Once both the overlord and middle managers are rolled back, a new set of tasks should be started, which will work properly. However, the current set of tasks may fail during a roll back. Please see #4815 for more info.

Interface Changes for Extension Developers

The ColumnSelectorFactory API has changed. Aggregator extension authors and any others who use ColumnSelectorFactory will need to update their code accordingly. Please see https://github.com/druid-io/druid/pull/4886 for more details.

The Aggregator.reset() method has been removed because it was deprecated and unused. Please see https://github.com/druid-io/druid/pull/5177 for more info.

The DataSegmentPusher interface has changed, and the push() method now has an additional replaceExisting parameter. Please see https://github.com/druid-io/druid/pull/5187 for details.

The Escalator interface has changed: the createEscalatedJettyClient method has been removed. Please see https://github.com/druid-io/druid/pull/5322 for more details.

Credits

Thanks to everyone who contributed to this release!

@a2l007
@akashdw
@AlexanderSaydakov
@b-slim
@ben-manes
@benvogan
@chuanlei
@chunghochen
@clintropolis
@daniel-tcell
@dclim
@dpenas
@drcrallen
@egor-ryashin
@elloooooo
@Fokko
@fuji-151a
@gianm
@gvsmirnov
@himanshug
@hzy001
@Igosuki
@jihoonson
@jon-wei
@KenjiTakahashi
@kevinconaway
@leventov
@mh2753
@niketh
@nishantmonu51
@pjain1
@QiuMM
@QubitPi
@quenlang
@Shimi
@skyler-tao
@xanec
@yuppie-flu
@zhangxinyu1

druid -

Published by gianm almost 7 years ago

Druid 0.11.0 contains over a hundred performance improvements, stability improvements, and bug fixes from almost 40 contributors. This release adds two major security features, TLS support and extension points for authentication and authorization.

Major new features include:

  • TLS (a.k.a. SSL) support
  • Extension points for authentication and authorization
  • Double columns support
  • cachingCost Balancer Strategy
  • jq expression support in JSON parser
  • Redis cache extension
  • GroupBy performance improvements
  • Various improvements to Druid SQL

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.11.0

Documentation for this release is at: http://druid.io/docs/0.11.0/

Highlights

TLS support

Druid now supports TLS, enabling encrypted client and inter-node communications. Please see http://druid.io/docs/0.11.0/operations/tls-support.html for details on configuration and related extensions.

Added by @pjain1 in https://github.com/druid-io/druid/pull/4270.

Authentication/authorization extension points

Extension points for authenticating and authorizing requests have been added to Druid. Please see http://druid.io/docs/0.11.0/configuration/auth.html for information on configuration and extension implementation.

The existing Kerberos authentication extension has been updated to implement the new Authenticator interface, please see the "Kerberos configuration changes" section under "Updating from 0.10.1 and earlier" for more information if you are using the Kerberos extension.

Added by @jon-wei in https://github.com/druid-io/druid/pull/4271

Double columns support

Druid now supports Double type aggregator columns. Please see http://druid.io/docs/0.11.0/querying/aggregations.html for documentation on the new Double aggregators.

Added by @b-slim in https://github.com/druid-io/druid/pull/4491.

cachingCost Balancer Strategy

Users upgrading to 0.11.0 are encouraged to try the new cachingCost segment balancing strategy on their coordinators. This strategy offers large performance improvements over the existing cost balancer strategy, and it is planned to become the default strategy in the release following 0.11.0.

This strategy can be selected by setting the following property on coordinators:

druid.coordinator.balancer.strategy=cachingCost

Added by @dgolitsyn in https://github.com/druid-io/druid/pull/4731

jq expression support in JSON parser

Druid's JSON input parser now supports jq expressions using jackson-jq, enabling more input transforms before ingestion. Please see http://druid.io/docs/0.11.0/ingestion/flatten-json.html for more details.

Added by @knoguchi in https://github.com/druid-io/druid/pull/4171.

Redis cache extension

A new cache implementation using Redis has been added in an extension, added by @QiuMM in https://github.com/druid-io/druid/pull/4615. Please refer to the preceding pull request for more details.

GroupBy performance improvements

Several new performance optimizations have been added to the GroupBy query by @jihoonson in the following PRs:

https://github.com/druid-io/druid/pull/4660 Parallel sort for ConcurrentGrouper
https://github.com/druid-io/druid/pull/4576 Array-based aggregation for groupBy query
https://github.com/druid-io/druid/pull/4668 Add IntGrouper to avoid unnecessary boxing/unboxing in array-based aggregation

PR #4660 offers a general improvement by parallelizing partial result sorting, while PR #4576 and #4668 offer significant improvements when grouping on a single String column.

SQL improvements

Various improvements and features have been added to Druid SQL, by @gianm in the following PRs:

https://github.com/druid-io/druid/pull/4750 - TRIM support
https://github.com/druid-io/druid/pull/4720 - Rounding for count distinct
https://github.com/druid-io/druid/pull/4561 - Metrics for SQL queries
https://github.com/druid-io/druid/pull/4360 - SQL expressions support

And much more!

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.11.0

Updating from 0.10.1 and earlier

Please see below for changes between 0.10.1 and 0.11.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.10.1, please see release notes of the relevant intermediate versions for additional notes.

Upgrading coordinators and overlords

The following patch changes the way coordinator->overlord redirects are handled:
https://github.com/druid-io/druid/pull/5037

The overlord leader election algorithm has changed in 0.11.0: https://github.com/druid-io/druid/pull/4699.

As a result of the two patches above, special care is needed when upgrading Coordinator or Overlord to 0.11.0. All coordinators and overlords must be shut down and upgraded together.

For example, to upgrade Coordinators, you would shutdown all coordinators, upgrade them to 0.11.0 and then start them. Overlords should be upgraded in a similar way.

During the upgrade process, there must not be any time period where a non-0.11.0 coordinator or overlord is running simultaneously with an 0.11.0 coordinator or overlord.

Note that at least one overlord should be brought up as quickly as possible after shutting them all down so that peons, tranquility etc continue to work after some retries.

Also note that the druid.zk.paths.indexer.leaderLatchPath property is no longer used now.

Service name changes

In earlier versions of Druid, / characters in service names defined by druid.service would be replaced by : characters because these service names were used in Zookeeper paths. Druid 0.11.0 no longer performs these character replacements.

Example:1 - if the old configuration had a broker with service name test/broker:
druid.service=test/broker

and a Router was configured assuming that / will be replaced with : in the broker service name,
druid.router.tierToBrokerMap={"hot":"test:broker","_default_tier":"test:broker"}

the Router configuration should be updated to remove that assumption:
druid.router.tierToBrokerMap={"hot":"test/broker","_default_tier":"test/broker"}

Example:2 - If the old configuration had overlord with service Name test/overlord then value of druid.coordinator.asOverlord.overlordService or druid.selectors.indexing.serviceName should be test/overlord and not test:overlord

Example:3 - If the old configuration had overlord with service Name test:overlord then value of druid.coordinator.asOverlord.overlordService or druid.selectors.indexing.serviceName should be test:overlord and not test/overlord

Following service name-related configurations are also affected and should be updated to exactly match the value of druid.service property on other node being discovered.

druid.coordinator.asOverlord.overlordService
druid.selectors.coordinator.serviceName
druid.selectors.indexing.serviceName
druid.router.defaultBrokerServiceName
druid.router.coordinatorServiceName
druid.router.tierToBrokerMap

Please see https://github.com/druid-io/druid/issues/4992 for more details.

Kerberos configuration changes

The Kerberos authentication configuration format has changed as a result of the new interfaces introduced by #4271. Please refer to http://druid.io/docs/0.11.0/development/extensions-core/druid-kerberos.html for the new configuration properties.

Users can point the Kerberos authenticator's authorizerName to an instance of an "allowAll" authorizer to replicate the pre-0.11.0 behavior of a cluster using Kerberos authentication with no authorization.

Lookups API path changes

The paths for the lookups configuration API have changed due to #5058.

Configuration paths that had the form /druid/coordinator/v1/lookups now have the form /druid/coordinator/v1/lookups/config.

Please see http://druid.io/docs/0.11.0/querying/lookups.html for the current API.

Migrating to Double columns

Prior to 0.11.0, the Double* aggregators would store column values on disk as Float while performing aggregations using Double representations.

PR #4491 allows the Double aggregators to store column values on disk as Doubles. Due to concerns related to rolling updates and version downgrades, this behavior is disabled by default and Druid will continue to store Double aggregators on disk as floats.

To enable Double column storage, set the following property in the common runtime properties:

druid.indexing.doubleStorage=double

Users should not set this property during an initial rolling upgrade to 0.11.0, as any nodes running pre-0.11.0 Druid will not be able to handle Double columns created during the upgrade period. Users will also need to reindex any segments with Double columns if downgrading from 0.11.0 to an older version. Please see #4944 and #4605 for more information.

Scan query changes

The Scan query has been moved from extensions-contrib to core Druid. As part of this migration: https://github.com/druid-io/druid/pull/4751, the scan query's handling of the time column has changed.

The time column is now is returned as "__time" rather than "timestamp", it is no longer included if you do not specifically ask for it in your "columns", and it is returned as a long rather than a string.

Users can revert the Scan query's time handling to the legacy extension behavior by setting "legacy" : true in their queries, or setting the property druid.query.scan.legacy = true. This is meant to provide a migration path for users that were formerly using the contrib extension.

Extension Interface Changes

Aggregator double column support

The Aggregator interface has gained a getDouble() method, which defaults to casting the result of getFloat(). The getDouble() method should be re-implemented for any custom aggregators that can support doubles.

See https://github.com/druid-io/druid/pull/4595 for more details.

QueryRunner interface change

The QueryRunner interface has changed and the old run() method has been removed, replaced by a new method that accepts a QueryPlus object.

Custom query extensions will need to implement the new interface.

Please see https://github.com/druid-io/druid/pull/4184 and https://github.com/druid-io/druid/pull/4482 for more details.

Filter interface change

The Filter.getBitmapResult() method no longer has a default implementation: https://github.com/druid-io/druid/pull/4481

Custom filter extensions will need to provide an implementation for getBitmapResult() now.

Other Notes

jvm/gc/time metric

The jvm/gc/time metric is no longer emitted, replaced by a new metric named jvm/gc/cpu for the reasons described here: https://github.com/druid-io/druid/pull/4480

Default worker select strategy

Please note that the default worker select strategy has changed from fillCapacity to equalDistribution. This change was introduced in 0.10.1, the previous release, but was not mentioned in the 0.10.1 release notes, so it is called out again here.

V8 segment creation removed

Druid will now always build V9 segments, creating V8 segments is no longer supported and the buildV9Directly property for ingestion tasks has been removed.

Please see https://github.com/druid-io/druid/pull/4420 for more details.

LogLevelAdjuster removed

Please note that the LogLevelAdjuster has been removed: https://github.com/druid-io/druid/pull/4236

Any user using mbeans to configure log levels should configure log4j2 using jmx instead.

Credits

Thanks to everyone who contributed to this release!

@a2l007
@akashdw
@andy256
@asifmansoora
@b-slim
@benvogan
@blugowski
@chrisgavin
@dclim
@dgolitsyn
@drcrallen
@egor-ryashin
@erikdubbelboer
@Fokko
@fuji-151a
@gaodayue
@gianm
@ginoledesma
@himanshug
@hzy001
@jihoonson
@jon-wei
@kevinconaway
@knoguchi
@leiwangx
@leventov
@michalmisiewicz
@niketh
@pjain1
@praveev
@QiuMM
@scan-the-automator
@solimant
@SpotXPeterCunningham
@tkyaw
@wywlds
@xanec
@yuusaku-t
@zhangxinyu1

druid -

Published by jon-wei about 7 years ago

Druid 0.10.1 contains hundreds of performance improvements, stability improvements, and bug fixes from over 40 contributors. Major new features include:

  • Large performance improvements and additional query metrics for TopN queries
  • The ability to push down limit clauses for GroupBy queries
  • More accurate query timeout handling
  • Hadoop indexing support for the Amazon S3A filesystem
  • Support for ingesting Protobuf data
  • A new Firehose that can read input via HTTP
  • Improved disk space management when indexing from cloud stores
  • Various improvements to coordinator lookups management
  • A new Kafka metrics emitter
  • A new dimension comparison filter
  • Various improvements to Druid SQL

If you are upgrading from a previous version of Druid, please see "Updating from 0.10.0 and earlier" below for upgrade notes, including some backwards incompatible changes.

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.10.1

Documentation for this release is at: http://druid.io/docs/0.10.1/

Highlights

TopN performance improvements

Processing for TopN queries with 1-2 aggregators on historical nodes is now 2-4 times faster. This is accomplished with new runtime inspection logic that generates monomorphic implementations of query processing classes, reducing polymorphism in the TopN query execution path.

Added in a series of PRs described here by @leventov: https://github.com/druid-io/druid/issues/3798.

Limit clause push down for GroupBy

Druid can now optimize limit clauses in GroupBy queries by distributing the limit application to historical/realtime nodes, applying the limit to partial result sets before they are sent to the broker for merging. This reduces network traffic within the cluster and reduces the merging workload on broker nodes. Please refer to http://druid.io/docs/0.10.1/querying/groupbyquery.html#query-context for more information.

Added in https://github.com/druid-io/druid/pull/3873 by @jon-wei.

Hadoop indexing support for Amazon S3A

Amazon's S3A filesystem is now supported for deep storage and as an input source for batch ingestion tasks. Please refer to <> for documentation.

Added in https://github.com/druid-io/druid/pull/4116 by @b-slim.

Protobuf 3.0 support and other enhancements

Support for ingesting Protobuf 3.0 data has been added, along with other enhancements such as reading Protobuf descriptors from a URL. Protobuf-supporting code has been moved into its own core extension as well. See http://druid.io/docs/0.10.1/development/extensions-core/protobuf.html for documentation.

Added in https://github.com/druid-io/druid/pull/4039 by @knoguchi.

HTTP Firehose

A new Firehose for realtime ingestion that reads data from a list of URLs via HTTP has been added. Please see http://druid.io/docs/latest/ingestion/firehose.html#httpfirehose for documentation.

Added in https://github.com/druid-io/druid/pull/4297 by @jihoonson.

Improved disk space management for realtime indexing from cloud stores

The Firehose implementations for Microsoft Azure, Rackspace Cloud Files, Google Cloud Storage, and Amazon S3 now support caching and prefetching of data. These firehoses can now operate on portions of the input data and pull new data as needed, instead of having to fully read the firehose's input to disk.

Please refer to the following links for documentation:
http://druid.io/docs/0.10.1/development/extensions-contrib/azure.html
http://druid.io/docs/0.10.1/development/extensions-contrib/cloudfiles.html
http://druid.io/docs/0.10.1/development/extensions-contrib/google.html
http://druid.io/docs/0.10.1/development/extensions-core/s3.html

Added in https://github.com/druid-io/druid/pull/4193 by @jihoonson.

Improvements to coordinator lookups management

Several enhancements have been made to the state management/synchronization logic for query-time lookups, including versioning of lookup specs. Please see http://druid.io/docs/0.10.1/querying/lookups.html for documentation.

Added in https://github.com/druid-io/druid/pull/3855 by @himanshug.

Kafka metrics emitter

A new metrics emitter that sends metrics data to Kafka in JSON format has been added. See http://druid.io/docs/0.10.1/development/extensions-contrib/kafka-emitter.html

Added in https://github.com/druid-io/druid/pull/3860 by @dkhwangbo.

Column comparison filter

A new column comparison filter has been added. This filter allows the user to compare values across columns within a row, like a "WHERE columnA = columnB" clause in SQL. See http://druid.io/docs/0.10.1/querying/filters.html#column-comparison-filter for documentation.

Added in https://github.com/druid-io/druid/pull/3928 by @erikdubbelboer.

Druid SQL improvements

Druid 0.10.1 has a number of enhancements to Druid SQL, such as support for lookups (PRs by @gianm):

https://github.com/druid-io/druid/pull/4368 - More forgiving Avatica server
https://github.com/druid-io/druid/pull/4109 - Support for another form of filtered aggregator
https://github.com/druid-io/druid/pull/4085 - Rule to collapse sort chains
https://github.com/druid-io/druid/pull/4055 - Add SQL REGEXP_EXTRACT function
https://github.com/druid-io/druid/pull/3991 - Make row extractions extensible and add one for lookups
https://github.com/druid-io/druid/pull/4028 - Support for coercing to DECIMAL
https://github.com/druid-io/druid/pull/3999 - Ability to generate exact distinct count queries

Other performance improvements

Druid 0.10.1 has a number of other performance improvements, including:

https://github.com/druid-io/druid/pull/4364 - Uncompress streams without having to download to tmp first, by @niketh
https://github.com/druid-io/druid/pull/4315 - Server selector improvement, by @dgolitsyn
https://github.com/druid-io/druid/pull/4110 - Remove "granularity" from IngestSegmentFirehose, by @gianm
https://github.com/druid-io/druid/pull/4038 - serialize DateTime As Long to improve json serde performance, by @kaijianding

And much more!

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.10.1

Updating from 0.10.0 and earlier

Please see below for changes between 0.10.0 and 0.10.1 that you should be aware of before upgrading. If you're updating from an earlier version than 0.10.0, please see release notes of the relevant intermediate versions for additional notes.

Deprecation of support for Hadoop versions < 2.6.0

To add support for Amazon's S3A filesystem, Druid is now built against Hadoop 2.7.3 libraries, and we are deprecating support for Hadoop versions older than 2.6.0.

For users running a Hadoop version older than 2.6.0, it is possible to continue running Druid 0.10.1 with the older Hadoop version using a workaround.

The user would need to downgrade hadoop.compile.version in the main Druid pom.xml, remove the hadoop-aws dependency from pom.xml in the druid-hdfs-storage core extension, and then rebuild Druid.

Users are strongly encouraged to upgrade their Hadoop clusters to a 2.6.0+ version as of this release, as support for Hadoop <2.6.0 may be dropped completely in future releases.

If users wish to use Hadoop 2.7.3 as default for ingestion tasks, users should double check any existing druid.indexer.task.defaultHadoopCoordinates configurations.

Kafka Broker Changes

Due to changes from https://github.com/druid-io/druid/pull/4115, the Kafka indexing service is no longer compatible with version 0.9.x Kafka brokers. Users will need to upgrade their Kafka brokers to an 0.10.x version.

Coordinator Lookup Management Changes

https://github.com/druid-io/druid/pull/3855 introduces various improvements to coordinator lookup propagation behavior. Please see http://druid.io/docs/0.10.1/querying/lookups.html for details. Note the changes to coordinator HTTP API regarding lookups management.

If Lookups are being used in prior deployment, then as part of upgrade to 0.10.1, All coordinators should be stopped, upgraded, and then started with version 0.10.1 at one time rather than upgrading them one at a time. There should never be a situation where one coordinator is running 0.10.0 while other coordinator is running 0.10.1 at the same time.

During the course of the cluster upgrade, lookup query nodes will report an error starting with got notice to load lookup [LookupExtractorFactoryContainer{version='null'. This is not actually an error and is a side effect of the update. See https://github.com/druid-io/druid/issues/4603 for details.

Off-heap query-time lookup cache

Please note that the off-heap query-time lookup cache is broken at this time because of an excessive memory use issue, and must not be used:
https://github.com/druid-io/druid/issues/3663

Default worker select strategy

Please note that the default worker select strategy has changed from fillCapacity to equalDistribution.

Rolling updates

The standard Druid update process described by http://druid.io/docs/0.10.1/operations/rolling-updates.html should be followed for rolling updates.

Credits

Thanks to everyone who contributed to this release!

@akashdw
@amarjayr
@asdf2014
@asrayousuf
@b-slim
@cesure
@cxmcc
@dclim
@dgolitsyn
@dkhwangbo
@drcrallen
@elloooooo
@erikdubbelboer
@fanjieqi
@Fokko
@freakyzoidberg
@fuji-151a
@gianm
@gkc2104
@himanshug
@hzy001
@JackyWoo
@Jdban
@jeffhartley
@jerchung
@jihoonson
@jon-wei
@kaijianding
@KenjiTakahashi
@knoguchi
@leventov
@licl2014
@logarithm
@niketh
@nishantmonu51
@pjain1
@praveev
@ramiyer
@sascha-coenen
@satishbhor
@sixtus
@sjvs
@skyler-tao
@xanec
@zhihuij
@zwang180

druid -

Published by gianm over 7 years ago

Druid 0.10.0 contains hundreds of performance improvements, stability improvements, and bug fixes from over 40 contributors. Major new features include a built-in SQL layer, numeric dimensions, Kerberos authentication support, a revamp of the "index" task, a new "like" filter, large columns, ability to run the coordinator and overlord as a single service, better performing defaults, and eight new extensions.

If you are upgrading from a previous version of Druid, please see "Updating from 0.9.2 and earlier" below for upgrade notes, including some backwards incompatible changes.

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.10.0

Documentation for this release is at: http://druid.io/docs/0.10.0/

Highlights

Built-in SQL

Druid now includes a built-in SQL server powered by Apache Calcite. Druid provides two SQL APIs: HTTP POST and JDBC. This provides an alternative to Druid's native JSON API which is more familiar to new developers, and which makes it easier to integrate pre-existing applications that natively speak SQL. Not all Druid and SQL features are supported by the SQL layer in this initial release, but we intend to expand both in future releases.

SQL support can be enabled by setting druid.sql.enable=true in your configuration. See http://druid.io/docs/0.10.0/querying/sql.html for details and documentation.

Added in #3682 by @gianm.

Numeric dimensions

Druid now supports numeric dimensions at ingestion and query time. Users can ingest long and float columns as dimensions (i.e., treating the numeric columns as part of the ingestion-time grouping key instead of as aggregators, if rollup is enabled). Additionally, Druid queries can now accept any long or float column as a dimension for grouping or for filtering.

There are performance tradeoffs between string and numeric columns. Numeric columns are generally faster to group on than string columns. Numeric columns don't have indexes, so they are generally slower to filter on than string columns.

See http://druid.io/docs/0.10.0/ingestion/schema-design.html#numeric-dimensions for ingestion documentation and http://druid.io/docs/0.10.0/querying/dimensionspecs.html for query documentation.

Added in #3838, #3966, and other patches by @jon-wei.

Kerberos authentication support

Added a new extension named 'druid-kerberos' which adds support for User Authentication for Druid Nodes using Kerberos. It uses the simple and protected GSSAPI negotiation mechanism, SPNEGO(https://en.wikipedia.org/wiki/SPNEGO) for authentication via HTTP.

See http://druid.io/docs/0.10.0/development/extensions-core/druid-kerberos.html for documentation on how to configure kerberos authentication.

Added in #3853 by @nishantmonu51.

Index task revamp

The indexing task was re-written to improve performance, particularly for jobs spanning multiple intervals that generated many shards. The segmentGranularity intervals can now be automatically determined and no longer needs to be specified, but ingestion time can be reduced if both intervals and numShards are provided.

Additionally, the indexing task now supports an appendToExisting flag which causes the data to be indexed as an additional shard of the current version rather than as a new version overshadowing the previous version.

See http://druid.io/docs/0.10.0/ingestion/tasks.html#index-task for documentation.

Added in #3611 by @dclim.

Like filter

Druid now includes a "like" filter that enables SQL LIKE-style filtering, such as foo LIKE 'bar%'. The implementation is generally faster than regex filters, and is encouraged over regex filters when possible. In particular, like filters on prefixes such as bar% are significantly faster than equivalent regex filters such as ^bar.*.

See http://druid.io/docs/0.10.0/querying/filters.html#like-filter for documentation.

Added in #3642 by @gianm.

Large columns

Druid now supports individual columns larger than 2GB. This feature is not typically required, since general guidance is that segments should generally be 500MB–1GB in size, but is useful in situations where one column is much larger than all the others (for example, large sketches).

This functionality is available to all Druid users and no special configuration is necessary when using the built-in column types. If you have developed a custom metric column type as a Druid extension, you can enable large column support by overriding getSerializer in your ComplexMetricsSerde.

Added in #3743 by @akashdw.

Coordinator/Overlord combination option

Druid deployments can now be simplified by combining the Coordinator and Overlord functions into the Coordinator process. To do this, set druid.coordinator.asOverlord.enabled and druid.coordinator.asOverlord.overlordService appropriately on your Coordinators and then stop your Overlords.
Overlord console would be available on http://coordinator-host:port/console.html.

This is currently an experimental feature and is off by default. We intend to consider making this the default in a future version of Druid.

See http://druid.io/docs/0.10.0/configuration/coordinator.html for documentation on this feature and configuration options.

Added in #3711 by @himanshug.

Better performing defaults

This release changes two default settings to improve out-of-the-box performance:

  • The buildV9Directly option introduced in Druid 0.9.0 is now enabled by default. This option improves performance of indexing by creating the v9 data format directly rather than creating v8 first and then converting to v9. If necessary, you can roll back to the old code by setting "buildV9Directly" to false in your indexing tasks.

  • The v2 groupBy engine introduced in Druid 0.9.2 is now enabled by default. This new groupBy engine was rewritten from the ground up for better performance and memory management. If necessary, you can roll back to the old engine by setting either "druid.groupBy.query.defaultStrategy" in your runtime.properties, or "groupByStrategy" in your query context, to "v1". See http://druid.io/docs/0.10.0/querying/groupbyquery.html for details on the differences between groupBy v1 and v2.

Other performance improvements

In addition to better performing defaults, Druid 0.10.0 has a number of other performance improvements, including:

  • Concise bitset union, intersection, and iteration optimization (#3883) by @leventov
  • DimensionSelector-based value matching optimization (#3858) by @leventov
  • Search query strategy for choosing index-based vs. cursor-based execution (#3792) by @jihoonson
  • Bitset iteration optimization (#3753) by @leventov
  • GroupBy optimization for granularity "all" (#3740) by @gianm
  • Disable flush after every DefaultObjectMapper write (#3748) by @jon-wei
  • Short-circuiting AND filter (#3676) by @gianm
  • Improved performance of IndexMergerV9 (#3440) by @leventov

New extensions

  • ambari-metrics-emitter (#3767) by @nishantmonu51
  • druid-kerberos (#3853) by @nishantmonu51
  • google-extensions (#2458) by @erikdubbelboer
  • scan-query (#3307) by @kaijianding
  • sqlserver-metadata-storage (#3421) by @mark1900
  • thrift-extensions (#3418) by @du00cs
  • time-min-max (#3299) by @sirpkt
  • virtual-columns (#2511) by @navis

And much more!

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.10.0

Updating from 0.9.2 and earlier

Please see below for changes between 0.9.2 and 0.10.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.9.2, please see release notes of the relevant intermediate versions for additional notes.

Rolling updates

The standard Druid update process described by http://druid.io/docs/0.10.0/operations/rolling-updates.html should be followed for rolling updates.

Query API changes

Please note the following backwards-incompatible query API changes when updating. Some queries may need to be adjusted to continue to behave as expected.

  • JavaScript query features are now disabled by default for security reasons (#3818). If you use these features, you can re-enable them by setting druid.javascript.enabled=true in your runtime properties. See http://druid.io/docs/0.10.0/development/javascript.html for details, including security considerations.

  • GroupBy queries no longer allow __time as the output name of a dimension, aggregator, or post-aggregator (#3967).

  • Select query pagingSpecs now default to fromNext: true behavior when fromNext is not specified (#3986). Behavior is unchanged for Select queries that did have fromNext specified. If you prefer the old default, then you can change this through the druid.query.select.enableFromNextDefault runtime property. See http://druid.io/docs/0.10.0/querying/select-query.html for details.

  • SegmentMetadata queries no longer include "size" analysis by default (#3773). You can still request "size" analysis by adding "size" to "analysisTypes" at query time.

Deployment and configuration changes

Please note the following deployment-related changes when updating.

  • Druid now requires Java 8 to run (#3914). If you are currently running on Java 7, we suggest upgrading Java first and then Druid.

  • Druid now defaults to the "v2" engine for groupBy rather than the legacy "v1" engine. As part of this, memory usage limits have changed from row-based to byte-based limits, so it is possible that some queries which met resource limits before will now exceed them and fail. You can avoid this by tuning the new groupBy engine appropriately. If necessary, you can roll back to the old engine by setting either "druid.groupBy.query.defaultStrategy" in your runtime.properties, or "groupByStrategy" in your query context, to "v1". See http://druid.io/docs/0.10.0/querying/groupbyquery.html for more details on the differences between groupBy v1 and v2.This new groupBy engine was rewritten from the ground up for better performance and memory management. The query API and results are compatible between the two engines; however, there are some differences from a cluster configuration perspective: (1) groupBy v2 uses primarily off-heap memory instead of on-heap memory; (2) it requires one merge buffer for each concurrent query; (3) if you've configured caching for groupBy, it must be on historicals; and (4) it doesn't support chunkPeriod.

  • Druid now has a non-zero default value for druid.processing.numMergeBuffers (#3953). This will increase the amount of direct memory used in its default configuration. This change was made in connection with changing the default groupBy engine, since the new "v2" engine requires merge buffers. If you were formerly using groupBy v1, you should be able to offset this by reducing your JVM heap, since groupBy v2 uses off-heap memory rather than on-heap memory.

  • Druid query-related processes (broker, historical, indexing tasks) now eagerly allocate druid.processing.numThreads number of DirectByteBuffer instances of size druid.processing.buffer.sizeBytes for query processing at startup (#3628). This allocation was lazy in earlier versions of Druid. This change does not affect memory use of a long-running Druid cluster, but it means that memory for processing buffers must be available when Druid starts up.

  • When running in non-UTC timezones, behavior of predefined segmentGranularity constants such as "day" will change when upgrading. Formerly, this would generate segments using local days. In Druid 0.10.0, this will generate segments using UTC days. If you were running Druid in the UTC timezone, this change has no effect. Please note that running Druid in a non-UTC timezone is not officially supported (see http://druid.io/docs/0.10.0/configuration/index.html) and we recommend always running all processes in UTC timezones. To create local-day segments using Druid 0.10.0, you can use a local-time segmentGranularity such as:

"segmentGranularity" : {
     "type" : "period",
     "period" : "P1D",
     "timeZone" : "America/Los_Angeles"
}

Extension changes

  • AggregatorFactory, BufferAggregator, and PostAggregator interfaces have changed (#3894, #3899, #3957, #4071). If you have deployed a custom extension that includes an AggregatorFactory or PostAggregator, it will need to be recompiled. Druid's built-in extensions have all been updated for this change.

  • Extensions targeting Druid 0.10.x must be compiled with JDK 8.

Credits

Thanks to everyone who contributed to this release!

@akashdw
@Aveplatter
@b-slim
@baruchz
@cheddar
@clintropolis
@DaimonPl
@dclim
@dpenas
@drcrallen
@du00cs
@erikdubbelboer
@Fokko
@freakyzoidberg
@gabrielpage
@gianm
@gvsmirnov
@himanshug
@hland
@hzy001
@jaehc
@jihoonson
@jisookim0513
@jkukul
@joanvr
@jon-wei
@kaijianding
@leventov
@mark1900
@michaelschiff
@navis
@ncolomer
@niketh
@nishantmonu51
@pjain1
@praveev
@sirpkt
@tranv94
@xiaoyao1991
@yuusaku-t
@zhxiaogg

druid -

Published by gianm almost 8 years ago

Druid 0.9.2 contains hundreds of performance improvements, stability improvements, and bug fixes from over 30 contributors. Major new features include a new groupBy engine, ability to disable rollup at ingestion time, ability to filter on longs, new encoding options for long-typed columns, performance improvements for HyperUnique and DataSketches, a query cache implementation based on Caffeine, a new lookup extension exposing fine grained caching strategies, support for reading ORC files, and new aggregators for variance and standard deviation.

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.9.2

Documentation for this release is here: http://druid.io/docs/0.9.2/

Highlights

New groupBy engine

Druid now includes a new groupBy engine, rewritten from the ground up for better performance and memory management. Benchmarks show a 2–5x performance boost on our test datasets. The new engine also supports strict limits on memory usage and the option to spill to disk when memory is exhausted, avoiding result row count limitations and potential OOMEs generated by the previous engine.

The new engine is off by default, but you can enable it through configuration or query context parameters. We intend to enable it by default in a future version of Druid.

See "implementation details" on http://druid.io/docs/0.9.2/querying/groupbyquery.html#implementation-details for documentation and configuration.

Added in #2998 by @gianm.

Ability to disable rollup

Since its inception, Druid has had a concept of "dimensions" and "metrics" that applied both at ingestion time and at query time. Druid is unique in that it is one of the only databases that supports aggregation at data loading time, which we call "rollup". But, for some use cases, ingestion-time rollup is not desired, and it's better to load the original data as-is. With rollup disabled, one row in Druid will be created for each input row.

Query-time aggregation is, of course, still supported through the groupBy, topN, and timeseries queries.

See the "rollup" flag on http://druid.io/docs/0.9.2/ingestion/index.html for documentation. By default, rollup remains enabled.

Added in #3020 by @kaijianding.

Ability to filter on longs

Druid now supports sophisticated filtering on integer-typed columns, including long metrics and the special __time column. This opens up a number of new capabilities:

Druid does not yet support grouping on longs. We intend to add this capability in a future release.

Added in #3180 by @jon-wei.

New long encodings

Until now, all integer-typed columns in Druid, including long metrics and the special __time column, were stored as 64-bit longs optionally compressed in blocks with LZ4. Druid 0.9.2 adds new encoding options which, in many cases, can reduce file sizes and improve performance:

  • Long encoding option "auto", which potentially uses table or delta encoding to use fewer than 64 bits per row. The "longs" encoding option is the default behavior, which always uses 64 bits.
  • Compression option "none", which is like the old "uncompressed" option, except it offers a speedup by bypassing block copying.

The default remains "longs" encoding + "lz4" compression. In our testing, two options that often yield useful benefits are "auto" + "lz4" (generally smaller than longs + lz4) and "auto" + "none" (generally faster than longs + lz4, file size impact varies). See the PR for full test results.

See "metricCompression" and "longEncoding" on http://druid.io/docs/0.9.2/ingestion/batch-ingestion.html for documentation.

Added in #3148 by @acslk.

Sketch performance improvements

  • DataSketches speedups of up to 80% from #3471.
  • HyperUnique speedups of 19–30% from #3314, used for "hyperUnique" and "cardinality" aggregators.

New extensions

And much more!

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.9.2

Updating from 0.9.1.1

Rolling updates

The standard Druid update process described by http://druid.io/docs/0.9.2/operations/rolling-updates.html should be followed for rolling updates.

Query time lookups

The druid-namespace-lookup extension, which was deprecated in 0.9.1 in favor of druid-lookups-cached-global, has been removed in 0.9.2. If you are using druid-namespace-lookup, migrate to druid-lookups-cached-global before upgrading to 0.9.2. See our migration guide for details: http://druid.io/docs/0.9.1.1/development/extensions-core/namespaced-lookup.html#transitioning-to-lookups-cached-global

Other notes

Please note the following changes:

  • Druid now ships Guice 4.1.0 rather than 4.0-beta (#3222). This conflicts with the version shipped in some Hadoop distributions, so for Hadoop indexing you may need to adjust your mapreduce.job.classloader or mapreduce.job.user.classpath.first options. In testing we have found this to be an effective workaround. See http://druid.io/docs/0.9.2/operations/other-hadoop.html for details.
  • If you are using Roaring bitmaps, note that compressRunOnSerialization now defaults to true. As a result, segments written will not be readable by Druid 0.8.1 or earlier. If you need segments written by Druid 0.9.2 to be readable by 0.8.1, and you are using Roaring bitmaps, you must set compressRunOnSerialization = false. By default, bitmaps are Concise, not Roaring, so this point will not apply to you unless you overrode that. See #3228 for details.
  • If you use the new long encoding or compression options, segments written by Druid will not be readable by any version older than 0.9.2. If you don't use the new options, segments will remain backwards compatible.
  • If you are using the experimental Kafka indexing service, there is a known issue that may cause task supervision to hang when it tries to stop all running tasks simultaneously during the upgrade process. To prevent this from happening, you can shutdown all supervisors and wait for the indexing tasks to complete before updating your overlord. Alternatively, you can set chatThreads in the supervisor tuning configuration to a value greater than the number of running tasks as a workaround.

Credits

Thanks to everyone who contributed to this release!

@acslk
@AlexanderSaydakov
@ashishawasthi
@b-slim
@chtefi
@dclim
@drcrallen
@du00cs
@ecesena
@erikdubbelboer
@fjy
@Fokko
@gianm
@giaosudau
@guobingkun
@gvsmirnov
@hamlet-lee
@himanshug
@HyukjinKwon
@jaehc
@jianran
@jon-wei
@kaijianding
@leventov
@linbojin
@michaelschiff
@navis
@nishantmonu51
@pjain1
@rajk-tetration
@SainathB
@sirpkt
@vogievetsky
@xvrl
@yuppie-flu

Package Rankings
Top 1.41% on Repo1.maven.org
Badges
Extracted from project README
Coverage Status Docker Helm codeql-config codeql cron-job-its labeler reusable-revised-its reusable-standard-its reusable-unit-tests revised-its standard-its static-checks unit-and-integration-tests-unified unit-tests Website Twitter Download Get Started Documentation Community Build Contribute License data loader Kafka management query view combo
Related Projects
maven-doxia

Apache Maven Doxia base

21 May 2009 28
maven-doxia-sitetools

Apache Maven Doxia Sitetools

21 May 2009 8
allura

Apache Allura - a software forge to manage source code repositories, bug reports, discussions, wi...

05 Apr 2014 128
maven-doxia-converter

Apache Maven Doxia Converter

04 Nov 2017 2
maven-doxia-site

Apache Maven Doxia site

04 Jan 2018 2
maven-doxia-linkcheck

Apache Maven Doxia LinkCheck

04 Nov 2017 2
maven-doxia-book-renderer

Apache Maven Doxia Book Renderer

23 Jan 2020 1