Apache Druid: a high performance real-time analytics database.
APACHE-2.0 License
Bot releases are visible (Hide)
Published by jihoonson over 4 years ago
Apache Druid 0.18.0 contains over 200 new features, performance enhancements, bug fixes, and major documentation improvements from 42 contributors. Check out the complete list of changes and everything tagged to the milestone.
Join is a key operation in data analytics. Prior to 0.18.0, Druid supported some join-related features, such as Lookups or semi-joins in SQL. However, the use cases for those features were pretty limited and, for other join use cases, users had to denormalize their datasources when they ingest data instead of joining them at query time, which could result in exploding data volume and long ingestion time.
Druid 0.18.0 supports real joins for the first time ever in its history. Druid supports INNER, LEFT, and CROSS joins for now. For native queries, the join
datasource has been newly introduced to represent a join of two datasources. Currently, only the left-deep join is allowed. That means, only a table
or another join
datasource is allowed for the left datasource. For the right datasource, lookup
, inline
, or query
datasources are allowed. Note that join of Druid datasources is not supported yet. There should be only one table
datasource in the same join query.
Druid SQL also supports joins. Under the covers, SQL join queries are translated into one or several native queries that include join datasources. See Query translation for more details of SQL translation and best practices to write efficient queries.
When a join query is issued, the Broker first evaluates all datasources except for the base datasource which is the only table
datasource in the query. The evaluation can include executing subqueries for query
datasources. Once the Broker evaluates all non-base datasources, it replaces them with inline
datasources and sends the rewritten query to data nodes (see the below "Query inlining in Brokers" section for more details). Data nodes use the hash join to process join queries. They build a hash table for each non-primary leaf datasource unless it already exists. Note that only lookup
datasource currently has a pre-built hash table. See Query execution for more details about join query execution.
Joins can affect performance of your queries. In general, any queries including joins can be slower than equivalent queries against a denormalized datasource. The LOOKUP
function could perform better than joins with lookup datasources. See Join performance for more details about join query performance and future plans for performance improvement.
https://github.com/apache/druid/pull/8728
https://github.com/apache/druid/pull/9545
https://github.com/apache/druid/pull/9111
Druid is now able to execute a nested query by inlining subqueries. Any type of subquery can be on top of any type of another, such as in the following example:
topN
|
(join datasource)
/ \
(table datasource) groupBy
To execute this query, the Broker first evaluates the leaf groupBy subquery; it sends the subquery to data nodes and collects the result. The collected result is materialized in the Broker memory. Once the Broker collects all results for the groupBy query, it rewrites the topN query by replacing the leaf groupBy with an inline datasource which has the result of the groupBy query. Finally, the rewritten query is sent to data nodes to execute the topN query.
When you run multiple queries of heterogenous workloads at a time, you may sometimes want to control the resource commitment for a query based on its priority. For example, you would want to limit the resources assigned to less important queries, so that important queries can be executed in time without being disrupted by less important ones.
Query laning allows you to control capacity utilization for heterogeneous query workloads. With laning, the broker examines and classifies a query for the purpose of assigning it to a 'lane'. Lanes have capacity limits, enforced by the Broker, that can be used to ensure sufficient resources are available for other lanes or for interactive queries (with no lane), or to limit overall throughput for queries within the lane.
Automatic query prioritization determines the query priority based on the configured strategy. The threshold-based prioritization strategy has been added; it automatically lowers the priority of queries that cross any of a configurable set of thresholds, such as how far in the past the data is, how large of an interval a query covers, or the number of segments taking part in a query.
See Query prioritization and laning for more details.
https://github.com/apache/druid/issues/6993
https://github.com/apache/druid/pull/9407
https://github.com/apache/druid/pull/9493
Since a native query containing subqueries can be executed part-by-part, a new subQueryId
has been introduced. Each subquery has different subQueryId
s but same queryId
. The subQueryId
is available as a new dimension in query metrics.
A new druid.server.http.maxSubqueryRows
configuration controls the maximum number of rows materialized in the Broker memory.
Please see Query execution for more details.
https://github.com/apache/druid/pull/9533
GROUPING SETS is now supported, allowing you to combine multiple GROUP BY clauses into one GROUP BY clause. This GROUPING SETS clause is internally translated into the groupBy query with subtotalsSpec
. The LIMIT clause is now applied after subtotalsSpec, rather than applied to each grouping set.
https://github.com/apache/druid/pull/9122
Druid now supports dynamic parameters for SQL. To use dynamic parameters, replace any literal in the query with a question mark (?
) character. These question marks represent the places where the parameters will be bound at execution time. See SQL dynamic parameters for more details.
https://github.com/apache/druid/pull/6974
applyLimitPushDownToSegments
is disabled by defaultapplyLimitPushDownToSegments
was added in 0.17.0 to push down limit evaluation to queryable nodes, limiting results during segment scan for groupBy v2. This can lead to performance degradation, as reported in https://github.com/apache/druid/issues/9689, if many segments are involved in query processing. This is because “limit push down to segment scan” initializes an aggregation buffer per segment, the overhead for which is not negligible. Enable this configuration only if your query involves a relatively small number of segments per historical or realtime task.
https://github.com/apache/druid/pull/9711
Druid supports two bitmap types, i.e., Roaring and CONCISE. Since Roaring bitmaps provide a better out-of-box experience (faster query speed in general), the default bitmap type is now switched to Roaring bitmaps. See Segment compression for more details about bitmaps.
https://github.com/apache/druid/pull/9548
When SQL-compatible null handling is disabled, the behavior of complex metric aggregation at ingestion time has now changed to be consistent with that at query time. The complex metrics are aggregated to the default 0 values for nulls instead of skipping them during ingestion.
https://github.com/apache/druid/pull/9484
Druid expression now supports typed constructors for creating arrays. Arrays can be defined with an explicit type. For example, <LONG>[1, 2, null]
creates an array of LONG
type containing 1
, 2
, and null
. Note that you can still create an array without an explicit type. For example, [1, 2, null]
is still a valid syntax to create an equivalent array. In this case, Druid will infer the type of array from its elements. This new syntax applies to empty arrays as well. <STRING>[]
, <DOUBLE>[]
, and <LONG>[]
will create an empty array of STRING
, DOUBLE
, and LONG
type, respectively.
https://github.com/apache/druid/pull/9367
The pendingSegments
table in the metadata store is used to create unique new segment IDs for appending tasks such as Kafka/Kinesis indexing tasks or batch tasks of appending mode. Automatic pending segments cleanup was introduced in 0.12.0, but has been disabled by default prior to 0.18.0. This configuration is now enabled by default.
https://github.com/apache/druid/pull/9385
The Parallel task now can create better splits. Each split can contain multiple input files based on their size. Empty files will be ignored. The split size is controllable with the new split hint spec. See Split hint spec for more details.
https://github.com/apache/druid/pull/9360
https://github.com/apache/druid/pull/9450
Transform
is an Interface
that represents a transformation to be applied to each row at ingestion time. This interface is now an Extension point. Please see Writing your own extensions for how to add your custom Transform
.
https://github.com/apache/druid/pull/9319
chunkPeriod
query context is removedchunkPeriod
has been deprecated since 0.14.0 because of its limited usage (it was sometimes useful for only groupBy v1). This query context is now removed in 0.18.0.
https://github.com/apache/druid/pull/9216
Druid now experimentally supports Java 11. You can run the same Druid binary distribution with Java 11 which is compiled with Java 8. Our tests on Travis include:
Performance testing results are not available yet.
Since Java 9, it issues a warning when it is found that some libraries use reflection to illegally access internal APIs of the JDK. These warnings will be fixed by modifying Druid codes or upgrading library versions in future releases. For now, these warnings can be suppressed by adding JVM options such as --add-opens
or --add-exports
. See JDK 11 Migration Guide for more details.
Some of the warnings are:
2020-01-22T21:30:08,893 WARN [main] org.apache.druid.java.util.metrics.AllocationMetricCollectors - Cannot initialize org.apache.druid.java.util.metrics.AllocationMetricCollector
java.lang.reflect.InaccessibleObjectException: Unable to make public long[] com.sun.management.internal.HotSpotThreadImpl.getThreadAllocatedBytes(long[]) accessible: module jdk.management does not "exports com.sun.management.internal" to unnamed module @6955cb39
This warning can be suppressed by adding --add-exports jdk.management/com.sun.management.internal=ALL-UNNAMED
.
2020-01-22T21:30:08,902 WARN [main] org.apache.druid.java.util.metrics.JvmMonitor - Cannot initialize GC counters. If running JDK11 and above, add --add-exports java.base/jdk.internal.perf=ALL-UNNAMED to the JVM arguments to enable GC counters.
This warning can be suppressed by adding --add-exports java.base/jdk.internal.perf=ALL-UNNAMED
.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
This warning can be suppressed by adding --add-opens java.base/java.lang=ALL-UNNAMED
.
https://github.com/apache/druid/pull/7306
https://github.com/apache/druid/pull/9491
A new extension is added in 0.18.0 to enable OpenID Connect based Authentication for Druid Processes. This can be used with any authentication server that supports same e.g. Okta. This extension should only be used at the router node to enable a group of users in existing authentication server to interact with Druid cluster, using the Web Console.
https://github.com/apache/druid/pull/8992
CVE-2020-1958 has been reported recently and fixed in 0.18.0 and 0.17.1. When LDAP authentication is enabled, callers of Druid APIs can bypass the credentialsValidator.userSearch filter barrier or retrieve any LDAP attribute values of users that exist on the LDAP server, so long as that information is visible to the Druid server. Please see the description in the link for more details. It is strongly recommended to upgrade to 0.18.0 or 0.17.1 if you are using LDAP authentication with Druid.
https://github.com/apache/druid/pull/9600
Kafka client library has been updated to 2.2.2, in which CVE-2019-12399 is fixed.
https://github.com/apache/druid/pull/9259
Druid 0.18.0 includes 40 bug fixes. Please see https://github.com/apache/druid/pulls?page=1&q=is%3Apr+milestone%3A0.18.0+is%3Aclosed+label%3ABug for the full list of bug fixes.
Be aware of the following changes between 0.17.1 and 0.18.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.17.1, please see the release notes of the relevant intermediate versions.
The S3 storage extension now supports cleanup of stale task logs and segments. When deploying 0.18.0, please ensure that your extensions
directory does not have any older versions of druid-s3-extensions
extension.
https://github.com/apache/druid/pull/9459
The Azure storage extension has been promoted to a core extension. It also supports cleanup of stale task logs and segments now. When deploying 0.18.0, please ensure that your extensions-contrib
directory does not have any older versions of druid-azure-extensions
extension.
https://github.com/apache/druid/pull/9394
https://github.com/apache/druid/pull/9523
The Google storage extension now supports cleanup of stale task logs and segments. When deploying 0.18.0, please ensure that your extensions
directory does not have any older versions of druid-google-extensions
extension.
https://github.com/apache/druid/pull/9519
Hadoop AWS library is now included in the binary distribution for better out-of-box experience. When deploying 0.18.0, please ensure that your hadoop-dependencies
directory or any other directories in the classpath does not have duplicate libraries.
PostgreSQL JDBC driver for Lookups is now included in the binary distribution for better out-of-box experience. When deploying 0.18.0, please ensure that your extensions/druid-lookups-cached-single
directory or any other directories in the classpath does not have duplicate JDBC drivers.
https://github.com/apache/druid/pull/9399
topN
or groupBy
on scan
with multi-valued columnsQuery inlining in Brokers is newly introduced in 0.18.0 but has a bug that queries with topN
or groupBy
on top of scan
fail if the scan query selects multi-valued dimensions. See https://github.com/apache/druid/issues/9697 for more details.
Avro parser doesn't work with Kafka indexing service because of a wrong null check. See https://github.com/apache/druid/issues/9728 for more details.
segment/unavailable/count
metric during handoffThis metric is supposed to take the number of segments served by realtime tasks into consideration as well, but it isn't now. As a result, it appears that unavailability spikes up before the new segments are loaded by historicals, even if all segments actually are continuously available on some combination of realtime tasks and historicals.
https://github.com/apache/druid/issues/9677
explain plan for
query and the actual execution planThe result of explain plan for
can be slightly different from what Druid actually executes when the query includes joins or subqueries. The difference can be found in that each part of the query plan would be represented as if it was its own native query in the result of explain plan for
. For example, for a join of a datasource d1
and a groupBy subquery on datasource d2
, the explain plan for
could return a plan like below
join
/ \
scan groupBy
| |
d1 d2
whereas the actual query plan Druid would execute is
join
/ \
d1 groupBy
|
d2
For a full list of open issues, please see https://github.com/apache/druid/labels/Bug.
Thanks to everyone who contributed to this release!
@a2l007
@abhishekrb19
@aditya-r-m
@AlexanderSaydakov
@als-sdin
@aP0StAl
@asdf2014
@benhopp
@bjozet
@capistrant
@Caroline1000
@ccaominh
@clintropolis
@dampcake
@fjy
@Fokko
@frnidito
@gianm
@himanshug
@JaeGeunBang
@jihoonson
@jon-wei
@JulianJaffePinterest
@kou64yama
@lamber-ken
@leventov
@liutang123
@maytasm
@mcbrewster
@mgill25
@mitchlloyd
@mrsrinivas
@nvolungis
@prabcs
@samarthjain
@sthetland
@suneet-s
@themaric
@vogievetsky
@xvrl
@zachjsh
@zhenxiao
Published by jon-wei over 4 years ago
Apache Druid 0.17.1 is a security bug fix release that addresses the following CVE for LDAP authentication:
Published by jon-wei over 4 years ago
Apache Druid 0.17.0 contains over 250 new features, performance enhancements, bug fixes, and major documentation improvements from 52 contributors. Check out the complete list of changes and everything tagged to the milestone.
Druid 0.17.0 includes a significant update to the native batch ingestion system. This update adds the internal framework to support non-text binary formats, with initial support for ORC and Parquet. Additionally, native batch tasks can now read data from HDFS.
This rework changes how the ingestion source and data format are specified in the ingestion task. To use the new features, please refer to the documentation on InputSources and InputFormats.
Please see the following documentation for details:
https://druid.apache.org/docs/0.17.0/ingestion/data-formats.html#input-format
https://druid.apache.org/docs/0.17.0/ingestion/native-batch.html#input-sources
https://druid.apache.org/docs/0.17.0/ingestion/native-batch.html#partitionsspec
https://github.com/apache/druid/issues/8812
The parallel index task now supports the single_dim
type partitions spec, which allows for range-based partitioning on a single dimension.
Please see https://druid.apache.org/docs/0.17.0/ingestion/native-batch.html for details.
The parallel indexing task now has a new configuration, splitHintSpec
, in the tuningConfig
to allow for operators to provide hints to control the amount of data that each first phase subtask reads. There is currently one split hint spec type, SegmentsSplitHintSpec
, used for re-ingesting Druid segments.
Auto-compaction can now use the parallel indexing task, allowing for greater compaction throughput.
To control the level of parallelism, the auto-compactiontuningConfig
has new parameters, maxNumConcurrentSubTasks
and splitHintSpec
.
Please see https://druid.apache.org/docs/0.17.0/configuration/index.html#compaction-dynamic-configuration for details.
https://github.com/apache/incubator-druid/pull/8570
Auto-compaction now uses the partitionSpec to track changes made by previous compaction tasks, allowing the coordinator to reduce redundant compaction operations.
Please see https://github.com/apache/druid/issues/8489 for details.
If you have auto-compaction enabled, please see the information under "Stateful auto-compaction changes" in the "Upgrading to Druid 0.17.0" section before upgrading.
The Druid broker can now opportunistically merge query results in parallel using multiple threads.
Please see druid.processing.merge.useParallelMergePool
in the Broker section of the configuration reference for details on how to configure this new feature.
Parallel merging is enabled by default (controlled by the druid.processing.merge.useParallelMergePool
property), and most users should not have to change any of the advanced configuration properties described in the configuration reference.
Additionally, merge parallelism can be controlled on a per-query basis using the query context. Information about the new query context parameters can be found at https://druid.apache.org/docs/0.17.0/querying/query-context.html.
https://github.com/apache/incubator-druid/pull/8578
In 0.17.0, we have added official documentation for Druid's SQL-compatible null handling mode.
Please see https://druid.apache.org/docs/0.17.0/configuration/index.html#sql-compatible-null-handling and https://druid.apache.org/docs/0.17.0/design/segments.html#sql-compatible-null-handling for details.
Several bugs that existed in this previously undocumented mode have been fixed, particularly around null handling in numeric columns. We recommend that users begin to consider transitioning their clusters to this new mode after upgrading to 0.17.0.
The full list of null handling bugs fixed in 0.17.0 can be found at https://github.com/apache/druid/issues?utf8=%E2%9C%93&q=label%3A%22Area+-+Null+Handling%22+milestone%3A0.17.0+
Druid now supports LDAP authentication. Authorization using LDAP groups is also supported by mapping LDAP groups to Druid roles.
LDAP integration requires the druid-basic-security
core extension. Please see https://druid.apache.org/docs/0.17.0/development/extensions-core/druid-basic-security.html for details.
As this is the first release with LDAP support, and there are a large variety of LDAP ecosystems, some LDAP use cases and features may not be supported yet. Please file an issue if you need enhancements to this new functionality.
https://github.com/apache/incubator-druid/pull/6972
A new Dropwizard metrics emitter has been added as a contrib extension.
The currently supported Dropwizard metrics types are counter, gauge, meter, timer and histogram. These metrics can be emitted using either a Console or JMX reporter.
Please see https://druid.apache.org/docs/0.17.0/design/extensions-contrib/dropwizard.html for details.
https://github.com/apache/incubator-druid/pull/7363
A new pair of endpoints have been added to all Druid services that return information about whether the Druid service has received a confirmation that the service has been added to the cluster, from the central service discovery mechanism (currently ZooKeeper). These endpoints can be useful as health/ready checks.
The new endpoints are:
/status/selfDiscovered/status
/status/selfDiscovered
Please see the Druid API reference for details.
https://github.com/apache/incubator-druid/pull/6702
https://github.com/apache/incubator-druid/pull/9005
Task supervisors (e.g. Kafka or Kinesis supervisors) are now recorded in the system tables in a new sys.supervisors
table.
Please see https://druid.apache.org/docs/0.17.0/querying/sql.html#supervisors-table for details.
https://github.com/apache/incubator-druid/pull/8547
A new boolean configuration property for historicals, druid.segmentCache.lazyLoadOnStart
, has been added.
This new property allows historicals to defer loading of a segment until the first time that segment is queried, which can significantly decrease historical startup times for clusters with a large number of segments.
Please see the configuration reference for details.
https://github.com/apache/incubator-druid/pull/6988
A new historical property, druid.segmentCache.locationSelectorStrategy
, has been added.
If there are multiple segment storage locations specified in druid.segmentCache.locations
, the new locationSelectorStrategy
property allows the user to specify what strategy is used to fill the locations. Currently supported options are roundRobin
and leastBytesUsed
.
Please see the configuration reference for details.
https://github.com/apache/incubator-druid/pull/8038
A new Broker endpoint has been added: /druid/broker/v1/readiness
.
A new Historical endpoint has been added: /druid/historical/v1/readiness
.
These endpoints are similar to the existing /druid/broker/v1/loadstatus
and /druid/historical/v1/loadstatus
endpoints.
They differ in that they do not require authentication/authorization checks, and instead of a JSON body they only return a 200 success or 503 HTTP response code.
https://github.com/apache/incubator-druid/pull/8841
It is now possible to define a "category" name property for each MiddleManager. New worker select strategies that are category-aware have been added, allowing the user to control how tasks are assigned to MiddleManagers based on the configured categories.
Please see the documentation for druid.worker.category
in the configuration reference, and the following links, for more details:
https://druid.apache.org/docs/0.17.0/configuration/index.htmlEqual-Distribution-With-Category-Spec
https://druid.apache.org/docs/0.17.0/configuration/index.html#Fill-Capacity-With-Category-Spec
https://druid.apache.org/docs/0.17.0/configuration/index.html#WorkerCategorySpec
https://github.com/apache/druid/pull/7066
A large number of dependencies have been updated to newer versions to address security vulnerabilities.
Please see the PRs below for details:
The deprecated Select native query type has been removed in 0.17.0.
If you have native queries that use Select, you need to modify them to use Scan instead. See the Scan query documentation (https://druid.apache.org/docs/0.17.0/querying/scan-query.html) for syntax and output format details.
For Druid SQL queries that use Select, no changes are needed; the SQL planner already uses the Scan query type under the covers for Select queries.
https://github.com/apache/incubator-druid/pull/8739
The legacy coordinator and overlord consoles have been removed, and they have been replaced with the new web console on the coordinator and overlord.
https://github.com/apache/incubator-druid/pull/8838
Druid 0.17.0 updates Calcite to version 1.21. This newer version of Calcite can make additional optimizations that assume SQL-compliant null handling behavior when planning queries.
If you use Druid SQL and rely on null handling behavior, please read the information at https://druid.apache.org/docs/0.17.0/configuration/index.html#sql-compatible-null-handling and ensure that your Druid cluster is running in the SQL-compliant null handling mode before upgrading.
https://github.com/apache/incubator-druid/pull/8566
Druid 0.17.0 has tidied up its lifecycle, querying, and ingestion logging.
Please see https://github.com/apache/incubator-druid/pull/8889 for a detailed list of changes. If you relied on specific log messages for external integrations, please review the new logging changes before upgrading.
The full set of log messages can still be seen when logging is set to DEBUG level. Template log4j2 configuration files that show how to enable per-package DEBUG logging are provided in the _common
configuration folder in the example clusters under conf/druid
.
The auto-compaction scheduling logic in 0.17.0 tracks additional segment partitioning information in Druid's metadata store that is not present in older versions. This information is used to determine whether a set of segments has already been compacted under the cluster's current auto-compaction configurations.
When this new metadata is not present, a set of segments will always be scheduled for an initial compaction and this new metadata will be created after they are compacted, allowing the scheduler to skip them later if auto-compaction config is unchanged.
Since this additional segment partitioning metadata is not present before 0.17.0, the auto-compaction scheduling logic will re-compact all segments within a datasource once after the upgrade to 0.17.0.
This re-compaction on the entire set of segments for each datasource that has auto-compaction enabled means that:
Users are advised to be aware of the temporary increase in scheduled compaction tasks and the impact on deep storage usage. Documentation on removing old segments is located at https://druid.apache.org/docs/0.17.0/ingestion/data-management.html#deleting-data
targetCompactionSizeBytes
property removedThe targetCompactionSizeBytes
property has been removed from the compaction task and auto-compaction configuration. For auto-compaction, maxRowsPerSegment
is now a mandatory configuration. For non-auto compaction tasks, any partitionsSpec
can be used.
https://github.com/apache/incubator-druid/pull/8573
Due to the parallel auto-compaction changes introduced by #8570, any manually submitted compaction task specs need to be updated to use an index_parallel type for the tuningConfig section instead of index. These spec changes should be applied after the cluster is upgraded to 0.17.0.
Existing auto-compaction configs can remain unchanged after the update; the auto-compaction will create non-parallel compaction tasks until the auto-compaction configs are updated to use parallelism post-upgrade.
To control the level of parallelism, the auto-compactiontuningConfig
has new parameters, maxNumConcurrentSubTasks
and splitHintSpec
.
Please see https://druid.apache.org/docs/0.17.0/configuration/index.html#compaction-dynamic-configuration for details.
The compaction task now requires an ioConfig
in the task spec.
Please see https://druid.apache.org/docs/0.17.0/ingestion/data-management.html#compaction-ioconfig for details.
ioConfig
does not have to be added to existing auto-compaction configurations, the coordinator after the upgrade will automatically create task specs with ioConfig
sections.
https://github.com/apache/incubator-druid/pull/8571
The targetPartitionSize
and maxSegmentSize
fields in the partition specs have been deprecated. They have been renamed to targetNumRowsPerSegment
and maxRowsPerSegment
respectively.
https://github.com/apache/incubator-druid/pull/8507
Cache metrics are now disabled by default. To enable cache metrics, add "org.apache.druid.client.cache.CacheMonitor"
to the druid.monitoring.monitors
property.
https://github.com/apache/incubator-druid/pull/8561
Supervisor task specs should now put the dataSchema
, tuningConfig
, and ioConfig
sections as subfields of a spec
field. Please see https://github.com/apache/incubator-druid/pull/8810 for examples.
The old format is still accepted in 0.17.0.
The /datasources/{dataSourceName}/segments
endpoint on the Coordinator now returns all used segments (including overshadowed) on the specified intervals, rather than only visible ones.
https://github.com/apache/incubator-druid/pull/8564
The druid.emitter.http.basicAuthentication
property now accepts a password provider. We recommend updating your configurations to use a password provider if using the HTTP emitter.
https://github.com/apache/incubator-druid/pull/8618
Reusing multi-valued columns in expressions will no longer result in unnecessary cartesian explosions. Please see the following links for details.
https://github.com/apache/druid/issues/8947
https://github.com/apache/incubator-druid/pull/8957
During a rolling upgrade, if there are tasks running 0.17.0 and overlords running older versions, and a task made progress reading data from its stream but rejected all the records it saw (e.g., all were unparseable), you will see NullPointerExceptions on overlords running older versions when the task updates the overlord with its current stream offsets.
Previously, there was a bug in this area (https://github.com/apache/incubator-druid/issues/8765) where such tasks would fail to communicate their current offsets to the overlord. The task/overlord publishing protocol has been updated to fix this, but older overlords do not recognize this protocol change.
This condition should be fairly rare.
Druid 0.17.0 now bundles Hadoop 2.8.5 libraries instead of 2.8.3.
If you were referencing the bundled 2.8.3 libraries in your configuration via the druid.indexer.task.defaultHadoopCoordinates
property, or the hadoopDependencyCoordinates
property in your Hadoop ingestion specs, you will need to update these references to point to the bundled 2.8.5 libraries instead.
If you were maintaining a custom extension that provides an implementation for the ParseSpec
interface, the verify
method has been removed, and the @Override
annotation on the method will need to be removed in any custom implementations.
https://github.com/apache/druid/pull/8744
We are currently aware of a bug with applying certain filter types on null values from long columns when SQL-compatible null handling is enabled (https://github.com/apache/druid/issues/9255).
Please file an issue if you encounter any other null handling problems.
The preview specs shown for native batch ingestion tasks created in the Data Loader of the web console are not correctly formatted and will fail if you copy them and submit them manually. Submitting these specs through the Data Loader submits a correctly formatted spec, however.
https://github.com/apache/druid/issues/9144
For a full list of open issues, please see https://github.com/apache/druid/labels/Bug
Thanks to everyone who contributed to this release!
@a2l007
@abhishekrb19
@aditya-r-m
@AlexanderSaydakov
@asdf2014
@capistrant
@ccaominh
@clintropolis
@denever
@elloooooo
@FaxianZhao
@fjy
@Fokko
@fstolba
@gianm
@gkc2104
@glasser
@GraceKoo
@himanshug
@jihoonson
@jnaous
@jon-wei
@karthikbhat13
@legendtkl
@leventov
@maytasm3
@mcbrewster
@mitchlloyd
@mohammadjkhan
@nishantmonu51
@pdeva
@pjain1
@pzhdfy
@QiuMM
@qutang1
@renevan10
@richardstartin
@samarthjain
@SandishKumarHN
@sashidhar
@sekingme
@SEKIRO-J
@suneet-s
@surekhasaharan
@tijoparacka
@vogievetsky
@xingbowu
@xvrl
@yuanlihan
@yunwan
@yurmix
@zhenxiao
Published by jon-wei almost 5 years ago
Apache Druid 0.16.1-incubating is a bug fix and user experience improvement release that fixes a rolling upgrade issue, improves the startup scripts, and updates licensing information.
Bug Fixes
#8682 implement FiniteFirehoseFactory in InlineFirehose
#8905 Retrying with a backward compatible task type on unknown task type error in parallel indexing
User Experience Improvements
#8792 Use bundled ZooKeeper in tutorials.
#8794 Startup scripts: verify Java 8 (exactly), improve port/java verification messages.
#8942 Improve verify-default-ports to check both INADDR_ANY and 127.0.0.1.
#8798 Fix verify script.
Licensing Update
#8944 Add license for tutorial wiki data
#8968 Add licenses.yaml entry for Wikipedia sample data
Other
#8419 Bump Apache Thrift to 0.10.0
Updating from 0.16.0-incubating and earlier
PR #8905 fixes an issue with rolling upgrades when updating from earlier versions.
Credits
Thanks to everyone who contributed to this release!
@aditya-r-m
@clintropolis
@Fokko
@gianm
@jihoonson
@jon-wei
Apache Druid (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Published by clintropolis about 5 years ago
Apache Druid 0.16.0-incubating contains over 350 new features, performance enhancements, bug fixes, and major documentation improvements from 50 contributors. Check out the complete list of changes and everything tagged to the milestone.
An experimental 'vectorized' query execution engine is new in 0.16.0, which can provide a speed increase in the range of 1.3-3x for timeseries and group by v2 queries. It operates on the principle of batching operations on rows instead of processing a single row at a time, e.g. iterating bitmaps in batches instead of per row, reading column values in batches, filtering in batches, aggregating values in batches, and so on. This results in significantly fewer method calls, better memory locality, and increased cache efficiency.
This is an experimental feature, but we view it as the path forward for Druid query processing and are excited for feedback as we continue to improve and fill out missing features in upcoming releases.
The feature can be enabled by setting "vectorize": true
your query context (the default is false
). This works both for Druid SQL and for native queries. When set to true
, vectorization will be used if possible; otherwise, Druid will fall back to its non-vectorized query engine. You can also set it to "force"
, which will return an error if the query cannot be fully vectorized. This is helpful for confirming that vectorization is indeed being used.
You can control the block size during execution by setting the vectorSize
query context parameter (default is 1000
).
https://github.com/apache/incubator-druid/issues/7093
https://github.com/apache/incubator-druid/pull/6794
groupBy v2 queries now use an array-based representation of result rows, rather than the map-based representation used by prior versions of Druid. This provides faster generation and processing of result sets. Out of the box this change is invisible and backwards-compatible; you will not have to change any configuration to reap the benefits of this more efficient format, and it will have no impact on cached results. Internally this format will always be utilized automatically by the broker in the queries that it issues to historicals. By default the results will be translated back to the existing 'map' based format at the broker before sending them back to the client.
However, if you would like to avoid the overhead of this translation, and get even faster results,resultAsArray
may be set on the query context to directly pass through the new array based result row format. The schema is as follows, in order:
https://github.com/apache/incubator-druid/issues/8118
https://github.com/apache/incubator-druid/pull/8196
The complete set of pull requests tagged as performance enhancements for 0.16 can be found here.
Users of the Kafka indexing service and compaction and who get a trickle of late data, can find a huge improvement in the form of a new concept called 'minor' compaction. Enabled by internal changes to how data segments are versioned, minor compaction is based on the idea of 'segment' based locking at indexing time instead of the current Druid locking behavior (which is now referred to as 'time chunk' locking). Segment locking as you might expect allows only the segments which are being compacted to be locked, while still allowing new 'appending' indexing tasks (like Kafka indexing tasks) to continue to run and create new segments, simulataneously. This is a big deal if you get a lot of late data, because the current behavior results in compaction tasks starving as higher priority realtime tasks hog the locks. This prevention of compaction tasks from optimizing the datasources segment sizes results in reduced overall performance.
To enable segment locking, you will need to set forceTimeChunkLock
to false
in the task context, or set druid.indexer.tasklock.forceTimeChunkLock=false
in the Overlord configuration. However, beware, after enabling this feature, due to the changes in segment versioning, there is no rollback path built in, so once you upgrade to 0.16, you cannot downgrade to an older version of Druid. Because of this, we highly recommend confirming that Druid 0.16 is stable in your cluster before enabling this feature.
It has a humble name, but the changes of minor compaction run deep, and it is not possible to adequately describe the mechanisms that drive this in these release notes, so check out the proposal and PR for more details.
https://github.com/apache/incubator-druid/issues/7491
https://github.com/apache/incubator-druid/pull/7547
The new Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system and to better enable resource sharing across tasks.
The advantage of the Indexer is that it allows query processing resources, lookups, cached authentication/authorization information, and much more to be shared between all running indexing task threads, giving each individual task access to a larger pool of resources and far fewer redundant actions done than is possible with the Peon model of execution where each task is isolated in its own process.
Using Indexer does come with one downside: the loss of process isolation provided by Peon processes means that a single task can potentially affect all running indexing tasks on that Indexer. The druid.worker.globalIngestionHeapLimitBytes
and druid.worker.numConcurrentMerges
configurations are meant to help minimize this. Additionally, task logs for indexer processes will be inline with the Indexer process log, and not persisted to deep storage.
You can start using indexing by supplying server indexer
as the command-line argument to org.apache.druid.cli.Main
when starting the service. To use Indexer in place of a MiddleManager and Peon, you should be able to adapt values from the configuration into the Indexer configuration, lifting druid.indexer.fork.property.
configurations directly to the Indexer, and sizing heap and direct memory based on the Peon sizes multiplied by the number of task slots (unlike a MiddleManager, it does not accept the configurations druid.indexer.runner.javaOpts
or druid.indexer.runner.javaOptsArray
). See the indexer documentation for details.
https://github.com/apache/incubator-druid/pull/8107
In 0.16.0, Druid's index_parallel
native parallel batch indexing task now supports 'perfect' rollup with the implementation of a 2 stage shuffle process.
Tasks in stage 1 perform a secondary partitioning of rows on top of the standard time based partitioning of segment granularity, creating an intermediary data segment for each partition. Stage 2 tasks are each assigned a set of the partitionings created during stage 1, and will collect and combine the set of intermediary data segments which belong to that partitioning, allowing it to achieve complete rollup when building the final segments. At this time, only hash-based partitioning is supported.
This can be enabled by setting forceGuaranteedRollup
to true
in the tuningConfig
; numShards
in partitionsSpec
and intervals
in granularitySpec
must also be set.
The Druid MiddleManager (or the new Indexer) processes have a new responsibility for these indexing tasks, serving the intermediary partition segments output of stage 1 into the stage 2 tasks, so depending on configuration and cluster size, the MiddleManager jvm configuration might need to be adjusted to increase heap allocation and http threads. These numbers are expected to scale with cluster size, as all MiddleManager or Indexer processes involved in a shuffle will need the ability to communicate with each other, but we do not expect the footprint to be significantly larger than it is currently. Optimistically we suggest trying with your existing configurations, and bumping up heap and http thread count only if issues are encountered.
https://github.com/apache/incubator-druid/issues/8061
The console data loader, introduced in 0.15, has been expanded in 0.16 to support Kafka, Kinesis, segment reindexing, and even "inline" data which can be pasted in directly.
https://github.com/apache/incubator-druid/pull/7643
https://github.com/apache/incubator-druid/pull/8181
https://github.com/apache/incubator-druid/pull/8056
https://github.com/apache/incubator-druid/pull/7947
The query view on the web console has received a major upgrade for 0.16, transitioning into an interactive point-and-click SQL editor. There is a new column picker sidebar and the result output table let you directly manipulate the SQL query without needing to actually type SQL.
There is also a query history view and a the ability to fully edit the query context.
https://github.com/apache/incubator-druid/pull/7905
https://github.com/apache/incubator-druid/pull/7934
https://github.com/apache/incubator-druid/pull/8251
https://github.com/apache/incubator-druid/pull/7816
The "Data servers" view has been renamed to "Servers", and now displays a list of all Druid processes which are discovered as part of the cluster, using the sys.servers
system table.
This should make it much more convenient to at a glance ensure that all your Druid servers are up and reporting to the rest of the cluster.
https://github.com/apache/incubator-druid/pull/7770
https://github.com/apache/incubator-druid/pull/7654
The datasource view adds a new segment timeline visualization, allowing segment size and distribution to be gauged visually.
If you previously have used what is now the legacy coordinator console, do not be alarmed, the timeline is reversed and the newest segments are now on the right side of the visualization!
https://github.com/apache/incubator-druid/pull/8202
The tasks view has been significantly improved to assist a cluster operators ability to manage Druid supervisors and indexing tasks. The supervisors display now includes status information to allow determining their overall status and displaying error messaging if a supervisor is in error.
https://github.com/apache/incubator-druid/pull/7428
https://github.com/apache/incubator-druid/pull/7799
Druid SQL and native query expression 'virtual columns' can now correctly utilize multi-value string columns, either operating on them as individual VARCHAR
values for parity with native Druid query behavior, or as an array like type, with a new set of multi-value string functions. Note that so far this is only supported in query time expressions; use of multi-value expressions at ingestion time via a transformSpec
will be available in a future release. The complete list of added functions and behavior changes is far too large to list here, so check out the SQL and expression documentation for more details.
https://github.com/apache/incubator-druid/issues/7525
https://github.com/apache/incubator-druid/pull/7588
https://github.com/apache/incubator-druid/pull/7950
https://github.com/apache/incubator-druid/pull/7973
https://github.com/apache/incubator-druid/pull/7974
https://github.com/apache/incubator-druid/pull/8011
To complement TIMESTAMPADD
which can modify timestamps by time units, a TIMESTAMPDIFF
which can compute the signed number of time units between two timestamps. For syntax information, check out the SQL documentation
https://github.com/apache/incubator-druid/pull/7695
TIME_CEIL
, a time specific, more flexible version of the CEIL
function, has also been added to Druid 0.16.0. This function can round a timestamp up by an ISO8601 period, like P3M (quarters) or PT12H (half-days), optionally for a specific timezone. See SQL documentation for additional information.
https://github.com/apache/incubator-druid/pull/8027
Druid 0.16.0 also adds specialized SQL operators and native expressions for dealing with IPv4 internet addresses in dotted-decimal string or integer format. The new operators are IPV4_MATCH(address, subnet)
, IPV4_PARSE(address)
, and IPV4_STRINGIFY(address)
, which can match IP addresses to subnets in CIDR notation, translate dotted-decimal string format to integer format, and translate integer format into dotted-decimal string format, respectively. See SQL documentation for details.
https://github.com/apache/incubator-druid/pull/8223
To increase permissiveness and SQL compatibility with users coming to Druid with experience in other databases, an alias for the COALESCE
function has been added in the form of NVL
, which some SQL dialects use instead. See SQL documentation for details
https://github.com/apache/incubator-druid/pull/7965
Long, double, and float sum, min, and max aggregators now will permissively work when used with string columns, performing a best-effort parsing to try and translate them to the correct numerical type.
https://github.com/apache/incubator-druid/pull/8243
https://github.com/apache/incubator-druid/pull/8319
An official "convenience binary" Docker image will now be offered for every release starting with 0.16, available at https://hub.docker.com/r/apache/incubator-druid.
The documentation section of the Druid website has received a major upgrade, transitioning to using Docusaurus, which creates much more beautiful and functional pages than we have currently. Each documentation page now has a left-hand collapsible table of contents showing the outline of the overall docs, and a right-hand table of contents showing the outline of that particular doc page, vastly improving navigability.
Alongside this, the ingestion documentation has been totally refreshed, beginning with a new ingestion/index.md
doc that introduces all the key ingestion spec concepts, and describes the most popular ingestion methods. This was a much needed rework of many existing documentation pages, which had grown organically over time and have become difficult to follow, into a simpler set of fewer, larger, more cross-referenced pages. They are also a bit more 'opinionated', pushing new people towards Kafka, Kinesis, Hadoop, and native batch ingestion. They discuss Tranquility but don't present it as something highly recommended since it is effectively in minimal maintenance mode.
Check out these improvements (and many more) here.
https://github.com/apache/incubator-druid/pull/8311
The druid-datasketches
extension, built on top of Apache Datasketches (incubating), has been expanded with 3 new post aggregators, quantilesDoublesSketchToRank
which computes an approximation to the rank of a given value that is the fraction of the distribution less than that value, and quantilesDoublesSketchToCDF
which computes an approximation to the Cumulative Distribution Function given an array of split points that define the edges of the bins.
Another post aggregation, thetaSketchToString
which will print a summary of sketch has been added to assist in debugging. See Datasketches extension documentation to learn more.
https://github.com/apache/incubator-druid/pull/7550
https://github.com/apache/incubator-druid/pull/7937
The HLLSketch
aggregator has been improved with a query-time only round
option to support rounding values into whole numbers, to give it feature parity with the built-in cardinality
and hyperUnique
aggregators.
https://github.com/apache/incubator-druid/pull/8023
Finally, users of HllSketch
should also see a performance improvement due to some changes made which allow Druid to precompute an empty sketch and copy that into the aggregation buffers, greatly decreasing time to initialize the aggregator during query processing.
https://github.com/apache/incubator-druid/pull/8194
The druid-stats
core extension has been enhanced with SQL support, exposing VAR_POP
and VAR_SAMP
to compute variance population and sample with the variance aggregator, as well as STDDEV_POP
and STDDEV_SAMP
to compute standard deviation population and sample using the standard deviation post aggregator. Additionally, VARIANCE
and STDDEV
functions are added as aliases for VAR_SAMP
and STDDEV_SAMP
respectively. See SQL documentation and stats extension documentation for more details.
https://github.com/apache/incubator-druid/pull/7801
The 'fixed bucket' histogram aggregator of the druid-histogram
extension has a new added property finalizeAsBase64Binary
that enables serializing the resulting histogram as a base64 string instead of a human readable JSON summary, making it consistent with the approximate histogram aggregator of the same extension. See the documentation for further information.
https://github.com/apache/incubator-druid/pull/7784
The statsd-emitter
extension has a new option, druid.emitter.statsd.dogstatsdServiceAsTag
which enables emitting the Druid service name as a 'tag', such as druid_service:druid/broker
instead of part of the metric name as in druid.broker.query.time
. druid
is used as the prefix. so the example of druid.broker.query.time
would instead be druid.query.time
, allowing consolidation metrics across Druid service types and discriminating by tag instead. druid.emitter.statsd.dogstatsd
must be set to true
for this setting to take effect. See the documentation for more details.
https://github.com/apache/incubator-druid/pull/8238
https://github.com/apache/incubator-druid/pull/8472
A new set of approximate sketch aggregators for computing quantiles and the like and based on t-digest has been added in Druid 0.16. T-digest was designed for parallel programming use cases like distributed aggregations or map reduce jobs by making combining two intermediate t-digests easy and efficient. It serves to complement existing algorithms provided by the Apache Datasketches extension and moments sketch extension. See the extension documentation for more details.
https://github.com/apache/incubator-druid/issues/7303
https://github.com/apache/incubator-druid/pull/7331
A new Druid emitter extension to allow sending Druid metrics to influxdb over HTTP has also been added in 0.16. Currently this emitter only emits service metric events to InfluxDB (See Druid metrics for a list of metrics). When a metric event is fired it is added to a queue of events. After a configurable amount of time, the events on the queue are transformed to InfluxDB's line protocol and POSTed to the InfluxDB HTTP API. The entire queue is flushed at this point. The queue is also flushed as the emitter is shutdown. See the extension docs for details.
https://github.com/apache/incubator-druid/pull/7717
In addition to the experimental vectorized query engine and new indexer process type, 0.16 also has some additional features available to allow potentially fine tuning indexing and query performance via experimentation.
First up, the ability to independently control what compression is used (or disable it) when persisting intermediary segments during indexing. This configuration available to the indexSpec
property, and can be added to tuningConfig
as:
"indexSpecForIntermediatePersists": {
"dimensionCompression": "uncompressed",
"metricCompression": "none"
}
for example to disable compression entirely for intermediary segments. One potential reason to consider 'uncompressed' intermediary segments is to ease up on the amount of Java 'direct' memory required to perform the final merge of intermediary segments before they are published and pushed to deep storage, as reading data from uncompressed columns does not require the 64kb direct buffers which are used to decode lz4 and other encoded columns. Of course this is a trade-off of storage space and page cache footprint, so we recommend experimenting with this before settling on a configuration to use for your production workloads.
https://github.com/apache/incubator-druid/pull/7919
Bitmap indexes are usually a huge performance boost for Druid, but under some scenarios can result in slower query speeds, particularly in cases of computationally expensive filters on very high cardinality dimensions. In Druid 0.16, a new mechanism to provide some manual control over when bitmap indexes are utilized, and when a filter will be done as a row scan are now in place, and available on a per filter, per query basis. Most filters will accept a new property, filterTuning
, which might look something like this:
"filterTuning": {
"useBitmapIndex": true
}
useBitmapIndex
if set to false will disallow a filter to utilize bitmap indexes. This property is optional, and default behavior if filterTuning
is not supplied remains unchanged. Note that this feature is not documented in user facing documentation, considered experimental, and subject to change in any future release.
https://github.com/apache/incubator-druid/pull/8209
If you would have liked to enable Druid request logging, but use Druid SQL and find them a bit too chatty due to all the metadata queries, you are luck with 0.16 due to a new configuration option that allows selectively muting specific types of queries from request logging. The option, druid.request.logging.mutedQueryTypes
, accepts a list of "queryType" strings as defined by Druid's native JSON query API, and defaults to an empty list (so nothing is ignored). For example,
druid.request.logging.mutedQueryTypes=["segmentMetadata", "timeBoundary"]
would mute all request logs for segmentMetadata
and timeBoundary
queries.
https://github.com/apache/incubator-druid/pull/7562
After upgrading to 0.16.0, the task logs will require the same S3 permissions as pushing segments to deep storage, which has some additional logic to possibly get the bucket owner (by fetching the bucket ACL) and set the ACL of the segment object to give the bucket owner full access. If you wish to avoid providing these additional permissions, the existing behavior can be retained by setting druid.indexer.logs.disableAcl=true
.
https://github.com/apache/incubator-druid/pull/7907
Druid SQL is now enabled on the router by default. Note that this may put some additional pressure on broker processes, due to additional metadata queries required to maintain datasource schemas for query planning, and can be disabled to retain the previous behavior by setting druid.sql.enable=false
.
https://github.com/apache/incubator-druid/pull/7808
Druid SQL queries that use the LOOKUP
function will now take advantage of injective
lookups that are a one-to-one transformation, which allows an optimization to be performed that avoids pushing down lookup evaluation to historical processes, instead allowing the transformation to be done later. The drawback of this change is that lookups which are incorrectly defined as being one-to-one, but are not in fact a one-to-one transformation at evaluation time will produce incorrect query results.
https://github.com/apache/incubator-druid/pull/7655
The metadata API provided by the coordinator has been modified to provide more consistent HTTP responses. The changes are minor, but backwards incompatible if you depend on a specific API.
/druid/coordinator/v1/datasources/{dataSourceName}
/druid/coordinator/v1/datasources/{dataSourceName}
/druid/coordinator/v1/datasources/{dataSourceName}/markUnused
/druid/coordinator/v1/datasources/{dataSourceName}/markUsed
now return a JSON object of the form {"numChangedSegments": N}
instead of 204 (empty response) when no segments were changed. On the other
hand, 500 (server error) is not returned instead of 204 (empty response).
/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}
/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}
now return a JSON object of the form {"segmentStateChanged": true/false}
.
/druid/coordinator/v1/metadata/datasources?includeDisabled
is now /druid/coordinator/v1/metadata/datasources?includeUnused
. includeDisabled
is still accepted, but a warning is emitted in a log.
https://github.com/apache/incubator-druid/pull/7653
The keepSegmentGranularity
option would create compaction tasks that ignored interval boundary of segments before compacting them, but was deprecated because it was not very useful in most cases. If you did find this behavior useful, it can still be achieved by setting segmentGranularity
to ALL
.
https://github.com/apache/incubator-druid/pull/7747
There is a known issue with all Druid versions that support 'auto' compaction via the Coordinator, where auto compaction can get stuck repeatedly trying the same interval. This issue can now be mitigated in 0.16 by setting targetCompactionSizeBytes
, but is still present when using maxRowsPerSegment
or maxTotalRows
. A future version of Druid will fully resolve this issue.
https://github.com/apache/incubator-druid/issues/8481
https://github.com/apache/incubator-druid/pull/8495
All native indexing task types now use PartitonsSpec
to define secondary partitioning to be consistent with other indexing task types. The changes are not backwards incompatible, but users should consider migrating maxRowsPerSegment
, maxTotalRows
, numShards
, and partitionDimensions
to the partitions spec of native, kafka, and kinesis indexing tasks.
https://github.com/apache/incubator-druid/pull/8141
Incremental publishing of segments for the Apache Kafka indexing service was introduced in Druid 0.12, at which time the existing Kafka indexing service was retained as a 'legacy' format to allow rolling update from older versions of Druid. This 'legacy' codebase has now been removed in Druid 0.16.0, which means that a rolling-update from a Druid version older than 0.12.0 is not supported.
https://github.com/apache/incubator-druid/pull/7735
The fromPigAvroStorage
option has been removed from the Apache Avro extension in Druid 0.16, in order to clean up dependencies. This option provided a special transformation of GenericData.Array
, but this column type will now be handled by the default List
implementation.
https://github.com/apache/incubator-druid/pull/7810
druid.auth.authenticator.kerberos.excludedPaths
superseded by druid.auth.unsecuredPaths
which applies across all authenticators/authorizers, so use this configuration property instead.
https://github.com/apache/incubator-druid/pull/7745
The setting druid.zk.service.terminateDruidProcessOnConnectFail
has been removed and this is now the default behavior - Druid processes will exit when an unhandled Curator client exception occurs instead of continuing to run in a 'zombie' state.
Additionally, a new optional setting druid.zk.service.connectionTimeoutMs
, which wires into the Curator client connectionTimeoutMs
, setting is now available to customize. By default it will continue to use the Curator default of 15000
milliseconds.
https://github.com/apache/incubator-druid/pull/8458
The deprecated standalone 'realtime' process has been removed from Druid. Use of this has been discouraged for some time, but now middleManager
and the new index
processes should process all of your realtime indexing tasks.
https://github.com/apache/incubator-druid/pull/7915
https://github.com/apache/incubator-druid/pull/8020
Thanks to everyone who contributed to this release!
@a2l007
@abossert
@AlexanderSaydakov
@AlexandreYang
@andrewluotechnologies
@asdf2014
@awelsh93
@blugowski
@capistrant
@ccaominh
@clintropolis
@dclim
@dene14
@dinsaw
@Dylan1312
@esevastyanov
@fjy
@Fokko
@gianm
@gocho1
@himanshug
@ilhanadiyaman
@jennyzzz
@jihoonson
@Jinstarry
@jon-wei
@justinborromeo
@kamaci
@khwj
@legoscia
@leventov
@litao91
@lml2468
@mcbrewster
@nieroda
@nishantmonu51
@pjain1
@pphust
@samarthjain
@SandishKumarHN
@sashidhar
@satybald
@sekingme
@shuqi7
@surekhasaharan
@viongpanzi
@vogievetsky
@xueyumusic
@xvrl
@yurmix
Published by clintropolis about 5 years ago
Apache Druid 0.15.1-incubating is a bug fix release that includes important fixes for Apache Zookeeper based segment loading, the 'druid-datasketches' extension, and much more.
#8137 coordinator throwing exception trying to load segments (fixed by #8140)
#7886 Middlemanager fails startup due to corrupt task files (fixed by #7917)
#8085 fix forking task runner task shutdown to be more graceful
#7777 timestamp_ceil function is either wrong or misleading (fixed by #7823)
#7820 subtotalsSpec and filtering returns no results (fixed by #7827)
#8013 Fix ExpressionVirtualColumn capabilities; fix groupBy's improper uses of StorageAdapter#getColumnCapabilities.
#6786 apache-druid-0.13.0-incubating router /druid/router/v1/brokers (fixed by #8026)
#8044 SupervisorManager: Add authorization checks to bulk endpoints.
#8204 HttpPostEmitter throw Class cast exception when using emitAndReturnBatch (fixed by #8205)
#7666 sketches-core-0.13.4
#8055 force native order when wrapping ByteBuffer
#7830 Kinesis: Fix getPartitionIds, should be checking isHasMoreShards.
#7999 Druid moving average query results in circular reference error (fixed by #8192)
#8002 Improve pull-deps reference in extensions page.
#8003 Add missing reference to Materialized-View extension.
#8079 Fix documentation formatting
#8087 fix references to bin/supervise in tutorial docs
Due to issue #8137, when updating from specifically 0.15.0-incubating to 0.15.1-incubating, it is recommended to update the Coordinator before the Historical servers to prevent segment unavailability during an upgrade (this is typically reversed). Upgrading from any version older than 0.15.0-incubating does not have these concerns and can be done normally.
Building Docker images is currently broken and will be fixed in the next release, see #8054 which is fixed by #8237 for more details.
Thanks to everyone who contributed to this release!
@AlexanderSaydakov
@ArtyomyuS
@ccl0326
@clintropolis
@gianm
@himanshug
@jihoonson
@legoscia
@leventov
@pjain1
@yurmix
@xueyumusic
Apache Druid (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Published by jihoonson over 5 years ago
Apache Druid 0.15.0-incubating contains over 250 new features, performance/stability/documentation improvements, and bug fixes from 39 contributors. Major new features and improvements include:
The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+is%3Aclosed+milestone%3A0.15.0
Documentation for this release is at: http://druid.apache.org/docs/0.15.0-incubating/
Druid has a new Data Loader UI which is integrated with the Druid Console. The new Data Loader UI shows some sampled data to easily verify the ingestion spec and generates the final ingestion spec automatically. The users are expected to easily issue batch index tasks instead of writing a JSON spec by themselves.
Added by @vogievetsky and @dclim in https://github.com/apache/incubator-druid/pull/7572 and https://github.com/apache/incubator-druid/pull/7531, respectively.
The Kafka indexing service now supports Kafka Transactional Topics.
Please note that only Kafka 0.11.0 or later versions are supported after this change.
Added by @surekhasaharan in https://github.com/apache/incubator-druid/pull/6496.
A new query type was introduced to compute moving average.
Please see http://druid.apache.org/docs/0.15.0-incubating/development/extensions-contrib/moving-average-query.html for more details.
Added by @yurmix in https://github.com/apache/incubator-druid/pull/6430.
The Scan query type now supports time ordering. Please see http://druid.apache.org/docs/0.15.0-incubating/querying/scan-query.html#time-ordering for more details.
Added by @justinborromeo in https://github.com/apache/incubator-druid/pull/7133.
The Moments Sketch is a new sketch type for approximate quantile computation. Please see http://druid.apache.org/docs/0.15.0-incubating/development/extensions-contrib/momentsketch-quantiles.html for more details.
Added by @edgan8 in https://github.com/apache/incubator-druid/pull/6581.
Druid community has been striving to enhance SQL support and now it's no longer experimental.
Druid Console now supports autocomplete for SQL.
Added by @shuqi7 in https://github.com/apache/incubator-druid/pull/7244.
Druid SQL supports time-ordered scan query.
Added by @justinborromeo in https://github.com/apache/incubator-druid/pull/7373.
You can now configure your lookups from the web console directly.
Added by @shuqi7 in https://github.com/apache/incubator-druid/pull/7259.
The web console now has a backup mode that allows it to function as best as it can if DruidSQL is disabled or unavailable.
You can now configure the auto compaction settings for a data source from the Datasource view.
The console query view will now (by default) wrap DruidSQL queries with a SELECT * FROM (...) LIMIT 1000
allowing you to enter queries like SELECT * FROM your_table
without worrying about the impact to the cluster. You can still send 'raw' queries by selecting the option from the ...
menu.
You can now click on the ...
menu in the query view to get an explanation of the DruidSQL query.
is_overshadowed
as a column in the segments table https://github.com/apache/incubator-druid/pull/7555 , https://github.com/apache/incubator-druid/pull/7425 [@shuqi7][@surekhasaharan]is_overshadowed
column represents that this segment is overshadowed by any published segments. It can be useful to see what segments should be loaded by historicals. Please see http://druid.apache.org/docs/0.15.0-incubating/querying/sql.html for more details.
This PR condenses the actions list into a tidy menu and lets you see the detailed status for supervisors and tasks. New actions for datasources around loading and dropping data by interval has also been added.
Light lookup module was introduced for Routers and they now need only minimum amount of memory. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/basic-cluster-tuning.html#router for basic memory tuning.
Added by @clintropolis in https://github.com/apache/incubator-druid/pull/7222.
ORC extension is now promoted to a core extension. Please read the below 'Updating from 0.14.0-incubating and earlier' section if you are using the ORC extension in an earlier version of Druid.
Added by @clintropolis in https://github.com/apache/incubator-druid/pull/7138.
GCP extension is now promoted to a core extension. Please read the below 'Updating from 0.14.0-incubating and earlier' section if you are using the GCP extension in an earlier version of Druid.
Added by @drcrallen in https://github.com/apache/incubator-druid/pull/6953.
Several configurations and scripts were added for easy single machine setup. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/single-server.html for details.
Added by @jon-wei in https://github.com/apache/incubator-druid/pull/7590.
A new tool was added for easy migration from single machine to a cluster environment. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/deep-storage-migration.html for details.
Added by @jon-wei in https://github.com/apache/incubator-druid/pull/7598.
Documents for basic tuning guide was added. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/basic-cluster-tuning.html for details.
Added by @jon-wei in https://github.com/apache/incubator-druid/pull/7629.
The Druid system table now requires only mandatory permissions instead of the read permission for the whole sys
database. Please see http://druid.apache.org/docs/0.15.0-incubating/development/extensions-core/druid-basic-security.html for details.
Added by @jon-wei in https://github.com/apache/incubator-druid/pull/7579.
The automatic segment merge by the coordinator is not supported anymore. Please use auto compaction instead.
Added by @jihoonson in #6883.
insert-segment-to-db
toolIn Druid 0.14.x or earlier, Druid stores segment metadata (descriptor.json
file) in deep storage in addition to metadata store. This behavior has changed in 0.15.0 and it doesn't store segment metadata file in deep storage anymore. As a result, insert-segment-to-db
tool is no longer supported as well since it works based on descriptor.json
files in deep storage. Please see http://druid.apache.org/docs/0.15.0-incubating/operations/insert-segment-db.html for details.
Please note that kill task will fail if you're using HDFS as deep storage and descriptor.json
file is missing in 0.14.x or earlier versions.
Added by @jihoonson in https://github.com/apache/incubator-druid/pull/6911.
This option was removed since it generates unscalable query plans and doesn't work with some SQL functions.
Added by @gianm in https://github.com/apache/incubator-druid/pull/7567.
CompressionUtils
for extension developerspublic static void gunzip(File pulledFile, File outDir)
was removed in https://github.com/apache/incubator-druid/pull/6908 by @clintropolis.
A new configuration (druid.coordinator.segment.awaitInitializationOnStart
) was added to make Coordinator wait for segment view initialization. This option is enabled by default.
Added by @QiuMM in https://github.com/apache/incubator-druid/pull/6847.
The coordinator periodically polls segment metadata information from metadata store and caches them in memory. In Druid 0.14.x or earlier, removing segments via coordinator APIs (/druid/coordinator/v1/datasources/{dataSourceName}
and /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}
) immediately updates the segment cache in memory as well as metadata store. But this behavior has changed in 0.15.0 and the cache is updated per poll rather than being updated immediately on removal. The below APIs can return removed segments via the above API calls until the cache is updated in the next poll.
/druid/coordinator/v1/metadata/datasources/{dataSourceName}
/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments
/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments/{segmentId}
/druid/coordinator/v1/metadata/datasources
/druid/coordinator/v1/loadstatus
The below metrics can also contain removed segments via the above API calls until the cache is updated in the next poll.
segment/unavailable/count
segment/underReplicated/count
This behavior was changed in https://github.com/apache/incubator-druid/pull/7595 by @surekhasaharan.
The /druid/coordinator/v1/lookups/config
API now returns a list of tiers currently active in the cluster in addition to ones known in the dynamic configuration.
Added by @clintropolis in https://github.com/apache/incubator-druid/pull/7647.
With a new configuration (druid.zk.service.terminateDruidProcessOnConnectFail
), Druid processes can terminate itself on disconnection to ZooKeeper.
Added by @michael-trelinski in https://github.com/apache/incubator-druid/pull/6740.
Kafka 0.11.x or later versions are only supported after https://github.com/apache/incubator-druid/pull/6496. Please consider updating Kafka version if you're using an older one.
The ORC extension has been promoted to a core
extension. When deploying 0.15.0-incubating, please ensure that your extensions
directory does not have any older versions of druid-orc-extensions
extension.
Additionally, even though the new core extension can index any data the old contrib
extension could, the JSON spec for the ingestion task is incompatible, and will need modified to work with the newer core
extension.
To migrate to 0.15.0-incubating:
inputSpec
of ioConfig
, inputFormat
must be changed from "org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat"
to"org.apache.orc.mapreduce.OrcInputFormat"
contrib
extension supported a typeString
property, which provided the schema of thecore
extension, column renaming can be achieved with flattenSpec
expressions.contrib
extension supported a mapFieldNameFormat
property, which provided a way to specify a dimension to flatten OrcMap
columns with primitive types. This functionality has also been replaced with flattenSpec
expressions.For more details and examples, please see http://druid.apache.org/docs/0.15.0-incubating/development/extensions-core/orc.html.
The GCP extension has been promoted to a core
extension. When deploying 0.15.0-incubating, please ensure that your extensions
directory does not have any older versions of the druid-google-extensions
extension.
The coordinator configuration for auto segment merge (druid.coordinator.merge.on
) is not supported anymore. Please use auto compaction instead.
descriptor.json
metadata file in deep storageThe segment metadata file (descriptor.json
) is not stored in deep storage any more. If you are using HDFS as your deep storage and need to roll back to 0.14.x or earlier, then please consider that the kill task could fail because of the missing descriptor.json
files.
Thanks to everyone who contributed to this release!
@a2l007
@asdf2014
@capistrant
@clintropolis
@dampcake
@dclim
@donbowman
@drcrallen
@Dylan1312
@edgan8
@es1220
@esevastyanov
@FaxianZhao
@fjy
@gianm
@glasser
@hpandeycodeit
@jihoonson
@jon-wei
@jorbay-au
@justinborromeo
@kamaci
@KazuhitoT
@leventov
@lxqfy
@michael-trelinski
@peferron
@puneetjaiswal
@QiuMM
@richardstartin
@samarthjain
@scrawfor
@shuqi7
@surekhasaharan
@venkatramanp
@vogievetsky
@xueyumusic
@xvrl
@yurmix
Apache Druid 0.14.2-incubating is a bug fix release that includes important fixes for the 'druid-datasketches' extension and the broker 'result' level caching.
Thanks to everyone who contributed to this release!
@AlexanderSaydakov
@clintropolis
@jihoonson
@jon-wei
Apache Druid (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Apache Druid 0.14.1-incubating is a small patch release that includes a handful of bug and documentation fixes from 16 contributors.
This release fixes an issue with druid-datasketches
extension with quantile sketches, but introduces another one with theta sketches that was confirmed after the release was finalized, caused by #7320 and described in #7607. If you utilize theta sketches, we recommend not upgrading to this release. This will be fixed in the next release of Druid by #7619.
Updating from version 0.13.0-incubating or earlier directly to 0.14.1-incubating will not require downtime like the migration path to 0.14.0-incubating due to the issue described in #6958, which has been fixed for this release in #7212. Likewise, rolling updates from version 0.13.0-incubating and earlier should also work properly due to #7512.
Updating from 0.13.0-incubating directly to 0.14.1-incubating will not encounter any issues during a rolling update with mixed versions of middle managers due to the fixes in #7520, as could be experienced when updating to 0.14.0-incubating.
Thanks to everyone who contributed to this release!
@AlexanderSaydakov
@b-slim
@benhopp
@chrishardis
@clintropolis
@ferristseng
@es1220
@gianm
@jihoonson
@jon-wei
@justinborromeo
@kaka11chen
@samarthjain
@surekhasaharan
@zhaojiandong
@zhztheplayer
Apache Druid (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Apache Druid (incubating) 0.14.0-incubating contains over 200 new features, performance/stability/documentation improvements, and bug fixes from 54 contributors. Major new features and improvements include:
The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+is%3Amerged+milestone%3A0.14.0
Documentation for this release is at: http://druid.io/docs/0.14.0-incubating/
Druid has a new web console that provides functionality that was previously split between the coordinator and overlord consoles.
The new console allows the user to manage datasources, segments, tasks, data processes (Historicals and MiddleManagers), and coordinator dynamic configuration. The user can also run SQL and native Druid queries within the console.
For more details, please see http://druid.io/docs/0.14.0-incubating/operations/management-uis.html
Added by @vogievetsky in #6923.
Druid now supports ingestion from Kinesis streams, provided by the new druid-kinesis-indexing-service
core extension.
Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/kinesis-ingestion.html for details.
Added by @jsun98 in #6431.
Historical processes can now be put into a "decommissioning" mode, where the coordinator will no longer consider the Historical process as a target for segment replication. The coordinator will also move segments off the decommissioning Historical.
This is controlled via Coordinator dynamic configuration. For more details, please see http://druid.io/docs/0.14.0-incubating/configuration/index.html#dynamic-configuration.
Added by @egor-ryashin in #6349.
The Druid Broker now has the ability to maintain a cache of published segments via polling the Coordinator, which can significantly improve response time for metadata queries on the sys.segments
system table.
Please see http://druid.io/docs/0.14.0-incubating/querying/sql.html#retrieving-metadata for details.
Added by @surekhasaharan in #6901
A new aggregator for constructing Bloom filters at query time and support for performing Bloom filter checks within Druid expressions have been added to the druid-bloom-filter
extension.
Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/bloom-filter.html
Added by @clintropolis in #6904 and #6397
druid-extensions-parquet
has been moved into the core extension set from the contrib extensions and now supports flattening and int96 values.
Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/parquet.html for details.
Added by @clintropolis in #6360
Outer query execution for nested GroupBy queries can now be pushed down to Historical processes; previously, the outer queries would always be executed on the Broker.
Please see https://github.com/apache/incubator-druid/pull/5471 for details.
Added by @samarthjain in #5471.
Segment handoff will now ignore segments that would be dropped by a datasource's retention rules, avoiding ingestion failures caused by issue #5868.
Period load rules will now include the future by default.
A new "Period Drop Before" rule has been added. Please see http://druid.io/docs/0.14.0-incubating/operations/rule-configuration.html#period-drop-before-rule for details.
Added by @QiuMM in #6676, #6414, and #6415.
Druid will now automatically terminate MapReduce jobs created by Hadoop batch ingestion tasks when the ingestion task is killed.
Added by @ankit0811 in #6828.
The statsd-emitter
extension now supports DogStatsD-style tags. Please see http://druid.io/docs/0.14.0-incubating/development/extensions-contrib/statsd.html
Added by @deiwin in #6605, with support for constant tags added by @glasser in #6791.
A new API for retrieving all lookup specs for all tiers has been added. Please see http://druid.io/docs/0.14.0-incubating/querying/lookups.html#get-all-lookups for details.
Added by @jihoonson in #7025.
Auto-compaction now supports the maxRowsPerSegment
option. Please see http://druid.io/docs/0.14.0-incubating/design/coordinator.html#compacting-segments for details.
The compaction task now supports a new segmentGranularity
option, deprecating the older keepSegmentGranularity
option for controlling the segment granularity of compacted segments. Please see the segmentGranularity
table in http://druid.io/docs/0.14.0-incubating/ingestion/compaction.html for more information on these properties.
Added by @jihoonson in #6758 and #6780.
The cachingCost
Coordinator segment balancing strategy will now only consider Historical processes for balancing decisions. Previously the strategy would unnecessarily consider active worker tasks as well, which are not targets for segment replication.
Added by @QiuMM in #6879.
jvm/heapAlloc/bytes
, added by @egor-ryashin in #6710.query/count
, added by @QiuMM in #6473.sqlQuery/bytes
and sqlQuery/time
, added by @gaodayue in #6302.ingest/kafka/maxLag
and ingest/kafka/avgLag
, added by @QiuMM in #6587task/success/count
, task/failed/count
, task/running/count
, task/pending/count
, task/waiting/count
, added by @QiuMM in #6657It is now possible to control the fields in RequestLogEvent
, emitted by EmittingRequestLogger
. Please see #6477 for details. Added by @leventov.
An extension point for custom TLS certificate checks has been added. Please see http://druid.io/docs/0.14.0-incubating/operations/tls-support.html#custom-tls-certificate-checks for details. Added by @jon-wei in #6432.
The Kafka Indexing Service extension has been moved out of experimental status.
The dsql
command line client now supports CLI history, basic autocomplete, and specifying query timeouts in the query context.
Added in #6929 by @gianm.
SQL queries now have an ID, and native queries executed as part of a SQL query will have the associated SQL query ID in the native query's request logs. SQL queries will now be logged in the request logs.
Two new metrics, sqlQuery/time
and sqlQuery/bytes
, are now emitted for SQL queries.
Please see http://druid.io/docs/0.14.0-incubating/configuration/index.html#request-logging and http://druid.io/docs/0.14.0-incubating/querying/sql.html#sql-metrics for details.
Added by @gaodayue in #6302
The follow aggregators are now supported in SQL:
Added by @jon-wei in #6951 and @clintropolis in #6502
Added by @gianm.
Due to the issue described in #6958, existing Kafka indexing tasks can be terminated unnecessarily during a rolling upgrade of the Overlord. The terminated tasks will be restarted by the Overlord and will function correctly after the initial restart.
The druid-parquet-extensions
extension has been moved from contrib
to core
. When deploying 0.14.0-incubating, please ensure that your extensions-contrib
directory does not have any older versions of the Parquet extension.
Additionally, there are now two styles of Parquet parsers in the extension:
parquet-avro
: Converts Parquet to Avro, and then parses the Avro representation. This was the existing parser prior to 0.14.0-incubating.parquet
: A new parser that parses the Parquet format directly. Only this new parser supports int96 values.Prior to 0.14.0-incubating, a specifying a parquet
type parser would have a task use the Avro-converting parser. In 0.14.0-incubating, to continue using the Avro-converting parser, you will need to update your ingestion specs to use parquet-avro
instead.
The inputFormat
field in the inputSpec
for tasks using Parquet input must also match the choice of parser:
parquet
: org.apache.druid.data.input.parquet.DruidParquetInputFormat
parquet-avro
: org.apache.druid.data.input.parquet.DruidParquetInputFormat
Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/parquet.html for details.
If you plan to use Druid 0.14.0-incubating with Hadoop versions other than 2.8.3, you may need to do the following:
Tip #3: Use specific versions of Hadoop libraries
.hadoop.compile.version
in the main Druid pom.xml
and then following the standard build instructions.Old task entries in the metadata storage will now be cleaned up automatically together with their task logs. Please see http:/druid.io/docs/0.14.0-incubating/development/extensions-core/configuration/index.html#task-logging and #6592 for details.
The druid.processing.buffer.sizeBytes
property has new default behavior if it is not set. Druid will now automatically choose a value for the processing buffer size using the following formula:
processingBufferSize = totalDirectMemory / (numMergeBuffers + numProcessingThreads + 1)
processingBufferSize = min(processingBufferSize, 1GB)
Where:
-XX:MaxDirectMemorySize
druid.processing.numMergeBuffers
.druid.processing.numThreads
.At most, Druid will use 1GB for the automatically chosen processing buffer size. The processing buffer size can still be specified manually.
Please see #6588 for details.
Please be aware that new retention rules will now include the future by default. Please see #6414 for details.
The druid.announcer.type
property used for choosing between Zookeeper or HTTP-based segment management/discovery has been moved to druid.serverview.type
. If you were using http
prior to 0.14.0-incubating, you will need to update your configs to use the new druid.serverview.type
.
Please see the following for details:
The druid.peon.defaultSegmentWriteOutMediumFactory.@type
property has been fixed. The property is now druid.peon.defaultSegmentWriteOutMediumFactory.type
without the "@".
Please see #6656 for details.
The ApproximateHistogram aggregator has been deprecated; it is a distribution-dependent algorithm without formal error bounds and has significant accuracy issues.
The DataSketches quantiles aggregator should be used instead for quantile and histogram use cases.
Please see Histogram and Quantiles Aggregators
The Cardinality and HyperUnique aggregators have been deprecated in favor of the DataSketches HLL aggregator and Theta Sketch aggregator. These aggregators have better accuracy and performance characteristics.
Please see Count Distinct Aggregators for details.
The chunkPeriod
query context configuration is now deprecated, along with the associated query/intervalChunk/time
metric. Please see #6591 for details.
keepSegmentGranularity
for CompactionThe keepSegmentGranularity
option for compaction tasks has been deprecated. Please see #6758 and the segmentGranularity
table in http://druid.io/docs/0.14.0-incubating/ingestion/compaction.html for more information on these properties.
SegmentId
classDruid now uses a SegmentId
class instead of plain Strings to represent segment IDs. Please see #6370 for details.
Added by @leventov.
druid-api
, druid-common
, java-util
moved to druid-core
The druid-api
, druid-common
, java-util
modules have been moved into druid-core
. Please update your dependencies accordingly if your project depended on these libraries.
Please see #6443 for details.
Thanks to everyone who contributed to this release!
@a2l007
@AlexanderSaydakov
@anantmf
@ankit0811
@asdf2014
@awelsh93
@benhopp
@Caroline1000
@clintropolis
@dclim
@deiwin
@DiegoEliasCosta
@drcrallen
@dyf6372
@Dylan1312
@egor-ryashin
@elloooooo
@evans
@FaxianZhao
@gaodayue
@gianm
@glasser
@Guadrado
@hate13
@hoesler
@hpandeycodeit
@janeklb
@jihoonson
@jon-wei
@jorbay-au
@jsun98
@justinborromeo
@kamaci
@leventov
@lxqfy
@mirkojotic
@navkumar
@niketh
@patelh
@pzhdfy
@QiuMM
@rcgarcia74
@richardstartin
@robertervin
@samarthjain
@seoeun25
@Shimi
@surekhasaharan
@taiii
@thomask
@VincentNewkirk
@vogievetsky
@yunwan
@zhaojiandong
Published by jon-wei over 5 years ago
Published by dclim almost 6 years ago
Druid 0.13.0-incubating contains over 400 new features, performance/stability/documentation improvements, and bug fixes from 81 contributors. It is the first release of Druid in the Apache Incubator program. Major new features and improvements include:
The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+is%3Aclosed+milestone%3A0.13.0
Documentation for this release is at: http://druid.io/docs/0.13.0-incubating/
Introduces the index_parallel
supervisor which manages the parallel batch ingestion of splittable sources without requiring a dependency on Hadoop. See http://druid.io/docs/latest/ingestion/native_tasks.html for more information.
Note: This is the initial single-phase implementation and has limitations on how it expects the input data to be partitioned. Notably, it does not have a shuffle implementation which will be added in the next iteration of this feature. For more details, see the proposal at #5543.
Added by @jihoonson in #5492.
Previously, compacting small segments into optimally-sized ones to improve query performance required submitting and running compaction or re-indexing tasks. This was often a manual process or required an external scheduler to handle the periodic submission of tasks. This patch implements automatic segment compaction managed by the coordinator service.
Note: This is the initial implementation and has limitations on interoperability with realtime ingestion tasks. Indexing tasks currently require acquisition of a lock on the portion of the timeline they will be modifying to prevent inconsistencies from concurrent operations. This implementation uses low-priority locks to ensure that it never interrupts realtime ingestion, but this also means that compaction may fail to make any progress if the realtime tasks are continually acquiring locks on the time interval being compacted. This will be improved in the next iteration of this feature with finer-grained locking. For more details, see the proposal at #4479.
Documentation for this feature: http://druid.io/docs/0.13.0-incubating/design/coordinator.html#compacting-segments
Added by @jihoonson in #5102.
Adds a system schema to the SQL interface which contains tables exposing information on served and published segments, nodes of the cluster, and information on running and completed indexing tasks.
Note: This implementation contains some known overhead inefficiencies that will be addressed in a future patch.
Documentation for this feature: http://druid.io/docs/0.13.0-incubating/querying/sql.html#system-schema
Added by @surekhasaharan in #6094.
Improves the performance and detail of the ingestion-related APIs which were previously quite opaque making it difficult to determine the cause of parse exceptions, task failures, and the actual output from a completed task. Also adds improved ingestion metric reporting including moving average throughput statistics.
Added by @surekhasaharan and @jon-wei in #5801, #5418, and #5748.
Improves Druid's handling of null values by treating them as missing values instead of being equivalent to empty strings or a zero-value. This makes Druid more SQL compatible and improves integration with external BI tools supporting ODBC/JDBC. See #4349 for proposal.
To enable this feature, you will need to set the system-wide property druid.generic.useDefaultValueForNull=false
.
Added by @nishantmonu51 in #5278 and #5958.
Implements result-level caching on brokers which can operate concurrently with the traditional segment-level cache. See #4843 for proposal.
Documentation for this feature: http://druid.io/docs/0.13.0-incubating/configuration/index.html#broker-caching
Added by @a2l007 in #5028.
Introduces a sql
firehose which supports data ingestion directly from an RDBMS.
Added by @a2l007 in #5441.
Adds support for optimizing Druid queries by applying a Bloom filter generated by an external system such as Apache Hive. In the future, #6397 will support generation of Bloom filters as the result of Druid queries which can then be used to optimize future queries.
Added by @nishantmonu51 in #6222.
Adds result formats for line-based JSON and CSV and additionally X-Druid-Column-Names
and X-Druid-Column-Types
response headers containing a list of columns contained in the result.
Added by @gianm in #6191.
Introduces two complementary aggregators, stringLast
and stringFirst
which operate on string columns and return the value with the maximum and minimum timestamp respectively.
Added by @andresgomezfrr in #5789.
Adds support for numeric Tuple sketches, which extend the functionality of the count distinct Theta sketches by adding arrays of double values associated with unique keys.
Added by @AlexanderSaydakov in #5148.
Adds a configurable implementation of a count distinct aggregator based on HllSketch from https://github.com/DataSketches. Comparison to Druid's native HyperLogLogCollector shows improved accuracy, efficiency, and speed: https://datasketches.github.io/docs/HLL/HllSketchVsDruidHyperLogLogCollector.html
Added by @AlexanderSaydakov in #5712.
Adds support for the subtotalsSpec
groupBy parameter which allows Druid to be efficient by reusing intermediate results at the broker level when running multiple queries that group by subsets of the same set of columns. See proposal in #5179 for more information.
Added by @himanshug in #5280.
Adds support for mutual TLS (server certificate validation + client certificate validation). See: https://en.wikipedia.org/wiki/Mutual_authentication
Added by @jon-wei in #6076.
Adds an HTTP-based indexing task management implementation to replace the previous one based on ZooKeeper. Part of a set of improvements to reduce and eventually eliminate Druid's dependency on ZooKeeper. See #4996 for proposal.
Added by @himanshug in #5104.
Allows the broker to exert backpressure on data-serving nodes to prevent the broker from crashing under memory pressure when results are coming in faster than they are being read by clients.
Added by @gianm in #6313.
Previously, a major tuning parameter for indexing task memory management was the maxRowsInMemory
configuration, which determined the threshold for spilling the contents of memory to disk. This was difficult to properly configure since the 'size' of a row varied based on multiple factors. maxBytesInMemory
makes this configuration byte-based instead of row-based.
Added by @surekhasaharan in #5583.
Supports the creation of materialized views which can improve query performance in certain situations at the cost of additional storage. See http://druid.io/docs/latest/development/extensions-contrib/materialized-view.html for more information.
Note: This is a community-contributed extension and is not automatically included in the Druid distribution. We welcome feedback for deciding when to promote this to a core extension. For more information, see Community Extensions.
Added by @zhangxinyu1 in #5556.
Adds support for ingesting the Influx Line Protocol data format. For more information, see: https://docs.influxdata.com/influxdb/v1.6/write_protocols/line_protocol_tutorial/
Note: This is a community-contributed extension and is not automatically included in the Druid distribution. We welcome feedback for deciding when to promote this to a core extension. For more information, see Community Extensions.
Added by @njhartwell in #5440.
Adds support for emitting Druid metrics to OpenTSDB.
Note: This is a community-contributed extension and is not automatically included in the Druid distribution. We welcome feedback for deciding when to promote this to a core extension. For more information, see Community Extensions.
Added by @QiuMM in #5380.
Please see below for changes between 0.12.3 and 0.13.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.12.3, please see release notes of the relevant intermediate versions for additional notes.
The MySQL metadata storage extension is now packaged together with the Druid distribution but without the required MySQL JDBC driver (due to licensing restrictions). To use this extension, the driver will need to be downloaded separately and added to the extension directory.
See http://druid.io/docs/latest/development/extensions-core/mysql.html for more details.
As a result of switching from jets3t to the AWS SDK (#5382), users of the S3 extension are now required to explicitly set the target region. This can be done by setting the JVM system property aws.region
or the environment variable AWS_REGION
.
As an example, to set the region to 'us-east-1' through system properties:
-Daws.region=us-east-1
to the jvm.config file for all Druid services-Daws.region=us-east-1
to druid.indexer.runner.javaOpts
in middleManager/runtime.properties so that the property will be passed to peon (worker) processesAs a result of renaming packaging from io.druid
to org.apache.druid
, ingestion specs that reference classes by their fully-qualified class name will need to be modified accordingly.
As an example, if you are using the Parquet extension with Hadoop indexing, the inputFormat
field of the inputSpec
will need to change from io.druid.data.input.parquet.DruidParquetInputFormat
to org.apache.druid.data.input.parquet.DruidParquetInputFormat
.
task/action/log/time
- Milliseconds taken to log a task action to the audit log (#5714)task/action/run/time
- Milliseconds taken to execute a task action (#5714)query/node/backpressure
- Nanoseconds the channel is unreadable due to backpressure being applied (#6335) (Note that this is not enabled by default and requires a custom implementation of QueryMetrics
to emit)taskId
and taskType
added to task-related metrics (#5664)HttpPostEmitterMonitor
no longer emits maxTime and minTime if no times were recorded (#6418)64-bit doubles aggregators are now used by default (see #5478). Support for 64-bit floating point columns was release in Druid 0.11.0, so if this is enabled, versions older than 0.11.0 will not be able to read the data segments.
To disable and keep the old format, you will need to set the system-wide property druid.indexing.doubleStorage=float
.
0.13.0 adds support for disabling bitmap indexes on a per-column basis, which can save space in cases where bitmap indexes add no value. This is done by setting the 'createBitmapIndex' field in the dimension schema. Segments written with this option will not be backwards compatible with older versions of Druid (#5402).
Druid's package names have all moved from io.druid
to org.apache.druid
. This affects the name of the Java main class that you should run when starting up services, which is now org.apache.druid.cli.Main
. It may also affect installation and configuration of extensions and monitors.
There is no longer a default ParseSpec (previously the DelimitedParseSpec). Ingestion specs now require parseSpec
to be specified. If you previously did not provide a parseSpec
, you should use one with "format": "tsv"
to maintain the existing behavior (#6310).
The default value for druid.sql.avatica.maxRowsPerFrame
was reduced from 100k to 5k to minimize out of memory errors (#5409).
As a result of #5595, routers may now select an undesired broker in configurations where there are different tiers of brokers that are intended to be dedicated to queries on different time ranges. See #1362 for discussion.
Timestamps parsed using a TimestampSpec with format 'ruby' no longer truncates the millisecond component. If you were using this parser and wanted a query granularity of SECOND, ensure that it is configured appropriately in your indexing specs (#6217).
The datasource name was added to TaskAnnouncement
which will result in a small per task increase in the amount of data stored in ZooKeeper (#5511).
Aggregators of type 'filtered' now support a 'name' field. Previously, the filtered aggregator inherited the name of the aggregator it wrapped. If you have provided the 'name' field for both the filtered aggregator and the wrapped aggregator, it will prefer the name of the filtered aggregator. It will use the name of the wrapped aggregator if the name of the filtered aggregator is missing or empty (#6219).
For upgrade instructions, use the ALTER DATABASE
and ALTER TABLE
instructions as described here: https://dev.mysql.com/doc/refman/5.7/en/charset-unicode-conversion.html.
For motivation and reference, see #5377 and #5411.
druid.indexer.runner.tlsStartPort
has been removed (#6194).druid.indexer.runner.separateIngestionEndpoint
has been removed (#6263).Packaging has been renamed from io.druid
to org.apache.druid
. All third-party extensions will need to rename their META-INF/io.druid.initialization.DruidModule
to org.apache.druid.initialization.DruidModule
and update their extension's packaging appropriately (#6266).
The DataSegmentPuller
interface has been removed (#5461).
A number of functions under java-util
have been removed (#5461).
The constructor of the Metadata
class has changed (#5613).
The 'spark2' Maven profile has been removed (#5581).
/druid/indexer/v1/supervisor/{id}/shutdown
endpoint has been deprecated in favor of /druid/indexer/v1/supervisor/{id}/terminate
(#6272 and #6234)./druid/indexer/v1/task/{taskId}/segments
endpoint has been deprecated (#6368).status
field returned by /druid/indexer/v1/task/ID/status
has been deprecated in favor of statusCode
(#6334).reportParseExceptions
and ignoreInvalidRows
parameters for ingestion tuning configurations have been deprecated in favor of logParseExceptions
and maxParseExceptions
(#5418)./druid/v2/datasources/{dataSourceName}/dimensions
endpoint has been deprecated. A segment metadata query or the INFORMATION_SCHEMA SQL table should be used instead (#6361)./druid/v2/datasources/{dataSourceName}/metrics
endpoint has been deprecated. A segment metadata query or the INFORMATION_SCHEMA SQL table should be used instead (#6361).Thanks to everyone who contributed to this release!
@a2l007
@adursun
@ak08
@akashdw
@aleksi75
@AlexanderSaydakov
@alperkokmen
@amalakar
@andresgomezfrr
@apollotonkosmo
@asdf2014
@awelsh93
@b-slim
@bolkedebruin
@Caroline1000
@chengchengpei
@clintropolis
@dansuzuki
@dclim
@DiegoEliasCosta
@dragonls
@drcrallen
@dyanarose
@dyf6372
@Dylan1312
@erikdubbelboer
@es1220
@evasomething
@fjy
@Fokko
@gaodayue
@gianm
@hate13
@himanshug
@hoesler
@jaihind213
@jcollado
@jihoonson
@jim-slattery-rs
@jkukul
@jon-wei
@josephglanville
@jsun98
@kaijianding
@KenjiTakahashi
@kevinconaway
@korvit0
@leventov
@lssenthilkumar
@mhshimul
@niketh
@NirajaM
@nishantmonu51
@njhartwell
@palanieppan-m
@pdeva
@pjain1
@QiuMM
@redlion99
@rpless
@samarthjain
@Scorpil
@scrawfor
@shiroari
@shivtools
@siddharths
@SpyderRivera
@spyk
@stuartmclean
@surekhasaharan
@susielu
@varaga
@vineshcpaul
@vvc11
@wysstartgo
@xvrl
@yunwan
@yuppie-flu
@yurmix
@zhangxinyu1
@zhztheplayer
Published by jon-wei about 6 years ago
Druid 0.12.3 contains stability improvements and bug fixes from 6 contributors. Major improvements include:
The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+milestone%3A0.12.3+is%3Aclosed
Documentation for this release is at: http://druid.io/docs/0.12.3
0.12.3 fixes a serious issue where the Kafka indexing service would incorrectly delete published segments in certain situations. Please see #6155, contributed by @gianm, for details.
Other stability improvements include:
0.12.3 includes a memory allocation adjustment for the GroupBy query that should reduce heap usage. Added by @gaodayue in #6256.
This release also contains several fixes for the TopN and GroupBy queries when numeric dimension output types are used. Added by @gianm in #6220.
Additionally, 0.12.3 includes the following Druid SQL bug fixes:
0.12.3 fixes an issue with the druid-basic-security
extension where non-coordinator nodes would sometimes fail when updating their cached views of the authentication and/or authorization tables from the coordinator. Fixed by @gaodayue in #6270.
The full change list can be found here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+milestone%3A0.12.3+is%3Aclosed
0.12.3 is a minor release and compatible with 0.12.2. If you're updating from an earlier version than 0.12.2, please see release notes of the relevant intermediate versions for additional notes.
Thanks to everyone who contributed to this release!
@gaodayue
@gianm
@jihoonson
@jon-wei
@QiuMM
@vogievetsky
@wpcnjupt
Published by jihoonson about 6 years ago
Druid 0.12.2 contains stability improvements and bug fixes from 13 contributors. Major improvements include:
The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+milestone%3A0.12.2+is%3Aclosed
Documentation for this release is at: http://druid.io/docs/0.12.2-rc1
We have fixed a bunch of bugs in Kafka indexing service, which are mostly race conditions when incrementally publishing segments.
Added by @jihoonson in #5805.
Added by @surekhasaharan in #5899.
Added by @surekhasaharan in #5900.
Added by @jihoonson in #5905.
Added by @jihoonson in #5907.
Added by @jihoonson in #5996.
We also have fixed some bugs in general data ingestion logics. Especially we have fixed a bug of wrong segment data when you use auto
encoded long columns with any compression.
Added by @jihoonson in #5932.
Added by @clintropolis in #6045.
Coordinator is now capable of more stable segment management especially for segment balancing. We have fixed an unexpected segment imbalancing caused by the conflicted decisions of Coordinator rule runner and balancer.
Added by @clintropolis in #5528.
Added by @clintropolis in #5529.
Added by @clintropolis in #5532.
Added by @clintropolis in #5555.
Added by @clintropolis in #5591.
Added by @clintropolis in #5888.
Added by @clintropolis in #5928.
We've fixed the wrong lexicographic sort of topN queries and the wrong filter application for the nested queries. The bug of ClassCastException
when caching topN queries with Float dimensions has also fixed.
Added by @drcrallen in #5650.
Added by @gianm in #5653.
The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+milestone%3A0.12.2+is%3Aclosed
0.12.2 is a minor release and compatible with 0.12.1. If you're updating from an earlier version than 0.12.1, please see release notes of the relevant intermediate versions for additional notes.
Thanks to everyone who contributed to this release!
@acdn-ekeddy
@awelsh93
@clintropolis
@drcrallen
@gianm
@jihoonson
@jon-wei
@kaijianding
@leventov
@michas2
@Oooocean
@samarthjain
@surekhasaharan
Published by jihoonson over 6 years ago
Druid 0.12.1 contains stability improvements and bug fixes from 10 contributors. Major improvements include:
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+milestone%3A0.12.1
Documentation for this release is at: http://druid.io/docs/0.12.1
The loadstatus
API of Coordinators returns the percentage of segments actually loaded in the cluster versus segments that should be loaded in the cluster. The performance of this API has greatly been improved.
Added by @jon-wei in #5632.
Druid now can limit the amount of memory used by HttpPostEmitter to 10% of the available JVM heap, thereby avoiding OutOfMemory errors from buffered events.
Added by @jon-wei in #5300.
There were some bugs in Kerberos authentication like authentication failure without cookies or broken authentication when router is used. See https://github.com/druid-io/druid/pull/5596, https://github.com/druid-io/druid/pull/5706, and https://github.com/druid-io/druid/pull/5766 for more details.
Added by @nishantmonu51 in #5596.
Added by @b-slim in #5706.
Added by @jon-wei in #5766.
Coordinators could be stuck if it loses leadership while starting. This bug has been fixed now.
Added by @jihoonson in #5554.
SegmentMetadataQuery is supposed to use the interval of druid.query.segmentMetadata.defaultHistory
if the interval is not specified, but it queried all segments instead which incurs an unexpected performance hit. SegmentMetadataQuery now respects the defaultHistory
option again.
Added by @gianm in #5489.
Druid now supports the HTTP OPTIONS request by fixing its auth handling.
Added by @jon-wei in #5615.
Kafka indexing service allowed retrying tasks to overwrite the segments in deep storage written by the previous failed tasks. However, this caused another bug that the same segment ID could have different data on historicals and in deep storage. This bug has been fixed now by using unique segment paths for each Kafka index tasks.
Added by @dclim in #5692.
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+milestone%3A0.12.1
0.12.1 is a minor release and compatible with 0.12.0. If you're updating from an earlier version than 0.12.0, please see release notes of the relevant intermediate versions for additional notes.
Thanks to everyone who contributed to this release!
@dclim
@gianm
@JeKuOrdina
@jihoonson
@jon-wei
@leventov
@niketh
@nishantmonu51
@pdeva
Published by jon-wei over 6 years ago
Druid 0.12.0 contains over a hundred performance improvements, stability improvements, and bug fixes from almost 40 contributors. This release adds major improvements to the Kafka indexing service.
Other major new features include:
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.12.0
Documentation for this release is at: http://druid.io/docs/0.12.0/
The Kafka indexing service now supports incremental handoffs, as well as decoupling the number of segments created by a Kafka indexing task from the number of Kafka partitions. Please see https://github.com/druid-io/druid/pull/4815#issuecomment-346155552 for more information.
Added by @pjain1 in #4815.
Druid now supports priorities for indexing task locks. When an indexing task needs to acquire a lock on a datasource + interval, higher-priority tasks can now preempt lower-priority tasks. Please see http://druid.io/docs/0.12.0-rc1/ingestion/tasks.html#task-priority for more information.
Added by @jihoonson in https://github.com/druid-io/druid/pull/4550.
Indexing tasks create entries in the "pendingSegments" table in the metadata store; prior to 0.12.0, these temporary entries were not automatically cleaned up, leading to possible cluster performance degradation over time. Druid 0.12.0 allows the coordinator to automatically clean up unused entries in the pending segments table. This feature is enabled by setting druid.coordinator.kill.pendingSegments.on=true
in coordinator properties.
Added by @jihoonson in #5149.
Compacting segments (merging a set of segments within a given interval to create a set with larger but fewer segments) is a common Druid batch ingestion use case. Druid 0.12.0 now supports a Compaction Task that merges all segments within a given interval into a single segment. Please see http://druid.io/docs/0.12.0-rc1/ingestion/tasks.html#compaction-task for more details.
Added by @jihoonson in #4985.
New z-score and p-value test statistics post-aggregators have been added to the druid-stats
extension. Please see http://druid.io/docs/0.12.0-rc1/development/extensions-core/test-stats.html for more details.
Added by @chunghochen in #4532.
A numeric quantiles sketch aggregator has been added to the druid-datasketches
extension.
Added by @AlexanderSaydakov in #5002.
Druid 0.12.0 includes a new authentication/authorization extension that provides Basic HTTP authentication and simple role-based access control. Please see http://druid.io/docs/0.12.0-rc1/development/extensions-core/druid-basic-security.html for more information.
Added by @jon-wei in #5099.
Currently clients can overwhelm a broker inadvertently by sending too many requests which get queued in an unbounded Jetty worker pool queue. Clients typically close the connection after a certain client-side timeout but the broker will continue to process these requests, giving the appearance of being unresponsive. Meanwhile, clients would continue to retry, continuing to add requests to an already overloaded broker..
The newly introduced properties druid.server.http.queueSize
and druid.server.http.enableRequestLimit
in the broker configuration and historical configuration allow users to configure request rejection to prevent clients from overwhelming brokers and historicals with queries.
Added by @himanshug in #4540.
For developers of custom ingestion parsing extensions, it is now possible for InputRowParsers to return multiple InputRows from a single input row. This can simplify ingestion pipelines by reducing the need for input transformations outside of Druid.
Added by @pjain1 in #5081.
When creating new segments, Druid stores some pre-processed data in temporary buffers. Prior to 0.12.0, these buffers were always kept in temporary files on disk. In 0.12.0, PR #4762 by @leventov allows these temporary buffers to be stored in off-heap memory, thus reducing the number of disk I/O operations during ingestion. To enable using off-heap memory for these buffers, the druid.peon.defaultSegmentWriteOutMediumFactory
property needs to be configured accordingly. If using off-heap memory for the temporary buffers, please ensure that -XX:MaxDirectMemorySize
is increased to accommodate the higher direct memory usage.
Please see http://druid.io/docs/0.12.0-rc1/configuration/indexing-service.html#SegmentWriteOutMediumFactory for configuration details.
PR #4704 by @jihoonson allows the user to configure a number of processing threads to be used for parallel merging of intermediate GroupBy results that have been spilled to disk. Prior to 0.12.0, this merging step would always take place within a single thread.
Please see http://druid.io/docs/0.12.0-rc1/configuration/querying/groupbyquery.html#parallel-combine for configuration details.
Various improvements and features have been added to Druid SQL, by @gianm in the following PRs:
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.12.0
Please see below for changes between 0.11.0 and 0.12.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.11.0, please see release notes of the relevant intermediate versions for additional notes.
Please note that after upgrading to 0.12.0, it is no longer possible to downgrade to a version older than 0.11.0, due to changes made in https://github.com/druid-io/druid/pull/4762. It is still possible to roll back to version 0.11.0.
The Metamarkets java-util library has been brought into Druid. As a result, the following package references have changed:
com.metamx.common
-> io.druid.java.util.common
com.metamx.emitter
-> io.druid.java.util.emitter
com.metamx.http
-> io.druid.java.util.http
com.metamx.metrics
-> io.druid.java.util.metrics
This will affect the druid.monitoring.monitors
configuration. References to monitor classes under the old com.metamx.metrics.*
package will need to be updated to reference io.druid.java.metrics.*
instead, e.g. io.druid.java.util.metrics.JvmMonitor
.
If classes under the the com.metamx
packages shown above are referenced in other configurations such as log4j2.xml, those references will need to be updated as well.
Extension developers will need to update their code to use the new Druid packages as well.
The Caffeine cache extension has been moved out of an extension, into core Druid. In addition, the Caffeine cache is now the default cache implementation. Please remove druid-caffeine-cache
if present from the extension list when upgrading to 0.12.0. More information can be found at https://github.com/druid-io/druid/pull/4810.
The semantics of the earlyMessageRejectPeriod
configuration have changed. The earlyMessageRejectPeriod
will now be added to (task start time + task duration)
instead of just (task start time)
when determining the bounds of the message window. Please see https://github.com/druid-io/druid/pull/4990 for more information.
In 0.12.0, there are protocol changes between the Kafka supervisor and Kafka Indexing task and also some changes to the metadata formats persisted on disk. Therefore, to support rolling upgrade, all the Middle Managers will need to be upgraded first before the Overlord. Note that this ordering is different from the standard order of upgrade, also note that this ordering is only necessary when using the Kafka Indexing Service. If one is not using Kafka Indexing Service or can handle down time for Kafka Supervisor then one can upgrade in any order.
Until the point in time Overlord is upgraded, all the Kafka Indexing Task will behave in same manner (even if they are upgraded) as earlier which means no decoupling and incremental hand-offs. Once, Overlord is upgraded, the new tasks started by the upgraded Overlord will support the new features.
Please see https://github.com/druid-io/druid/pull/4815 for more info.
Once both the overlord and middle managers are rolled back, a new set of tasks should be started, which will work properly. However, the current set of tasks may fail during a roll back. Please see #4815 for more info.
The ColumnSelectorFactory
API has changed. Aggregator extension authors and any others who use ColumnSelectorFactory
will need to update their code accordingly. Please see https://github.com/druid-io/druid/pull/4886 for more details.
The Aggregator.reset()
method has been removed because it was deprecated and unused. Please see https://github.com/druid-io/druid/pull/5177 for more info.
The DataSegmentPusher
interface has changed, and the push()
method now has an additional replaceExisting
parameter. Please see https://github.com/druid-io/druid/pull/5187 for details.
The Escalator
interface has changed: the createEscalatedJettyClient
method has been removed. Please see https://github.com/druid-io/druid/pull/5322 for more details.
Thanks to everyone who contributed to this release!
@a2l007
@akashdw
@AlexanderSaydakov
@b-slim
@ben-manes
@benvogan
@chuanlei
@chunghochen
@clintropolis
@daniel-tcell
@dclim
@dpenas
@drcrallen
@egor-ryashin
@elloooooo
@Fokko
@fuji-151a
@gianm
@gvsmirnov
@himanshug
@hzy001
@Igosuki
@jihoonson
@jon-wei
@KenjiTakahashi
@kevinconaway
@leventov
@mh2753
@niketh
@nishantmonu51
@pjain1
@QiuMM
@QubitPi
@quenlang
@Shimi
@skyler-tao
@xanec
@yuppie-flu
@zhangxinyu1
Druid 0.11.0 contains over a hundred performance improvements, stability improvements, and bug fixes from almost 40 contributors. This release adds two major security features, TLS support and extension points for authentication and authorization.
Major new features include:
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.11.0
Documentation for this release is at: http://druid.io/docs/0.11.0/
Druid now supports TLS, enabling encrypted client and inter-node communications. Please see http://druid.io/docs/0.11.0/operations/tls-support.html for details on configuration and related extensions.
Added by @pjain1 in https://github.com/druid-io/druid/pull/4270.
Extension points for authenticating and authorizing requests have been added to Druid. Please see http://druid.io/docs/0.11.0/configuration/auth.html for information on configuration and extension implementation.
The existing Kerberos authentication extension has been updated to implement the new Authenticator interface, please see the "Kerberos configuration changes" section under "Updating from 0.10.1 and earlier" for more information if you are using the Kerberos extension.
Added by @jon-wei in https://github.com/druid-io/druid/pull/4271
Druid now supports Double type aggregator columns. Please see http://druid.io/docs/0.11.0/querying/aggregations.html for documentation on the new Double aggregators.
Added by @b-slim in https://github.com/druid-io/druid/pull/4491.
Users upgrading to 0.11.0 are encouraged to try the new cachingCost
segment balancing strategy on their coordinators. This strategy offers large performance improvements over the existing cost balancer strategy, and it is planned to become the default strategy in the release following 0.11.0.
This strategy can be selected by setting the following property on coordinators:
druid.coordinator.balancer.strategy=cachingCost
Added by @dgolitsyn in https://github.com/druid-io/druid/pull/4731
Druid's JSON input parser now supports jq expressions using jackson-jq, enabling more input transforms before ingestion. Please see http://druid.io/docs/0.11.0/ingestion/flatten-json.html for more details.
Added by @knoguchi in https://github.com/druid-io/druid/pull/4171.
A new cache implementation using Redis has been added in an extension, added by @QiuMM in https://github.com/druid-io/druid/pull/4615. Please refer to the preceding pull request for more details.
Several new performance optimizations have been added to the GroupBy query by @jihoonson in the following PRs:
https://github.com/druid-io/druid/pull/4660 Parallel sort for ConcurrentGrouper
https://github.com/druid-io/druid/pull/4576 Array-based aggregation for groupBy query
https://github.com/druid-io/druid/pull/4668 Add IntGrouper to avoid unnecessary boxing/unboxing in array-based aggregation
PR #4660 offers a general improvement by parallelizing partial result sorting, while PR #4576 and #4668 offer significant improvements when grouping on a single String column.
Various improvements and features have been added to Druid SQL, by @gianm in the following PRs:
https://github.com/druid-io/druid/pull/4750 - TRIM support
https://github.com/druid-io/druid/pull/4720 - Rounding for count distinct
https://github.com/druid-io/druid/pull/4561 - Metrics for SQL queries
https://github.com/druid-io/druid/pull/4360 - SQL expressions support
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.11.0
Please see below for changes between 0.10.1 and 0.11.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.10.1, please see release notes of the relevant intermediate versions for additional notes.
The following patch changes the way coordinator->overlord redirects are handled:
https://github.com/druid-io/druid/pull/5037
The overlord leader election algorithm has changed in 0.11.0: https://github.com/druid-io/druid/pull/4699.
As a result of the two patches above, special care is needed when upgrading Coordinator or Overlord to 0.11.0. All coordinators and overlords must be shut down and upgraded together.
For example, to upgrade Coordinators, you would shutdown all coordinators, upgrade them to 0.11.0 and then start them. Overlords should be upgraded in a similar way.
During the upgrade process, there must not be any time period where a non-0.11.0 coordinator or overlord is running simultaneously with an 0.11.0 coordinator or overlord.
Note that at least one overlord should be brought up as quickly as possible after shutting them all down so that peons, tranquility etc continue to work after some retries.
Also note that the druid.zk.paths.indexer.leaderLatchPath
property is no longer used now.
In earlier versions of Druid, /
characters in service names defined by druid.service
would be replaced by :
characters because these service names were used in Zookeeper paths. Druid 0.11.0 no longer performs these character replacements.
Example:1 - if the old configuration had a broker with service name test/broker
:
druid.service=test/broker
and a Router was configured assuming that /
will be replaced with :
in the broker service name,
druid.router.tierToBrokerMap={"hot":"test:broker","_default_tier":"test:broker"}
the Router configuration should be updated to remove that assumption:
druid.router.tierToBrokerMap={"hot":"test/broker","_default_tier":"test/broker"}
Example:2 - If the old configuration had overlord with service Name test/overlord
then value of druid.coordinator.asOverlord.overlordService
or druid.selectors.indexing.serviceName
should be test/overlord
and not test:overlord
Example:3 - If the old configuration had overlord with service Name test:overlord
then value of druid.coordinator.asOverlord.overlordService
or druid.selectors.indexing.serviceName
should be test:overlord
and not test/overlord
Following service name-related configurations are also affected and should be updated to exactly match the value of druid.service
property on other node being discovered.
druid.coordinator.asOverlord.overlordService
druid.selectors.coordinator.serviceName
druid.selectors.indexing.serviceName
druid.router.defaultBrokerServiceName
druid.router.coordinatorServiceName
druid.router.tierToBrokerMap
Please see https://github.com/druid-io/druid/issues/4992 for more details.
The Kerberos authentication configuration format has changed as a result of the new interfaces introduced by #4271. Please refer to http://druid.io/docs/0.11.0/development/extensions-core/druid-kerberos.html for the new configuration properties.
Users can point the Kerberos authenticator's authorizerName to an instance of an "allowAll" authorizer to replicate the pre-0.11.0 behavior of a cluster using Kerberos authentication with no authorization.
The paths for the lookups configuration API have changed due to #5058.
Configuration paths that had the form /druid/coordinator/v1/lookups
now have the form /druid/coordinator/v1/lookups/config
.
Please see http://druid.io/docs/0.11.0/querying/lookups.html for the current API.
Prior to 0.11.0, the Double* aggregators would store column values on disk as Float while performing aggregations using Double representations.
PR #4491 allows the Double aggregators to store column values on disk as Doubles. Due to concerns related to rolling updates and version downgrades, this behavior is disabled by default and Druid will continue to store Double aggregators on disk as floats.
To enable Double column storage, set the following property in the common runtime properties:
druid.indexing.doubleStorage=double
Users should not set this property during an initial rolling upgrade to 0.11.0, as any nodes running pre-0.11.0 Druid will not be able to handle Double columns created during the upgrade period. Users will also need to reindex any segments with Double columns if downgrading from 0.11.0 to an older version. Please see #4944 and #4605 for more information.
The Scan query has been moved from extensions-contrib to core Druid. As part of this migration: https://github.com/druid-io/druid/pull/4751, the scan query's handling of the time column has changed.
The time column is now is returned as "__time" rather than "timestamp", it is no longer included if you do not specifically ask for it in your "columns", and it is returned as a long rather than a string.
Users can revert the Scan query's time handling to the legacy extension behavior by setting "legacy" : true
in their queries, or setting the property druid.query.scan.legacy = true
. This is meant to provide a migration path for users that were formerly using the contrib extension.
The Aggregator interface has gained a getDouble()
method, which defaults to casting the result of getFloat()
. The getDouble()
method should be re-implemented for any custom aggregators that can support doubles.
See https://github.com/druid-io/druid/pull/4595 for more details.
The QueryRunner interface has changed and the old run() method has been removed, replaced by a new method that accepts a QueryPlus object.
Custom query extensions will need to implement the new interface.
Please see https://github.com/druid-io/druid/pull/4184 and https://github.com/druid-io/druid/pull/4482 for more details.
The Filter.getBitmapResult() method no longer has a default implementation: https://github.com/druid-io/druid/pull/4481
Custom filter extensions will need to provide an implementation for getBitmapResult() now.
The jvm/gc/time
metric is no longer emitted, replaced by a new metric named jvm/gc/cpu
for the reasons described here: https://github.com/druid-io/druid/pull/4480
Please note that the default worker select strategy has changed from fillCapacity
to equalDistribution
. This change was introduced in 0.10.1, the previous release, but was not mentioned in the 0.10.1 release notes, so it is called out again here.
Druid will now always build V9 segments, creating V8 segments is no longer supported and the buildV9Directly
property for ingestion tasks has been removed.
Please see https://github.com/druid-io/druid/pull/4420 for more details.
Please note that the LogLevelAdjuster has been removed: https://github.com/druid-io/druid/pull/4236
Any user using mbeans to configure log levels should configure log4j2 using jmx instead.
Thanks to everyone who contributed to this release!
@a2l007
@akashdw
@andy256
@asifmansoora
@b-slim
@benvogan
@blugowski
@chrisgavin
@dclim
@dgolitsyn
@drcrallen
@egor-ryashin
@erikdubbelboer
@Fokko
@fuji-151a
@gaodayue
@gianm
@ginoledesma
@himanshug
@hzy001
@jihoonson
@jon-wei
@kevinconaway
@knoguchi
@leiwangx
@leventov
@michalmisiewicz
@niketh
@pjain1
@praveev
@QiuMM
@scan-the-automator
@solimant
@SpotXPeterCunningham
@tkyaw
@wywlds
@xanec
@yuusaku-t
@zhangxinyu1
Druid 0.10.1 contains hundreds of performance improvements, stability improvements, and bug fixes from over 40 contributors. Major new features include:
If you are upgrading from a previous version of Druid, please see "Updating from 0.10.0 and earlier" below for upgrade notes, including some backwards incompatible changes.
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.10.1
Documentation for this release is at: http://druid.io/docs/0.10.1/
Processing for TopN queries with 1-2 aggregators on historical nodes is now 2-4 times faster. This is accomplished with new runtime inspection logic that generates monomorphic implementations of query processing classes, reducing polymorphism in the TopN query execution path.
Added in a series of PRs described here by @leventov: https://github.com/druid-io/druid/issues/3798.
Druid can now optimize limit clauses in GroupBy queries by distributing the limit application to historical/realtime nodes, applying the limit to partial result sets before they are sent to the broker for merging. This reduces network traffic within the cluster and reduces the merging workload on broker nodes. Please refer to http://druid.io/docs/0.10.1/querying/groupbyquery.html#query-context for more information.
Added in https://github.com/druid-io/druid/pull/3873 by @jon-wei.
Amazon's S3A filesystem is now supported for deep storage and as an input source for batch ingestion tasks. Please refer to <> for documentation.
Added in https://github.com/druid-io/druid/pull/4116 by @b-slim.
Support for ingesting Protobuf 3.0 data has been added, along with other enhancements such as reading Protobuf descriptors from a URL. Protobuf-supporting code has been moved into its own core extension as well. See http://druid.io/docs/0.10.1/development/extensions-core/protobuf.html for documentation.
Added in https://github.com/druid-io/druid/pull/4039 by @knoguchi.
A new Firehose for realtime ingestion that reads data from a list of URLs via HTTP has been added. Please see http://druid.io/docs/latest/ingestion/firehose.html#httpfirehose for documentation.
Added in https://github.com/druid-io/druid/pull/4297 by @jihoonson.
The Firehose implementations for Microsoft Azure, Rackspace Cloud Files, Google Cloud Storage, and Amazon S3 now support caching and prefetching of data. These firehoses can now operate on portions of the input data and pull new data as needed, instead of having to fully read the firehose's input to disk.
Please refer to the following links for documentation:
http://druid.io/docs/0.10.1/development/extensions-contrib/azure.html
http://druid.io/docs/0.10.1/development/extensions-contrib/cloudfiles.html
http://druid.io/docs/0.10.1/development/extensions-contrib/google.html
http://druid.io/docs/0.10.1/development/extensions-core/s3.html
Added in https://github.com/druid-io/druid/pull/4193 by @jihoonson.
Several enhancements have been made to the state management/synchronization logic for query-time lookups, including versioning of lookup specs. Please see http://druid.io/docs/0.10.1/querying/lookups.html for documentation.
Added in https://github.com/druid-io/druid/pull/3855 by @himanshug.
A new metrics emitter that sends metrics data to Kafka in JSON format has been added. See http://druid.io/docs/0.10.1/development/extensions-contrib/kafka-emitter.html
Added in https://github.com/druid-io/druid/pull/3860 by @dkhwangbo.
A new column comparison filter has been added. This filter allows the user to compare values across columns within a row, like a "WHERE columnA = columnB" clause in SQL. See http://druid.io/docs/0.10.1/querying/filters.html#column-comparison-filter for documentation.
Added in https://github.com/druid-io/druid/pull/3928 by @erikdubbelboer.
Druid 0.10.1 has a number of enhancements to Druid SQL, such as support for lookups (PRs by @gianm):
https://github.com/druid-io/druid/pull/4368 - More forgiving Avatica server
https://github.com/druid-io/druid/pull/4109 - Support for another form of filtered aggregator
https://github.com/druid-io/druid/pull/4085 - Rule to collapse sort chains
https://github.com/druid-io/druid/pull/4055 - Add SQL REGEXP_EXTRACT function
https://github.com/druid-io/druid/pull/3991 - Make row extractions extensible and add one for lookups
https://github.com/druid-io/druid/pull/4028 - Support for coercing to DECIMAL
https://github.com/druid-io/druid/pull/3999 - Ability to generate exact distinct count queries
Druid 0.10.1 has a number of other performance improvements, including:
https://github.com/druid-io/druid/pull/4364 - Uncompress streams without having to download to tmp first, by @niketh
https://github.com/druid-io/druid/pull/4315 - Server selector improvement, by @dgolitsyn
https://github.com/druid-io/druid/pull/4110 - Remove "granularity" from IngestSegmentFirehose, by @gianm
https://github.com/druid-io/druid/pull/4038 - serialize DateTime As Long to improve json serde performance, by @kaijianding
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.10.1
Please see below for changes between 0.10.0 and 0.10.1 that you should be aware of before upgrading. If you're updating from an earlier version than 0.10.0, please see release notes of the relevant intermediate versions for additional notes.
To add support for Amazon's S3A filesystem, Druid is now built against Hadoop 2.7.3 libraries, and we are deprecating support for Hadoop versions older than 2.6.0.
For users running a Hadoop version older than 2.6.0, it is possible to continue running Druid 0.10.1 with the older Hadoop version using a workaround.
The user would need to downgrade hadoop.compile.version
in the main Druid pom.xml, remove the hadoop-aws
dependency from pom.xml in the druid-hdfs-storage
core extension, and then rebuild Druid.
Users are strongly encouraged to upgrade their Hadoop clusters to a 2.6.0+ version as of this release, as support for Hadoop <2.6.0 may be dropped completely in future releases.
If users wish to use Hadoop 2.7.3 as default for ingestion tasks, users should double check any existing druid.indexer.task.defaultHadoopCoordinates
configurations.
Due to changes from https://github.com/druid-io/druid/pull/4115, the Kafka indexing service is no longer compatible with version 0.9.x Kafka brokers. Users will need to upgrade their Kafka brokers to an 0.10.x version.
https://github.com/druid-io/druid/pull/3855 introduces various improvements to coordinator lookup propagation behavior. Please see http://druid.io/docs/0.10.1/querying/lookups.html for details. Note the changes to coordinator HTTP API regarding lookups management.
If Lookups are being used in prior deployment, then as part of upgrade to 0.10.1, All coordinators should be stopped, upgraded, and then started with version 0.10.1 at one time rather than upgrading them one at a time. There should never be a situation where one coordinator is running 0.10.0 while other coordinator is running 0.10.1 at the same time.
During the course of the cluster upgrade, lookup query nodes will report an error starting with got notice to load lookup [LookupExtractorFactoryContainer{version='null'
. This is not actually an error and is a side effect of the update. See https://github.com/druid-io/druid/issues/4603 for details.
Please note that the off-heap query-time lookup cache is broken at this time because of an excessive memory use issue, and must not be used:
https://github.com/druid-io/druid/issues/3663
Please note that the default worker select strategy has changed from fillCapacity
to equalDistribution
.
The standard Druid update process described by http://druid.io/docs/0.10.1/operations/rolling-updates.html should be followed for rolling updates.
Thanks to everyone who contributed to this release!
@akashdw
@amarjayr
@asdf2014
@asrayousuf
@b-slim
@cesure
@cxmcc
@dclim
@dgolitsyn
@dkhwangbo
@drcrallen
@elloooooo
@erikdubbelboer
@fanjieqi
@Fokko
@freakyzoidberg
@fuji-151a
@gianm
@gkc2104
@himanshug
@hzy001
@JackyWoo
@Jdban
@jeffhartley
@jerchung
@jihoonson
@jon-wei
@kaijianding
@KenjiTakahashi
@knoguchi
@leventov
@licl2014
@logarithm
@niketh
@nishantmonu51
@pjain1
@praveev
@ramiyer
@sascha-coenen
@satishbhor
@sixtus
@sjvs
@skyler-tao
@xanec
@zhihuij
@zwang180
Druid 0.10.0 contains hundreds of performance improvements, stability improvements, and bug fixes from over 40 contributors. Major new features include a built-in SQL layer, numeric dimensions, Kerberos authentication support, a revamp of the "index" task, a new "like" filter, large columns, ability to run the coordinator and overlord as a single service, better performing defaults, and eight new extensions.
If you are upgrading from a previous version of Druid, please see "Updating from 0.9.2 and earlier" below for upgrade notes, including some backwards incompatible changes.
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.10.0
Documentation for this release is at: http://druid.io/docs/0.10.0/
Druid now includes a built-in SQL server powered by Apache Calcite. Druid provides two SQL APIs: HTTP POST and JDBC. This provides an alternative to Druid's native JSON API which is more familiar to new developers, and which makes it easier to integrate pre-existing applications that natively speak SQL. Not all Druid and SQL features are supported by the SQL layer in this initial release, but we intend to expand both in future releases.
SQL support can be enabled by setting druid.sql.enable=true
in your configuration. See http://druid.io/docs/0.10.0/querying/sql.html for details and documentation.
Added in #3682 by @gianm.
Druid now supports numeric dimensions at ingestion and query time. Users can ingest long and float columns as dimensions (i.e., treating the numeric columns as part of the ingestion-time grouping key instead of as aggregators, if rollup is enabled). Additionally, Druid queries can now accept any long or float column as a dimension for grouping or for filtering.
There are performance tradeoffs between string and numeric columns. Numeric columns are generally faster to group on than string columns. Numeric columns don't have indexes, so they are generally slower to filter on than string columns.
See http://druid.io/docs/0.10.0/ingestion/schema-design.html#numeric-dimensions for ingestion documentation and http://druid.io/docs/0.10.0/querying/dimensionspecs.html for query documentation.
Added in #3838, #3966, and other patches by @jon-wei.
Added a new extension named 'druid-kerberos' which adds support for User Authentication for Druid Nodes using Kerberos. It uses the simple and protected GSSAPI negotiation mechanism, SPNEGO(https://en.wikipedia.org/wiki/SPNEGO) for authentication via HTTP.
See http://druid.io/docs/0.10.0/development/extensions-core/druid-kerberos.html for documentation on how to configure kerberos authentication.
Added in #3853 by @nishantmonu51.
The indexing task was re-written to improve performance, particularly for jobs spanning multiple intervals that generated many shards. The segmentGranularity intervals
can now be automatically determined and no longer needs to be specified, but ingestion time can be reduced if both intervals
and numShards
are provided.
Additionally, the indexing task now supports an appendToExisting
flag which causes the data to be indexed as an additional shard of the current version rather than as a new version overshadowing the previous version.
See http://druid.io/docs/0.10.0/ingestion/tasks.html#index-task for documentation.
Added in #3611 by @dclim.
Druid now includes a "like" filter that enables SQL LIKE-style filtering, such as foo LIKE 'bar%'
. The implementation is generally faster than regex filters, and is encouraged over regex filters when possible. In particular, like filters on prefixes such as bar%
are significantly faster than equivalent regex filters such as ^bar.*
.
See http://druid.io/docs/0.10.0/querying/filters.html#like-filter for documentation.
Added in #3642 by @gianm.
Druid now supports individual columns larger than 2GB. This feature is not typically required, since general guidance is that segments should generally be 500MB–1GB in size, but is useful in situations where one column is much larger than all the others (for example, large sketches).
This functionality is available to all Druid users and no special configuration is necessary when using the built-in column types. If you have developed a custom metric column type as a Druid extension, you can enable large column support by overriding getSerializer
in your ComplexMetricsSerde
.
Added in #3743 by @akashdw.
Druid deployments can now be simplified by combining the Coordinator and Overlord functions into the Coordinator process. To do this, set druid.coordinator.asOverlord.enabled
and druid.coordinator.asOverlord.overlordService
appropriately on your Coordinators and then stop your Overlords.
Overlord console would be available on http://coordinator-host:port/console.html
.
This is currently an experimental feature and is off by default. We intend to consider making this the default in a future version of Druid.
See http://druid.io/docs/0.10.0/configuration/coordinator.html for documentation on this feature and configuration options.
Added in #3711 by @himanshug.
This release changes two default settings to improve out-of-the-box performance:
The buildV9Directly option introduced in Druid 0.9.0 is now enabled by default. This option improves performance of indexing by creating the v9 data format directly rather than creating v8 first and then converting to v9. If necessary, you can roll back to the old code by setting "buildV9Directly" to false in your indexing tasks.
The v2 groupBy engine introduced in Druid 0.9.2 is now enabled by default. This new groupBy engine was rewritten from the ground up for better performance and memory management. If necessary, you can roll back to the old engine by setting either "druid.groupBy.query.defaultStrategy" in your runtime.properties, or "groupByStrategy" in your query context, to "v1". See http://druid.io/docs/0.10.0/querying/groupbyquery.html for details on the differences between groupBy v1 and v2.
In addition to better performing defaults, Druid 0.10.0 has a number of other performance improvements, including:
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.10.0
Please see below for changes between 0.9.2 and 0.10.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.9.2, please see release notes of the relevant intermediate versions for additional notes.
The standard Druid update process described by http://druid.io/docs/0.10.0/operations/rolling-updates.html should be followed for rolling updates.
Please note the following backwards-incompatible query API changes when updating. Some queries may need to be adjusted to continue to behave as expected.
JavaScript query features are now disabled by default for security reasons (#3818). If you use these features, you can re-enable them by setting druid.javascript.enabled=true
in your runtime properties. See http://druid.io/docs/0.10.0/development/javascript.html for details, including security considerations.
GroupBy queries no longer allow __time
as the output name of a dimension, aggregator, or post-aggregator (#3967).
Select query pagingSpecs now default to fromNext: true
behavior when fromNext
is not specified (#3986). Behavior is unchanged for Select queries that did have fromNext
specified. If you prefer the old default, then you can change this through the druid.query.select.enableFromNextDefault
runtime property. See http://druid.io/docs/0.10.0/querying/select-query.html for details.
SegmentMetadata queries no longer include "size" analysis by default (#3773). You can still request "size" analysis by adding "size" to "analysisTypes" at query time.
Please note the following deployment-related changes when updating.
Druid now requires Java 8 to run (#3914). If you are currently running on Java 7, we suggest upgrading Java first and then Druid.
Druid now defaults to the "v2" engine for groupBy rather than the legacy "v1" engine. As part of this, memory usage limits have changed from row-based to byte-based limits, so it is possible that some queries which met resource limits before will now exceed them and fail. You can avoid this by tuning the new groupBy engine appropriately. If necessary, you can roll back to the old engine by setting either "druid.groupBy.query.defaultStrategy" in your runtime.properties, or "groupByStrategy" in your query context, to "v1". See http://druid.io/docs/0.10.0/querying/groupbyquery.html for more details on the differences between groupBy v1 and v2.This new groupBy engine was rewritten from the ground up for better performance and memory management. The query API and results are compatible between the two engines; however, there are some differences from a cluster configuration perspective: (1) groupBy v2 uses primarily off-heap memory instead of on-heap memory; (2) it requires one merge buffer for each concurrent query; (3) if you've configured caching for groupBy, it must be on historicals; and (4) it doesn't support chunkPeriod.
Druid now has a non-zero default value for druid.processing.numMergeBuffers
(#3953). This will increase the amount of direct memory used in its default configuration. This change was made in connection with changing the default groupBy engine, since the new "v2" engine requires merge buffers. If you were formerly using groupBy v1, you should be able to offset this by reducing your JVM heap, since groupBy v2 uses off-heap memory rather than on-heap memory.
Druid query-related processes (broker, historical, indexing tasks) now eagerly allocate druid.processing.numThreads
number of DirectByteBuffer instances of size druid.processing.buffer.sizeBytes
for query processing at startup (#3628). This allocation was lazy in earlier versions of Druid. This change does not affect memory use of a long-running Druid cluster, but it means that memory for processing buffers must be available when Druid starts up.
When running in non-UTC timezones, behavior of predefined segmentGranularity constants such as "day" will change when upgrading. Formerly, this would generate segments using local days. In Druid 0.10.0, this will generate segments using UTC days. If you were running Druid in the UTC timezone, this change has no effect. Please note that running Druid in a non-UTC timezone is not officially supported (see http://druid.io/docs/0.10.0/configuration/index.html) and we recommend always running all processes in UTC timezones. To create local-day segments using Druid 0.10.0, you can use a local-time segmentGranularity such as:
"segmentGranularity" : {
"type" : "period",
"period" : "P1D",
"timeZone" : "America/Los_Angeles"
}
AggregatorFactory, BufferAggregator, and PostAggregator interfaces have changed (#3894, #3899, #3957, #4071). If you have deployed a custom extension that includes an AggregatorFactory or PostAggregator, it will need to be recompiled. Druid's built-in extensions have all been updated for this change.
Extensions targeting Druid 0.10.x must be compiled with JDK 8.
Thanks to everyone who contributed to this release!
@akashdw
@Aveplatter
@b-slim
@baruchz
@cheddar
@clintropolis
@DaimonPl
@dclim
@dpenas
@drcrallen
@du00cs
@erikdubbelboer
@Fokko
@freakyzoidberg
@gabrielpage
@gianm
@gvsmirnov
@himanshug
@hland
@hzy001
@jaehc
@jihoonson
@jisookim0513
@jkukul
@joanvr
@jon-wei
@kaijianding
@leventov
@mark1900
@michaelschiff
@navis
@ncolomer
@niketh
@nishantmonu51
@pjain1
@praveev
@sirpkt
@tranv94
@xiaoyao1991
@yuusaku-t
@zhxiaogg
Druid 0.9.2 contains hundreds of performance improvements, stability improvements, and bug fixes from over 30 contributors. Major new features include a new groupBy engine, ability to disable rollup at ingestion time, ability to filter on longs, new encoding options for long-typed columns, performance improvements for HyperUnique and DataSketches, a query cache implementation based on Caffeine, a new lookup extension exposing fine grained caching strategies, support for reading ORC files, and new aggregators for variance and standard deviation.
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.9.2
Documentation for this release is here: http://druid.io/docs/0.9.2/
Druid now includes a new groupBy engine, rewritten from the ground up for better performance and memory management. Benchmarks show a 2–5x performance boost on our test datasets. The new engine also supports strict limits on memory usage and the option to spill to disk when memory is exhausted, avoiding result row count limitations and potential OOMEs generated by the previous engine.
The new engine is off by default, but you can enable it through configuration or query context parameters. We intend to enable it by default in a future version of Druid.
See "implementation details" on http://druid.io/docs/0.9.2/querying/groupbyquery.html#implementation-details for documentation and configuration.
Added in #2998 by @gianm.
Since its inception, Druid has had a concept of "dimensions" and "metrics" that applied both at ingestion time and at query time. Druid is unique in that it is one of the only databases that supports aggregation at data loading time, which we call "rollup". But, for some use cases, ingestion-time rollup is not desired, and it's better to load the original data as-is. With rollup disabled, one row in Druid will be created for each input row.
Query-time aggregation is, of course, still supported through the groupBy, topN, and timeseries queries.
See the "rollup" flag on http://druid.io/docs/0.9.2/ingestion/index.html for documentation. By default, rollup remains enabled.
Added in #3020 by @kaijianding.
Druid now supports sophisticated filtering on integer-typed columns, including long metrics and the special __time column. This opens up a number of new capabilities:
Druid does not yet support grouping on longs. We intend to add this capability in a future release.
Added in #3180 by @jon-wei.
Until now, all integer-typed columns in Druid, including long metrics and the special __time column, were stored as 64-bit longs optionally compressed in blocks with LZ4. Druid 0.9.2 adds new encoding options which, in many cases, can reduce file sizes and improve performance:
The default remains "longs" encoding + "lz4" compression. In our testing, two options that often yield useful benefits are "auto" + "lz4" (generally smaller than longs + lz4) and "auto" + "none" (generally faster than longs + lz4, file size impact varies). See the PR for full test results.
See "metricCompression" and "longEncoding" on http://druid.io/docs/0.9.2/ingestion/batch-ingestion.html for documentation.
Added in #3148 by @acslk.
The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.9.2
The standard Druid update process described by http://druid.io/docs/0.9.2/operations/rolling-updates.html should be followed for rolling updates.
The druid-namespace-lookup extension, which was deprecated in 0.9.1 in favor of druid-lookups-cached-global, has been removed in 0.9.2. If you are using druid-namespace-lookup, migrate to druid-lookups-cached-global before upgrading to 0.9.2. See our migration guide for details: http://druid.io/docs/0.9.1.1/development/extensions-core/namespaced-lookup.html#transitioning-to-lookups-cached-global
Please note the following changes:
mapreduce.job.classloader
or mapreduce.job.user.classpath.first
options. In testing we have found this to be an effective workaround. See http://druid.io/docs/0.9.2/operations/other-hadoop.html for details.chatThreads
in the supervisor tuning configuration to a value greater than the number of running tasks as a workaround.Thanks to everyone who contributed to this release!
@acslk
@AlexanderSaydakov
@ashishawasthi
@b-slim
@chtefi
@dclim
@drcrallen
@du00cs
@ecesena
@erikdubbelboer
@fjy
@Fokko
@gianm
@giaosudau
@guobingkun
@gvsmirnov
@hamlet-lee
@himanshug
@HyukjinKwon
@jaehc
@jianran
@jon-wei
@kaijianding
@leventov
@linbojin
@michaelschiff
@navis
@nishantmonu51
@pjain1
@rajk-tetration
@SainathB
@sirpkt
@vogievetsky
@xvrl
@yuppie-flu