What Changed

This release comes with several features, SQL /UI/Perf enhancements Bugfixes across areas ranging from Multistage Query Engine to Ingestion, Storage format, SQL support, etc.

Multi-stage Query Engine

Features

Support RelDistribution-based trait Planning (#11976, #12079)

• Adds support for RelDistribution optimization for more accurate leaf-stage direct exchange/shuffle. Also extends partition optimization beyond leaf stage to entire query plan.
• Applies optimization based on distribution trait in the mailbox/worker assignment stage
  • Fixes previous direct exchange which was decided based on the table partition hint. Now direct exchange is decided via distribution trait: it will applied if-and-only-if the trait propagated matches the exchange requirement.
  • As a side effect, is_colocated_by_join_keysquery option is reintroduced to ensure dynamic broadcast which can also benefit from direct exchange optimization
  • Allows propagation of partition distribution trait info across the tree to be used during Physical Planning phase. It can be used in the following scenarios (will follow up in separate PRs)
• Note on backward incompatbility
  • is_colocated_by_join_keys hint is now required for making colocated joins
    • it should only affect semi-join b/c it is the only one utilizing broadcast exchange but were pulled to act as direct exchange.
    • inner/left/right/full join should automatically apply colocation thus the backward incompatibility should not affect these.

Leaf stage planning with multi-semi join support (#11937)

• Solves the limitation of pinotQuery that supports limited amount of PlanNodes.
• Splits the ServerRequest planning into 2 stages
  • First plan as much as possible into PinotQuery
  • for any remainder nodes that cannot be planned into PinotQuery, will be run together with the LeafStageTransferrableBlockOperator as the input locally.

Support for ArrayAgg aggregation function (#11822)

• Usage: ArrayAgg(column, 'dataType' [, 'isDistinct'])
• Float type column is treated as Double in the multistage engine, so FLOAT type is not supported.
• Supports data BOOLEAN, INT, LONG, FLOAT(only in V1), DOUBLE, STRING, TIMESTAMP.
  E.g. ArrayAgg(intCol, 'INT') returns ARRAY<INT>

Enhancements

• Canonicalize SqlKind.OTHERS and SqlKind.OTHER_FUNCTIONS and support

  concat as || operator (#12025)

• Capability for constant filter in QueryContext, with support for server to handle it (#11956)

• Tests for filter pushdown (#11994)

• Enhancements to query plan tests (#11966)

• Refactor PlanFragmenter to make the logic clear (#11912)

• Observability enhancements to emit metrics for grpc request and multi-stage leaf stage (#11838)

  • pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)
  • pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)

• Security enhancement to add RBAC authorization checks for multi-stage query engine (#11830)

• Enhancement to leaf-stage execution stats NPE handling (#11805)

• Enhancement to add a framework to back-propagate metadata across opChains (#11746)

• Use of BinaryArray to wire proto for multi-stage engine bytes literal handling (#11738)

• Enable dynamic broadcast for SEMI joins. Adds a fallback option to enable hash table join using joinOptions(join_strategy = 'hash_table')(#11696)

• Improvements to dispatch exception handling (#11688)

• Allow malformed dateTime string to return default value configurable in the function signature (#11258)

  • fromDateTime(colContainsMalformedStr, '<dateTimeFormat>', '<timezone>', <default_value>)
    

• Improvement in multi-stage aggregation to directly store column index as identifier (#11617)

• Perf optimization to avoid unnecessary rows conversion in aggregation (#11607)

• Enhance SegmentPartitionMetadataManager to handle new segment (#11585)

• Optimize mailbox info in query plan to reduce memory footprint (#12382)

  • This PR changes the proto object structure, which will cause backward incompatibility when broker and server are running different version.

• Optimizations to query plan serialization (#12370)

• Optimization for parallel execution of Ser/de stage plan (#12363)

• Optimizations in query dispatch (#12358)

• Perf optimization for group-by and join for single key scenario (#11630)

Bugfixes, Refactoring, Cleanups, Tests

• Bugfix for evaluation of chained literal functions (#12248)
• Fixes to sort copy rule (#12251 and #12237)
• Fixes duplicate results for literal queries (#12240)
• Bugfix to use UTF-8 encoding for default Charset (#12213)
• Bugfix to escape table name when routing queries (#12212)
• Refactoring of planner code and removing unnecessary rules (#12070, #12052)
• Fix to remove unnecessar project after agg during relBuilder (#12058)
• Fixes issues multi-semi-join (#12038)
• Fixes leaf limit refactor issue (#12001)
• Add back filter merge after rule (#11989)
• Fix operator EOS pull (#11970)
• Fix type cast issue with dateTimeConvert scalar function (#11839, #11971)
• Fix to set explicit warning flags set on each stage stats (#11936)
• Fix mailbox visitor mismatch receive/send (#11908)
• Fix eliminate multiple exchanges in nested semi-join queries (#11882)
• Bugfix for multiple consecutive Exchange returning empty response (#11885)
• Fixing unit-test-2 build (#11889)
• Fix issue with realtime partition mismatch metric (#11871)
• Fix the NPE for rebalance retry (#11883)
• Bugfix to make Agg literal attach happen after BASIC_RULES (#11863)
• Fix NPE by init execution stats map (#11801)
• Test cases for special column escape (#11737)
• Fix StPoint scalar function usage in multi-stage engine intermediate stage (#11731)
• Clean up for transform function type (#11726)
• Add capability to ignore test (#11703)
• Fix custom property naming (#11675)
• Log warning when multi-stage engine planning throws exception (#11595)
• Fix usage of metadata overrides (#11587)
• Test change to enable metadata manager by default for colocated join quickstart (#11579)
• Tests for IN/NOT-IN operation (#12349)
• Fix stage id in stage plan (#12366)
• Bugfix for IN and NOT IN filters within case statements (#12305)

Notable Features

Server-level throttling for realtime consumption (#12292)

• Use server config pinot.server.consumption.rate.limit to enable this feature
• Server rate limiter is disabled by default (default value 0)

Reduce segment generation disk footprint for Minion Tasks (#12220)

• Supported in MergeRollupTask and RealtimeToOfflineSegmentsTask minion tasks
• Use taskConfig segmentMapperFileSizeThresholdInBytes to specify the threshold size

"task": {
  "taskTypeConfigsMap": {
    "<task_name>": {
      "segmentMapperFileSizeThresholdInBytes": "1000000000"
    }
  }
}

Support for swapping of TLS keystore/truststore (#12277, #12325)

• Security feature that makes the keystore/truststore swappable.
• Auto-reloads keystore/truststore (without need for a restart) if they are local files

Sticky Query Routing (#12276)

• Adds support for deterministic and sticky routing for a query / table / broker. This setting would lead to same server / set of servers (for MultiStageReplicaGroupSelector) being used for all queries of a given table.

• Query option (takes precedence over fixed routing setting at table / broker config level)
  SET "useFixedReplica"=true;

• Table config (takes precedence over fixed routing setting at broker config level)

  "routing": {
     ...          
     "useFixedReplica": true
  }
  

• Broker conf - pinot.broker.use.fixed.replica=true

Table Config to disallow duplicate primary key for dimension tables (#12290)

• Use tableConfig dimensionTableConfig.errorOnDuplicatePrimaryKey=true to enable this behavior
• Disabled by default

Partition-Level ForceCommit for realtime tables (#12088)

• Support to force-commit specific partitions of a realtime table.
• Partitions can be specified to the forceCommit API as a comma separated list of partition names or consuming segment names

Support initializing broker tags from config (#12175)

• Support to give the broker initial tags on startup.
• Automatically updates brokerResource when broker joins the cluster for the first time
• Broker tags are provided as comma-separated values in pinot.broker.instance.tags

Support for StreamNative OAuth2 authentication for pulsar (#12068)

• StreamNative (the cloud SAAS offering of Pulsar) uses OAuth2 to authenticate clients to their Pulsar clusters.
• For more information, see how to Configure OAuth2 authentication in Pulsar clients
• Can be configured by adding the following properties to streamConfigs:

"stream.pulsar.issuerUrl": "https://auth.streamnative.cloud"
"stream.pulsar.credsFilePath": "file:///path/to/private_creds_file
"stream.pulsar.audience": "urn:sn:pulsar:test:test-cluster"

Introduce low disk mode to table rebalance (#12072)

• Introduces a new table rebalance boolean config lowDiskMode.Default value is false.
• Applicable for rebalance with downtime=false.
• When enabled, segments will first be offloaded from servers, then added to servers after offload is done. It may increase the total time of the rebalance, but can be useful when servers are low on disk space, and we want to scale up the cluster and rebalance the table to more servers.
• #12112 adds the UI capability to toggle this option

Support Vector index and Hierarchical Navigable Small Worlds (HNSW) (#11977)

• Supports Vector Index on float array/multi-value columnz
• Add predicate and function to retrieve topK closest vector. Example query

SELECT ProductId, UserId, l2_distance(embedding, ARRAY[-0.0013143676,-0.011042999,...]) AS l2_dist, n_tokens, combined
FROM fineFoodReviews
WHERE VECTOR_SIMILARITY(embedding, ARRAY[-0.0013143676,-0.011042999,...], 5)  
ORDER by l2_dist ASC 
LIMIT 10

• The function VectorSimilarity will return a double value where the first parameter is the embedding column and the second parameter is the search term embedding literal.
• Since VectorSimilarity is a predicate, once config the topK, this predicate will return topk rows per segment. Then if you are using this index with other predicate, you may not get expected number of rows since the records matching other predicate might not in the topk rows.

Support for retention on deleted keys of upsert tables (#12037)

• Adds an upsert config deletedKeysTTL which will remove deleted keys from in-memory hashmap and mark the validDocID as invalid after the deletedKeysTTL threshold period.
• Disabled by default. Enabled only if a valid value for deletedKeysTTL is set
• More details in the design document

Configurable Lucene analyzer (#12027)

• Introduces the capability to specify a custom Lucene analyzer used by text index for indexing and search on an individual column basis.
• Sample usage

fieldConfigList: [
   {
        "name": "columnName",
        "indexType": "TEXT",
        "indexTypes": [
          "TEXT"
        ],
        "properties": {
          "luceneAnalyzerClass": "org.apache.lucene.analysis.core.KeywordAnalyzer"
        },
      }
  ]

• Default Behavior falls back to using the standardAnalyzer unless the luceneAnalyzerClass property is specified.

Support for murmur3 as a partition function (#12049)

• Murmur3 support with optional fields seed and variant for the hash in functionConfig field of columnPartitionMap.Default value for seed is 0.

• Added support for 2 variants of Murmur3: x86_32 and x64_32 configurable using the variant field in functionConfig. If no variant is provided we choose to keep the x86_32 variant as it was part of the original implementation.

• Examples of functionConfig;

  "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3 
            },
            ..
          }
        }
  

  Here there is no functionConfig configured, so the seed value will be 0 and variant will be x86_32.

  "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3,
              "functionConfig": {
                 "seed": "9001"
               },
            },
            ..
          }
        }
  

  Here the seed is configured as 9001 but as no variant is provided, x86_32 will be picked up.

   "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3,
              "functionConfig" :{
                 "seed": "9001"
                 "variant": "x64_32"
               },
            },
            ..
          }
        }
  

  Here the variant is mentioned so Murmur3 will use the x64_32 variant with 9001 as seed.

• Note on users using Debezium and Murmur3 as partitioning function :

  • The partitioning key should be set up on either of byte[], String or long[] columns.
  • On pinot variant should be set as x64_32 and seed should be set as 9001.

New optimized MV forward index to only store unique MV values

• Adds new MV dictionary encoded forward index format that only stores the unique MV entries.

• This new index format can significantly reduce the index size when the MV entries repeat a lot

• The new index format can be enabled during index creation, derived column creation, and segment reload

• To enable the new index format, set the compression codec in the FieldConfig:

  {
    "name": "myCol",
    "encodingType": "DICTIONARY",
    "compressionCodec": "MV_ENTRY_DICT"
  }
  

  Or use the new index JSON:

  {
    "name": "myCol",
    "encodingType": "DICTIONARY",
    "indexes": {
      "forward": {
        "dictIdCompressionType": "MV_ENTRY_DICT"
      }
    }
  }
  

Support for explicit null handling modes (#11960)

• Adds support for 2 possible ways to handle null:
  • Table mode - which already exists
  • Column mode, which means that each column specifies its own nullability in the FieldSpec
• Column mode can be enabled by the below config.
• The default value for enableColumnBasedNullHandling is false. When set to true, Pinot will ignore TableConfig.IndexingConfig.nullHandlingEnabled and columns will be nullable if and only if FieldSpec.notNull is false, which is also the default value.

{
  "schemaName": "blablabla",
  "dimensionFieldSpecs": [
    {
      "dataType": "INT",
      "name": "nullableField",
      "notNull": false
    },
    {
      "dataType": "INT",
      "name": "notNullableField",
      "notNull": true
    },
    {
      "dataType": "INT",
      "name": "defaultNullableField"
    },
    ...
  ],
  "enableColumnBasedNullHandling": true/false
}

Support tracking out of order events in Upsert (#11877)

• Adds a new upsert config outOfOrderRecordColumn
• When set to a non-null value, we check whether an event is OOO or not and then accordingly update the corresponding column value to true / false.
• This will help in tracking which event is out-of-order while using skipUpsert

Compression configuration support for aggregationConfigs to StartreeIndexConfigs (#11744)

• Can be used to save space. For eg: when a functionColumnPairs has a output type of bytes, such as when you use distinctcountrawhll.
• Sample config

"starTreeIndexConfigs": [
        {
          "dimensionsSplitOrder": [
            "a",
            "b",
            "c"
          ],
          "skipStarNodeCreationForDimensions": [],
          "functionColumnPairs": [],
          "aggregationConfigs": [
            {
              "columnName": "column1",
              "aggregationFunction": "SUM",
              "compressionCodec": "SNAPPY"
            },
            {
              "columnName": "column2",
              "aggregationFunction": "distinctcounthll",
              "compressionCodec": "LZ4"
            }
          ],
          "maxLeafRecords": 10000
        }
      ]

Preconfiguration based mirror instance assignment (#11578)

• Supports instance assignment based pre-configured instance assignment map.
• The assignment will always respect the mirrored servers in the pre-configured map
• More details here
• Sample table config

"instanceAssignmentConfigMap": {
  "CONSUMING": {
    "partitionSelector": "MIRROR_SERVER_SET_PARTITION_SELECTOR",
    "replicaGroupPartitionConfig": { ... },
     "tagPoolConfig": {
       ...
       "tag": "mt1_REALTIME"
     }
     ...
 }
 "COMPLETED": {
   "partitionSelector": "MIRROR_SERVER_SET_PARTITION_SELECTOR",
   "replicaGroupPartitionConfig": { ... },
    "tagPoolConfig": {
       ...
       "tag": "mt1_OFFLINE"
     }
     ...
 },
 "instancePartitionsMap": {
      "CONSUMING": “mt1_CONSUMING"
      "COMPLETED": "mt1_OFFLINE"
 },

Support for Listing Dimension Tables (#11859)

• Adds dimension as a valid option to table "type" in the /tables controller API

Support in upsert for dropping out of order events (#11811)

• This patch adds a new config for upsert: dropOutOfOrderRecord
• If set to true, pinot doesn't persist out-of-order events in the segment.
• This feature is useful to
  • Save disk-usage
  • Avoid any confusion when using skipUpsert for partial-upsert tables as nulls start showing up for columns where a previous non-null was encountered and we don't know if it's an out-of-order event or not.

Support to retry failed table rebalance tasks (#11740)

• New configs for the RebalanceChecker periodic task:
  • controller.rebalance.checker.frequencyPeriod: 5min by default ; -1 to disable
  • controller.rebalanceChecker.initialDelayInSeconds: 2min+ by default
• New configs added for RebalanceConfig:
  • heartbeatIntervalInMs: 300_000 i.e. 5min
  • heartbeatTimeoutInMs: 3600_000 i.e. 1hr
  • maxAttempts: 3 by default, i.e. the original run plus two retries
  • retryInitialDelayInMs: 300_000 i.e. 5min, for exponential backoff w/ jitters
• New metrics to monitor rebalance and its retries:
  • TABLE_REBALANCE_FAILURE("TableRebalanceFailure", false), emit from TableRebalancer.rebalanceTable()
  • TABLE_REBALANCE_EXECUTION_TIME_MS("tableRebalanceExecutionTimeMs", false), emit from TableRebalancer.rebalanceTable()
  • TABLE_REBALANCE_FAILURE_DETECTED("TableRebalanceFailureDetected", false), emit from RebalanceChecker
  • TABLE_REBALANCE_RETRY("TableRebalanceRetry", false), emit from RebalanceChecker
• New restful API
  • DELETE /tables/{tableName}/rebalance API to stop rebalance. In comparison, POST /tables/{tableName}/rebalance was used to start one.

Support for UltraLogLog (#11835)

• UltraLogLog aggregations for Count Distinct (distinctCountULL and distinctCountRawULL)
• UltraLogLog creation via Transform Function
• UltraLogLog merging in MergeRollup
• Support for UltraLogLog in Star-Tree indexes

Support for Apache Datasketches CPC sketch (#11774)

• Ingestion via transformation function
• Extracting estimates via query aggregation functions
• Segment rollup aggregation
• StarTree aggregation

Support to reduce DirectMemory OOM chances on broker (#11710)

• Broadly there are two configs that will enable this feature:

  • maxServerResponseSizeBytes: Maximum serialized response size across all servers for a query. This value is equally divided across all servers processing the query.
  • maxQueryResponseSizeBytes: Maximum length of the serialized response per server for a query

• Configs are available as queryOption, tableConfig and Broker config. The priority of enforcement is as follows:

  The overriding order of priority is:
  1. QueryOption  -> maxServerResponseSizeBytes
  2. QueryOption  -> maxQueryResponseSizeBytes
  3. TableConfig  -> maxServerResponseSizeBytes
  4. TableConfig  -> maxQueryResponseSizeBytes
  5. BrokerConfig -> pinot.broker.max.server.response.size.bytes
  6. BrokerConfig -> pinot.broker.max.query.response.size.bytes
  

UI Support to Allow schema to be created with JSON config (#11809)

• This is helpful when user has the entire JSON handy
• UI still keeps Form Way to add Schema along with JSON view

Support in JSON index for ignoring values longer than a given length (#11604)

• Use option maxValueLength in jsonIndexConfig to restrict length of values
• A value of 0 (or when the key is omitted) means there is no restriction

Support for MultiValue VarByte V4 index writer (#11674)

• Supports serializing and writing MV columns in VarByteChunkForwardIndexWriterV4
• Supports V4 reader that can be used to read SV var length, MV fixed length and MV var length buffers encoded with V4 writer

Improved scalar function support for Multivalue columns(#11555, #11654)

arrayIndexOfInt(int[] value, int valToFind)
arrayIndexOfLong(int[] value, long valToFind)
arrayIndexOfFloat(int[] value, float valToFind)
arrayIndexOfDouble(int[] value, double valToFind)
arrayIndexOfString(int[] value, String valToFind)
intersectIndices(int[] values1, int[] values2)

Support for FrequentStringsSketch and FrequentLonsSketch aggregation functions (#11098)

• Approximation aggregation functions for estimating the frequencies of items a dataset in a memory efficient way. More details in Apache Datasketches library.

FREQUENTLONGSSKETCH(col, maxMapSize=256) -> Base64 encoded sketch object
FREQUENTSTRINGSSKETCH(col, maxMapSize=256) -> Base64 encoded sketch object

Controller API for Table Indexe (#11576)

• Table index api to get the aggregate index details of all segments for a table.

  • URL/tables/{tableName}/indexes

• Response format

  {
      "totalSegments": 31,
      "columnToIndexesCount":
      {
          "col1":
          {
              "dictionary": 31,
              "bloom": 0,
              "null": 0,
              "forward": 31,
              ...
              "inverted": 0,
              "some-dynamically-injected-index-type": 31,
          },
          "col2":
          {
              ...
          }
          ...
  }
  

Support for configurable rebalance delay at lead controller (#11509)

• The lead controller rebalance delay is now configurable with controller.resource.rebalance.delay_ms
• Changing rebalance configurations will now update the lead controller resource

Support for configuration through environment variables (#12307)

• Adds support for Pinot configuration through ENV variables with Dynamic mapping.
• More details in issue: #10651
• Sample configs through ENV

export PINOT_CONTROLLER_HOST=host
export PINOT_SERVER_PROPERTY_WHATEVER=whatever_property
export ANOTHER_VARIABLE=random

Add hyperLogLogPlus aggregation function for distinct count (#11346)

• HLL++ has higher accuracy than HLL when cardinality of dimension is at 10k-100k.
• More details here

DISTINCTCOUNTHLLPLUS(some_id, 12)

Support for clpMatch

• Adds query rewriting logic to transform a "virtual" UDF, clpMatch, into a boolean expression on the columns of a CLP-encoded field.
• To use the rewriter, modify broker config to add org.apache.pinot.sql.parsers.rewriter.ClpRewriter to pinot.broker.query.rewriter.class.names.

Support for DATETIMECONVERTWINDOWHOP function (#11773)

Support for JSON_EXTRACT_INDEX transform function to leverage json index for json value extraction (#11739)

Support for ArrayAgg aggregation function (#11822)

GenerateData command support for generating data in JSON format (#11778)

Enhancements

SQL

• Support ARRAY function as a literal evaluation (#12278)
• Support for ARRAY literal transform functions (#12118)
• Theta Sketch Aggregation enhancements (#12042)
  • Adds configuration options for DistinctCountThetaSketchAggregationFunction
  • Respects ordering for existing Theta sketches to use "early-stop" optimisations for unions
• Add query option override for Broker MinGroupTrimSize (#11984)
• Support for 2 new scalar functions for bytes: toUUIDBytes and fromUUIDBytes (#11988)
• Config option to make groupBy trim size configurable at Broker (#11958)
• Pre-aggregation support for distinct count hll++ (#11747)
• Add float type into literal thrift to preserve literal type conforming to SQL standards (#11697)
• Enhancement to add query function override for Aggregate functions of multi valued columns (#11307)
• Perf optimization in IN clause evaluation (#11557)
• Add TextMatchFilterOptimizer to maximally push down text_match filters to Lucene (#12339

UI

• Async rendering of UI elements to load UI elements async resulting in faster page loads (#12210)
• Make the table name link clickable in task details (#12253)
• Swagger UI enhancements to resumeConsumption API call (#12200)
• Adds support for CTRL key as a modifier for Query shortcuts (#12087)
• UI enhancement to show partial index in reload (#11913)
• UI improvement to add Links to Instance in Table and Segment View (#11807)
• Fixes reload to use the right indexes API instead of fetching all segment metadata (#11793)
• Enhancement to add toggle to hide/show query exceptions (#11611)

Misc

• Enhancement to reduce the heap usage of String Dictionaries that are loaded on-heap (#12223)
• Wire soft upsert delete for Compaction task (12330)
• Upsert compaction debuggability APIs for validDocId metadata (#12275)
• Make server resource classes configurable (#12324)
• Shared aggregations for Startree index - mapping from aggregation used in the query to aggregation used to store pre-aggregated values (#12164)
• Increased fetch timeout for Kineses to prevent stuck kinesis consumers
• Metric to track table rebalance (#12270)
• Allow server-level configs for upsert metadata (#18851)
• Support to dynamically initialize Kafka client SSL configs (#12249)
• Optimize segment metadata file creation without having to download full segment (#12255)
• Allow string / numeric data type for deleteRecordColumn config (#12222)
• Atomic and Deterministic snapshots of validDocId for upsert tables (#12232, #12246)
• Observability enhancement to add column name when JSON index building fails (#12151)
• Creation of DateTimeGenerator for DATE_TIME field type columns (#12206)
• Add singleton registry for all controller and minion metrics (#12119)
• Support helm chart server separate liveness and readiness probe endpoints (#11800)
• Observability enhancement to add metrics for Table Disabled and Consumption Paused (#12000)
• Support for SegmentGenerationAndPushTask to push segment to realtime table (#12084)
• Enhancement to make the deep store upload retry async with configurable parallelism (#12017)
• Optimizations in segment commit to not read partition group metadata (#11943)
• Replace timer with scheduled executor service in IngestionDelayTracker to reduce number of threads (#11849)
• Adds an option skipControllerCertValidation to skip controller cert validation in AddTableCommand (#11967)
• Adds instrumentation for DataTable Creation (#11942)
• Improve performance of ZkBasicAuthAccessFactory by caching Bcrypt password (#11904)
• Adds support to to fetch metadata for specific list of segments (#11949)
• Allow user specify local temp directory for quickstart (#11961)
• Optimization for server to directly return final result for queries hitting single server (#11938)
• Explain plan optimization to early release AcquireReleaseColumnsSegmentOperator (#11945)
• Observability metric to track query timeouts (#11892)
• Add support for auth in QueryRunner (#11897)
• Allow users to pass custom RecordTransformers to SegmentProcessorFramework (#11887)
• Add isPartialResult flag to broker response (#11592)
• Add new configs to Google Cloud Storage (GCS) connector: jsonKey (#11890)
  • jsonKey is the GCP credential key in string format (either in plain string or base64 encoded string). Refer Creating and managing service account keys to download the keys.
• Performance enhancement to build segments in column orientation (#11776)
  • Disabled by default. Can be enabled by setting table config columnMajorSegmentBuilderEnabled
• Observability enhancements to emit metrics for grpc request and multi-stage leaf stage (#11838)
  • pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)
  • pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)
• Observability improvement to expose GRPC metrics (#11842)
• Improvements to response format for reload API to be pretty printed (#11608)
• Enhancements to support Java 21 (#11672)
• Add more information in RequestContext class (#11708)
• Support to read exact buffer byte ranges corresponding to a given forward index doc id (#11729)
• Enhance Broker reducer to handle expression format change (#11762)
• Capture build scans on ge.apache.org to benefit from deep build insights (#11767)
• Performance enhancement in multiple places by updating initial capacity of HashMap (#11709)
• Support for building indexes post segment file creation, allowing indexes that may depend on a completed segment to be built as part of the segment creation process (#11711)
• Support excluding time values in SimpleSegmentNameGenerator (#11650)
• Perf enhancement to reduce cpu usage by avoiding throwing an exception during query execution (#11715)
• Added framework for supporting nulls in ScalarTransformFunctionWrapper in the future (#11653)
• Observability change to metrics to export netty direct memory used and max (#11575)
• Observability change to add a metric to measure total thread cpu time for a table (#11713)
• Observability change to use SlidingTimeWindowArrayReservoirin dropwizard metrics (#11695)
• Minor improvements to upsert preload (#11694)
• Observability changes to expose additional Realtime Ingestion Metrics (#11685)
• Perf enhancement to remove the global lock in SegmentCompletionManager (#11679)
• Enhancements to unify tmp file naming format and delete tmp files at a regular cadence by extending the ControllerPeriodicTask (#10815)
  • controller.realtime.segment.tmpFileAsyncDeletionEnabled (default false)
  • controller.realtime.segment.tmpFileRetentionInSeconds (default 3600)
• Improvements to skip unparseable records in the csv record reader (#11540, #11594)
• Enhancements to allow override/force options when add schema (#11572)
• Enhancement to handle direct memory OOM on brokers (#11496)
• Enhancement to metadata API to return upsert partition to primary key count map for both controller and server APIs (#12334)
• Enhancements to peer server segment download by retrying both peer discovery and download. (#12317)
• Helper functions in StarTreeBuilderUtils and StarTreeV2BuilderConfig (#12361)
• Perf optimizations to release all segments of a table in releaseAndRemoveAllSegments method (#12297)
• Enhancement to Maintain pool selection for the minimizeDataMovement instance partition assignment strategy (#11953)
• Upsert enhancement to assign segments for with respect to ideal state (#11628)
• Observability change to export Additional Upsert Metrics to Prom (#11660)
• Observibility enhancement to add CPU metrics for minion purge task (#12337)
• Add HttpHeaders in broker event listener requestContext (#12258)

Bugfixes, Refactoring, Cleanups, Deprecations

• Upsert bugfix in "rewind()" for CompactedPinotSegmentRecordReader (#12329)
• Fix error message format for Preconditions.checks failures(#12327)
• Bugfix to distribute Pinot as a multi-release JAR (#12131, #12300)
• Fixes in upsert metadata manager (#12319)
• Security fix to allow querying tables with table-type suffix (#12310)
• Bugfix to ensure tagConfigOverride config is null for upsert tables (#12233 and #12311)
• Increased fetch timeout for Kineses to prevent stuck kinesis consumers(#12214)
• Fixes to catch-all Regex for JXM -> Prom Exporter (#12073 and #12295)
• Fixes lucene index errors when using QuickStart (#12289)
• Null handling bugfix for sketch group-by queries (#12259)
• Null pointer exception fixes in Controller SQL resource (#12211)
• Synchronization fixes to replace upsert segments (#12105 and #12241)
• Bugfix for S3 connection pool error when AWS session tokens expire after an hour (#12221)
• FileWriter fixes to append headerline only for required formats like csv (#12208)
• Security bugfix for pulsar OAuth2 authentication (#12195)
• Bugfix to appropriately compute "segment.flush.threshold.size" when force-committing realtime segments (#12188)
• Fixes rebalance converge check that reports success before rebalance completes (#12182)
• Fixes upsertPrimaryKeysCount metric reporting when table is deleted (#12169)
• Update LICENSE-binary for commons-configuration2 upgrade (#12165)
• Improve error logging when preloading segments not exist on server (#12153)
• Fixes to file access resource leaks (#12129)
• Ingestion bugfix to avoid unnecessary transformers in CompositeTransformer (#12138)
• Improve logging to print OS name during service statup (#12135)
• Improve logging in multiple files (#12134, #12137, #12127, #12121)
• Test fixes for ExprMinMaxRewriterTest.testQueryRewrite (#12047)
• Fixes default path of log4j in helmchart (#12069, #12083)
• Fix default brokerUpdateFrequencyInMillis for connector (#12093)
• Updates to README file (#12075)
• Fix to remove unnecessary locking during segment preloading (#12077)
• Fix bug with silently ignoring force commit call failures (#12044)
• Upsert bugfix to allow optional segments that can be skipped by servers without failing the query (#11978)
• Fix incorrect handling of consumer creation errors (#12045)
• Fix the memory leak issue on CommonsConfigurationUtils (#12056)
• Fix rebalance on upsert table (#12054)
• Add new Transformer to transform -0.0 and NaN (#12032)
• Improve inverted index validation in table config to enhance user experience (#12043)
• Fixes test flakiness by replacing HashSet/HashMap with LinkedHashSet/LinkedHashMap (#11941)
• Flaky test fix for ServerRoutingStatsManagerTest.testQuerySubmitAndCompletionStats (#12029)
• Fix derived column from MV column (#12028)
• Support for leveraging StarTree index in conjunction with filtered aggregations (#11886)
• Improves tableConfig validation for enabling size based threshold for realtime tables (#12016)
• Fix flaky PinotTenantRestletResourceTest (#12026)
• Fix flaky Fix PinotTenantRestletResourceTest (#12019)
• Fix the race condition of concurrent modification to segment data managers (#12004)
• Fix the misuse of star-tree when all predicates are always false under OR (#12003)
• Fix the test failures caused by instance drop failure (#12002)
• Fix fromULL scalar function (#11995)
• Fix to exclude module-info.class during shade operations (#11975)
• Fix the wrong import for Preconditions (#11979)
• Add check for illegal character '/' in taskName (#11955)
• Bugfix to only register new segments when it's fully initalized by partitionUpsertMetadataManager (#11964)
• Obervability fix to add logs to track sequence of events for table creation (#11946)
• Fix the NPE in minimizeDataMovement instance assignment strategy (#11952)
• Fix to add catch all logging for exception during DQL/DML process (#11944)
• Fix bug where we don't handle cases that a upsert table has both upsert deletion and upsert ttl configs (#11791)
• Removing direct dependencies on commons-logging and replacing with jcl-over-slf4j (#11920)
• Fix NPE for IN clause on constant STRING dictionary (#11930)
• Fix flaky OfflineClusterIntegrationTest on server response size tests (#11926)
• Avoid npe when checking mirror server set assignment (#11915)
• Deprecate _segmentAssignmentStrategy in favor of SegmentsValidationAndRetentionConfig #11869
• Bugfix to capture auth phase timing even if access is denied (#11884)
• Bugfix to mark rows as invalid in case primary time column is out of range (#11907)
• Fix to radomize server port to avoid port already bind issue (#11861)
• Add LazyRow abstraction for previously indexed record (#11826)
• Config Validation for upsert table to not assign COMPLETED segments to another server (#11852)
• Bugfix to resolve dependency conflict in pinot-protobuf module (#11867)
• Fix case of useMultistageEngine property reference in JsonAsyncHttpPinotClientTransportFactory (#11820)
• Bugfix to add woodstox-core to pinot-s3 dependencies and fix stack trace (#11799)
• Fix to move pinot-segment-local test from unit test suite 1 to 2 (#11865)
• Observability fix to log upsert config when initializing the metadata manager (#11864)
• Fix to improve tests when errors are received in the consumer thread (#11858)
• Fix for flaky ArrayAgg test (#11860)
• Fix for flaky tests in TupleSelectionTransformFunctionsTest (#11848)
• Fix for arrayAgg null support (#11853)
• Fix the bug of reading decimal value stored in int32 or int64 (#11840)
• Remove duplicate pinot-integration-tests from unit test suite 2 (#11844)
• Fix for a null handling error in queries (#11829)
• Fix the way of fetching the segment zk metadata for task generators (#11832)
• Make testInvalidateCachedControllerLeader times based on getMinInvalidateIntervalMs (#11815)
• Update doap to reflect latest release (#11827)
• Clean up integration test pom file (#11817)
• Bugfix to exclude OFFLINE segments when reading server to segments map (#11818)
• Add tests for zstd compressed parquet files (#11808)
• Fix job submission time for reload and foce commit job (#11803)
• Remove actually unsupported config that selectively enable nullable columns (#10653)
• Fix LLCRealtimeClusterIntegrationTest.testReset (#11806)
• Use expected version in api for table config read modify write change (#11782)
• Move jobId out of rebalanceConfig (#11790)
• Fix PeerServerSegmentFinder not respecting HTTPS port (#11752)
• Enhanced geospatial v2 integration tests (#11741)
• Add integration test for rebalance in upsert tables (#11568)
• Fix trivy CI issue (#11757)
• Cleanup rebalance configs by adding a RebalanceConfig class (#11730)
• Fix a protobuf comment to be more precise (#11735)
• Move scala dependencies to root pom (#11671)
• Fix ProtoBuf inputformat plug-in handling for null values (#11723)
• Bugfix where segment download URI is invalid after same CRC refresh using tar push (#11720)
• Fix in TableCacheTest (#11717)
• Add more test for broker jersey bounded thread pool (#11705)
• Fix bug in gapfill with SumAvgGapfillProcessor. (#11714)
• Bugfix to allow GcsPinotFS to work with granular permissions (#11655)
• Fix default log4j2 config file path in helm chart (#11707)
• Refactor code and doc occurrences of argmin/max -> exprmin/max (#11700)
• Make constructor and functions public to be used from scheduler plugins (#11699)
• Bugfix to change json_format to return java null when java null is received (#11673)
• Fix the potential access to upsert metadata manager after it is closed (#11692)
• Bugfix to use isOptional instead of the deprecated hasOptional Keyword (#11682)
• Fix logging issue in RealtimeTableDataManager (#11693)
• Cleanup some reader/writer logic for raw forward index (#11669)
• Do not execute spotless in Java 21 (#11670)
• Update license-maven-plugin (#11665)
• Bugfix to allow deletion of local files with special characters (#11664)
• Clean up CaseTransformFunction::constructStatementListLegacy. (#11339)
• Bugfix to force FileChannel to commit data to disk (#11625)
• Remove the old deprecated commit end without metadata (#11662)
• Fix for a jackson vulnerability (#11619)
• Refactor BasicAuthUtils from pinot-core to pinot-common and remove pinot-core dependency from pinot-jdbc-client (#11620)
• Bugfix to support several extensions for different indexes (#11600)
• Fix the alias handling in single-stage engine (#11610)
• Fix to use constant null place holder (#11615)
• Refactor to move all BlockValSet into the same package (#11616)
• Remove deprecated Request class from pinot-java-client (#11614)
• Refactoring to remove old thirdeye files. (#11609)
• Testing fix to use builder method in integration test (#11564)
• Fix the broken Pinot JDBC client. (#11606)
• Bugfix to change the Forbidden error to Unauthorized (#11501)
• Fix for schema add UI issue that passing wrong data in the request header (#11602)
• Remove/Deprecate HLC handling code (#11590)
• Fix the bug of using push time to identify new created segment (#11599)
• Bugfix in CSVRecordReader when using line iterator (#11581)
• Remove split commit and some deprecated config for real-time protocol on controller (#11663)Improved validation for single argument aggregation functions (#11556)
• Fix to not emit lag once tabledatamanager shutdown (#11534)
• Bugfix to fail reload if derived columns can't be created (#11559)
• Fix the double unescape of property value (#12405)
• Fix for the backward compatible issue that existing metadata may contain unescaped characters (#12393)
• Skip invalid json string rather than throwing error during json indexing (#12238)
• Fixing the multiple files concurrent write issue when reloading SSLFactory (#12384)
• Fix memory leaking issue by making thread local variable static (#12242)
• Bugfixfor Upsert compaction task generator (#12380)
• Log information about SSLFactory renewal (#12357)
• Fixing array literal usage for vector (#12365)
• Fixing quickstart table baseballStats minion ingestion (#12371)
• Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction (#12347)
• Bugfix to skip instead of throwing error on 'getValidDocIdMetadata' (#12360)
• Fix to clean up segment metadata when the associated segment gets deleted from remote store (#12350)
• Fix getBigDecimal() scale throwing rounding error (#12326)
• Workaround fix for the problem of Helix sending 2 transitions for CONSUMING -> DROPPED (#12351)
• Bugfix for making nonLeaderForTables exhaustive (#12345)
• Bugfixes for graceful interrupt handling of mutable lucene index (#11558,#12274)
• Remove split commit and some deprecated config for real-time protocol on controller (#11663)
• Update the table config in quick start (#11652)
• Deprecate k8s skaffold scripts and move helm to project root directory (#11648)
• Fix NPE in SingleColumnKeySelector (#11644)
• Simplify kafka build and remove old kafka 0.9 files (#11638)
• Adding comments for docker image tags, make a hyper link of helmChart from root directory (#11646)
• Improve the error response on controller. (#11624)
• Simplify authrozation for table config get (#11640)
• Bugfix to remove segments with empty download url in UpsertCompactionTask (#12320)
• Test changes to make taskManager resources protected for derived classes to override in their setUp() method. (#12335)

Backward incompatible Changes

• Fix a race condition for upsert compaction (#12346). Notes on backward incompatibility below:

  • This PR is introducing backward incompatibility for UpsertCompactionTask. Previously, we allowed to configure the compaction task without the snapshot enabled. We found that using in-memory based validDocIds is a bit dangerous as it will not give us the consistency (e.g. fetching validDocIds bitmap while the server is restarting & updating validDocIds).

    We now enforce the enableSnapshot=true for UpsertCompactionTask if the advanced customer wants to run the compaction with the in-memory validDocId bitmap.

    {
      "upsertConfig": {
        "mode": "FULL",
        "enableSnapshot": true
      }
    }
    ...
    "task": {
      "taskTypeConfigsMap": {
        "UpsertCompactionTask": {
          "schedule": "0 */5 * ? * *",
          "bufferTimePeriod": "7d",
          "invalidRecordsThresholdPercent": "30",
          "invalidRecordsThresholdCount": "100000",
          "invalidDocIdsType": "SNAPSHOT/IN_MEMORY/IN_MEMORY_WITH_DELETE"
        }
      }
    }
    

    Also, we allow to configure invalidDocIdsType to UpsertCompactionTask for advanced user.

    1. snapshot: Default validDocIds type. This indicates that the validDocIds bitmap is loaded from the snapshot from the Pinot segment. UpsertConfig's enableSnapshot must be enabled for this type.
       1. onHeap: the validDocIds bitmap will be fetched from the server.
       2. onHeapWithDelete: the validDocIds bitmap will be fetched from the server. This will also take account into the deleted documents. UpsertConfig's deleteRecordColumn must be provided for this type.
• Removal of the feature flag allow.table.name.with.database (#12402)
• Error handling to throw exception when schema name doesn't match table name during table creation (#11591)
• Fix type cast issue with dateTimeConvert scalar function (#11839, #11971)
• Incompatible API fix to remove table state update operation in GET call (#11621)
• Use string to represent BigDecimal datatype in JSON response (#11716)
• Single quoted literal will not have its type auto-derived to maintain SQL compatibility (#11763)
• Changes to always use split commit on server and disables the option to disable it (#11680, #11687)
• Change to not allow NaN as default value for Float and Double in Schemas (#11661)
• Code cleanup and refactor that removes TableDataManagerConfig (#12189)
• Fix partition handling for consistency of values between query and segment (#12115)
• Changes for migration to commons-configuration2 (#11985)
• Cleanup to simplify the upsert metadata manager constructor (#12120)
• Fixes typo in pom.xml (#11997)
• JDBC Driver fixes to support Jetbrains Intellij/Datagrip database tooling (#11814)
• Fix regression in ForwardIndexType for noDictionaryConfig and noDictionaryColumns (#11784)
• Separate pr test scripts and codecov (#11804)
• Bugfix to make reload status should only count online/consuming segments (#11787)
• Fix flaky TableViewsTest (#11770)
• Fix a flaky test (#11771)
• Cleanup to fee more disk for trivy job (#11780)
• Fix schema name in table config during controller startup (#11574)
• Prevent NPE when attempt to fetch partition information fails (#11769)
• Added UTs for null handling in CaseTransform function. (#11721)
• Bugfix to disallow peer download when replication is < 2 (#11469)
• Update s todocker image and github action scripts (#12378)
• Enhancements to queries test framework (#12215)

Library Upgrades and dependencies

• update maven-jar-plugin and maven-enforcer-plugin version (#11637)
• Update testng as the test provider explicitly instead of relying on the classpath. (#11612)
• Update compatibility verifier version (#11684)
• Upgrade Avro dependency to 1.10.2 (#11698)
• Upgrade testng version to 7.8.0 (#11462)
• Update lombok version and config (#11742)
• Upgrading Apache Helix to 1.3.1 version (#11754)
• Upgrade spark from 3.2 to 3.5 (#11702)
• Added commons-configuration2 dependency. (#11792)
• Upgrade confluent libraries to 7.2.6 to fix some errors related to optional proto fields (#11753)
• Upgrade lucene to 9.8.0 and upgrade text index version (#11857)
• Upgrade the PinotConfiguartion to commons-configuartion2(#11916)
• Pre PinotConfig commons-configuartions2 upgrade (#11868)
• Bump commons-codec:commons-codec from 1.15 to 1.16.0 (#12204)
• Bump flink.version from 1.12.0 to 1.14.6 (#12202)
• Bump com.yscope.clp:clp-ffi from 0.4.3 to 0.4.4 (#12203)
• Bump org.apache.spark:spark-launcher_2.12 from 3.2.1 to 3.5.0 (#12199)
• Bump io.grpc:grpc-context from 1.59.0 to 1.60.1 (#12198)
• Bump com.azure:azure-core from 1.37.0 to 1.45.1 (#12193)
• Bump org.freemarker:freemarker from 2.3.30 to 2.3.32 (#12192)
• Bump com.google.auto.service:auto-service from 1.0.1 to 1.1.1 (#12183)
• Bump dropwizard-metrics.version from 4.2.22 to 4.2.23 (#12178)
• Bump org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0 (#12170)
• Bump com.gradle:common-custom-user-data-maven-extension (#12171)
• Bump org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14 (#12172)
• Bump org.glassfish.tyrus.bundles:tyrus-standalone-client (#12162)
• Bump com.google.api.grpc:proto-google-common-protos (#12159)
• Bump org.apache.datasketches:datasketches-java from 4.1.0 to 5.0.0 (#12161)
• Bump org.apache.zookeeper:zookeeper from 3.6.3 to 3.7.2 (#12152)
• Bump org.apache.commons:commons-collections4 from 4.1 to 4.4 (#12149)
• Bump log4j.version from 2.20.0 to 2.22.0 (#12143)
• Bump com.github.luben:zstd-jni from 1.5.5-6 to 1.5.5-11 (#12125)
• Bump com.google.guava:guava from 32.0.1-jre to 32.1.3-jre (#12124)
• Bump org.apache.avro:avro from 1.10.2 to 1.11.3 (#12116)
• Bump org.apache.maven.plugins:maven-assembly-plugin from 3.1.1 to 3.6.0 (#12109)
• Bump net.java.dev.javacc:javacc from 7.0.10 to 7.0.13 (#12103)
• Bump com.azure:azure-identity from 1.8.1 to 1.11.1 (#12095)
• Bump xml-apis:xml-apis from 1.4.01 to 2.0.2 (#12082)
• Bump up the parquet version to 1.13.1 (#12076)
• Bump io.grpc:grpc-context from 1.14.0 to 1.59.0 (#12034)
• Bump org.reactivestreams:reactive-streams from 1.0.3 to 1.0.4 (#12033)
• Bump org.codehaus.mojo:appassembler-maven-plugin from 1.10 to 2.1.0 (#12030)
• Bump com.google.code.findbugs:jsr305 from 3.0.0 to 3.0.2 (#12031)
• Bump org.jacoco:jacoco-maven-plugin from 0.8.9 to 0.8.11 (#12024)
• Bump dropwizard-metrics.version from 4.2.2 to 4.2.22 (#12022)
• Bump grpc.version from 1.53.0 to 1.59.0 (#12023)
• Bump com.google.code.gson:gson from 2.2.4 to 2.10.1 (#12009)
• Bump net.nicoulaj.maven.plugins:checksum-maven-plugin from 1.8 to 1.11 (#12008)
• Bump circe.version from 0.14.2 to 0.14.6 (#12006)
• Bump com.mercateo:test-clock from 1.0.2 to 1.0.4 (#12005)
• Bump simpleclient_common.version from 0.8.1 to 0.16.0 (#11986)
• Bump com.jayway.jsonpath:json-path from 2.7.0 to 2.8.0 (#11987)
• Bump commons-net:commons-net from 3.1 to 3.10.0 (#11982)
• Bump org.scalatest:scalatest-maven-plugin from 1.0 to 2.2.0 (#11973)
• Bump io.netty:netty-bom from 4.1.94.Final to 4.1.100.Final (#11972)
• Bump com.google.errorprone:error_prone_annotations from 2.3.4 to 2.23.0 (#11905)
• Bump net.minidev:json-smart from 2.4.10 to 2.5.0 (#11875)
• Bump org.yaml:snakeyaml from 2.0 to 2.2 (#11876)
• Bump browserify-sign in /pinot-controller/src/main/resources (#11896)
• Bump org.easymock:easymock from 4.2 to 5.2.0 (#11854)
• Bump org.codehaus.mojo:exec-maven-plugin from 1.5.0 to 3.1.0 (#11856)
• Bump com.github.luben:zstd-jni from 1.5.2-3 to 1.5.5-6 (#11855)
• Bump aws.sdk.version from 2.20.94 to 2.20.137 (#11463)
• Bump org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.4 (#11678)

What's changed

Multi-stage Query Engine

Features

Support RelDistribution-based trait Planning (#11976, #12079)

• Adds support for RelDistribution optimization for more accurate leaf-stage direct exchange/shuffle. Also extends partition optimization beyond leaf stage to entire query plan.
• Applies optimization based on distribution trait in the mailbox/worker assignment stage
  • Fixes previous direct exchange which was decided based on the table partition hint. Now direct exchange is decided via distribution trait: it will applied if-and-only-if the trait propagated matches the exchange requirement.
  • As a side effect, is_colocated_by_join_keysquery option is reintroduced to ensure dynamic broadcast which can also benefit from direct exchange optimization
  • Allows propagation of partition distribution trait info across the tree to be used during Physical Planning phase. It can be used in the following scenarios (will follow up in separate PRs)
• Note on backward incompatbility
  • is_colocated_by_join_keys hint is now required for making colocated joins
    • it should only affect semi-join b/c it is the only one utilizing broadcast exchange but were pulled to act as direct exchange.
    • inner/left/right/full join should automatically apply colocation thus the backward incompatibility should not affect these.

Leaf stage planning with multi-semi join support (#11937)

• Solves the limitation of pinotQuery that supports limited amount of PlanNodes.
• Splits the ServerRequest planning into 2 stages
  • First plan as much as possible into PinotQuery
  • for any remainder nodes that cannot be planned into PinotQuery, will be run together with the LeafStageTransferrableBlockOperator as the input locally.

Support for ArrayAgg aggregation function (#11822)

• Usage: ArrayAgg(column, 'dataType' [, 'isDistinct'])
• Float type column is treated as Double in the multistage engine, so FLOAT type is not supported.
• Supports data BOOLEAN, INT, LONG, FLOAT(only in V1), DOUBLE, STRING, TIMESTAMP.
  E.g. ArrayAgg(intCol, 'INT') returns ARRAY<INT>

Multi-stage Query Engine Enhancements

• Canonicalize SqlKind.OTHERS and SqlKind.OTHER_FUNCTIONS and support

  concat as || operator (#12025)

• Capability for constant filter in QueryContext, with support for server to handle it (#11956)

• Tests for filter pushdown (#11994)

• Enhancements to query plan tests (#11966)

• Refactor PlanFragmenter to make the logic clear (#11912)

• Observability enhancements to emit metrics for grpc request and multi-stage leaf stage (#11838)

  • pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)
  • pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)

• Security enhancement to add RBAC authorization checks for multi-stage query engine (#11830)

• Enhancement to leaf-stage execution stats NPE handling (#11805)

• Enhancement to add a framework to back-propagate metadata across opChains (#11746)

• Use of BinaryArray to wire proto for multi-stage engine bytes literal handling (#11738)

• Enable dynamic broadcast for SEMI joins. Adds a fallback option to enable hash table join using joinOptions(join_strategy = 'hash_table')(#11696)

• Improvements to dispatch exception handling (#11688)

• Allow malformed dateTime string to return default value configurable in the function signature (#11258)

  • fromDateTime(colContainsMalformedStr, '<dateTimeFormat>', '<timezone>', <default_value>)
    

• Improvement in multi-stage aggregation to directly store column index as identifier (#11617)

• Perf optimization to avoid unnecessary rows conversion in aggregation (#11607)

• Enhance SegmentPartitionMetadataManager to handle new segment (#11585)

• Optimize mailbox info in query plan to reduce memory footprint (#12382)

  • This PR changes the proto object structure, which will cause backward incompatibility when broker and server are running different version.

• Optimizations to query plan serialization (#12370)

• Optimization for parallel execution of Ser/de stage plan (#12363)

• Optimizations in query dispatch (#12358)

• Perf optimization for group-by and join for single key scenario (#11630)

Multi-stage Query Engine: Bugfixes, Refactoring, Cleanups, Tests

• Bugfix for evaluation of chained literal functions (#12248)
• Fixes to sort copy rule (#12251 and #12237)
• Fixes duplicate results for literal queries (#12240)
• Bugfix to use UTF-8 encoding for default Charset (#12213)
• Bugfix to escape table name when routing queries (#12212)
• Refactoring of planner code and removing unnecessary rules (#12070, #12052)
• Fix to remove unnecessar project after agg during relBuilder (#12058)
• Fixes issues multi-semi-join (#12038)
• Fixes leaf limit refactor issue (#12001)
• Add back filter merge after rule (#11989)
• Fix operator EOS pull (#11970)
• Backward incompatible: Fix type cast issue with dateTimeConvert scalar function (#11839, #11971)
• Fix to set explicit warning flags set on each stage stats (#11936)
• Fix mailbox visitor mismatch receive/send (#11908)
• Fix eliminate multiple exchanges in nested semi-join queries (#11882)
• Bugfix for multiple consecutive Exchange returning empty response (#11885)
• Fixing unit-test-2 build (#11889)
• Fix issue with realtime partition mismatch metric (#11871)
• Fix the NPE for rebalance retry (#11883)
• Bugfix to make Agg literal attach happen after BASIC_RULES (#11863)
• Fix NPE by init execution stats map (#11801)
• Test cases for special column escape (#11737)
• Fix StPoint scalar function usage in multi-stage engine intermediate stage (#11731)
• Clean up for transform function type (#11726)
• Add capability to ignore test (#11703)
• Fix custom property naming (#11675)
• Log warning when multi-stage engine planning throws exception (#11595)
• Fix usage of metadata overrides (#11587)
• Test change to enable metadata manager by default for colocated join quickstart (#11579)
• Tests for IN/NOT-IN operation (#12349)
• Fix stage id in stage plan (#12366)
• Bugfix for IN and NOT IN filters within case statements (#12305)

Notable Features

Server-level throttling for realtime consumption (#12292)

• Use server config pinot.server.consumption.rate.limit to enable this feature
• Server rate limiter is disabled by default (default value 0)

Reduce segment generation disk footprint for Minion Tasks (#12220)

• Supported in MergeRollupTask and RealtimeToOfflineSegmentsTask minion tasks
• Use taskConfig segmentMapperFileSizeThresholdInBytes to specify the threshold size

"task": {
  "taskTypeConfigsMap": {
    "<task_name>": {
      "segmentMapperFileSizeThresholdInBytes": "1000000000"
    }
  }
}

Support for swapping of TLS keystore/truststore (#12277, #12325)

• Security feature that makes the keystore/truststore swappable.
• Auto-reloads keystore/truststore (without need for a restart) if they are local files

Sticky Query Routing (#12276)

• Adds support for deterministic and sticky routing for a query / table / broker. This setting would lead to same server / set of servers (for MultiStageReplicaGroupSelector) being used for all queries of a given table.

• Query option (takes precedence over fixed routing setting at table / broker config level)
  SET "useFixedReplica"=true;

• Table config (takes precedence over fixed routing setting at broker config level)

  "routing": {
     ...          
     "useFixedReplica": true
  }
  

• Broker conf - pinot.broker.use.fixed.replica=true

Table Config to disallow duplicate primary key for dimension tables (#12290)

• Use tableConfig dimensionTableConfig.errorOnDuplicatePrimaryKey=true to enable this behavior
• Disabled by default

Partition-Level ForceCommit for realtime tables (#12088)

• Support to force-commit specific partitions of a realtime table.
• Partitions can be specified to the forceCommit API as a comma separated list of partition names or consuming segment names

Support initializing broker tags from config (#12175)

• Support to give the broker initial tags on startup.
• Automatically updates brokerResource when broker joins the cluster for the first time
• Broker tags are provided as comma-separated values in pinot.broker.instance.tags

Support for StreamNative OAuth2 authentication for pulsar (#12068)

• StreamNative (the cloud SAAS offering of Pulsar) uses OAuth2 to authenticate clients to their Pulsar clusters.
• For more information, see how to Configure OAuth2 authentication in Pulsar clients
• Can be configured by adding the following properties to streamConfigs:

"stream.pulsar.issuerUrl": "https://auth.streamnative.cloud"
"stream.pulsar.credsFilePath": "file:///path/to/private_creds_file
"stream.pulsar.audience": "urn:sn:pulsar:test:test-cluster"

Introduce low disk mode to table rebalance (#12072)

• Introduces a new table rebalance boolean config lowDiskMode.Default value is false.
• Applicable for rebalance with downtime=false.
• When enabled, segments will first be offloaded from servers, then added to servers after offload is done. It may increase the total time of the rebalance, but can be useful when servers are low on disk space, and we want to scale up the cluster and rebalance the table to more servers.
• #12112 adds the UI capability to toggle this option

Support Vector index and Hierarchical Navigable Small Worlds (HNSW) (#11977)

• Supports Vector Index on float array/multi-value columnz
• Add predicate and function to retrieve topK closest vector. Example query

SELECT ProductId, UserId, l2_distance(embedding, ARRAY[-0.0013143676,-0.011042999,...]) AS l2_dist, n_tokens, combined
FROM fineFoodReviews
WHERE VECTOR_SIMILARITY(embedding, ARRAY[-0.0013143676,-0.011042999,...], 5)  
ORDER by l2_dist ASC 
LIMIT 10

• The function VectorSimilarity will return a double value where the first parameter is the embedding column and the second parameter is the search term embedding literal.
• Since VectorSimilarity is a predicate, once config the topK, this predicate will return topk rows per segment. Then if you are using this index with other predicate, you may not get expected number of rows since the records matching other predicate might not in the topk rows.

Support for retention on deleted keys of upsert tables (#12037)

• Adds an upsert config deletedKeysTTL which will remove deleted keys from in-memory hashmap and mark the validDocID as invalid after the deletedKeysTTL threshold period.
• Disabled by default. Enabled only if a valid value for deletedKeysTTL is set
• More details in the design document

Configurable Lucene analyzer (#12027)

• Introduces the capability to specify a custom Lucene analyzer used by text index for indexing and search on an individual column basis.
• Sample usage

fieldConfigList: [
   {
        "name": "columnName",
        "indexType": "TEXT",
        "indexTypes": [
          "TEXT"
        ],
        "properties": {
          "luceneAnalyzerClass": "org.apache.lucene.analysis.core.KeywordAnalyzer"
        },
      }
  ]

• Default Behavior falls back to using the standardAnalyzer unless the luceneAnalyzerClass property is specified.

Support for murmur3 as a partition function (#12049)

• Murmur3 support with optional fields seed and variant for the hash in functionConfig field of columnPartitionMap.Default value for seed is 0.

• Added support for 2 variants of Murmur3: x86_32 and x64_32 configurable using the variant field in functionConfig. If no variant is provided we choose to keep the x86_32 variant as it was part of the original implementation.

• Examples of functionConfig;

  "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3 
            },
            ..
          }
        }
  

  Here there is no functionConfig configured, so the seed value will be 0 and variant will be x86_32.

  "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3,
              "functionConfig": {
                 "seed": "9001"
               },
            },
            ..
          }
        }
  

  Here the seed is configured as 9001 but as no variant is provided, x86_32 will be picked up.

   "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3,
              "functionConfig" :{
                 "seed": "9001"
                 "variant": "x64_32"
               },
            },
            ..
          }
        }
  

  Here the variant is mentioned so Murmur3 will use the x64_32 variant with 9001 as seed.

• Note on users using Debezium and Murmur3 as partitioning function :

  • The partitioning key should be set up on either of byte[], String or long[] columns.
  • On pinot variant should be set as x64_32 and seed should be set as 9001.

New optimized MV forward index to only store unique MV values

• Adds new MV dictionary encoded forward index format that only stores the unique MV entries.

• This new index format can significantly reduce the index size when the MV entries repeat a lot

• The new index format can be enabled during index creation, derived column creation, and segment reload

• To enable the new index format, set the compression codec in the FieldConfig:

  {
    "name": "myCol",
    "encodingType": "DICTIONARY",
    "compressionCodec": "MV_ENTRY_DICT"
  }
  

  Or use the new index JSON:

  {
    "name": "myCol",
    "encodingType": "DICTIONARY",
    "indexes": {
      "forward": {
        "dictIdCompressionType": "MV_ENTRY_DICT"
      }
    }
  }
  

Support for explicit null handling modes (#11960)

• Adds support for 2 possible ways to handle null:
  • Table mode - which already exists
  • Column mode, which means that each column specifies its own nullability in the FieldSpec
• Column mode can be enabled by the below config.
• The default value for enableColumnBasedNullHandling is false. When set to true, Pinot will ignore TableConfig.IndexingConfig.nullHandlingEnabled and columns will be nullable if and only if FieldSpec.notNull is false, which is also the default value.

{
  "schemaName": "blablabla",
  "dimensionFieldSpecs": [
    {
      "dataType": "INT",
      "name": "nullableField",
      "notNull": false
    },
    {
      "dataType": "INT",
      "name": "notNullableField",
      "notNull": true
    },
    {
      "dataType": "INT",
      "name": "defaultNullableField"
    },
    ...
  ],
  "enableColumnBasedNullHandling": true/false
}

Support tracking out of order events in Upsert (#11877)

• Adds a new upsert config outOfOrderRecordColumn
• When set to a non-null value, we check whether an event is OOO or not and then accordingly update the corresponding column value to true / false.
• This will help in tracking which event is out-of-order while using skipUpsert

Compression configuration support for aggregationConfigs to StartreeIndexConfigs (#11744)

• Can be used to save space. For eg: when a functionColumnPairs has a output type of bytes, such as when you use distinctcountrawhll.
• Sample config

"starTreeIndexConfigs": [
        {
          "dimensionsSplitOrder": [
            "a",
            "b",
            "c"
          ],
          "skipStarNodeCreationForDimensions": [],
          "functionColumnPairs": [],
          "aggregationConfigs": [
            {
              "columnName": "column1",
              "aggregationFunction": "SUM",
              "compressionCodec": "SNAPPY"
            },
            {
              "columnName": "column2",
              "aggregationFunction": "distinctcounthll",
              "compressionCodec": "LZ4"
            }
          ],
          "maxLeafRecords": 10000
        }
      ]

Preconfiguration based mirror instance assignment (#11578)

• Supports instance assignment based pre-configured instance assignment map.
• The assignment will always respect the mirrored servers in the pre-configured map
• More details here
• Sample table config

"instanceAssignmentConfigMap": {
  "CONSUMING": {
    "partitionSelector": "MIRROR_SERVER_SET_PARTITION_SELECTOR",
    "replicaGroupPartitionConfig": { ... },
     "tagPoolConfig": {
       ...
       "tag": "mt1_REALTIME"
     }
     ...
 }
 "COMPLETED": {
   "partitionSelector": "MIRROR_SERVER_SET_PARTITION_SELECTOR",
   "replicaGroupPartitionConfig": { ... },
    "tagPoolConfig": {
       ...
       "tag": "mt1_OFFLINE"
     }
     ...
 },
 "instancePartitionsMap": {
      "CONSUMING": “mt1_CONSUMING"
      "COMPLETED": "mt1_OFFLINE"
 },

Support for Listing Dimension Tables (#11859)

• Adds dimension as a valid option to table "type" in the /tables controller API

Support in upsert for dropping out of order events (#11811)

• This patch adds a new config for upsert: dropOutOfOrderRecord
• If set to true, pinot doesn't persist out-of-order events in the segment.
• This feature is useful to
  • Save disk-usage
  • Avoid any confusion when using skipUpsert for partial-upsert tables as nulls start showing up for columns where a previous non-null was encountered and we don't know if it's an out-of-order event or not.

Support to retry failed table rebalance tasks (#11740)

• New configs for the RebalanceChecker periodic task:
  • controller.rebalance.checker.frequencyPeriod: 5min by default ; -1 to disable
  • controller.rebalanceChecker.initialDelayInSeconds: 2min+ by default
• New configs added for RebalanceConfig:
  • heartbeatIntervalInMs: 300_000 i.e. 5min
  • heartbeatTimeoutInMs: 3600_000 i.e. 1hr
  • maxAttempts: 3 by default, i.e. the original run plus two retries
  • retryInitialDelayInMs: 300_000 i.e. 5min, for exponential backoff w/ jitters
• New metrics to monitor rebalance and its retries:
  • TABLE_REBALANCE_FAILURE("TableRebalanceFailure", false), emit from TableRebalancer.rebalanceTable()
  • TABLE_REBALANCE_EXECUTION_TIME_MS("tableRebalanceExecutionTimeMs", false), emit from TableRebalancer.rebalanceTable()
  • TABLE_REBALANCE_FAILURE_DETECTED("TableRebalanceFailureDetected", false), emit from RebalanceChecker
  • TABLE_REBALANCE_RETRY("TableRebalanceRetry", false), emit from RebalanceChecker
• New restful API
  • DELETE /tables/{tableName}/rebalance API to stop rebalance. In comparison, POST /tables/{tableName}/rebalance was used to start one.

Support for UltraLogLog (#11835)

• UltraLogLog aggregations for Count Distinct (distinctCountULL and distinctCountRawULL)
• UltraLogLog creation via Transform Function
• UltraLogLog merging in MergeRollup
• Support for UltraLogLog in Star-Tree indexes

Support for Apache Datasketches CPC sketch (#11774)

• Ingestion via transformation function
• Extracting estimates via query aggregation functions
• Segment rollup aggregation
• StarTree aggregation

Support to reduce DirectMemory OOM chances on broker (#11710)

• Broadly there are two configs that will enable this feature:

  • maxServerResponseSizeBytes: Maximum serialized response size across all servers for a query. This value is equally divided across all servers processing the query.
  • maxQueryResponseSizeBytes: Maximum length of the serialized response per server for a query

• Configs are available as queryOption, tableConfig and Broker config. The priority of enforcement is as follows:

  The overriding order of priority is:
  1. QueryOption  -> maxServerResponseSizeBytes
  2. QueryOption  -> maxQueryResponseSizeBytes
  3. TableConfig  -> maxServerResponseSizeBytes
  4. TableConfig  -> maxQueryResponseSizeBytes
  5. BrokerConfig -> pinot.broker.max.server.response.size.bytes
  6. BrokerConfig -> pinot.broker.max.query.response.size.bytes
  

UI Support to Allow schema to be created with JSON config (#11809)

• This is helpful when user has the entire JSON handy
• UI still keeps Form Way to add Schema along with JSON view

Support in JSON index for ignoring values longer than a given length (#11604)

• Use option maxValueLength in jsonIndexConfig to restrict length of values
• A value of 0 (or when the key is omitted) means there is no restriction

Support for MultiValue VarByte V4 index writer (#11674)

• Supports serializing and writing MV columns in VarByteChunkForwardIndexWriterV4
• Supports V4 reader that can be used to read SV var length, MV fixed length and MV var length buffers encoded with V4 writer

Improved scalar function support for Multivalue columns(#11555, #11654)

arrayIndexOfInt(int[] value, int valToFind)
arrayIndexOfLong(int[] value, long valToFind)
arrayIndexOfFloat(int[] value, float valToFind)
arrayIndexOfDouble(int[] value, double valToFind)
arrayIndexOfString(int[] value, String valToFind)
intersectIndices(int[] values1, int[] values2)

Support for FrequentStringsSketch and FrequentLonsSketch aggregation functions (#11098)

• Approximation aggregation functions for estimating the frequencies of items a dataset in a memory efficient way. More details in Apache Datasketches library.

FREQUENTLONGSSKETCH(col, maxMapSize=256) -> Base64 encoded sketch object
FREQUENTSTRINGSSKETCH(col, maxMapSize=256) -> Base64 encoded sketch object

Controller API for Table Indexe (#11576)

• Table index api to get the aggregate index details of all segments for a table.

  • URL/tables/{tableName}/indexes

• Response format

  {
      "totalSegments": 31,
      "columnToIndexesCount":
      {
          "col1":
          {
              "dictionary": 31,
              "bloom": 0,
              "null": 0,
              "forward": 31,
              ...
              "inverted": 0,
              "some-dynamically-injected-index-type": 31,
          },
          "col2":
          {
              ...
          }
          ...
  }
  

Support for configurable rebalance delay at lead controller (#11509)

• The lead controller rebalance delay is now configurable with controller.resource.rebalance.delay_ms
• Changing rebalance configurations will now update the lead controller resource

Support for configuration through environment variables (#12307)

• Adds support for Pinot configuration through ENV variables with Dynamic mapping.
• More details in issue: #10651
• Sample configs through ENV

export PINOT_CONTROLLER_HOST=host
export PINOT_SERVER_PROPERTY_WHATEVER=whatever_property
export ANOTHER_VARIABLE=random

Add hyperLogLogPlus aggregation function for distinct count (#11346)

• HLL++ has higher accuracy than HLL when cardinality of dimension is at 10k-100k.
• More details here

DISTINCTCOUNTHLLPLUS(some_id, 12)

Support for clpMatch

• Adds query rewriting logic to transform a "virtual" UDF, clpMatch, into a boolean expression on the columns of a CLP-encoded field.
• To use the rewriter, modify broker config to add org.apache.pinot.sql.parsers.rewriter.ClpRewriter to pinot.broker.query.rewriter.class.names.

Support for DATETIMECONVERTWINDOWHOP function (#11773)

Support for JSON_EXTRACT_INDEX transform function to leverage json index for json value extraction (#11739)

Support for ArrayAgg aggregation function (#11822)

GenerateData command support for generating data in JSON format (#11778)

Enhancements

SQL

• Support ARRAY function as a literal evaluation (#12278)
• Support for ARRAY literal transform functions (#12118)
• Theta Sketch Aggregation enhancements (#12042)
  • Adds configuration options for DistinctCountThetaSketchAggregationFunction
  • Respects ordering for existing Theta sketches to use "early-stop" optimisations for unions
• Add query option override for Broker MinGroupTrimSize (#11984)
• Support for 2 new scalar functions for bytes: toUUIDBytes and fromUUIDBytes (#11988)
• Config option to make groupBy trim size configurable at Broker (#11958)
• Pre-aggregation support for distinct count hll++ (#11747)
• Add float type into literal thrift to preserve literal type conforming to SQL standards (#11697)
• Enhancement to add query function override for Aggregate functions of multi valued columns (#11307)
• Perf optimization in IN clause evaluation (#11557)
• Add TextMatchFilterOptimizer to maximally push down text_match filters to Lucene (#12339

UI

• Async rendering of UI elements to load UI elements async resulting in faster page loads (#12210)
• Make the table name link clickable in task details (#12253)
• Swagger UI enhancements to resumeConsumption API call (#12200)
• Adds support for CTRL key as a modifier for Query shortcuts (#12087)
• UI enhancement to show partial index in reload (#11913)
• UI improvement to add Links to Instance in Table and Segment View (#11807)
• Fixes reload to use the right indexes API instead of fetching all segment metadata (#11793)
• Enhancement to add toggle to hide/show query exceptions (#11611)

Misc

• Enhancement to reduce the heap usage of String Dictionaries that are loaded on-heap (#12223)
• Wire soft upsert delete for Compaction task (12330)
• Upsert compaction debuggability APIs for validDocId metadata (#12275)
• Make server resource classes configurable (#12324)
• Shared aggregations for Startree index - mapping from aggregation used in the query to aggregation used to store pre-aggregated values (#12164)
• Increased fetch timeout for Kineses to prevent stuck kinesis consumers
• Metric to track table rebalance (#12270)
• Allow server-level configs for upsert metadata (#18851)
• Support to dynamically initialize Kafka client SSL configs (#12249)
• Optimize segment metadata file creation without having to download full segment (#12255)
• Allow string / numeric data type for deleteRecordColumn config (#12222)
• Atomic and Deterministic snapshots of validDocId for upsert tables (#12232, #12246)
• Observability enhancement to add column name when JSON index building fails (#12151)
• Creation of DateTimeGenerator for DATE_TIME field type columns (#12206)
• Add singleton registry for all controller and minion metrics (#12119)
• Support helm chart server separate liveness and readiness probe endpoints (#11800)
• Observability enhancement to add metrics for Table Disabled and Consumption Paused (#12000)
• Support for SegmentGenerationAndPushTask to push segment to realtime table (#12084)
• Enhancement to make the deep store upload retry async with configurable parallelism (#12017)
• Optimizations in segment commit to not read partition group metadata (#11943)
• Replace timer with scheduled executor service in IngestionDelayTracker to reduce number of threads (#11849)
• Adds an option skipControllerCertValidation to skip controller cert validation in AddTableCommand (#11967)
• Adds instrumentation for DataTable Creation (#11942)
• Improve performance of ZkBasicAuthAccessFactory by caching Bcrypt password (#11904)
• Adds support to to fetch metadata for specific list of segments (#11949)
• Allow user specify local temp directory for quickstart (#11961)
• Optimization for server to directly return final result for queries hitting single server (#11938)
• Explain plan optimization to early release AcquireReleaseColumnsSegmentOperator (#11945)
• Observability metric to track query timeouts (#11892)
• Add support for auth in QueryRunner (#11897)
• Allow users to pass custom RecordTransformers to SegmentProcessorFramework (#11887)
• Add isPartialResult flag to broker response (#11592)
• Add new configs to Google Cloud Storage (GCS) connector: jsonKey (#11890)
  • jsonKey is the GCP credential key in string format (either in plain string or base64 encoded string). Refer Creating and managing service account keys to download the keys.
• Performance enhancement to build segments in column orientation (#11776)
  • Disabled by default. Can be enabled by setting table config columnMajorSegmentBuilderEnabled
• Observability enhancements to emit metrics for grpc request and multi-stage leaf stage (#11838)
  • pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)
  • pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)
• Observability improvement to expose GRPC metrics (#11842)
• Improvements to response format for reload API to be pretty printed (#11608)
• Enhancements to support Java 21 (#11672)
• Add more information in RequestContext class (#11708)
• Support to read exact buffer byte ranges corresponding to a given forward index doc id (#11729)
• Enhance Broker reducer to handle expression format change (#11762)
• Capture build scans on ge.apache.org to benefit from deep build insights (#11767)
• Performance enhancement in multiple places by updating initial capacity of HashMap (#11709)
• Support for building indexes post segment file creation, allowing indexes that may depend on a completed segment to be built as part of the segment creation process (#11711)
• Support excluding time values in SimpleSegmentNameGenerator (#11650)
• Perf enhancement to reduce cpu usage by avoiding throwing an exception during query execution (#11715)
• Added framework for supporting nulls in ScalarTransformFunctionWrapper in the future (#11653)
• Observability change to metrics to export netty direct memory used and max (#11575)
• Observability change to add a metric to measure total thread cpu time for a table (#11713)
• Observability change to use SlidingTimeWindowArrayReservoirin dropwizard metrics (#11695)
• Minor improvements to upsert preload (#11694)
• Observability changes to expose additional Realtime Ingestion Metrics (#11685)
• Perf enhancement to remove the global lock in SegmentCompletionManager (#11679)
• Enhancements to unify tmp file naming format and delete tmp files at a regular cadence by extending the ControllerPeriodicTask (#10815)
  • controller.realtime.segment.tmpFileAsyncDeletionEnabled (default false)
  • controller.realtime.segment.tmpFileRetentionInSeconds (default 3600)
• Improvements to skip unparseable records in the csv record reader (#11540, #11594)
• Enhancements to allow override/force options when add schema (#11572)
• Enhancement to handle direct memory OOM on brokers (#11496)
• Enhancement to metadata API to return upsert partition to primary key count map for both controller and server APIs (#12334)
• Enhancements to peer server segment download by retrying both peer discovery and download. (#12317)
• Helper functions in StarTreeBuilderUtils and StarTreeV2BuilderConfig (#12361)
• Perf optimizations to release all segments of a table in releaseAndRemoveAllSegments method (#12297)
• Enhancement to Maintain pool selection for the minimizeDataMovement instance partition assignment strategy (#11953)
• Upsert enhancement to assign segments for with respect to ideal state (#11628)
• Observability change to export Additional Upsert Metrics to Prom (#11660)
• Observibility enhancement to add CPU metrics for minion purge task (#12337)
• Add HttpHeaders in broker event listener requestContext (#12258)

Bugfixes, Refactoring, Cleanups, Deprecations

• Upsert bugfix in "rewind()" for CompactedPinotSegmentRecordReader (#12329)
• Fix error message format for Preconditions.checks failures(#12327)
• Bugfix to distribute Pinot as a multi-release JAR (#12131, #12300)
• Fixes in upsert metadata manager (#12319)
• Security fix to allow querying tables with table-type suffix (#12310)
• Bugfix to ensure tagConfigOverride config is null for upsert tables (#12233 and #12311)
• Increased fetch timeout for Kineses to prevent stuck kinesis consumers(#12214)
• Fixes to catch-all Regex for JXM -> Prom Exporter (#12073 and #12295)
• Fixes lucene index errors when using QuickStart (#12289)
• Null handling bugfix for sketch group-by queries (#12259)
• Null pointer exception fixes in Controller SQL resource (#12211)
• Synchronization fixes to replace upsert segments (#12105 and #12241)
• Bugfix for S3 connection pool error when AWS session tokens expire after an hour (#12221)
• FileWriter fixes to append headerline only for required formats like csv (#12208)
• Security bugfix for pulsar OAuth2 authentication (#12195)
• Bugfix to appropriately compute "segment.flush.threshold.size" when force-committing realtime segments (#12188)
• Fixes rebalance converge check that reports success before rebalance completes (#12182)
• Fixes upsertPrimaryKeysCount metric reporting when table is deleted (#12169)
• Update LICENSE-binary for commons-configuration2 upgrade (#12165)
• Improve error logging when preloading segments not exist on server (#12153)
• Fixes to file access resource leaks (#12129)
• Ingestion bugfix to avoid unnecessary transformers in CompositeTransformer (#12138)
• Improve logging to print OS name during service statup (#12135)
• Improve logging in multiple files (#12134, #12137, #12127, #12121)
• Test fixes for ExprMinMaxRewriterTest.testQueryRewrite (#12047)
• Fixes default path of log4j in helmchart (#12069, #12083)
• Fix default brokerUpdateFrequencyInMillis for connector (#12093)
• Updates to README file (#12075)
• Fix to remove unnecessary locking during segment preloading (#12077)
• Fix bug with silently ignoring force commit call failures (#12044)
• Upsert bugfix to allow optional segments that can be skipped by servers without failing the query (#11978)
• Fix incorrect handling of consumer creation errors (#12045)
• Fix the memory leak issue on CommonsConfigurationUtils (#12056)
• Fix rebalance on upsert table (#12054)
• Add new Transformer to transform -0.0 and NaN (#12032)
• Improve inverted index validation in table config to enhance user experience (#12043)
• Fixes test flakiness by replacing HashSet/HashMap with LinkedHashSet/LinkedHashMap (#11941)
• Flaky test fix for ServerRoutingStatsManagerTest.testQuerySubmitAndCompletionStats (#12029)
• Fix derived column from MV column (#12028)
• Support for leveraging StarTree index in conjunction with filtered aggregations (#11886)
• Improves tableConfig validation for enabling size based threshold for realtime tables (#12016)
• Fix flaky PinotTenantRestletResourceTest (#12026)
• Fix flaky Fix PinotTenantRestletResourceTest (#12019)
• Fix the race condition of concurrent modification to segment data managers (#12004)
• Fix the misuse of star-tree when all predicates are always false under OR (#12003)
• Fix the test failures caused by instance drop failure (#12002)
• Fix fromULL scalar function (#11995)
• Fix to exclude module-info.class during shade operations (#11975)
• Fix the wrong import for Preconditions (#11979)
• Add check for illegal character '/' in taskName (#11955)
• Bugfix to only register new segments when it's fully initalized by partitionUpsertMetadataManager (#11964)
• Obervability fix to add logs to track sequence of events for table creation (#11946)
• Fix the NPE in minimizeDataMovement instance assignment strategy (#11952)
• Fix to add catch all logging for exception during DQL/DML process (#11944)
• Fix bug where we don't handle cases that a upsert table has both upsert deletion and upsert ttl configs (#11791)
• Removing direct dependencies on commons-logging and replacing with jcl-over-slf4j (#11920)
• Fix NPE for IN clause on constant STRING dictionary (#11930)
• Fix flaky OfflineClusterIntegrationTest on server response size tests (#11926)
• Avoid npe when checking mirror server set assignment (#11915)
• Deprecate _segmentAssignmentStrategy in favor of SegmentsValidationAndRetentionConfig #11869
• Bugfix to capture auth phase timing even if access is denied (#11884)
• Bugfix to mark rows as invalid in case primary time column is out of range (#11907)
• Fix to radomize server port to avoid port already bind issue (#11861)
• Add LazyRow abstraction for previously indexed record (#11826)
• Config Validation for upsert table to not assign COMPLETED segments to another server (#11852)
• Bugfix to resolve dependency conflict in pinot-protobuf module (#11867)
• Fix case of useMultistageEngine property reference in JsonAsyncHttpPinotClientTransportFactory (#11820)
• Bugfix to add woodstox-core to pinot-s3 dependencies and fix stack trace (#11799)
• Fix to move pinot-segment-local test from unit test suite 1 to 2 (#11865)
• Observability fix to log upsert config when initializing the metadata manager (#11864)
• Fix to improve tests when errors are received in the consumer thread (#11858)
• Fix for flaky ArrayAgg test (#11860)
• Fix for flaky tests in TupleSelectionTransformFunctionsTest (#11848)
• Fix for arrayAgg null support (#11853)
• Fix the bug of reading decimal value stored in int32 or int64 (#11840)
• Remove duplicate pinot-integration-tests from unit test suite 2 (#11844)
• Fix for a null handling error in queries (#11829)
• Fix the way of fetching the segment zk metadata for task generators (#11832)
• Make testInvalidateCachedControllerLeader times based on getMinInvalidateIntervalMs (#11815)
• Update doap to reflect latest release (#11827)
• Clean up integration test pom file (#11817)
• Bugfix to exclude OFFLINE segments when reading server to segments map (#11818)
• Add tests for zstd compressed parquet files (#11808)
• Fix job submission time for reload and foce commit job (#11803)
• Remove actually unsupported config that selectively enable nullable columns (#10653)
• Fix LLCRealtimeClusterIntegrationTest.testReset (#11806)
• Use expected version in api for table config read modify write change (#11782)
• Move jobId out of rebalanceConfig (#11790)
• Fix PeerServerSegmentFinder not respecting HTTPS port (#11752)
• Enhanced geospatial v2 integration tests (#11741)
• Add integration test for rebalance in upsert tables (#11568)
• Fix trivy CI issue (#11757)
• Cleanup rebalance configs by adding a RebalanceConfig class (#11730)
• Error handling to throw exception when schema name doesn't match table name during table creation (#11591)
• Fix a protobuf comment to be more precise (#11735)
• Move scala dependencies to root pom (#11671)
• Fix ProtoBuf inputformat plug-in handling for null values (#11723)
• Bugfix where segment download URI is invalid after same CRC refresh using tar push (#11720)
• Fix in TableCacheTest (#11717)
• Add more test for broker jersey bounded thread pool (#11705)
• Fix bug in gapfill with SumAvgGapfillProcessor. (#11714)
• Bugfix to allow GcsPinotFS to work with granular permissions (#11655)
• Fix default log4j2 config file path in helm chart (#11707)
• Refactor code and doc occurrences of argmin/max -> exprmin/max (#11700)
• Make constructor and functions public to be used from scheduler plugins (#11699)
• Bugfix to change json_format to return java null when java null is received (#11673)
• Fix the potential access to upsert metadata manager after it is closed (#11692)
• Bugfix to use isOptional instead of the deprecated hasOptional Keyword (#11682)
• Fix logging issue in RealtimeTableDataManager (#11693)
• Cleanup some reader/writer logic for raw forward index (#11669)
• Do not execute spotless in Java 21 (#11670)
• Update license-maven-plugin (#11665)
• Bugfix to allow deletion of local files with special characters (#11664)
• Clean up CaseTransformFunction::constructStatementListLegacy. (#11339)
• Bugfix to force FileChannel to commit data to disk (#11625)
• Remove the old deprecated commit end without metadata (#11662)
• Fix for a jackson vulnerability (#11619)
• Refactor BasicAuthUtils from pinot-core to pinot-common and remove pinot-core dependency from pinot-jdbc-client (#11620)
• Bugfix to support several extensions for different indexes (#11600)
• Fix the alias handling in single-stage engine (#11610)
• Fix to use constant null place holder (#11615)
• Refactor to move all BlockValSet into the same package (#11616)
• Remove deprecated Request class from pinot-java-client (#11614)
• Refactoring to remove old thirdeye files. (#11609)
• Testing fix to use builder method in integration test (#11564)
• Fix the broken Pinot JDBC client. (#11606)
• Bugfix to change the Forbidden error to Unauthorized (#11501)
• Fix for schema add UI issue that passing wrong data in the request header (#11602)
• Remove/Deprecate HLC handling code (#11590)
• Fix the bug of using push time to identify new created segment (#11599)
• Bugfix in CSVRecordReader when using line iterator (#11581)
• Remove split commit and some deprecated config for real-time protocol on controller (#11663)Improved validation for single argument aggregation functions (#11556)
• Fix to not emit lag once tabledatamanager shutdown (#11534)
• Bugfix to fail reload if derived columns can't be created (#11559)
• Fix the double unescape of property value (#12405)
• Fix for the backward compatible issue that existing metadata may contain unescaped characters (#12393)
• Skip invalid json string rather than throwing error during json indexing (#12238)
• Fixing the multiple files concurrent write issue when reloading SSLFactory (#12384)
• Fix memory leaking issue by making thread local variable static (#12242)
• Bugfixfor Upsert compaction task generator (#12380)
• Log information about SSLFactory renewal (#12357)
• Fixing array literal usage for vector (#12365)
• Fixing quickstart table baseballStats minion ingestion (#12371)
• Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction (#12347)
• Bugfix to skip instead of throwing error on 'getValidDocIdMetadata' (#12360)
• Fix to clean up segment metadata when the associated segment gets deleted from remote store (#12350)
• Fix getBigDecimal() scale throwing rounding error (#12326)
• Workaround fix for the problem of Helix sending 2 transitions for CONSUMING -> DROPPED (#12351)
• Bugfix for making nonLeaderForTables exhaustive (#12345)
• Bugfixes for graceful interrupt handling of mutable lucene index (#11558,#12274)
• Remove split commit and some deprecated config for real-time protocol on controller (#11663)
• Update the table config in quick start (#11652)
• Deprecate k8s skaffold scripts and move helm to project root directory (#11648)
• Fix NPE in SingleColumnKeySelector (#11644)
• Simplify kafka build and remove old kafka 0.9 files (#11638)
• Adding comments for docker image tags, make a hyper link of helmChart from root directory (#11646)
• Improve the error response on controller. (#11624)
• Simplify authrozation for table config get (#11640)
• Bugfix to remove segments with empty download url in UpsertCompactionTask (#12320)
• Test changes to make taskManager resources protected for derived classes to override in their setUp() method. (#12335)

Backward incompatible Changes

• Fix a race condition for upsert compaction (#12346). Notes on backward incompatibility below:

  • This PR is introducing backward incompatibility for UpsertCompactionTask. Previously, we allowed to configure the compaction task without the snapshot enabled. We found that using in-memory based validDocIds is a bit dangerous as it will not give us the consistency (e.g. fetching validDocIds bitmap while the server is restarting & updating validDocIds).

    We now enforce the enableSnapshot=true for UpsertCompactionTask if the advanced customer wants to run the compaction with the in-memory validDocId bitmap.

    {
      "upsertConfig": {
        "mode": "FULL",
        "enableSnapshot": true
      }
    }
    ...
    "task": {
      "taskTypeConfigsMap": {
        "UpsertCompactionTask": {
          "schedule": "0 */5 * ? * *",
          "bufferTimePeriod": "7d",
          "invalidRecordsThresholdPercent": "30",
          "invalidRecordsThresholdCount": "100000",
          "invalidDocIdsType": "SNAPSHOT/IN_MEMORY/IN_MEMORY_WITH_DELETE"
        }
      }
    }
    

    Also, we allow to configure invalidDocIdsType to UpsertCompactionTask for advanced user.

    1. snapshot: Default validDocIds type. This indicates that the validDocIds bitmap is loaded from the snapshot from the Pinot segment. UpsertConfig's enableSnapshot must be enabled for this type.
       1. onHeap: the validDocIds bitmap will be fetched from the server.
       2. onHeapWithDelete: the validDocIds bitmap will be fetched from the server. This will also take account into the deleted documents. UpsertConfig's deleteRecordColumn must be provided for this type.
• Removal of the feature flag allow.table.name.with.database (#12402)
• Incompatible API fix to remove table state update operation in GET call (#11621)
• Use string to represent BigDecimal datatype in JSON response (#11716)
• Single quoted literal will not have its type auto-derived to maintain SQL compatibility (#11763)
• Changes to always use split commit on server and disables the option to disable it (#11680, #11687)
• Change to not allow NaN as default value for Float and Double in Schemas (#11661)
• Code cleanup and refactor that removes TableDataManagerConfig (#12189)
• Fix partition handling for consistency of values between query and segment (#12115)
• Changes for migration to commons-configuration2 (#11985)
• Cleanup to simplify the upsert metadata manager constructor (#12120)
• Fixes typo in pom.xml (#11997)
• JDBC Driver fixes to support Jetbrains Intellij/Datagrip database tooling (#11814)
• Fix regression in ForwardIndexType for noDictionaryConfig and noDictionaryColumns (#11784)
• Separate pr test scripts and codecov (#11804)
• Bugfix to make reload status should only count online/consuming segments (#11787)
• Fix flaky TableViewsTest (#11770)
• Fix a flaky test (#11771)
• Cleanup to fee more disk for trivy job (#11780)
• Fix schema name in table config during controller startup (#11574)
• Prevent NPE when attempt to fetch partition information fails (#11769)
• Added UTs for null handling in CaseTransform function. (#11721)
• Bugfix to disallow peer download when replication is < 2 (#11469)
• Update s todocker image and github action scripts (#12378)
• Enhancements to queries test framework (#12215)

Library Upgrades and dependencies

• update maven-jar-plugin and maven-enforcer-plugin version (#11637)
• Update testng as the test provider explicitly instead of relying on the classpath. (#11612)
• Update compatibility verifier version (#11684)
• Upgrade Avro dependency to 1.10.2 (#11698)
• Upgrade testng version to 7.8.0 (#11462)
• Update lombok version and config (#11742)
• Upgrading Apache Helix to 1.3.1 version (#11754)
• Upgrade spark from 3.2 to 3.5 (#11702)
• Added commons-configuration2 dependency. (#11792)
• Upgrade confluent libraries to 7.2.6 to fix some errors related to optional proto fields (#11753)
• Upgrade lucene to 9.8.0 and upgrade text index version (#11857)
• Upgrade the PinotConfiguartion to commons-configuartion2(#11916)
• Pre PinotConfig commons-configuartions2 upgrade (#11868)
• Bump commons-codec:commons-codec from 1.15 to 1.16.0 (#12204)
• Bump flink.version from 1.12.0 to 1.14.6 (#12202)
• Bump com.yscope.clp:clp-ffi from 0.4.3 to 0.4.4 (#12203)
• Bump org.apache.spark:spark-launcher_2.12 from 3.2.1 to 3.5.0 (#12199)
• Bump io.grpc:grpc-context from 1.59.0 to 1.60.1 (#12198)
• Bump com.azure:azure-core from 1.37.0 to 1.45.1 (#12193)
• Bump org.freemarker:freemarker from 2.3.30 to 2.3.32 (#12192)
• Bump com.google.auto.service:auto-service from 1.0.1 to 1.1.1 (#12183)
• Bump dropwizard-metrics.version from 4.2.22 to 4.2.23 (#12178)
• Bump org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0 (#12170)
• Bump com.gradle:common-custom-user-data-maven-extension (#12171)
• Bump org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14 (#12172)
• Bump org.glassfish.tyrus.bundles:tyrus-standalone-client (#12162)
• Bump com.google.api.grpc:proto-google-common-protos (#12159)
• Bump org.apache.datasketches:datasketches-java from 4.1.0 to 5.0.0 (#12161)
• Bump org.apache.zookeeper:zookeeper from 3.6.3 to 3.7.2 (#12152)
• Bump org.apache.commons:commons-collections4 from 4.1 to 4.4 (#12149)
• Bump log4j.version from 2.20.0 to 2.22.0 (#12143)
• Bump com.github.luben:zstd-jni from 1.5.5-6 to 1.5.5-11 (#12125)
• Bump com.google.guava:guava from 32.0.1-jre to 32.1.3-jre (#12124)
• Bump org.apache.avro:avro from 1.10.2 to 1.11.3 (#12116)
• Bump org.apache.maven.plugins:maven-assembly-plugin from 3.1.1 to 3.6.0 (#12109)
• Bump net.java.dev.javacc:javacc from 7.0.10 to 7.0.13 (#12103)
• Bump com.azure:azure-identity from 1.8.1 to 1.11.1 (#12095)
• Bump xml-apis:xml-apis from 1.4.01 to 2.0.2 (#12082)
• Bump up the parquet version to 1.13.1 (#12076)
• Bump io.grpc:grpc-context from 1.14.0 to 1.59.0 (#12034)
• Bump org.reactivestreams:reactive-streams from 1.0.3 to 1.0.4 (#12033)
• Bump org.codehaus.mojo:appassembler-maven-plugin from 1.10 to 2.1.0 (#12030)
• Bump com.google.code.findbugs:jsr305 from 3.0.0 to 3.0.2 (#12031)
• Bump org.jacoco:jacoco-maven-plugin from 0.8.9 to 0.8.11 (#12024)
• Bump dropwizard-metrics.version from 4.2.2 to 4.2.22 (#12022)
• Bump grpc.version from 1.53.0 to 1.59.0 (#12023)
• Bump com.google.code.gson:gson from 2.2.4 to 2.10.1 (#12009)
• Bump net.nicoulaj.maven.plugins:checksum-maven-plugin from 1.8 to 1.11 (#12008)
• Bump circe.version from 0.14.2 to 0.14.6 (#12006)
• Bump com.mercateo:test-clock from 1.0.2 to 1.0.4 (#12005)
• Bump simpleclient_common.version from 0.8.1 to 0.16.0 (#11986)
• Bump com.jayway.jsonpath:json-path from 2.7.0 to 2.8.0 (#11987)
• Bump commons-net:commons-net from 3.1 to 3.10.0 (#11982)
• Bump org.scalatest:scalatest-maven-plugin from 1.0 to 2.2.0 (#11973)
• Bump io.netty:netty-bom from 4.1.94.Final to 4.1.100.Final (#11972)
• Bump com.google.errorprone:error_prone_annotations from 2.3.4 to 2.23.0 (#11905)
• Bump net.minidev:json-smart from 2.4.10 to 2.5.0 (#11875)
• Bump org.yaml:snakeyaml from 2.0 to 2.2 (#11876)
• Bump browserify-sign in /pinot-controller/src/main/resources (#11896)
• Bump org.easymock:easymock from 4.2 to 5.2.0 (#11854)
• Bump org.codehaus.mojo:exec-maven-plugin from 1.5.0 to 3.1.0 (#11856)
• Bump com.github.luben:zstd-jni from 1.5.2-3 to 1.5.5-6 (#11855)
• Bump aws.sdk.version from 2.20.94 to 2.20.137 (#11463)
• Bump org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.4 (#11678)

What's changed

Major features and updates

Multistage Engine

Multistage engine new features:

Support for Window Functions
Initial (phase 1) Query runtime for window functions with ORDER BY within the OVER() clause (#10449)
Support for the ranking ROW_NUMBER() window function (#10527, #10587)
Set Operations Support
Support SetOperations (UNION, INTERSECT, MINUS) compilation in query planner (#10535)
Timestamp and Date Operations
Support TIMESTAMP type and date ops functions (#11350)
Aggregate Functions
Support more aggregation functions that are currently implementable (#11208)
Support multi-value aggregation functions (#11216)
Support Sketch based functions (#)
Make Intermediate Stage Worker Assignment Tenant Aware (#10617)
Evaluate literal expressions during query parsing, enabling more efficient query execution. (#11438)
Added support for partition parallelism in partitioned table scans, allowing for more efficient data retrieval (#11266).
[multistage]Adding more tuple sketch scalar functions and integration tests (#11517)

Multistage engine enhancements

Turn on v2 engine by default (#10543)
Introduced the ability to stream leaf stage blocks for more efficient data processing (#11472).
Early terminate SortOperator if there is a limit (#11334)
Implement ordering for SortExchange (#10408)
Table level Access Validation, QPS Quota, Phase Metrics for multistage queries (#10534)
Support partition based leaf stage processing (#11234)
Populate queryOption down to leaf (#10626)
Pushdown explain plan queries from the controller to the broker (#10505)
Enhanced the multi-stage group-by executor to support limiting the number of groups, improving query performance and resource utilization (#11424).
Improved resilience and reliability of the multi-stage join operator, now with added support for hash join right table protection (#11401).

Multistage engine bug fixes

Fix Predicate Pushdown by Using Rule Collection (#10409)
Try fixing mailbox cancel race condition (#10432)
Catch Throwable to Propagate Proper Error Message (#10438)
Fix tenant detection issues (#10546)
Handle Integer.MIN_VALUE in hashCode based FieldSelectionKeySelector (#10596)
Improve error message in case of non-existent table queried from the controller (#10599)
Derive SUM return type to be PostgreSQL compatible (#11151)

Index SPI

Add the ability to include new index types at runtime in Apache Pinot. This opens the ability of adding third party indexes, including proprietary indexes. More details here

Null value support for pinot queries

NULL support for ORDER BY, DISTINCT, GROUP BY, value transform functions and filtering.

Upsert enhancements

Delete support in upsert enabled tables (#10703)

Support added to extend upserts and allow deleting records from a realtime table. The design details can be found here.

Preload segments with upsert snapshots to speedup table loading (#11020)

Adds a feature to preload segments from a table that uses the upsert snapshot feature. The segments with validDocIds snapshots can be preloaded in a more efficient manner to speed up the table loading (thus server restarts).

TTL configs for upsert primary keys (#10915)

Adds support for specifying expiry TTL for upsert primary key metadata cleanup.

Segment compaction for upsert real-time tables (#10463)

Adds a new minion task to compact segments belonging to a real-time table with upserts.

Pinot Spark Connector for Spark3 (#10394)

• Added spark3 support for Pinot Spark Connector (#10394)
• Also added support to pass pinot query options to spark connector (#10443)

PinotDataBufferFactory and new PinotDataBuffer implementations (#10528)

Adds new implementations of PinotDataBuffer that uses Unsafe java APIs and foreign memory APIs. Also added support for PinotDataBufferFactory to allow plugging in custom PinotDataBuffer implementations.

Query functions enhancements

• Add PercentileKLL aggregation function (#10643)
• Support for ARG_MIN and ARG_MAX Functions (#10636)
• refactor argmin/max to exprmin/max and make it calcite compliant (#11296)
• Integer Tuple Sketch support (#10427)
• Adding vector scalar functions (#11222)
• [feature] multi-value datetime transform variants (#10841)
• FUNNEL_COUNT Aggregation Function (#10867)
• [multistage] Add support for RANK and DENSE_RANK ranking window functions (#10700)
• add theta sketch scalar (#11153)
• Register dateTimeConverter,timeConvert,dateTrunc, regexpReplace to v2 functions (#11097)
• Add extract(quarter/dow/doy) support (#11388)
• Funnel Count - Multiple Strategies (no partitioning requisites) (#11092)
• Add Boolean assertion transform functions. (#11547)

JSON and CLP encoded message ingestion and querying

• Add clpDecode transform function for decoding CLP-encoded fields. (#10885)
• Add CLPDecodeRewriter to make it easier to call clpDecode with a column-group name rather than the individual columns. (#11006)
• Add SchemaConformingTransformer to transform records with varying keys to fit a table's schema without dropping fields. (#11210)

Tier level index config override (#10553)

Allows overriding index configs at tier level, allowing for more flexible index configurations for different tiers.

Ingestion connectors and features

• Kinesis stream header extraction (#9713)
• Extract record keys, headers and metadata from Pulsar sources (#10995)
• Realtime pre-aggregation for Distinct Count HLL & Big Decimal (#10926)
• Added support to skip unparseable records in the csv record reader (#11487)
• Null support for protobuf ingestion. (#11553)

UI enhancements

• Adds persistence of authentication details in the browser session. This means that even if you refresh the app, you will still be logged in until the authentication session expires (#10389)
• AuthProvider logic updated to decode the access token and extract user name and email. This information will now be available in the app for features to consume. (#10925)

Pinot docker image improvements and enhancements

• Make Pinot base build and runtime images support Amazon Corretto and MS OpenJDK (#10422)
• Support multi-arch pinot docker image (#10429)
• Update dockerfile with recent jdk distro changes (#10963)

Operational improvements

Rebalance

• Rebalance status API (#10359)
• Tenant level rebalance API Tenant rebalance and status tracking APIs (#11128)

Config to use customized broker query thread pool (#10614)

Added new configuration options below which allow use of a bounded thread pool and allocate capacities for it.

pinot.broker.enable.bounded.http.async.executor
pinot.broker.http.async.executor.max.pool.size
pinot.broker.http.async.executor.core.pool.size
pinot.broker.http.async.executor.queue.size

This feature allows better management of broker resources.

Drop results support (#10419)

Adds a parameter to queryOptions to drop the resultTable from the response. This mode can be used to troubleshoot a query (which may have sensitive data in the result) using metadata only.

Make column order deterministic in segment (#10468)

In segment metadata and index map, store columns in alphabetical order so that the result is deterministic. Segments generated before/after this PR will have different CRC, so during the upgrade, we might get segments with different CRC from old and new consuming servers. For the segment consumed during the upgrade, some downloads might be needed.

Allow configuring helix timeouts for EV dropped in Instance manager (#10510)

Adds options to configure helix timeouts
external.view.dropped.max.wait.ms`` - The duration of time in milliseconds to wait for the external view to be dropped. Default - 20 minutes. external.view.check.interval.ms`` - The period in milliseconds in which to ping ZK for latest EV state.

Enable case insensitivity by default (#10771)

This PR makes Pinot case insensitive be default, and removes the deprecated property enable.case.insensitive.pql

Newly added APIs and client methods

• Add Server API to get tenant pools (#11273)
• Add new broker query point for querying multi-stage engine (#11341)
• Add a new controller endpoint for segment deletion with a time window (#10758)
• New API to get tenant tags (#10937)
• Instance retag validation check api (#11077)
• Use PUT request to enable/disable table/instance (#11109)
• Update the pinot tenants tables api to support returning broker tagged tables (#11184)
• Add requestId for BrokerResponse in pinot-broker and java-client (#10943)
• Provide results in CompletableFuture for java clients and expose metrics (#10326)

Cleanup and backward incompatible changes

High level consumers are no longer supported

• Cleanup HLC code (#11326)
• Remove support for High level consumers in Apache Pinot (#11017)

Type information preservation of query literals

• [feature] [backward-incompat] [null support # 2] Preserve null literal information in literal context and literal transform (#10380)
  String versions of numerical values are no longer accepted. For example, "123" won't be treated as a numerical anymore.

Controller job status ZNode path update

• Moving Zk updates for reload, force_commit to their own Znodes which … (#10451)
  The status of previously completed reload jobs will not be available after this change is deployed.

Metric names for mutable indexes to change

• Implement mutable index using index SPI (#10687)
  Due to a change in the IndexType enum used for some logs and metrics in mutable indexes, the metric names may change slightly.

Update in controller API to enable / disable / drop instances

• Update getTenantInstances call for controller and separate POST operations on it (#10993)

Change in substring query function definition

• Change substring to comply with standard sql definition (#11502)

Full list of features added

• Allow queries on multiple tables of same tenant to be executed from controller UI #10336
• Encapsulate changes in IndexLoadingConfig and SegmentGeneratorConfig #10352
• [Index SPI] IndexType (#10191)
• Simplify filtered aggregate transform operator creation (#10410)
• Introduce BaseProjectOperator and ValueBlock (#10405)
• Add support to create realtime segment in local (#10433)
• Refactor: Pass context instead on individual arguments to operator (#10413)
• Add "processAll" mode for MergeRollupTask (#10387)
• Upgrade h2 version from 1.x to 2.x (#10456)
• Added optional force param to the table configs update API (#10441)
• Enhance broker reduce to handle different column names from server response (#10454)
• Adding fields to enable/disable dictionary optimization. (#10484)
• Remove converted H2 type NUMERIC(200, 100) from BIG_DECIMAL (#10483)
• Add JOIN support to PinotQuery (#10421)
• Add testng on verifier (#10491)
• Clean up temp consuming segment files during server start (#10489)
• make pinot k8s sts and deployment start command configurable (#10509)
• Fix Bottleneck for Server Bootstrap by Making maxConnsPerRoute Configurable (#10487)
• Type match between resultType and function's dataType (#10472)
• create segment zk metadata cache (#10455)
• Allow ValueBlock length to increase in TransformFunction (#10515)
• Allow configuring helix timeouts for EV dropped in Instance manager (#10510)
• Enhance error reporting (#10531)
• Combine "GET /segments" API & "GET /segments/{tableName}/select" (#10412)
• Exposed the CSV header map as part of CSVRecordReader (#10542)
• Moving Zk updates for reload,force_commit to their own Znodes which will spread out Zk write load across jobTypes (#10451)
• Enabling dictionary override optimization on the segment reload path as well. (#10557)
• Make broker's rest resource packages configurable (#10588)
• Check EV not exist before allowing creating the table (#10593)
• Adding an parameter (toSegments) to the endSegmentReplacement API (#10630)
• update target tier for segments if tierConfigs is provided (#10642)
• Add support for custom compression factor for Percentile TDigest aggregation functions (#10649)
• Utility to convert table config into updated format (#10623)
• Segment lifecycle event listener support (#10536)
• Add server metrics to capture gRPC activity (#10678)
• Separate and parallelize BloomFilter based semgment pruner (#10660)
• API to expose the contract/rules imposed by pinot on tableConfig #10655
• Add description field to metrics in Pinot (#10744)
• changing the dedup store to become pluggable #10639
• Make the TimeUnit in the DATETRUNC function case insensitive. (#10750)
• [feature] Consider tierConfigs when assigning new offline segment #10746
• Compress idealstate according to estimated size #10766
• 10689: Update for pinot helm release version 0.2.7 (#10723)
• Fail the query if a filter's rhs contains NULL. (#11188)
• Support Off Heap for Native Text Indices (#10842)
• refine segment reload executor to avoid creating threads unbounded #10837
• compress nullvector bitmap upon seal (#10852)
• Enable case insensitivity by default (#10771)
• Push out-of-order events metrics for full upsert (#10944)
• [feature] add requestId for BrokerResponse in pinot-broker and java-client #10943
• Provide results in CompletableFuture for java clients and expose metrics #10326
• Add minion observability for segment upload/download failures (#10978)
• Enhance early terminate for combine operator (#10988)
• Add fromController method that accepts a PinotClientTransport (#11013)
• Ensure min/max value generation in the segment metadata. (#10891)
• Apply some allocation optimizations on GrpcSendingMailbox (#11015)
• When enable case-insensitive, don't allow to add newly column name which have the same lowercase name with existed columns. (#10991)
• Replace Long attributes with primitive values to reduce boxing (#11059)
• retry KafkaConsumer creation in KafkaPartitionLevelConnectionHandler.java (#253) (#11040)
• Support for new dataTime format in DateTimeGranularitySpec without explicitly setting size (#11057)
• Returning 403 status code in case of authorization failures (#11136)
• Simplify compatible test to avoid test against itself (#11163)
• Updated code for setting value of segment min/max property. (#10990)
• Add stat to track number of segments that have valid doc id snapshots (#11110)
• Add brokerId and brokerReduceTimeMs to the broker response stats (#11142)
• safely multiply integers to prevent overflow (#11186)
• Move largest comparison value update logic out of map access (#11157)
• Optimize DimensionTableDataManager to abort unnecesarry loading (#11192)
• Refine isNullsLast and isAsc functions. (#11199)
• Update the pinot tenants tables api to support returning broker tagged tables (#11184)
• add multi-value support for native text index (#11204)
• Add percentiles report in QuerySummary (#11299)
• Add meter for broker responses with unavailable segments (#11301)
• Enhance Minion task management (#11315)
• add additional lucene index configs (#11354)
• Add DECIMAL data type to orc record reader (#11377)
• add configuration to fail server startup on non-good status checker (#11347)
• allow passing freshness checker after an idle threshold (#11345)
• Add broker validation for hybrid tableConfig creation (#7908)
• Support partition parallelism for partitioned table scan (#11266)

Vulnerability fixes, bugfixes, cleanups and deprecations

• Remove support for High level consumers in Apache Pinot (#11017)
• Fix JDBC driver check for username (#10416)
• [Clean up] Remove getColumnName() from AggregationFunction interface (#10431)
• fix jersey TerminalWriterInterceptor MessageBodyWriter not found issue (#10462)
• Bug fix: Start counting operator execution time from first NoOp block (#10450)
• Fix unavailable instances issues for StrictReplicaGroup (#10466)
• Change shell to bash (#10469)
• Fix the double destroy of segment data manager during server shutdown (#10475)
• Remove "isSorted()" precondition check in the ForwardIndexHandler (#10476)
• Fix null handling in streaming selection operator (#10453)
• Fix jackson dependencies (#10477)
• Startree index build enhancement (#10905)
• optimize queries where lhs and rhs of predicate are equal (#10444)
• Trivial fix on a warning detected by static checker (#10492)
• wait for full segment commit protocol on force commit (#10479)
• Fix bug and add test for noDict -> Dict conversion for sorted column (#10497)
• Make column order deterministic in segment (#10468)
• Type match between resultType and function's dataType (#10472)
• Allow empty segmentsTo for segment replacement protocol (#10511)
• Use string as default compatible type for coalesce (#10516)
• Use threadlocal variable for genericRow to make the MemoryOptimizedTable threadsafe (#10502)
• Fix shading in spark2 connector pom file (#10490)
• Fix ramping delay caused by long lasting sequence of unfiltered messa… (#10418)
• Do not serialize metrics in each Operator (#10473)
• Make pinot-controller apply webpack production mode when bin-dist profile is used. (#10525)
• Fix FS props handling when using /ingestFromUri (#10480)
• Clean up v0_deprecated batch ingestion jobs (#10532)
• Deprecate kafka 0.9 support (#10522)
• safely multiply integers to prevent overflow (#11186)
• Reduce timeout for codecov and not fail the job in any case (#10547)
• Fix DataTableV3 serde bug for empty array (#10583)
• Do not record operator stats when tracing is enabled (#10447)
• Forward auth token for logger APIs from controller to other controllers and brokers (#10590)
• Bug fix: Partial upsert default strategy is null (#10610)
• Fix flaky test caused by EV check during table creation (#10616)
• Fix withDissabledTrue typo (#10624)
• Cleanup unnecessary mailbox id ser/de (#10629)
• no error metric for queries where all segments are pruned (#10589)
• bug fix: to keep QueryParser thread safe when handling many read requests on class RealtimeLuceneTextIndex (#10620)
• Fix static DictionaryIndexConfig.DEFAULT_OFFHEAP being actually onheap (#10632)
• 10567: [cleanup pinot-integration-test-base], clean query generations and some other refactoring. (#10648)
• Fixes backward incompatability with SegmentGenerationJobSpec for segment push job runners (#10645)
• Bug fix to get the toSegments list correctly (#10659)
• 10661: Fix for failing numeric comparison in where clause for IllegalStateException. (#10662)
• Fixes partial upsert not reflecting multiple comparison column values (#10693)
• Fix Bug in Reporting Timer Value for Min Consuming Freshness (#10690)
• Fix typo of rowSize -> columnSize (#10699)
• update segment target tier before table rebalance (#10695)
• Fix a bug in star-tree filter operator which can incorrecly filter documents (#10707)
• Enhance the instrumentation for a corner case where the query doesn't go through DocIdSetOp (#10729)
• bug fix: add missing properties when edit instance config (#10741)
• Making segmentMapper do the init and cleanup of RecordReader (#10874)
• Fix githubEvents table for quickstart recipes (#10716)
• Minor Realtime Segment Commit Upload Improvements (#10725)
• Return 503 for all interrupted queries. Refactor the query killing code. (#10683)
• Add decoder initialization error to the server's error cache (#10773)
• bug fix: add @JsonProperty to SegmentAssignmentConfig (#10759)
• ensure we wait the full no query timeout before shutting down (#10784)
• Clean up KLL functions with deprecated convention (#10795)
• Redefine the semantics of SEGMENT_STREAMED_DOWNLOAD_UNTAR_FAILURES metric to count individual segment fetch failures. (#10777)
• fix excpetion during exchange routing causes stucked pipeline (#10802)
• [bugfix] fix floating point and integral type backward incompatible issue (#10650)
• [pinot-core] Start consumption after creating segment data manager (#11227)
• Fix IndexOutOfBoundException in filtered aggregation group-by (#11231)
• Fix null pointer exception in segment debug endpoint #11228
• Clean up RangeIndexBasedFilterOperator. (#11219)
• Fix the escape/unescape issue for property value in metadata (#11223)
• Fix a bug in the order by comparator (#10818)
• Keeps nullness attributes of merged in comparison column values (#10704)
• Add required JSON annotation in H3IndexResolution (#10792)
• Fix a bug in SELECT DISTINCT ORDER BY. (#10827)
• jsonPathString should return null instead of string literal "null" (#10855)
• Bug Fix: Segment Purger cannot purge old segments after schema evolution (#10869)
• Fix #10713 by giving metainfo more priority than config (#10851)
• Close PinotFS after Data Manager Shutdowns (#10888)
• bump awssdk version for a bugfix on http conn leakage (#10898)
• Fix MultiNodesOfflineClusterIntegrationTest.testServerHardFailure() (#10909)
• Fix a bug in SELECT DISTINCT ORDER BY LIMIT. (#10887)
• Fix an integer overflow bug. (#10940)
• Return true when _resultSet is not null (#10899)
• Fixing table name extraction for lateral join queries (#10933)
• Fix casting when prefetching mmap'd segment larger than 2GB (#10936)
• Null check before closing reader (#10954)
• Fixes SQL wildcard escaping in LIKE queries (#10897)
• [Clean up] Do not count DISTINCT as aggregation (#10985)
• do not readd lucene readers to queue if segment is destroyed #10989
• Message batch ingestion lag fix (#10983)
• Fix a typo in snapshot lock (#11007)
• When extracting root-level field name for complex type handling, use the whole delimiter (#11005)
• update jersey to fix Denial of Service (DoS) (#11021)
• Update getTenantInstances call for controller and separate POST operations on it (#10993)
• update freemaker to fix Server-side Template Injection (#11019)
• format double 0 properly to compare with h2 results (#11049)
• Fix double-checked locking in ConnectionFactory (#11014)
• Remove presto-pinot-driver and pinot-java-client-jdk8 module (#11051)
• Make RequestUtils always return a string array when getTableNames (#11069)
• Fix BOOL_AND and BOOL_OR result type (#11033)
• [cleanup] Consolidate some query and controller/broker methods in integration tests (#11064)
• Fix grpc regression on multi-stage engine (#11086)
• Delete an obsolete TODO. (#11080)
• Minor fix on AddTableCommand.toString() (#11082)
• Allow using Lucene text indexes on mutable MV columns. (#11093)
• Allow offloading multiple segments from same table in parallel (#11107)
• Added serviceAccount to minion-stateless (#11095)
• Bug fix: TableUpsertMetadataManager is null (#11129)
• Fix reload bug (#11131)
• Allow extra aggregation types in RealtimeToOfflineSegmentsTask (#10982)
• Fix a bug when use range index to solve EQ predicate (#11146)
• Sanitise API inputs used as file path variables (#11132)
• Fix NPE when nested query doesn't have gapfill (#11155)
• Fix the NPE when query response error stream is null (#11154)
• Make interface methods non private, for java 8 compatibility (#11164)
• Increment nextDocId even if geo indexing fails (#11158)
• Fix the issue of consuming segment entering ERROR state due to stream connection errors (#11166)
• In TableRebalancer, remove instance partitions only when reassigning instances (#11169)
• Remove JDK 8 unsupported code (#11176)
• Fix compat test by adding -am flag to build pinot-integration-tests (#11181)
• dont duplicate register scalar function in CalciteSchema (#11190)
• Fix the storage quota check for metadata push (#11193)
• Delete filtering NULL support dead code paths. (#11198)
• [bugfix] Do not move real-time segments to working dir on restart (#11226)
• Fix a bug in ExpressionScanDocIdIterator for multi-value. (#11253)
• Exclude NULLs when PredicateEvaluator::isAlwaysTrue is true. (#11261)
• UI: fix sql query options seperator (#10770)
• Fix a NullPointerException bug in ScalarTransformFunctionWrapper. (#11309)
• [refactor] improve disk read for partial upsert handler (#10927)
• Fix the wrong query time when the response is empty (#11349)
• getMessageAtIndex should actually return the value in the streamMessage for compatibility (#11355)
• Remove presto jdk8 related dependencies (#11285)
• Remove special routing handling for multiple consuming segments (#11371)
• Properly handle shutdown of TableDataManager (#11380)
• Fixing the stale pinot ServerInstance in _tableTenantServersMap (#11386)
• Fix the thread safety issue for mutable forward index (#11392)
• Fix RawStringDistinctExecutor integer overflow (#11403)
• [logging] fix consume rate logging bug to respect 1 minute threshold (#11421)

What's changed

Major features and updates

Multistage Engine

Multistage engine new features:

Support for Window Functions
Initial (phase 1) Query runtime for window functions with ORDER BY within the OVER() clause (#10449)
Support for the ranking ROW_NUMBER() window function (#10527, #10587)
Set Operations Support
Support SetOperations (UNION, INTERSECT, MINUS) compilation in query planner (#10535)
Timestamp and Date Operations
Support TIMESTAMP type and date ops functions (#11350)
Aggregate Functions
Support more aggregation functions that are currently implementable (#11208)
Support multi-value aggregation functions (#11216)
Support Sketch based functions (#)
Make Intermediate Stage Worker Assignment Tenant Aware (#10617)
Evaluate literal expressions during query parsing, enabling more efficient query execution. (#11438)
Added support for partition parallelism in partitioned table scans, allowing for more efficient data retrieval (#11266).
[multistage]Adding more tuple sketch scalar functions and integration tests (#11517)

Multistage engine enhancements

Turn on v2 engine by default (#10543)
Introduced the ability to stream leaf stage blocks for more efficient data processing (#11472).
Early terminate SortOperator if there is a limit (#11334)
Implement ordering for SortExchange (#10408)
Table level Access Validation, QPS Quota, Phase Metrics for multistage queries (#10534)
Support partition based leaf stage processing (#11234)
Populate queryOption down to leaf (#10626)
Pushdown explain plan queries from the controller to the broker (#10505)
Enhanced the multi-stage group-by executor to support limiting the number of groups, improving query performance and resource utilization (#11424).
Improved resilience and reliability of the multi-stage join operator, now with added support for hash join right table protection (#11401).

Multistage engine bug fixes

Fix Predicate Pushdown by Using Rule Collection (#10409)
Try fixing mailbox cancel race condition (#10432)
Catch Throwable to Propagate Proper Error Message (#10438)
Fix tenant detection issues (#10546)
Handle Integer.MIN_VALUE in hashCode based FieldSelectionKeySelector (#10596)
Improve error message in case of non-existent table queried from the controller (#10599)
Derive SUM return type to be PostgreSQL compatible (#11151)

Index SPI

Add the ability to include new index types at runtime in Apache Pinot. This opens the ability of adding third party indexes, including proprietary indexes. More details here

Null value support for pinot queries

NULL support for ORDER BY, DISTINCT, GROUP BY, value transform functions and filtering.

Upsert enhancements

Delete support in upsert enabled tables (#10703)

Support added to extend upserts and allow deleting records from a realtime table. The design details can be found here.

Preload segments with upsert snapshots to speedup table loading (#11020)

Adds a feature to preload segments from a table that uses the upsert snapshot feature. The segments with validDocIds snapshots can be preloaded in a more efficient manner to speed up the table loading (thus server restarts).

TTL configs for upsert primary keys (#10915)

Adds support for specifying expiry TTL for upsert primary key metadata cleanup.

Segment compaction for upsert real-time tables (#10463)

Adds a new minion task to compact segments belonging to a real-time table with upserts.

Pinot Spark Connector for Spark3 (#10394)

• Added spark3 support for Pinot Spark Connector (#10394)
• Also added support to pass pinot query options to spark connector (#10443)

PinotDataBufferFactory and new PinotDataBuffer implementations (#10528)

Adds new implementations of PinotDataBuffer that uses Unsafe java APIs and foreign memory APIs. Also added support for PinotDataBufferFactory to allow plugging in custom PinotDataBuffer implementations.

Query functions enhancements

• Add PercentileKLL aggregation function (#10643)
• Support for ARG_MIN and ARG_MAX Functions (#10636)
• refactor argmin/max to exprmin/max and make it calcite compliant (#11296)
• Integer Tuple Sketch support (#10427)
• Adding vector scalar functions (#11222)
• [feature] multi-value datetime transform variants (#10841)
• FUNNEL_COUNT Aggregation Function (#10867)
• [multistage] Add support for RANK and DENSE_RANK ranking window functions (#10700)
• add theta sketch scalar (#11153)
• Register dateTimeConverter,timeConvert,dateTrunc, regexpReplace to v2 functions (#11097)
• Add extract(quarter/dow/doy) support (#11388)
• Funnel Count - Multiple Strategies (no partitioning requisites) (#11092)
• Add Boolean assertion transform functions. (#11547)

JSON and CLP encoded message ingestion and querying

• Add clpDecode transform function for decoding CLP-encoded fields. (#10885)
• Add CLPDecodeRewriter to make it easier to call clpDecode with a column-group name rather than the individual columns. (#11006)
• Add SchemaConformingTransformer to transform records with varying keys to fit a table's schema without dropping fields. (#11210)

Tier level index config override (#10553)

Allows overriding index configs at tier level, allowing for more flexible index configurations for different tiers.

Ingestion connectors and features

• Kinesis stream header extraction (#9713)
• Extract record keys, headers and metadata from Pulsar sources (#10995)
• Realtime pre-aggregation for Distinct Count HLL & Big Decimal (#10926)
• Added support to skip unparseable records in the csv record reader (#11487)
• Null support for protobuf ingestion. (#11553)

UI enhancements

• Adds persistence of authentication details in the browser session. This means that even if you refresh the app, you will still be logged in until the authentication session expires (#10389)
• AuthProvider logic updated to decode the access token and extract user name and email. This information will now be available in the app for features to consume. (#10925)

Pinot docker image improvements and enhancements

• Make Pinot base build and runtime images support Amazon Corretto and MS OpenJDK (#10422)
• Support multi-arch pinot docker image (#10429)
• Update dockerfile with recent jdk distro changes (#10963)

Operational improvements

Rebalance

• Rebalance status API (#10359)
• Tenant level rebalance API Tenant rebalance and status tracking APIs (#11128)

Config to use customized broker query thread pool (#10614)

Added new configuration options below which allow use of a bounded thread pool and allocate capacities for it.

pinot.broker.enable.bounded.http.async.executor
pinot.broker.http.async.executor.max.pool.size
pinot.broker.http.async.executor.core.pool.size
pinot.broker.http.async.executor.queue.size

This feature allows better management of broker resources.

Drop results support (#10419)

Adds a parameter to queryOptions to drop the resultTable from the response. This mode can be used to troubleshoot a query (which may have sensitive data in the result) using metadata only.

Make column order deterministic in segment (#10468)

In segment metadata and index map, store columns in alphabetical order so that the result is deterministic. Segments generated before/after this PR will have different CRC, so during the upgrade, we might get segments with different CRC from old and new consuming servers. For the segment consumed during the upgrade, some downloads might be needed.

Allow configuring helix timeouts for EV dropped in Instance manager (#10510)

Adds options to configure helix timeouts
external.view.dropped.max.wait.ms`` - The duration of time in milliseconds to wait for the external view to be dropped. Default - 20 minutes. external.view.check.interval.ms`` - The period in milliseconds in which to ping ZK for latest EV state.

Enable case insensitivity by default (#10771)

This PR makes Pinot case insensitive be default, and removes the deprecated property enable.case.insensitive.pql

Newly added APIs and client methods

• Add Server API to get tenant pools (#11273)
• Add new broker query point for querying multi-stage engine (#11341)
• Add a new controller endpoint for segment deletion with a time window (#10758)
• New API to get tenant tags (#10937)
• Instance retag validation check api (#11077)
• Use PUT request to enable/disable table/instance (#11109)
• Update the pinot tenants tables api to support returning broker tagged tables (#11184)
• Add requestId for BrokerResponse in pinot-broker and java-client (#10943)
• Provide results in CompletableFuture for java clients and expose metrics (#10326)

Cleanup and backward incompatible changes

High level consumers are no longer supported

• Cleanup HLC code (#11326)
• Remove support for High level consumers in Apache Pinot (#11017)

Type information preservation of query literals

• [feature] [backward-incompat] [null support # 2] Preserve null literal information in literal context and literal transform (#10380)
  String versions of numerical values are no longer accepted. For example, "123" won't be treated as a numerical anymore.

Controller job status ZNode path update

• Moving Zk updates for reload, force_commit to their own Znodes which … (#10451)
  The status of previously completed reload jobs will not be available after this change is deployed.

Metric names for mutable indexes to change

• Implement mutable index using index SPI (#10687)
  Due to a change in the IndexType enum used for some logs and metrics in mutable indexes, the metric names may change slightly.

Update in controller API to enable / disable / drop instances

• Update getTenantInstances call for controller and separate POST operations on it (#10993)

Change in substring query function definition

• Change substring to comply with standard sql definition (#11502)

Full list of features added

• Allow queries on multiple tables of same tenant to be executed from controller UI #10336
• Encapsulate changes in IndexLoadingConfig and SegmentGeneratorConfig #10352
• [Index SPI] IndexType (#10191)
• Simplify filtered aggregate transform operator creation (#10410)
• Introduce BaseProjectOperator and ValueBlock (#10405)
• Add support to create realtime segment in local (#10433)
• Refactor: Pass context instead on individual arguments to operator (#10413)
• Add "processAll" mode for MergeRollupTask (#10387)
• Upgrade h2 version from 1.x to 2.x (#10456)
• Added optional force param to the table configs update API (#10441)
• Enhance broker reduce to handle different column names from server response (#10454)
• Adding fields to enable/disable dictionary optimization. (#10484)
• Remove converted H2 type NUMERIC(200, 100) from BIG_DECIMAL (#10483)
• Add JOIN support to PinotQuery (#10421)
• Add testng on verifier (#10491)
• Clean up temp consuming segment files during server start (#10489)
• make pinot k8s sts and deployment start command configurable (#10509)
• Fix Bottleneck for Server Bootstrap by Making maxConnsPerRoute Configurable (#10487)
• Type match between resultType and function's dataType (#10472)
• create segment zk metadata cache (#10455)
• Allow ValueBlock length to increase in TransformFunction (#10515)
• Allow configuring helix timeouts for EV dropped in Instance manager (#10510)
• Enhance error reporting (#10531)
• Combine "GET /segments" API & "GET /segments/{tableName}/select" (#10412)
• Exposed the CSV header map as part of CSVRecordReader (#10542)
• Moving Zk updates for reload,force_commit to their own Znodes which will spread out Zk write load across jobTypes (#10451)
• Enabling dictionary override optimization on the segment reload path as well. (#10557)
• Make broker's rest resource packages configurable (#10588)
• Check EV not exist before allowing creating the table (#10593)
• Adding an parameter (toSegments) to the endSegmentReplacement API (#10630)
• update target tier for segments if tierConfigs is provided (#10642)
• Add support for custom compression factor for Percentile TDigest aggregation functions (#10649)
• Utility to convert table config into updated format (#10623)
• Segment lifecycle event listener support (#10536)
• Add server metrics to capture gRPC activity (#10678)
• Separate and parallelize BloomFilter based semgment pruner (#10660)
• API to expose the contract/rules imposed by pinot on tableConfig #10655
• Add description field to metrics in Pinot (#10744)
• changing the dedup store to become pluggable #10639
• Make the TimeUnit in the DATETRUNC function case insensitive. (#10750)
• [feature] Consider tierConfigs when assigning new offline segment #10746
• Compress idealstate according to estimated size #10766
• 10689: Update for pinot helm release version 0.2.7 (#10723)
• Fail the query if a filter's rhs contains NULL. (#11188)
• Support Off Heap for Native Text Indices (#10842)
• refine segment reload executor to avoid creating threads unbounded #10837
• compress nullvector bitmap upon seal (#10852)
• Enable case insensitivity by default (#10771)
• Push out-of-order events metrics for full upsert (#10944)
• [feature] add requestId for BrokerResponse in pinot-broker and java-client #10943
• Provide results in CompletableFuture for java clients and expose metrics #10326
• Add minion observability for segment upload/download failures (#10978)
• Enhance early terminate for combine operator (#10988)
• Add fromController method that accepts a PinotClientTransport (#11013)
• Ensure min/max value generation in the segment metadata. (#10891)
• Apply some allocation optimizations on GrpcSendingMailbox (#11015)
• When enable case-insensitive, don't allow to add newly column name which have the same lowercase name with existed columns. (#10991)
• Replace Long attributes with primitive values to reduce boxing (#11059)
• retry KafkaConsumer creation in KafkaPartitionLevelConnectionHandler.java (#253) (#11040)
• Support for new dataTime format in DateTimeGranularitySpec without explicitly setting size (#11057)
• Returning 403 status code in case of authorization failures (#11136)
• Simplify compatible test to avoid test against itself (#11163)
• Updated code for setting value of segment min/max property. (#10990)
• Add stat to track number of segments that have valid doc id snapshots (#11110)
• Add brokerId and brokerReduceTimeMs to the broker response stats (#11142)
• safely multiply integers to prevent overflow (#11186)
• Move largest comparison value update logic out of map access (#11157)
• Optimize DimensionTableDataManager to abort unnecesarry loading (#11192)
• Refine isNullsLast and isAsc functions. (#11199)
• Update the pinot tenants tables api to support returning broker tagged tables (#11184)
• add multi-value support for native text index (#11204)
• Add percentiles report in QuerySummary (#11299)
• Add meter for broker responses with unavailable segments (#11301)
• Enhance Minion task management (#11315)
• add additional lucene index configs (#11354)
• Add DECIMAL data type to orc record reader (#11377)
• add configuration to fail server startup on non-good status checker (#11347)
• allow passing freshness checker after an idle threshold (#11345)
• Add broker validation for hybrid tableConfig creation (#7908)
• Support partition parallelism for partitioned table scan (#11266)

Vulnerability fixes, bugfixes, cleanups and deprecations

• Remove support for High level consumers in Apache Pinot (#11017)
• Fix JDBC driver check for username (#10416)
• [Clean up] Remove getColumnName() from AggregationFunction interface (#10431)
• fix jersey TerminalWriterInterceptor MessageBodyWriter not found issue (#10462)
• Bug fix: Start counting operator execution time from first NoOp block (#10450)
• Fix unavailable instances issues for StrictReplicaGroup (#10466)
• Change shell to bash (#10469)
• Fix the double destroy of segment data manager during server shutdown (#10475)
• Remove "isSorted()" precondition check in the ForwardIndexHandler (#10476)
• Fix null handling in streaming selection operator (#10453)
• Fix jackson dependencies (#10477)
• Startree index build enhancement (#10905)
• optimize queries where lhs and rhs of predicate are equal (#10444)
• Trivial fix on a warning detected by static checker (#10492)
• wait for full segment commit protocol on force commit (#10479)
• Fix bug and add test for noDict -> Dict conversion for sorted column (#10497)
• Make column order deterministic in segment (#10468)
• Type match between resultType and function's dataType (#10472)
• Allow empty segmentsTo for segment replacement protocol (#10511)
• Use string as default compatible type for coalesce (#10516)
• Use threadlocal variable for genericRow to make the MemoryOptimizedTable threadsafe (#10502)
• Fix shading in spark2 connector pom file (#10490)
• Fix ramping delay caused by long lasting sequence of unfiltered messa… (#10418)
• Do not serialize metrics in each Operator (#10473)
• Make pinot-controller apply webpack production mode when bin-dist profile is used. (#10525)
• Fix FS props handling when using /ingestFromUri (#10480)
• Clean up v0_deprecated batch ingestion jobs (#10532)
• Deprecate kafka 0.9 support (#10522)
• safely multiply integers to prevent overflow (#11186)
• Reduce timeout for codecov and not fail the job in any case (#10547)
• Fix DataTableV3 serde bug for empty array (#10583)
• Do not record operator stats when tracing is enabled (#10447)
• Forward auth token for logger APIs from controller to other controllers and brokers (#10590)
• Bug fix: Partial upsert default strategy is null (#10610)
• Fix flaky test caused by EV check during table creation (#10616)
• Fix withDissabledTrue typo (#10624)
• Cleanup unnecessary mailbox id ser/de (#10629)
• no error metric for queries where all segments are pruned (#10589)
• bug fix: to keep QueryParser thread safe when handling many read requests on class RealtimeLuceneTextIndex (#10620)
• Fix static DictionaryIndexConfig.DEFAULT_OFFHEAP being actually onheap (#10632)
• 10567: [cleanup pinot-integration-test-base], clean query generations and some other refactoring. (#10648)
• Fixes backward incompatability with SegmentGenerationJobSpec for segment push job runners (#10645)
• Bug fix to get the toSegments list correctly (#10659)
• 10661: Fix for failing numeric comparison in where clause for IllegalStateException. (#10662)
• Fixes partial upsert not reflecting multiple comparison column values (#10693)
• Fix Bug in Reporting Timer Value for Min Consuming Freshness (#10690)
• Fix typo of rowSize -> columnSize (#10699)
• update segment target tier before table rebalance (#10695)
• Fix a bug in star-tree filter operator which can incorrecly filter documents (#10707)
• Enhance the instrumentation for a corner case where the query doesn't go through DocIdSetOp (#10729)
• bug fix: add missing properties when edit instance config (#10741)
• Making segmentMapper do the init and cleanup of RecordReader (#10874)
• Fix githubEvents table for quickstart recipes (#10716)
• Minor Realtime Segment Commit Upload Improvements (#10725)
• Return 503 for all interrupted queries. Refactor the query killing code. (#10683)
• Add decoder initialization error to the server's error cache (#10773)
• bug fix: add @JsonProperty to SegmentAssignmentConfig (#10759)
• ensure we wait the full no query timeout before shutting down (#10784)
• Clean up KLL functions with deprecated convention (#10795)
• Redefine the semantics of SEGMENT_STREAMED_DOWNLOAD_UNTAR_FAILURES metric to count individual segment fetch failures. (#10777)
• fix excpetion during exchange routing causes stucked pipeline (#10802)
• [bugfix] fix floating point and integral type backward incompatible issue (#10650)
• [pinot-core] Start consumption after creating segment data manager (#11227)
• Fix IndexOutOfBoundException in filtered aggregation group-by (#11231)
• Fix null pointer exception in segment debug endpoint #11228
• Clean up RangeIndexBasedFilterOperator. (#11219)
• Fix the escape/unescape issue for property value in metadata (#11223)
• Fix a bug in the order by comparator (#10818)
• Keeps nullness attributes of merged in comparison column values (#10704)
• Add required JSON annotation in H3IndexResolution (#10792)
• Fix a bug in SELECT DISTINCT ORDER BY. (#10827)
• jsonPathString should return null instead of string literal "null" (#10855)
• Bug Fix: Segment Purger cannot purge old segments after schema evolution (#10869)
• Fix #10713 by giving metainfo more priority than config (#10851)
• Close PinotFS after Data Manager Shutdowns (#10888)
• bump awssdk version for a bugfix on http conn leakage (#10898)
• Fix MultiNodesOfflineClusterIntegrationTest.testServerHardFailure() (#10909)
• Fix a bug in SELECT DISTINCT ORDER BY LIMIT. (#10887)
• Fix an integer overflow bug. (#10940)
• Return true when _resultSet is not null (#10899)
• Fixing table name extraction for lateral join queries (#10933)
• Fix casting when prefetching mmap'd segment larger than 2GB (#10936)
• Null check before closing reader (#10954)
• Fixes SQL wildcard escaping in LIKE queries (#10897)
• [Clean up] Do not count DISTINCT as aggregation (#10985)
• do not readd lucene readers to queue if segment is destroyed #10989
• Message batch ingestion lag fix (#10983)
• Fix a typo in snapshot lock (#11007)
• When extracting root-level field name for complex type handling, use the whole delimiter (#11005)
• update jersey to fix Denial of Service (DoS) (#11021)
• Update getTenantInstances call for controller and separate POST operations on it (#10993)
• update freemaker to fix Server-side Template Injection (#11019)
• format double 0 properly to compare with h2 results (#11049)
• Fix double-checked locking in ConnectionFactory (#11014)
• Remove presto-pinot-driver and pinot-java-client-jdk8 module (#11051)
• Make RequestUtils always return a string array when getTableNames (#11069)
• Fix BOOL_AND and BOOL_OR result type (#11033)
• [cleanup] Consolidate some query and controller/broker methods in integration tests (#11064)
• Fix grpc regression on multi-stage engine (#11086)
• Delete an obsolete TODO. (#11080)
• Minor fix on AddTableCommand.toString() (#11082)
• Allow using Lucene text indexes on mutable MV columns. (#11093)
• Allow offloading multiple segments from same table in parallel (#11107)
• Added serviceAccount to minion-stateless (#11095)
• Bug fix: TableUpsertMetadataManager is null (#11129)
• Fix reload bug (#11131)
• Allow extra aggregation types in RealtimeToOfflineSegmentsTask (#10982)
• Fix a bug when use range index to solve EQ predicate (#11146)
• Sanitise API inputs used as file path variables (#11132)
• Fix NPE when nested query doesn't have gapfill (#11155)
• Fix the NPE when query response error stream is null (#11154)
• Make interface methods non private, for java 8 compatibility (#11164)
• Increment nextDocId even if geo indexing fails (#11158)
• Fix the issue of consuming segment entering ERROR state due to stream connection errors (#11166)
• In TableRebalancer, remove instance partitions only when reassigning instances (#11169)
• Remove JDK 8 unsupported code (#11176)
• Fix compat test by adding -am flag to build pinot-integration-tests (#11181)
• dont duplicate register scalar function in CalciteSchema (#11190)
• Fix the storage quota check for metadata push (#11193)
• Delete filtering NULL support dead code paths. (#11198)
• [bugfix] Do not move real-time segments to working dir on restart (#11226)
• Fix a bug in ExpressionScanDocIdIterator for multi-value. (#11253)
• Exclude NULLs when PredicateEvaluator::isAlwaysTrue is true. (#11261)
• UI: fix sql query options seperator (#10770)
• Fix a NullPointerException bug in ScalarTransformFunctionWrapper. (#11309)
• [refactor] improve disk read for partial upsert handler (#10927)
• Fix the wrong query time when the response is empty (#11349)
• getMessageAtIndex should actually return the value in the streamMessage for compatibility (#11355)
• Remove presto jdk8 related dependencies (#11285)
• Remove special routing handling for multiple consuming segments (#11371)
• Properly handle shutdown of TableDataManager (#11380)
• Fixing the stale pinot ServerInstance in _tableTenantServersMap (#11386)
• Fix the thread safety issue for mutable forward index (#11392)
• Fix RawStringDistinctExecutor integer overflow (#11403)
• [logging] fix consume rate logging bug to respect 1 minute threshold (#11421)

What's Changed

Major updates

• Fixed backward compatibility issues related to CASE-WHEN statement. https://github.com/apache/pinot/pull/10291

What's Changed

Major updates

• Force commit consuming segments by @sajjad-moradi in https://github.com/apache/pinot/pull/9197
• add a freshness based consumption status checker by @jadami10 in https://github.com/apache/pinot/pull/9244
• Add metrics to track controller segment download and upload requests in progress by @gviedma in https://github.com/apache/pinot/pull/9258
• Adding endpoint to download local log files for each component by @xiangfu0 in https://github.com/apache/pinot/pull/9259
• [Feature] Add an option to search input files recursively in ingestion job. The default is set to true to be backward compatible. by @61yao in https://github.com/apache/pinot/pull/9265
• add query cancel APIs on controller backed by those on brokers by @klsince in https://github.com/apache/pinot/pull/9276
• Add Spark Job Launcher tool by @KKcorps in https://github.com/apache/pinot/pull/9288
• Enable Consistent Data Push for Standalone Segment Push Job Runners by @yuanbenson in https://github.com/apache/pinot/pull/9295
• Allow server to directly return the final aggregation result by @Jackie-Jiang in https://github.com/apache/pinot/pull/9304
• TierBasedSegmentDirectoryLoader to keep segments in multi-datadir by @klsince in https://github.com/apache/pinot/pull/9306
• Adaptive Server Selection by @vvivekiyer in https://github.com/apache/pinot/pull/9311
• [Feature] Support IsDistinctFrom and IsNotDistinctFrom by @61yao in https://github.com/apache/pinot/pull/9312
• Allow ingestion of errored records with incorrect datatype by @KKcorps in https://github.com/apache/pinot/pull/9320
• Allow setting custom time boundary for hybrid table queries by @saurabhd336 in https://github.com/apache/pinot/pull/9356
• skip late cron job with max allowed delay by @klsince in https://github.com/apache/pinot/pull/9372
• Do not allow implicit cast for BOOLEAN and TIMESTAMP by @Jackie-Jiang in https://github.com/apache/pinot/pull/9385
• Add missing properties in CSV plugin by @KKcorps in https://github.com/apache/pinot/pull/9399
• set MDC so that one can route minion task logs to separate files cleanly by @klsince in https://github.com/apache/pinot/pull/9400
• Add a new API to fix segment date time in metadata by @KKcorps in https://github.com/apache/pinot/pull/9413
• Update get bytes to return raw bytes of string and support getBytesMV by @61yao in https://github.com/apache/pinot/pull/9441
• Exposing consumer's record lag in /consumingSegmentsInfo by @navina in https://github.com/apache/pinot/pull/9515
• Do not create dictionary for high-cardinality columns by @KKcorps in https://github.com/apache/pinot/pull/9527
• get task runtime configs tracked in Helix by @klsince in https://github.com/apache/pinot/pull/9540
• Add more options to json index by @Jackie-Jiang in https://github.com/apache/pinot/pull/9543
• add SegmentTierAssigner and refine restful APIs to get segment tier info by @klsince in https://github.com/apache/pinot/pull/9598
• Add segment level debug API by @saurabhd336 in https://github.com/apache/pinot/pull/9609
• Add record availability lag for Kafka connector by @navina in https://github.com/apache/pinot/pull/9621
• notify servers that need to move segments to new tiers via SegmentReloadMessage by @klsince in https://github.com/apache/pinot/pull/9624
• Allow to configure multi-datadirs as instance configs and a Quickstart example about them by @klsince in https://github.com/apache/pinot/pull/9705
• Customize stopword for Lucene Index by @jasperjiaguo in https://github.com/apache/pinot/pull/9708
• Add memory optimized dimension table by @KKcorps in https://github.com/apache/pinot/pull/9802
• ADLS file system upgrade by @xiangfu0 in https://github.com/apache/pinot/pull/9855
• Added Delete Schema/Table pinot admin commands by @bagipriyank in https://github.com/apache/pinot/pull/9857
• Adding new ADLSPinotFS auth type: DEFAULT by @xiangfu0 in https://github.com/apache/pinot/pull/9860
• Add rate limit to Kinesis requests by @KKcorps in https://github.com/apache/pinot/pull/9863
• Adding configs for zk client timeout by @xiangfu0 in https://github.com/apache/pinot/pull/9975

Other features/changes

• Show most recent scheduling errors by @satishwaghela in https://github.com/apache/pinot/pull/9161
• Do not use aggregation result for distinct query in IntermediateResultsBlock by @Jackie-Jiang in https://github.com/apache/pinot/pull/9262
• Emit metrics for ratio of actual consumption rate to rate limit in realtime tables by @sajjad-moradi in https://github.com/apache/pinot/pull/9201
• add metrics entry offlineTableCount by @walterddr in https://github.com/apache/pinot/pull/9270
• refine query cancel resp msg by @klsince in https://github.com/apache/pinot/pull/9242
• add @ManualAuthorization annotation for non-standard endpoints by @apucher in https://github.com/apache/pinot/pull/9252
• Optimize ser/de to avoid using output stream by @Jackie-Jiang in https://github.com/apache/pinot/pull/9278
• Add Support for Covariance Function by @SabrinaZhaozyf in https://github.com/apache/pinot/pull/9236
• Throw an exception when MV columns are present in the order-by expression list in selection order-by only queries by @somandal in https://github.com/apache/pinot/pull/9078
• Improve server query cancellation and timeout checking during execution by @jasperjiaguo in https://github.com/apache/pinot/pull/9286
• Add capabilities to ingest from another stream without disabling the realtime table by @sajjad-moradi in https://github.com/apache/pinot/pull/9289
• Add minMaxInvalid flag to avoid unnecessary needPreprocess by @npawar in https://github.com/apache/pinot/pull/9238
• Add array cardinality function by @walterddr in https://github.com/apache/pinot/pull/9300
• TierBasedSegmentDirectoryLoader to keep segments in multi-datadir by @klsince in https://github.com/apache/pinot/pull/9306
• Add support for custom null values in CSV record reader by @KKcorps in https://github.com/apache/pinot/pull/9318
• Infer parquet reader type based on file metadata by @saurabhd336 in https://github.com/apache/pinot/pull/9294
• Include fmpp plugin module inside the src assembly file by @xiangfu0 in https://github.com/apache/pinot/pull/9321
• Add Support for Cast Function on MV Columns by @SabrinaZhaozyf in https://github.com/apache/pinot/pull/9296
• Allow ingestion of errored records with incorrect datatype by @KKcorps in https://github.com/apache/pinot/pull/9320
• [Feature] Not Operator Transformation by @61yao in https://github.com/apache/pinot/pull/9330
• Handle null string in CSV decoder by @KKcorps in https://github.com/apache/pinot/pull/9340
• [Feature] Not scalar function by @61yao in https://github.com/apache/pinot/pull/9338
• Add support for EXTRACT syntax and converts it to appropriate Pinot expression by @tanmesh in https://github.com/apache/pinot/pull/9184
• Add support for Auth in controller requests in java query client by @KKcorps in https://github.com/apache/pinot/pull/9230
• delete all related minion task metadata when deleting a table by @zhtaoxiang in https://github.com/apache/pinot/pull/9339
• BloomFilterRule should only recommend for supported column type by @yuanbenson in https://github.com/apache/pinot/pull/9364
• Support all the types in ParquetNativeRecordReader by @xiangfu0 in https://github.com/apache/pinot/pull/9352
• Improve segment name check in metadata push by @zhtaoxiang in https://github.com/apache/pinot/pull/9359
• Allow expression transformer cotinue on error by @xiangfu0 in https://github.com/apache/pinot/pull/9376
• skip late cron job with max allowed delay by @klsince in https://github.com/apache/pinot/pull/9372
• Enhance and filter predicate evaluation efficiency by @jasperjiaguo in https://github.com/apache/pinot/pull/9336
• Deprecate instanceId Config For Broker/Minion Specific Configs by @ankitsultana in https://github.com/apache/pinot/pull/9308
• Optimize combine operator to fully utilize threads by @Jackie-Jiang in https://github.com/apache/pinot/pull/9387
• Terminate the query after plan generation if timeout by @jasperjiaguo in https://github.com/apache/pinot/pull/9386
• [Feature] Support IsDistinctFrom and IsNotDistinctFrom by @61yao in https://github.com/apache/pinot/pull/9312
• [Feature] Support Coalesce for Column Names by @61yao in https://github.com/apache/pinot/pull/9327
• Disable logging for interrupted exceptions in kinesis by @KKcorps in https://github.com/apache/pinot/pull/9405
• Benchmark thread cpu time by @jasperjiaguo in https://github.com/apache/pinot/pull/9408
• Use ISODateTimeFormat as default for SIMPLE_DATE_FORMAT by @KKcorps in https://github.com/apache/pinot/pull/9378
• Extract the common logic for upsert metadata manager by @Jackie-Jiang in https://github.com/apache/pinot/pull/9435
• Make minion task metadata manager methods more generic by @saurabhd336 in https://github.com/apache/pinot/pull/9436
• Always pass clientId to kafka's consumer properties by @navina in https://github.com/apache/pinot/pull/9444
• Adaptive Server Selection by @vvivekiyer in https://github.com/apache/pinot/pull/9311
• Refine IndexHandler methods a bit to make them reentrant by @klsince in https://github.com/apache/pinot/pull/9440
• use MinionEventObserver to track finer grained task progress status on worker by @klsince in https://github.com/apache/pinot/pull/9432
• Allow spaces in input file paths by @KKcorps in https://github.com/apache/pinot/pull/9426
• Add support for gracefully handling the errors while transformations by @KKcorps in https://github.com/apache/pinot/pull/9377
• Cache Deleted Segment Names in Server to Avoid SegmentMissingError by @ankitsultana in https://github.com/apache/pinot/pull/9423
• Handle Invalid timestamps by @KKcorps in https://github.com/apache/pinot/pull/9355
• refine minion worker event observer to track finer grained progress for tasks by @klsince in https://github.com/apache/pinot/pull/9449
• spark-connector should use v2/brokers endpoint by @itschrispeck in https://github.com/apache/pinot/pull/9451
• Remove netty server query support from presto-pinot-driver to remove pinot-core and pinot-segment-local dependencies by @xiangfu0 in https://github.com/apache/pinot/pull/9455
• Adaptive Server Selection: Address pending review comments by @vvivekiyer in https://github.com/apache/pinot/pull/9462
• track progress from within segment processor framework by @klsince in https://github.com/apache/pinot/pull/9457
• Decouple ser/de from DataTable by @Jackie-Jiang in https://github.com/apache/pinot/pull/9468
• collect file info like mtime, length while listing files for free by @klsince in https://github.com/apache/pinot/pull/9466
• Extract record keys, headers and metadata from Stream sources by @navina in https://github.com/apache/pinot/pull/9224
• [pinot-spark-connector] Bump spark connector max inbound message size by @cbalci in https://github.com/apache/pinot/pull/9475
• refine the minion task progress api a bit by @klsince in https://github.com/apache/pinot/pull/9482
• add parsing for AT TIME ZONE by @agavra in https://github.com/apache/pinot/pull/9477
• Eliminate explosion of metrics due to gapfill queries by @elonazoulay in https://github.com/apache/pinot/pull/9490
• ForwardIndexHandler: Change compressionType during segmentReload by @vvivekiyer in https://github.com/apache/pinot/pull/9454
• Introduce Segment AssignmentStrategy Interface by @GSharayu in https://github.com/apache/pinot/pull/9309
• Add query interruption flag check to broker groupby reduction by @jasperjiaguo in https://github.com/apache/pinot/pull/9499
• adding optional client payload by @walterddr in https://github.com/apache/pinot/pull/9465
• [feature] distinct from scalar functions by @61yao in https://github.com/apache/pinot/pull/9486
• Check data table version on server only for null handling by @Jackie-Jiang in https://github.com/apache/pinot/pull/9508
• Add docId and column name to segment read exception by @KKcorps in https://github.com/apache/pinot/pull/9512
• Sort scanning based operators by cardinality in AndDocIdSet evaluation by @jasperjiaguo in https://github.com/apache/pinot/pull/9420
• Do not fail CI when codecov upload fails by @Jackie-Jiang in https://github.com/apache/pinot/pull/9522
• [Upsert] persist validDocsIndex snapshot for Pinot upsert optimization by @deemoliu in https://github.com/apache/pinot/pull/9062
• broker filter by @dongxiaoman in https://github.com/apache/pinot/pull/9391
• [feature] coalesce scalar by @61yao in https://github.com/apache/pinot/pull/9487
• Allow setting custom time boundary for hybrid table queries by @saurabhd336 in https://github.com/apache/pinot/pull/9356
• [GHA] add cache timeout by @walterddr in https://github.com/apache/pinot/pull/9524
• Optimize PinotHelixResourceManager.hasTable() by @Jackie-Jiang in https://github.com/apache/pinot/pull/9526
• Include exception when upsert metadata manager cannot be created by @Jackie-Jiang in https://github.com/apache/pinot/pull/9532
• allow to config task expire time by @klsince in https://github.com/apache/pinot/pull/9530
• expose task finish time via debug API by @klsince in https://github.com/apache/pinot/pull/9534
• Remove the wrong warning log in KafkaPartitionLevelConsumer by @Jackie-Jiang in https://github.com/apache/pinot/pull/9536
• starting http server for minion worker conditionally by @klsince in https://github.com/apache/pinot/pull/9542
• Make StreamMessage generic and a bug fix by @vvivekiyer in https://github.com/apache/pinot/pull/9544
• Improve primary key serialization performance by @KKcorps in https://github.com/apache/pinot/pull/9538
• [Upsert] Skip removing upsert metadata when shutting down the server by @Jackie-Jiang in https://github.com/apache/pinot/pull/9551
• add array element at function by @walterddr in https://github.com/apache/pinot/pull/9554
• Handle the case when enableNullHandling is true and an aggregation function is used w/ a column that has an empty null bitmap by @nizarhejazi in https://github.com/apache/pinot/pull/9566
• Support segment storage format without forward index by @somandal in https://github.com/apache/pinot/pull/9333
• Adding SegmentNameGenerator type inference if not explicitly set in config by @timsants in https://github.com/apache/pinot/pull/9550
• add version information to JMX metrics & component logs by @agavra in https://github.com/apache/pinot/pull/9578
• remove unused RecordTransform/RecordFilter classes by @agavra in https://github.com/apache/pinot/pull/9607
• Support rewriting forward index upon changing compression type for existing raw MV column by @vvivekiyer in https://github.com/apache/pinot/pull/9510
• Support Avro's Fixed data type by @sajjad-moradi in https://github.com/apache/pinot/pull/9642
• [feature] [kubernetes] add loadBalancerSourceRanges to service-external.yaml for controller and broker by @jameskelleher in https://github.com/apache/pinot/pull/9494
• Limit up to 10 unavailable segments to be printed in the query exception by @Jackie-Jiang in https://github.com/apache/pinot/pull/9617
• remove more unused filter code by @agavra in https://github.com/apache/pinot/pull/9620
• Do not cache record reader in segment by @Jackie-Jiang in https://github.com/apache/pinot/pull/9604
• make first part of user agent header configurable by @rino-kadijk in https://github.com/apache/pinot/pull/9471
• optimize order by sorted ASC, unsorted and order by DESC cases by @gortiz in https://github.com/apache/pinot/pull/8979
• Enhance cluster config update API to handle non-string values properly by @Jackie-Jiang in https://github.com/apache/pinot/pull/9635
• Reverts recommender REST API back to PUT (reverts PR #9326) by @yuanbenson in https://github.com/apache/pinot/pull/9638
• Remove invalid pruner names from server config by @Jackie-Jiang in https://github.com/apache/pinot/pull/9646
• Using usageHelp instead of deprecated help in picocli commands by @navina in https://github.com/apache/pinot/pull/9608
• Handle unique query id on server by @Jackie-Jiang in https://github.com/apache/pinot/pull/9648
• stateless group marker missing several by @walterddr in https://github.com/apache/pinot/pull/9673
• Support reloading consuming segment using force commit by @Jackie-Jiang in https://github.com/apache/pinot/pull/9640
• Improve star-tree to use star-node when the predicate matches all the non-star nodes by @Jackie-Jiang in https://github.com/apache/pinot/pull/9667
• add FetchPlanner interface to decide what column index to prefetch by @klsince in https://github.com/apache/pinot/pull/9668
• Improve star-tree traversal using ArrayDeque by @Jackie-Jiang in https://github.com/apache/pinot/pull/9688
• Handle errors in combine operator by @Jackie-Jiang in https://github.com/apache/pinot/pull/9689
• return different error code if old version is not on master by @SabrinaZhaozyf in https://github.com/apache/pinot/pull/9686
• Support creating dictionary at runtime for an existing column by @vvivekiyer in https://github.com/apache/pinot/pull/9678
• check mutable segment explicitly instead of checking existence of indexDir by @klsince in https://github.com/apache/pinot/pull/9718
• Remove leftover file before downloading segmentTar by @npawar in https://github.com/apache/pinot/pull/9719
• add index key and size map to segment metadata by @walterddr in https://github.com/apache/pinot/pull/9712
• Use ideal state as source of truth for segment existence by @Jackie-Jiang in https://github.com/apache/pinot/pull/9735
• Close Filesystem on exit with Minion Tasks by @KKcorps in https://github.com/apache/pinot/pull/9681
• render the tables list even as the table sizes are loading by @jadami10 in https://github.com/apache/pinot/pull/9741
• Add Support for IP Address Function by @SabrinaZhaozyf in https://github.com/apache/pinot/pull/9501
• bubble up error messages from broker by @agavra in https://github.com/apache/pinot/pull/9754
• Add support to disable the forward index for existing columns by @somandal in https://github.com/apache/pinot/pull/9740
• show table metadata info in aggregate index size form by @walterddr in https://github.com/apache/pinot/pull/9733
• Preprocess immutable segments from REALTIME table conditionally when loading them by @klsince in https://github.com/apache/pinot/pull/9772
• revert default timeout nano change in QueryConfig by @agavra in https://github.com/apache/pinot/pull/9790
• AdaptiveServerSelection: Update stats for servers that have not responded by @vvivekiyer in https://github.com/apache/pinot/pull/9801
• Add null value index for default column by @KKcorps in https://github.com/apache/pinot/pull/9777
• [MergeRollupTask] include partition info into segment name by @zhtaoxiang in https://github.com/apache/pinot/pull/9815
• Adding a consumer lag as metric via a periodic task in controller by @navina in https://github.com/apache/pinot/pull/9800
• Deserialize Hyperloglog objects more optimally by @priyen in https://github.com/apache/pinot/pull/9749
• Download offline segments from peers by @wirybeaver in https://github.com/apache/pinot/pull/9710
• Thread Level Usage Accounting and Query Killing on Server by @jasperjiaguo in https://github.com/apache/pinot/pull/9727
• Add max merger and min mergers for partial upsert by @deemoliu in https://github.com/apache/pinot/pull/9665
• #9518 added pinot helm 0.2.6 with secure version pinot 0.11.0 by @bagipriyank in https://github.com/apache/pinot/pull/9519
• Combine the read access for replication config by @snleee in https://github.com/apache/pinot/pull/9849
• add v1 ingress in helm chart by @jhisse in https://github.com/apache/pinot/pull/9862
• Optimize AdaptiveServerSelection for replicaGroup based routing by @vvivekiyer in https://github.com/apache/pinot/pull/9803
• Do not sort the instances in InstancePartitions by @Jackie-Jiang in https://github.com/apache/pinot/pull/9866
• Merge new columns in existing record with default merge strategy by @navina in https://github.com/apache/pinot/pull/9851
• Support disabling dictionary at runtime for an existing column by @vvivekiyer in https://github.com/apache/pinot/pull/9868
• support BOOL_AND and BOOL_OR aggregate functions by @agavra in https://github.com/apache/pinot/pull/9848
• Use Pulsar AdminClient to delete unused subscriptions by @navina in https://github.com/apache/pinot/pull/9859
• add table sort function for table size by @jadami10 in https://github.com/apache/pinot/pull/9844
• In Kafka consumer, seek offset only when needed by @Jackie-Jiang in https://github.com/apache/pinot/pull/9896
• fallback if no broker found for the specified table name by @klsince in https://github.com/apache/pinot/pull/9914
• Allow liveness check during server shutting down by @Jackie-Jiang in https://github.com/apache/pinot/pull/9915
• Allow segment upload via Metadata in MergeRollup Minion task by @KKcorps in https://github.com/apache/pinot/pull/9825
• Add back the Helix workaround for missing IS change by @Jackie-Jiang in https://github.com/apache/pinot/pull/9921
• Allow uploading realtime segments via CLI by @KKcorps in https://github.com/apache/pinot/pull/9861
• Add capability to update and delete table config via CLI by @KKcorps in https://github.com/apache/pinot/pull/9852
• default to TAR if push mode is not set by @klsince in https://github.com/apache/pinot/pull/9935
• load startree index via segment reader interface by @klsince in https://github.com/apache/pinot/pull/9828
• Allow collections for MV transform functions by @saurabhd336 in https://github.com/apache/pinot/pull/9908
• Construct new IndexLoadingConfig when loading completed realtime segments by @vvivekiyer in https://github.com/apache/pinot/pull/9938
• Make GET /tableConfigs backwards compatible in case schema does not match raw table name by @timsants in https://github.com/apache/pinot/pull/9922
• feat: add compressed file support for ORCRecordReader by @etolbakov in https://github.com/apache/pinot/pull/9884
• Add Variance and Standard Deviation Aggregation Functions by @snleee in https://github.com/apache/pinot/pull/9910
• enable MergeRollupTask on realtime tables by @zhtaoxiang in https://github.com/apache/pinot/pull/9890
• Update cardinality when converting raw column to dict based by @vvivekiyer in https://github.com/apache/pinot/pull/9875
• Add back auth token for UploadSegmentCommand by @timsants in https://github.com/apache/pinot/pull/9960
• Improving gz support for avro record readers by @snleee in https://github.com/apache/pinot/pull/9951
• Default column handling of noForwardIndex and regeneration of forward index on reload path by @somandal in https://github.com/apache/pinot/pull/9810
• [Feature] Support coalesce literal by @61yao in https://github.com/apache/pinot/pull/9958
• Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly by @npawar in https://github.com/apache/pinot/pull/9988
• handle pending minion tasks properly when getting the task progress status by @klsince in https://github.com/apache/pinot/pull/9911
• allow gauge stored in metric registry to be updated by @zhtaoxiang in https://github.com/apache/pinot/pull/9961
• support case-insensitive query options in SET syntax by @agavra in https://github.com/apache/pinot/pull/9912
• pin versions-maven-plugin to 2.13.0 by @jadami10 in https://github.com/apache/pinot/pull/9993
• Pulsar Connection handler should not spin up a consumer / reader by @navina in https://github.com/apache/pinot/pull/9893
• Handle in-memory segment metadata for index checking by @Jackie-Jiang in https://github.com/apache/pinot/pull/10017
• Support the cross-account access using IAM role for S3 PinotFS by @snleee in https://github.com/apache/pinot/pull/10009
• report minion task metadata last update time as metric by @zhtaoxiang in https://github.com/apache/pinot/pull/9954
• support SKEWNESS and KURTOSIS aggregates by @agavra in https://github.com/apache/pinot/pull/10021
• emit minion task generation time and error metrics by @zhtaoxiang in https://github.com/apache/pinot/pull/10026
• Use the same default time value for all replicas by @Jackie-Jiang in https://github.com/apache/pinot/pull/10029
• Reduce the number of segments to wait for convergence when rebalancing by @saurabhd336 in https://github.com/apache/pinot/pull/10028

UI Update & Improvement

• Allow hiding query console tab based on cluster config (#9261)
• Allow hiding pinot broker swagger UI by config (#9343)
• Add UI to show fine-grained minion task progress (#9488)
• Add UI to track segment reload progress (#9521)
• Show minion task runtime config details in UI (#9652)
• Redefine the segment status (#9699)
• Show an option to reload the segments during edit schema (#9762)
• Load schema UI async (#9781)
• Fix blank screen when redirect to unknown app route (#9888)

Multi-Stage Query Engine

New join semantics support

• Left join (#9466)
• In-equi join (#9448)
• Full join (#9907)
• Right join (#9907)
• Semi join (#9367)
• Using keyword (#9373)

New sql semantics support:

• Having (#9274)
• Order by (#9279)
• In/NotIn clause (#9374)
• Cast (#9384)
• LIke/Rexlike (#9654)
• Range predicate (#9445)

Performance enhancement

• Thread safe query planning (#9344)
• Partial query execution and round robin scheduling (#9753)
• Improve data table serde (#9731)

Library version upgrade

• Upgrade h3 lib from 3.7.2 to 4.0.0 to lower glibc requirement (#9335)
• Upgrade ZK version to 3.6.3 (#9612)
• Upgrade snakeyaml from 1.30 to 1.33 (#9464)
• Upgrade RoaringBitmap from 0.9.28 to 0.9.35 (#9730)
• Upgrade spotless-maven-plugin from 2.9.0 to 2.28.0 (#9877)
• Upgrade decode-uri-component from 0.2.0 to 0.2.2 (#9941)

BugFixes

• Fix bug with logging request headers by @abhs50 in https://github.com/apache/pinot/pull/9247
• Fix a UT that only shows up on host with more cores by @klsince in https://github.com/apache/pinot/pull/9257
• Fix message count by @Jackie-Jiang in https://github.com/apache/pinot/pull/9271
• Fix issue with auth AccessType in Schema REST endpoints by @sajjad-moradi in https://github.com/apache/pinot/pull/9293
• Fix PerfBenchmarkRunner to skip the tmp dir by @Jackie-Jiang in https://github.com/apache/pinot/pull/9298
• Fix thrift deserializer thread safety issue by @saurabhd336 in https://github.com/apache/pinot/pull/9299
• Fix transformation to string for BOOLEAN and TIMESTAMP by @Jackie-Jiang in https://github.com/apache/pinot/pull/9287
• [hotfix] Add VARBINARY column to switch case branch by @walterddr in https://github.com/apache/pinot/pull/9313
• Fix annotation for "/recommender" endpoint by @sajjad-moradi in https://github.com/apache/pinot/pull/9326
• Fix jdk8 build issue due to missing pom dependency by @somandal in https://github.com/apache/pinot/pull/9351
• Fix pom to use pinot-common-jdk8 for pinot-connector jkd8 java client by @somandal in https://github.com/apache/pinot/pull/9353
• Fix log to reflect job type by @KKcorps in https://github.com/apache/pinot/pull/9381
• [Bugfix] schema update bug fix by @MeihanLi in https://github.com/apache/pinot/pull/9382
• fix histogram null pointer exception by @jasperjiaguo in https://github.com/apache/pinot/pull/9428
• Fix thread safety issues with SDF (WIP) by @saurabhd336 in https://github.com/apache/pinot/pull/9425
• Bug fix: failure status in ingestion jobs doesn't reflect in exit code by @KKcorps in https://github.com/apache/pinot/pull/9410
• Fix skip segment logic in MinMaxValueBasedSelectionOrderByCombineOperator by @Jackie-Jiang in https://github.com/apache/pinot/pull/9434
• Fix the bug of hybrid table request using the same request id by @Jackie-Jiang in https://github.com/apache/pinot/pull/9443
• Fix the range check for range index on raw column by @Jackie-Jiang in https://github.com/apache/pinot/pull/9453
• Fix Data-Correctness Bug in GTE Comparison in BinaryOperatorTransformFunction by @ankitsultana in https://github.com/apache/pinot/pull/9461
• extend PinotFS impls with listFilesWithMetadata and some bugfix by @klsince in https://github.com/apache/pinot/pull/9478
• fix null transform bound check by @walterddr in https://github.com/apache/pinot/pull/9495
• Fix JsonExtractScalar when no value is extracted by @Jackie-Jiang in https://github.com/apache/pinot/pull/9500
• Fix AddTable for realtime tables by @npawar in https://github.com/apache/pinot/pull/9506
• Fix some type convert scalar functions by @Jackie-Jiang in https://github.com/apache/pinot/pull/9509
• fix spammy logs for ConfluentSchemaRegistryRealtimeClusterIntegrationTest [MINOR] by @agavra in https://github.com/apache/pinot/pull/9516
• Fix timestamp index on column of preserved key by @Jackie-Jiang in https://github.com/apache/pinot/pull/9533
• Fix record extractor when ByteBuffer can be reused by @Jackie-Jiang in https://github.com/apache/pinot/pull/9549
• Fix explain plan ALL_SEGMENTS_PRUNED_ON_SERVER node by @somandal in https://github.com/apache/pinot/pull/9572
• Fix time validation when data type needs to be converted by @Jackie-Jiang in https://github.com/apache/pinot/pull/9569
• UI: fix incorrect task finish time by @jayeshchoudhary in https://github.com/apache/pinot/pull/9557
• Fix the bug where uploaded segments cannot be deleted on real-time table by @Jackie-Jiang in https://github.com/apache/pinot/pull/9579
• [bugfix] correct the dir for building segments in FileIngestionHelper by @zhtaoxiang in https://github.com/apache/pinot/pull/9591
• Fix NonAggregationGroupByToDistinctQueryRewriter by @Jackie-Jiang in https://github.com/apache/pinot/pull/9605
• fix distinct result return by @walterddr in https://github.com/apache/pinot/pull/9582
• Fix GcsPinotFS by @lfernandez93 in https://github.com/apache/pinot/pull/9556
• fix DataSchema thread-safe issue by @walterddr in https://github.com/apache/pinot/pull/9619
• Bug fix: Add missing table config fetch for /tableConfigs list all by @timsants in https://github.com/apache/pinot/pull/9603
• Fix re-uploading segment when the previous upload failed by @Jackie-Jiang in https://github.com/apache/pinot/pull/9631
• Fix string split which should be on whole separator by @Jackie-Jiang in https://github.com/apache/pinot/pull/9650
• Fix server request sent delay to be non-negative by @Jackie-Jiang in https://github.com/apache/pinot/pull/9656
• bugfix: Add missing BIG_DECIMAL support for GenericRow serde by @timsants in https://github.com/apache/pinot/pull/9661
• Fix extra restlet resource test which should be stateless by @Jackie-Jiang in https://github.com/apache/pinot/pull/9674
• AdaptiveServerSelection: Fix timer by @vvivekiyer in https://github.com/apache/pinot/pull/9697
• fix PinotVersion to be compatible with prometheus by @agavra in https://github.com/apache/pinot/pull/9701
• Fix the setup for ControllerTest shared cluster by @Jackie-Jiang in https://github.com/apache/pinot/pull/9704
• [hotfix]groovy class cache leak by @walterddr in https://github.com/apache/pinot/pull/9716
• Fix TIMESTAMP index handling in SegmentMapper by @Jackie-Jiang in https://github.com/apache/pinot/pull/9722
• Fix the server admin endpoint cache to reflect the config changes by @Jackie-Jiang in https://github.com/apache/pinot/pull/9734
• [bugfix] fix case-when issue by @walterddr in https://github.com/apache/pinot/pull/9702
• [bugfix] Let StartControllerCommand also handle "pinot.zk.server", "pinot.cluster.name" in default conf/pinot-controller.conf by @thangnd197 in https://github.com/apache/pinot/pull/9739
• [hotfix] semi-join opt by @walterddr in https://github.com/apache/pinot/pull/9779
• Fixing the rebalance issue for real-time table with tier by @snleee in https://github.com/apache/pinot/pull/9780
• UI: show segment debug details when segment is in bad state by @jayeshchoudhary in https://github.com/apache/pinot/pull/9700
• Fix the replication in segment assignment strategy by @GSharayu in https://github.com/apache/pinot/pull/9816
• fix potential fd leakage for SegmentProcessorFramework by @klsince in https://github.com/apache/pinot/pull/9797
• Fix NPE when reading ZK address from controller config by @Jackie-Jiang in https://github.com/apache/pinot/pull/9751
• have query table list show search bar; fix InstancesTables filter by @jadami10 in https://github.com/apache/pinot/pull/9742
• [pinot-spark-connector] Fix empty data table handling in GRPC reader by @cbalci in https://github.com/apache/pinot/pull/9837
• [bugfix] fix mergeRollupTask metrics by @zhtaoxiang in https://github.com/apache/pinot/pull/9864
• Bug fix: Get correct primary key count by @KKcorps in https://github.com/apache/pinot/pull/9876
• Fix issues for realtime table reload by @Jackie-Jiang in https://github.com/apache/pinot/pull/9885
• UI: fix segment status color remains same in different table page by @jayeshchoudhary in https://github.com/apache/pinot/pull/9891
• Fix bloom filter creation on BYTES by @Jackie-Jiang in https://github.com/apache/pinot/pull/9898
• [hotfix] broker selection not using table name by @walterddr in https://github.com/apache/pinot/pull/9902
• Fix race condition when 2 segment upload occurred for the same segment by @jackjlli in https://github.com/apache/pinot/pull/9905
• fix timezone_hour/timezone_minute functions by @agavra in https://github.com/apache/pinot/pull/9949
• [Bugfix] Move brokerId extraction to BaseBrokerStarter by @jackjlli in https://github.com/apache/pinot/pull/9965
• Fix ser/de for StringLongPair by @Jackie-Jiang in https://github.com/apache/pinot/pull/9985
• bugfix dir check for HadoopPinotFS.copyFromLocalDir by @klsince in https://github.com/apache/pinot/pull/9979
• Bugfix: Use correct exception import in TableRebalancer. by @mayankshriv in https://github.com/apache/pinot/pull/10025
• Fix NPE in AbstractMetrics From Race Condition by @ankitsultana in https://github.com/apache/pinot/pull/10022

Summary

Apache Pinot 0.11.0 has introduced many new features to extend the query abilities, e.g. the Multi-Stage query engine enables Pinot to do distributed joins, more sql syntax(DML support), query functions and indexes(Text index, Timestamp index) supported for new use cases. And as always, more integrations with other systems(E.g. Spark3, Flink).

Note: there is a major upgrade for Apache Helix to 1.0.4, so please make sure you upgrade the system in the order of:
Helix Controller -> Pinot Controller -> Pinot Broker -> Pinot server

Multi-Stage Query Engine

The new multi-stage query engine (a.k.a V2 query engine) is designed to support more complex SQL semantics such as JOIN, OVER window, MATCH_RECOGNIZE and eventually, make Pinot support closer to full ANSI SQL semantics.

More to read: https://docs.pinot.apache.org/developers/advanced/v2-multi-stage-query-engine

Pause Stream Consumption on Apache Pinot

Pinot operators can pause realtime consumption of events while queries are being executed, and then resume consumption when ready to do so again.

More to read: https://medium.com/apache-pinot-developer-blog/pause-stream-consumption-on-apache-pinot-772a971ef403

Gap-filling function

The gapfilling functions allow users to interpolate data and perform powerful aggregations and data processing over time series data.
More to read: https://www.startree.ai/blog/gapfill-function-for-time-series-datasets-in-pinot

Add support for Spark 3.x (#8560)

Long waiting feature for segment generation on Spark 3.x.

Add Flink Pinot connector (#8233)

Similar to the Spark Pinot connector, this allows Flink users to dump data from the Flink application to Pinot.

Show running queries and cancel query by id (#9175)

This feature allows better fine-grained control on pinot queries.

Timestamp Index (#8343)

This allows users to have better query performance on the timestamp column for lower granularity. See: https://docs.pinot.apache.org/basics/indexing/timestamp-index

Native Text Indices (#8384)

Wanna search text in realtime? The new text indexing engine in Pinot supports the following capabilities:

1. New operator: LIKE

select * FROM foo where text_col LIKE 'a%'

2. New operator: CONTAINS

select * from foo where text_col CONTAINS 'bar'

3. Native text index, built from the ground up, focusing on Pinot’s time series use cases and utilizing existing Pinot indices and structures(inverted index, bitmap storage).
4. Real Time Text Index

Read more: https://medium.com/@atri.jiit/text-search-time-series-style-681af37ba42e

Adding DML definition and parse SQL InsertFile (#8557)

Now you can use INSERT INTO [database.]table FROM FILE dataDirURI OPTION ( k=v ) [, OPTION (k=v)]* to load data into Pinot from a file using Minion. See: https://docs.pinot.apache.org/basics/data-import/from-query-console

Deduplication (#8708)

This feature supports enabling deduplication for realtime tables, via a top-level table config. At a high level, primaryKey (as defined in the table schema) hashes are stored into in-memory data structures, and each incoming row is validated against it. Duplicate rows are dropped.

The expectation while using this feature is for the stream to be partitioned by the primary key, strictReplicaGroup routing to be enabled, and the configured stream consumer type to be low level. These requirements are therefore mandated via table config API's input validations.

Functions support and changes:

• Add support for functions arrayConcatLong, arrayConcatFloat, arrayConcatDouble (#9131)
• Add support for regexpReplace scalar function (#9123)
• Add support for Base64 Encode/Decode Scalar Functions (#9114)
• Optimize like to regexp conversion to do not include unnecessary ^._ and ._$ (#8893)
• Support DISTINCT on multiple MV columns (#8873)
• Support DISTINCT on single MV column (#8857)
• Add histogram aggregation function (#8724)
• Optimize dateTimeConvert scalar function to only parse the format once (#8939)
• Support conjugates for scalar functions, add more scalar functions (#8582)
• add FIRSTWITHTIME aggregate function support #7647 (#8181)
• Add PercentileSmartTDigestAggregationFunction (#8565)
• Simplify the parameters for DistinctCountSmartHLLAggregationFunction (#8566)
• add scalar function for cast so it can be calculated at compile time (#8535)
• Scalable Gapfill Implementation for Avg/Count/Sum (#8647)
• Add commonly used math, string and date scalar functions in Pinot (#8304)
• Datetime transform functions (#8397)
• Scalar function for url encoding and decoding (#8378)
• Add support for IS NULL and NOT IS NULL in transform functions (#8264)
• Support st_contains using H3 index (#8498)

The full list of features introduced in this release

• add query cancel APIs on controller backed by those on brokers (#9276)
• Add an option to search input files recursively in ingestion job. The default is set to true to be backward compatible. (#9265)
• Adding endpoint to download local log files for each component (#9259)
• Add metrics to track controller segment download and upload requests in progress (#9258)
• add a freshness based consumption status checker (#9244)
• Force commit consuming segments (#9197)
• Adding kafka offset support for period and timestamp (#9193)
• Make upsert metadata manager pluggable (#9186)
• Adding logger utils and allow change logger level at runtime (#9180)
• Proper null handling in equality, inequality and membership operators for all SV column data types (#9173)
• support to show running queries and cancel query by id (#9171)
• Enhance upsert metadata handling (#9095)
• Proper null handling in Aggregation functions for SV data types (#9086)
• Add support for IAM role based credentials in Kinesis Plugin (#9071)
• Task genrator debug api (#9058)
• Add Segment Lineage List API #9005 (#9006)
• [colocated-join] Adds Support for instancePartitionsMap in Table Config (#8989)
• Support pause/resume consumption of realtime tables (#8986)
• #8970 Minion tab in Pinot UI (#8978)
• Add Protocol Buffer Stream Decoder (#8972)
• Update minion task metadata ZNode path (#8959)
• add /tasks/{taskType}/{tableNameWithType}/debug API (#8949)
• Defined a new broker metric for total query processing time (#8941)
• Proper null handling in SELECT, ORDER BY, DISTINCT, and GROUP BY (#8927)
• fixing REGEX OPTION parser (#8905)
• Enable key value byte stitching in PulsarMessageBatch (#8897)
• Add property to skip adding hadoop jars to package (#8888)
• Support DISTINCT on multiple MV columns (#8873)
• Implement Mutable FST Index (#8861)
• Support DISTINCT on single MV column (#8857)
• Add controller API for reload segment task status (#8828)
• Spark Connector, support for TIMESTAMP and BOOLEAN fields (#8825)
• Allow moveToFinalLocation in METADATA push based on config (#8823) (#8815)
• allow up to 4GB per bitmap index (#8796)
• Deprecate debug options and always use query options (#8768)
• Streamed segment download & untar with rate limiter to control disk usage (#8753)
• Improve the Explain Plan accuracy (#8738)
• allow to set https as the default scheme (#8729)
• Add histogram aggregation function (#8724)
• Allow table name with dots by a PinotConfiguration switch (#8713)
• Disable Groovy function by default (#8711)
• Deduplication (#8708)
• Add pluggable client auth provider (#8670)
• Adding pinot file system command (#8659)
• Allow broker to automatically rewrite expensive function to its approximate counterpart (#8655)
• allow to take data outside the time window by negating the window filter (#8640)
• Support BigDecimal raw value forward index; Support BigDecimal in many transforms and operators (#8622)
• Ingestion Aggregation Feature (#8611)
• Enable uploading segments to realtime tables (#8584)
• Package kafka 0.9 shaded jar to pinot-distribution (#8569)
• Simplify the parameters for DistinctCountSmartHLLAggregationFunction (#8566)
• Add PercentileSmartTDigestAggregationFunction (#8565)
• Add support for Spark 3.x (#8560)
• Adding DML definition and parse SQL InsertFile (#8557)
• endpoints to get and delete minion task metadata (#8551)
• Add query option to use more replica groups (#8550)
• Only discover public methods annotated with @ScalarFunction (#8544)
• Support single-valued BigDecimal in schema, type conversion, SQL statements and minimum set of transforms. (#8503)
• Add connection based FailureDetector (#8491)
• Add endpoints for some finer control on minion tasks (#8486)
• Add adhoc minion task creation endpoint (#8465)
• Rewrite PinotQuery based on expression hints at instance/segment level (#8451)
• Allow disabling dict generation for High cardinality columns (#8398)
• add segment size metric on segment push (#8387)
• Implement Native Text Operator (#8384)
• Change default memory allocation for consuming segments from on-heap to off-heap (#8380)
• New Pinot storage metrics for compressed tar.gz and table size w/o replicas (#8358)
• add a experiment API for upsert heap memory estimation (#8355)
• Timestamp type index (#8343)
• Upgrade Helix to 1.0.4 in Pinot (#8325)
• Allow overriding expression in query through query config (#8319)
• Always handle null time values (#8310)
• Add prefixesToRename config for renaming fields upon ingestion (#8273)
• Added multi column partitioning for offline table (#8255)
• Automatically update broker resource on broker changes (#8249)

Vulnerability fixs

Pinot has resolved all the high-level vulnerabilities issues:

• Add a new workflow to check vulnerabilities using trivy (#9044)
• Disable Groovy function by default (#8711)
• Upgrade netty due to security vulnerability (#8328)
• Upgrade protobuf as the current version has security vulnerability (#8287)
• Upgrade to hadoop 2.10.1 due to cves (#8478)
• Upgrade Helix to 1.0.4 (#8325)
• Upgrade thrift to 0.15.0 (#8427)
• Upgrade jetty due to security issue (#8348)
• Upgrade netty (#8346)
• Upgrade snappy version (#8494)

Bug fixs

• Nested arrays and map not handled correctly for complex types (#9235)
• Fix empty data block not returning schema (#9222)
• Allow mvn build with development webpack; fix instances default value (#9179)
• Fix the race condition of reflection scanning classes (#9167)
• Fix ingress manifest for controller and broker (#9135)
• Fix jvm processors count (#9138)
• Fix grpc query server not setting max inbound msg size (#9126)
• Fix upsert replace (#9132)
• Fix the race condition for partial upsert record read (#9130)
• Fix log msg, as it missed one param value (#9124)
• Fix authentication issue when auth annotation is not required (#9110)
• Fix segment pruning that can break server subquery (#9090)
• Fix the NPE for ADLSGen2PinotFS (#9088)
• Fix cross merge (#9087)
• Fix LaunchDataIngestionJobCommand auth header (#9070)
• Fix catalog skipping (#9069)
• Fix adding util for getting URL from InstanceConfig (#8856)
• Fix string length in MutableColumnStatistics (#9059)
• Fix instance details page loading table for tenant (#9035)
• Fix thread safety issue with java client (#8971)
• Fix allSegmentLoaded check (#9010)
• Fix bug in segmentDetails table name parsing; style the new indexes table (#8958)
• Fix pulsar close bug (#8913)
• Fix REGEX OPTION parser (#8905)
• Avoid reporting negative values for server latency. (#8892)
• Fix getConfigOverrides in MinionQuickstart (#8858)
• Fix segment generation error handling (#8812)
• Fix multi stage engine serde (#8689)
• Fix server discovery (#8664)
• Fix Upsert config validation to check for metrics aggregation (#8781)
• Fix multi value column index creation (#8848)
• Fix grpc port assignment in multiple server quickstart (#8834)
• Spark Connector GRPC reader fix for reading realtime tables (#8824)
• Fix auth provider for minion (#8831)
• Fix metadata push mode in IngestionUtils (#8802)
• Misc fixes on segment validation for uploaded real-time segments (#8786)
• Fix a typo in ServerInstance.startQueryServer() (#8794)
• Fix the issue of server opening up query server prematurely (#8785)
• Fix regression where case order was reversed, add regression test (#8748)
• Fix dimension table load when server restart or reload table (#8721)
• Fix when there're two index filter operator h3 inclusion index throw exception (#8707)
• Fix the race condition of reading time boundary info (#8685)
• Fix pruning in expressions by max/min/bloom (#8672)
• Fix GcsPinotFs listFiles by using bucket directly (#8656)
• Fix column data type store for data table (#8648)
• Fix the potential NPE for timestamp index rewrite (#8633)
• Fix on timeout string format in KinesisDataProducer (#8631)
• Fix bug in segment rebalance with replica group segment assignment (#8598)
• Fix the upsert metadata bug when adding segment with same comparison value (#8590)
• Fix the deadlock in ClusterChangeMediator (#8572)
• Fix BigDecimal ser/de on negative scale (#8553)
• Fix table creation bug for invalid realtime consumer props (#8509)
• Fix the bug of missing dot to extract sub props from ingestion job filesytem spec and minion segmentNameGeneratorSpec (#8511)
• Fix to query inconsistencies under heavy upsert load (resolves #7958) (#7971)
• Fix ChildTraceId when using multiple child threads, make them unique (#8443)
• Fix the group-by reduce handling when query times out (#8450)
• Fix a typo in BaseBrokerRequestHandler (#8448)
• Fix TIMESTAMP data type usage during segment creation (#8407)
• Fix async-profiler install (#8404)
• Fix ingestion transform config bugs. (#8394)
• Fix upsert inconsistency by snapshotting the validDocIds before reading the numDocs (#8392)
• Fix bug when importing files with the same name in different directories (#8337)
• Fix the missing NOT handling (#8366)
• Fix setting of metrics compression type in RealtimeSegmentConverter (#8350)
• Fix segment status checker to skip push in-progress segments (#8323)
• Fix datetime truncate for multi-day (#8327)
• Fix redirections for routes with access-token (#8285)
• Fix CSV files surrounding space issue (#9028)
• Fix suppressed exceptions in GrpcBrokerRequestHandler(#8272)

Summary

This release introduces some new great features, performance enhancements, UI improvments, and bug fixes which are described in details in the following sections.
The release was cut from this commit fd9c58a.

Dependency Graph

The dependency graph for plug-and-play architecture that was introduced in release 0.3.0 has been extended and now it contains new nodes for Pinot Segment SPI.

SQL Improvements

• Implement NOT Operator (#8148)
• Add DistinctCountSmartHLLAggregationFunction which automatically store distinct values in Set or HyperLogLog based on cardinality (#8189)
• Add LEAST and GREATEST functions (#8100)
• Handle SELECT * with extra columns (#7959)
• Add FILTER clauses for aggregates (#7916)
• Add ST_Within function (#7990)
• Handle semicolon in query (#7861)
• Add EXPLAIN PLAN (#7568)

UI Enhancements

• Show Reported Size and Estimated Size in human readable format in UI (#8199)
• Make query console state URL based (#8194)
• Improve query console to not show query result when multiple columns have the same name (#8131)
• Improve Pinot dashboard tenant view to show correct amount of servers and brokers (#8115)
• Fix issue with opening new tabs from Pinot Dashboard (#8021)
• Fix issue with Query console going blank on syntax error (#8006)
• Make query stats always show even there's error (#7981)
• Implement OIDC auth workflow in UI (#7121)
• Add tooltip and modal for table status (#7899)
• Add option to wrap lines in custom code mirror (#7857)
• Add ability to comment out queries with cmd + / (#7841)
• Return exception when unavailable segments on empty broker response (#7823)
• Properly handle the case where segments are missing in externalview (#7803)
• Add TIMESTAMP to datetime column Type (#7746)

Performance Improvements

• Reuse regex matcher in dictionary based LIKE queries (#8261)
• Early terminate orderby when columns already sorted (#8228)
• Do not do another pass of Query Automaton Minimization (#8237)
• Improve RangeBitmap by upgrading RoaringBitmap (#8206)
• Optimize geometry serializer usage when literal is available (#8167)
• Improve performance of no-dictionary group by (#8195)
• Allocation free DataBlockCache lookups (#8140)
• Prune unselected THEN statements in CaseTransformFunction (#8138)
• Aggregation delay conversion to double (#8139)
• Reduce object allocation rate in ExpressionContext or FunctionContext (#8124)
• Lock free DimensionDataTableManager (#8102)
• Improve json path performance during ingestion by upgrading JsonPath (#7819)
• Reduce allocations and speed up StringUtil.sanitizeString (#8013)
• Faster metric scans - ForwardIndexReader (#7920)
• Unpeel group by 3 ways to enable vectorization (#7949)
• Power of 2 fixed size chunks (#7934)
• Don't use mmap for compression except for huge chunks (#7931)
• Exit group-by marking loop early (#7935)
• Improve performance of base chunk forward index write (#7930)
• Cache JsonPaths to prevent compilation per segment (#7826)
• Use LZ4 as default compression mode (#7797)
• Peel off special case for 1 dimensional groupby (#7777)
• Bump roaringbitmap version to improve range queries performance (#7734)

Other Notable Features

• Adding NoopPinotMetricFactory and corresponding changes (#8270)
• Allow to specify fixed segment name for SegmentProcessorFramework (#8269)
• Move all prestodb dependencies into a separated module (#8266)
• Include docIds in Projection and Transform block (#8262)
• Automatically update broker resource on broker changes (#8249)
• Update ScalarFunction annotation from name to names to support function alias. (#8252)
• Implemented BoundedColumnValue partition function (#8224)
• Add copy recursive API to pinotFS (#8200)
• Add Support for Getting Live Brokers for a Table (without type suffix) (#8188)
• Pinot docker image - cache prometheus rules (#8241)
• In BrokerRequestToQueryContextConverter, remove unused filterExpressionContext (#8238)
• Adding retention period to segment delete REST API (#8122)
• Pinot docker image - upgrade prometheus and scope rulesets to components (#8227)
• Allow segment name postfix for SegmentProcessorFramework (#8230)
• Superset docker image - update pinotdb version in superset image (#8231)
• Add retention period to deleted segment files and allow table level overrides (#8176)
• Remove incubator from pinot and superset (#8223)
• Adding table config overrides for disabling groovy (#8196)
• Optimise sorted docId iteration order in mutable segments (#8213)
• Adding secure grpc query server support (#8207)
• Move Tls configs and utils from pinot-core to pinot-common (#8210)
• Reduce allocation rate in LookupTransformFunction (#8204)
• Allow subclass to customize what happens pre/post segment uploading (#8203)
• Enable controller service auto-discovery in Jersey framework (#8193)
• Add support for pushFileNamePattern in pushJobSpec (#8191)
• Add additionalMatchLabels to helm chart (#7177)
• Simulate rsvps after meetup.com retired the feed (#8180)
• Adding more checkstyle rules (#8197)
• Add persistence.extraVolumeMounts and persistence.extraVolumes to Kubernetes statefulsets (#7486)
• Adding scala profile for kafka 2.x build and remove root pom scala dependencies (#8174)
• Allow realtime data providers to accept non-kafka producers (#8190)
• Enhance revertReplaceSegments api (#8166)
• Adding broker level config for disabling Pinot queries with Groovy (#8159)
• Make presto driver query pinot server with SQL (#8186)
• Adding controller config for disabling Groovy in ingestionConfig (#8169)
• Adding main method for LaunchDataIngestionJobCommand for spark-submit command (#8168)
• Add auth token for segment replace rest APIs (#8146)
• Add allowRefresh option to UploadSegment (#8125)
• Add Ingress to Broker and Controller helm charts (#7997)
• Improve progress reporter in SegmentCreationMapper (#8129)
• St_* function error messages + support literal transform functions (#8001)
• Add schema and segment crc to SegmentDirectoryContext (#8127)
• Extend enableParallePushProtection support in UploadSegment API (#8110)
• Support BOOLEAN type in Config Recommendation Engine (#8055)
• Add a broker metric to distinguish exception happens when acquire channel lock or when send request to server (#8105)
• Add pinot.minion prefix on minion configs for consistency (#8109)
• Enable broker service auto-discovery in Jersey framework (#8107)
• Timeout if waiting server channel lock takes a long time (#8083)
• Wire EmptySegmentPruner to routing config (#8067)
• Support for TIMESTAMP data type in Config Recommendation Engine (#8087)
• Listener TLS customization (#8082)
• Add consumption rate limiter for LLConsumer (#6291)
• Implement Real Time Mutable FST (#8016)
• Allow quickstart to get table files from filesystem (#8093)
• Add support for instant segment deletion (#8077)
• Add a config file to override quickstart configs (#8059)
• Add pinot server grpc metadata acl (#8030)
• Move compatibility verifier to a separate module (#8049)
• Move hadoop and spark ingestion libs from plugins directory to external-plugins (#8048)
• Add global strategy for partial upsert (#7906)
• Upgrade kafka to 2.8.1 (#7883)
• Created EmptyQuickstart command (#8024)
• Allow SegmentPushUtil to push realtime segment (#8032)
• Add ignoreMerger for partial upsert (#7907)
• Make task timeout and concurrency configurable (#8028)
• Return 503 response from health check on shut down (#7892)
• Pinot-druid-benchmark: set the multiValueDelimiterEnabled to false when importing TPC-H data (#8012)
• Cleanup: Remove remaining occurrences of incubator. (#8023)
• Refactor segment loading logic in BaseTableDataManager to decouple it with local segment directory (#7969)
• Improving segment replacement/revert protocol (#7995)
• PinotConfigProvider interface (#7984)
• Enhance listSegments API to exclude the provided segments from the output (#7878)
• Remove outdated broker metric definitions (#7962)
• Add skip key for realtimeToOffline job validation (#7921)
• Upgrade async-http-client (#7968)
• Allow Reloading Segments with Multiple Threads (#7893)
• Ignore query options in commented out queries (#7894)
• Remove TableConfigCache which does not listen on ZK changes (#7943)
• Switch to zookeeper of helm 3.0x (#7955)
• Use a single react hook for table status modal (#7952)
• Add debug logging for realtime ingestion (#7946)
• Separate the exception for transform and indexing for consuming records (#7926)
• Disable JsonStatementOptimizer (#7919)
• Make index readers/loaders pluggable (#7897)
• Make index creator provision pluggable (#7885)
• Support loading plugins from multiple directories (#7871)
• Update helm charts to honour readinessEnabled probes flags on the Controller, Broker, Server and Minion StatefulSets (#7891)
• Support non-selection-only GRPC server request handler (#7839)
• GRPC broker request handler (#7838)
• Add validator for SDF (#7804)
• Support large payload in zk put API (#7364)
• Push JSON Path evaluation down to storage layer (#7820)
• When upserting new record, index the record before updating the upsert metadata (#7860)
• Add Post-Aggregation Gapfilling functionality. (#7781)
• Clean up deprecated fields from segment metadata (#7853)
• Remove deprecated method from StreamMetadataProvider (#7852)
• Obtain replication factor from tenant configuration in case of dimension table (#7848)
• Use valid bucket end time instead of segment end time for merge/rollup delay metrics (#7827)
• Make pinot start components command extensible (#7847)
• Make upsert inner segment update atomic (#7844)
• Clean up deprecated ZK metadata keys and methods (#7846)
• Add extraEnv, envFrom to statefulset help template (#7833)
• Make openjdk image name configurable (#7832)
• Add getPredicate() to PredicateEvaluator interface (#7840)
• Make split commit the default commit protocol (#7780)
• Pass Pinot connection properties from JDBC driver (#7822)
• Add Pinot client connection config to allow skip fail on broker response exception (#7816)
• Change default range index version to v2 (#7815)
• Put thread timer measuring inside of wall clock timer measuring (#7809)
• Add getRevertReplaceSegmentRequest method in FileUploadDownloadClient (#7796)
• Add JAVA_OPTS env var in docker image (#7799)
• Split thread cpu time into three metrics (#7724)
• Add config for enabling realtime offset based consumption status checker (#7753)
• Add timeColumn, timeUnit and totalDocs to the json segment metadata (#7765)
• Set default Dockerfile CMD to -help (#7767)
• Add getName() to PartitionFunction interface (#7760)
• Support Native FST As An Index Subtype for FST Indices (#7729)
• Add forceCleanup option for 'startReplaceSegments' API (#7744)
• Add config for keystore types, switch tls to native implementation, and add authorization for server-broker tls channel (#7653)
• Extend FileUploadDownloadClient to send post request with json body (#7751)

Major Bug Fixes

• Fix string comparisons (#8253)
• Bugfix for order-by all sorted optimization (#8263)
• Fix dockerfile (#8239)
• Ensure partition function never return negative partition (#8221)
• Handle indexing failures without corrupting inverted indexes (#8211)
• Fixed broken HashCode partitioning (#8216)
• Fix segment replace test (#8209)
• Fix filtered aggregation when it is mixed with regular aggregation (#8172)
• Fix FST Like query benchmark to remove SQL parsing from the measurement (#8097)
• Do not identify function types by throwing exceptions (#8137)
• Fix regression bug caused by sharing TSerializer across multiple threads (#8160)
• Fix validation before creating a table (#8103)
• Check cron schedules from table configs after subscribing child changes (#8113)
• Disallow duplicate segment name in tar file (#8119)
• Fix storage quota checker NPE for Dimension Tables (#8132)
• Fix TraceContext NPE issue (#8126)
• Update gcloud libraries to fix underlying issue with api's with CMEK (#8121)
• Fix error handling in jsonPathArray (#8120)
• Fix error handling in json functions with default values (#8111)
• Fix controller config validation failure for customized TLS listeners (#8106)
• Validate the numbers of input and output files in HadoopSegmentCreationJob (#8098)
• Broker Side validation for the query with aggregation and col but without group by (#7972)
• Improve the proactive segment clean-up for REVERTED (#8071)
• Allow JSON forward indexes (#8073)
• Fix the PinotLLCRealtimeSegmentManager on segment name check (#8058)
• Always use smallest offset for new partitionGroups (#8053)
• Fix RealtimeToOfflineSegmentsTaskExecutor to handle time gap (#8054)
• Refine segment consistency checks during segment load (#8035)
• Fixes for various JDBC issues (#7784)
• Delete tmp- segment directories on server startup (#7961)
• Fix ByteArray datatype column metadata getMaxValue NPE bug and expose maxNumMultiValues (#7918)
• Fix the issues that Pinot upsert table's uploaded segments get deleted when a server restarts. (#7979)
• Fixed segment upload error return (#7957)
• Fix QuerySchedulerFactory to plug in custom scheduler (#7945)
• Fix the issue with grpc broker request handler not started correctly (#7950)
• Fix realtime ingestion when an entire batch of messages is filtered out (#7927)
• Move decode method before calling acquireSegment to avoid reference count leak (#7938)
• Fix semaphore issue in consuming segments (#7886)
• Add bootstrap mode for PinotServiceManager to avoid glitch for health check (#7880)
• Fix the broker routing when segment is deleted (#7817)
• Fix obfuscator not capturing secretkey and keytab (#7794)
• Fix segment merge delay metric when there is empty bucket (#7761)
• Fix QuickStart by adding types for invalid/missing type (#7768)
• Use oldest offset on newly detected partitions (#7756)
• Fix javadoc to compatible with jdk8 source (#7754)
• Handle null segment lineage ZNRecord for getSelectedSegments API (#7752)
• Handle fields missing in the source in ParquetNativeRecordReader (#7742)

Backward Incompatible Changes

• Fix the issue with HashCode partitioning function (#8216)
• Fix the issue with validation on table creation (#8103)
• Change PinotFS API's (#8603)

This is a bug fixing release contains:

Update Log4j to 2.17.0 to address CVE-2021-45105 (#7933)

The release is based on the release 0.9.2 with the following cherry-picks:

93c0404da6bcbf9bf4e165f1e2cbba069abcc872

Summary

This is a bug fixing release contains:

• Upgrade log4j to 2.16.0 to fix CVE-2021-45046 (#7903)
• Upgrade swagger-ui to 3.23.11 to fix CVE-2019-17495 (#7902)
• Fix the bug that RealtimeToOfflineTask failed to progress with large time bucket gaps (#7814).

The release is based on the release 0.9.1 with the following cherry-picks:

9ed6498cdf9d32a65ebcbcce9158acab64a8c0d7
50e1613503cd74b26cf78873efcbdd6e8516bd8f
767aa8abfb5bf085ba0a7ae5ff4024679f27816e

Summary

This release fixes the major issue of CVE-2021-44228 and a major bug fixing of pinot admin exit code issue(https://github.com/apache/pinot/pull/7798).

The release is based on the release 0.9.0 with the following cherry-picks:

e44d2e46f2eaba5f75d789d92ce767fbee96feba
af2858aff26e169f348605e61d0c5e21ddd73dd9

Summary

This release introduces a new features: Segment Merge and Rollup to simplify users day to day operational work. A new metrics plugin is added to support dropwizard. As usual, new functionalities and many UI/ Performance improvements.

The release was cut from the following commit: 13c9ee9 and the following cherry-picks: 668b5e0, ee887b9

Support Segment Merge and Roll-up

LinkedIn operates a large multi-tenant cluster that serves a business metrics dashboard, and noticed that their tables consisted of millions of small segments. This was leading to slow operations in Helix/Zookeeper, long running queries due to having too many tasks to process, as well as using more space because of a lack of compression.

To solve this problem they added the Segment Merge task, which compresses segments based on timestamps and rolls up/aggregates older data. The task can be run on a schedule or triggered manually via the Pinot REST API.

At the moment this feature is only available for offline tables, but will be added for real-time tables in a future release.

Major Changes:

• Integrate enhanced SegmentProcessorFramework into MergeRollupTaskExecutor (#7180)
• Merge/Rollup task scheduler for offline tables. (#7178)
• Fix MergeRollupTask uploading segments not updating their metadata (#7289)
• MergeRollupTask integration tests (#7283)
• Add mergeRollupTask delay metrics (#7368)
• MergeRollupTaskGenerator enhancement: enable parallel buckets scheduling (#7481)
• Use maxEndTimeMs for merge/roll-up delay metrics. (#7617)

UI Improvement

This release also sees improvements to Pinot’s query console UI.

• Cmd+Enter shortcut to run query in query console (#7359)
• Showing tooltip in SQL Editor (#7387)
• Make the SQL Editor box expandable (#7381)
• Fix tables ordering by number of segments (#7564)

SQL Improvements

There have also been improvements and additions to Pinot’s SQL implementation.

New functions:

• IN (#7542)
• LASTWITHTIME (#7584)
• ID_SET on MV columns (#7355)
• Raw results for Percentile TDigest and Est (#7226),
• Add timezone as argument in function toDateTime (#7552)

New predicates are supported:

• LIKE(#7214)
• REGEXP_EXTRACT(#7114)
• FILTER(#7566)

Query compatibility improvements:

• Infer data type for Literal (#7332)
• Support logical identifier in predicate (#7347)
• Support JSON queries with top-level array path expression. (#7511)
• Support configurable group by trim size to improve results accuracy (#7241)

Performance Improvements

This release contains many performance improvement, you may sense it for you day to day queries. Thanks to all the great contributions listed below:

• Reduce the disk usage for segment conversion task (#7193)
• Simplify association between Java Class and PinotDataType for faster mapping (#7402)
• Avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation (#7412)
• Replace MINUS with STRCMP (#7394)
• Bit-sliced range index for int, long, float, double, dictionarized SV columns (#7454)
• Use MethodHandle to access vectorized unsigned comparison on JDK9+ (#7487)
• Add option to limit thread usage per query (#7492)
• Improved range queries (#7513)
• Faster bitmap scans (#7530)
• Optimize EmptySegmentPruner to skip pruning when there is no empty segments (#7531)
• Map bitmaps through a bounded window to avoid excessive disk pressure (#7535)
• Allow RLE compression of bitmaps for smaller file sizes (#7582)
• Support raw index properties for columns with JSON and RANGE indexes (#7615)
• Enhance BloomFilter rule to include IN predicate(#7444) (#7624)
• Introduce LZ4_WITH_LENGTH chunk compression type (#7655)
• Enhance ColumnValueSegmentPruner and support bloom filter prefetch (#7654)
• Apply the optimization on dictIds within the segment to DistinctCountHLL aggregation func (#7630)
• During segment pruning, release the bloom filter after each segment is processed (#7668)
• Fix JSONPath cache inefficient issue (#7409)
• Optimize getUnpaddedString with SWAR padding search (#7708)
• Lighter weight LiteralTransformFunction, avoid excessive array fills (#7707)
• Inline binary comparison ops to prevent function call overhead (#7709)
• Memoize literals in query context in order to deduplicate them (#7720)

Other Notable New Features and Changes

• Human Readable Controller Configs (#7173)
• Add the support of geoToH3 function (#7182)
• Add Apache Pulsar as Pinot Plugin (#7223) (#7247)
• Add dropwizard metrics plugin (#7263)
• Introduce OR Predicate Execution On Star Tree Index (#7184)
• Allow to extract values from array of objects with jsonPathArray (#7208)
• Add Realtime table metadata and indexes API. (#7169)
• Support array with mixing data types (#7234)
• Support force download segment in reload API (#7249)
• Show uncompressed znRecord from zk api (#7304)
• Add debug endpoint to get minion task status. (#7300)
• Validate CSV Header For Configured Delimiter (#7237)
• Add auth tokens and user/password support to ingestion job command (#7233)
• Add option to store the hash of the upsert primary key (#7246)
• Add null support for time column (#7269)
• Add mode aggregation function (#7318)
• Support disable swagger in Pinot servers (#7341)
• Delete metadata properly on table deletion (#7329)
• Add basic Obfuscator Support (#7407)
• Add AWS sts dependency to enable auth using web identity token. (#7017)(#7445)
• Mask credentials in debug endpoint /appconfigs (#7452)
• Fix /sql query endpoint now compatible with auth (#7230)
• Fix case sensitive issue in BasicAuthPrincipal permission check (#7354)
• Fix auth token injection in SegmentGenerationAndPushTaskExecutor (#7464)
• Add segmentNameGeneratorType config to IndexingConfig (#7346)
• Support trigger PeriodicTask manually (#7174)
• Add endpoint to check minion task status for a single task. (#7353)
• Showing partial status of segment and counting CONSUMING state as good segment status (#7327)
• Add "num rows in segments" and "num segments queried per host" to the output of Realtime Provisioning Rule (#7282)
• Check schema backward-compatibility when updating schema through addSchema with override (#7374)
• Optimize IndexedTable (#7373)
• Support indices remove in V3 segment format (#7301)
• Optimize TableResizer (#7392)
• Introduce resultSize in IndexedTable (#7420)
• Offset based realtime consumption status checker (#7267)
• Add causes to stack trace return (#7460)
• Create controller resource packages config key (#7488)
• Enhance TableCache to support schema name different from table name (#7525)
• Add validation for realtimeToOffline task (#7523)
• Unify CombineOperator multi-threading logic (#7450)
• Support no downtime rebalance for table with 1 replica in TableRebalancer (#7532)
• Introduce MinionConf, move END_REPLACE_SEGMENTS_TIMEOUT_MS to minion config instead of task config. (#7516)
• Adjust tuner api (#7553)
• Adding config for metrics library (#7551)
• Add geo type conversion scalar functions (#7573)
• Add BOOLEAN_ARRAY and TIMESTAMP_ARRAY types (#7581)
• Add MV raw forward index and MV BYTES data type (#7595)
• Enhance TableRebalancer to offload the segments from most loaded instances first (#7574)
• Improve get tenant API to differentiate offline and realtime tenants (#7548)
• Refactor query rewriter to interfaces and implementations to allow customization (#7576)
• In ServiceStartable, apply global cluster config in ZK to instance config (#7593)
• Make dimension tables creation bypass tenant validation (#7559)
• Allow Metadata and Dictionary Based Plans for No Op Filters (#7563)
• Reject query with identifiers not in schema (#7590)
• Round Robin IP addresses when retry uploading/downloading segments (#7585)
• Support multi-value derived column in offline table reload (#7632)
• Support segmentNamePostfix in segment name (#7646)
• Add select segments API (#7651)
• Controller getTableInstance() call now returns the list of live brokers of a table. (#7556)
• Allow MV Field Support For Raw Columns in Text Indices (#7638)
• Allow override distinctCount to segmentPartitionedDistinctCount (#7664)
• Add a quick start with both UPSERT and JSON index (#7669)
• Add revertSegmentReplacement API (#7662)
• Smooth segment reloading with non blocking semantic (#7675)
• Clear the reused record in PartitionUpsertMetadataManager (#7676)
• Replace args4j with picocli (#7665)
• Handle datetime column consistently (#7645)(#7705)
• Allow to carry headers with query requests (#7696) (#7712)
• Allow adding JSON data type for dimension column types (#7718)
• Separate SegmentDirectoryLoader and tierBackend concepts (#7737)
• Implement size balanced V4 raw chunk format (#7661)
• Add presto-pinot-driver lib (#7384)

Major Bug fixes

• Fix null pointer exception for non-existed metric columns in schema for JDBC driver (#7175)
• Fix the config key for TASK_MANAGER_FREQUENCY_PERIOD (#7198)
• Fixed pinot java client to add zkClient close (#7196)
• Ignore query json parse errors (#7165)
• Fix shutdown hook for PinotServiceManager (#7251) (#7253)
• Make STRING to BOOLEAN data type change as backward compatible schema change (#7259)
• Replace gcp hardcoded values with generic annotations (#6985)
• Fix segment conversion executor for in-place conversion (#7265)
• Fix reporting consuming rate when the Kafka partition level consumer isn't stopped (#7322)
• Fix the issue with concurrent modification for segment lineage (#7343)
• Fix TableNotFound error message in PinotHelixResourceManager (#7340)
• Fix upload LLC segment endpoint truncated download URL (#7361)
• Fix task scheduling on table update (#7362)
• Fix metric method for ONLINE_MINION_INSTANCES metric (#7363)
• Fix JsonToPinotSchema behavior to be consistent with AvroSchemaToPinotSchema (#7366)
• Fix currentOffset volatility in consuming segment(#7365)
• Fix misleading error msg for missing URI (#7367)
• Fix the correctness of getColumnIndices method (#7370)
• Fix SegmentZKMetadta time handling (#7375)
• Fix retention for cleaning up segment lineage (#7424)
• Fix segment generator to not return illegal filenames (#7085)
• Fix missing LLC segments in segment store by adding controller periodic task to upload them (#6778)
• Fix parsing error messages returned to FileUploadDownloadClient (#7428)
• Fix manifest scan which drives /version endpoint (#7456)
• Fix missing rate limiter if brokerResourceEV becomes null due to ZK connection (#7470)
• Fix race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks: (#7427)
• Fix pql double quote checker exception (#7485)
• Fix minion metrics exporter config (#7496)
• Fix segment unable to retry issue by catching timeout exception during segment replace (#7509)
• Add Exception to Broker Response When Not All Segments Are Available (Partial Response) (#7397)
• Fix segment generation commands (#7527)
• Return non zero from main with exception (#7482)
• Fix parquet plugin shading error (#7570)
• Fix the lowest partition id is not 0 for LLC (#7066)
• Fix star-tree index map when column name contains '.' (#7623)
• Fix cluster manager URLs encoding issue(#7639)
• Fix fieldConfig nullable validation (#7648)
• Fix verifyHostname issue in FileUploadDownloadClient (#7703)
• Fix TableCache schema to include the built-in virtual columns (#7706)
• Fix DISTINCT with AS function (#7678)
• Fix SDF pattern in DataPreprocessingHelper (#7721)
• Fix fields missing issue in the source in ParquetNativeRecordReader (#7742)

Summary

This release introduced several awesome new features, including compatibility tests, enhanced complex type and Json support, partial upsert support, and new stream ingestion plugins (AWS Kinesis, Apache Pulsar). It contains a lot of query enhancements such as new timestamp and boolean type support and flexible numerical column comparison. It also includes many key bug fixes. See details below.

The release was cut from the following commit: fe83e95aa9124ee59787c580846793ff7456eaa5

and the following cherry-picks:

• 668b5e01d7c263e3cdb3081e0a947a43e7d7f782
• ee887b97e77ef7a132d3d6d60f83a800e52d4555
• c2f7fccefcbd9930c995808cf9947c61b4223786
• c1ac8a18b65841fc666722496cc5f4f9347b3dd4
• 4da1dae06aef50f0a7c96b5a22019e541310fdd9
• 573651b28a6f89bd4895c992a5e8fa8e23df4615
• c6c407d24c3e02e6b3f628e691118f379575d6da
• 0d96c7f5f58191a823956cc1b1a8c93914fd73b3
• c2637d139dd5ba79682b4244cde316dacb0852ee

Notable New Features

• Extract time handling for SegmentProcessorFramework (#7158)
• Add Apache Pulsar low level and high level connector (#7026)
• Enable parallel builds for compat checker (#7149)
• Add controller/server API to fetch aggregated segment metadata (#7102)
• Support Dictionary Based Plan For DISTINCT (#7141)
• Provide HTTP client to kinesis builder (#7148)
• Add datetime function with 2 arguments (#7116)
• Adding ability to check ingestion status for Offline Pinot table (#7070)
• Add timestamp datatype support in JDBC (#7117)
• Allow updating controller and broker helix hostname (#7064)
• Cancel running Kinesis consumer tasks when timeout occurs (#7109)
• Implement Append merger for partial upsert (#7087)
• SegmentProcessorFramework Enhancement (#7092)
• Added TaskMetricsEmitted periodic controler job (#7091)
• Support json path expressions in query. (#6998)
• Support data preprocessing for AVRO and ORC formats (#7062)
• Add partial upsert config and mergers (#6899)
• Add support for range index rule recommendation(#7034) (#7063)
• Allow reloading consuming segment by default (#7078)
• Add LZ4 Compression Codec (#6804) (#7035)
• Make Pinot JDK 11 Compilable (#6424)
• Introduce in-Segment Trim for GroupBy OrderBy Query (#6991)
• Produce GenericRow file in segment processing mapper (#7013)
• Add ago() scalar transform function (#6820)
• Add Bloom Filter support for IN predicate(#7005) (#7007)
• Add genericRow file reader and writer (#6997)
• Normalize LHS and RHS numerical types for >, >=, <, and <= operators. (#6927)
• Add Kinesis Stream Ingestion Plugin (#6661)
• feature/#6766 JSON and Startree index information in API (#6873)
• Support null value fields in generic row ser/de (#6968)
• Implement PassThroughTransformOperator to optimize select queries(#6972) (#6973)
• Optimize TIME_CONVERT/DATE_TIME_CONVERT predicates (#6957)
• Prefetch call to fetch buffers of columns seen in the query (#6967)
• Enabling compatibility tests in the script (#6959)
• Add collectionToJsonMode to schema inference (#6946)
• Add the complex-type support to decoder/reader (#6945)
• Adding a new Controller API to retrieve ingestion status for realtime… (#6890)
• Add support for Long in Modulo partition function. (#6929)
• Enhance PinotSegmentRecordReader to preserve null values (#6922)
• add complex-type support to avro-to-pinot schema inference (#6928)
• Add correct yaml files for real time data(#6787) (#6916)
• Add complex-type transformation to offline segment creation (#6914)
• Add config File support(#6787) (#6901)
• Enhance JSON index to support nested array (#6877)
• Add debug endpoint for tables. (#6897)
• JSON column datatype support. (#6878)
• Allow empty string in MV column (#6879)
• Add Zstandard compression support with JMH benchmarking(#6804) (#6876)
• Normalize LHS and RHS numerical types for = and != operator. (#6811)
• Change ConcatCollector implementation to use off-heap (#6847)
• [PQL Deprecation] Clean up the old BrokerRequestOptimizer (#6859)
• [PQL Deprecation] Do not compile PQL broker request for SQL query (#6855)
• Add TIMESTAMP and BOOLEAN data type support (#6719)
• Add admin endpoint for Pinot Minon. (#6822)
• Remove the usage of PQL compiler (#6808
• Add endpoints in Pinot Controller, Broker and Server to get system and application configs. (#6817)
• Support IN predicate in ColumnValue SegmentPruner(#6756) (#6776)
• Enable adding new segments to a upsert-enabled realtime table (#6567)
• Interface changes for Kinesis connector (#6667)
• Pinot Minion SegmentGenerationAndPush task: PinotFS configs inside taskSpec is always temporary and has higher priority than default PinotFS created by the minion server configs (#6744)
• DataTable V3 implementation and measure data table serialization cost on server (#6710)
• add uploadLLCSegment endpoint in TableResource (#6653)
• File-based SegmentWriter implementation (#6718)
• Basic Auth for pinot-controller (#6613)
• UI integration with Authentication API and added login page (#6686)
• Support data ingestion for offline segment in one pass (#6479)
• SumPrecision: support all data types and star-tree (#6668)
• complete compatibility regression testing (#6650)
• Kinesis implementation Part 1: Rename partitionId to partitionGroupId (#6655)
• Make Pinot metrics pluggable (#6640)
• Recover the segment from controller when LLC table cannot load it (#6647)
• Adding a new API for validating specified TableConfig and Schema (#6620)
• Introduce a metric for query/response size on broker. (#6590)
• Adding a controller periodic task to clean up dead minion instances (#6543)
• Adding new validation for Json, TEXT indexing (#6541)
• Always return a response from query execution. (#6596)

Special notes

• After the 0.8.0 release, we will officially support jdk 11, and can now safely start to use jdk 11 features. Code is still compilable with jdk 8 (#6424)
• RealtimeToOfflineSegmentsTask config has some backward incompatible changes (#7158)
  — timeColumnTransformFunction is removed (backward-incompatible, but rollup is not supported anyway)
  — Deprecate collectorType and replace it with mergeType
  — Add roundBucketTimePeriod and partitionBucketTimePeriod to config the time bucket for round and partition
• Regex path for pluggable MinionEventObserverFactory is changed from org.apache.pinot.*.event.* to org.apache.pinot.*.plugin.minion.tasks.* (#6980)
• Moved all pinot built-in minion tasks to the pinot-minion-builtin-tasks module and package them into a shaded jar (#6618)
• Reloading consuming segment flag pinot.server.instance.reload.consumingSegment will be true by default (#7078)
• Move JSON decoder from pinot-kafka to pinot-json package. (#7021)
• Backward incompatible schema change through controller rest API PUT /schemas/{schemaName} will be blocked. (#6737)
• Deprecated /tables/validateTableAndSchema in favor of the new configs/validate API and introduced new APIs for /tableConfigs to operate on the realtime table config, offline table config and schema in one shot. (#6840)

Major Bug fixes

• Fix race condition in MinionInstancesCleanupTask (#7122)
• Fix custom instance id for controller/broker/minion (#7127)
• Fix UpsertConfig JSON deserialization. (#7125)
• Fix the memory issue for selection query with large limit (#7112)
• Fix the deleted segments directory not exist warning (#7097)
• Fixing docker build scripts by providing JDK_VERSION as parameter (#7095)
• Misc fixes for json data type (#7057)
• Fix handling of date time columns in query recommender(#7018) (#7031)
• fixing pinot-hadoop and pinot-spark test (#7030)
• Fixing HadoopPinotFS listFiles method to always contain scheme (#7027)
• fixed GenericRow compare for different _fieldToValueMap size (#6964)
• Fix NPE in NumericalFilterOptimizer due to IS NULL and IS NOT NULL operator. (#7001)
• Fix the race condition in realtime text index refresh thread (#6858) (#6990)
• Fix deep store directory structure (#6976)
• Fix NPE issue when consumed kafka message is null or the record value is null. (#6950)
• Mitigate calcite NPE bug. (#6908)
• Fix the exception thrown in the case that a specified table name does not exist (#6328) (#6765)
• Fix CAST transform function for chained transforms (#6941)
• Fixed failing pinot-controller npm build (#6795)

Summary

This release introduced several awesome new features, including JSON index, lookup-based join support, geospatial support, TLS support for pinot connections, and various performance optimizations and improvements. It also adds several new APIs to better manage the segments and upload data to offline table. It also contains many key bug fixes. See details below.

The release was cut from the following commit: 78152cdb2892cf8c2df5b8a4d04e2aa897333487

and the following cherry-picks:

• b527af353e78f26d0c4388cab89e4fe18d5290f6
• 84d59e3ba27a3cdf0eaecbfe0eeec9b47060a2e3
• a18dc60dca09bd2a1d33a8bc6b787d7ceb8e1749
• 4ec38f79315d4017e5e2ac45e8989fa7fc4584fa
• b48dac07dfce0769ad4acf1a643f3e9aba53e18b
• 5d2bc0c6d83825e152d30fb4774464fabf3b3e8b
• 913492e443d71d758f0b88b17cb144b5a5a5fb57
• 50a4531b33475327bc9fe3c0199e7003f0a4c882
• 1f21403a29a40c751fceb2211437a1e27c58b5e1
• 8dbb70ba08daf90f5e9067fcec545203ffefe215

Notable New Features

• Add a server metric: queriesDisabled to check if queries disabled or not. (#6586)
• Optimization on GroupKey to save the overhead of ser/de the group keys (#6593) (#6559)
• Support validation for jsonExtractKey and jsonExtractScalar functions (#6246) (#6594)
• Real-Time Provisioning Helper tool improvement to take data characteristics as input instead of an actual segment (#6546)
• Add the isolation level config isolation.level to Kafka consumer (2.0) to ingest transactionally committed messages only (#6580)
• Enhance StarTreeIndexViewer to support multiple trees (#6569)
• Improves ADLSGen2PinotFS with service principal-based auth, auto-create container on the initial run. It's backward compatible with key-based auth. (#6531)
• Add metrics for minion tasks status (#6549)
• Use minion data directory as tmp directory for SegmentGenerationAndPushTask to ensure directory is always cleaned up (#6560)
• Add optional HTTP basic auth to pinot broker, which enables user- and table-level authentication of incoming queries. (#6552)
• Add Access Control for REST endpoints of Controller (#6507)
• Add date_trunc to scalar functions to support date_trunc during ingestion (#6538)
• Allow tar gz with > 8gb size (#6533)
• Add Lookup UDF Join support (#6530), (#6465), (#6383) (#6286)
• Add cron scheduler metrics reporting (#6502)
• Support generating derived column during segment load, so that derived columns can be added on-the-fly (#6494)
• Support chained transform functions (#6495)
• Add scalar function JsonPathArray to extract arrays from json (#6490)
• Add a guard against multiple consuming segments for same partition (#6483)
• Remove the usage of deprecated range delimiter (#6475)
• Handle scheduler calls with a proper response when it's disabled. (#6474)
• Simplify SegmentGenerationAndPushTask handling getting schema and table config (#6469)
• Add a cluster config to config number of concurrent tasks per instance for minion task: SegmentGenerationAndPushTaskGenerator (#6468)
• Replace BrokerRequestOptimizer with QueryOptimizer to also optimize the PinotQuery (#6423)
• Add additional string scalar functions (#6458)
• Add additional scalar functions for array type (#6446)
• Add CRON scheduler for Pinot tasks (#6451)
• Set default Data Type while setting type in Add Schema UI dialog (#6452)
• Add ImportData sub command in pinot admin (#6396)
• H3-based geospatial index (#6409) (#6306)
• Add JSON index support (#6408) (#6216) (#6346)
• Make minion tasks pluggable via reflection (#6395)
• Add compatibility test for segment operations upload and delete (#6382)
• Add segment reset API that disables and then enables the segment (#6336)
• Add Pinot minion segment generation and push task. (#6340)
• Add a version option to pinot admin to show all the component versions (#6380)
• Add FST index using Lucene lib to speedup REGEXP_LIKE operator on text (#6120)
• Add APIs for uploading data to an offline table. (#6354)
• Allow the use of environment variables in stream configs (#6373)
• Enhance task schedule api for single type/table support (#6352)
• Add broker time range based pruner for routing. Query operators supported: RANGE, =, <, <=, >, >=, AND, OR(#6259)
• Add json path functions to extract values from json object (#6347)
• Create a pluggable interface for Table config tuner (#6255)
• Add a Controller endpoint to return table creation time (#6331)
• Add tooltips, ability to enable-disable table state to the UI (#6327)
• Add Pinot Minion client (#6339)
• Add more efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator (#6320)
• Add decimal percentile support. (#6323)
• Add API to get status of consumption of a table (#6322)
• Add support to add offline and realtime tables, individually able to add schema and schema listing in UI (#6296)
• Improve performance for distinct queries (#6285)
• Allow adding custom configs during the segment creation phase (#6299)
• Use sorted index based filtering only for dictionary encoded column (#6288)
• Enhance forward index reader for better performance (#6262)
• Support for text index without raw (#6284)
• Add api for cluster manager to get table state (#6211)
• Perf optimization for SQL GROUP BY ORDER BY (#6225)
• Add support using environment variables in the format of ${VAR_NAME:DEFAULT_VALUE} in Pinot table configs. (#6271)

Special notes

• Pinot controller metrics prefix is fixed to add a missing dot (#6499). This is a backward-incompatible change that JMX query on controller metrics must be updated

• Legacy group key delimiter (\t) was removed to be backward-compatible with release 0.5.0 (#6589)

• Upgrade zookeeper version to 3.5.8 to fix ZOOKEEPER-2184: Zookeeper Client should re-resolve hosts when connection attempts fail. (#6558)

• Add TLS-support for client-pinot and pinot-internode connections (#6418)
  Upgrades to a TLS-enabled cluster can be performed safely and without downtime. To achieve a live-upgrade, go through the following steps:

  • First, configure alternate ingress ports for https/netty-tls on brokers, controllers, and servers. Restart the components with a rolling strategy to avoid cluster downtime.
  • Second, verify manually that https access to controllers and brokers is live. Then, configure all components to prefer TLS-enabled connections (while still allowing unsecured access). Restart the individual components.
  • Third, disable insecure connections via configuration. You may also have to set controller.vip.protocol and controller.vip.port and update the configuration files of any ingestion jobs. Restart components a final time and verify that insecure ingress via http is not available anymore.

• PQL endpoint on Broker is deprecated (#6607)

  • Apache Pinot has adopted SQL syntax and semantics. Legacy PQL (Pinot Query Language) is deprecated and no longer supported. Please use SQL syntax to query Pinot on broker endpoint /query/sql and controller endpoint /sql

Major Bug fixes

• Fix the SIGSEGV for large index (#6577)
• Handle creation of segments with 0 rows so segment creation does not fail if the data source has 0 rows. (#6466)
• Fix QueryRunner tool for multiple runs (#6582)
• Use URL encoding for the generated segment tar name to handle characters that cannot be parsed to URI. (#6571)
• Fix a bug of miscounting the top nodes in StarTreeIndexViewer (#6569)
• Fix the raw bytes column in real-time segment (#6574)
• Fixes a bug to allow using JSON_MATCH predicate in SQL queries (#6535)
• Fix the overflow issue when loading the large dictionary into the buffer (#6476)
• Fix empty data table for distinct query (#6363)
• Fix the default map return value in DictionaryBasedGroupKeyGenerator (#6712)
• Fix log message in ControllerPeriodicTask (#6709)
• Fix bug #6671: RealtimeTableDataManager shuts down SegmentBuildTimeLeaseExtender for all tables in the host (#6682)
• Fix license headers and plugin checks

Summary

This release introduced some excellent new features, including upsert, tiered storage, pinot-spark-connector, support of having clause, more validations on table config and schema, support of ordinals in GROUP BY and ORDER BY clause, array transform functions, adding push job type of segment metadata only mode, and some new APIs like updating instance tags, new health check endpoint. It also contains many key bug fixes. See details below.

The release was cut from the following commit:
e5c9bec
and the following cherry-picks:

• d033a11

Notable New Features

• Tiered storage (#5793)
• Upsert feature (#6096, #6113, #6141, #6149, #6167)
• Pre-generate aggregation functions in QueryContext (#5805)
• Adding controller healthcheck endpoint: /health (#5846)
• Add pinot-spark-connector (#5787)
• Support multi-value non-dictionary group by (#5851)
• Support type conversion for all scalar functions (#5849)
• Add additional datetime functionality (#5438)
• Support post-aggregation in ORDER-BY (#5856)
• Support post-aggregation in SELECT (#5867)
• Add RANGE FilterKind to support merging ranges for SQL (#5898)
• Add HAVING support (#5889)
• Support for exact distinct count for non int data types (#5872)
• Add max qps bucket count (#5922)
• Add Range Indexing support for raw values (#5853)
• Add IdSet and IdSetAggregationFunction (#5926)
• [Deepstore by-pass]Add a Deepstore bypass integration test with minor bug fixes. (#5857)
• Add Hadoop counters for detecting schema mismatch (#5873)
• Add RawThetaSketchAggregationFunction (#5970)
• Instance API to directly updateTags (#5902)
• Add streaming query handler (#5717)
• Add InIdSetTransformFunction (#5973)
• Add ingestion descriptor in the header (#5995)
• Zookeeper put api (#5949)
• Feature/#5390 segment indexing reload status api (#5718)
• Segment processing framework (#5934)
• Support streaming query in QueryExecutor (#6027)
• Add list of allowed tables for emitting table level metrics (#6037)
• Add FilterOptimizer which supports optimizing both PQL and SQL query filter (#6056)
• Adding push job type of segment metadata only mode (#5967)
• Minion taskExecutor for RealtimeToOfflineSegments task (#6050, #6124)
• Adding array transform functions: array_average, array_max, array_min, array_sum (#6084)
• Allow modifying/removing existing star-trees during segment reload (#6100)
• Implement off-heap bloom filter reader (#6118)
• Support for multi-threaded Group By reducer for SQL. (#6044)
• Add OnHeapGuavaBloomFilterReader (#6147)
• Support using ordinals in GROUP BY and ORDER BY clause (#6152)
• Merge common APIs for Dictionary (#6176)
• Add table level lock for segment upload (#6165)
• Added recursive functions validation check for group by (#6186)
• Add StrictReplicaGroupInstanceSelector (#6208)
• Add IN_SUBQUERY support (#6022)
• Add IN_PARTITIONED_SUBQUERY support (#6043)
• Some UI features (#5810, #5981, #6117, #6215)

Special notes

• Brokers should be upgraded before servers in order to keep backward-compatible:
  — Change group key delimiter from '\t' to '\0' (#5858)
  — Support for exact distinct count for non int data types (#5872)
• Pinot Components have to be deployed in the following order:
  (PinotServiceManager -> Bootstrap services in role ServiceRole.CONTROLLER -> All remaining bootstrap services in parallel)
  — Starts Broker and Server in parallel when using ServiceManager (#5917)
• New settings introduced and old ones deprecated:
  — Make realtime threshold property names less ambiguous (#5953)
  — Change Signature of Broker API in Controller (#6119)
• This aggregation function is still in beta version. This PR involves change on the format of data sent from server to broker, so it works only when both broker and server are upgraded to the new version:
  — Enhance DistinctCountThetaSketchAggregationFunction (#6004)

Major Bug fixes

• Improve performance of DistinctCountThetaSketch by eliminating empty sketches and unions. (#5798)
• Enhance VarByteChunkSVForwardIndexReader to directly read from data buffer for uncompressed data (#5816)
• Fixing backward-compatible issue of schema fetch call (#5885)
• Fix race condition in MetricsHelper (#5887)
• Fixing the race condition that segment finished before ControllerLeaderLocator created. (#5864)
• Fix CSV and JSON converter on BYTES column (#5931)
• Fixing the issue that transform UDFs are parsed as function name 'OTHER', not the real function names (#5940)
• Incorporating embedded exception while trying to fetch stream offset (#5956)
• Use query timeout for planning phase (#5990)
• Add null check while fetching the schema (#5994)
• Validate timeColumnName when adding/updating schema/tableConfig (#5966)
• Handle the partitioning mismatch between table config and stream (#6031)
• Fix built-in virtual columns for immutable segment (#6042)
• Refresh the routing when realtime segment is committed (#6078)
• Add support for Decimal with Precision Sum aggregation (#6053)
• Fixing the calls to Helix to throw exception if zk connection is broken (#6069)
• Allow modifying/removing existing star-trees during segment reload (#6100)
• Add max length support in schema builder (#6112)
• Enhance star-tree to skip matching-all predicate on non-star-tree dimension (#6109)

Backward Incompatible Changes

• Make realtime threshold property names less ambiguous (#5953)
• Enhance DistinctCountThetaSketchAggregationFunction (#6004)
• Deep Extraction Support for ORC, Thrift, and ProtoBuf Records (#6046)

Summary

This release includes many new features on Pinot ingestion and connectors (e.g., support for filtering during ingestion which is configurable in table config; support for json during ingestion; proto buf input format support and a new Pinot JDBC client), query capability (e.g., a new GROOVY transform function UDF) and admin functions (a revamped Cluster Manager UI & Query Console UI). It also contains many key bug fixes. See details below.

The release was cut from the following commit:
d1b4586
and the following cherry-picks:

• 63a4fd4
• a7f7f46
• dafbef1
• ced3a70
• d902c1a

Notable New Features

• Allowing update on an existing instance config: PUT /instances/{instanceName} with Instance object as the pay-load (#PR4952)
• Add PinotServiceManager to start Pinot components (#PR5266)
• Support for protocol buffers input format. (#PR5293)
• Add GenericTransformFunction wrapper for simple ScalarFunctions (PR#5440)
  — Adding support to invoke any scalar function via GenericTransformFunction
• Add Support for SQL CASE Statement (PR#5461)
• Support distinctCountRawThetaSketch aggregation that returns serialized sketch. (PR#5465)
• Add multi-value support to SegmentDumpTool (PR#5487)
  — add segment dump tool as part of the pinot-tool.sh script
• Add json_format function to convert json object to string during ingestion. (PR#5492)
  — Can be used to store complex objects as a json string (which can later be queries using jsonExtractScalar)
• Support escaping single quote for SQL literal (PR#5501)
  — This is especially useful for DistinctCountThetaSketch because it stores expression as literal
  E.g. DistinctCountThetaSketch(..., 'foo=''bar''', ...)
• Support expression as the left-hand side for BETWEEN and IN clause (PR#5502)
• Add a new field IngestionConfig in TableConfig
  — FilterConfig: ingestion level filtering of records, based on filter function. (PR#5597)
  — TransformConfig: ingestion level column transformations. This was previously introduced in Schema (FieldSpec#transformFunction), and has now been moved to TableConfig. It continues to remain under schema, but we recommend users to set it in the TableConfig starting this release (PR#5681).
• Allow star-tree creation during segment load (#PR5641)
  — Introduced a new boolean config enableDynamicStarTreeCreation in IndexingConfig to enable/disable star-tree creation during segment load.
• Support for Pinot clients using JDBC connection (#PR5602)
• Support customized accuracy for distinctCountHLL, distinctCountHLLMV functions by adding log2m value as the second parameter in the function. (#PR5564)
  —Adding cluster config: default.hyperloglog.log2m to allow user set default log2m value.
• Add segment encryption on Controller based on table config (PR#5617)
• Add a constraint to the message queue for all instances in Helix, with a large default value of 100000. (PR#5631)
• Support order-by aggregations not present in SELECT (PR#5637)
  — Example: "select subject from transcript group by subject order by count() desc"
  This is equivalent to the following query but the return response should not contain count().
  "select subject, count() from transcript group by subject order by count() desc"
• Add geo support for Pinot queries (PR#5654)
  — Added geo-spatial data model and geospatial functions
• Cluster Manager UI & Query Console UI revamp (PR#5684 and PR#5732)
  — updated cluster manage UI and added table details page and segment details page
• Add Controller API to explore Zookeeper (PR#5687)
• Support BYTES type for dictinctCount and group-by (PR#5701 and PR#5708)
  —Add BYTES type support to DistinctCountAggregationFunction
  —Correctly handle BYTES type in DictionaryBasedAggregationOperator for DistinctCount
• Support for ingestion job spec in JSON format (#PR5729)
• Improvements to RealtimeProvisioningHelper command (#PR5737)
  — Improved docs related to ingestion and plugins
• Added GROOVY transform function UDF (#PR5748)
  — Ability to run a groovy script in the query as a UDF. e.g. string concatenation:
  SELECT GROOVY('{"returnType": "INT", "isSingleValue": true}', 'arg0 + " " + arg1', columnA, columnB) FROM myTable

Special notes

• Changed the stream and metadata interface (PR#5542)
  — This PR concludes the work for the issue #5359 to extend offset support for other streams
• TransformConfig: ingestion level column transformations. This was previously introduced in Schema (FieldSpec#transformFunction), and has now been moved to TableConfig. It continues to remain under schema, but we recommend users to set it in the TableConfig starting this release (PR#5681).
• Config key enable.case.insensitive.pql in Helix cluster config is deprecated, and replaced with enable.case.insensitive. (#PR5546)
• Change default segment load mode to MMAP. (PR#5539)
  —The load mode for segments currently defaults to heap.

Major Bug fixes

• Fix bug in distinctCountRawHLL on SQL path (#5494)
• Fix backward incompatibility for existing stream implementations (#5549)
• Fix backward incompatibility in StreamFactoryConsumerProvider (#5557)
• Fix logic in isLiteralOnlyExpression. (#5611)
• Fix double memory allocation during operator setup (#5619)
• Allow segment download url in Zookeeper to be deep store uri instead of hardcoded controller uri (#5639)
• Fix a backward compatible issue of converting BrokerRequest to QueryContext when querying from Presto segment splits (#5676)
• Fix the issue that PinotSegmentToAvroConverter does not handle BYTES data type. (#5789)

Backward Incompatible Changes

• PQL queries with HAVING clause will no longer be accepted for the following reasons: (#PR5570)
  — HAVING clause does not apply to PQL GROUP-BY semantic where each aggregation column is ordered individually
  — The current behavior can produce inaccurate results without any notice
  — HAVING support will be added for SQL queries in the next release
• Because of the standardization of the DistinctCountThetaSketch predicate strings, please upgrade Broker before Server. The new Broker can handle both standard and non-standard predicate strings for backward-compatibility. (#PR5613)

Summary

This release introduced various new features, including the theta-sketch based distinct count aggregation function, an S3 filesystem plugin, a unified star-tree index implementation, deprecation of TimeFieldSpec in favor of DateTimeFieldSpec, etc. Miscellaneous refactoring, performance improvement and bug fixes were also included in this release. See details below.

The release was cut from this commit:
https://github.com/apache/incubator-pinot/commit/008be2db874dd1c0d7877ce712842abd818d89d1
with cherry-picking the following patches:

• https://github.com/apache/incubator-pinot/commit/7f10c5c9a571ba1ca8d8ebee5d2b74fc3b05cd7b
• https://github.com/apache/incubator-pinot/commit/ee21e793c0365f538bcb3b801f47122c59fc0e04
• https://github.com/apache/incubator-pinot/commit/a314d42e8744549e5c182383445f11c60ac4ae4a

Notable New Features

• Made DateTimeFieldSpecs mainstream and deprecated TimeFieldSpec (#2756)
  • Used time column from table config instead of schema (#5320)
  • Included dateTimeFieldSpec in schema columns of Pinot Query Console #5392
  • Used DATE_TIME as the primary time column for Pinot tables (#5399)
• Supported range queries using indexes (#5240)
• Supported complex aggregation functions
  • Supported Aggregation functions with multiple arguments (#5261)
  • Added api in AggregationFunction to get compiled input expressions (#5339)
• Added a simple PinotFS benchmark driver (#5160)
• Supported default star-tree (#5147)
• Added an initial implementation for theta-sketch based distinct count aggregation function (#5316)
  • One minor side effect: DataSchemaPruner won't work for DistinctCountThetaSketchAggregatinoFunction (#5382)
• Added access control for Pinot server segment download api (#5260)
• Added Pinot S3 Filesystem Plugin (#5249)
• Text search improvement
  • Pruned stop words for text index (#5297)
  • Used 8byte offsets in chunk based raw index creator (#5285)
  • Derived num docs per chunk from max column value length for varbyte raw index creator (#5256)
  • Added inter segment tests for text search and fixed a bug for Lucene query parser creation (#5226)
  • Made text index query cache a configurable option (#5176)
  • Added Lucene DocId to PinotDocId cache to improve performance (#5177)
  • Removed the construction of second bitmap in text index reader to improve performance (#5199)
• Tooling/usability improvement
  • Added template support for Pinot Ingestion Job Spec (#5341)
  • Allowed user to specify zk data dir and don't do clean up during zk shutdown (#5295)
  • Allowed configuring minion task timeout in the PinotTaskGenerator (#5317)
  • Update JVM settings for scripts (#5127)
  • Added Stream github events demo (#5189)
  • Moved docs link from gitbook to docs.pinot.apache.org (#5193)
• Re-implemented ORCRecordReader (#5267)
• Evaluated schema transform expressions during ingestion (#5238)
• Handled count distinct query in selection list (#5223)
• Enabled async processing in pinot broker query api (#5229)
• Supported bootstrap mode for table rebalance (#5224)
• Supported order-by on BYTES column (#5213)
• Added Nightly publish to binary (#5190)
• Shuffled the segments when rebalancing the table to avoid creating hotspot servers (#5197)
• Supported inbuilt transform functions (#5312)
  • Added date time transform functions (#5326)
• Deepstore by-pass in LLC: introduced segment uploader (#5277, #5314)
• APIs Additions/Changes
  • Added a new server api for download of segments
    • /GET /segments/{tableNameWithType}/{segmentName}
• Upgraded helix to 0.9.7 (#5411)
• Added support to execute functions during query compilation (#5406)
• Other notable refactoring
  • Moved table config into pinot-spi (#5194)
  • Cleaned up integration tests. Standardized the creation of schema, table config and segments (#5385)
  • Added jsonExtractScalar function to extract field from json object (#4597)
  • Added template support for Pinot Ingestion Job Spec #5372
  • Cleaned up AggregationFunctionContext (#5364)
  • Optimized real-time range predicate when cardinality is high (#5331)
  • Made PinotOutputFormat use table config and schema to create segments (#5350)
  • Tracked unavailable segments in InstanceSelector (#5337)
  • Added a new best effort segment uploader with bounded upload time (#5314)
  • In SegmentPurger, used table config to generate the segment (#5325)
  • Decoupled schema from RecordReader and StreamMessageDecoder (#5309)
  • Implemented ARRAYLENGTH UDF for multi-valued columns (#5301)
  • Improved GroupBy query performance (#5291)
  • Optimized ExpressionFilterOperator (#5132)

Major Bug Fixes

• Do not release the PinotDataBuffer when closing the index (#5400)
• Handled a no-arg function in query parsing and expression tree (#5375)
• Fixed compatibility issues during rolling upgrade due to unknown json fields (#5376)
• Fixed missing error message from pinot-admin command (#5305)
• Fixed HDFS copy logic (#5218)
• Fixed spark ingestion issue (#5216)
• Fixed the capacity of the DistinctTable (#5204)
• Fixed various links in the Pinot website

Work in Progress

• Upsert: support overriding data in the real-time table (#4261).
  • Add pinot upsert features to pinot common (#5175)
• Enhancements for theta-sketch, e.g. multiValue aggregation support, complex predicates, performance tuning, etc

Backward Incompatible Changes

• TableConfig no longer support de-serialization from json string of nested json string (i.e. no \" inside the json) (#5194)
• The following APIs are changed in AggregationFunction (use TransformExpressionTree instead of String as the key of blockValSetMap) (#5371):

  void aggregate(int length, AggregationResultHolder aggregationResultHolder, Map<TransformExpressionTree, BlockValSet> blockValSetMap);
  void aggregateGroupBySV(int length, int[] groupKeyArray, GroupByResultHolder groupByResultHolder, Map<TransformExpressionTree, BlockValSet> blockValSetMap);
  void aggregateGroupByMV(int length, int[][] groupKeysArray, GroupByResultHolder groupByResultHolder, Map<TransformExpressionTree, BlockValSet> blockValSetMap);

• A different segment writing logic was introduced in #5256. Although this is backward compatible in a sense that the old segments can be read by the new code, rollback would be tricky since new segments after the upgrade would have been written in the new format, and the old code cannot read those new segments.

What's the big change?

The reason behind the architectural change from the previous release (0.2.0) and this release (0.3.0), is the possibility of extending Apache Pinot. The 0.2.0 release was not flexible enough to support new storage types nor new stream types. Basically, inserting a new functionality required to change too much code. Thus, the Pinot team went through an extensive refactoring and improvement of the source code.

For instance, the picture below shows the module dependencies of the 0.2.X or previous releases. If we wanted to support a new storage type, we would have had to change several modules. Pretty bad, huh?

In order to conquer this challenge, below major changes are made:

• Refactored common interfaces to pinot-spi module
• Concluded four types of modules:
  • Pinot input format: How to read records from various data/file formats: e.g. Avro/CSV/JSON/ORC/Parquet/Thrift
  • Pinot filesystem: How to operate files on various filesystems: e.g. Azure Data Lake/Google Cloud Storage/S3/HDFS
  • Pinot stream ingestion: How to ingest data stream from various upstream systems, e.g. Kafka/Kinesis/Eventhub
  • Pinot batch ingestion: How to run Pinot batch ingestion jobs in various frameworks, like Standalone, Hadoop, Spark.
• Built shaded jars for each individual plugin
• Added support to dynamically load pinot plugins at server startup time

Now the architecture supports a plug-and-play fashion, where new tools can be supported with little and simple extensions, without affecting big chunks of code. Integrations with new streaming services and data formats can be developed in a much more simple and convenient way.

Below is current supported Pinot Plugins module structure:

• pinot-input-format
  • pinot-avro
  • pinot-csv
  • pinot-json
  • pinot-orc
  • pinot-parquet
  • pinot-thrift
• pinot-file-system
  • pinot-adls
  • pinot-gcs
  • pinot-hdfs
• pinot-stream-ingestion
  • pinot-kafka-0.9
  • pinot-kafka-2.0
• pinot-batch-ingestion
  • pinot-batch-ingestion-hadoop
  • pinot-batch-ingestion-spark
  • pinot-batch-ingestion-standalone

Notable New Features

• Added support for DISTINCT (#4535)
• Added support default value for BYTES column (#4583)
• JDK 11 Support
• Added support to tune size vs accuracy for approximation aggregation functions: DistinctCountHLL, PercentileEst, PercentileTDigest (#4666)
• Added Data Anonymizer Tool (#4747)
• Deprecated pinot-hadoop and pinot-spark modules, replace with pinot-batch-ingestion-hadoop and pinot-batch-ingestion-spark
• Support STRING and BYTES for no dictionary columns in realtime consuming segments (#4791)
• Make pinot-distribution to build a pinot-all jar and assemble it (#4977)
• Added support for PQL case insensitive (#4983)
• Enhanced TableRebalancer logics
  • Moved to new rebalance strategy (#4695)
  • Supported rebalancing tables under any condition(#4990)
  • Supported reassigning completed segments along with Consuming segments for LLC realtime table (#5015)
• Added experimental support for Text Search‌ (#4993)
• Upgraded Helix to version 0.9.4, task management now works as expected (#5020)
• Added date_trunc transformation function. (#4740)
• Support schema evolution for consuming segment. (#4954)
• SQL Support
  • Added Calcite SQL compiler
  • Added SQL response format (#4694, #4877)
  • Added support for GROUP BY with ORDER BY (#4602)
  • Query console defaults to use SQL syntax (#4994)
  • Support column alias (#5016, #5033)
  • Added SQL query endpoint: /query/sql (#4964)
  • Support arithmetic operators (#5018)
  • Support non-literal expressions for right-side operand in predicate comparison(#5070)
• APIs Additions/Changes
  • Pinot Admin Command
    • Added -queryType option in PinotAdmin PostQuery subcommand (#4726)
    • Added -schemaFile as option in AddTable command (#4959)
    • Added OperateClusterConfig sub command in PinotAdmin (#5073)
  • Pinot Controller Rest APIs
    • Get Table leader controller resource (#4545)
    • Support HTTP POST/PUT to upload JSON encoded schema (#4639)
    • Table rebalance API now requires both table name and type as parameters. (#4824)
    • Refactored Segments APIs (#4806)
    • Added segment batch deletion REST API (#4828)
    • Update schema API to reload table on schema change when applicable (#4838)
    • Enhance the task related REST APIs (#5054)
    • Added PinotClusterConfig REST APIs (#5073)
      • GET /cluster/configs
      • POST /cluster/configs
      • DELETE /cluster/configs/{configName}
• Configurations Additions/Changes
  • Config: controller.host is now optional in Pinot Controller
  • Added instance config: queriesDisabled to disable query sending to a running server (#4767)
  • Added broker config: pinot.broker.enable.query.limit.override configurable max query response size (#5040)
  • Removed deprecated server configs (#4903)
    • pinot.server.starter.enableSegmentsLoadingCheck
    • pinot.server.starter.timeoutInSeconds
    • pinot.server.instance.enable.shutdown.delay
    • pinot.server.instance.starter.maxShutdownWaitTime
    • pinot.server.instance.starter.checkIntervalTime
  • Decouple server instance id with hostname/port config. (#4995)
  • Add FieldConfig to encapsulate encoding, indexing info for a field.(#5006)

Major Bug Fixes

• Fixed the bug of releasing the segment when there are still threads working on it. (#4764)
• Fixed the bug of uneven task distribution for threads (#4793)
• Fixed encryption for .tar.gz segment file upload (#4855)
• Fixed controller rest API to download segment from non local FS. (#4808)
• Fixed the bug of not releasing segment lock if segment recovery throws exception (#4882)
• Fixed the issue of server not registering state model factory before connecting the Helix manager (#4929)
• Fixed the exception in server instance when Helix starts a new ZK session (#4976)
• Fixed ThreadLocal DocIdSet issue in ExpressionFilterOperator (#5114)
• Fixed the bug in default value provider classes (#5137)
• Fixed the bug when no segment exists in RealtimeSegmentSelector (#5138)

Work in Progress

• We are in the process of supporting text search query functionalities.
• We are in the process of supporting null value (#4230), currently limited query feature is supported
  • Added Presence Vector to represent null value (#4585)
  • Added null predicate support for leaf predicates (#4943)

Backward Incompatible Changes

• It’s a disruptive upgrade from version 0.1.0 to this because of the protocol changes between Pinot Broker and Pinot Server. Please ensure that you upgrade to release 0.2.0 first, then upgrade to this version.
• If you build your own startable or war without using scripts generated in Pinot-distribution module. For Java 8, an environment variable “plugins.dir” is required for Pinot to find out where to load all the Pinot plugin jars. For Java 11, plugins directory is required to be explicitly set into classpath. Please see pinot-admin.sh as an example.
• As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.
• Kafka 0.9 is no longer included in the release distribution.
• Pull request #4806 introduces a backward incompatible API change for segments management.
  • Removed segment toggle APIs
  • Removed list all segments in cluster APIs
  • Deprecated below APIs:
    • GET /tables/{tableName}/segments
    • GET /tables/{tableName}/segments/metadata
    • GET /tables/{tableName}/segments/crc
    • GET /tables/{tableName}/segments/{segmentName}
    • GET /tables/{tableName}/segments/{segmentName}/metadata
    • GET /tables/{tableName}/segments/{segmentName}/reload
    • POST /tables/{tableName}/segments/{segmentName}/reload
    • GET /tables/{tableName}/segments/reload
    • POST /tables/{tableName}/segments/reload
• Pull request #5054 deprecated below task related APIs:
  • GET:
    • /tasks/taskqueues: List all task queues
    • /tasks/taskqueuestate/{taskType} -> /tasks/{taskType}/state
    • /tasks/tasks/{taskType} -> /tasks/{taskType}/tasks
    • /tasks/taskstates/{taskType} -> /tasks/{taskType}/taskstates
    • /tasks/taskstate/{taskName} -> /tasks/task/{taskName}/taskstate
    • /tasks/taskconfig/{taskName} -> /tasks/task/{taskName}/taskconfig
  • PUT:
    • /tasks/scheduletasks -> POST /tasks/schedule
    • /tasks/cleanuptasks/{taskType} -> /tasks/{taskType}/cleanup
    • /tasks/taskqueue/{taskType}: Toggle a task queue
  • DELETE:
    • /tasks/taskqueue/{taskType} -> /tasks/{taskType}
• Deprecated modules pinot-hadoop and pinot-spark and replaced with pinot-batch-ingestion-hadoop and pinot-batch-ingestion-spark.
• Introduced new Pinot batch ingestion jobs and yaml based job specs to define segment generation jobs and segment push jobs.
• You may see exceptions like below in pinot-brokers during cluster upgrade, but it's safe to ignore them.

2020/03/09 23:37:19.879 ERROR [HelixTaskExecutor] [CallbackProcessor@b808af5-pinot] [pinot-broker] [] Message cannot be processed: 78816abe-5288-4f08-88c0-f8aa596114fe, {CREATE_TIMESTAMP=1583797034542, MSG_ID=78816abe-5288-4f08-88c0-f8aa596114fe, MSG_STATE=unprocessable, MSG_SUBTYPE=REFRESH_SEGMENT, MSG_TYPE=USER_DEFINE_MSG, PARTITION_NAME=fooBar_OFFLINE, RESOURCE_NAME=brokerResource, RETRY_COUNT=0, SRC_CLUSTER=pinot, SRC_INSTANCE_TYPE=PARTICIPANT, SRC_NAME=Controller_hostname.domain,com_9000, TGT_NAME=Broker_hostname,domain.com_6998, TGT_SESSION_ID=f6e19a457b80db5, TIMEOUT=-1, segmentName=fooBar_559, tableName=fooBar_OFFLINE}{}{}
java.lang.UnsupportedOperationException: Unsupported user defined message sub type: REFRESH_SEGMENT
        at org.apache.pinot.broker.broker.helix.TimeboundaryRefreshMessageHandlerFactory.createHandler(TimeboundaryRefreshMessageHandlerFactory.java:68) ~[pinot-broker-0.2.1172.jar:0.3.0-SNAPSHOT-c9d88e47e02d799dc334d7dd1446a38d9ce161a3]
        at org.apache.helix.messaging.handling.HelixTaskExecutor.createMessageHandler(HelixTaskExecutor.java:1096) ~[helix-core-0.9.1.509.jar:0.9.1.509]
        at org.apache.helix.messaging.handling.HelixTaskExecutor.onMessage(HelixTaskExecutor.java:866) [helix-core-0.9.1.509.jar:0.9.1.509]

Major changes/feature additions since 0.1.0:

• Added support for Kafka 2.0.

• Table rebalancer now supports a minimum number of serving replicas during rebalance.

• Added support for UDF in filter predicates and selection.

• Added support to use hex string as the representation of byte array for queries (#4041)

• Added support for parquet reader (#3852)

• Introduced interface stability and audience annotations (#4063)

• Refactor HelixBrokerStarter to separate constructor and start() (#4100) - backwards incompatible

• Admin tool for listing segments with invalid intervals for offline tables

• Migrated to log4j2 (#4139)

• Added simple avro msg decoder

• Added support for passing headers in pinot client

• Table rebalancer now supports a minimum number of serving replicas during rebalance.

• Support transform functions with AVG aggregation function (#4557)

• Configurations additions/changes

  • Allow customized metrics prefix (#4392)
  • Controller.enable.batch.message.mode to false by default (#3928)
  • RetentionManager and OfflineSegmentIntervalChecker initial delays configurable (#3946)
  • Config to control kafka fetcher size and increase default (#3869)
  • Added a percent threshold to consider startup of services (#4011)
  • Make SingleConnectionBrokerRequestHandler as default (#4048)
  • Always enable default column feature, remove the configuration (#4074)
  • Remove redundant default broker configurations (#4106)
  • Removed some config keys in server (#4222)
  • Add config to disable HLC realtime segment (#4235)
  • Make RetentionManager and OfflineSegmentIntervalChecker initial delays configurable (#3946)
  • The following config variables are deprecated and will be removed in the next release:
    • pinot.broker.requestHandlerType will be removed, in favor of using the "singleConnection"
      broker request handler. If you have set this configuration, please remove it and use the default
      type ("singleConnection") for broker request handler.

Work in progress

• We are in the process of separating Helix and Pinot controllers, so that
  admninistrators can have the option of running independent Helix
  controllers and Pinot controllers.

• We are in the process of moving towards supporting SQL query format and results.

• We are in the process of seperating instance and segment assignment using instance
  pools to optimize the number of Helix state transitions in Pinot clusters with
  thousands of tables.

Other important notes

• Task management does not work correctly in this relese, due to bugs in Helix.
  We will upgrade to Helix 0.9.2 (or later) version to get this fixed.

• You must upgrade to this release before moving onto newer versions of Pinot
  release. The protocol between pinot-broker and pinot-server has been changed
  and this release has the code to retain compatibility moving forward.
  Skipping this release may (depending on your environment) cause query errors
  if brokers are upgraded and servers are in the process of being upgraded.

• As always, we recommend that you upgrade controllers first, and then brokers
  and lastly the servers in order to have zero downtime in production clusters.

• PR #4100 introduces a backwards incompatible change to pinot broker. If you
  use the Java constructor on HelixBrokerStarter class, then you will face a
  compilation error with this version. You will need to construct the object
  and call start() method in order to start the broker.

• PR #4139 introduces a backwards incompatible change for log4j configuration.
  If you used a custom log4j configuration (log4j.xml), you need to write a new
  log4j2 configuration (log4j2.xml). In addition, you may need to change the
  arguments on the command line to start pinot components.

  If you used pinot-admin command to start pinot components, you don't need any
  change. If you used your own commands to start pinot components, you will
  need to pass the new log4j2 config as a jvm parameter (i.e. substitute
  -Dlog4j.configuration or -Dlog4j.configurationFile argument with
  -Dlog4j2.configurationFile=log4j2.xml).

Pinot 0.1.0

This is the first official release of Apache Pinot.

Requirements

• Java 8 (It has been reported that Pinot cannot be compiled with Java 11 due to a missing package for sun.misc , #3625)

Highlights

Pluggable Storage

We have added a Pinot filesystem abstraction that provides users the option to plug-in their own storage backend. Currently, we support HDFS, NFS, and Azure Data Lake.

• Documentation (https://pinot.readthedocs.io/en/latest/pluggable_storage.html)

Pluggable Realtime Streams

We have decoupled Pinot from Kafka for realtime ingestion and added the abstraction for Streams. Users can now add their own plugins to read from any pub-sub systems.

• Documentation (https://pinot.readthedocs.io/en/latest/pluggable_streams.html)

Native Byte Array and TDigest Support

Pinot now can support byte[] type natively. Also, Pinot can accept byte serialized TDigest object (com.tdunning.math.stats.TDigest) and this can be queried to compute approximate percentiles as follows.

select percentileTDigest95(tDigestColumn) from myTable where ... group by... top N

Compatibility

Since this is the first official Apache release, these notes applies to people who have used Pinot in production by building from the source code.

Backward Incompatible Changes

• All the packages have been renamed from com.linkedin to org.apache.
• Method signatures has been changed in interface ControllerRestApi and SegmentNameGenerator.

Rollback Restrictions

• If you have used PartitionAwareRouting feature, we have changed the format of partition values from the segment metadata and zk segment metadata. (e.g. partitionRanges: [0 0], [1 1] -> partitions: 0, 1) The new code is backward compatible with the old format; however, old code will not be able to understand the new format in case of rollback.

Deprecation

• N/A

Credits

Thanks for everyone who have contributed to Pinot.

We would also like to express our gratitude to our Apache mentors @kishoreg, @felixcheung, @jimjag, @olamy and @rvs.