Fixed

• #4795 Query: Fix deadlock in endpointset.

Added

• #4228 Tools thanos bucket inspect: Add flag --output to provide output method (table,csv,tsv).
• #4282 Query: breaking ⚠️ Add --endpoint flag to the querier. The --store flag will eventually be replaced.
• #4680 Query: Add exemplar.partial-response flag to control partial response.
• #4679 Query: Add enable-feature flag to enable negative offsets and @ modifier, similar to Prometheus.
• #4696 Query: Add cache name to tracing spans.
• #4710 Store: Add metric to capture timestamp of the last loaded block.
• #4736 S3: Add capability to use custom AWS STS Endpoint.
• #4764 Compact: Add block-viewer.global.sync-block-timeout flag to set the timeout of synchronization block metas.
• #4801 Compact: Add Prometheus metrics for tracking the progress of compaction and downsampling.
• #4444 UI: Add mark deletion and no compaction to the Block UI.
• #4576 UI: Add filter compaction level to the Block UI.
• #4731 Rule: Add stateless mode to ruler.
• #4612 Sidecar: Add --prometheus.http-client and --prometheus.http-client-file flag for sidecar to connect to Prometheus with basic auth or TLS.
• #4847 Query: Add --alert.query-url which is used in the UI for rules/alerts pages. By default the HTTP listen address is used for this URL.
• #4856 Mixin: Add Query Frontend to Grafana dashboard.
• #4874 Query: Add --endpoint-strict flag to statically configure Thanos API server endpoints. It is similar to --store-strict but supports passing any Thanos gRPC APIs: StoreAPI, MetadataAPI, RulesAPI, TargetsAPI and ExemplarsAPI.
• #4868 Rule: Support ruleGroup limit introduced by Prometheus v2.31.0.
• #4897 Query: Add validation for querier address flags.

Fixed

• #4508 Sidecar, Mixin: Rename ThanosSidecarUnhealthy to ThanosSidecarNoConnectionToStartedPrometheus; Remove ThanosSidecarPrometheusDown alert; Remove unused thanos_sidecar_last_heartbeat_success_time_seconds metrics.
• #4663 Fetcher: Fix discovered data races.
• #4754 Query: Fix possible panic on stores endpoint.
• #4753 Store: validate block sync concurrency parameter.
• #4779 Examples: Fix the interactive test for MacOS users.
• #4792 Store: Fix data race in BucketedBytes pool.
• #4769 Query Frontend: Add "X-Request-ID" field and other fields to start call log.
• #4709 Store: Fix panic when the application is stopped.
• #4777 Query: Fix data race in exemplars server.
• #4811 Query: Fix data race in metadata, rules, and targets servers.
• #4795 Query: Fix deadlock in endpointset.

Changed

• #4864 UI: Remove the old PromQL editor.
• #4708 Receive: Remove gRPC message size limit, which fixes errors commonly seen when receivers forward messages within a hashring.

Fixed

• #4714 EndpointSet: Do not use unimplemented yet new InfoAPI to obtain metadata (avoids unnecessary HTTP roundtrip, instrumentation/alerts spam and logs)

Thanks to all 44 contributors, who had committed good work to this release! 🤗

@GiedriusS @bwplotka @Nexucis @Namanl2001 @hanjm @clyang82 @Hangzhi @pracucci @metalmatze @iamrajiv @roystchiang @saswatamcode @adzshaf @Abhishek357 @Bacto @AnaisUrlichs @2pk03 @vanugrah @arajkumar @galan @ethernoy @hitanshu-mehta @ianbillett @ilangofman @jesusvazquez @jmichalek132 @OGKevin @matej-g @56quarters @nishidhaSri @PhilipGough @phillebaba @JRavi2 @crsandeep @simonpasquier @thanatchakromsang @weyert @wiardvanrij @huyan0 @yangyang919 @zhyChesterCheung @rhassanein @songjiayang @yahaa

Notable changes:

• Query-frontend overhead reduced 20% (change your max_idle_conns_per_host to at least 100!)
• Massive enhancement over Block UI: Search capabilities, overlap finder and more!
• Added Baidu BOS object storage support
• Improved tracing metadata. It now shows exact series, samples and overall bytes processes per component in the path of StoreAPI.Series query.
• Easier use of non Store APIs: auto-discovery through endpoint flag. Old components will work as they were, no action is required unless you want to explicitly disable certain APIs (e.g rules, exemplars, metadata etc) if you don't want them.
• Added few Prometheus PromQL functions (upgraded Prometheus deps)

Added

• #4453 Tools thanos bucket web: Add flag --selector.relabel-config-file / --selector.relabel-config / --max-time / --min-time to filter served blocks.
• #4482 Store: Add http_config option for COS object store client.
• #4487 Query/Store: Add memcached auto discovery support for all caching clients.
• #4444 UI: Add search to the Block UI.
• #4509 Logging: Add duration_ms in int64 to the logs for easier log filtering.
• #4462 UI: Highlighting blocks overlap in the Block UI.
• #4469 Compact: Add flag compact.skip-block-with-out-of-order-chunks to skip blocks with out-of-order chunks during compaction instead of halting.
• #4506 Store: Add Baidu BOS object storage, see documents for further information.
• #4552 Compact: Add thanos_compact_downsample_duration_seconds histogram metric.
• #4594 Reloader: Expose metrics in config reloader to give info on the last operation.
• #4619 Tracing: Added consistent tags to Series call from Querier about number important series statistics: processed.series, processed.samples, processed.samples and processed.bytes. This will give admin idea of how much data each component processes per query.
• #4623 Query-frontend: Make HTTP downstream tripper (client) configurable via parameters --query-range.downstream-tripper-config and --query-range.downstream-tripper-config-file. If your downstream URL is localhost or 127.0.0.1 then it is strongly recommended to bump max_idle_conns_per_host to at least 100 so that query-frontend could properly use HTTP keep-alive connections and thus reduce the latency of query-frontend by about 20%.

Fixed

• #4468 Rule: Fix temporary rule filename composition issue.
• #4476 UI: Fix incorrect html escape sequence used for '>' symbol.
• #4532 Mixin: Fix "all jobs" selector in thanos mixin dashboards.
• #4607 Azure: Fix Azure MSI Rate Limit.

Changed

• #4519 Query: Switch to miekgdns DNS resolver as the default one.
• #4586 Update Prometheus/Cortex dependencies and implement LabelNames() pushdown as a result; provides massive speed-up for the labels API in Thanos Query.
• #4421 breaking ⚠️: --store (in the future, to be renamed to --endpoints) now supports passing any APIs from Thanos gRPC APIs: StoreAPI, MetadataAPI, RulesAPI, TargetsAPI and ExemplarsAPI (in oppose in the past you have to put it in hidden --targets, --rules etc flags). --store will now automatically detect what APIs server exposes.
• #4669 Moved Prometheus dependency to v2.30.

Fixes couple of critical bugs vs rc.0 (:

Added

• #4453 Tools thanos bucket web: Add flag --selector.relabel-config-file / --selector.relabel-config / --max-time / --min-time to filter served blocks.
• #4482 Store: Add http_config option for COS object store client.
• #4487 Query/Store: Add memcached auto discovery support for all caching clients.
• #4444 UI: Add search to the Block UI.
• #4509 Logging: Add duration_ms in int64 to the logs for easier log filtering.
• #4462 UI: Highlighting blocks overlap in the Block UI.
• #4469 Compact: Add flag compact.skip-block-with-out-of-order-chunks to skip blocks with out-of-order chunks during compaction instead of halting.
• #4506 Store: Add Baidu BOS object storage, see documents for further information.
• #4552 Compact: Add thanos_compact_downsample_duration_seconds histogram metric.
• #4594 Reloader: Expose metrics in config reloader to give info on the last operation.
• #4619 Tracing: Added consistent tags to Series call from Querier about number important series statistics: processed.series, processed.samples, processed.samples and processed.bytes. This will give admin idea of how much data each component processes per query.
• #4623 Query-frontend: Make HTTP downstream tripper (client) configurable via parameters --query-range.downstream-tripper-config and --query-range.downstream-tripper-config-file. If your downstream URL is localhost or 127.0.0.1 then it is strongly recommended to bump max_idle_conns_per_host to at least 100 so that query-frontend could properly use HTTP keep-alive connections and thus reduce the latency of query-frontend by about 20%.

Fixed

• #4468 Rule: Fix temporary rule filename composition issue.
• #4476 UI: Fix incorrect html escape sequence used for '>' symbol.
• #4532 Mixin: Fix "all jobs" selector in thanos mixin dashboards.
• #4607 Azure: Fix Azure MSI Rate Limit.

Changed

• #4519 Query: Switch to miekgdns DNS resolver as the default one.
• #4586 Update Prometheus/Cortex dependencies and implement LabelNames() pushdown as a result; provides massive speed-up for the labels API in Thanos Query.
• #4421 breaking ⚠️: --store (in the future, to be renamed to --endpoints) now supports passing any APIs from Thanos gRPC APIs: StoreAPI, MetadataAPI, RulesAPI, TargetsAPI and ExemplarsAPI (in oppose in the past you have to put it in hidden --targets, --rules etc flags). --store will now automatically detect what APIs server exposes.
• #4669 Moved Prometheus dependency to v2.30.

Added

• #4453 Tools thanos bucket web: Add flag --selector.relabel-config-file / --selector.relabel-config / --max-time / --min-time to filter served blocks.
• #4482 Store: Add http_config option for COS object store client.
• #4487 Query/Store: Add memcached auto discovery support for all caching clients.
• #4444 UI: Add search to the Block UI.
• #4509 Logging: Add duration_ms in int64 to the logs for easier log filtering.
• #4462 UI: Highlighting blocks overlap in the Block UI.
• #4469 Compact: Add flag compact.skip-block-with-out-of-order-chunks to skip blocks with out-of-order chunks during compaction instead of halting.
• #4506 Store: Add Baidu BOS object storage, see documents for further information.
• #4552 Compact: Add thanos_compact_downsample_duration_seconds histogram metric.
• #4594 Reloader: Expose metrics in config reloader to give info on the last operation.
• #4619 Tracing: Added consistent tags to Series call from Querier about number important series statistics: processed.series, processed.samples, processed.samples and processed.bytes. This will give admin idea of how much data each component processes per query.
• #4623 Query-frontend: Make HTTP downstream tripper (client) configurable via parameters --query-range.downstream-tripper-config and --query-range.downstream-tripper-config-file. If your downstream URL is localhost or 127.0.0.1 then it is strongly recommended to bump max_idle_conns_per_host to at least 100 so that query-frontend could properly use HTTP keep-alive connections and thus reduce the latency of query-frontend by about 20%.

Fixed

• #4468 Rule: Fix temporary rule filename composition issue.
• #4476 UI: Fix incorrect html escape sequence used for '>' symbol.
• #4532 Mixin: Fix "all jobs" selector in thanos mixin dashboards.
• #4607 Azure: Fix Azure MSI Rate Limit.

Changed

• #4519 Query: Switch to miekgdns DNS resolver as the default one.
• #4586 Update Prometheus/Cortex dependencies and implement LabelNames() pushdown as a result; provides massive speed-up for the labels API in Thanos Query.
• #4421: breaking ⚠️: --store (in the future, to be renamed to --endpoints) now supports passing any APIs from Thanos gRPC APIs: StoreAPI, MetadataAPI, RulesAPI, TargetsAPI and ExemplarsAPI (in oppose in the past you have to put it in hidden --targets, --rules etc flags). --store will now automatically detect what APIs server exposes.

Highlights

• Added offline deduplication that helps saving even more space on the remote object storage & increases query performance
• The receive dual-mode has been implemented
• Receive now is able to ingest exemplars and it is possible to query them
• Receive now uses a few percent less of CPU time with the same load
• Azure object storage has been refactored to support MSI authentication & it's now much more configurable

This release contains improvements from 28 authors (git log --pretty="%ae" origin/release-0.21...origin/release-0.22 | sort | uniq | wc -l)! Thanks to all of them! Thanos wouldn't be what it is without all of their effort and time! ❤️

Added

• #4394 Add error logs to receiver when write request rejected with invalid replica
• #4403 UI: Add sorting and filtering to flags page
• #4299 Tracing: Add tracing to exemplar APIs.
• #4327 Add environment variable substitution to all YAML configuration flags.
• #4239 Add penalty based deduplication mode for compactor.
• #4292 Receive: Enable exemplars ingestion and querying.
• #4392 Tools: Added --delete-blocks to bucket rewrite tool to mark the original blocks for deletion after rewriting is done.
• #3970 Azure: Adds more configuration options for Azure blob storage. This allows for pipeline and reader specific configuration. Implements HTTP transport configuration options. These options allows for more fine-grained control on timeouts and retries. Implements MSI authentication as second method of authentication via a service principal token.
• #4406 Tools: Add retention command for applying retention policy on the bucket.
• #4430 Compact: Add flag downsample.concurrency to specify the concurrency of downsampling blocks.
• #4231 Receive: Implemented the receive dual-mode. This means that it is now you can run receive as ingester (TSDB only, no hashring awareness), as router (no TSDB, only forwarding mode) and old mode (router + ingester) in one service. Consider splitting functionality if you want to scale your hashring into multiple rings or you want horizontally autoscale your ingester more often.
• #3678 UI: add support for new duration format in graph range input. Now it is possible to use composite durations such as 5m30s.

Fixed

• #4384 Fix the experimental PromQL editor when used on multiple line.
• #4342 ThanosSidecarUnhealthy doesn't fire if the sidecar is never healthy
• #4388 Receive: fix bug in forwarding remote-write requests within the hashring via gRPC when TLS is enabled on the HTTP server but not on the gRPC server.
• #4340 UI: now displays the duration and all annotations of an alert in the alerts page.
• #4348 Fixed parsing of the port in the log middleware.
• #4417 UI: fixed the night mode in Bucket UI.
• #4442 Ruler: fix SIGHUP reload signal not working.

Changed

• #4354 Receive: use the S2 library for decoding Snappy data; saves about 5-7% of CPU time in the Receive component when handling incoming remote write requests
• #4369 Build: Pin upgrade Alpine's version.
• #4404 Receive: added extra validation for the tenant's label name. Some unsupported formats could have passed before.

Highlights

• Added offline deduplication that helps saving even more space on the remote object storage & increases query performance
• The receive dual-mode has been implemented
• Receive now is able to ingest exemplars and it is possible to query them
• Receive now uses a few percent less of CPU time with the same load

This release contains improvements from 27 authors (git log --pretty="%ae" origin/release-0.21...origin/release-0.22 | sort | uniq | wc -l)! Thanks to all of them! Thanos wouldn't be what it is without all of their effort and time! ❤️

Added

• #4394 Add error logs to receiver when write request rejected with invalid replica
• #4403 UI: Add sorting and filtering to flags page
• #4299 Tracing: Add tracing to exemplar APIs.
• #4327 Add environment variable substitution to all YAML configuration flags.
• #4239 Add penalty based deduplication mode for compactor.
• #4292 Receive: Enable exemplars ingestion and querying.
• #4392 Tools: Added --delete-blocks to bucket rewrite tool to mark the original blocks for deletion after rewriting is done.
• #3970 Azure: Adds more configuration options for Azure blob storage. This allows for pipeline and reader specific configuration. Implements HTTP transport configuration options. These options allows for more fine-grained control on timeouts and retries. Implements MSI authentication as second method of authentication via a service principal token.
• #4406 Tools: Add retention command for applying retention policy on the bucket.
• #4430 Compact: Add flag downsample.concurrency to specify the concurrency of downsampling blocks.
• #4231 Receive: Implemented the receive dual-mode. This means that it is now you can run receive as ingester (TSDB only, no hashring awareness), as router (no TSDB, only forwarding mode) and old mode (router + ingester) in one service. Consider splitting functionality if you want to scale your hashring into multiple rings or you want horizontally autoscale your ingester more often.
• #3678 UI: add support for new duration format in graph range input. Now it is possible to use composite durations such as 5m30s.

Fixed

• #4384 Fix the experimental PromQL editor when used on multiple line.
• #4342 ThanosSidecarUnhealthy doesn't fire if the sidecar is never healthy
• #4388 Receive: fix bug in forwarding remote-write requests within the hashring via gRPC when TLS is enabled on the HTTP server but not on the gRPC server.
• #4340 UI: now displays the duration and all annotations of an alert in the alerts page.
• #4348 Fixed parsing of the port in the log middleware.
• #4417 UI: fixed the night mode in Bucket UI.

Changed

• #4354 Receive: use the S2 library for decoding Snappy data; saves about 5-7% of CPU time in the Receive component when handling incoming remote write requests
• #4369 Build: Pin upgrade Alpine's version.
• #4404 Receive: added extra validation for the tenant's label name. Some unsupported formats could have passed before.

Fixed

• #4308 Sidecar: reloader: fix output config file permission

Highlights

• Add TLS and basic authentication to Thanos APIs.
• The UI has a dark theme support now.
• Label Matchers for various APIs
• Extra statistics for query API

Added

• #4117 Mixin: new alert ThanosReceiveTrafficBelowThreshold to flag if the ingestion average of the last hour dips below 50% of the ingestion average for the last 12 hours.
• #4107 Store: LabelNames and LabelValues now support label matchers.
• #3940 Sidecar: Added matchers support to LabelValues
• #4171 Docker: Busybox image updated to latest (1.33.1)
• #4175 Added Tag Configuration Support Lightstep Tracing
• #4176 Query API: Adds optional Stats param to return stats for query APIs
• #4125 Rule: Add --alert.relabel-config / --alert.relabel-config-file allowing to specify alert relabel configurations like Prometheus
• #4211 Add TLS and basic authentication to Thanos APIs
• #4249 UI: add dark theme
• #3707 Tools: Added --rewrite.to-relabel-config to bucket rewrite tool to support series relabel from given blocks.

Fixed

• #4105 Tools: Add glob support for filepath in tools command

Changed

• #4223 Query: federated exemplars API only add replica labels to series labels, not to exemplar labels.

Highlights

• Add TLS and basic authentication to Thanos APIs.
• The UI has a dark theme support now.

Added

• #4117 Mixin: new alert ThanosReceiveTrafficBelowThreshold to flag if the ingestion average of the last hour dips below 50% of the ingestion average for the last 12 hours.
• #4107 Store: LabelNames and LabelValues now support label matchers.
• #3940 Sidecar: Added matchers support to LabelValues
• #4171 Docker: Busybox image updated to latest (1.33.1)
• #4175 Added Tag Configuration Support Lightstep Tracing
• #4176 Query API: Adds optional Stats param to return stats for query APIs
• #4125 Rule: Add --alert.relabel-config / --alert.relabel-config-file allowing to specify alert relabel configurations like Prometheus
• #4211 Add TLS and basic authentication to Thanos APIs
• #4249 UI: add dark theme
• #3707 Tools: Added --rewrite.to-relabel-config to bucket rewrite tool to support series relabel from given blocks.

Fixed

• #4105 Tools: Add glob support for filepath in tools command

Changed

• #4223 Query: federated exemplars API only add replica labels to series labels, not to exemplar labels.

Fixed

• #4208 UI: Fix infinite redirection loop on root (/).

Fixed

• #4123 Query: match external labels for exemplars API.

Highlights 🚀 🎉

• Upgrade Go version to 1.16
• Upgrade Prometheus & Alertmanager.
• Query: Added federated exemplars API support.
• UI: Make the ReactUI the default one.
• UI: Enable Targets page in Querier UI.
• UI: Add new codemirror-promql-based expression editor.
• Query Frontend: Support max_item_size in Query frontend Memcached cache.
• Tools: Added a flag to support rewrite Prometheus TSDB blocks.

Added

• #4029 Mixin: Remove dependency on the rule dashboard when generating the compact dashboard
• #4019 Query: Adds query range histogram.
• #3846 Query: Added federated exemplars API support.
• #3350 Query/Sidecar: Added targets API support. You can now configure your Querier to fetch Prometheus targets from leaf Prometheus-es!
• #3977 Expose exemplars for http_request_duration_seconds histogram if tracing is enabled.
• #3903 Store: Returning custom gRPC code when reaching series/chunk limits.
• #3919 Allow to disable automatically setting CORS headers using --web.disable-cors flag in each component that exposes an API.
• #3840 Tools: Added a flag to support rewrite Prometheus TSDB blocks.
• #3920 Query Frontend: Support max_item_size in Query frontend Memcached cache.

Fixed

• #3204 Mixin: Use sidecar's metric timestamp for healthcheck.
• #3922 *: Fix panic in http logging middleware.
• #3960 Ruler: Fix deduplication of equal alerts with different labels.
• #3937 Store: Fix race condition in chunk pool.
• #4017 Query Frontend: fix downsampling iterator returning duplicate samples.
• #4041 Logging: fix the HTTP logger.

Changed

• #3929 Store: Adds the name of the instantiated memcached client to log info.
• #3827 Upgrade Go version to 1.16
• #3948 Receiver: Adjust http_request_duration_seconds buckets for low latency requests.
• #3856 Mixin: breaking ⚠️ Introduce flexible multi-cluster/namespace mode for alerts and dashboards. Removes jobPrefix config option. Removes namespace by default.
• #3937 Store: Reduce memory usage for range queries.
• #4045 UI: Enable Targets page in Querier UI.
• #4062 Flags: Sort flags alphabetically.
• #4081 UI: Make the ReactUI the default one.
• #4094 *: Upgrade Prometheus & Alertmanager.

Highlights

• Upgrade Go version to 1.16
• Upgrade Prometheus & Alertmanager.
• Query: Added federated exemplars API support.
• UI: Make the ReactUI the default one.
• UI: Enable Targets page in Querier UI.
• UI: Add new codemirror-promql-based expression editor.
• Query Frontend: Support max_item_size in Query frontend Memcached cache.
• Tools: Added a flag to support rewrite Prometheus TSDB blocks.

Added

• #4029 Mixin: Remove dependency on the rule dashboard when generating the compact dashboard
• #4019 Query: Adds query range histogram.
• #3846 Query: Added federated exemplars API support.
• #3350 Query/Sidecar: Added targets API support. You can now configure your Querier to fetch Prometheus targets from leaf Prometheus-es!
• #3977 Expose exemplars for http_request_duration_seconds histogram if tracing is enabled.
• #3903 Store: Returning custom gRPC code when reaching series/chunk limits.
• #3919 Allow to disable automatically setting CORS headers using --web.disable-cors flag in each component that exposes an API.
• #3840 Tools: Added a flag to support rewrite Prometheus TSDB blocks.
• #3920 Query Frontend: Support max_item_size in Query frontend Memcached cache.

Fixed

• #3204 Mixin: Use sidecar's metric timestamp for healthcheck.
• #3922 *: Fix panic in http logging middleware.
• #3960 Ruler: Fix deduplication of equal alerts with different labels.
• #3937 Store: Fix race condition in chunk pool.
• #4017 Query Frontend: fix downsampling iterator returning duplicate samples.
• #4041 Logging: fix the HTTP logger.

Changed

• #3929 Store: Adds the name of the instantiated memcached client to log info.
• #3827 Upgrade Go version to 1.16
• #3948 Receiver: Adjust http_request_duration_seconds buckets for low latency requests.
• #3856 Mixin: breaking ⚠️ Introduce flexible multi-cluster/namespace mode for alerts and dashboards. Removes jobPrefix config option. Removes namespace by default.
• #3937 Store: Reduce memory usage for range queries.
• #4045 UI: Enable Targets page in Querier UI.
• #4062 Flags: Sort flags alphabetically.
• #4081 UI: Make the ReactUI the default one.
• #4094 *: Upgrade Prometheus & Alertmanager.

Added

• #3700 Compact/Web: Make old bucket viewer UI work with vanilla Prometheus blocks.
• #3657 *: It's now possible to configure HTTP transport options for S3 client.
• #3752 Compact/Store: Added --block-meta-fetch-concurrency allowing to configure number of go routines for block metadata synchronization.
• #3723 Query Frontend: Added --query-range.request-downsampled flag enabling additional queries for downsampled data in case of empty or incomplete response to range request.
• #3579 Cache: Added inmemory cache for caching bucket.
• #3792 Receiver: Added --tsdb.allow-overlapping-blocks flag to allow overlapping tsdb blocks and enable vertical compaction.
• #3740 Query: Added --query.default-step flag to set default step. Useful when your tenant scrape interval is stable and far from default UI's 1s.
• #3686 Query/Sidecar: Added metric metadata API support. You can now configure you Querier to fetch Prometheus metrics metadata from leaf Prometheus-es!
• #3031 Compact/Sidecar/Receive/Rule: Added --hash-func. If some function has been specified, writers calculate hashes using that function of each file in a block before uploading them. If those hashes exist in the meta.json file then Compact does not download the files if they already exist on disk and with the same hash. This also means that the data directory passed to Thanos Compact is only cleared once at boot or if everything succeeds. So, if you, for example, use persistent volumes on k8s and your Thanos Compact crashes or fails to make an iteration properly then the last downloaded files are not wiped from the disk. The directories that were created the last time are only wiped again after a successful iteration or if the previously picked up blocks have disappeared.

Fixed

• #3705 Store: Fix race condition leading to failing queries or possibly incorrect query results.
• #3661 Compact: Deletion-mark.json is deleted as the last one, which could in theory lead to potential store gateway load or query error for such in-deletion block.
• #3760 Store: Fix panic caused by a race condition happening on concurrent index-header reader usage and unload, when --store.enable-index-header-lazy-reader is enabled.
• #3759 Store: Fix panic caused by a race condition happening on concurrent index-header lazy load and unload, when --store.enable-index-header-lazy-reader is enabled.
• #3773 Compact: Fixed compaction planner size check, making sure we don't create too large blocks.
• #3814 Store: Decreased memory utilisation while fetching block's chunks.
• #3815 Receive: Improve handling of empty time series from clients
• #3795 s3: A truncated "get object" response is reported as error.
• #3899 Receive: Correct the inference of client gRPC configuration.
• #3943 Receive: Fixed memory regression introduced in v0.17.0.
• #3960 Query: Fixed deduplication of equal alerts with different labels.

Changed

• #3804 Ruler, Receive, Querier: Updated Prometheus dependency. TSDB characteristics might have changed.

Added

• #3700 Compact/Web: Make old bucket viewer UI work with vanilla Prometheus blocks.
• #3657 *: It's now possible to configure HTTP transport options for S3 client.
• #3752 Compact/Store: Added --block-meta-fetch-concurrency allowing to configure number of go routines for block metadata synchronization.
• #3723 Query Frontend: Added --query-range.request-downsampled flag enabling additional queries for downsampled data in case of empty or incomplete response to range request.
• #3579 Cache: Added inmemory cache for caching bucket.
• #3792 Receiver: Added --tsdb.allow-overlapping-blocks flag to allow overlapping tsdb blocks and enable vertical compaction.
• #3740 Query: Added --query.default-step flag to set default step. Useful when your tenant scrape interval is stable and far from default UI's 1s.
• #3686 Query/Sidecar: Added metric metadata API support. You can now configure you Querier to fetch Prometheus metrics metadata from leaf Prometheus-es!
• #3031 Compact/Sidecar/Receive/Rule: Added --hash-func. If some function has been specified, writers calculate hashes using that function of each file in a block before uploading them. If those hashes exist in the meta.json file then Compact does not download the files if they already exist on disk and with the same hash. This also means that the data directory passed to Thanos Compact is only cleared once at boot or if everything succeeds. So, if you, for example, use persistent volumes on k8s and your Thanos Compact crashes or fails to make an iteration properly then the last downloaded files are not wiped from the disk. The directories that were created the last time are only wiped again after a successful iteration or if the previously picked up blocks have disappeared.

Fixed

• #3705 Store: Fix race condition leading to failing queries or possibly incorrect query results.
• #3661 Compact: Deletion-mark.json is deleted as the last one, which could in theory lead to potential store gateway load or query error for such in-deletion block.
• #3760 Store: Fix panic caused by a race condition happening on concurrent index-header reader usage and unload, when --store.enable-index-header-lazy-reader is enabled.
• #3759 Store: Fix panic caused by a race condition happening on concurrent index-header lazy load and unload, when --store.enable-index-header-lazy-reader is enabled.
• #3773 Compact: Fixed compaction planner size check, making sure we don't create too large blocks.
• #3814 Store: Decreased memory utilisation while fetching block's chunks.
• #3815 Receive: Improve handling of empty time series from clients
• #3795 s3: A truncated "get object" response is reported as error.
• #3899 Receive: Correct the inference of client gRPC configuration.
• #3943: Receive: Fixed memory regression introduced in v0.17.0.

Changed

• #3804 Ruler, Receive, Querier: Updated Prometheus dependency. TSDB characteristics might have changed.

Added

• #3700 Compact/Web: Make old bucket viewer UI work with vanilla Prometheus blocks.
• #3657 *: It's now possible to configure HTTP transport options for S3 client.
• #3752 Compact/Store: Added --block-meta-fetch-concurrency allowing to configure number of go routines for block metadata synchronization.
• #2641 Query Frontend: Added --query-range.request-downsampled flag enabling additional queries for downsampled data in case of empty or incomplete response to range request.
• #3579 Cache: Added inmemory cache for caching bucket.
• #3792 Receiver: Added --tsdb.allow-overlapping-blocks flag to allow overlapping tsdb blocks and enable vertical compaction.
• #3740 Query: Added --query.default-step flag to set default step. Useful when your tenant scrape interval is stable and far from default UI's 1s.
• #3686 Query,Sidecar: Added metric metadata API support. You can now configure you Querier to fetch Prometheus metrics metadata from leaf Prometheus-es!
• #3031 Compact/Sidecar/Receive/Rule: Added --hash-func. If some function has been specified, writers calculate hashes using that function of each file in a block before uploading them. If those hashes exist in the meta.json file then Compact does not download the files if they already exist on disk and with the same hash. This also means that the data directory passed to Thanos Compact is only cleared once at boot or if everything succeeds. So, if you, for example, use persistent volumes on k8s and your Thanos Compact crashes or fails to make an iteration properly then the last downloaded files are not wiped from the disk. The directories that were created the last time are only wiped again after a successful iteration or if the previously picked up blocks have disappeared.

Fixed

• #3705 Store: Fix race condition leading to failing queries or possibly incorrect query results.
• #3661 Compact: Deletion-mark.json is deleted as the last one, which could in theory lead to potential store gateway load or query error for such in-deletion block.
• #3760 Store: Fix panic caused by a race condition happening on concurrent index-header reader usage and unload, when --store.enable-index-header-lazy-reader is enabled.
• #3759 Store: Fix panic caused by a race condition happening on concurrent index-header lazy load and unload, when --store.enable-index-header-lazy-reader is enabled.
• #3773 Compact: Fixed compaction planner size check, making sure we don't create too large blocks.
• #3814 Store: Decreased memory utilisation while fetching block's chunks.
• #3815 Receive: Improve handling of empty time series from clients
• #3795 s3: A truncated "get object" response is reported as error.

Changed

• #3804 Ruler, Receive, Querier: Updated Prometheus dependency. TSDB characteristics might have changed.

Highlights

• Several big optimizations speeding up query performance were introduced!
• A new command was added, thanos tools bucket rewrite, enabling the deletion of series from given block.
• The Query Frontend now supports proxying requests to the labels and series API endpoints.
• thanos tools bucket replicate can now copy particular blocks by ID.
• The number of series touched by the Store component when servicing a Series call can now be limited using a CLI flag.

Added

• #3380 Mixin: Add block deletion panels for compactor dashboards.
• #3568 Store: Optimized inject label stage of index lookup.
• #3566 StoreAPI: Support label matchers in labels API.
• #3531 Store: Optimized common cases for time selecting smaller amount of series by avoiding looking up symbols.
• #3469 StoreAPI: Added hints field to LabelNamesRequest and LabelValuesRequest. Hints are an opaque data structure that can be used to carry additional information from the store and its content is implementation-specific.
• #3421 Tools: Added thanos tools bucket rewrite command allowing to delete series from given block.
• #3509 Store: Added a CLI flag to limit the number of series that are touched.
• #3444 Query Frontend: Make POST request to downstream URL for labels and series API endpoints.
• #3388 Tools: Bucket replicator now can specify block IDs to copy.
• #3385 Tools: Bucket prints extra statistics for block index with debug log-level.
• #3121 Receive: Added --receive.hashrings alternative to receive.hashrings-file flag (lower priority). The flag expects the literal hashring configuration in JSON format.

Fixed

• #3567 Mixin: Reintroduce thanos_objstore_bucket_operation_failures_total alert.
• #3527 Query Frontend: Fix query_range behavior when start/end times are the same
• #3560 Query Frontend: Allow separate label cache
• #3672 Rule: Prevent crashing due to no such host error when using dnssrv+ or dnssrvnoa+.
• #3461 Compact, Shipper, Store: Fixed panic when no external labels are set in block metadata.

Changed

• #3496 S3: Respect SignatureV2 flag for all credential providers.
• #2732 Swift: Switched to a new library ncw/swift providing large objects support.
  By default, segments will be uploaded to the same container directory segments/ if the file is bigger than 1GB.
  To change the defaults see the docs.
• #3626 Shipper: Failed upload of meta.json file doesn't cause block cleanup anymore. This has a potential to generate corrupted blocks under specific conditions. Partial block is left in bucket for later cleanup.