Bot releases are visible (Hide)

tempo - v2.4.1 Latest Release

Published by grafanabot 6 months ago

[BUGFIX] Fix compaction/retention in AWS S3 and GCS when a prefix is configured. #3465 (@bpfoster)

tempo - v2.4.1

Published by grafanabot 7 months ago

[BUGFIX] Fix compaction/retention in AWS S3 and GCS when a prefix is configured. #3465 (@bpfoster)

tempo - v2.4.0

Published by grafanabot 8 months ago

Breaking Changes

Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
Deprecating vParquet v1 #3377 (@ie-pham)
vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
Deprecate the following fields. These have all been migrated to a top level "cache:" field.
```
storage:
  trace:
    cache:
    search:
      cache_control:
    background_cache:
    memcached:
    redis:
```

Changes

[CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
[CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
[CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
[CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
[CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
[CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
[CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

[FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
[FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
[FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.
```
storage:
  trace:
    cache:
    search:
      cache_control:
    background_cache:
    memcached:
    redis:
```

Enhancements

[ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
[ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
[ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
[ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
[ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
[ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
[ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
[ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
[ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
[ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
[ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
[ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
[ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
[ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
[ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
[ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
[ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
[ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
[ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303https://github.com/grafana/tempo/pull/3303 (@zalegrala)
[ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
[ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
[ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
[ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)
[ENHANCEMENT] Config: Adds query-frontend.log-query-request-headers to enable logging of request headers in query logs. #3383 (@jmichalek132)

Bugfixes

[BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
[BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
[BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
[BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
[BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
[BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
[BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
[BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
[BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
[BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
[BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
[BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
[BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
[BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
[BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)
[BUGFIX] Fix some instances where spanmetrics histograms could be inconsistent #3412 (@mdisibio)

tempo - v2.4.0-rc.0

Published by grafanabot 8 months ago

Breaking Changes

Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
Deprecating vParquet v1 #3377 (@ie-pham)
vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
Deprecate the following fields. These have all been migrated to a top level "cache:" field.
```
storage:
  trace:
    cache:
    search:
      cache_control:
    background_cache:
    memcached:
    redis:
```

Changes

[CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
[CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
[CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
[CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
[CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
[CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
[CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

[FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
[FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
[FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.
```
storage:
  trace:
    cache:
    search:
      cache_control:
    background_cache:
    memcached:
    redis:
```

Enhancements

[ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
[ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
[ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
[ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
[ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
[ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
[ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
[ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
[ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
[ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
[ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
[ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
[ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
[ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
[ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
[ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
[ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
[ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
[ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303https://github.com/grafana/tempo/pull/3303 (@zalegrala)
[ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
[ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
[ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
[ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)

Bugfixes

[BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
[BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
[BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
[BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
[BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
[BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
[BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
[BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
[BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
[BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
[BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
[BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
[BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
[BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
[BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)

tempo - v2.3.1

Published by grafanabot 11 months ago

Bugfixes

[BUGFIX] Include statusMessage intrinsic attribute in tag search. #3084 (@rcrowe)
[BUGFIX] Fix compactor ignore configured S3 headers #3149 (@Batkilin)
[BUGFIX] Readd session token to s3 credentials. #3144 (@farodin91)

tempo - v2.3.0

Published by grafanabot 12 months ago

Breaking Changes

Removed deprecated config option: distributor.log_received_traces. Use the distributor.log_received_spans config block instead.
Removed frontend metrics tempo_query_frontend_queries_total{op="searchtags|metrics"}

Changes

[CHANGE] Update Go to 1.21 #2486 (@zalegrala)
[CHANGE] Moved the tempo_ingester_traces_created_total metric to be incremented when a trace is cut to the wal #2884 (@joe-elliott)
[CHANGE] Upgrade from deprecated azure-storage-blob-go SDK to azure-sdk-for-go #2835 (@LasseHels)
[CHANGE] Metrics summary API validate the requested time range #2902 (@mdisibio)
[CHANGE] Restructure Azure backends into versioned backends. Introduce use_v2_sdk config option for switching. #2952 (@zalegrala)
v1: azure-storage-blob-go original (now deprecated) SDK
v2: azure-sdk-for-go
[CHANGE] Adjust trace size estimation to better honor row group size settings. #3038 (@joe-elliott)
[CHANGE] Update alpine image version to 3.18. #3046 (@joe-elliott)
[CHANGE] Overrides module refactor #2688 (@mapno)
Added new defaults block to the overrides' module. Overrides change to indented syntax.
Old config:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

New config:

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]

Features

[FEATURE] New TraceQL structural operators ancestor (<<), parent (<) #2877 (@kousikmitra)
[FEATURE] Add the /api/status/buildinfo endpoint #2702 (@fabrizio-grafana)
[FEATURE] New encoding vParquet3 with support for dedicated attribute columns (@mapno, @stoewer) #2649
[FEATURE] Add filtering support to Generic Forwarding #2742 (@Blinkuu)
[FEATURE] Add cli command to print out summary of large traces #2775 (@ie-pham)
[FEATURE] Added not structural operators to TraceQL: !>, !<, and !~ #2993 (@joe-elliott)

Enhancements

[ENHANCEMENT] Make metrics-generator ingestion slack per tenant #2589 (@ie-pham)
[ENHANCEMENT] Support quoted attribute name in TraceQL #3004 (@kousikmitra)
[ENHANCEMENT] Add support for searching by span status message using statusMessage keyword #2848 (@kousikmitra)
[ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup #2697 (@mdisibio)
[ENHANCEMENT] Assert ingestion rate limits as early as possible #2640 (@mghildiy)
[ENHANCEMENT] Add several metrics-generator fields to user-configurable overrides #2711 (@kvrhdn)
[ENHANCEMENT] Update /api/metrics/summary to correctly handle missing attributes and improve performance of TraceQL select() queries. #2765 (@mdisibio)
[ENHANCEMENT] Tempo CLI command to convert from vParquet2 -> 3. #2828 (@joe-elliott)
[ENHANCEMENT] Add TempoUserConfigurableOverridesReloadFailing alert #2784 (@kvrhdn)
[ENHANCEMENT] Add RootSpanName and RootServiceName to log about discarded spans #2816 (@marcinginszt)
[ENHANCEMENT] Add UserID to log message about rate limiting #2850 (@lshippy)
[ENHANCEMENT] Requests to Azure Blob Storage will now be retried once instead of zero times #2835 (@LasseHels)
[ENHANCEMENT] Add span metrics filter policies to user-configurable overrides #2906 (@rlankfo)
[ENHANCEMENT] Add collection-interval to metrics-generator config in user-configurable overrides #2899 (@rlankfo)
[ENHANCEMENT] Enforce max trace size on the trace by id path. #2935 (@joe-elliott)
[ENHANCEMENT] Add target_info_excluded_dimensions to user-config api #2945 (@ie-pham)
[ENHANCEMENT] User-configurable overrides: add scope query parameter to return merged overrides for tenant #2915 #3018 (@kvrhdn)
[ENHANCEMENT] Add histogram buckets to metrics-generator config in user-configurable overrides #2928 (@mar4uk)
[ENHANCEMENT] Adds websocket support for search streaming. #2971 (@joe-elliott)
[ENHANCEMENT] Add new config block to distributors to produce debug metrics. #3008 (@joe-elliott)
Breaking Change Removed deprecated config option: distributor.log_received_traces
[ENHANCEMENT] added a metrics generator config option to enable/disable X-Scope-OrgID headers on remote write. #2974 (@vineetjp)
[ENHANCEMENT] Correctly return RetryInfo to Otel Collector/Grafana Agent on ResourceExhausted. This allows the agents to honor their own retry settings. #3019 (@joe-elliott)

Bugfixes

[BUGFIX] Unescape tag names #2894 (@fabrizio-grafana)
[BUGFIX] Load defaults for the internal server #3041 (@rubenvp8510)
[BUGFIX] Fix pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions #3012 (@electron0zero)
[BUGFIX] Fix panic in metrics summary api #2738 (@mdisibio)
[BUGFIX] Fix rare deadlock when uploading blocks to Azure Blob Storage #2129 (@LasseHels)
[BUGFIX] Only search ingester blocks that fall within the request time range. #2783 (@joe-elliott)
[BUGFIX] Align tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. #2840 (@joe-elliott)
This query will now correctly tell you %age of requests that are within SLO:
```
sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op) 
/
sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
```
BREAKING CHANGE Removed: tempo_query_frontend_queries_total{op="searchtags|metrics"}.
[BUGFIX] To support blob storage in Azure Stack Hub as backend. #2853 (@chlislb)
[BUGFIX] Respect spss on GRPC streaming. #2971 (@joe-elliott)
[BUGFIX] Moved empty root span substitution from querier to query-frontend. #2671 (@galalen)
[BUGFIX] Correctly propagate ingester errors on the query path #2935 (@joe-elliott)
[BUGFIX] Fix issue where ingester doesn't stop query after timeout #3031 (@mdisibio)
[BUGFIX] Fix cases where empty filter {} wouldn't return expected results #2498 (@mdisibio)
[BUGFIX] Reorder S3 credential chain and upgrade minio-go. native_aws_auth_enabled is deprecated #3006 (@ekristen, @mapno)

tempo - v2.2.4

Published by grafanabot 12 months ago

Changes

[CHANGE] Update alpine image version to 3.18 to patch CVE-2022-48174. #3046 (@joe-elliott)
[CHANGE] Bump Jaeger query docker image to 1.50.0 #2998 (@pavolloffay)

tempo - v2.3.0-rc.0

Published by grafanabot 12 months ago

Breaking Changes

Removed deprecated config option: distributor.log_received_traces. Use the distributor.log_received_spans config block instead.
Removed frontend metrics tempo_query_frontend_queries_total{op="searchtags|metrics"}

Changes

[CHANGE] Update Go to 1.21 #2486 (@zalegrala)
[CHANGE] Make metrics-generator ingestion slack per tenant #2589 (@ie-pham)
[CHANGE] Moved the tempo_ingester_traces_created_total metric to be incremented when a trace is cut to the wal #2884 (@joe-elliott)
[CHANGE] Upgrade from deprecated azure-storage-blob-go SDK to azure-sdk-for-go #2835 (@LasseHels)
[CHANGE] Metrics summary API validate the requested time range #2902 (@mdisibio)
[CHANGE] Restructure Azure backends into versioned backends. Introduce use_v2_sdk config option for switching. #2952 (@zalegrala)
v1: azure-storage-blob-go original (now deprecated) SDK
v2: azure-sdk-for-go
[CHANGE] Adjust trace size estimation to better honor row group size settings. #3038 (@joe-elliott)
[CHANGE] Update alpine image version to 3.18. #3046 (@joe-elliott)
[CHANGE] Overrides module refactor #2688 (@mapno)
Added new defaults block to the overrides' module. Overrides change to indented syntax.
Old config:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

New config:

overrides:
defaults:
  ingestion:
    rate_strategy: local
    rate_limit_bytes: 12345
    burst_size_bytes: 67890
  read:
    max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator:
    processors: [service-graphs, span-metrics]

Features

[FEATURE] New TraceQL structural operators ancestor (<<), parent (<) #2877 (@kousikmitra)
[FEATURE] Add the /api/status/buildinfo endpoint #2702 (@fabrizio-grafana)
[FEATURE] New encoding vParquet3 with support for dedicated attribute columns (@mapno, @stoewer) #2649
[FEATURE] Add filtering support to Generic Forwarding #2742 (@Blinkuu)
[FEATURE] Add cli command to print out summary of large traces #2775 (@ie-pham)
[FEATURE] Added not structural operators to TraceQL: !>, !<, and !~ #2993 (@joe-elliott)

Enhancements

[ENHANCEMENT] Support quoted attribute name in TraceQL #3004 (@kousikmitra)
[ENHANCEMENT] Unescape tag names #2894 (@fabrizio-grafana)
[ENHANCEMENT] Add support for searching by span status message using statusMessage keyword #2848 (@kousikmitra)
[ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup #2697 (@mdisibio)
[ENHANCEMENT] Assert ingestion rate limits as early as possible #2640 (@mghildiy)
[ENHANCEMENT] Add several metrics-generator fields to user-configurable overrides #2711 (@kvrhdn)
[ENHANCEMENT] Update /api/metrics/summary to correctly handle missing attributes and improve performance of TraceQL select() queries. #2765 (@mdisibio)
[ENHANCEMENT] Tempo CLI command to convert from vParquet2 -> 3. #2828 (@joe-elliott)
[ENHANCEMENT] Add TempoUserConfigurableOverridesReloadFailing alert #2784 (@kvrhdn)
[ENHANCEMENT] Add RootSpanName and RootServiceName to log about discarded spans #2816 (@marcinginszt)
[ENHANCEMENT] Add UserID to log message about rate limiting #2850 (@lshippy)
[ENHANCEMENT] Requests to Azure Blob Storage will now be retried once instead of zero times #2835 (@LasseHels)
[ENHANCEMENT] Add span metrics filter policies to user-configurable overrides #2906 (@rlankfo)
[ENHANCEMENT] Add collection-interval to metrics-generator config in user-configurable overrides #2899 (@rlankfo)
[ENHANCEMENT] Enforce max trace size on the trace by id path. #2935 (@joe-elliott)
[ENHANCEMENT] Add target_info_excluded_dimensions to user-config api #2945 (@ie-pham)
[ENHANCEMENT] User-configurable overrides: add scope query parameter to return merged overrides for tenant #2915 #3018 (@kvrhdn)
[ENHANCEMENT] Add histogram buckets to metrics-generator config in user-configurable overrides #2928 (@mar4uk)
[ENHANCEMENT] Adds websocket support for search streaming. #2971 (@joe-elliott)
[ENHANCEMENT] Add new config block to distributors to produce debug metrics. #3008 (@joe-elliott)
Breaking Change Removed deprecated config option: distributor.log_received_traces
[ENHANCEMENT] added a metrics generator config option to enable/disable X-Scope-OrgID headers on remote write. #2974 (@vineetjp)
[ENHANCEMENT] Correctly return RetryInfo to Otel Collector/Grafana Agent on ResourceExhausted. This allows the agents to honor their own retry
settings. #3019 (@joe-elliott)

Bugfixes

[BUGFIX] Load defaults for the internal server #3041 (@rubenvp8510)
[BUGFIX] Fix pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions #3012 (@electron0zero)
[BUGFIX] Fix panic in metrics summary api #2738 (@mdisibio)
[BUGFIX] Fix rare deadlock when uploading blocks to Azure Blob Storage #2129 (@LasseHels)
[BUGFIX] Only search ingester blocks that fall within the request time range. #2783 (@joe-elliott)
[BUGFIX] Align tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. #2840 (@joe-elliott)
This query will now correctly tell you %age of requests that are within SLO:
```
sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op) 
/
sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
```
BREAKING CHANGE Removed: tempo_query_frontend_queries_total{op="searchtags|metrics"}.
[BUGFIX] To support blob storage in Azure Stack Hub as backend. #2853 (@chlislb)
[BUGFIX] Respect spss on GRPC streaming. #2971 (@joe-elliott)
[BUGFIX] Moved empty root span substitution from querier to query-frontend. #2671 (@galalen)
[BUGFIX] Correctly propagate ingester errors on the query path #2935 (@joe-elliott)
[BUGFIX] Fix issue where ingester doesn't stop query after timeout #3031 (@mdisibio)
[BUGFIX] Fix cases where empty filter {} wouldn't return expected results #2498 (@mdisibio)
[BUGFIX] Reorder S3 credential chain and upgrade minio-go. native_aws_auth_enabled is deprecated #3006 (@ekristen, @mapno)

tempo - v2.2.3

Published by grafanabot about 1 year ago

Bugfixes

Fix S3 credentials providers configuration https://github.com/grafana/tempo/pull/2889 (@mapno)

tempo - v2.2.2

Published by grafanabot about 1 year ago

Bugfixes

[BUGFIX] Fix node role auth IDMSv1 #2760 (@coufalja)

tempo - v2.2.1

Published by grafanabot about 1 year ago

Bugfixes

[BUGFIX] Fix incorrect metrics for index failures https://github.com/grafana/tempo/pull/2781 (@zalegrala)
[BUGFIX] Fix panic in the metrics-generator when using multiple tenants with default overrides https://github.com/grafana/tempo/pull/2786 (@kvrhdn)
[BUGFIX] Restore tenant_header_key removed in #2414. https://github.com/grafana/tempo/pull/2795 (@joe-elliott)
[BUGFIX] Disable streaming over http by default. https://github.com/grafana/tempo/pull/2803 (@joe-elliott)

tempo - v2.2.0

Published by grafanabot about 1 year ago

Breaking Changes

[CHANGE] Breaking Change Remove support tolerate_failed_blocks. #2416 (@joe-elliott)
Removed config option:
```
query_frontend:
  tolerate_failed_blocks: <int>
```
[CHANGE] Breaking Change Rename s3.insecure_skip_verify #2407 (@zalegrala)

storage:
  trace:
    s3:
      insecure_skip_verify: true   // renamed to tls_insecure_skip_verify

[CHANGE] Breaking Change Convert metrics generator from deployment to a statefulset in jsonnet. Refer to the PR for seamless migration instructions. #2533 #2467 (@zalegrala)

Changes

[CHANGE] Make vParquet2 the default block format #2526 (@stoewer)
[CHANGE] Change log level of two compactor messages from debug to info. #2443 (@dylanguedes)
[CHANGE] Remove tenant_header_key option from tempo-query config #2414 (@kousikmitra)
[CHANGE] Prefix service graph extra dimensions labels with server_ and client_ if enable_client_server_prefix is enabled #2335 (@domasx2)
[CHANGE] Ignore context canceled errors in the queriers #2440 (@joe-elliott)
[CHANGE] Start flush queue worker after wal replay and block rediscovery #2456 (@ie-pham)
[CHANGE] Update Go to 1.20.4 #2486 (@ie-pham)

Features

[FEATURE] New experimental API to derive on-demand RED metrics grouped by any attribute, and new metrics generator processor #2368 #2418 #2424 #2442 #2480 #2481 #2501 #2579 #2582 (@mdisibio @zalegrala)
[FEATURE] New TraceQL structural operators descendant (>>), child (>), and sibling (~) #2625 #2660 (@mdisibio)
[FEATURE] Add user-configurable overrides module #2543 (@electron0zero @kvrhdn)
[FEATURE] Add support for q query param in /api/v2/search/<tag.name>/values to filter results based on a TraceQL query #2253 (@mapno)
To make use of filtering, configure autocomplete_filtering_enabled.
[FEATURE] Add support for by() and coalesce() to TraceQL. #2490
[FEATURE] Add a GRPC streaming endpoint for traceql search #2366 (@joe-elliott)
[FEATURE] Add select() to TraceQL #2494 (@joe-elliott)
[FEATURE] Add traceDuration, rootName and rootServiceName intrinsics to TraceQL #2503 (@joe-elliott)

Enhancements

[ENHANCEMENT] Add support for query batching between frontend and queriers to improve throughput #2677 (@joe-elliott)
[ENHANCEMENT] Add initial RBAC support for serverless backend queries, limited to Google CloudRun #2487 (@modulitos)
[ENHANCEMENT] Add capability to flush all remaining traces to backend when ingester is stopped #2538(@mghildiy)
[ENHANCEMENT] Fill parent ID column and nested set columns #2487 (@stoewer)
[ENHANCEMENT] Add metrics generator config option to allow customizable ring port #2399 (@mdisibio)
[ENHANCEMENT] Improve performance of TraceQL regex #2484 (@mdisibio)
[ENHANCEMENT] log client ip to help identify which client is no org id #2436 (@Komey)
[ENHANCEMENT] Add spss parameter to /api/search/tags to configure the spans per span set in response#2308 (@stoewer)
[ENHANCEMENT] Continue polling tenants on error with configurable threshold #2540 (@mdisibio)
[ENHANCEMENT] Fully skip over parquet row groups with no matches in the column dictionaries #2676 (@mdisibio)
[ENHANCEMENT] Add prefix configuration option for s3 #2362 (@kousikmitra)
[ENHANCEMENT] Add support to filter using negated regex operator !~ #2410 (@kousikmitra)
[ENHANCEMENT] Add prefix configuration option to storage.trace.azure and storage.trace.gcs #2386 (@kousikmitra)
[ENHANCEMENT] Add prefix configuration option to storage.trace.s3 #2362 (@kousikmitra)
[ENHANCEMENT] Add support for concurrent_shards under trace_by_id #2416 (@joe-elliott)
```
query_frontend:
  trace_by_id:
    concurrent_shards: 3
```

[ENHANCEMENT] Enable cross cluster querying by adding two config options. #2598 (@joe-elliott)

querier:
  secondary_ingester_ring: <string>
metrics_generator:
  override_ring_key: <string>

[ENHANCEMENT] Add scope parameter to /api/search/tags #2282 (@joe-elliott)
Create new endpoint /api/v2/search/tags that returns all tags organized by scope.
[ENHANCEMENT] Ability to toggle off latency or count metrics in metrics-generator #2070 (@AlexDHoffer)
[ENHANCEMENT] Extend /flush to support flushing a single tenant #2260 (@kvrhdn)
[ENHANCEMENT] Add override to limit number of blocks inspected in tag value search #2358 (@mapno)
[ENHANCEMENT] New synchronous read mode for vParquet and vParquet2 #2165 #2535 (@mdisibio)
[ENHANCEMENT] Add option to override metrics-generator ring port #2399 (@mdisibio)
[ENHANCEMENT] Add support for IPv6 #1555 (@zalegrala)
[ENHANCEMENT] Add span filtering to spanmetrics processor #2274 (@zalegrala)
[ENHANCEMENT] Add ability to detect virtual nodes in the servicegraph processor #2365 (@mapno)
[ENHANCEMENT] Introduce overrides.Interface to decouple implementation from usage #2482 (@kvrhdn)
[ENHANCEMENT] Improve TraceQL throughput by asynchronously creating jobs #2530 (@joe-elliott)

Bugfixes

[BUGFIX] Fix Search SLO by routing tags to a new handler. #2468 (@electron0zero)
[BUGFIX] tempodb integer divide by zero error #2167 (@kroksys)
[BUGFIX] metrics-generator: ensure Prometheus will scale up shards when remote write is lagging behind #2463 (@kvrhdn)
[BUGFIX] Fixes issue where matches and other spanset level attributes were not persisted to the TraceQL results. #2490
[BUGFIX] Fixes issue where ingester search could occasionally fail with file does not exist error #2534 (@mdisibio)
[BUGFIX] Tempo failed to find meta.json path after adding prefix in S3/GCS/Azure configuration. #2585 (@WildCatFish)
[BUGFIX] Delay logging config warnings until the logger has been initialized #2645 (@kvrhdn)
[BUGFIX] Fix issue where metrics-generator was setting wrong labels for traces_target_info #2546 (@ie-pham)

tempo - v2.2.0-rc.0

Published by grafanabot about 1 year ago

Breaking Changes

[CHANGE] Breaking Change Remove support tolerate_failed_blocks. #2416 (@joe-elliott)
Removed config option:
```
query_frontend:
  tolerate_failed_blocks: <int>
```
[CHANGE] Breaking Change Rename s3.insecure_skip_verify #2407 (@zalegrala)

storage:
  trace:
    s3:
      insecure_skip_verify: true   // renamed to tls_insecure_skip_verify

[CHANGE] Breaking Change Convert metrics generator from deployment to a statefulset in jsonnet. Refer to the PR for seamless migration instructions. #2533 #2467 (@zalegrala)

Changes

[CHANGE] Make vParquet2 the default block format #2526 (@stoewer)
[CHANGE] Change log level of two compactor messages from debug to info. #2443 (@dylanguedes)
[CHANGE] Remove tenant_header_key option from tempo-query config #2414 (@kousikmitra)
[CHANGE] Prefix service graph extra dimensions labels with server_ and client_ if enable_client_server_prefix is enabled #2335 (@domasx2)
[CHANGE] Ignore context canceled errors in the queriers #2440 (@joe-elliott)
[CHANGE] Start flush queue worker after wal replay and block rediscovery #2456 (@ie-pham)
[CHANGE] Update Go to 1.20.4 #2486 (@ie-pham)

Features

[FEATURE] New experimental API to derive on-demand RED metrics grouped by any attribute, and new metrics generator processor #2368 #2418 #2424 #2442 #2480 #2481 #2501 #2579 #2582 (@mdisibio @zalegrala)
[FEATURE] New TraceQL structural operators descendant (>>), child (>), and sibling (~) #2625 #2660 (@mdisibio)
[FEATURE] Add user-configurable overrides module #2543 (@electron0zero @kvrhdn)
[FEATURE] Add support for q query param in /api/v2/search/<tag.name>/values to filter results based on a TraceQL query #2253 (@mapno)
To make use of filtering, configure autocomplete_filtering_enabled.
[FEATURE] Add support for by() and coalesce() to TraceQL. #2490
[FEATURE] Add a GRPC streaming endpoint for traceql search #2366 (@joe-elliott)
[FEATURE] Add select() to TraceQL #2494 (@joe-elliott)
[FEATURE] Add traceDuration, rootName and rootServiceName intrinsics to TraceQL #2503 (@joe-elliott)

Enhancements

[ENHANCEMENT] Add support for query batching between frontend and queriers to improve throughput #2677 (@joe-elliott)
[ENHANCEMENT] Add initial RBAC support for serverless backend queries, limited to Google CloudRun #2487 (@modulitos)
[ENHANCEMENT] Add capability to flush all remaining traces to backend when ingester is stopped #2538(@mghildiy)
[ENHANCEMENT] Fill parent ID column and nested set columns #2487 (@stoewer)
[ENHANCEMENT] Add metrics generator config option to allow customizable ring port #2399 (@mdisibio)
[ENHANCEMENT] Improve performance of TraceQL regex #2484 (@mdisibio)
[ENHANCEMENT] log client ip to help identify which client is no org id #2436 (@Komey)
[ENHANCEMENT] Add spss parameter to /api/search/tags to configure the spans per span set in response#2308 (@stoewer)
[ENHANCEMENT] Continue polling tenants on error with configurable threshold #2540 (@mdisibio)
[ENHANCEMENT] Fully skip over parquet row groups with no matches in the column dictionaries #2676 (@mdisibio)
[ENHANCEMENT] Add prefix configuration option for s3 #2362 (@kousikmitra)
[ENHANCEMENT] Add support to filter using negated regex operator !~ #2410 (@kousikmitra)
[ENHANCEMENT] Add prefix configuration option to storage.trace.azure and storage.trace.gcs #2386 (@kousikmitra)
[ENHANCEMENT] Add prefix configuration option to storage.trace.s3 #2362 (@kousikmitra)
[ENHANCEMENT] Add support for concurrent_shards under trace_by_id #2416 (@joe-elliott)
```
query_frontend:
  trace_by_id:
    concurrent_shards: 3
```

[ENHANCEMENT] Enable cross cluster querying by adding two config options. #2598 (@joe-elliott)

querier:
  secondary_ingester_ring: <string>
metrics_generator:
  override_ring_key: <string>

[ENHANCEMENT] Add scope parameter to /api/search/tags #2282 (@joe-elliott)
Create new endpoint /api/v2/search/tags that returns all tags organized by scope.
[ENHANCEMENT] Ability to toggle off latency or count metrics in metrics-generator #2070 (@AlexDHoffer)
[ENHANCEMENT] Extend /flush to support flushing a single tenant #2260 (@kvrhdn)
[ENHANCEMENT] Add override to limit number of blocks inspected in tag value search #2358 (@mapno)
[ENHANCEMENT] New synchronous read mode for vParquet and vParquet2 #2165 #2535 (@mdisibio)
[ENHANCEMENT] Add option to override metrics-generator ring port #2399 (@mdisibio)
[ENHANCEMENT] Add support for IPv6 #1555 (@zalegrala)
[ENHANCEMENT] Add span filtering to spanmetrics processor #2274 (@zalegrala)
[ENHANCEMENT] Add ability to detect virtual nodes in the servicegraph processor #2365 (@mapno)
[ENHANCEMENT] Introduce overrides.Interface to decouple implementation from usage #2482 (@kvrhdn)
[ENHANCEMENT] Improve TraceQL throughput by asynchronously creating jobs #2530 (@joe-elliott)

Bugfixes

[BUGFIX] Fix Search SLO by routing tags to a new handler. #2468 (@electron0zero)
[BUGFIX] tempodb integer divide by zero error #2167 (@kroksys)
[BUGFIX] metrics-generator: ensure Prometheus will scale up shards when remote write is lagging behind #2463 (@kvrhdn)
[BUGFIX] Fixes issue where matches and other spanset level attributes were not persisted to the TraceQL results. #2490
[BUGFIX] Fixes issue where ingester search could occasionally fail with file does not exist error #2534 (@mdisibio)
[BUGFIX] Tempo failed to find meta.json path after adding prefix in S3/GCS/Azure configuration. #2585 (@WildCatFish)
[BUGFIX] Delay logging config warnings until the logger has been initialized #2645 (@kvrhdn)
[BUGFIX] Fix issue where metrics-generator was setting wrong labels for traces_target_info #2546 (@ie-pham)

tempo - v2.1.1

Published by grafanabot over 1 year ago

Bugfixes

[BUGFIX] Fix issue where Tempo sometimes flips booleans from false->true at storage time. #2400 (@joe-elliott)

tempo - v2.1.0

Published by grafanabot over 1 year ago

Breaking Changes

[CHANGE] Breaking Change Remove support for search on v2 blocks. #2159 (@joe-elliott)
Removed config options:
```
overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:
```
Removed metrics:
tempo_ingester_trace_search_bytes_discarded_total
[CHANGE] BREAKING CHANGE Change metrics prefixed with cortex_ to tempo_ #2204 (@mapno)
[ENHANCEMENT] Add Throughput and SLO Metrics with SLOConfig in Query Frontend #2008 (@electron0zero)
- BREAKING CHANGE query_frontend_result_metrics_inspected_bytes metric removed in favour of query_frontend_bytes_processed_per_second

Changes

[CHANGE] Stop caching parquet files for search #2164 (@mapno)
[CHANGE] Update Go to 1.20 #2079 (@scalalang2)
[CHANGE] Upgrade OTel to v0.74.0 #2317 (@mapno)

Features

[FEATURE] New parquet based block format vParquet2 #2244 (@stoewer)
[FEATURE] Add support for Azure Workload Identity authentication #2195 (@LambArchie)
[FEATURE] Add flag to check configuration #2131 (@robertscherbarth @agrib-01)
[FEATURE] Add flag to optionally enable all available Go runtime metrics #2005 (@andreasgerstmayr)
[FEATURE] Add support for span kind to TraceQL #2217 (@joe-elliott)
[FEATURE] Add support for min/max/sum aggregates to TraceQL#2255 (@joe-elliott)

Enhancements

[ENHANCEMENT] Capture and update search metrics for TraceQL #2087 (@electron0zero)
[ENHANCEMENT] Add support for s3 session token in static config #2093 (@farodin91)
[ENHANCEMENT] Metrics generator to make use of counters earlier #2068 (@zalegrala)
[ENHANCEMENT] Log when a trace is too large to compact #2105 (@scalalang2)
[ENHANCEMENT] Add support for arbitrary arithemtic to TraceQL queries #2146 (@joe-elliott)
[ENHANCEMENT] tempo-cli: add command to migrate a tenant #2130 (@kvrhdn)
[ENHANCEMENT] Added the ability to multiple span metrics by an attribute such as X-SampleRatio #2172 (@altanozlu)

Bugfixes

[BUGFIX] Correctly connect context during compaction #2220 (@ie-pham)
[BUGFIX] Apply rate() to bytes/s panel in tenant's dashboard. #2081 (@mapno)
[BUGFIX] Retry copy operations during compaction in GCS backend #2111 (@mapno)
[BUGFIX] Fix float/int comparisons in TraceQL. #2139 (@joe-elliott)
[BUGFIX] Improve locking and search head block in SearchTagValuesV2 #2164 (@mapno)
[BUGFIX] Fix not closing WAL block file before attempting to delete the folder. #2152 (@kostya9)
[BUGFIX] Stop searching for virtual tags if there are any hits.
This prevents invalid values from showing up for intrinsics like status #2219 (@joe-elliott)
[BUGFIX] Correctly return unique spans when &&ing and ||ing spansets. #2254 (@joe-elliott)
[BUGFIX] Support negative values on aggregate filters like count() > -1. #2289 (@joe-elliott)
[BUGFIX] Support float as duration like {duration > 1.5s} [#2304]https://github.com/grafana/tempo/pull/2304 (@ie-pham)

tempo - v2.1.0-rc.0

Published by grafanabot over 1 year ago

Breaking Changes

[CHANGE] Breaking Change Remove support for search on v2 blocks. #2159 (@joe-elliott)
Removed config options:
```
overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:
```
Removed metrics:
tempo_ingester_trace_search_bytes_discarded_total
[CHANGE] BREAKING CHANGE Change metrics prefixed with cortex_ to tempo_ #2204 (@mapno)
[ENHANCEMENT] Add Throughput and SLO Metrics with SLOConfig in Query Frontend #2008 (@electron0zero)
- BREAKING CHANGE query_frontend_result_metrics_inspected_bytes metric removed in favour of query_frontend_bytes_processed_per_second

Changes

[CHANGE] Stop caching parquet files for search #2164 (@mapno)
[CHANGE] Update Go to 1.20 #2079 (@scalalang2)
[CHANGE] Upgrade OTel to v0.74.0 #2317 (@mapno)

Features

[FEATURE] New parquet based block format vParquet2 #2244 (@stoewer)
[FEATURE] Add support for Azure Workload Identity authentication #2195 (@LambArchie)
[FEATURE] Add flag to check configuration #2131 (@robertscherbarth @agrib-01)
[FEATURE] Add flag to optionally enable all available Go runtime metrics #2005 (@andreasgerstmayr)
[FEATURE] Add support for span kind to TraceQL #2217 (@joe-elliott)
[FEATURE] Add support for min/max/sum aggregates to TraceQL#2255 (@joe-elliott)

Enhancements

[ENHANCEMENT] Capture and update search metrics for TraceQL #2087 (@electron0zero)
[ENHANCEMENT] Add support for s3 session token in static config #2093 (@farodin91)
[ENHANCEMENT] Metrics generator to make use of counters earlier #2068 (@zalegrala)
[ENHANCEMENT] Log when a trace is too large to compact #2105 (@scalalang2)
[ENHANCEMENT] Add support for arbitrary arithemtic to TraceQL queries #2146 (@joe-elliott)
[ENHANCEMENT] tempo-cli: add command to migrate a tenant #2130 (@kvrhdn)
[ENHANCEMENT] Added the ability to multiple span metrics by an attribute such as X-SampleRatio #2172 (@altanozlu)

Bugfixes

[BUGFIX] Correctly connect context during compaction #2220 (@ie-pham)
[BUGFIX] Apply rate() to bytes/s panel in tenant's dashboard. #2081 (@mapno)
[BUGFIX] Retry copy operations during compaction in GCS backend #2111 (@mapno)
[BUGFIX] Fix float/int comparisons in TraceQL. #2139 (@joe-elliott)
[BUGFIX] Improve locking and search head block in SearchTagValuesV2 #2164 (@mapno)
[BUGFIX] Fix not closing WAL block file before attempting to delete the folder. #2139 (@kostya9)
[BUGFIX] Stop searching for virtual tags if there are any hits.
This prevents invalid values from showing up for intrinsics like status #2219 (@joe-elliott)
[BUGFIX] Correctly return unique spans when &&ing and ||ing spansets. #2254 (@joe-elliott)
[BUGFIX] Support negative values on aggregate filters like count() > -1. #2289 (@joe-elliott)
[BUGFIX] Support float as duration like {duration > 1.5s} [#2304]https://github.com/grafana/tempo/pull/2304 (@ie-pham)

tempo - v2.0.1

Published by grafanabot over 1 year ago

Changes

[CHANGE] No longer return status.code from /api/search/tags unless it is an attribute present in the data #2059 (@mdisibio)
[CHANGE] Pad leading zeroes in span id to always be 16 chars #2062 (@ie-pham)

Bugfixes

[BUGFIX] Suppress logspam in single binary mode when metrics generator is disabled. #2058 (@joe-elliott)
[BUGFIX] Error more gracefully while reading some blocks written by an interim commit between 1.5 and 2.0 #2055 (@mdisibio)
[BUGFIX] Correctly coalesce trace level data when combining Parquet traces. #2095 (@joe-elliott)
[BUGFIX] Unescape query parameters in AWS Lambda to allow TraceQL queries to work. #2114 (@joe-elliott)

tempo - v2.0.0

Published by grafanabot over 1 year ago

Breaking Changes

Config

[CHANGE] BREAKING CHANGE Use snake case on Azure Storage config #1879 (@faustodavid)
Example of using snake case on Azure Storage config:

# config.yaml
storage:
  trace:
    azure:
      storage_account_name:
      storage_account_key:
      container_name:

[CHANGE] Config updates to prepare for Tempo 2.0. #1978 (@joe-elliott)

query_frontend:
  query_shards:                  // removed. use trace_by_id.query_shards
querier:
    query_timeout:               // removed. use trace_by_id.query_timeout
compactor:
    compaction:
        chunk_size_bytes:        // renamed to v2_in_buffer_bytes
        flush_size_bytes:        // renamed to v2_out_buffer_bytes
        iterator_buffer_size:    // renamed to v2_prefetch_traces_count
ingester:
    use_flatbuffer_search:       // removed. automatically set based on block type
storage:
  traces:
    wal:
        encoding:                // renamed to v2_encoding
        version:                 // removed and pinned to block.version
    block:
        index_downsample_bytes:  // renamed to v2_index_downsample_bytes
        index_page_size_bytes:   // renamed to v2_index_page_size_bytes
        encoding:                // renamed to v2_encoding
        row_group_size_bytes:    // renamed to parquet_row_group_size_bytes

[CHANGE] BREAKING CHANGE Remove search_enabled and metrics_generator_enabled. Both default to true. #2004 (@joe-elliott)
[CHANGE] BREAKING CHANGE Parquet is the new default block version #1678
To continue using the v2 backend set:
```
storage:
  trace:
    block:
      version: v2
```

Jsonnet

[CHANGE] Delete TempoRequestErrors alert from mixin #1810 (@zalegrala)
- BREAKING CHANGE Any jsonnet users relying on this alert should copy this into their own environment.

Metrics Generator

[CHANGE] metrics-generator: handle collisions between user defined and default dimensions #1794 (@stoewer)
BREAKING CHANGE Custom dimensions colliding with intrinsic dimensions will be prefixed with __.

Changes

[CHANGE] Increase default values for server.grpc_server_max_recv_msg_size and server.grpc_server_max_send_msg_size from 4MB to 16MB #1688 (@mapno)
[CHANGE] Update Go to 1.19 #1665 (@ie-pham)
[CHANGE] Update alpine image version to 3.16. #1784 (@zalegrala)

[CHANGE] Config updates to prepare for Tempo 2.0. #1978 (@joe-elliott)
Defaults updated:

query_frontend:
  max_oustanding_per_tenant: 2000
  search:
      concurrent_jobs: 1000
      target_bytes_per_job: 104857600
      max_duration: 168h
      query_ingesters_until: 30m
  trace_by_id:
      query_shards: 50
querier:
    max_concurrent_queries: 20
    search:
        prefer_self: 10
ingester:
    concurrent_flushes: 4
    max_block_duration: 30m
    max_block_bytes: 524288000
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        search:
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576

Features

[FEATURE] TraceQL Phase 1 support. A new query language for traces!
[FEATURE] Parquet backend is GA and default. By default Tempo will create Parquet blocks to enhance search performance
and to provide users with their data in an open format. See breaking changes above for a note on how to continue using older backends.

[FEATURE] Add generic forwarder and implement otlpgrpc forwarder #1775 (@Blinkuu)
New config options and example configuration:

# config.yaml
distributor:
  forwarders:
    - name: "otel-forwarder"
      backend: "otlpgrpc"
      otlpgrpc:
        endpoints: ['otelcol:4317']
        tls:
          insecure: true

# overrides.yaml
overrides:
  "example-tenant-1":
    forwarders: ['otel-forwarder']
  "example-tenant-2":
    forwarders: ['otel-forwarder']

Enhancements

[ENHANCEMENT] Add /status/usage-stats endpoint to show usage stats data #1782 (@electron0zero)
[ENHANCEMENT] Add TLS support to jaeger query plugin. #1999 (@rubenvp8510)
[ENHANCEMENT] Collect inspectedBytes from SearchMetrics #1975 (@electron0zero)
[ENHANCEMENT] Add zone awareness replication for ingesters. #1936 (@manohar-koukuntla)

# use the following fields in _config field of jsonnet config, to enable zone aware ingester
    multi_zone_ingester_enabled: false,
    multi_zone_ingester_migration_enabled: false,
    multi_zone_ingester_replicas: 0,
    multi_zone_ingester_max_unavailable: 25,

[ENHANCEMENT] Add new data-type aware searchtagvalues v2 api #1956 (@mdisibio)
[ENHANCEMENT] Filter namespace by cluster in tempo dashboards variables #1771 (@electron0zero)
[ENHANCEMENT] Exit early from sharded search requests #1742 (@electron0zero)
[ENHANCEMENT] Avoid running tempodb pool jobs with a cancelled context #1852 (@zalegrala)
[ENHANCEMENT] Add config flag to allow for compactor disablement for debug purposes #1850 (@zalegrala)
[ENHANCEMENT] Identify bloom that could not be retrieved from backend block #1737 (@AlexDHoffer)
[ENHANCEMENT] tempo: check configuration returns now a list of warnings #1663 (@frzifus)
[ENHANCEMENT] Make DNS address fully qualified to reduce DNS lookups in Kubernetes #1687 (@electron0zero)
[ENHANCEMENT] Return 200 instead of 206 when blocks failed is < tolerate_failed_blocks. #1725 (@joe-elliott)
[ENHANCEMENT] Add GOMEMLIMIT variable to compactor jsonnet and set the value to equal compactor memory limit. #1758 (@ie-pham)
[ENHANCEMENT] Add capability to configure the used S3 Storage Class #1697 (@amitsetty)
[ENHANCEMENT] cache: expose username and sentinel_username redis configuration options for ACL-based Redis Auth support #1708 (@jsievenpiper)
[ENHANCEMENT] metrics-generator: expose span size as a metric #1662 (@ie-pham)
[ENHANCEMENT] Set Max Idle connections to 100 for Azure, should reduce DNS errors in Azure #1632 (@electron0zero)
[ENHANCEMENT] Add PodDisruptionBudget to ingesters in jsonnet #1691 (@joe-elliott)
[ENHANCEMENT] Add a cli command to convert a block to the current parquet schema. #1707 (@joe-elliott)
[ENHANCEMENT] metrics-generator: filter out older spans before metrics are aggregated #1612 (@ie-pham)
[ENHANCEMENT] Add hedging to trace by ID lookups created by the frontend. #1735 (@mapno)
New config options and defaults:

query_frontend:
  trace_by_id:
    hedge_requests_at: 5s
    hedge_requests_up_to: 3

[ENHANCEMENT] Vulture now has improved distribution of the random traces it searches. #1763 (@rfratto)
[ENHANCEMENT] Add TLS support to the vulture #1874 (@zalegrala)
[ENHANCEMENT] metrics-generator: extract status_message field from spans #1786, #1794 (@stoewer)
[ENHANCEMENT] metrics-generator: handle collisions between user defined and default dimensions #1794 (@stoewer)
[ENHANCEMENT] metrics-generator: make intrinsic dimensions configurable and disable status_message by default #1960 (@stoewer)
[ENHANCEMENT] distributor: Log span names when distributor.log_received_spans.include_all_attributes is on #1790 (@suraciii)
[ENHANCEMENT] metrics-generator: truncate label names and values exceeding a configurable length #1897 (@kvrhdn)
[ENHANCEMENT] Convert last few Jsonnet alerts with per_cluster_label #2000 (@Whyeasy)
[ENHANCEMENT] New tenant dashboard #1901 (@mapno)
[ENHANCEMENT] Upgrade opentelemetry-proto submodule to v0.18.0 Internal types are updated to use scope instead of instrumentation_library.
This is a breaking change in trace by ID queries if JSON is requested. #1754 (@mapno)

Bugfixes

[BUGFIX] Stop distributors on Otel receiver fatal error#1887 (@rdooley)
[BUGFIX] New wal file separator '+' for the NTFS filesystem and backward compatibility with the old separator ':' #1700 (@kilian-kier)
[BUGFIX] Honor caching and buffering settings when finding traces by id #1697 (@joe-elliott)
[BUGFIX] Correctly propagate errors from the iterator layer up through the queriers #1723 (@joe-elliott)
[BUGFIX] Make multitenancy work with HTTP #1781 (@gouthamve)
[BUGFIX] Fix parquet search bug fix on http.status_code that may cause incorrect results to be returned #1799 (@mdisibio)
[BUGFIX] tempo-mixin: tweak dashboards to support metrics without cluster label present #1913 (@kvrhdn)
[BUGFIX] Fix docker-compose examples not running on Apple M1 hardware #1920 (@stoewer)
[BUGFIX] Don't persist tenants without blocks in the ingester#1947 (@joe-elliott)
[BUGFIX] Return more consistent search results by combining partial traces # (@mapno)

tempo - v2.0.0-rc.0

Published by joe-elliott over 1 year ago

Breaking Changes

Config

[CHANGE] BREAKING CHANGE Use snake case on Azure Storage config #1879 (@faustodavid)
Example of using snake case on Azure Storage config:

# config.yaml
storage:
  trace:
    azure:
      storage_account_name:
      storage_account_key:
      container_name:

[CHANGE] Config updates to prepare for Tempo 2.0. #1978 (@joe-elliott)

query_frontend:
  query_shards:                  // removed. use trace_by_id.query_shards
querier:
    query_timeout:               // removed. use trace_by_id.query_timeout
compactor:
    compaction:
        chunk_size_bytes:        // renamed to v2_in_buffer_bytes
        flush_size_bytes:        // renamed to v2_out_buffer_bytes
        iterator_buffer_size:    // renamed to v2_prefetch_traces_count
ingester:
    use_flatbuffer_search:       // removed. automatically set based on block type
storage:
  traces:
    wal:
        encoding:                // renamed to v2_encoding
        version:                 // removed and pinned to block.version
    block:
        index_downsample_bytes:  // renamed to v2_index_downsample_bytes
        index_page_size_bytes:   // renamed to v2_index_page_size_bytes
        encoding:                // renamed to v2_encoding
        row_group_size_bytes:    // renamed to parquet_row_group_size_bytes

[CHANGE] BREAKING CHANGE Remove search_enabled and metrics_generator_enabled. Both default to true. #2004 (@joe-elliott)
[CHANGE] BREAKING CHANGE Parquet is the new default block version #1678
To continue using the v2 backend set:
```
storage:
  trace:
    block:
      version: v2
```

Jsonnet

[CHANGE] Delete TempoRequestErrors alert from mixin #1810 (@zalegrala)
- BREAKING CHANGE Any jsonnet users relying on this alert should copy this into their own environment.

Metrics Generator

[CHANGE] metrics-generator: handle collisions between user defined and default dimensions #1794 (@stoewer)
BREAKING CHANGE Custom dimensions colliding with intrinsic dimensions will be prefixed with __.

Changes

[CHANGE] Increase default values for server.grpc_server_max_recv_msg_size and server.grpc_server_max_send_msg_size from 4MB to 16MB #1688 (@mapno)
[CHANGE] Update Go to 1.19 #1665 (@ie-pham)
[CHANGE] Update alpine image version to 3.16. #1784 (@zalegrala)

[CHANGE] Config updates to prepare for Tempo 2.0. #1978 (@joe-elliott)
Defaults updated:

query_frontend:
  max_oustanding_per_tenant: 2000
  search:
      concurrent_jobs: 1000
      target_bytes_per_job: 104857600
      max_duration: 168h
      query_ingesters_until: 30m
  trace_by_id:
      query_shards: 50
querier:
    max_concurrent_queries: 20
    search:
        prefer_self: 10
ingester:
    concurrent_flushes: 4
    max_block_duration: 30m
    max_block_bytes: 524288000
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        search:
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576

Features

[FEATURE] TraceQL Phase 1 support. A new query language for traces!
[FEATURE] Parquet backend is GA and default. By default Tempo will create Parquet blocks to enhance search performance
and to provide users with their data in an open format. See breaking changes above for a note on how to continue using older backends.

[FEATURE] Add generic forwarder and implement otlpgrpc forwarder #1775 (@Blinkuu)
New config options and example configuration:

# config.yaml
distributor:
  forwarders:
    - name: "otel-forwarder"
      backend: "otlpgrpc"
      otlpgrpc:
        endpoints: ['otelcol:4317']
        tls:
          insecure: true

# overrides.yaml
overrides:
  "example-tenant-1":
    forwarders: ['otel-forwarder']
  "example-tenant-2":
    forwarders: ['otel-forwarder']

Enhancements

[ENHANCEMENT] Add /status/usage-stats endpoint to show usage stats data #1782 (@electron0zero)
[ENHANCEMENT] Add TLS support to jaeger query plugin. #1999 (@rubenvp8510)
[ENHANCEMENT] Collect inspectedBytes from SearchMetrics #1975 (@electron0zero)
[ENHANCEMENT] Add zone awareness replication for ingesters. #1936 (@manohar-koukuntla)

# use the following fields in _config field of jsonnet config, to enable zone aware ingester
    multi_zone_ingester_enabled: false,
    multi_zone_ingester_migration_enabled: false,
    multi_zone_ingester_replicas: 0,
    multi_zone_ingester_max_unavailable: 25,

[ENHANCEMENT] Add new data-type aware searchtagvalues v2 api #1956 (@mdisibio)
[ENHANCEMENT] Filter namespace by cluster in tempo dashboards variables #1771 (@electron0zero)
[ENHANCEMENT] Exit early from sharded search requests #1742 (@electron0zero)
[ENHANCEMENT] Avoid running tempodb pool jobs with a cancelled context #1852 (@zalegrala)
[ENHANCEMENT] Add config flag to allow for compactor disablement for debug purposes #1850 (@zalegrala)
[ENHANCEMENT] Identify bloom that could not be retrieved from backend block #1737 (@AlexDHoffer)
[ENHANCEMENT] tempo: check configuration returns now a list of warnings #1663 (@frzifus)
[ENHANCEMENT] Make DNS address fully qualified to reduce DNS lookups in Kubernetes #1687 (@electron0zero)
[ENHANCEMENT] Return 200 instead of 206 when blocks failed is < tolerate_failed_blocks. #1725 (@joe-elliott)
[ENHANCEMENT] Add GOMEMLIMIT variable to compactor jsonnet and set the value to equal compactor memory limit. #1758 (@ie-pham)
[ENHANCEMENT] Add capability to configure the used S3 Storage Class #1697 (@amitsetty)
[ENHANCEMENT] cache: expose username and sentinel_username redis configuration options for ACL-based Redis Auth support #1708 (@jsievenpiper)
[ENHANCEMENT] metrics-generator: expose span size as a metric #1662 (@ie-pham)
[ENHANCEMENT] Set Max Idle connections to 100 for Azure, should reduce DNS errors in Azure #1632 (@electron0zero)
[ENHANCEMENT] Add PodDisruptionBudget to ingesters in jsonnet #1691 (@joe-elliott)
[ENHANCEMENT] Add a cli command to convert a block to the current parquet schema. #1707 (@joe-elliott)
[ENHANCEMENT] metrics-generator: filter out older spans before metrics are aggregated #1612 (@ie-pham)
[ENHANCEMENT] Add hedging to trace by ID lookups created by the frontend. #1735 (@mapno)
New config options and defaults:

query_frontend:
  trace_by_id:
    hedge_requests_at: 5s
    hedge_requests_up_to: 3

[ENHANCEMENT] Vulture now has improved distribution of the random traces it searches. #1763 (@rfratto)
[ENHANCEMENT] Add TLS support to the vulture #1874 (@zalegrala)
[ENHANCEMENT] metrics-generator: extract status_message field from spans #1786, #1794 (@stoewer)
[ENHANCEMENT] metrics-generator: handle collisions between user defined and default dimensions #1794 (@stoewer)
[ENHANCEMENT] metrics-generator: make intrinsic dimensions configurable and disable status_message by default #1960 (@stoewer)
[ENHANCEMENT] distributor: Log span names when distributor.log_received_spans.include_all_attributes is on #1790 (@suraciii)
[ENHANCEMENT] metrics-generator: truncate label names and values exceeding a configurable length #1897 (@kvrhdn)
[ENHANCEMENT] Convert last few Jsonnet alerts with per_cluster_label #2000 (@Whyeasy)
[ENHANCEMENT] New tenant dashboard #1901 (@mapno)
[ENHANCEMENT] Upgrade opentelemetry-proto submodule to v0.18.0 Internal types are updated to use scope instead of instrumentation_library.
This is a breaking change in trace by ID queries if JSON is requested. #1754 (@mapno)

Bugfixes

[BUGFIX] Stop distributors on Otel receiver fatal error#1887 (@rdooley)
[BUGFIX] New wal file separator '+' for the NTFS filesystem and backward compatibility with the old separator ':' #1700 (@kilian-kier)
[BUGFIX] Honor caching and buffering settings when finding traces by id #1697 (@joe-elliott)
[BUGFIX] Correctly propagate errors from the iterator layer up through the queriers #1723 (@joe-elliott)
[BUGFIX] Make multitenancy work with HTTP #1781 (@gouthamve)
[BUGFIX] Fix parquet search bug fix on http.status_code that may cause incorrect results to be returned #1799 (@mdisibio)
[BUGFIX] tempo-mixin: tweak dashboards to support metrics without cluster label present #1913 (@kvrhdn)
[BUGFIX] Fix docker-compose examples not running on Apple M1 hardware #1920 (@stoewer)
[BUGFIX] Don't persist tenants without blocks in the ingester#1947 (@joe-elliott)
[BUGFIX] Return more consistent search results by combining partial traces # (@mapno)

tempo - v1.5.0

Published by grafanabot about 2 years ago

Breaking Changes

(#1478) In order to build advanced visualization features into Grafana we have decided to change our spanmetric names to match OTel conventions. This way any functionality added to Grafana will work whether you use Tempo, Grafana Agent or the OTel Collector to generate metrics. Details in the span metrics documentation.
(#1556) Jsonnet users will need to specify ephemeral storage requests and limits for the metrics generator.
(#1481) Anonymous usage reporting has been added. Distributors and metrics generators will now require permissions to object storage equivalent to compactors and ingesters. This feature is enabled by default but can be disabled easily.
(#1558) Deprecated metrics tempodb_(gcs|s3|azure)_request_duration_seconds have been removed in favor of tempodb_backend_request_duration_seconds.

Changes

[CHANGE] metrics-generator: Changed added metric label instance to __metrics_gen_instance to reduce collisions with custom dimensions. #1439 (@joe-elliott)
[CHANGE] Don't enforce max_bytes_per_tag_values_query when set to 0. #1447 (@joe-elliott)
[CHANGE] Add new querier service in deployment jsonnet to serve /status endpoint. #1474 (@annanay25)
[CHANGE] Swapped out Google Cloud Functions serverless docs and build for Google Cloud Run. #1483 (@joe-elliott)
[CHANGE] BREAKING CHANGE Change spanmetrics metric names and labels to match OTel conventions. #1478 (@mapno)
Old metric names:

traces_spanmetrics_duration_seconds_{sum,count,bucket}

New metric names:

traces_spanmetrics_latency_{sum,count,bucket}

Additionally, default label span_status is renamed to status_code.

[CHANGE] Update to Go 1.18 #1504 (@annanay25)
[CHANGE] Change tag/value lookups to return partial results when reaching response size limit instead of failing #1517 (@mdisibio)
[CHANGE] Change search to be case-sensitive #1547 (@mdisibio)

[CHANGE] Relax Hedged request defaults for external endpoints. #1566 (@joe-elliott)

querier:
  search:
    external_hedge_requests_at: 4s    -> 8s
    external_hedge_requests_up_to: 3  -> 2

[CHANGE] BREAKING CHANGE Include emptyDir for metrics generator wal storage in jsonnet #1556 (@zalegrala)
Jsonnet users will now need to specify a storage request and limit for the generator wal.

    _config+:: {
      metrics_generator+: {
        ephemeral_storage_request_size: '10Gi',
        ephemeral_storage_limit_size: '11Gi',
      },
    }

[CHANGE] Two additional latency buckets added to the default settings for generated spanmetrics. Note that this will increase cardinality when using the defaults. #1593 (@fredr)
[CHANGE] Mark log_received_traces as deprecated. New flag is log_received_spans.
Extend distributor spans logger with optional features to include span attributes and a filter by error status. #1465 (@faustodavid)

Features

[FEATURE] Add parquet block format #1479 #1531 #1564 (@annanay25, @mdisibio)
[FEATURE] Add anonymous usage reporting, enabled by default. #1481 (@zalegrala)
BREAKING CHANGE As part of the usage stats inclusion, the distributor will also require access to the store. This is required so the distirbutor can know which cluster it should be reporting membership of.
[FEATURE] Include messaging systems and databases in service graphs. #1576 (@kvrhdn)

Enhancements

[ENHANCEMENT] Added the ability to have a per tenant max search duration. #1421 (@joe-elliott)
[ENHANCEMENT] metrics-generator: expose max_active_series as a metric #1471 (@kvrhdn)
[ENHANCEMENT] Azure Backend: Add support for authentication with Managed Identities. #1457 (@joe-elliott)
[ENHANCEMENT] Add metric to track feature enablement #1459 (@zalegrala)
[ENHANCEMENT] Added s3 config option insecure_skip_verify #1470 (@zalegrala)
[ENHANCEMENT] Added polling option to reduce issues in Azure blocklist_poll_jitter_ms #1518 (@joe-elliott)
[ENHANCEMENT] Add a config to query single ingester instance based on trace id hash for Trace By ID API. (1484)[https://github.com/grafana/tempo/pull/1484] (@sagarwala, @bikashmishra100, @ashwinidulams)
[ENHANCEMENT] Add blocklist metrics for total backend objects and total backend bytes #1519 (@ie-pham)
[ENHANCEMENT] Adds tempo_querier_external_endpoint_hedged_roundtrips_total to count the total hedged requests #1558 (@joe-elliott)
BREAKING CHANGE Removed deprecated metrics tempodb_(gcs|s3|azure)_request_duration_seconds in favor of tempodb_backend_request_duration_seconds. These metrics
have been deprecated since v1.1.
[ENHANCEMENT] Add tags option for s3 backends. This allows new objects to be written with the configured tags. #1442 (@stevenbrookes)
[ENHANCEMENT] metrics-generator: support per-tenant processor configuration #1434 (@kvrhdn)
[ENHANCEMENT] Include rollout dashboard #1456 (@zalegrala)
[ENHANCEMENT] Add SentinelPassword configuration for Redis #1463 (@zalegrala)
[ENHANCEMENT] Add support for time picker in jaeger query plugin. #1631 (@rubenvp8510)

Bugfixes

[BUGFIX] Fix nil pointer panic when the trace by id path errors. #1441 (@joe-elliott)
[BUGFIX] Update tempo microservices Helm values example which missed the 'enabled' key for thriftHttp. #1472 (@hajowieland)
[BUGFIX] Fix race condition in forwarder overrides loop. 1468 (@mapno)
[BUGFIX] Fix v2 backend check on span name to be substring #1538 (@mdisibio)
[BUGFIX] Fix wal check on span name to be substring #1548 (@mdisibio)
[BUGFIX] Prevent ingester panic "cannot grow buffer" #1258 (@mdisibio)
[BUGFIX] metrics-generator: do not remove x-scope-orgid header in single tenant modus #1554 (@kvrhdn)
[BUGFIX] Fixed issue where backend does not support root.name and root.service.name #1589 (@kvrhdn)
[BUGFIX] Fixed ingester to continue starting up after block replay error #1603 (@mdisibio)