• Fix CVE-2021-22133
• Make TRANSACTION_IGNORE_URLS dynamically configurable: #872

• module/apmsql: add tracingDriver.Unwrap method to get underlying driver #849
• module/apmgopgv10: add support for github.com/go-pg/pg/v10 #857 (thanks @macnibblet!)
• Enable central configuration of "sanitize_field_names" #856
• module/apmgrpc: set span destination context #861

• module/apmgoredisv8: introduce new package to support go-redis v8 (#780)
• module/apmhttp: introduce httptrace client option (#788)
• module/apmsql: add support for database/sql/driver.Validator (#791)
• Record sample rate on transactions and spans, propagate through tracestate (#804)
• module/apmredigo: change redigo dependency to v1.8.2 (#807)
• Deprecate IGNORE_URLS, replace with TRANSACTION_IGNORE_URLS (#811)
• Tracer.Close now waits for the transport goroutine to end before returning (#816)
• Relax Kubernetes pod UID discovery rules (#819)
• Add transaction and span outcome (#820)
• Add cloud metadata, configurable with ELASTIC_APM_CLOUD_PROVIDER (#823)
• Round ELASTIC_APM_SAMPLING_RATE with 4 digits precision (#828)
• module/apmhttp: implement io.ReaderFrom in wrapped http.ResponseWriter (#830)
• Fixed Transaction.Discard so that it sets TransactionData to nil (#836)
• module/apmsql/pgxv4: add support for pgx driver (#831 -- thanks @Deepak13245!)
• module/apmgormv2: add support for gorm.io (GORM v2) (#825 -- thanks @Deepak13245!)

• Add "recording" config option, to dynamically disable event recording (#737)
• Enable central configuration of "stack_frames_min_duration" and "stack_trace_limit" (#742)
• Implement "CloseIdleConnections" on the Elasticsearch RoundTripper (#750 - thanks @pebrc!)
• Fix apmot nil pointer dereference in Tracer.Inject (#763 - thanks @randomswdev!)

• Update cucumber/godog to 0.8.1 (#733)

• Fix segfault on 32-bit architectures (#728)

• Add span.context.destination.* #664
• transport: fix Content-Type for pprof data #679
• Add "tracestate" propagation #690
• Add support for API Key auth #698
• module/apmsql: report rows affected #700

• module/apmhttp: add WithClientRequestName option (#609 - thanks @lobotomist!)
• module/apmhttp: add WithPanicPropagation function (#611 - thanks @lobotomist!)
• module/apmgoredis: add Client.RedisClient (#613 - thanks @wins1908!)
• Introduce apm.TraceFormatter, for formatting trace IDs (#635)
• Report error cause(s), add support for errors.Unwrap (#638)
• Setting ELASTIC_APM_TRANSACTION_MAX_SPANS to 0 now disables all spans (#640)
• module/apmzerolog: add Writer.MinLevel (#641)
• Introduce SetLabel and deprecate SetTag (#642)
• Support central config for ELASTIC_APM_CAPTURE_BODY and ELASTIC_APM_TRANSACTION_MAX_SPANS (#648)
• module/apmgorm: sql.ErrNoRows is no longer reported as an error (#645 - thanks @vutung2311!)
• Server URL path is cleaned/canonicalizsed in order to avoid 301 redirects (#658)
• context.request.socket.remote_address now reports the peer address (#662)
• Experimental support for periodic CPU/heap profiling (#666)
• module/apmnegroni: introduce tracing Negroni middleware (#671 - thanks @earlzo!)
• Unescape hyphens in k8s pod UIDs when the systemd cgroup driver is used (#672)
• Read and propagate the standard W3C "traceparent" header (#674)

• Add Context.SetCustom (#581)
• Add support for extracting UUID-like container IDs (#577)
• Introduce transaction/span breakdown metrics (#564)
• Optimised HTTP request body capture (#592)
• Fixed transaction encoding to drop tags (and other context) for non-sampled transactions (#593)
• Introduce central config polling (#591)
• Fixed apmgrpc client interceptor, propagating trace context for non-sampled transactions (#602)

• Update opentracing-go dependency to v1.1.0
• Update HTTP routers to return " unknown route" if route cannot be matched (#486)
• module/apmchi: introduce instrumentation for go-chi/chi router (#495 - thanks @rliebz!)
• module/apmgoredis: introduce instrumentation for the go-redis/redis client (#505 - thanks @aspacca!)
• module/apmsql: exposed the QuerySignature function (#515)
• module/apmgopg: introduce instrumentation for the go-pg/pg ORM (#516 - thanks @macnibblet!)
• module/apmmongo: set minimum Go version to Go 1.10 (#522)
• internal/sqlscanner: bug fix for multi-byte rune handling (#535)
• module/apmgrpc: added WithServerRequestIgnorer server option (#531 - thanks @omrishtam!)
• Introduce ELASTIC_APM_GLOBAL_LABELS config (#539)
• module/apmgorm: register row_query callbacks (#532 - thanks @Vilsol!)
• Introduce ELASTIC_APM_STACK_TRACE_LIMIT config (#559)
• Include agent name/version and Go version in User-Agent (#560)
• Truncate error.culprit at 1024 chars (#561)

• Rename "metricset.labels" to "metricset.tags" (#438)
• Introduce ELASTIC_APM_DISABLE_METRICS to disable metrics with matching names (#439)
• module/apmelasticsearch: introduce instrumentation for Elasticsearch clients (#445)
• module/apmmongo: introduce instrumentation for the MongoDB Go Driver (#452)
• Introduce ErrorDetailer interface (#453)
• module/apmhttp: add CloseIdleConnectons and CancelRequest to RoundTripper (#457)
• Allow specifying transaction (span) ID via TransactionOptions/SpanOptions (#463)
• module/apmzerolog: introduce zerolog log correlation and exception-tracking writer (#428)
• module/apmelasticsearch: capture body for _msearch, template and rollup search (#470)
• Ended Transactions/Spans may now be used as parents (#478)
• Introduce apm.DetachedContext for async/fire-and-forget trace propagation (#481)
• module/apmechov4: add a copy of apmecho supporting echo/v4 (#477, thanks @1995parham!)

• Add "transaction.sampled" to errors (#410)
• Enforce license header in source files with go-licenser (#411)
• module/apmot: ignore "follows-from" span references (#414)
• module/apmot: report error log records (#415)
• Introduce ELASTIC_APM_CAPTURE_HEADERS to control HTTP header capture (#418)
• module/apmzap: introduce zap log correlation and exception-tracking hook (#426)
• type Error implements error interface (#399)
• Add "transaction.type" to errors (#433)
• Added instrumentation-specific Go modules (i.e. one for each package under apm/module) (#405)

This is a minor fix release, adding support for the new github.com/pkg/errors.Frame type (see #409). We continue to support github.com/pkg/errors v0.8.x, but will also support master/v0.9.0 when it comes.

Fixed a minor data race between Tracer.Active and Tracer.loop (#406). The only significant symptom of this race would have been race test failures, such as observed in apm-server.

Fixes a bug in the builtin metrics: CPU% metrics are now correctly reported in the range [0,1], no longer scaled by 100.

Major changes / new instrumentation modules

module/apmlogrus: introduce Logrus error-tracking hook, and log correlation (#381)

We have introduced module/apmlogrus, which provides two integrations with Logrus: a logrus.Hook implementation which sends error log records to Elastic APM, and a function for adding trace, transaction, and span ID fields to log records.

module/apmbeego: introduce Beego instrumentation module (#386)

This new instrumentation module provides tracing middleware for the Beego web framework. This currently covers the web requests, and not beego/orm.

Stop pooling Transaction/Span/Error, introduce internal pooled objects (#319)

Prior to this release, Transaction, Span and Error objects returned by StartTransaction, StartSpan and NewError respectively were pooled to minimise memory allocation overhead, at the cost of having to handle the objects with care; once a transaction was ended, it would no longer be valid for use. The intention was to educate users to handle these objects carefully.

It turned out to be too easy to access ended transactions, particularly when used with context objects with cancellation. For example, an instrumented HTTP handler might pass its context onto another goroutine; some time later that goroutine attempts to start a span using the context, but the request has timed out and accordingly ended the transaction.

We have changed how pooling works, so that it operates internally within the Transaction, Span, and Error objects. This means a slightly higher allocation overhead, but it is now safe to access an ended Transaction, Span, or Error.

Bug fixes

apm.StartSpanOptions fixed to stop ignoring options (#326)

apm.StartSpanOptions was ignoring the options provided, which meant, for example, you could not provide the start timestamp. Many thanks to @ChristophPech for the fix!

module/apmsql: don't report driver.ErrBadConn, context.Canceled (#346, #348)

module/apmsql will no longer report "bad connection" or "context canceled" errors.

"Bad connection" errors occur when a pooled database connection becomes stale/unusable, which can occur in normal usage. Internally, database/sql will attempt to fetch another pooled connection, and finally attempt to make a new connection, under these circumstances. Since these are not indicative of application errors, and are not actionable, we will stop reporting them to the user.

Similarly, we were reporting "context canceled" errors from module/apmsql, as these may be intentional and not related to any application logic error.

Other changes

• ELASTIC_APM_SERVER_CERT enables server certificate pinning (#325)
• Add Docker container ID to metadata (#330)
• Add Kubernetes pod info to metadata (#342)
• Added distributed trace context propagation to apmgrpc (#335)
• Introduce Span.Subtype, Span.Action (#332)
• Enable metrics collection with default interval of 30s (#322)
• Added ErrorLogRecord.Error field, for associating an error value with a log record (#380)
• module/apmhttp: report status code for client spans (#388)

Major changes

The Go agent is now generally available. This version requires APM Server v6.5 or later.

• The canonical import path is now "go.elastic.co/apm", and top-level package name is "apm" (#202)
• The v2 intake protocol is now being used, and support for distributed tracing added (#180)
• We've added support for multiple APM Server URLs with failover (#233)
• All HTTP request/response headers are now reported, with configurable filtering (#280)
• ELASTIC_APM_LOG_FILE and ELASTIC_APM_LOG_LEVEL introduced for logging agent behaviour (#313)
• We've added support for mixing OpenTracing spans with native transactions/spans (#235)

New instrumentation modules

• module/apmredigo: introduce gomodule/redigo instrumentation (#248)
• module/apmrestful: introduce emicklei/go-restful instrumentation (#270)

Other changes

• Unexported Transaction.Timestamp and Span.Timestamp (#207)
• Dropped Context.SetCustom (#284)
• Add support for span tags (#213)
• Add jitter (+/-10%) to backoff on transport error (#212)
• Require units for size configuration (#223)
• Require units for duration configuration (#211)
• Drop SetHTTPResponseHeadersSent and SetHTTPResponseFinished methods from Context (#238)
• Stop setting custom context (gin.handler) in apmgin (#238)
• Set response context in errors reported by web modules (#238)
• Update Sampler interface to take TraceContext (#243)
• Truncate SQL statements to a maximum of 10000 chars, all other strings to 1024 (#244, #276)
• Add leading slash to URLs in transaction/span context (#250)
• Add Transaction.Context method for setting framework (#252)
• Timestamps are now reported as usec since epoch, spans no longer use "start" offset (#257)
• Fix panic handling in web instrumentations (#273)
• Migrate internal/fastjson to go.elastic.co/fastjson (#275)
• Reuse memory for tags (#286)
• Return a more helpful error message when /intake/v2/events 404s, to detect old servers (#290)
• Implement test service for w3c/distributed-tracing test harness (#293)
• End HTTP client spans on response body closure (#289)
• module/apmgrpc requires Go 1.9+ (#300)
• Invalid tag key characters are replaced with underscores (#308)
• ELASTIC_APM_SANITIZE_FIELD_NAMES and ELASTIC_APM_IGNORE_URLS now use wildcard matching (#260)

Fixed premature Span.End() in apmgorm callback, causing a data-race with captured errors (#229)

Fixed a bug causing error stacktraces and culprit to sometimes not be set (#204)

• ELASTIC_APM_SERVER_URL now defaults to "http://localhost:8200" (#122)
• Transport.SetUserAgent method added, enabling the User-Agent to be set programatically (#124)
• Inlined functions are now properly reported in stacktraces (#127)
• Support for the experimental metrics API added (#94)
• module/apmsql: SQL is parsed to generate more useful span names (#129)
• Basic vgo module added (#136)
• module/apmhttprouter: added a wrapper type for httprouter.Router to simplify adding routes (#140)
• Add Transaction.Context methods for setting user IDs (#144)
• module/apmgocql: new instrumentation module, providing an observer for gocql (#148)
• Add ELASTIC_APM_SERVER_TIMEOUT config (#157)
• Add ELASTIC_APM_IGNORE_URLS config (#158)
• module/apmsql: fix a bug preventing errors from being captured (#160)
• Introduce Tracer.StartTransactionOptions, drop variadic args from Tracer.StartTransaction (#165)
• module/apmgorm: introduce GORM instrumentation module (#169, #170)
• module/apmhttp: record outgoing request URLs in span context (#172)
• module/apmot: introduce OpenTracing implementation (#173)

In the next release we will be changing the canonical import path prefix to "go.elastic.co/apm", and the top-level package name to "apm". Please consider vendoring if this will break your builds.