6.26.0 ships the same features as 7.26.0 except for the Python versions it supports.

Please refer to the 7.26.0 changelog

Prelude

Release on: 2021-03-02

• Please refer to the 7.26.0 tag on
  integrations-core
  for the list of changes on the Core Checks

Upgrade Notes

• forwarder_retry_queue_payloads_max_size takes precedence over the
  deprecated forwarder_retry_queue_max_size. If
  forwarder_retry_queue_max_size is not set, you are not affected by
  this change. If forwarder_retry_queue_max_size is set, but
  forwarder_retry_queue_payloads_max_size is not set, the Agent uses
  forwarder_retry_queue_max_size * 2MB as the value of
  forwarder_retry_queue_payloads_max_size. It is recommended to
  configure forwarder_retry_queue_payloads_max_size and remove
  forwarder_retry_queue_max_size from the Agent configuration.
• Docker image: remove Docker volumes for /etc/datadog-agent and
  /tmp as it prevents to inherit from Datadog Agent image. It was
  originally done to allow read-only rootfs on Kubernetes, so in order
  to continue supporting this feature, relevant volumes are created in
  newer Kubernetes manifest or Helm chart >= 2.6.9

New Features

• APM: Support SQL obfuscator feature to replace consecutive digits in
  table names.
• APM: Add an endpoint to receive apm stats from tracers.
• Agent discovers by itself which container AD features and checks
  should be scheduled without having to specify any configuration.
  This works for Docker, Containerd, ECS/EKS Fargate and Kubernetes.
  It also allows to support heterogeneous nodes with a single
  configuration (for instance a Kubernetes DaemonSet could cover nodes
  running Containerd and/or Docker - activating relevant configuration
  depending on node configuration). This feature is activated by
  default and can be de-activated by setting environment variable
  AUTCONFIG_FROM_ENVIRONMENT=false.
• Adds a new agent command stream-logs to stream the logs being
  processed by the agent. This will help diagnose issues with log
  integrations.
• Submit host tags with log events for a configurable time duration to
  avoid potential race conditions where some tags might not be
  available to all backend services on freshly provisioned instances.
• Added no_proxy_nonexact_match as a configuration setting which
  allows non-exact URL and IP address matching. The new behavior uses
  the go http proxy function documented here
  https://godoc.org/golang.org/x/net/http/httpproxy#Config If the
  new behavior is disabled, a warning will be logged if a url or IP
  proxy behavior will change in the future.
• The Quality of Service of pods is now collected and sent to the
  orchestration endpoint.
• Runtime-security new command line allowing to trigger a process
  cache dump..
• Support Prometheus Autodiscovery for Kubernetes Pods.
• The core agent now exposes a gRPC API to expose tags to the other
  agents. The following settings are now introduced to allow each of
  the agents to use this API (they all default to false):
  • apm_config.remote_tagger
  • logs_config.remote_tagger
  • process_config.remote_tagger
• New perf map usage metrics.
• Add unofficial arm64 support to network tracer in system-probe.
• system-probe: Add optional runtime compilation of eBPF programs.

Enhancement Notes

• APM: Sublayer metrics (trace.<SPAN_NAME>.duration and
  derivatives) computation in agent can be disabled with feature flags
  disable_sublayer_spans, disable_sublayer_stats. Reach out to
  support with questions about this metric.
• APM: Automatically activate non-local trafic (i.e. listening on
  0.0.0.0) for APM in containerized environment if no explicit setting
  is set (bind_host or apm_non_local_traffic)
• APM: Add a tag allowing trace metrics from synthetic data to be
  aggregated independently.
• Consider the task level resource limits if the container level
  resource limits aren't defined on ECS Fargate.
• Use the default agent transport for host metadata calls. This allows
  usage of the config no_proxy setting for host metadata calls. By
  default cloud provider IPs are added to the transport's no_proxy
  list. Added config flag use_proxy_for_cloud_metadata to disable
  this behavior.
• GOMAXPROCS is now set automatically to match the allocated CPU
  cgroup quota. GOMAXPROCS can now also be manually specified and
  overridden in millicore units. If no quota or GOMAXPROCS value is
  set it will default to the original behavior.
• Added --flare flag to jmx (list|collect) commands to save check
  results to the agent logs directory. This enables flare to pick up
  jmx command results.
• Kubernetes events are now tagged with kube_service,
  kube_daemon_set, kube_job and kube_cronjob. Note: Other object
  kinds are already supported (pod_name, kube_deployment,
  kube_replica_set).
• Expose logs agent pipeline latency in the status page.
• Individual DEB packages are now signed.
• Docker container, when not running in a Kubernetes environment may
  now be tailed from their log file. The Agent must have read access
  to /var/lib/docker/containers and Docker containers must use the
  JSON logging driver. This new option can be activated using the new
  configuration flag logs_config.docker_container_use_file.
• File tailing from a kubernetes pod annotation is now supported. Note
  that the file path is relative to the Agent and not the
  pod/container bearing the annotation.

Bug Fixes

• APM: Group arrays of consecutive '?' identifiers
• Fix agent panic when UDP port is busy and dogstatsd_so_rcvbuf is
  configured.
• Fix a bug that prevents from reading the correct container resource
  limits on ECS Fargate.
• Fix parsing of dogstatsd event strings that contained negative
  lengths for event title and/or event text length.
• Fix sending duplicated kubernetes events.
• Do not invoke the secret backend command (if configured) when the
  agent health command/agent container liveness probe is called.
• Fix parsing of CLI options of the agent health command

Other Notes

• Bump gstatus version from 1.0.4 to 1.0.5.
• JMXFetch upgraded from
  0.41.0 to
  0.42.0

6.25.1 ships the same features as 7.25.1 except for the Python versions it supports.

Please refer to the 7.25.1 changelog.

Prelude

Release on: 2021-01-26

Bug Fixes

• Fix "fatal error: concurrent map read and map write" due to reads of
  a concurrently mutated map in inventories.payload.MarshalJSON()
• Fix an issue on arm64 where non-gauge metrics from Python checks
  were treated as gauges.
• On Windows, fixes uninstall/upgrade problem if core agent is not
  running but other services are.
• Fix NPM UDP destination address decoding when source address ends
  with .8 during offset guessing.
• On Windows, changes the password generating algorithm to have a
  minimum length of 16 and a maximum length of 20 (from 12-18).
  Improves compatibility with environments that have longer password
  requirements.

6.25.0 ships the same features as 7.25.0 except for the Python versions it supports.

Please refer to the 7.25.0 changelog.

Prelude

Release on: 2021-01-14

• Please refer to the 7.25.0 tag on
  integrations-core
  for the list of changes on the Core Checks

New Features

• Add com.datadoghq.ad.tags container
  auto-discovery label in AWS Fargate environment.
• Package the gstatus command line tool binary for GlusterFS
  integration metric collection.
• Queried domain can be tracked as part of DNS stats
• APM: The agent is now able to skip top-level span computation in
  cases when the client has marked them by means of the
  Datadog-Client-Computed-Top-Level header.
• APM: The maximum allowed key length for tags has been increased from
  100 to 200.
• APM: Improve Oracle SQL obfuscation support.
• APM: Added support for Windows pipes. To enable it, set the pipe
  path using DD_APM_WINDOWS_PIPE_NAME. For more details check PR
  #6615
• Pause containers are now detected and auto excluded based on the
  io.kubernetes container labels.
• APM: new datadog_agent.obfuscate_sql_exec_plan
  function exposed to python checks to enable obfuscation of
  json-encoded SQL Query Execution Plans.
• APM: new obfuscate_sql_values
  option in apm_config.obfuscation
  enabling optional obfuscation of SQL queries contained in JSON data
  collected from some APM services (ES & Mongo)

Enhancement Notes

• Support the ddog-gov.com site option in the Windows GUI installer.
• Adds config setting for ECS metadata endpoint client timeout
  (ecs_metadata_timeout), value in milliseconds.
• Add loader config to allow selecting
  specific loader at runtime. This config is available at init_config and instances level.
• Added additional container information to the status page when
  collect all container logs is enabled in agent status.
• On Windows, it will no longer be required to supply the ddagentuser
  name on upgrade. Previously, if a non-default or domain user was
  used, the same user had to be provided on subsequent upgrades.
• Added --flare flag to agent check to save check results to the
  agent logs directory. This enables flare to pick up check results.
• Added new config option for JMXFetch collect_default_jvm_metrics
  that enables/disables default JVM metric collection.
• Allow empty message for DogStatsD events (e.g. "_e{10,0}:test
  title|")
• Expires the cache key for availability of ECS metadata endpoint used
  to fetch EC2 resource tags every 5 minutes.
• Data coming from kubernetes pods now have new kube_ownerref_kind
  and kube_ownerref_name tags for each of the pod's OwnerRef
  property, indicating its Kind and Name, respectively.
• We improved the way Agents get the Kubernetes cluster ID from the
  Cluster Agent. It used to be that the cluster agent would create a
  configmap which had to be mounted as an env variable in the agent
  daemonset, blocking the process-agent from starting if not found.
  Now the process-agent will start, only the Kubernetes Resources
  collection will be blocked.
• Events sent by the runtime security agent to the backend use a new
  taxonomy.
• Scrub container args as well for orchestrator explorer.
• Support custom autodiscovery identifiers on Kubernetes using the
  ad.datadoghq.com/<container_name>.check.id
  pod annotation.
• The CPU check now collects system-wide context switches on Linux.
• Add --table option to agent check command to output results in
  condensed tabular format instead of JSON.
• APM: improve performance by changing the msgpack serialization
  implementation.
• APM: improve the performance of the msgpack deserialization for the
  v0.5 payload format.
• APM: improve performance of trace processing by removing some heap
  allocations.
• APM: improve sublayer computation performance by reducing the number
  of heap allocations.
• APM: improved stats computation performance by removing some string
  concatenations.
• APM: improved trace signature computation by avoiding heap
  allocations.
• APM: improve stats computation performance.
• Update from alpine:3.10 to alpine:3.12 the base image in Dogstatsd's
  Dockerfiles.

Deprecation Notes

• APM: remove the already deprecated apm_config.extra_aggregators
  config option.

Bug Fixes

• Fix macos dlopen failures by ensuring
  cmake preserves the required runtime search path.
• Fix memory leak on check unscheduling, which could be noticeable for
  checks submitting large amounts of metrics/tags.
• Exclude pause containers using the cdk/pause.* image.
• Fixed missing some Agent environment variables in the flare
• Fix a bug that prevented the logs Agent from discovering the correct
  init containers source and service on Kubernetes.
• The logs agent now uses the container image name as logs source
  instead of kubernetes when a standard
  service value was defined for the container.
• Fixes panic on concurrent map access in Kubernetes metadata tag
  collector.
• Fixed a bug that could potentially cause missing container tags for
  check metrics.
• Fix a potential panic on ECS when the ECS API is returning empty
  docker ID
• Fix systemd check id to handle multiple instances. The fix will make
  check id unique for each different instances.
• Fix missing tags on pods that were not seen with a running container
  yet.
• Fix snmp listener subnet loop to use correct subnet pointer when
  creating snmpJob object.
• Upgrade the embedded pip version to 20.3.3 to get a newer vendored
  version of urllib3.

Other Notes

• The Agent, Logs Agent and the system-probe are now compiled with Go
  1.14.12
• Upgrade embedded libkrb5 Kerberos library to v1.18.3. This version
  drops support for the encryption types marked as "weak" in the docs
  of the
  library

6.24.1 ships the same features as 7.24.1 except for the Python versions it supports.

Please refer to the 7.24.1 changelog.

7.24.1

Bug Fixes

• Fix a bug when parsing the current version of an integration that prevented
  upgrading from an alpha or beta prerelease version.

• During a domain installation in a child domain, the Windows installer can now use a user from a parent domain.

• The Datadog Agent had a memory leak where some tags would be collected but
  never cleaned up after their entities were removed from a Kubernetes
  cluster due to their IDs not being recognized. This has now been fixed, and
  all tags are garbage collected when their entities are removed.

Other Notes

• Updated the shipped CA certs to latest (2020-12-08)

Release Notes

1.10.0

Prelude

Released on: 2020-12-10
Pinned to datadog-agent v7.24.0: CHANGELOG.

New Features

• Add a new command 'datadog-cluster-agent health' to show the cluster
  agent's health, similar to the already existing agent health.

• collect node information for the orchestrator explorer

• Fill DatadogMetric AutoscalerReferences field to ease usage/investigation of DatadogMetrics

• The Cluster Agent can now collect stats from Cluster Level Check runners
  to optimize its dispatching logic and rebalance the scheduled checks.

• Allow providing custom tags to orchestrator resources.

Enhancement Notes

• Add new configuration parameter to allow 'GroupExec' permission on the secret-backend command.
  The new parameter ('secret_backend_command_allow_group_exec_perm') is now enabled by default in the cluster-agent image.

• Add resolve option to endpoint checks through new annotation ad.datadoghq.com/endpoints.resolve. With ip value, it allows endpoint checks to target static pods

• Expose metrics for the cluster level checks advanced dispatching.

Bug Fixes

• Fix 'readsecret.sh' permission in Cluster-Agent dockerfiles that removes other permission.

• Fix issue in Cluster Agent when using external metrics without DatadogMetrics where multiple HPAs using the same metricName + Labels would prevent all HPAs (except 1st one) to get values from Datadog

• Ensure that leader election runs if orchestrator_explorer and leader_election are enabled.

• Rename node role tag from "node_role" to "kube_node_role" in orchestrator_explorer collection.

6.24.0 ships the same features as 7.24.0 except for the Python versions it supports.

Please refer to the 7.24.0 changelog.

7.24.0

Prelude

Release on: 2020-12-03

• Please refer to the 7.24.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• tcp_queue_length check: the previous metrics reported by this check (tcp_queue.rqueue.size, tcp_queue.rqueue.min, tcp_queue.rqueue.max, tcp_queue.wqueue.size, tcp_queue.wqueue.min, tcp_queue.wqueue.max) were generating too much data because there was one time series generated per TCP connection.
  Those metrics have been replaced by tcp_queue.read_buffer_max_usage_pct, tcp_queue.write_buffer_max_usage_pct which are aggregating all the connections of a container.
  These metrics are reporting the maximum usage in percent (amount of data divided by the queue capacity) of the busiest buffer.
  Additionally, only_count_nb_context option from the tcp_queue_length check configuration has been removed and will be ignored from now on.

New Features

• Added new configuration flag,
  system_probe_config.enable_conntrack_all_namespaces,
  false by default. When set to true, this will allow system
  probe to monitor conntrack entries (for NAT info) in all
  namespaces that are peers of the root namespace.

• Added JMX version and java runtime version to agent status page

• kubernetes_pod_annotations_as_tags (DD_KUBERNETES_POD_ANNOTATIONS_AS_TAGS) now support regex wildcards:
  '{"*":"<PREFIX>_%%annotation%%"}' can be used as value to collect all pod annotations as tags.
  kubernetes_node_labels_as_tags (DD_KUBERNETES_NODE_LABELS_AS_TAGS) now support regex wildcards:
  '{"*":"<PREFIX>_%%label%%"}' can be used as value to collect all node labels as tags.
  Note: kubernetes_pod_labels_as_tags (DD_KUBERNETES_POD_LABELS_AS_TAGS) supports this already.

• Listening for conntrack updates from all network namespaces
  (system_probe_config.enable_conntrack_all_namespaces flag) is now turned
  on by default

Enhancement Notes

• Expand pause container image filter

• Adds misconfig check for hidepid=2 option on proc mount.

• It's possible to ignore auto_conf.yaml configuration files using ignore_autoconf or DD_IGNORE_AUTOCONF.
  Example: DD_IGNORE_AUTOCONF="redisdb kubernetes_state"

• APM: The trace-agent now automatically sets the GOMAXPROCS value in
  Linux containers to match allocated CPU quota, as opposed to the matching
  the entire node's quota.

• APM: Lowered CPU usage when using analytics.

• APM: Move UTF-8 validation from the span normalizer to the trace decoder, which reduces the number of times each distinct string will be validated to once, which is beneficial when the v0.5 trace format is used.

• Add the config forwarder_retry_queue_payloads_max_size which defines the
  maximum size in bytes of all the payloads in the forwarder's retry queue.

• When enabled, JMXFetch now logs to its own log file. Defaults to jmxfetch.log
  in the default agent log directory, and can be configured with jmx_log_file.

• Added UDS support for JMXFetch
  JMXFetch upgraded to 0.40.3

• dogstatsd_mapper_profiles may now be defined as an environment variable DD_DOGSTATSD_MAPPER_PROFILES formatted as JSON

• Add orchestrator explorer related section into DCA Status

• Added byte count per log source and display it on the status page.

• APM: refactored the SQL obfuscator to be significantly more efficient.

Deprecation Notes

• IO check: device_blacklist_re has been deprecated in favor of device_exclude_re.

• The config options tracemalloc_whitelist and tracemalloc_blacklist have been
  deprecated in favor of tracemalloc_include and tracemalloc_exclude.

Bug Fixes

• APM: Fix a bug where non-float64 numeric values in apm_config.analyzed_spans
  would disable this functionality.

• Disable stack protector on system-probe to make it buildable on the environments which stack protector is enabled by default.

  Some linux distributions like Alpine Linux enable stack protector by default which is not available on eBPF.

• Fix a panic in containerd if retrieved ociSpec is nil

• Fix random panic in Kubelet searchPodForContainerID due to concurrent modification of pod.Status.AllContainers

• Add retries to Kubernetes host tags retrievals, minimize the chance of missing/changing host tags every 30mins

• Fix rtloader build on strict posix environment, e.g. musl libc on Alpine Linux.

• Allows system_probe to be enabled without enabling network performance monitoring.

  Set network_config.enabled=false in your system-probe.yaml when running the system-probe without networks enabled.

• Fixes truncated output for status of compliance checks in Security Agent.

• Under some circumstances, the Agent would delete all tags for a workload if
  they were collected from different sources, such as the kubelet and docker,
  but deleted from only one of them. Now, the agent keeps track of tags per
  collector correctly.

Other Notes

• The utilities provided by the sysstat package have been removed: the iostat,
  mpstat, pidstat, sar, sadf, cifsiostat and nfsiostat-sysstat
  binaries have been removed from the packaged Agent. This has no effect on the
  behavior of the Agent and official integrations, but your custom checks may be
  affected if they rely on these embedded binaries.

• Activate security-agent service by default in the Linux packages of the Agent (RPM/DEB). The security-agent won't be started if the file /etc/datadog-agent/security-agent.yaml does not exist.

Release Notes

1.9.1

Prelude

Released on: 2020-10-21
Pinned to datadog-agent v7.23.1: CHANGELOG

Bug Fixes

• Support of secrets in JSON environment variables, added in 7.23.0, is
  reverted due to a side effect (e.g. a string value of "-" would be loaded as a list). This
  feature will be fixed and added again in a future release.

Release Notes

7.23.1

Prelude

Release on: 2020-10-21

Bug Fixes

• The ec2_prefer_imdsv2 parameter was ignored when fetching EC2 tags from the metadata endpoint. This fixes a misleading warning log that was logged even when ec2_prefer_imdsv2 was left disabled in the Agent configuration.
• Support of secrets in JSON environment variables, added in 7.23.0, is reverted due to a side effect (e.g. a string value of "-" would be loaded as a list). This feature will be fixed and added again in a future release.
• The Windows installer can now install on domains where the domain name is different from the Netbios name.

Release Notes

1.9.0

Prelude

Pinned to datadog-agent v7.23.0: CHANGELOG.

New Features

• Collect the node and cluster resource in Kubernetes for the Orchestrator Explorer (#6297).
• Add resolve option to the endpoint checks (#5918).
• Add health command (#6144).
• Add options to configure the External Metrics Server (#6406).

Enhancement Notes

• Fill DatadogMetric AutoscalerReferences field to ease usage/investigation of DatadogMetrics (#6367).
• Only run compliance checks on the Cluster Agent leader (#6311).
• Add orchestrator_explorer configuration to enable the cluster-id ConfigMap creation and Orchestrator Explorer instanciation (#6189).

Bug Fixes

• Fix transformer for gibiBytes and gigaBytes (#6437).
• Fix cluster-agent commands to allow executing the readsecret.sh script for the secret backend feature (#6445).
• Fix issue with External Metrics when several HPAs use the same query (#6412).

6.23.1 ships the same features as 7.23.1 except for the Python versions it supports.

Please refer to the 7.23.1 changelog.

Release Notes

7.23.0

Prelude

Release on: 2020-10-06

• Please refer to the 7.23.0 tag on
  integrations-core
  for the list of changes on the Core Checks

Upgrade Notes

• Network monitoring: enable DNS stats collection by default.

New Features

• APM: Decoding errors reported by the datadog.trace-agent.receiver.error and
  datadog.trace_agent.normalizer.traces_dropped
  contain more detailed reason tags in case of EOFs and timeouts.
• Running the agent flare with the -p flag now includes profiles for
  the trace-agent.
• APM: An SQL query obfuscation cache was added under the feature flag
  DD_APM_FEATURES=sql_cache. In most cases where SQL queries are
  repeated or prepared, this can significantly reduce CPU work.
• Secrets handles are not supported inside JSON value set through
  environment variables. For example setting a secret in a list DD_FLARE_STRIPPED_KEYS='["ENC[auth_token_name]"]'
  datadog-agent run
• Add basic support for UTF16 (BE and LE) encoding. It should be
  manually enabled in a log configuration using encoding: utf-16-be
  or encoding: utf-16-le other values are unsupported and ignored by
  the agent.

Enhancement Notes

• Add new configuration parameter to allow 'GroupExec' permission on
  the secret-backend command. Set to 'true' the new parameter
  'secret_backend_command_allow_group_exec_perm' to activate it.
• Add a map from DNS rcode to count of replies received with that
  rcode
• Enforces a size limit of 64MB to uncompressed sketch payloads
  (distribution metrics). Payloads above this size will be split into
  smaller payloads before being sent.
• APM: Span normalization speed has been increased by 15%.
• Improve the kubelet check error reporting in the output of
  agent status in the case where the agent cannot properly connect
  to the kubelet.
• Add space_id, space_name, org_id and org_name as tags to both autodiscovered
  containers as well as checks found through autodiscovery on Cloud
  Foundry/Tanzu.
• Improves compliance check status view in the security-agent status
  command.
• Include compliance benchmarks from
  github.com/DataDog/security-agent-policies in the Agent packages and
  the Cluster Agent image.
• Windows Docker image is now based on Windows Server Nano instead of
  Windows Server Core.
• Allow sending the GCP project ID under the project_id: host tag
  key, in addition to the project: host tag key, with the
  gce_send_project_id_tag config setting.
• Add kubeconfig to GCE excluded host
  tags (used on GKE)
• The cluster name can now be longer than 40 characters, however the
  combined length of the host name and cluster name must not exceed
  254 characters.
• When requesting EC2 metadata, you can use IMDSv2 by turning on a new
  configuration option (ec2_prefer_imdsv2).
• When tailing logs from container in a kubernetes environment long
  lines (>16kB usually) that got split by the container runtime
  (docker & containerd at least) are now reassembled pending they do
  not exceed the upper message length limit (256kB).
• Move the cluster-id ConfigMap creation, and Orchestrator Explorer
  controller instantiation behind the orchestrator_explorer config
  flag to avoid it failing and generating error logs.
• Add caching for sending kubernetes resources for live containers
• Agent log format improvement: logs can have kv-pairs as context to
  make it easier to get all logs for a given context Sample:
  2020-09-17 12:17:17 UTC | CORE | INFO |
  (pkg/collector/runner/runner.go:327 in work) | check:io | Done
  running check
• The CRI check now supports container exclusion based on container
  name, image and kubernetes namespace.
• Added a network_config config to the system-probe that allows the
  network module to be selectively enabled/disabled. Also added a
  corresponding DD_SYSTEM_PROBE_NETWORK_ENABLED env var. The
  network module will only be disabled if the network_config exists
  and has enabled set to false, or if the env var is set to false. To
  maintain compatibility with previous configs, the network module
  will be enabled in all other cases.
• Log a warning when a log file is rotated but has not finished
  tailing the file.
• The NTP check now uses the cloud provider's recommended NTP servers
  by default, if the Agent detects that it's running on said cloud
  provider.

Deprecation Notes

• process_config.orchestrator_additional_endpoints
  and process_config.orchestrator_dd_url are
  deprecated in favor of: orchestrator_explorer.orchestrator_additional_endpoints
  and orchestrator_explorer.orchestrator_dd_url.

Bug Fixes

• Fixed an issue where the Datadog Agent would improperly filter all
  remaining traces in a payload after a trace matching an
  ignore_resources filter was matched.
• Allow agent integration install to
  work even if the datadog agent configuration file doesn't exist.
  This is typically the case when this command is run from a
  Dockerfile in order to build a custom image from the datadog
  official one.
• Implement variable interpolation in the tagger when inferring the
  standard tags from the DD_ENV, DD_SERVICE and DD_VERSION
  environment variables
• Fix a bug that was causing not picking checks and logs for
  containers targeted by container-image-based autodiscovery. Or
  picking checks and logs for containers that were not targeted by
  container-image-based autodiscovery. This happened when several
  image names were pointing to the same image digest.
• APM: Allow digits in SQL literal identifiers (e.g. 1sad123jk)
• Fixes an issue with not always reporting ECS Fargate task_arn tag
  due to a race condition in the tag collector.
• The SUSE SysVInit service now correctly starts the Agent as the
  dd-agent user instead of root.
• APM: Allow double-colon operator in SQL obfuscator.
• UDP packets can be sent in two ways. In the "connected" way, a connect call is made first to assign the
  remote/destination address, and then packets get sent with the send function or sendto function with destination address
  set to NULL. In the "unconnected" way, packets get sent using sendto function with a non NULL destination
  address. This fix addresss a bug where network stats were not being
  generated for UDP packets sent using the "unconnected" way.
• Fix the Windows systray not appearing sometimes (bug introduced with
  6.20.0).
• The Chocolatey package now uses a fixed URL to the MSI installer.
• Fix logs tagging inconsistency for restarted containers.
• On macOS, in Agent v6, the unversioned python binaries in
  /opt/datadog-agent/embedded/bin (example: python, pip) now
  correctly point to the Python 2 binaries.
• Fix truncated cgroup name on copy with bpf_probe_read_str in OOM
  kill and TCP queue length checks.
• Use double-precision floats for metric values submitted from Python
  checks.
• On Windows, the ddtray executable now has a digital signature
• Updates the logs package to get the short image name from Kubernetes
  ContainerSpec, rather than ContainerStatus. This works around a
  known issue where the image name in the ContainerStatus may be
  incorrect.
• On Windows, the Agent now responds to control signals from the OS
  and shuts down gracefully. Coincidentally, a Windows Agent Container
  will now gracefully stop when receiving the stop command.

Other Notes

• All Agents binaries are now compiled with Go 1.14.7
• JMXFetch upgraded from
  0.38.2 to
  0.39.1
• Move the orchestrator related settings process_config.orchestrator_additional_endpoints
  and process_config.orchestrator_dd_url into
  the orchestrator_explorer section.

6.22.1 ships the same features as 7.22.1 except for the Python versions it supports.

Please refer to the 7.22.1 changelog.

Prelude

Release on: 2020-09-17

• Please refer to the 7.22.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Define a default logs file (security-agent.log) for the security-agent.
• Fix segfault when listing Garden containers that are in error state.
• Do not activate security-agent service by default in the Linux packages of the Agent (RPM/DEB).
  The security-agent was already properly starting and exiting if not activated in configuration.

6.22.0 ships the same features as 7.22.0 except for the Python versions it supports.

Please refer to the 7.22.0 changelog.

Release Notes

7.22.0

Prelude

Release on: 2020-08-25

• Please refer to the 7.22.0 tag on
  integrations-core
  for the list of changes on the Core Checks

New Features

• Implements agent-side compliance rule evaluation in security agent
  using expressions.
• Add IO operations monitoring for Docker check
  (docker.io.read/write_operations)
• Track TCP connection churn on system-probe
• The new Runtime Security Agent collects file integrity monitoring
  events. It is disabled by default and only available for Linux for
  now.
• Make security-agent part of automatically started agents in
  RPM/DEB/etc. packages (will do nothing and exit 0 by default)
• Add support for receiving and processing SNMP traps, and forwarding
  them as logs to Datadog.
• APM: A new trace ingestion endpoint was introduced at /v0.5/traces which
  supports a more compact payload format, greatly improving resource usage.
  The spec for the new wire format can be viewed at here.
  Tracers supporting this change, will automatically use the new endpoint.

Enhancement Notes

• Adds a gauge for system.mem.slab_reclaimable. This is part
  of slab memory that might be reclaimed (i.e. caches). Datadog 7.x
  adds SReclaimable memory, if
  available on the system, to the system.mem.cached gauge by default. This
  may lead to inconsistent metrics for clients migrating from Datadog
  5.x, where system.mem.cached didn't
  include SReclaimable memory. Adding a
  gauge for system.mem.slab_reclaimable allows inverse
  calculation to remove this value from the system.mem.cached gauge.

• Expand GCR pause container image filter

• Kubernetes events for pods, replicasets and deployments now have
  tags that match the metrics metadata. Namely, pod_name, kube_deployment, kube_replicas_set.

• Enabled the collection of the kubernetes resource requirements
  (requests and limits) by bumping the agent-payload dep. and
  collecting the resource requirements.

• Implements resource fallbacks for complex compliance check
  assertions.

• Add system.cpu.num_cores metric with the number of CPU cores
  (windows/linux)

• compliance: Add support for Go custom compliance checks and
  implement two for CIS Kubernetes

• Make DSD Mapper also map metrics that already contain tags.

• If the retrieval of the AWS EC2 instance ID or hostname fails,
  previously-retrieved values are now sent, which should mitigate host
  aliases flapping issues in-app.

• Increase default timeout on AWS EC2 metadata endpoints, and make it
  configurable with ec2_metadata_timeout

• Add container incl./excl. lists support for ECS Fargate
  (process-agent)

• Adds support for a heap profile and cpu profile (of configurable
  length) to be created and included in the flare.

• Upgrade embedded Python 3 to 3.8.5. Link to Python 3.8 changelog:
  https://docs.python.org/3/whatsnew/3.8.html

  Note that the Python 2 version shipped in Agent v6 continues to be
  version 2.7.18 (unchanged).

• Upgrade pip to v20.1.1. Link to pip 20.1.1 changelog:
  https://pip.pypa.io/en/stable/news/#id54

• Upgrade pip-tools to v5.3.1. Link to pip-tools 5.3.1 changelog:
  https://github.com/jazzband/pip-tools/blob/master/CHANGELOG.md

• Introduces support for resolving pathFrom from in File and Audit
  checks.

• On Windows, always add the user to the required groups during
  installation.

• APM: A series of changes to internal algorithms were made which reduced
  CPU usage between 20-40% based on throughput.

Bug Fixes

• Allow integration commands to work for pre-release versions.
• [Windows] Ensure PYTHONPATH variable is ignored correctly when
  initializing the Python runtime.
• Enable listening for conntrack info from all namespaces in system
  probe
• Fix cases where the resolution of secrets in integration configs
  would not be performed for autodiscovered containers.
• Fixes submission of containers blkio metrics that may modify array
  after being already used by aggregator. Can cause missing tags on
  containerd.* metrics
• Restore support of JSON-formatted lists for configuration options
  passed as environment variables.
• Don't allow pressing the disable button on checks twice.
• Fix container_include_metrics
  support for all container checks
• Fix a bug where the Agent disables collecting tags when the cluster
  checks advanced dispatching is enabled in the Daemonset Agent.
• Fixes a bug where the ECS metadata endpoint V2 would get queried
  even though it was not configured with the configuration option
  cloud_provider_metadata.
• Fix a bug when a kubernetes job has exited after some time the
  tagger does not update it even if it did change its state.
• Fixes the Agent failing to start on sysvinit on systems with
  dpkg >= 1.19.3
• The agent was collecting docker container logs (metrics) even if
  they are matching DD_CONTAINER_EXCLUDE_LOGS (resp. DD_CONTAINER_EXCLUDE_METRICS) if they
  were started before the agent. This is now fixed.
• Fix a bug where the Agent would not remove tags for pods that no
  longer exist, potentially causing unbounded memory growth.
• Fix pidfile support on security-agent
• Fixed system-probe not working on CentOS/RHEL 8 due to our custom
  SELinux policy. We now install the custom policy only on CentOS/RHEL
  7, where the system-probe is known not to work with the default. On
  other platform the default will be used.
• Stop sending payload for Cloud Foundry applications containers that
  have no container_name tag attached
  to avoid them showing up in the UI with empty name.

Other Notes

• APM: datadog.trace_agent.receiver.* metrics are now also tagged by
  endpoint_version