Prelude

Release on: 2022-06-28

Bug Fixes

• Fixes issue where proxy config was ignored by the trace-agent.

Prelude

Release on: 2022-06-27

• Please refer to the 7.37.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• OTLP ingest: Support for the deprecated experimental.otlp section and the DD_OTLP_GRPC_PORT and DD_OTLP_HTTP_PORT environment variables has been removed. Use the otlp_config section or the DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT and DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT environment variables instead.
• OTLP: Deprecated settings otlp_config.metrics.report_quantiles and otlp_config.metrics.send_monotonic_counter have been removed in favor of otlp_config.metrics.summaries.mode and otlp_config.metrics.sums.cumulative_monotonic_mode respectively.

New Features

• Adds User-level service unit filtering support for Journald log collection via include_user_units and exclude_user_units.
• A wildcard (*) can be used in either exclude_units or exclude_user_units if only a particular type of Journald log is desired.
• A new troubleshooting section has been added to the Agent CLI. This section will hold helpers to understand the Agent behavior. For now, the section only has two command to print the different metadata payloads sent by the Agent (v5 and inventory).
• APM: Incoming OTLP traces are now allowed to set their own sampling priority.
• Enable NPM NAT gateway lookup by default.
• Partial support of IPv6 on EKS clusters
  • Fix the kubelet client when the IP of the host is IPv6.
  • Fix the substitution of %%host%% patterns inside the auto-discovery annotations: If the concerned pod has an IPv6 and the %%host%% pattern appears inside an URL context, then the IPv6 is surrounded by square brackets.
• OTLP ingest now supports the same settings and protocol version as the OpenTelemetry Collector OTLP receiver v0.50.0.
• The Cloud Workload Security agent can now monitor and evaluate rules on bind syscall.
• [corechecks/snmp] add scale factor option to metric configurations
• Evaluate memory.usage metrics based on collected metrics.

Enhancement Notes

• APM: DD_APM_FILTER_TAGS_REQUIRE and DD_APM_FILTER_TAGS_REJECT can now be a literal JSON array. e.g. ["someKey:someValue"] This allows for matching tag values with the space character in them.
• SNMP Traps are now sent to a dedicated intake via the epforwarder.
• Update SNMP traps database to include integer enumerations.
• The Agent now supports a single com.datadoghq.ad.checks label in Docker, containerd, and Podman containers. It merges the contents of the existing check_names, init_configs (now optional), and instances annotations into a single JSON value.
• Add a new Agent telemetry metric autodiscovery_poll_duration (histogram) to monitor configuration poll duration in Autodiscovery.
• APM: Added /config/set endpoint in trace-agent to change configuration settings during runtime. Supports changing log level(log_level).
• APM: When the X-Datadog-Trace-Count contains an invalid value, an error will be issued.
• Upgrade to Docker client 20.10, reducing the duration of docker check on Windows (requires Docker >= 20.10 on the host).
• The Agent maintains scheduled cluster and endpoint checks when the Cluster Agent is unavailable.
• The Cluster Agent followers now forward queries to the Cluster Agent leaders themselves. This allows a reduction in the overall number of connections to the Cluster Agent and better spreads the load between leader and forwarders.
• The kube_namespace tag is now included in all metrics, events, and service checks generated by the Helm check.
• Include install_info to version-history.json
• Allow nightly builds install on non-prod repos
• Add a kubernetes_node_annotations_as_tags parameter to use Kubernetes node annotations as host tags.
• Add more detailed logging around leadership status failures.
• Move the experimental SNMP Traps Listener configuration under network_devices.
• Add support for the DNS Monitoring feature of NPM to Linux kernels older than 4.1.
• Adds segment_name and segment_id tags to PCF containers that belong to an isolation segment.
• Make logs agent additional_endpoints reliable by default. This can be disabled by setting is_reliable: false on the additional endpoint.
• On Windows, if a datadog.yaml file is found during an installation or upgrade, the dialogs collecting the API Key and Site are skipped.
• Resolve SNMP trap variables with integer enumerations to their string representation.
• [corechecks/snmp] Add profile static_tags config
• Report telemetry metrics about the retry queue capacity: datadog.agent.retry_queue_duration.capacity_secs, datadog.agent.retry_queue_duration.bytes_per_sec and datadog.agent.retry_queue_duration.capacity_bytes
• Updated cloud providers to add the Instance ID as a host alias for EC2 instances, matching what other cloud providers do. This should help with correctly identifying hosts where the customer has changed the hostname to be different from the Instance ID.
• NTP check: Include /etc/ntpd.conf and /etc/openntpd/ntpd.conf for use_local_defined_servers.
• Kubernetes pod with short-lived containers do not have log lines duplicated with both container tags (the stopped one and the running one) when logs are collected. This feature is enabled by default, set logs_config.validate_pod_container_id to false to disable it.

Security Notes

• The Agent is built with Go 1.17.11.

Bug Fixes

• Updates defaults for the port and binding host of the experimental traps listener.
• APM: The Agent is now performing rare span detection on all spans, as opposed to only dropped spans. This change will slightly reduce the number of rare spans kept unnecessarily.
• APM OTLP: This change ensures that the ingest now standardizes certain attribute keys to their correct Datadog tag counter parts, such as: container tags, "operation.name", "service.name", etc.
• APM: Fix a bug where the APM section of the GUI would not show up in older Internet Explorer versions on Windows.
• Support dynamic Auth Tokens in Kubernetes v1.22+ (Bound Service Account Token Volume).
• The %%host%% autodiscovery tag now works properly when using containerd, but only on Linux and when using IP v4 addresses.
• Enhanced the coverage of pause-containers filtering on Containerd.
• APM: Fix the loss of trace metric container information when large payloads need to be split.
• Fix cri check producing no metrics when running on OpenShift / cri-o.
• Fix missing health status from Docker containers in Live Container View.
• Fix Agent startup failure when running as a non-privileged user (for instance, when running on OpenShift with restricted SCC).
• Fix missing container metrics (container, containerd checks and live container view) on AWS Bottlerocket.
• APM: Fixed an issue where "CPU threshold exceeded" logs would show the wrong user CPU usage by a factor of 100.
• Ensures that when kubernetes_namespace_labels_as_tags is set, the namespace labels are always attached to metrics and logs, even when the pod is not ready yet.
• Add missing support for UDPv6 receive path to NPM.
• The agent workload-list --verbose command and the workload-list.log file in the flare do not show containers' environment variables anymore. Except for DD_SERVICE, DD_ENV and DD_VERSION.
• Fixed a potential deadlock in the Python check runner during agent shutdown.
• Fixes issue where trace-agent would not report any version info.
• The DCA and the cluster runners no longer write warning logs to /tmp.
• Fixes an issue where the Agent would panic when trying to inspect Docker containers while the Docker daemon was unavailable or taking too long to respond.

Other Notes

• Exclude teradata on Mac agents.

Prelude

Release on: 2022-05-31

• Please refer to the 7.36.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fixes issue where proxy config was ignored by the trace-agent.
• This fixes a regression introduced in 7.36.0 where some logs sources attached to a container/pod would not be unscheduled on container/pod stop if multiple logs configs were attached to the container/pod. This could lead to duplicate log entries being created on container/pod restart as there would be more than one tailer tailing the targeted source.

Prelude

Release on: 2022-05-24

• Please refer to the 7.36.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Debian packages are now built on Debian 8. Newly built DEBs are supported on Debian >= 8 and Ubuntu >= 14.
• The OTLP endpoint will no longer enable the legacy OTLP/HTTP endpoint 0.0.0.0:55681 by default. To keep using the legacy endpoint, explicitly declare it via the otlp_config.receiver.protocols.http.endpoint configuration setting or its associated environment variable,DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT.
• Package signing keys were rotated:
  • DEB packages are now signed with key AD9589B7, a signing subkey of key F14F620E
  • RPM packages are now signed with key FD4BF915

New Features

• Adding support for IBM cloud. The agent will now detect that we're running on IBM cloud and collect host aliases (vm name and ID).
• Added event collection in the Helm check. The feature is disabled by default. To enable it, set the collect_events option to true.
• Adds a service check for the Helm check. The check fails for a release when its latest revision is in "failed" state.
• Adds a kube_qos (quality of service) tag to metrics associated with kubernetes pods and their containers.
• CWS can now track network devices creation and load TC classifiers dynamically.
• CWS can now track network namespaces.
• The DNS event type was added to CWS.
• The OTLP ingest endpoint is now considered GA for metrics.

Enhancement Notes

• Traps OIDs are now resolved to names using user-provided 'traps db' files in snmp.d/traps_db/.
• The Agent now supports a single ad.datadoghq.com/$IDENTIFIER.checks annotation in Kubernetes Pods and Services to configure Autodiscovery checks. It merges the contents of the existing "check_names", init_configs (now optional), and instances annotations into a single JSON value.
• DD_URL environment variable can now be used to set the Datadog intake URL just like DD_DD_URL. If both DD_DD_URL and DD_URL are set, DD_DD_URL will be used to avoid breaking change.
• Added a process-agent version command, and made the output mimic the core agent.
• Windows: Add Datadog registry to Flare.
• Add --service flag to stream-logs command to filter streamed logs in detail.
• Support a simple date pattern for automatic multiline detection
• APM: The OTLP ingest stringification of non-standard Datadog values such as Arrays and KeyValues is now consistent with OpenTelemetry attribute stringification.
• APM: Connections to upload profiles to the Datadog intake are now closed after 47 seconds of idleness. Common tracer setups send one profile every 60 seconds, which coincides with the intake's connection timeout and would occasionally lead to errors.
• The Cluster Agent now exposes a new metric cluster_checks_configs_info. It exposes the node and the check ID as tags.
• KSM core check: add a new kubernetes_state.cronjob.complete service check that returns the status of the most recent job for a cronjob.
• Retry more HTTP status codes for the logs agent HTTP destination.
• COPYRIGHT-3rdparty.csv now contains each copyright statement exactly as it is shown on the original component.
• Adds sidecar_present and sidecar_count tags on Cloud Foundry containers that run apps with sidecar processes.
• Agent flare now includes output from the process and container checks.
• Add the --cfgpath parameter in the Process Agent replacing --config.
• Add the check subcommand in the Process Agent replacing --check (-check). Only warn once if the -version flag is used.
• Adds human readable output of process and container data in the check command for the Process Agent.
• The Agent flare command now collects Process Agent performance profile data in the flare bundle when the --profile flag is used.

Deprecation Notes

• Deprecated process-agent --vesion in favor of process-agent version.
• The logs configuration use_http and use_tcp flags have been deprecated in favor of force_use_http and force_use_tcp.
• OTLP ingest: metrics.send_monotonic_counter has been deprecated in favor of metrics.sums.cumulative_monotonic_mode. metrics.send_monotonic_counter will be removed in v7.37.
• OTLP ingest: metrics.report_quantiles has been deprecated in favor of metrics.summaries.mode. metrics.report_quantiles will be removed in v7.37 / v6.37.
• Remove the unused --ddconfig (-ddconfig) parameter. Deprecate the --config (-config) parameter (show warning on usage).
• Deprecate the --check (-check) parameter (show warning on usage).

Bug Fixes

• Bump GoSNMP to fix incomplete support of SNMP v3 INFORMs.
• APM: OTLP: Fixes an issue where attributes from different spans were merged leading to spans containing incorrect attributes.
• APM: OTLP: Fixed an inconsistency where the error message was left empty in cases where the "exception" event was not found. Now, the span status message is used as a fallback.
• Fixes an issue where some data coming from the Agent when running in ECS Fargate did not have task_*, ecs_cluster_name, region, and availability_zone tags.
• Collect the "0" value for resourceRequirements if it has been set
• Fix a bug introduced in 7.33 that could prevent auto-discovery variable %%port_<name>%% to not be resolved properly.
• Fix a panic in the Docker check when a failure happens early (when listing containers)
• Fix missing docker.memory.limit (and docker.memory.in_use) on Windows
• Fixes a conflict preventing NPM/USM and the TCP Queue Length check from being enabled at the same time.
• Fix permission of "/readsecret.sh" script in the agent Dockerfile when executing with dd-agent user (for cluster check runners)
• For Windows, fixes problem in upgrade wherein NPM driver is not automatically started by system probe.
• Fix Gohai not being able to fetch network information when running on a non-English windows (when the output of commands like ipconfig were not in English). gohai no longer relies on system commands but uses Golang net package instead (same as Linux hosts). This bug had the side effect of preventing network monitoring data to be linked back to the host.
• Time-based metrics (for example, kubernetes_state.pod.age, kubernetes_state.pod.uptime) are now comparable in the Kubernetes state core check.
• Fix a risk of panic when multiple KSM Core check instances run concurrently.
• For Windows, includes NPM driver 1.3.2, which has a fix for a BSOD on system probe shutdown.
• Adds new --json flag to check. process-agent check --json now outputs valid json.
• On Windows, includes NPM driver update which fixes performance problem when host is under high connection load.
• Previously, the Agent could not log the start or end of a check properly after the first five check runs. The Agent now can log the start and end of a check correctly.

Other Notes

• Include pre-generated trap db file in the conf.d/snmp.d/traps_db/ folder.
• Gohai dependency has been upgraded. This brings a newer version of gopsutil and a fix when fetching network information in non-english Windows (see fixes section).
• If users are using strict firewall rules, they should also exclude the new port 6162 from their firewall.

Prelude

Release on: 2022-05-05

Bug Fixes

• Fix a regression impacting CSPM metering

Prelude

Released on: 2022-04-12
Pinned to datadog-agent v7.35.0: CHANGELOG

New Features

• Collect ResourceRequirements on other K8s workloads as well for live containers (Deployment, StatefulSet, ReplicaSet, DaemonSet).
• Enable collection of Roles/RoleBindings/ClusterRoles/ClusterRoleBindings/ServiceAccounts by default in the orchestrator check.
• Add Ingress collection in the orchestrator check.

Bug Fixes

• Fix a bug that prevents scrubbing sensitive content on the DaemonSet resource.
• Fix a bug that prevents scrubbing sensitive content on the StatefulSet resource.

Prelude

Release on: 2022-04-12

Bug Fixes

• The weak dependency of datadog-agent, datadog-iot-agent and dogstatsd deb packages on the datadog-signing-keys package has been fixed to ensure proper upgrade to version 1:1.1.0.

Prelude

Release on: 2022-04-07

• Please refer to the 7.35.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Agent, Dogstatsd and IOT Agent RPMs now have proper preinstall dependencies. On AlmaLinux, Amazon Linux, CentOS, Fedora, RHEL and Rocky Linux, these are:

  • coreutils (provided by package coreutils-single on certain platforms)
  • grep
  • glibc-common
  • shadow-utils

  On OpenSUSE and SUSE, these are:

  • coreutils
  • grep
  • glibc
  • shadow

• APM Breaking change: The default head based sampling mechanism settings apm_config.max_traces_per_second or DD_APM_MAX_TPS, when set to 0, will be sending 0% of traces to Datadog, instead of 100% in previous Agent versions.

• The OTLP ingest endpoint is now considered stable for traces. Its configuration is located in the top-level otlp_config section.

  Support for the deprecated experimental.otlp section and the DD_OTLP_GRPC_PORT and DD_OTLP_HTTP_PORT environment variables will be removed in Agent 7.37. Use the otlp_config section or the DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT and DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT environment variables instead.

New Features

• The Cloud Workload Security agent can now monitor and evaluate rules on signals (kill syscall).

• CWS allows to write SECL rule on environment variable values.

• The security Agent now offers a command to directly download the policy file from the API.

• Policy can now define macros with items specified as a YAML list instead of a SECL expression, as

  In addition, macros and rules can now be updated in later loaded policies (default.policy is loaded first, the other policies in the folder are loaded in alphabetical order).

  The previous macro can be modified with:

  It can also be overriden with:

  Rules can now also be disabled with:

• Cloud Workload Security now works on Google's Container Optimized OS LTS versions, starting from v81.

• Allow setting variables to store states through rule actions. Action rules can now be defined as follows:

  These actions will be executed when the rule is triggered by an event. Right now, only set actions can be defined. name is the name of the variable that will be set by the actions. The value for the variable can be specified by using:

  • value for a predefined value (strings, integers, booleans, array of strings and array of integers are currently supported).
  • field for the value of an event field.

  Variable arrays can be modified by specifying append: true.

  Variables can be reused in rule expressions like a regular variable:

  By default, variables are global. They can be bounded to a specific process by using the process scope as follows:

  The variable can be referenced in other expressions as ${process.my_scoped_variable}. When the process dies, the variable with be automatically freed.

• Configuration process_config.enabled is now split into two settings: process_config.process_collection.enabled and process_config.container_collection.enabled. This will allow better control over the process Agent. process_config.enabled now translates to these new settings:

  • `process_config.enabled=true`: process_config.process_collection.enabled=true
  • `process_config.enabled=false`: process_config.container_collection.enabled=true and process_config.process_collection.enabled=false
  • `process_config.enabled=disabled`: process_config.container_collection.enabled=false and process_config.process_collection.enabled=false

• Expose additional CloudFoundry metadata in the DCA API that the PCF firehose nozzles can use to reduce the load on the CC API.

• Added new "Helm" cluster check that collects information about the Helm releases deployed in the cluster.

• Add the process_agent_runtime_config_dump.yaml file to the core Agent flare with process-agent runtime settings.

• Add process-agent status output to the core Agent status command.

• Added new process-agent status command to help with troubleshooting and for better consistency with the core Agent. This command is intended to eventually replace process-agent --info.

• CWS rules can now be written on kernel module loading and deletion events.

• The splice event type was added to CWS. It can be used to detect the Dirty Pipe vulnerability.

• Add two options under a new config prefix to send logs to Vector instead of Datadog. vector.logs.enabled must be set to true, along with vector.logs.url that should be set to point to a Vector configured accordingly. This overrides the main endpoints, additional endpoints remains fully functional.

• Adds new Windows system check, winkmem. This check reports the top users of paged and non-paged memory in the windows kernel.

Enhancement Notes

• Add support for the device_namespace tag in SNMP Traps.

• SNMP Trap Listener now also supports protocol versions 1 and 3 on top of the existing v2 support.

• The cluster agent has an external metrics provider feature to allow using Datadog queries in Kubernetes HorizontalPodAutoscalers. It sometimes faces issues like:

  2022-01-01 01:01:01 UTC | CLUSTER | ERROR |
  (pkg/util/kubernetes/autoscalers/datadogexternal.go:79 in
  queryDatadogExternal) | Error while executing metric query ...
  truncated... API returned error: Query timed out

  To mitigate this problem, use the new external_metrics_provider.chunk_size parameter to reduce the number of queries that are batched by the Agent and sent together to Datadog.

• Added a new implementation of the containerd check based on the container check. Several metrics are not emitted anymore: containerd.mem.current.max, containerd.mem.kernel.limit, containerd.mem.kernel.max, containerd.mem.kernel.failcnt, containerd.mem.swap.limit, containerd.mem.swap.max, containerd.mem.swap.failcnt, containerd.hugetlb.max, containerd.hugetlb.failcount, containerd.hugetlb.usage, containerd.mem.rsshuge, containerd.mem.dirty, containerd.blkio.merged_recursive, containerd.blkio.queued_recursive, containerd.blkio.sectors_recursive, containerd.blkio.service_recursive_bytes, containerd.blkio.time_recursive, containerd.blkio.serviced_recursive, containerd.blkio.wait_time_recursive, containerd.blkio.service_time_recursive. The containerd.image.size now reports all images present on the host, container tags are removed.

• Migrate the cri check to generic check infrastructure. No changes expected in metrics.

• Tags configured with DD_TAGS or DD_EXTRA_TAGS in an ECS Fargate or EKS Fargate environment are now attached to Dogstatsd metrics.

• Added a new implementation of the docker check based on the container check. Metrics produced do not change. Added the capability to run the docker check on Linux without access to /sys or /proc, although with a limited number of metrics.

• The DogstatsD protocol now supports a new field that contains the client's container ID. This allows enriching DogstatsD metrics with container tags.

• When ec2_collect_tags is enabled, the Agent now attempts to fetch data from the instance metadata service, falling back to the existing EC2-API-based method of fetching tags. Support for tags in the instance metadata service is an opt-in EC2 feature, so this functionality will not work automatically.

• Add support for ECS metadata v4 API https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html

• Agents are now built with Go 1.17.6.

• On ECS Fargate and EKS Fargate, Agent-configured tags (DD_TAGS/DD_EXTRA_TAGS) are now applied to all integration-collected metrics.

• Logs from JMXFetch will now be included in the Agent logfile, regardless of the log_level setting of the Agent.

• Addition of two knobs to configure JMXFetch statsd client:

  • jmx_statsd_client_queue_size to set the client queue size.
  • jmx_statsd_telemetry_enabled to enable the client telemetry.

• KSMCore node.ready service check now reports warning instead of unknown when a node enters an unknown state.

• Added DD_PROCESS_CONFIG_PROCESS_DD_URL and DD_PROCESS_AGENT_PROCESS_DD_URL environment variables

• Added DD_PROCESS_CONFIG_ADDITIONAL_ENDPOINTS and DD_PROCESS_AGENT_ADDITIONAL_ENDPOINTS environment variables

• Automatically extract the org.opencontainers.image.source container label into the git.repository_url tag.

• The experimental OTLP ingest endpoint now supports the same settings as the OpenTelemetry Collector OTLP receiver v0.43.1.

• The OTLP ingest endpoint now supports the same settings as the OpenTelemetry Collector OTLP receiver v0.44.0.

• The OTLP ingest endpoint can now be configured through environment variables.

• The OTLP ingest endpoint now always maps conventional metric resource-level attributes to metric tags.

• OTLP ingest: the k8s.pod.uid and container.id semantic conventions are now used for enriching tags in OTLP metrics.

• Add the DD_PROCESS_CONFIG_MAX_PER_MESSAGE env variable to set the process_config.max_per_message. Add the DD_PROCESS_CONFIG_MAX_CTR_PROCS_PER_MESSAGE env variable to set the process_config.max_ctr_procs_per_message.

• Add the DD_PROCESS_CONFIG_EXPVAR_PORT and DD_PROCESS_AGENT_EXPVAR_PORT env variables to set the process_config.expvar_port. Add the DD_PROCESS_CONFIG_CMD_PORT env variable to set the process_config.cmd_port.

• Add the DD_PROCESS_CONFIG_INTERNAL_PROFILING_ENABLED env variable to set the process_config.internal_profiling.enabled.

• Add the DD_PROCESS_CONFIG_SCRUB_ARGS and DD_PROCESS_AGENT_SCRUB_ARGS env variables to set the process_config.scrub_args. Add the DD_PROCESS_CONFIG_CUSTOM_SENSITIVE_WORDS and DD_PROCESS_AGENT_CUSTOM_SENSITIVE_WORDS env variables to set the process_config.custom_sensitive_words. Add the DD_PROCESS_CONFIG_STRIP_PROC_ARGUMENTS and DD_PROCESS_AGENT_STRIP_PROC_ARGUMENTS env variables to set the process_config.strip_proc_arguments.

• Added DD_PROCESS_CONFIG_WINDOWS_USE_PERF_COUNTERS and DD_PROCESS_AGENT_WINDOWS_USE_PERF_COUNTERS environment variables

• Add the DD_PROCESS_CONFIG_QUEUE_SIZE and DD_PROCESS_AGENT_QUEUE_SIZE env variables to set the process_config.queue_size. Add the DD_PROCESS_CONFIG_RT_QUEUE_SIZE and DD_PROCESS_AGENT_RT_QUEUE_SIZE env variables to set the process_config.rt_queue_size. Add the DD_PROCESS_CONFIG_PROCESS_QUEUE_BYTES and DD_PROCESS_AGENT_PROCESS_QUEUE_BYTES env variables to set the process_config.process_queue_bytes.

• Changes process payload chunking in the process Agent to take into account the size of process details such as CLI and user name. Adds the process_config.max_message_bytes setting for the target max (uncompressed) payload size.

• When ec2_collect_tags is configured, the Agent retries API calls to gather EC2 tags before giving up.

• Retry HTTP transaction when the HTTP status code is 404 (Not found).

• Validate SNMP namespace to ensure it respects length and illegal character rules.

• Include /etc/chrony.conf for use_local_defined_servers.

Deprecation Notes

• The security Agent commands check-policies and reload are deprecated. Use runtime policy check and runtime policy reload respectively instead.
• Configuration process_config.enabled is now deprecated. Use process_config.process_collection.enabled and process_config.container_collection.enabled settings instead to control container and process collection in the process Agent.
• Removed API_KEY environment variable from the process agent. Use DD_API_KEY instead
• Removes the DD_PROCESS_AGENT_CONTAINER_SOURCE environment variable from the Process Agent. The list of container sources now entirely depends on the activated features.
• Removed unused process_config.windows.args_refresh_interval config setting
• Removed unused process_config.windows.add_new_args config setting
• Removes the process_config.max_ctr_procs_per_message setting.

Bug Fixes

• APM: OTLP: Fixes an issue where attributes from different spans were merged leading to spans containing incorrect attributes.
• APM: Fixed an issue which caused a panic when receiving OTLP traces with invalid data (specifically duplicate SpanIDs).
• Silence the misleading error message No valid api key found, reporting the forwarder as unhealthy from the output of the agent check command.
• Fixed a deadlock in the Logs Agent.
• Exclude filters no longer apply to empty container names, images, or namespaces.
• Fix CPU limit calculation for Windows containers.
• Fix a rare panic in Gohai when collecting the system's Python version.
• For Windows, includes NPM driver 1.3.2, which has a fix for a BSOD on system probe shutdown.
• OTLP ingest now uses the exact sum and count values from OTLP Histograms when generating Datadog distributions.

Other Notes

• JMXFetch upgraded to 0.46.0 https://github.com/DataDog/jmxfetch/releases/0.46.0

Prelude

Released on: 2022-03-01
Pinned to datadog-agent v7.34.0: CHANGELOG

New Features

• Add an external_metrics_provider.endpoints parameter that allows to specify a list of external metrics provider endpoints.
  If the first one fails, the DCA will query the next ones.
• Support file-based endpoint checks.
• Enable collection of PV/PVCs by default in the orchestrator check
• File-based cluster checks support Autodiscovery.

Bug Fixes

• Fix the Admission Controller/Webhooks info section of the cluster agent agent status output on Kubernetes 1.22+.
  Although the cluster agent was able to register its webhook with both the v1beta1 and the v1 version of the Administrationregistration API, the agent status command was always using the v1beta1, which has been removed in Kubernetes 1.22.
• Improve error handling of deleted HPA objects.
• Fix an issue where scrubbing custom sensitive words would not work as intended for the orchestrator check.
• Fixed a bug that could prevent the Admission Controller from starting when the External Metrics Provider is enabled.
• Fix the caculation of orchestrator cache hits.

Prelude

Release on: 2022-03-02

• Please refer to the 7.34.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• CWS uses ** for subfolder matching instead of *. Previously, * was used to match files and subfolders. With this release, * will match only files and folders at the same level. Use`` at the end of a path to match files and subfolders. ` must be used at the end of the path. For example, the rule `open.file.path == "/etc/*" has to be converted to open.file.path == "/etc/**".
• additional_endpoints in the logs_config now uses the same compression configuration as the main endpoint when sending to HTTP destinations. Agents that relied on using different compression settings for additional_endpoints may need to be reconfigured.

New Features

• Autodiscovery of integrations now works with Podman containers. The minimum Podman version supported is 3.0.0.
• Cloud provider detection now support Oracle Cloud. This includes cloud provider detection, host aliases and NTP servers.
• APM: Add proxy endpoint to allow Instrumentation Libraries to submit telemetry data.
• CWS now allows to write SECL rule based on process ancestor args.
• CWS now exposes the first argument of exec event. Usually the name of the executed program.
• Add a new runtime reload command to the security-agent to dynamically reload CWS policies.
• Enables process discovery check to run by default in the process agent. Process discovery is a lightweight process metadata collection check enabling users to see recommendations for integrations running in their environments.
• APM: Adds a new endpoint to the Datadog Agent to forward pipeline stats to the Datadog backend.
• The Cloud Workload Security agent can now monitor and evaluate rules on mmap, mprotect and ptrace.
• Add support for Shift JIS (Japanese) encoding. It should be manually enabled in a log configuration using encoding: shift-jis.
• Extend SNMP profile syntax to support metadata definitions
• When running inside a container with the host /etc folder mounted to /host/etc, the agent will now report the distro informations of the host instead of the one from the container.
• Added telemetry for the workloadmeta store.

Enhancement Notes

• Add Autodiscovery telemetry.
• APM: Add the option to collect SQL comments and commands during obfuscation.
• Adds the process_config.disable_realtime_checks config setting in the process Agent allowing users to disable realtime process and container checks. Note: This prevents refresh of stats in the Live Processes and Live Containers pages for processes and containers reported by the Agent.
• [corechecks/snmp] Add additional metadata fields
• Reduce the memory usage when flushing series.
• Specifying auto_multi_line_detection: false in an integration's logs_config will now disable detection for that integration, even if detection is enabled globally.
• Make agent checkconfig an alias of agent configcheck
• Added possibility to watch all the namespaces when running on containerd outside Kubernetes. By default, the agent will report events and metrics from all the namespaces. In order to select a specific one, please set the containerd_namespace option.
• The container check now works for containers managed by runtimes that implement the CRI interface such as CRI-O.
• cri.* and container.* metrics can now be collected from the CRI API on Windows.
• When using site: ddog-gov.com, the agent now uses Agent-version-based URLs and api.ddog-gov.com as it has previously done for other Datadog domains.
• Add telemetry for ECS queries.
• Agents are now built with Go 1.16.12.
• Add Kubelet queries telemetry.
• Add the kubernetes_node_annotations_as_host_aliases parameter tospecify a list of Kubernetes node annotations that should be used as host aliases. If not set, it defaults to cluster.k8s.io/machine.
• The experimental OTLP endpoint now supports the same settings as the OpenTelemetry Collector OTLP receiver v0.41.0.
• OTLP metrics tags are enriched when experimental.otlp.metrics.tag_cardinality is set to orchestrator. This can also be controlled via the DD_OTLP_TAG_CARDINALITY environment variable.
• Make the Prometheus auto-discovery be able to schedule OpenMetrics V2 checks instead of legacy V1 ones.
  By default, the Prometheus annotations based auto-discovery will keep on scheduling openmetrics v1 check. But the agent now has a prometheus_scrape.version parameter that can be set to 2 to schedule the v2.
  The changes between the two versions of the check are described in https://datadoghq.dev/integrations-core/legacy/prometheus/#config-changes-between-versions
• Raised the max batch size of logs and events from <spanclass="title-ref">100 to 1000 elements. Improves performance in high volume scenarios.
• Add saturation metrics for network and memory.
• The Agent no longer logs spurious warnings regarding proxy-related environment variables DD_PROXY_NO_PROXY, DD_PROXY_HTTP, and DD_PROXY_HTTPS.
• [corechecks/snmp] Add agent host as tag when use_device_id_as_hostname is enabled.
• [corechecks/snmp] Add profile metadata match syntax
• [corechecks/snmp] Support multiple symbols for profile metadata
• On Windows, the installer now uses a zipped Python integration folder, which should result in faster install times.
• Add support for Windows 2022 in published Docker images

Bug Fixes

• APM: Fix SQL obfuscation error on statements using bind variables starting with digits
• Adds Windows NPM driver 1.3.1, which contains a fix for the system crash on system-probe shutdown under heavy load.
• DD_CLUSTER_NAME can be used to define the kube_cluster_name on EKS Fargate.
• On Windows the Agent now correctly detects Windows 11.
• Fixes an issue where the Docker check would undercount the number of stopped containers in the docker.containers.stopped and docker.containers.stopped.total metrics, accompanied by a "Cannot split the image name" error in the logs.
• Fixed a bug that caused a panic when running the docker check in cases where there are containers stuck in the "Removal in Progress" state.
• On EKS Fargate, the container check is scheduled while no suitable metrics collector is available, leading to excessive logging. Also fixes an issue with Liveness/Readiness probes failing regularly.
• Allow Prometheus scrape tls_verify to be set to false and change label_to_hostname type to string.
• Fixes truncated queries using temp tables in SQL Server.
• Fixes an NPM issue on Windows where if the first packet on a UDP flow is inbound, it is not counted correctly.
• On macOS, fix a bug where the Agent would not gracefully stop when sent a SIGTERM signal.
• Fix missing tags with eBPF checks (OOM Kill/TCP Queue Length) with some container runtimes (for instance, containerd 1.5).
• The experimental OTLP endpoint now ignores hostname attributes with localhost-like names for hostname resolution.
• Fixes an issue where cumulative-to-delta OTLP metrics conversion did not take the hostname into account.

Prelude

Release on: 2022-02-10

Bug Fixes

• Fixes a panic that happens occasionally when handling tags for deleted containers or pods.
• Fixes security module failing to start on kernels 4.14 and 4.15.

Prelude

Released on: 2022-01-26
Pinned to datadog-agent v7.33.0.

New Features

• Collect PVC tag on pending pods
• Add the ability to filter for check names in the cluster checks output.

Bug Fixes

• Add reworked status output for orchestrator section on CLC setups.

Security

• Fix the removal of the "kubectl.kubernetes.io/last-applied-configuration" annotation on new collected resources

Enhancement Notes

• Add autoscaler resource kind (hpa,wpa) inside the DatadogMetrics status references.

Prelude

Release on: 2022-01-26

• Please refer to the 7.33.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• APM: The apm_config.max_traces_per_second setting no longer affects error sampling. To change the TPS for errors, use apm_config.error_traces_per_second instead.
• Starting from this version of the Agent, the Agent does not run on SLES 11. The new minimum requirement is SLES >= 12 or OpenSUSE >= 15 (including OpenSUSE 42).
• Changed the default value of logs_config.docker_container_use_file to true. The agent will now prefer to use files for collecting docker logs and fall back to the docker socket when files are not available.
• Upgrade Docker base image to ubuntu:21.10 as new stable release.

New Features

• Autodiscovery of integrations now works with containerd.
• Metadata information sent by the Agent are now part of the flares. This will allow for easier troubleshooting of issues related to metadata.
• APM: Added credit card obfuscation. It is off by default and can be enabled using the env. var. DD_APM_OBFUSCATION_CREDIT_CARDS_ENABLED or apm_config.obfuscation.credit_cards.enabled. There is also an option to enable an additional Luhn checksum check in order to eliminate false negatives, but it comes with a performance cost and should not be used unless absolutely needed. The option is DD_APM_OBFUSCATION_CREDIT_CARDS_LUHN or apm_config.obfuscation.credit_cards.luhn.
• APM: The rare sampler can now be disabled using the environment variable DD_APM_DISABLE_RARE_SAMPLER or the apm_config.disable_rare_sampler configuration. By default the rare sampler catches 5 extra trace chunks per second on top of the head base sampling. The TPS is spread to catch all combinations of service, name, resource, http.status, error.type missed by head base sampling.
• APM: The error sampler TPS can be configured using the environment variable DD_APM_ERROR_TPS or the apm_config.error_traces_per_second configuration. It defaults to 10 extra trace chunks sampled per second on top of the base head sampling. The TPS is spread to catch all combinations of service, name, resource, http.status, and error.type.
• Add a generic container check. It generates container.* metrics based on all running containers, regardless of the container runtime used (among the supported ones).
• Added new option "container_labels_as_tags" that allows the Agent to extract container label values and set them as metric tags values. It's equivalent to the existing "docker_labels_as_tags", but it also works with containerd.
• CSPM: enable the usage of the print function in Rego rules.
• CSPM: add option to dump reports to file, when running checks manually. CSPM: constants can now be defined in rego rules and will be usable from rego rules.
• CWS: SECL expressions can now make use of predefined variables. ${process.pid} variable refers to the pid of the process that trigger the event.
• Enable NPM DNS domain collection by default.
• Exposed additional experimental configuration for OTLP metrics translation via experimental.otlp.metrics.
• Add two options under a new config prefix to send metrics to Vector instead of Datadog. vector.metrics.enabled must be set to true, along with vector.metrics.url that should be set to point to a Vector configured accordingly.
• The bpf syscall is now monitored by CWS; rules can be written on BPF commands.
• Add runtime settings support to the security-agent. Currenlty only the log-level is supported.
• APM: A new intake endpoint was added as /v0.6/traces, which accepts a new, more compact and efficient payload format. For more details, check: https://github.com/DataDog/datadog-agent/blob/7.33.0/pkg/trace/api/version.go#L78.

Enhancement Notes

• Adds Nomad namespace and datacenter to list of env vars extracted from Docker containers.
• Add a new On-disk storage section to agent status command.
• Run CSPM commands as a configurable user. Defaults to 'nobody'.
• CSPM: the findings query now defaults to data.datadog.findings
• The docker.exit service check has a new tag exit_code. The 143 exit code is considered OK by default, in addition to 0. The Docker check supports a parameter ok_exit_codes to allow choosing exit codes that are considered OK.
• Allow dogstatsd replay files to be fully loaded into memory as opposed to relying on MMAP. We still default to MMAPing replay targets.
• kubernetes_state.node.* metrics are tagged with kubelet_version, container_runtime_version, kernel_version, and os_image.
• The Kube State Metrics Core check uses ksm v2.1.
• Lowercase the cluster names discovered from cloud providers to ease moving between different Datadog products.
• On Windows, allow enabling process discovery in the process agent by providing PROCESS_DISCOVERY_ENABLED=true to the msiexec command.
• Automatically extract the org.opencontainers.image.revision container label into the git.commit.sha tag.
• The experimental OTLP endpoint now can be configured through the experimental.otlp.receiver section and supports the same settings as the OpenTelemetry Collector OTLP receiver v0.38.0.
• The Process, APM, and Security agent now use the remote tagger introduced in Agent 7.26 by default. To disable it in the respective agent, the following settings need to be set to `false`:
  • apm_config.remote_tagger
  • process_config.remote_tagger
  • security_agent.remote_tagger
• Allows the remote tagger timeout at startup to be configured by setting the remote_tagger_timeout_seconds config value. It also now defaults to 30 seconds instead of 5 minutes.
• Calls to cloud metadata APIs for metadata like hostnames and IP addresses are now cached and the existing values used when the metadata service returns an error. This will prevent such metadata from temporarily "disappearing" from hosts.
• Datadog Process Agent Service is started automatically by the core agent on Windows when process discovery is enabled in the config.
• All packages - datadog-agent, datadog-iot-agent and datadog-dogstatsd -now support AlmaLinux and Rocky Linux distributions.
• If unrecognized DD_.. environment variables are set, the agent will now log a warning at startup, to help catch deployment typos.
• Update the embedded pip version to 21.3.1 on Python 3 to allow the use of newer build backends.
• Metric series can now be submitted using the V2 API by setting use_v2_api.series to true. This value defaults to false, and should only be set to true in internal testing scenarios. The default will change in a future release.
• Add support for Windows 20H2 in published Docker images
• Add a new agent command to dump the content of the workloadmeta store agent workload-list. The output of agent workload-list --verbose is included in the agent flare.

Bug Fixes

• Strip special characters (n, r and t) from OctetString
• APM: Fix bug where obfuscation fails for autovacuum sql text. For example, SQL text like autovacuum: VACUUM ANALYZE fake.table will no longer fail obfuscation.
• APM: Fix SQL obfuscation failures on queries with literals that include non alpha-numeric characters
• APM: Fix obfuscation error on SQL queries using the '!' operator.
• Fixed Windows Dockerfile scripts to make the ECS Fargate Python check run when the agent is deployed in ECS Fargate Windows.
• Fixing deadlock when stopping the agent righ when a metadata provider is scheduled.
• Fix a bug where container_include/exclude_metrics was applied on Autodiscovery when using Docker, preventing logs collection configured through container_include/exclude_logs.
• Fix inclusion of registry.json file in flare
• Fixes an issue where the agent would remove tags from pods or containers around 5 minutes after startup of either the agent itself, or the pods or containers themselves.
• APM: SQL query obfuscation doesn't drop redacted literals from the obfuscated query when they are preceded by a SQL comment.
• The Kube State Metrics Core check supports VerticalPodAutoscaler metrics.
• The experimental OTLP endpoint now uses the StartTimestamp field for reset detection on cumulative metrics transformations.
• Allow configuring process discovery check in the process agent when both regular process and container checks are off.
• Fix disk check reporting /dev/root instead of the actual block device path and missing its tags when tag_by_label is enabled.
• Remove occasionally hanging autodiscovery errors from the agent status once a pod is deleted.

Other Notes

• The Windows installer only creates the datadog.yaml file on new installs.

Prelude

Release on: 2021-12-22

• JMXFetch: Remove all dependencies on log4j and use java.util.logging instead.

Prelude

Release on: 2021-12-15

Security Notes

• Upgrade the log4j dependency to 2.12.2 in JMXFetch to fully address CVE-2021-44228 and CVE-2021-45046

Prelude

Release on: 2021-12-11

Security Notes

• Set -Dlog4j2.formatMsgNoLookups=True when starting the JMXfetch process to mitigate vulnerability described in CVE-2021-44228

Prelude

Release on: 2021-11-18

Bug Fixes

• On ECS, fix the volume of calls to ListTagsForResource which led to ECS API throttling.
• Fix incorrect use of a namespaced PID with the host procfs when parsing mountinfo to ensure debugfs is mounted correctly. This issue was preventing system-probe startup in AWS ECS. This issue could also surface in other containerized environments where PID namespaces are in use and /host/proc is mounted.
• Fixes system-probe startup failure due to kernel version parsing on Linux 4.14.252+. This specifically was affecting versions of Amazon Linux 2, but could affect any Linux kernel in the 4.14 tree with sublevel >= 252.

Prelude

Released on: 2021-11-10
Pinned to datadog-agent v7.32.0

New Features

• Introduce the collection of the following resources: ClusterRole, ClusterRoleBinding, Role, RoleBinding, ServiceAccount.

Bug Fixes

• Fix tags for PV resources in the Orchestrator Explorer (type and phase).
• Fix an edge case in which the Cluster Agent's Admission Controller doesn't update the Webhook object according to specified configuration.

Prelude

Release on: 2021-11-09

• Please refer to the 7.32.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• APM: Change default profiling intake to use v2 endpoint.
• CSPM the check subcommand is now part of the security-agent
  compliance.

New Features

• On Kubernetes, add a kube_priority_class tag on metrics coming from pods with a priority class.
• Priority class name of pods are now collected and sent to the orchestration endpoint
• Autodiscovery can now resolve template variables and environment variables in log configurations.
• The Windows installer now offers US5 as a new site choice.
• APM: New telemetry was added to measure /v.*/traces endpoints latency and response size. These metrics are datadog.trace_agent.receiver.{rate_response_bytes,serve_traces_ms}.
• APM: Metrics are now available for Windows Pipes and UDS connections via datadog.trace_agent.receiver {uds_connections,pipe_connections}.
• Introduce a new configuration parameter container_env_as_tags to allow converting containerd containers' environment variables into tags.
• The "containerd" check is now supported on Windows.
• Add experimental support for writing agent-side CSPM compliance checks in Rego.
• Runtime security can now attach span/trace to event.
• Provides alternative implementation for process collection on Windows using performance counters.
• Add multi-line auto-sensing when tailing logs from file. It checks the 1000 first lines (or waits 30 seconds, whichever is first) when tailing for a list of known timestamp formats. If the number of matched lines is greater than the threshold it switches to the MultiLineHandler with the pattern matching the timestamp format. The pattern chosen is saved in the log config and is reused if the file rotates. Use the new global config parameter logs_config.auto_multi_line_detection to enable the feature for the whole agent, or the per log integration config parameter auto_multi_line_detection to enable the feature on a case by case basis.
• Added experimental support for OTLP metrics via experimental.otlp.{http_port,grpc_port} or their corresponding environment variables (DD_OTLP{HTTP,GRPC}_PORT).
• Created a new process discovery check. This is a lightweight check that runs every 4 hours by default, and collects process metadata, so that Datadog can suggest potential integrations for the user to enable.
• Added new executable readsecret_multiple_providers.sh that allows the agent to read secrets both from files and Kubernetes secrets. Please refer to the docs for more details.

Enhancement Notes

• KSM core check has a new labels_as_tags parameter to configure which pod labels should be used as datadog tag in an easier way than with the label_joins parameter.
• Add namespace to snmp listener config
• Remove network_devices from datadog.yaml configuration
• kubernetes state core check: add kubernetes_state.job.completion.succeeded and kubernetes_state.job.completion.failed metrics to report job completion as metrics in addition to the already existing service check.
• Add use_device_id_as_hostname in snmp check and snmp_listener configuration to use DeviceId as hostname for metrics and service checks
• APM: The maximum allowed tag value length has been increased to 25,000 bytes.
• Reduce memory usage when checks report new metrics every run. Most metrics are removed after two check runs without new samples. Rate, historate and monotonic count will be kept in memory for additional 25 hours after that. Number of check runs and the additional time can be changed with check_sampler_bucket_commits_count_expiry and check_sampler_stateful_metric_expiration_time. Metric expiration can be disabled entirely by setting check_sampler_expire_metrics to false.
• CSPM reports the agent version as part of the events
• Agents are now built with Go1.16. This will have one user-visible change: on Linux, the process-level RSS metric for agent processes will be reduced from earlier versions. This reflects a change in how memory usage is calculated, not a reduction in used memory, and is an artifact of the Go runtime switching from MADV_FREE to MADV_DONTNEED.
• Tag Kubernetes containers with image_id tag.
• Eliminates the need to synchronize state between regular and RT process collection.
• APM: Added a configuration option to set the API key separately for Live Debugger. It can be set via apm_config.debugger_api_key or DD_APM_DEBUGGER_API_KEY.
• Update EP forwarder config to use intake v2 for ndm metadata
• Remove the reason tag from the kubernetes_state.job.failed metric to reduce cardinality
• The runtime security module of system-probe is now powered by DataDog/ebpf-manager instead of DataDog/ebpf.
• Security Agent: use exponential backoff for log warning when the security agent fails to connect to the system probe.
• APM: OTLP traces now supports semantic conventions from version 1.5.0 of the OpenTelemetry specification.
• Show enabled autodiscovery sources in the agent status
• Add namespace to SNMP integration and SNMP Listener to disambiguate devices with same IP.
• Add snmp corecheck autodiscovery
• Enable SNMP device metadata collection by default
• Reduced CPU usage when origin detection is used.
• The Windows installer now prioritizes user name from the command line over stored registry entries

Bug Fixes

• Make sure DD_ENABLE_METADATA_COLLECTION="false" prevent all host metadata emission, including the initial one.
• Most checks are stripping tags with an empty value. KSM was missing this logic so that KSM specific metrics could have a tag with an empty value. They will now be stripped like for any other check.
• Fixed a regression that was preventing the Agent from retrying kubelet and docker connections in case of failure.
• Fix the cgroup collector to correctly pickup Cloud Foundry containers.
• Fix an issue where the orchestrator check would stop sending updates when run on as a cluster-check.
• Port python-tuf CVE fix on the embedded Python 2 see https://github.com/theupdateframework/python-tuf/security/advisories/GHSA-wjw6-2cqr-j4qr.
• Fix some string logging in the Windows installer.
• The flare command now correctly copies agent logs located in subdirectories of the agent's root log directory.
• Kubernetes state core check: job.status.succeeded and job.status.failed gauges were not sent when equal 0. 0 values are now sent.
• Tag Namespace and PV and PVC metrics correctly with phase instead of pod_phase in the Kube State Metrics Core check.

Prelude

Release on: 2021-09-28
Pinned to datadog-agent v7.31.1

Bug Fixes

• Fix CSPM not sending intake protocol causing lack of host tags.