Bot releases are hidden (Show)
Published by ebeahan over 2 years ago
markupsafe==2.0.1
to resolve ImportError
exception. #1804Published by ebeahan over 2 years ago
We're pleased to announce ECS 8.0.
Thank you to all the ECS contributors who help support the broader Elastic community.
ECS versioning now aligns with the Elastic platform beginning with 8.0.
ECS didn't follow the same release cadence as the Elastic platform when first introduced. Over time this approach added complexity for our users. For example, users might find themselves asking, "which Elastic version maps to ECS 1.6.0?". By aligning, it's clear what version of ECS maps to which Elastic platform version.
Power in simplicity. š
The following fields are removed in ECS 8.0:
Field | Migrate to* | Reference |
---|---|---|
log.original |
event.original |
RFC 0017 |
process.ppid |
process.parent.pid |
RFC 0022 |
host.user.* reuse |
user.* reuses |
user.* field set usage |
*Field aliases can help transition existing searches or visualizations depending on these removed fields.
ECS 1.x introduced wildcard
and match_only_text
as beta field types. As of ECS 8.0, these data types are now GA.
The field types selected for ECS provide the best default experience for most users. However, some users may see interoperable data types better fitting for their use cases, and they can read more about options here.
In 1.x, the project maintained sample index templates for two versions of Elasticsearch (6.x, 7.x). In 8.0, ECS now produces two sample template types: composable
and legacy.
In composable,
each ECS field set has a component template. An example component template, template.json,
references each field set component template. These artifacts work with the new index templates introduced in Elasticsearch 7.8.
The legacy
template will continue working with the legacy index template API.
--oss
flaghost.user.*
field reuse. #1439http.request.method
. #1443log.origin.file.line
from integer
to long
. #1533log.original
field. #1580process.ppid
field. #1596faas.*
field set as beta. #1628, #1755match_only_text
type field migration GA. #1584default_field: true
for Beats artifacts. #1633analyzer
definitions for text fields. #1737default_field
flag for root fields in Beats generator. #1711Published by ebeahan almost 3 years ago
object
as fallback for flattened
type. #1653Published by kgeller about 3 years ago
x509
order to correct nesting. ##1621Published by kgeller about 3 years ago
The following RFCs have advanced as a part of this release:
threat.*
field settext
to match_only_text
typeThere's also been a couple of new field additions in 1.12: file.fork_name
, service.address
, process.end
, code_signature.digest_algorithm
and code_signature.timestamp
.
Lastly, a couple tooling and documentation improvements. There now exists support for multi-field type fallback to better support ES 6 types as well as the new match_only_text
type. And finally, we updated examples within user
to better clarify things.
hash
order to correct nesting. #1603hash
reuses. #1604pe
order to correct nesting. #1605pe
reuses. #1606enrichments
to an array
type. #1608file.fork_name
field. #1288service.address
field. #1537service.environment
as a beta field. #1541process.end
field. #1544code_signature.digest_algorithm
and code_signature.timestamp
fields. #1557email.*
field set in the experimental fields. #1569keyword
fields to wildcard
. #1517threat.software.*
and threat.group.*
fields to GA. #1540user.name
and user.id
examples for clarity. #1566text
and .text
multi-fields to match_only_text
. #1532, #1571match_only_text
field types. #1528find | xargs rm
. #1588Published by ebeahan about 3 years ago
The following RFCs have advanced as part of this release:
elf
file fieldsthreat.*
field set with threat.software.*
and threat.group.*
fieldsThe event.agent_id_status
field is also new in 1.11 to reflect the status of the agent.id
verification performed by a receiving system or data pipeline.
Lastly, many tooling and documentation improvements, including the --exclude
flag. The --exclude
flag adds the ability to remove individual fields from the schema. More detail is available in the usage doc.
elf.*
field set added as beta. #1410beta
from orchestrator
field set. #1417threat.*
field set beta. #1438event.agent_id_status
field. #1454process.target
and process.target.parent
added to experimental schema. #1467threat.enrichments
beta fields. #1478, #1504relater.user
description. #1420cloud.region
and cloud.availability
fields. #1452event.kind
descriptions for alert
and signal
. #1548host.user.*
field reuse. #1422log.original
superseded by event.original
#1469ignore_above
when index: false
and doc_values: false
. #1483doc_values
is carried into Beats artifacts. #1488match_only_text
data type in Go code generator. #1418beta
attribute now supported on categorization allowed values. #1511Location
and Field Set
columns in Field Reuse
table for better readability. #1472, #1476Threat
schema #1505Published by kgeller over 3 years ago
A handful of new additions from the ECS RFC process are included in this release:
data_stream
fields moved to Stage 2, and are released for beta.In addition to RFC proposed changes, ECS 1.10.0 also adds some documentation updates, including the ability to add a short_override
to field reuses for a custom description.
Finally, there is now support for flattened and nested types in the Go code generator script.
data_stream
fieldset. #1307orchestrator
fieldset as beta fields. #1326threat.*
experimental fields with proposed changes from RFC 0018. #1344, #1351short_override
#1366user.*
field reuse descriptions. #1382nested
types in go code generator. #1254, #1350flattened
data type. #1302Published by ebeahan over 3 years ago
Several additions introduced from the ECS RFC process are included in this release:
Finished
status with user.changes.*
, user.effective.*
, and user.target.*
field reuses becoming GA.threat.indicator
fields, elf.*
fields, pe.*
extensions, and data_stream.*
fieldset are now in the experimental ECS schema.A new section has been added to the ECS event categorization documentation. Real-world example events are categorized to demonstrate using the event categorization fields to group and identify similar events from multiple data sources.
In addition to RFC proposed changes, ECS 1.9.0 also adds:
http.request.id
cloud.service.name
hash.ssdeep
code_signature.team_id
and code_signature.signing_id
geo.*
fieldset: geo.timezone
, geo.postal_code
, geo.continent_code
Finally, *.mac
field descriptions now suggest normalizing MAC address values to the RFC7042 format.
hash.ssdeep
. #1169cloud.service.name
. #1204http.request.id
. #1208data_stream.*
fieldset introduced in experimental schema and artifacts. #1215geo.timezone
, geo.postal_code
, and geo.continent_code
. #1229beta
host metrics fields. #1248code_signature.team_id
, code_signature.signing_id
. #1249pe
fields added to experimental schema. #1256elf
fieldset to experimental schema. #1261threat.indicator
fields to experimental schema. #1268user.changes.*
, user.effective.*
, and user.target.*
field reuses are GA. #1271Published by ebeahan over 3 years ago
In this release, two ECS RFCs are advancing. The multiple users in an event RFC proposed field reuses now appear in the ECS documentation as beta. The host metrics fields are also advancing and are available in the experimental schema and artifacts.
Accompanying the multiple user
changes, the user.*
fieldset adds ECS' first usage doc. The user usage page contains guidance on categorization, user ids, field reuse, and mapping examples.
The event categorization fields, with the initial set of allowed values, were introduced as beta in ECS 1.4.0. Over the past several ECS released, we've iterated and further fleshed out these fields and values. We're excited to announce that the event categorization fields are now generally available!
In addition to the event categorizations fields becoming GA, two additional event.category
allowed values have also been introduced: registry
and session.
A new field, os.type
, is intended to ease filtering for Windows, Unix, Linux, and macOS events.
Finally, a component template and composable templates (per fieldset) have been added as generated artifacts. The legacy index templates for Elasticsearch 6.x and 7.x are still being maintained. More details covered here.
event.reference
description. #1181scaled_float
type is used. #1250event.category
"registry". #1040event.category
"session". #1049user
fields. #1066user
fields at user.effective.*
, user.target.*
and user.changes.*
. #1066os.type
. #1111[
and ]
bracket characters may enclose a literal IPv6 address when populating url.domain
. #1131url.extension
. #1151host.user.*
fields for removal at the next major. #1066tracing
fields should be at root of Beats fields.ecs.yml
artifacts. #1164path
key when type is alias
, to support the alias field type. #877scaled_float
's mandatory parameter scaling_factor
. #1042constant_keyword
to keyword
. #1046wildcard
, version
, and constant_keyword
data types. #1050constant_keyword
's optional parameter value
. #1112tracing
fields are not nested under thetracing.
#1162Published by ebeahan almost 4 years ago
A few months ago, we introduced the RFC process. This process is meant to fully vet big additions or changes to ECS. A key aspect of this process is that proposals advance in stages. Each stage represents the vetting and maturity of the proposal.
We wonāt go over the process in detail here, but one of its key aspects is that accepted āstage 2ā proposals appear in āexperimentalā ECS artifacts. They donāt yet appear officially in ECS documentation. Proposals that reach āstage 3ā are the ones that will officially appear in ECS documentation.
ECS 1.7 is the first release that includes RFCs that have reached stage 2 / experimental changes. A new directory has therefore been added, where all the usual generated artifacts are published including the experimental changes. This is at experimental/generated.
This release includes experimental changes from two RFCs reaching stage 2:
keyword
type on many existing ECS fields with the new wildcard
type.user.effective.*
, user.target.*
, and user.changes.*
.Contrary to the new experimental changes described above, the following changes are reflected in the documentation.
Two new fields are introduced: http.[request|response].mime_type
/ and threat.technique.subtechnique
.
Both the network.direction
and event.category
fields add support for additional allowed values.
The ECS generator script adds two new arguments, --oss
and --strict
. See usage for more details and examples.
Lastly, we have changed the index pattern of the sample Elasticsearch template from ecs-*
to try-ecs-*
to avoid conflicting with Logstash' template when run in ECS compatibility mode.
protocol
allowed value under event.type
should not have the expected_event_types
defined. #964file.extension
(no dots). #1016threat.technique.subtechnique
to capture MITRE ATT&CKĀ® subtechniques. #951configuration
as an allowed event.category
. #963source.*
and destination.*
. #967.subdomain
under client
, destination
, server
, source
url
, to match its presence at dns.question.subdomain
. #981ecs-*
totry-ecs-*
to avoid conflicting with Logstash' ecs-logstash-*
. #1048as
value for their destination. #960event.original
index setting. #1053--strict
flag to perform stricter schema validation when running the generator script. #937--strict
that ensures composite types in example fields are quoted. #966ignore_above
and normalizer
support for keyword multi-fields. #971--oss
flag for users who want to generate ECS templates for use on OSS clusters. #991[discrete]
marker before each section header in field details. #989--ref
now loads experimental/schemas
based on git ref in addition to schemas
. #1063Published by ebeahan about 4 years ago
This release adds the x509.*
field set to capture common core fields for x509 certificates. Other notable schema changes include the introduction of event.reason
, adding span.id
to the transaction.*
field set, and new related.*
fields. Please see the full schema change details below.
Before this release, there was no way to reuse field sets as different names inside themselves. Now nesting fields within themselves, such as process
=> process.parent
, and defining nested sets using a different name are both available.
Did you know you can use the Python scripts in the ECS repository to generate Elasticsearch templates containing the only ECS fields you need + your custom fields? A lot of the changes in the "tooling and artifact" changelog below are about how we improved this experience. However you can jump directly to the new usage documentation to learn how to do this.
Finally in previous releases, reusable fields not expected at the root of documents were accidentally defined at the root in some generated artifacts. This incorrect behavior is fixed in this release.
registry.data.strings
should have been marked as an array field. #790x509.*
field set. #762agent.build.*
for extended agent version information. #764log.file.path
to capture the log file an event came from. #802pe
at process.parent.pe
#868span.id
to the tracing fieldset, for additional log correlation #882event.reason
for the reason why an event's outcome or action was taken. #907related.hosts
to capture all hostnames and host identifiers on an event. #913user.roles
to capture a list of role names that apply to the user. #917user.id
, it shouldobject_type=keyword
that was being applied to object
fields.dns.answers
,log.syslog
, network.inner
, observer.egress
, and observer.ingress
. #871dashed_name
in generated/ecs/*.yml
to also@
with -
. #871http.request.method
#840reusable.top_level:false
. This PR affects ecs_flat.yml
, the csv fileorder
attribute from the ecs_nested.yml
and ecs_flat.yml
files. #811ecs_nested.yml
, the array of strings that used to be in reusable.expected
name
and fields
keys at the top level. #873labels
to avoid YAML interpreting it, and having--include
or --subset
flags. #814index
is now correctly populated in the Beats field definition file. #824nestings
array in ecs_nested.yml
. #803enabled
field parameter. #824ref
option to generator allowing schemas to be built for a specific ECS version. #851template-settings
and mapping-settings
options to allow override of defaults in generated ES templates. #856--include
flag, it's no longer necessaryprocess
=> process.parent
). #864reused_here
is added in ecs_nested.yml
. It obsoletes thenestings
, and is able to fully capture details of othergroup
=> user
, then user
=> many places),reusable.order
. Thisgroup
. #864generated/ecs/ecs.yml
, which is a deeply nestedecs_flat.yml
and ecs_nested.yml
files are now generated for each individual subset,ecs_nested.yml
, we're deprecating the attribute nestings
. It will benestings
attribute was an array ofreused_here
, which is an array of objects.Published by webmat over 4 years ago
In this release, we continue fleshing out categorization by introducing the "network" and "iam" categories, with related event types.
We're adding new field sets: "dll", "pe", "code_signature", "interface" & "vlan". We're also adding a few fields here and there (check out the details below).
Implementers consuming ECS artifacts like generated/ecs/*.yml
programmatically will be happy to know that we now clearly identify which fields are expected to contain an array of values. Shout-out to contributors on the ecs-logging libraries for raising this šš¼.
Finally, starting with ECS 1.5.0, the project is using Python 3.7.
dll.*
fields #679related.hash
to keep track of all hashes seen on an event. #711code_signature
fieldset. #733hash
fields at process.parent.hash.*
. #739entity_id
to process
and process.parent
. #747rule.author
, rule.license
fields #754event.category
and three related values for event.type
. #756event.reference
and event.url
to hold link to additional event info/actions. #757file.mime_type
to include MIME type information on file structures #760event.category
value of network and associated event.type
values. #761default_field
growing too big. #687event.outcome
based on community feedback. #759Published by webmat almost 5 years ago
This release introduces two much-awaited changes.
The text
analyzer has been added to many existing fields. This enables full text search queries on fields that contain a lot of text, or semi-structured data (such as file paths and urls). Look at #575 and #680 to learn more. As an example, the field user_agent.original
can now service full text search queries at user_agent.original.text
.
We're also introducing the first set of allowed values for the 4 previously reserved fields (event.kind
, event.category
, event.type
and event.outcome
). We're calling them the "categorization fields". More allowed values will be released over time. You can preview future values, and provide feedback in this public document: https://ela.st/ecs-categories-draft. Learn more in the new "ECS Categorization Fields" section of the documentation.
text
analyzer as a multi-field to user_agent.original
. #575file.attributes
. #611file.drive_letter
. #620rule
fields. #665text
analyzer as a multi-field to around 25 more fields. #680registry.*
fieldset for the Windows registry. #673event.kind
, event.category
, event.type
and event.outcome
. #684, #691, #692related.user
#694Published by webmat almost 5 years ago
tls.server.supported_ciphers
. #662Published by webmat almost 5 years ago
vulnerability.*
fields to represent vulnerability information. #581event.ingested
as the ingest timestamp. #582package.reference
. #585package.build_version
. #586package.type
. #587host.domain
field. #591process.command_line
. #599process.exit_code
. #600tls.*
to support analysis of TLS protocol events. #606process.parent.*
. #612process.args_count
. #615schema.json
and the code generating it. #627generated/elasticsearch
, this PR only removes an obsolete file.Published by webmat about 5 years ago
threat.*
fields to apply a taxonomy to events and alerts. #505log.*
to allow for full Syslog mapping. #525package.*
to installed software packages. #532registered_domain
to url
, source
, destination
, client
, and server
. #533top_level_domain
field to url
, dns.question
, source
, destination
, client
, and server
. #542, #572group.domain
field. #547url.extension
. #551, #573observer.name
and observer.product
. #557, #571dns.question.subdomain
field. #561, #574error.stack_trace
field. #562log.origin.file.name
, log.origin.function
and log.origin.file.line
fields. #563, #568service.node.name
to allow distinction between different nodes of the same service running on the same host. #565error.type
field. #566Published by webmat about 5 years ago
as
fields for Autonomous System information (i.e. ASN). #341.bytes
fields and event.duration
. #385, #425hash.*
field set. #426dns.*
field set, to describe DNS traffic. #438event.code
, event.sequence
and event.provider
. #439file.name
and file.directory
. #441file.created
, and file.accessed
. #445process.uptime
and host.uptime
fields. #477domain
field to user. #486.nat.ip
and .nat.port
to source
, destination
, client
and server
. #491process.thread.name
field. #517trace.id
and transaction.id
fields for tracing across different services. #519log.logger
field. #521file
fields. #441service.id
description so it works better for clustered services. #502Published by webmat over 5 years ago
process.pid
and process.ppid
. #464, #470Published by webmat over 5 years ago
ECS is turning 1.0.0!
We've clarified a few field descriptions and examples, based on your feedback and questions.
user.group
keyword
field, introduced in #204. Instead,group
field set can be nested at user.group
. #308user.group
, insteadcloud.provider
. #330, #348url.port
type to long
. #339http
field set to url
field set. #330@timestamp
and event.created
. #329object_type
for field labels
. #331geo
field set. Not necessarily geo-ip based, since geo.name
. #333Published by ruflin almost 6 years ago
This is the second 1.0.0 pre-release of ECS. From 1.0.0-beta2 to 1.0.0 GA, no further breaking changes and no additions or new fields are planned. All new contributions must go into the master branch.
device.*
fields to observer.*
fields to eliminate user confusion. #238network.total.bytes
to network.bytes
and network.total.packets
network.packets
. #179network.inbound.bytes
, network.inbound.packets
,network.outbound.bytes
and network.outbound.packets
. #179event.type
definition to be only reserved. #242host.name
field and clarify usage of host.hostname
. #187event.start
and event.end
date fields. #185process.thread.id
field. #200host.name
field and clarify usage of host.hostname
.event.start
and event.end
date fields.related
field set with related.ip
. #206user.group
field. #204group
field set with group.id
and group.name
. #203url.full
field. #207process.executable
field. #209process.working_directory
and process.start
. #215http
. #237
http.response.body
to http.response.body.content
. #239http.request.body.content
. #239user.full_name
field. #201network.community_id
field. #208geo.country_name
and geo.region_iso_code
. #214event.kind
and event.outcome
. #242client
and server
objects and fields. #236user_agent
field set. #240, #262geo.name
for ad hoc location names. #248event.timezone
to allow for proper interpretation of incomplete timestamps. #258source.address
, destination.address
, client.address
, andserver.address
. #247os.full
to capture full OS name, including version. #259event.category
and event.action
. #242network.direction
. #212source.bytes
, source.packets
, destination.bytes
and destination.packets
. #179network.transport
, network.type
, network.application
,network.protocol
must be lowercase. #251http.request.method
must be lowercase. #251