IDMEF for Go

Intrusion Detection Message Exchange Format

Go library for authoring and parsing data in IDMEF format (IETF RFC 4765).

Why IDMEF?

IDMEF has a few characteristics which make it attrative to use.

1. It is an IETF RFC
2. It is well-defined
3. It is vendor agnostic

Some SIEMs can use it directly and it can be used as an intermediate format to translate vendor specific events to vendor specific consumers. Some formats of consideration are AWS CloudTrail (both CloudTrail>IDMEF and IDMEF>CloudTrail) and Google Chronicle (IDMEF>Chronicle).

Usage

There are two sets of Message structs, one for authoring and one for parsing. The reason is due to Go's lack of support for parsing XML with tag prefixes.

Note on JSON: since these are provided as Go structs, it is very easy to convert to JSON. Today, there are no JSON tags but they may be added in the future after some investigation. The things to investigate are use of camelCase vs. PascalCase for JSON property names: (a) whether both should be used to map to the XML structure or (b) whether only one should be used (likely lower camelCase) to be consistent with JSON.

Authoring

Use the go-idmef (idmef) package structs to create the idmef.Message struct and then call xml.Marshal() or idmef.Message.Bytes().

Example messages from the RFC are available in the testdata folder in both XML and Go code. These are compared in the tests](unmarshal/)

Parsing

See unmarshal.ReadFile() function for an example to parse aa IDMEF XML file.

Coverage

This list shows the defined objects. See the RFC model overview for reference.

• IDMEF-Message
  • Alert
    • Analyzer
      • Node
        • Address
      • Process
    • CreateTime
    • DetectTime
    • AnalyzerTime
    • Source
      • Node
        • Address
      • User
        • UserId
      • Process
      • Service
    • Target
      • Node
        • Address
      • User
        • UserId
      • Process
      • Service
      • File
        • FileAccess
          • Permission
        • Linkage
    • Classification
      • Reference
    • Assessment
    • CorrelationAlert
    • AdditionalData
  • Heartbeat
    • Analyzer
    • CreateTime
    • AdditionalData

Examples

The examples in RFC 4765 are included and tested in this repo. Go and XML representations are provided, parsed and compared. The following is a lists of the examples in RFC 4765. RFC descriptions are provided.

• Denial-of-Service Attacks: The following examples show how some common denial-of-service attacks could be represented in the IDMEF.
  • The "teardrop" Attack (xml): Network-based detection of the "teardrop" attack. This shows the basic format of an alert.
  • The "ping of death" Attack (xml): Network-based detection of the "ping of death" attack. Note the identification of multiple targets, and the identification of the source as a spoofed address. NOTE: The URL has been cut to fit the IETF formating requirements.
• Port Scanning Attacks: The following examples show how some common port scanning attacks could be represented in the IDMEF.
  • Connection to a Disallowed Service (xml): Host-based detection of a policy violation (attempt to obtain information via "finger"). Note the identification of the target service, as well as the originating user (obtained, e.g., through RFC 1413).
  • Simple Port Scanning (xml): Network-based detection of a port scan. This shows detection by a single analyzer; see Section 7.5 for the same attack as detected by a correlation engine. Note the use of to show the ports that were scanned.
• Local Attacks: The following examples show how some common local host attacks could
  be represented in the IDMEF.
  • The "loadmodule" Attack (xml): Host-based detection of the "loadmodule" exploit. This attack involves tricking the "loadmodule" program into running another program; since "loadmodule" is set-user-id "root", the executed program runs with super-user privileges. Note the use of and to identify the user attempting the exploit and how he's doing it.
  • The "loadmodule" Attack with root target user (xml): The Intrusion Detection System (IDS) could also indicate that the target user is the "root" user, and show the attempted command; the alert might then look like:
  • The "phf" Attack (xml): Network-based detection of the "phf" attack. Note the use of the element to provide more details about this particular attack.
  • File Modification (xml): Host-based detection of a race condition attack. Note the use of the to provide information about the files that are used to perform the attack.
• System Policy Violation (xml): In this example, logins are restricted to daytime hours. The alert reports a violation of this policy that occurs when a user logs in a little after 10:00 pm. Note the use of to provide information about the policy being violated.
• Correlated Alerts (xml): The following example shows how the port scan alert from Section 7.2.2 could be represented if it had been detected and sent from a correlation engine, instead of a single analyzer.
• Analyzer Assessments (xml): Host-based detection of a successful unauthorized acquisition of root access through the eject buffer overflow. Note the use of to provide information about the analyzer's evaluation of and reaction to the attack.
• Heartbeat (xml): This example shows a Heartbeat message that provides "I'm alive and working" information to the manager. Note the use of elements, with "meaning" attributes, to provide some additional information.

TODO

1. Furhter dtail struct

Notes

1. idmef is the authoring package and creates XML with the idmef tag.
2. unmarshal is the parsing package which reads in XML files but does not support the idmef tag prefix due to Go issue 9519. Unmarshal or parse a file using unmarshal to receive a *unmarshal.Message which can then be converted to an authoring struct with *unmarshal.Message.Common().

References

IDMEF

• IETF RFC 4765: Format Details
• IETF RFC 4766: Format Requirements
• IETF RFC 4767: Recommended Transport Protocol (IDXP)
• IDMEF on Wikipedia
• XML Schema-Based Minification for Communication of Security Information and Event
• awesome-idmef

Alternative Formats

• OSSEC: https://www.ossec.net/docs/formats/alerts.html
• OSSEM: https://github.com/OTRF/OSSEM

Go XML situation

1. encoding/xml: support for XML namespace prefixes
2. xml namespace prefix issue at go: "To fix that you need to use two structs, one for Unmarshalling and second to Marshalling data"
3. Unable to parse xml in GO with : in tags

Other Implementations

1. PHP - https://github.com/fpoirotte/php-idmef

Comparisons

1. Power of the IDMEF format
2. SDEE vs IDMEF?
3. IDMEF 'Lingua Franca' for Security Incident Management by Douglas Corner
4. Security Log Standard: Still an Open Question

Credits

1. timestamp.Timestamp is based on code from github.com/coreos/mantle under the Apache 2.0 license and MIT compatible. This is a large, archived codebase with many dependencies.
2. diffmatchpatch from github.com/sergi/go-diff was used during development to analyze failed test results.