Logprep

Commits

• 1ef1b78: Fix release pipeline (#628) (Jörg Zimmermann) #628

Logprep helm chart

Breaking

• This release limits the maximum python version to 3.12.3 because of the issue
  #612.
• Remove normalizer processor, as it's functionality was replaced by the grokker, timestamper and field_manager processors
• Remove elasticsearch_output connector to reduce maintenance effort

Features

• add a helm chart to install logprep in kubernetes based environments

Improvements

• add documentation about behavior of the timestamper on ISO8601 and UNIX time parsing
• add unit tests for helm chart templates
• add helm to github actions runner
• add helm chart release to release pipeline

Bugfix

• fixes a bug where it could happen that a config value could be overwritten by a default in a later configuration in a multi source config scenario
• fixes a bug in the field_manager where extending a non list target leads to a processing failure
• fixes a bug in pseudonymizer where a missing regex_mapping from an existing config_file causes logprep to crash continuously

Details

• set upper python version limit to <3.12.4 and revise github workflows by @dtrai2 in https://github.com/fkie-cad/Logprep/pull/614
• configuration overwrites with default values by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/613
• field_manager does not writes to a non list target field if extend_target_list by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/616
• remove normalizer processor by @dtrai2 in https://github.com/fkie-cad/Logprep/pull/400
• improve timestamper processor documentation by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/618
• Pseudonymizer late error on non existing regexmapping by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/617
• add helm chart by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/606
• remove elasticsearch_output connector by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/620
• add helm chart release to release pipeline by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/621
• prepare release 13 by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/623

Full Changelog: https://github.com/fkie-cad/Logprep/compare/v12.0.0...v13.0.0

Commits

• 567d546: Pseudonymizer late error on non existing regexmapping (#617) (Jörg Zimmermann) #617

Breaking

• pseudonymizer change rule config field pseudonyms to mapping
• clusterer change rule config field target to source_fields
• generic_resolver change rule config field append_to_list to extend_target_list
• hyperscan_resolver change rule config field append_to_list to extend_target_list
• calculator now adds the error tag _calculator_missing_field_warning to the events tag field instead of _calculator_failure in case of missing field in events
• domain_label_extractor now writes _domain_label_extractor_missing_field_warning tag to event tags in case of missing fields
• geoip_enricher now writes _geoip_enricher_missing_field_warning tag to event tags in case of missing fields
• grokker now writes _grokker_missing_field_warning tag to event tags instead of _grokker_failure in case of missing fields
• requester now writes _requester_missing_field_warning tag to event tags instead of _requester_failure in case of missing fields
• timestamp_differ now writes _timestamp_differ_missing_field_warning tag to event tags instead of _timestamp_differ_failure in case of missing fields
• timestamper now writes _timestamper_missing_field_warning tag to event tags instead of _timestamper_failure in case of missing fields
• rename --thread_count parameter to --thread-count in http generator
• removed --report parameter and feature from http generator
• when using extend_target_list in the field managerthe ordering of the given source fields is now preserved
• logprep now exits with a negative exit code if pipeline restart fails 5 times
  • this was implemented because further restart behavior should be configured on level of a system init service or container orchestrating service like k8s
  • the restart_count parameter is configurable. If you want the old behavior back, you can set this parameter to a negative number
• logprep now exits with a exit code of 2 on configuration errors

Features

• add UCL into the quickstart setup
• add logprep http output connector
• add pseudonymization tools to logprep -> see: logprep pseudo --help
• add restart_count parameter to configuration
• add option mode to pseudonymizer processor and to pseudonymization tools to chose the AES Mode for encryption and decryption
• add retry mechanism to opensearch parallel bulk, if opensearch returns 429 rejected_execution_exception

Improvements

• remove logger from Components and Factory signatures
• align processor architecture to use methods like write_to_target, add_field_to and get_dotted_field_value when reading and writing from and to events
  • required substantial refactoring of the hyperscan_resolver, generic_resolver and template_replacer
• change pseudonymizer, pre_detector, selective_extractor processors and pipeline to handle extra_data the same way
• refactor clusterer, pre_detector and pseudonymizer processors and change rule_tree so that the processor do not require process override
  • required substantial refactoring of the clusterer
• handle missing fields in processors via _handle_missing_fields from the field_manager
• add LogprepMPQueueListener to outsource logging to a separate process
• add a single Queuehandler to root logger to ensure all logs were handled by LogprepMPQueueListener
• refactor http_generator to use a logprep http output connector
• ensure all cached_properties are populated during setup time

Details

• remove logger from signatures by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/589
• add queuelistener for logging by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/590
• remove pex build by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/587
• Align processor architecture by @ppcad in https://github.com/fkie-cad/Logprep/pull/583
• integrate UCL into quickstart setup by @dtrai2 in https://github.com/fkie-cad/Logprep/pull/582
• remove arrow dependency by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/595
• refactor http generator output to http_output connector by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/591
• make extend_target_list preserve ordering by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/593
• add depseudo tools by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/599
• add calling setup during configuration verify by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/601
• fix quickstart compose setup for WSL by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/604
• revise restart behavior by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/602
• add option to pseudonymizer to use different encryption modes by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/600
• add retry mechanism to write_backlog by @dtrai2 in https://github.com/fkie-cad/Logprep/pull/605
• Prepare Release v12 by @dtrai2 in https://github.com/fkie-cad/Logprep/pull/607

Full Changelog: https://github.com/fkie-cad/Logprep/compare/v11.3.0...v12.0.0

Commits

• 7c9b2d1: update CHANGELOG.md (#607) (dtrai2) #607

Commits

• 28d7c54: make extend_target_list preserve ordering (#593) (Jörg Zimmermann) #593

Commits

• 8311e0f: integrate UCL into quickstart setup (#582) (dtrai2) #582

Commits

• 8329dbf: remove pex build (#587) (Jörg Zimmermann) #587

Features

• add gzip handling to http_input connector
• adds advanced logging configuration
  • add configurable log format
  • add configurable datetime formate in logs
  • makes hostname available in custom log formats
  • add fine grained log level configuration for every logger instance

Improvements

• rename logprep.event_generator module to logprep.generator
• shorten logger instance names

Bugfix

• fixes exposing OpenSearch/ElasticSearch stacktraces in log when errors happen by making loglevel configurable for loggers opensearch and elasticsearch
• fixes the logprep quickstart profile

Details

• Handle Opensearch Stacktraces by @dtrai2 in https://github.com/fkie-cad/Logprep/pull/577
• add gzip handling to http_input by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/581
• Add Flush Timeout to s3 Connector by @saegel in https://github.com/fkie-cad/Logprep/pull/553
• rename event generator module by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/585
• advanced logging configuration by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/584
• fix logprep quickstart profile by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/586
• prepare release 11.3.0 by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/588

Full Changelog: https://github.com/fkie-cad/Logprep/compare/v11.2.1...v11.3.0

Commits

• 9b406c4: rename event generator module (#585) (Jörg Zimmermann) #585

Commits

• a247564: Handle Opensearch Stacktraces (#577) (dtrai2) #577

Bugfix

• fixes bug, that leads to spawning exporter http server always on localhost

Details

• fix bug prometheus exporter server uvicorn only on localhost by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/580

Full Changelog: https://github.com/fkie-cad/Logprep/compare/v11.2.0...v11.2.1

Commits

• 2a6d699: fix bug prometheus exporter server uvicorn only on localhost (#580) (Jörg Zimmermann) #580

Features

• expose metrics via uvicorn webserver
  • makes all uvicorn configuration options possible
  • add security best practices to server configuration
• add following metrics to http_input connector
  • nummer_of_http_requests
  • message_backlog_size

Bugfix

• fixes a bug in grokker rules, where common field prefixes wasn't possible
• fixes bug where missing key in credentials file leads to AttributeError

Details

• Fix AttributeError due to missing key in credentials file by @djkhl in https://github.com/fkie-cad/Logprep/pull/574
• Fix grokker rules that have fields with common prefixes by @dtrai2 in https://github.com/fkie-cad/Logprep/pull/571
• export metrics via uvicorn asgi app by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/576
• add http input connector metrics by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/578

Full Changelog: https://github.com/fkie-cad/Logprep/compare/v11.1.0...v11.2.0

Commits

• f5b3d16: prepare release 11.2.0 (#579) (djkhl) #579

Commits

• 3376751: Fix grokker rules that have fields with common prefixes (#571) (dtrai2) #571

11.1.0

Features

• new documentation part with security best practices which compiles to user_manual/security/best_practices.html
  • also comes with excel export functionality of given best practices
• add basic auth to http_input

Bugfix

• fixes a bug in http connector leading to only first process working
• fixes the broken gracefull shutdown behaviour

Details

• fix http connector only first process working by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/566
• Add overview with best practices to docs by @dtrai2 in https://github.com/fkie-cad/Logprep/pull/559
• fix broken gracefull shutdown by @ekneg54 in https://github.com/fkie-cad/Logprep/pull/568
• Add Basic Auth to HTTP Connector by @herrfeder in https://github.com/fkie-cad/Logprep/pull/569

Full Changelog: https://github.com/fkie-cad/Logprep/compare/v11.0.1...v11.1.0

Commits

• d93cd99: Add Basic Auth to HTTP Connector (#569) (David Lassig) #569

Commits

• f66b548: fix broken gracefull shutdown (#568) (Jörg Zimmermann) #568