streamalert
StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
APACHE-2.0 License
Published by ryandeivert over 4 years ago
StreamAlert Release v3.2.0
Highlights
Cross-Account Lambda Output
The previously implemented AWS Lambda output was unable to perform cross-account execution of Lambda functions. The V2 version of the Lambda output now supports performing an AssumeRole call prior to invoking the Lambda output, enabling this sort of functionality.
A complete list of changes in v3.2.0 can be found here
Published by ryandeivert over 4 years ago
StreamAlert Release v3.1.0
New Features
Scheduled Queries
The concept of "stateful" alerting has always been a gap that StreamAlert has failed to bridge. We've introduced a feature we've dubbed Scheduled Queries as a way to help bridge that gap. Users can now write and deploy Athena queries that will run on a user-defined schedule. The results of these queries are then fed data back into StreamAlert's Rules Engine for further processing and alerting. See the documentation for more information on getting up and running with Scheduled Queries.
See also: #1209
Dynamic Outputs in Rules
It is now possible for rules to dynamically configure outputs based on information in a record. A new keyword argument of dynamic_outputs has been added to the @rule decorator to support this. For more information on how to leverage this for yourself, see the documentation. This is great addition that we've also wanted for a long time, so a huge thank you to @jack1902 for adding this!
AWS Simple Email Service Output
Support has been added for sending alerts to AWS Simple Email Service (SES). This enables sending richly formatted emails to recipients, as opposed to the previous method of using AWS SNS for sending only very simple emails. A huge thanks to @jack1902 for contributing this!
Microsoft Teams Output
Support has also been added for sending alerts to Microsoft Teams. A huge thanks (again!) to @jack1902 for contributing this!
Publisher Integration Tests
The Publishers testing implementation has been updated to support configuring tests for publishers directly within a test event file. For more information on how to add tests for Publishers, see the documentation.
See also: #1185
Improvements
Parquet for Data Retention
One of our biggest pain points in the StreamAlert ecosystem has been the speed of searches. This release adds support for Parquet as the storage format of data sent to S3 for historical data retention, and we're already seeing vast improvements in comparison to JSON. In addition to this, Athena tables are also now created and managed via Terraform, removing the need for users to reason about them during deployment time.
See also: #1202
Rule Integration Tests
In addition to the updates to integration tests made as part of #1181, a larger update to the framework has migrated tests out of the tests/integration directory. Integration test files for rules should now live beside the rule being tested. The documentation for tests includes more details.
New Rules
AWS Config Compliance and Remediation Rules
Thanks to @jack1902 for adding two new rules related to AWS Config!
SSH Activity via osquery
Thanks to @chunyong-lin for open-sourcing a rule to alert on SSH login activity captured by osquery.
Bug Fixes
To view the complete list of all of the bugs fixed in v3.1.0, including many not mentioned above, see here.
All Changes
To view the complete list of all changes included in v3.1.0, see here.
Published by ryandeivert over 4 years ago
StreamAlert Release v3.0.0
New Features
LookupTables v2
In StreamAlert version 2, we added undocumented support for utilizing files stored in S3 as a method to dynamically load information for use within rules. This release completely redesigns LookupTables and adds new support for DynamoDB as a backend storage mechanism. The big driver behind this decision was the need to alleviate the high memory consumption that came with reading and deserializing JSON files from S3, among other benefits. Additionally, the new implementation will enable easily extending LookupTables to support other, arbitrary backend types.
Terraform Remote State File Locking
Given that StreamAlert utilizes a shared, remote state file that is stored in S3, it was effectively possible for multiple users to be reading and writing a state file at the same time. This could lead to state file corruption, and the potential to botch your entire deployment. In this release, we have added support for remote state file locking, via DynamoDB, to ensure that this type of corruption never occurs.
Improvements
Python 3.7 Support
Perhaps the most important part of this release is the full conversion of the codebase to Python 3.7. This has been a long time coming and we know that many have been patiently awaiting this. The Getting Started instructions have been updated to be specific to Python 3, and there is no backward compatibility to Python 2.7. See the note below on migration. A huge thanks to @GarretReece and @btonic for their support in this endeavor.
See also: #986
Terraform 0.12 Support
The release of the new version of Terraform introduced support for proper for each loops and "dynamic" blocks, along with numerous other benefits. These features have already enabled us to fix longstanding bug in our S3 event notifications module, and we will continue to leverage the benefits of Terraform version 12 to improve our Terraform modules going forward. A huge thanks to @scoders for their support in making this possible.
Separated Schema Files
This release introduces the ability to break up your logs.json file into multiple different files, nested within a schemas directory. It is now possible to have, for example, definition files of schemas/osquery.json and/or schemas/aws.json to independently house all of your osquery and aws related log definitions, respectively. This feature is currently backwards compatible, so if your logs.json is not unmanageably large, or you are not ready to make the change, then your current definitions will continue to function just fine.
Moved Sources into Cluster Files
Version 3 of StreamAlert includes the migration of the cluster source definitions to reside within the individual cluster configuration files. Be sure to check the new format in the StreamAlert docs on the Datasource Configuration page and update your deployment configs accordingly.
To view the complete list of all of the improvements in v3.0.0, including many not mentioned above, see here.
Notable Fixes
AWS Resource Prefix Consistency
This release includes sweeping changes to correct any and all discrepancies related to prefixing or namespacing with the streamlert name. All IAM roles are now created under a valid namespace, each Kinesis Data Firehose is now prefixed properly, and even the pesky “stream_alert” naming convention has been thoroughly scrubbed in this update. We know that these changes may lead to many complications with upgrading, so please see the note below on migration.
See also: #1013, #1064
Alert Merger Optimization
The alert merger function has been updated to better handle massive alert throughput, reducing the potential for out-of-memory exceptions in the function.
Terraform Refactors
Various Terraform modules have also been refactored to reduce complexity or redundancy.
See also: #1041, #1069
StreamAlert Apps Updates
The G Suite StreamAlert Apps have been updated to support newer versions of the Python client library for Google. Additionally, the Box StreamAlert App dependencies have been updated to an official, non-alpha release that supports JSON Web Tokens (JWT) for authentication.
Bug Fixes
To view the complete list of all of the bugs fixed in v3.0.0, including many not mentioned above, see here.
All Changes
To view the complete list of all changes included in v3.0.0, see here.
Migration Steps
There are many changes in this release that will make it prohibitively difficult to do a direct "upgrade" from version 2 to version 3. Therefore, we recommend updating your prefix setting and doing a complete redeploy of StreamAlert version 3. This should be possible alongside a current deployment, thanks in part to the changes that are included in release 3.
After deploying version 3, you must also migrate over your data sources/producers as you see fit. This step is beyond the scope of the advice we're willing to provide here, but please reach out to us if you are stuck and need assistance.
At a minimum, any rules will need to be updated for new import paths (stream_alert --> streamalert) along with any Python 2 vs Python 3 syntax changes. We recommend using the Python 2to3 tool on all custom rules to begin your conversion, which should address most of the necessary changes.
Thanks
We would also like to give a special shout out to @jack1902 and @0xdabbad00 for their contributions to version 3.
Published by Ryxias about 5 years ago
A complete list of changes in v2.2.1 can be found here
Highlighted Changes
- Deployments of streamalert now require a certain configuration variable to specify SQS prefix names. This is not a breaking change, but will require specifying the configuration or Terraform builds will not work. Refer to this PR (#960) for more details.
Published by ryandeivert about 6 years ago
StreamAlert Release v2.0.0
New Features
Alert Merging
This release brings new support for merging similar alerts. The rate at which alerts can be triggered is occasionally unpredictable and be a potential pain for analysts down stream. Combing through a massive amount of alerts becomes tiresome, and could easily result in missing the needle in the haystack. We have introduced an alert merging feature that allows rules to dictate how numerous alerts generated by the rule in a given timeframe are merged together into a single, unified alert. This feature has proven to greatly reduce alert overload. See the rules page of the documentation for how to configure alert merging. See also: #612, #640, #642, #666
Rule Baking/Staging
Release 2.0.0 introduces a new feature we’ve appropriately deemed Rule Baking. Often, new rules have not been battle tested and need to be tuned over time to prevent alert fatigue. This feature introduces a staging period for new rules, along with a daily email digest. The digest includes statistics on alerts that have fired for staged rules, allowing analysts to get metrics on new rules before they go into production. As an added bonus, any rule, new or old, can be toggled into and out of staging on-demand without any code changes.
SNS & SQS Alerting Outputs
This release introduces support for sending alerts to AWS SNS (Simple Notification Service) and SQS (Simple Queue Service). SNS is ideal for sending email or text messages when alerts occur, while SQS better enables downstream and reliable alert processing by other tools.
Slack App
StreamAlert now supports reading files from S3 that contain lookup information to be used within rules. These JSON files can be gzip compressed, and will be refreshed every few minutes to ensure the AWS Lambda container always has an up-to-date intelligence set.
Salesforce App
Salesforce is a commonly used CRM service, and in the latest version of their API (at the time of this writing), it supports 42 types of event logs, and auditing/alerting on these events is no simple task. The new StreamAlert App will collect Console, Login, LoginAs, Report and ReportAs event logs and send to StreamAlert for rule processing and alerting.
Carbon Black Hash Banning
Thanks to @fusionrace for adding support for automated banning of potential malicious files via the new Carbon Black alert output. This feature enables rules to specify md5 hashes which should be banned, and the output handles the rest.
Lookup Tables via S3
StreamAlert now supports reading files from S3 that contain lookup information to be used within rules. These JSON files can be gzip compressed, and will be refreshed every few minutes to ensure the AWS Lambda container always has an up-to-date intelligence set.
Log Parsing Features
Occasionally, logs can be a JSON object containing an embedded csv log of interest. This is the case with AWS RDS logs that are captured via CloudWatch Logs log group and forwarded on to another destination with a CloudWatch Logs Subscription. This version of StreamAlert introduces the ability to extract embedded csv from a JSON object and parse it accordingly. It also adds support for encoded JSON within another JSON object. Various other added benefits were introduced as well; see also: #744, #745
To view a list of all of the new features, see here. Many of these features are sparsely documented at the moment, but look out for many updates to documentation in the coming weeks!
Improvements
Terraform Consolidation
A huge thanks to @austinbyers for refactoring the redundant Terraform Lambda code into a centralized module. The consolidation has enabled us to build new Lambda functions faster and reduced the amount of tests we need to write. The new module is easily adaptable and most of StreamAlert’s Lambda functions have already been ported to leverage it. See also: #638
Alert Processor Consolidation
As part of the move toward alert merging, ‘clustered’ alert processors became unnecessary and overly complex. This change removes multiple alert processors (one for each cluster) in favor of a single processor for all alerts. The architecture of StreamAlert has become more intuitive and simplified thanks to this change.
Rule Test Templates
Up until now, writing tests for rules in StreamAlert requires a copy of a 'real' log that will classify properly in the rules engine. This can be a little arduous if your rule only tests one or two conditions. This release introduce new functionality, where a user can supply a partial test event, and the testing framework will ‘intelligently’ fill the rest of the fields in with default data.
Firehose Optimizations
Batches of records being sent to AWS Firehose, for historical data retention, will now only have records that failed to send retried instead of the entire batch. Additionally, the data will only be serialized to JSON once at the start of the sending, instead of on every attempt. The client has also been tuned with connect and read timeouts to reduce latency when sending batches. See also: #649, #736
Threat Intel Improvements
This release introduces the ability to exclude specific indicators of compromise from the Threat Intel lookups. For instance, IP addresses that are known-goods could be omitted when querying the threat intel DynamoDB table. This can speed up rule processing and reduce cost. The ThreatIntel downloader has also been updated to support filtering out bad or unactionable indicators at the source, before they even land in DynamoDB. See also: #684
General Improvements(https://github.com/airbnb/streamalert/pull/725)
-
#604 - Speeding up Lambda function deploys by using the
-auto-approveflag with terraform. - #652 - Reducing the amount of API requests made to PagerDuty during PagerDuty incident creation. This change prevents potential throttling by the API.
- #653 - More efficient rule helper functions
- #691 - Backoff (aka retry) decorators are used fairly heavily throughout StreamAlert. This release centralizes the backoff handlers to be reused and reduces the amount of duplicate code & tests.
-
#707 - Almost every new component added to StreamAlert needs to load the configuration from disk. This release introduces shared logic for config loading that removes the need to write yet another
load_configfunction. - #725 - Alert processor status logging is now greatly simplified. Those writing new alert outputs no longer have to reason about success or failure logging.
- #784 - S3 server-side encryption adds a layer of protection to data in S3 buckets, and release 2.0.0 introduces this as the default for StreamAlert.
- #804 - Deduplicating gsuite events, thanks to @stoggi.
- #806 - Updating Athena partition refresh function to use SQS as a source. See also: #808, #810, #815
-
#818 - Logger instantiation consolidated to
sharedmodule.
To view the complete list of all of the improvements in v2.0.0, including many not mentioned above, see here.
Bug Fixes
-
#623 - Python
absolute importwarning within CloudWatch Logs during Lambda invocations. Fixes: #267 -
#657 - Long-standing and sporadic
Too many open filesbug that was never easily tracked down. Fixes: #587 - #667 - Large amounts of memory utilized during data normalization.
-
#683 -
TooManyUpdateserror when saving SSM parameter in StreamAlert Apps. Fixes: #682 - #688 - Unsupported types in log would not cause the log’s classification to fail. Fixes: #676
- #731 - Slack messages could be split on each character if no newline character existed, causing hundreds of message to be sent to Slack.
- #732 - Shredding the Lambda tmp directory before downloading S3 objects to prevent disk space from filling up.
-
#735 - Catching
KeyErrorduring classification that could occur if log is misclassified as the wrong type. - #741 - Supporting space-delimited logs in the csv parser by casting the unicode delimiter value to a string.
To view the complete list of all of the bugs fixed in v2.0.0, including many not mentioned above, see here.