StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
APACHE-2.0 License
Published by ryandeivert over 4 years ago
The previously implemented AWS Lambda output was unable to perform cross-account execution of Lambda functions. The V2 version of the Lambda output now supports performing an AssumeRole
call prior to invoking the Lambda output, enabling this sort of functionality.
A complete list of changes in v3.2.0 can be found here
Published by ryandeivert over 4 years ago
The concept of "stateful" alerting has always been a gap that StreamAlert has failed to bridge. We've introduced a feature we've dubbed Scheduled Queries as a way to help bridge that gap. Users can now write and deploy Athena queries that will run on a user-defined schedule. The results of these queries are then fed data back into StreamAlert's Rules Engine for further processing and alerting. See the documentation for more information on getting up and running with Scheduled Queries.
See also: #1209
It is now possible for rules to dynamically configure outputs based on information in a record. A new keyword argument of dynamic_outputs
has been added to the @rule
decorator to support this. For more information on how to leverage this for yourself, see the documentation. This is great addition that we've also wanted for a long time, so a huge thank you to @jack1902 for adding this!
Support has been added for sending alerts to AWS Simple Email Service (SES). This enables sending richly formatted emails to recipients, as opposed to the previous method of using AWS SNS for sending only very simple emails. A huge thanks to @jack1902 for contributing this!
Support has also been added for sending alerts to Microsoft Teams. A huge thanks (again!) to @jack1902 for contributing this!
The Publishers testing implementation has been updated to support configuring tests for publishers directly within a test event file. For more information on how to add tests for Publishers, see the documentation.
See also: #1185
One of our biggest pain points in the StreamAlert ecosystem has been the speed of searches. This release adds support for Parquet as the storage format of data sent to S3 for historical data retention, and we're already seeing vast improvements in comparison to JSON. In addition to this, Athena tables are also now created and managed via Terraform, removing the need for users to reason about them during deployment time.
See also: #1202
In addition to the updates to integration tests made as part of #1181, a larger update to the framework has migrated tests out of the tests/integration
directory. Integration test files for rules should now live beside the rule being tested. The documentation for tests includes more details.
Thanks to @jack1902 for adding two new rules related to AWS Config!
Thanks to @chunyong-lin for open-sourcing a rule to alert on SSH login activity captured by osquery.
To view the complete list of all of the bugs fixed in v3.1.0, including many not mentioned above, see here.
To view the complete list of all changes included in v3.1.0, see here.
Published by ryandeivert over 4 years ago
In StreamAlert version 2, we added undocumented support for utilizing files stored in S3 as a method to dynamically load information for use within rules. This release completely redesigns LookupTables
and adds new support for DynamoDB as a backend storage mechanism. The big driver behind this decision was the need to alleviate the high memory consumption that came with reading and deserializing JSON files from S3, among other benefits. Additionally, the new implementation will enable easily extending LookupTables
to support other, arbitrary backend types.
Given that StreamAlert utilizes a shared, remote state file that is stored in S3, it was effectively possible for multiple users to be reading and writing a state file at the same time. This could lead to state file corruption, and the potential to botch your entire deployment. In this release, we have added support for remote state file locking, via DynamoDB, to ensure that this type of corruption never occurs.
Perhaps the most important part of this release is the full conversion of the codebase to Python 3.7. This has been a long time coming and we know that many have been patiently awaiting this. The Getting Started instructions have been updated to be specific to Python 3, and there is no backward compatibility to Python 2.7. See the note below on migration. A huge thanks to @GarretReece and @btonic for their support in this endeavor.
See also: #986
The release of the new version of Terraform introduced support for proper for each
loops and "dynamic" blocks, along with numerous other benefits. These features have already enabled us to fix longstanding bug in our S3 event notifications module, and we will continue to leverage the benefits of Terraform version 12 to improve our Terraform modules going forward. A huge thanks to @scoders for their support in making this possible.
This release introduces the ability to break up your logs.json
file into multiple different files, nested within a schemas
directory. It is now possible to have, for example, definition files of schemas/osquery.json
and/or schemas/aws.json
to independently house all of your osquery and aws related log definitions, respectively. This feature is currently backwards compatible, so if your logs.json
is not unmanageably large, or you are not ready to make the change, then your current definitions will continue to function just fine.
Version 3 of StreamAlert includes the migration of the cluster source definitions to reside within the individual cluster configuration files. Be sure to check the new format in the StreamAlert docs on the Datasource Configuration page and update your deployment configs accordingly.
To view the complete list of all of the improvements in v3.0.0, including many not mentioned above, see here.
This release includes sweeping changes to correct any and all discrepancies related to prefixing or namespacing with the streamlert
name. All IAM roles are now created under a valid namespace, each Kinesis Data Firehose is now prefixed properly, and even the pesky “stream_alert” naming convention has been thoroughly scrubbed in this update. We know that these changes may lead to many complications with upgrading, so please see the note below on migration.
See also: #1013, #1064
The alert merger function has been updated to better handle massive alert throughput, reducing the potential for out-of-memory exceptions in the function.
Various Terraform modules have also been refactored to reduce complexity or redundancy.
See also: #1041, #1069
The G Suite StreamAlert Apps have been updated to support newer versions of the Python client library for Google. Additionally, the Box StreamAlert App dependencies have been updated to an official, non-alpha release that supports JSON Web Tokens (JWT) for authentication.
To view the complete list of all of the bugs fixed in v3.0.0, including many not mentioned above, see here.
To view the complete list of all changes included in v3.0.0, see here.
There are many changes in this release that will make it prohibitively difficult to do a direct "upgrade" from version 2 to version 3. Therefore, we recommend updating your prefix
setting and doing a complete redeploy of StreamAlert version 3. This should be possible alongside a current deployment, thanks in part to the changes that are included in release 3.
After deploying version 3, you must also migrate over your data sources/producers as you see fit. This step is beyond the scope of the advice we're willing to provide here, but please reach out to us if you are stuck and need assistance.
At a minimum, any rules
will need to be updated for new import paths (stream_alert
--> streamalert
) along with any Python 2 vs Python 3 syntax changes. We recommend using the Python 2to3 tool on all custom rules to begin your conversion, which should address most of the necessary changes.
We would also like to give a special shout out to @jack1902 and @0xdabbad00 for their contributions to version 3.
Published by Ryxias about 5 years ago
A complete list of changes in v2.2.1 can be found here
Published by ryandeivert about 6 years ago
StreamAlert Release v2.0.0
This release brings new support for merging similar alerts. The rate at which alerts can be triggered is occasionally unpredictable and be a potential pain for analysts down stream. Combing through a massive amount of alerts becomes tiresome, and could easily result in missing the needle in the haystack. We have introduced an alert merging feature that allows rules to dictate how numerous alerts generated by the rule in a given timeframe are merged together into a single, unified alert. This feature has proven to greatly reduce alert overload. See the rules page of the documentation for how to configure alert merging. See also: #612, #640, #642, #666
Release 2.0.0 introduces a new feature we’ve appropriately deemed Rule Baking. Often, new rules have not been battle tested and need to be tuned over time to prevent alert fatigue. This feature introduces a staging period for new rules, along with a daily email digest. The digest includes statistics on alerts that have fired for staged rules, allowing analysts to get metrics on new rules before they go into production. As an added bonus, any rule, new or old, can be toggled into and out of staging on-demand without any code changes.
This release introduces support for sending alerts to AWS SNS (Simple Notification Service) and SQS (Simple Queue Service). SNS is ideal for sending email or text messages when alerts occur, while SQS better enables downstream and reliable alert processing by other tools.
StreamAlert now supports reading files from S3 that contain lookup information to be used within rules. These JSON files can be gzip compressed, and will be refreshed every few minutes to ensure the AWS Lambda container always has an up-to-date intelligence set.
Salesforce is a commonly used CRM service, and in the latest version of their API (at the time of this writing), it supports 42 types of event logs, and auditing/alerting on these events is no simple task. The new StreamAlert App will collect Console, Login, LoginAs, Report and ReportAs event logs and send to StreamAlert for rule processing and alerting.
Thanks to @fusionrace for adding support for automated banning of potential malicious files via the new Carbon Black alert output. This feature enables rules to specify md5 hashes which should be banned, and the output handles the rest.
StreamAlert now supports reading files from S3 that contain lookup information to be used within rules. These JSON files can be gzip compressed, and will be refreshed every few minutes to ensure the AWS Lambda container always has an up-to-date intelligence set.
Occasionally, logs can be a JSON object containing an embedded csv log of interest. This is the case with AWS RDS logs that are captured via CloudWatch Logs log group and forwarded on to another destination with a CloudWatch Logs Subscription. This version of StreamAlert introduces the ability to extract embedded csv from a JSON object and parse it accordingly. It also adds support for encoded JSON within another JSON object. Various other added benefits were introduced as well; see also: #744, #745
To view a list of all of the new features, see here. Many of these features are sparsely documented at the moment, but look out for many updates to documentation in the coming weeks!
A huge thanks to @austinbyers for refactoring the redundant Terraform Lambda code into a centralized module. The consolidation has enabled us to build new Lambda functions faster and reduced the amount of tests we need to write. The new module is easily adaptable and most of StreamAlert’s Lambda functions have already been ported to leverage it. See also: #638
As part of the move toward alert merging, ‘clustered’ alert processors became unnecessary and overly complex. This change removes multiple alert processors (one for each cluster) in favor of a single processor for all alerts. The architecture of StreamAlert has become more intuitive and simplified thanks to this change.
Up until now, writing tests for rules in StreamAlert requires a copy of a 'real' log that will classify properly in the rules engine. This can be a little arduous if your rule only tests one or two conditions. This release introduce new functionality, where a user can supply a partial test event, and the testing framework will ‘intelligently’ fill the rest of the fields in with default data.
Batches of records being sent to AWS Firehose, for historical data retention, will now only have records that failed to send retried instead of the entire batch. Additionally, the data will only be serialized to JSON once at the start of the sending, instead of on every attempt. The client has also been tuned with connect and read timeouts to reduce latency when sending batches. See also: #649, #736
This release introduces the ability to exclude specific indicators of compromise from the Threat Intel lookups. For instance, IP addresses that are known-goods could be omitted when querying the threat intel DynamoDB table. This can speed up rule processing and reduce cost. The ThreatIntel downloader has also been updated to support filtering out bad or unactionable indicators at the source, before they even land in DynamoDB. See also: #684
-auto-approve
flag with terraform.load_config
function.shared
module.To view the complete list of all of the improvements in v2.0.0, including many not mentioned above, see here.
absolute import
warning within CloudWatch Logs during Lambda invocations. Fixes: #267Too many open files
bug that was never easily tracked down. Fixes: #587TooManyUpdates
error when saving SSM parameter in StreamAlert Apps. Fixes: #682KeyError
during classification that could occur if log is misclassified as the wrong type.To view the complete list of all of the bugs fixed in v2.0.0, including many not mentioned above, see here.