Highlights

• Added debug-iter-topology in makefile to debug running topology
• Added more stable topologies with iter-files-mails, that uses generator to send mails to topology. It's memory safe and stable
• Fixed many bug to avoid topology restarts
• Added three new topologies that use new spout

Highlights

• Resolved many bugs
• Getting only main headers fields from mails. This number can grow very fast.
• Added a field headers with all headers.
• Upgraded mail-parser and streamparse.
• Disabled Zemana integration from code (deprecated).

Highlights

• Added module that analyze SMTP dialects. Added dialect section in main configuration file
• Upgraded mail-parser and Apache Tika modules
• Resolved many bugs

Highlights

• Resolved many bug
• Performance improvement
• Added persistent store for RAM runtime filter. Now SpamScope saves the filter stores, so it can reload lists after a reboot
• Upgraded mail-parser library, now SpamScope stores all mail headers
• Added more logs
• Added filter for size for attachments

Highlights

• New documentation, with more details about configuration and installation.
• Added Ansible playbooks to install Apache Storm and SpamScope.
• Added priority for post processing.
• Added timeout for long time analysis function.

Highlights

• Fixed Travis CI configuration.
• New store_samples attachments post processing, to store samples on filesystem.
• Splitted Elasticsearch template for attachments and mails.
• Many fix to Elasticsearch template to upgrade at 6.x.
• Added whitelist to VirusTotal post processing.
• Storing analyzed mails in daily folders.
• Fixed many bugs.

⚠️ Breaking Changes ⚠️

• Changed main configuration file. All lists are under list key, so they are automatically loaded:

# Attachments bolt configuration
attachments:
    # The lists of all components must be under lists keyword to load them
    # automatically
    commons:
        lists:
            blacklist_content_types:
                # All content types to remove from results
                # Example in content_types/blacklist/generic.example.yml
                generic: /path/to/generic_content_types
                custom: /path/to/custom_content_types

            not_extract_content_types:
                # All content types that you don't want extract from archive
                # Example: application/java-archive (jar), you can save the jar
                # but do not extract the class inside.
                generic: /path/to/generic_content_types
                custom: /path/to/custom_content_types

Highlights

• New Travis CI configuration. Added automatic deploy.
• Fixed Elasticsearch template (index.mapping.ignore_malformed).
• Added Dockerfile example: debug images.
• Upgraded mail-parser and streamparse.
• New field to_domains in main report: domains of to header.
• Bugfix

Highlights

• More stable version. Solved three bug: heartbeat timeout when using Thug, getting custom header now not raise pystorm/serializers/json_serializer.py and you can avoid to set blacklist or whitelist in configuration.
• Removed waiting.sleep in configuration spout file to avoid Apache Storm timeout.
• Thug section has more important options.
• Added debug environment in config.json.
• Added docker-compose example for debug use.
• Splitted requirements in optional and not.
• Fixed Thug analysis integration: replaced os.kill with raise. Added more options.

Highlights

• Upgraded requirements
• Upgraded mail-parser
• Upgraded Apache Tika support to 1.16 version
• New installer that manages all Python dependencies. SpamScope is on PyPI
• Command line spamscope-elasticsearch can be used in the cases where Elasticsearch is behind a reverse proxy. You can use RFC-1738 formatted URLs.

⚠️ Breaking Changes ⚠️

• The mail format is different. See mail-parser for more details.

Highlights

• Support Outlook mail (msg format). Enable flag outlook: True in mailboxes main configuration for folders that have this mail format.
• Added SpamAssassin post processing. Every mails can be analyzed from SpamAssassin. Be careful the parsing can be very slow.
• Now you can extract custom headers from mail with headers: list in mailboxes main configuration (see configuration example in this repository).
• Merged urls bolts of body and attachments in one bolt.
• Fixed Elasticsearch template.
• Upgraded mail-parser and streamparse.
• Added headers receiveds in main output.
• The form in mails now are analyzed from phishing bolt.
• New code to analyze raw mail with third part tools.
• The binary attachments have a flag binary.
• SpamScope can extract samples from mailformed zip attachments.
• SpamScope can extract samples from quoted-printable headers mail.

Highlights

• Upgraded requirements: Apache Tika, Streamparse and Shodan
• Resolved bug JsonSerializer in Shodan report. Issue #368 Streamparse
• Resolved bug in Elasticsearch template to manage Shodan report
• Added .editorconfig to setup editor for SpamScope

Highlights

• Auto build docker images with Travis.
• Changed function to make base64 decode.
• Manage incorrect padding errors for base64 attachments and report in results.

Highlights

• Upgraded elasticsearch template.
• Manage HTTPError in Zemana integration.

Highlights

• Upgraded requirements: streamparse and elasticsearch.
• analisys_date field in attachments analysis.
• Bugfixes.

Highlights

• Added new network bolt: now it's possible to analyze the sender ip address.
• Updated Apache Storm version (1.1.0).
• Mark mail as phishing only if there are urls.
• Fixed Thug analysis for files in archive.
• Fixed attachments filter function.

⚠️ Breaking Changes ⚠️

• New flag for attachments: with_raw_attachments. If true mail has attachments.
  If with_attachments is true the mail has attachments that are not filtered.

⚠️ New configuration sections ⚠️

• Added two new fields to tokenizer bolt, to filter analysis of sender ip address:

+    # If True the same ip address is filtered and not analyzed.
 +    filter_network: True
 +
 +    # Max number of hashes saved for filter function
 +    maxlen_network: 1000000

• New section for network bolt:

+# Network bolt configuration
 +network:
 +    shodan:
 +        enabled: False
 +        api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 +
 +    virustotal:
 +        enabled: False
 +        api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Highlights

• Fixed bug Redis client: issue push messages empty.
• Fixed report Zeman: report Zemana only in attachments documents.

Highlights

• Fixed bug template Elasticsearch, to manage geoip and draw sender spam map.

Highlights

• Fixed bug topology Redis. Now it's possible use SpamScope with logstash.

Highlights

• Upgraded requirements: new version of mail-parser.
• Fixed bugs: now more stable.
• New Zemana AntiMalware integration: post processing attachments. New configuration section in attachments bolt.

+    zemana:
 +        enabled: False
 +
 +        PartnerId: xxxxx
 +        UserId: xxxxx
 +        ApiKey: xxxxx
 +        useragent: SpamScope

Highlights

• Upgraded requirements: new versions of mail-parser and tika-app.
• Fixed bugs: now more stable.
• More attachments processes: upgraded parallelism for attachment bolt.