msticpy

Microsoft Threat Intelligence Security Tools

OTHER License

Downloads
30.1K
Stars
1.8K
View Code on GitHub View on X
Ecosystems: VS Code Extension, TypeScript, Playwright, Windows, Windows UI Library (WinUI)

Bot releases are hidden (Show)

msticpy - AI documentation assistant, BinaryEdge TI provider and other misc fixes Latest Release

Published by ianhelle about 1 month ago

We've been quietly doing some work to introduce LLM/GPT/AI capabilities into msticpy.
@EileenG02 has helped us in that direction by building a document Q&A agent using Autogen.

You can try it out in a notebook using the following:

Load the magic extension

%load_ext msticpy.aiagents.mp_docs_rag_magic

Ask a question in a separate cell using the %%ask cell magic

%%ask
What are the three things that I need to connect to Azure Query Provider?

Awesome work @EileenG02!

There's also a new TI provider for BinaryEdge courtesy of @petebryan.

Alongside this there have been quite a few contributions to fix and improve things like:

  • Splunk improvements (thanks @Tatsuya-hasegawa)
  • Fixes for Sentinel provider get_alert_rules to use updated API (thanks @BWC-TomW)
  • A massive amount of type annotation work and fixes to context/TI providers by @FlorianBracq
  • Miscellaneous fixes to things like Sentinel TI provider, MSSentinel tidy-up to more consistently handle parameters,
    correct use of the term CountryOrRegionName from CountryName in geolocation contexts.

The gory details of the PRs follow:

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.12.0...v2.13.0

msticpy - Splunk and Sentinel Updates

Published by ianhelle 5 months ago

Sentinel updates

WorkspaceConfig and Sentinel QueryProvider (azure_monito_driver) have had a few updates:

  • handle both old (Kqlmagic) and standard connection string formats in WorkspaceConfig
  • removing a lot of legacy code from WorkspaceConfig
  • Allow additional connection parameters to be used with MSSentinel QueryProvider for
    authentication parameters (e.g. you can now supply authentication parameters like "client_id", "client_secret" to query_provider.connect())
  • msticpyconfig.yaml now supports using an "MSSentinel" key in place of "AzureSentinel"
  • Workspace entries in msticpyconfig.yaml support an Args subkey, where you can add authentication parameters - these will be supplied to the connect() method if not overridden on the command line. Like Args sections for other providers, the values here can be text or references to environment variables or Azure Key Vault secrets.
  • Fix to MSSentinel API update_incident to add full properties

Splunk Updates

  • Added jwt authentication token expiry check.

Other fixes

Fix for vtlookup3.py

  • Fixed problematic way of using nestasyncio - this was causing failures when run from a langchain agent.
    Fix for lookup/tilookup
  • If the progress parameter was not passed it would still try to cancel a non-existent progress task and cause an exception.
    QueryProviders
  • Fix split query time-ranges calculation - thanks to @pjain90 for spotting this.

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.11.0...v2.12.0

msticpy - Sentinel Split Query fix

Published by ianhelle 7 months ago

This is a minor release mainly to add a warning for Kusto/Sentinel queries that return partial results.
A close friend of MSTICPy (thx @Cyb3r-Monk) had spotted that MSTICPy does not report partial results when doing split queries so it's possible to lose data from the query range silently.

Due to an unfortunate admin error, the fix for this was committed direct to main, so no PR for this is available. :-(

If you want the query to fail (throw an exception) rather than just warn you can supply a new parameter fail_if_partial.
This only affects the Sentinel query provider and works for standard as well as split queries.

NOTE: the documentation has a typo and calls this fail_on_commit - we'll fix that in the next release to support both fail_if_partial and fail_on_partial

Example

qry_prov.exec_query(query_string, fail_if_partial=True)

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.10.0...v2.11.0

msticpy - v2.10.0

Published by ianhelle 8 months ago

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.9.0...v2.10.0

msticpy - Defender Advanced hunting, IPQualityScore TI provider

Published by ianhelle 11 months ago

Some of the highlights of this release:

IPQualityScore

New TI provider submitted by @petebryan - provides a lot of interesting stats on IPs.

Defender Advanced Hunting API

Thanks to @d3vzer0 our MS Defender client is now able to use the support Graph-based API rather than the legacy
APIs. To use this, for the moment use the DataEnvironment name M365DGraph when you create
query provider. In the next 0.x release we will switch the other aliases for M365D, MDE, MDATP to use this
new interface and deprecate the existing ones.

Startup errors when running in unexpected environments.

init_notebook made some (incorrect) assumptions about when it would be running in a Synapse environment.
Azure Machine Learning have recently changed their default compute to be a Synapse environment.
Fixes here will correct failures due to faulty detection of environment type.

Startup fixes and perf improvements

We've optimized some of the imports done within the package at startup so msticpy should be quicker to
load.

Azure env credentials fix

Although we previously supported the Azure EnvironmentCredential credential type, our implementation allowed
you to use only with ClientID + ClientSecret. The changes allow it to be used with other supported
credential formats - notably username + password and certificate authentication using a certificate file.

Improvements to Entities

Although these are not visible to most people, we try to keep our Entity definitions in sync with the official
Microsoft "V3" entity definitions. We've added a few entity types and updated some of the attributes
to bring this in line, while still allowing backwards compatible attributes to be used.

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.8.0...v2.9.0

msticpy - Stability release

Published by ianhelle about 1 year ago

A few bugs had crept in over the last couple of releases: some due to buggy coding, some due the world moving forward. So, many items in this release are to address these.

Among the feature improvements are the following:

  • Documentation and scripts from @ccianelli22 for creating a MSTICPy install for use in isolated (no Internet) environments. This is super useful for customers operating in sovereign clouds or other air-gapped high-security environments.
  • Added Splunk authentication method using security token rather than username/password - thanks @Tatsuya-hasegawa
  • Query yaml file validation by @FlorianBracq
  • Paging for large CyberReason queries by @FlorianBracq
  • Modern method to obtain cloud-specific URL endpoints for Azure services. Previously, we were relying on msrestazure, which is now deprecated for this purpose. Many thanks to @ccianelli22 for the work to do this.
  • Fix (by me) for a bug I'd introduced with the switch to using Azure-monitor-query library for MS Sentinel. When using a connection string with this new driver, the logic failed to parse and extract details from this correctly. Many thanks to @cindraw for reporting this bug.

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.7.0...v2.8.0

msticpy - 2.8.0 pre-release

Published by ianhelle about 1 year ago

Updated method to dynamically fetch Azure endpoints (rather than relying on deprecated msrestazure).
Updated version of Insight data provider

msticpy - TI Providers, Sentinel/Kusto Drivers, Query Editor

Published by ianhelle about 1 year ago

Main Changes in this release

Two new TI Providers

Two cool new providers to add to the growing family in MSTICPy:

  • CrowdSec is a commercial Malicious IP threat service
    with a free tier for limited threat lookups. (big thanks to @sbs2001 for submitting this)
  • AbuseIPDB - is an open/free provider of threat intel
    on malicious IP addresses, providing a central abuse list to lookup IP addresses that have
    been associated with malicious activity. (big thanks to @rrevuelta for submitting this.)

As with other providers, these are automatically enabled for use if you include settings
for the API keys in your msticpyconfig.yaml

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

In v2.5.0 we introduced replacement drivers for the MS Sentinel/LogAnalytics/Azure Monitor
and Kusto/Azure Data Explorer providers.

The new drivers are based on the Azure SDKs for each data service. You can read the release notes
for them here.

The new drivers give several advantages, like being able to run queries across multiple workspaces
or Kusto clusters in parallel. Splitting large queries by time chunks (split_query_by parameter)
will also run multiple segments in parallel, dramatically speeding up the query. The default
parallelism is 4 simultaneous threads but you can change this (although be wary of the impact
on the data service for highly parallel queries - this may affect other users and services accessing
the data).

The new drivers are now the default drivers for these providers. They are used by default for
the "MSSentinel" and "Kusto" data environment identifiers. For backward compatibility, they will
also continue to support the "MSSentinel_New" and "Kusto_New" identifiers.

To invoke the previous Kqlmagic-based drivers use "MSSentinel_Legacy" or "Kusto_Legacy".

This change also brings a dependency change for MSTICPy. The following packages are now
part of the core installed dependencies:

  • azure-kusto-data
  • azure-monitor-query

Kqlmagic and its dependencies are no longer installed by default but can be installed with the "kql" extra:

python -m pip install msticpy[kql]

See these links to read more about the MSSentinel provider and Kusto providers.

Query Editor

We've added an ipywidgets based query template editor .

note: this is somewhat provisional so please be sure to test and report bugs.

The query editor allows you to edit existing query files or create new ones and helps manage
the various query properties (like parameter definitions) and query metadata.

Check out the documentation on how to use this in the Extending section of the MSTICPy documentation.

Updates to Authentication.

The improvements here mainly affect the AzureData and MicrosoftSentinel classes but'
also bring some improvements to the core authentication - such as being able to specify
the Azure cloud from the az_connect function and authenticate by providing an
AzureCredential.

  • You can now authenticate by supplying an AzureCredential as a credential parameter
    for AzureData and MicrosoftSentinel connect methods.
  • The connect methods for both these classes also support cloud parameter to specify different sovereign clouds
  • The __init__ and connect methods are instrumented with logging to help debug issues:
import msticpy as mp
from msticpy.context.azure.sentinel_core import MicrosoftSentinel

mp.set_logging_level("INFO")
mssentinel = MicrosoftSentinel()
mssentinel.connect()

Other major items

  • MS Sentinel delete watchlist API added by @mbabinski
  • Splunk fixes added by @Tatsuya-hasegawa

Thanks

Our thanks to the following folks who contributed to this release.
@FlorianBracq
@sbs2001
@rrevuelta
@mbabinski
@Tatsuya-hasegawa

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.6.0...v2.7.0

msticpy - DataProviders, QueryEditor, CrowdSec and AbuseIPDB TIProviders

Published by ianhelle about 1 year ago

Preview release of 2.7.0

More detailed release notes in the full release.

Main Changes

Two new TI Providers:

  • CrowdSec (thanks to @sbs2001)
  • AbuseIPDB (thanks to @rrevuelta)

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

These were introduced in v2.5.0 but are now the default drivers for these providers.

Query Editor

ipywidgets based query template editor - this is somewhat provisional so please be sure to test and
report bugs.

Updates to Authentication - esp for the AzureData and MicrosoftSentinel API modules

  • You can now authenticate by supplying an AzureCredential as a credential parameter
  • The connect methods for these support cloud parameter to specify different sovreign clouds
  • The init and connect methods are instrumented with logging to help debug issues:
import msticpy as mp
from msticpy.context.azure.sentinel_core import MicrosoftSentinel

mp.set_logging_level("INFO")
mssentinel = MicrosoftSentinel()
mssentinel.connect()

Other items

  • MS Sentinel delete watchlist API added by @mbabinski
  • Splunk fixes added by @Tatsuya-hasegawa

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.6.0...v2.7.0.pre1

msticpy - v2.6.0 Parallel Queries, Velociraptor data

Published by ianhelle about 1 year ago

The three big changes in this release are:

  • Executing MS Sentinel and Kusto queries in parallel across multiple instance
  • Threaded (parallel) execution of time-split queries
  • Addition of data provider to query local (exported) Velociraptor logs

Many thanks to @d3vzer0 for inspiration and early work on the threaded query feature.
Many thanks @juju4 for inspiration and work on the Velociraptor support.

Support for running a query across multiple connections (with optional threaded operation)

It is common for data services to be spread across multiple tenants or workloads. E.g., multiple Sentinel workspaces,
Microsoft Defender subscriptions or Splunk instances. You can use the MSTICPy QueryProvider to run a query across multiple connections and return the results in a single DataFrame.

To create a multi-instance provider:

  • Create an instance of a QueryProvider for your data source and execute the connect() method to connect to the first instance of your data service.
  • Then use the add_connection() method. This takes the same parameters as the connect() method (the parameters for this method vary by data provider) to add additional instance connections.

add_connection() also supports an alias parameter to allow you to refer to the connection by a friendly name.


    qry_prov = QueryProvider("MSSentinel")
    qry_prov.connect(workspace="Workspace1")
    qry_prov.add_connection(workspace="Workspace2, alias="Workspace2")
    qry_prov.list_connections()

When you now run a query for this provider, the query will be run on all of the connections and the results will be returned as a single dataframe.


    test_query = '''
        SecurityAlert
        | take 5
        '''

    query_test = qry_prov.exec_query(query=test_query)
    query_test.head()

Some of the MSTICPy drivers support asynchronous execution of queries against multiple instances, so that the time taken to run the query is much reduced compared to running the queries sequentially. Drivers that support asynchronous queries will use this automatically. The initial set of multi-threaded drivers are:

  • MSSentinel_New (the new version of the MSSentinel driver)
  • Kusto_New (the new version of the Kusto/Azure Data Explorer driver)

By default, the queries will use at most 4 concurrent threads. You can override this by initializing the QueryProvider with the
max_threads parameter to set it to the number of threads you want. Although you should be cautious
about using too many simultaneous connections due to the potential impact on the cluster performance.


    qry_prov = QueryProvider("MSSentinel", max_threads=10)

Multi-threaded support for split/shared queries

MSTICPy has supported splitting large queries by time-slice for a while. This is documented here Splitting a Query into time chunks. With this release, we've added asynchronous support for this (if the driver supports threaded/async operation) so that multiple chunks of the query will run in parallel.


    qry_prov.SecurityAlert.list_alerts(start=start, end=end, split_by="1d")

Use the parameter split_query_by or split_by to specify a time range (the time unit uses the same syntax as pandas time intervals - e.g. "1D", "4h", etc. - the the pandas documentation for more details on this).

In this release sharding is also supported for ad hoc queries as long as you add "start" and "end" parameters to the query (this is still experimental, so let us know if you have issues with this).

Velociraptor Local Data Provider

The Velociraptor data provider can read Velociraptor log files and provide convenient query functions for each data set in the output logs.

The provider can read files from one or more hosts, stored in in separate folders. The files are read, converted to pandas DataFrames and grouped by table/event. Multiple log files of the same type (when reading in data from multiple hosts) are concatenated into a single DataFrame.

To use the Velociraptor provider, you need to create an QueryProvider instance, passing the string "Velociraptor" (or "VelociraptorLogs") as the data_environment parameter. You also need to add the data_paths parameter to specify specific folders that you want to search for log file (although you can set these paths in msticpyconfig.yaml, if you do this frequently).

You can specify multiple folders to have the logs from different hosts.

    qry_prov = mp.QueryProvider("VelociraptorLogs", data_paths=["~/my_logs"])

Calling the connect method triggers the provider to read the locations of the
log files (although the contents are not read until a query function is run).


    qry_prov.connect()


## Listing Velociraptor tables

```python3
    qry_prov.list_queries()

    ['velociraptor.Custom_Windows_NetBIOS',
    'velociraptor.Custom_Windows_Patches',
    'velociraptor.Custom_Windows_Sysinternals_PSInfo',
    'velociraptor.Custom_Windows_Sysinternals_PSLoggedOn',
   ....

Each query returns the table of data types retrieved from the logs.


    qry_prov.vc_prov.velociraptor.Windows_Forensics_ProcessInfo()
Name PebBaseAddress Pid ImagePathName CommandLine CurrentDirectory Env
LogonUI.exe 0x95bd3d2000 804 C:\Windows\system32\LogonUI.exe "LogonUI.exe" /flags:0x2 /state0:0xa3b92855 /state1:0x41c64e6d C:\Windows\system32\ {'ALLUSERSP
dwm.exe 0x6cf4351000 848 C:\Windows\system32\dwm.exe "dwm.exe" C:\Windows\system32\ {'ALLUSERSP
svchost.exe 0x6cd64d000 872 C:\Windows\System32\svchost.exe C:\Windows\System32\svchost.exe -k termsvcs C:\Windows\system32\ {'ALLUSERSP
svchost.exe 0x7d18e99000 912 C:\Windows\System32\svchost.exe C:\Windows\System32\svchost.exe -k LocalServiceNetworkRestricted C:\Windows\system32\ {'ALLUSERSP
svchost.exe 0x5c762eb000 920 C:\Windows\system32\svchost.exe C:\Windows\system32\svchost.exe -k LocalService C:\Windows\system32\ {'ALLUSERSP

What's Changed

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.5.3...v2.6.0

msticpy - v2.5.3: ipwidgets, Sentinel and Kusto driver fixes

Published by ianhelle over 1 year ago

Minor release addressing the following:

  • Azure-monitor-query release 1.2.0 changed the format of the endpoint URLs that it accepts. Fixed the azure_kusto driver (currently invoked with the "Kusto_New" data environment) so that it will provide the correct format for 1.2.0+ and pre 1.2.0 versions
  • Bug in the kql_driver (MS Sentinel) was causing the kusto_driver to fail when querying. The latter is a subclass of the former and was failing due to an attribute that was defined in the parent (kql_driver) but not in the child (kusto_driver). This affected the older (current) Kusto driver version and does not affect the new azure_kusto ("Kusto_New") driver.
  • Updated requirements to allow ipywidgets 8.x to install by default (this is now supported by vs code
  • Updated documentation for the new Sentinel and Kusto drivers to add instructions for manually installing the required SDK components (azure-monitor-query and azure-kusto-data)

What's Changed

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.5.2...v2.5.3

msticpy - v2.5.2: Hotfix for Holoviz panel compatibility

Published by ianhelle over 1 year ago

  • Release is mainly to align bokeh version requirements with the new release of Holoviz panel.
    • moved bokeh from <3.0.0 to < 4.0.0
  • Also fixes an issue with the MicrosoftSentinel attribute disappearing from msticpy

What's Changed

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.5.1...v2.5.2

msticpy - v2.5.1: Hotfix for import failure

Published by ianhelle over 1 year ago

Some minor fixes that address:

  • importing msticpy without some non-default azure packages installed failed
  • added more resiliency to query reader so that the whole thing does not fail if there is bad query file.
  • removed initialization dependency on azure-resourcegraph in MicrosoftSentinel class.

What's Changed

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.5.0...v2.5.1

msticpy - v2.5.0

Published by ianhelle over 1 year ago

Summary of main changes

  • New MS Sentinel and Azure Kusto drivers/data providers - these include support for multi-threaded parallel queries, proxies and user-defined query timeouts.
  • Extensibility model for MSTICPy - you can create private data providers, TI and Context providers and load them into MSTICPy alongside the built-in providers.
  • MS Sentinel repo query download - add current detection and hunting queries from the Sentinel repo as Sentinel queries runnable from MSTICPy/notebooks
  • OSQuery data provider - makes it easy to import OS Query logs to dataframes to do additional processing/analysis on them.
  • Panel tabulator now supported as default data viewer (a million times better than the one we built!)

More details on these changes below

Sentinel and Kusto provider new drivers

This change adds replacement drivers for the MSSentinel and Kusto data providers.
In place of Kqlmagic, these drivers use the azure-kusto-data and azure-monitor-query SDKs, respectively.

Currently these drivers are enabled alongside the existing versions - in a future version we will make these the defaults for Sentinel and Kusto.

Some of the main changes with these new versions:

  • They use the provider names MSSentinel_New and Kusto_New when creating a QueryProvider instance.
  • Both drivers support setting proxies for firewall-protected networks
  • Both drivers support custom configuration of the server timeout via a timeout parameter
  • Both drivers use integrated Azure authentication by default and support the auth_types and tenant_id parameters used elsewhere
    in MSTICPy
  • Both drivers support threaded execution for parallelizing queries (across multiple workspaces/clusters or split by time) - this functionality, however, will be exposed in v2.6.0 via a separate feature.
  • The MSSentinel_New driver allows you to execute the same query across multiple workspaces in parallel and returns the results as a combined dataframe.
  • Some of the previous parameters have been deprecated:
    • mp_az_auth is replaced by auth_types (the former still works but will be removed in a future release).
    • mp_az_auth_tenant_id is replaced by tenant_id (the former is not supported in the new providers).

Note: in order to use these new versions you must have the azure-kusto-data and/or azure-monitor-query Python packages
installed. You can install these using pip install msticpy[azure] or install them separately using pip.

For more details on how to use these providers, see:

Changes specific to the MS Sentinel provider

Connecting to multiple workspaces allows you to run queries across these workspaces and return the combined results as a single Pandas DataFrame. The workspaces must use common authentication credentials and should have the same data schema.

# use workspace names if these workspaces are configured in msticpyconfig.yaml
qry_prov.connect(workspaces=["Default", "MyOtherWorkspace"])

# or use a list of workspace IDs
qry_prov.connect(workspaces=["e6b4bc15-119b-45a2-8f3d-c39ed384ed37", "b17e0e5a...."])

# run query against connected workspaces
qry_prov.SecurityAlert.list_alerts()

Changes specific to the Kusto provider

  • The settings format has changed (although the existing format is still supported albeit with some limited functionality).
    See the Kusto provider documentation for details.
  • In the earlier implementation of driver you can specify a new cluster to connect to in when executing a query. This is no longer supported.
    Once the provider is connected to a cluster it will only execute queries against that cluster. (You can however, call the connect() function to
    connect the provider to a new cluster before running the query.)
  • Filtering pre-defined queries by cluster. If you have MSTICPy query definitions for the Kusto provider, these will all be attached as methods
    of the QueryProvider, when it is created. However, as soon as you connect to a specific cluster, the queries will be filtered down to show
    only the queries that are intended to run on that cluster.
  • New APIs (exposed via the query_provider):
    • get_database_names() - return list of databases for the connected cluster
    • get_database_schema() - return table schema for a database in the cluster
    • configured_clusters() - return a list of clusters configured in msticpyconfig.yaml
    • set_cluster() - switch connected to cluster to a different one (you can use the connect method to do this, which also lets you specify
      additional connection parameters).

Extend MSTICPy with Data provider, TI provider and Context provider plugins

This adds the ability to "side-load" data providers, TI providers and context providers. If you have a data/TI/context source that you want to use in MSTICPy you can write a provider (deriving from one of the base provider classes) and tell MSTICPy where to load it from.

In a future release we'll build on this framework to let you install plugins from external packages and provide some cookie-cutter templates to generate skelton provider classes.

Writing a TI provider or Context provider (partial example)


    class TIProviderHttpTest(HttpTIProvider):
        """Custom IT provider TI HTTP."""

        PROVIDER_NAME = "MyTIProvider"
        _BASE_URL = "https://api.service.com"
        _QUERIES = _QUERIES = {
            "ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"),
            "ipv6": APILookupParams(path="/api/v1/indicators/IPv6/{observable}/general"),

Telling MSTICPy to load the plugins

Load on demand


    import msticpy as mp

    mp.load_plugins(plugin_paths="/my_modules")

    # or multiple paths
    mp.load_plugins(
        plugin_paths=["./my_modules", "./my_other_modules"]
    )

Or specify in msticpyconfig.yaml

        ...
        Custom:
            - "testdata"
    PluginFolders:
        - tests/testdata/plugins
    Azure:
        ...

See the new Extending Msticpy section in our docs.
If you want to contribute any of the drivers you write, also check out the new Development section in the MSTICPy docs.

OS Query Provider

Great contribution from @juju4 here (with a bit of collaboration with @ianhelle).
Create a MSTICPy QueryProvider with the data environment name "OSQueryLogs" and load forensic logs from OSQuery.

# specify one or more paths to folders where the dumped JSON OSQuery logs can be found
qry_prov = mp.QueryProvider("OSQueryLogs", data_paths=["~/logs1", "~/logs2"])
qry_prov.connect()
qry_prov.list_queries()

['osquery.acpi_tables',
'osquery.device_nodes',
'osquery.dns_resolvers',
'osquery.events',
'osquery.fim',
'osquery.last',
'osquery.listening_ports',
'osquery.logged_in_users',
'osquery.mounts',
'osquery.open_sockets',
...

Each event type is available as a separate function that returns a pandas DataFrame with the combined events from the logs for that type

qry_prov.osquery.processes()

Downloading Sentinel Detection and Hunting queries for the Sentinel Query Provider

We haven't finished documenting this or integrating it fully, so will leave the full announcement of this until the next release. If you want to play around with it look at the following module:

from msticpy.data.drivers.sentinel_query_reader import download_and_write_sentinel_queries

download_and_write_sentinel_queries(
    query_type="Hunting",    # or "Detections"
    yaml_output_folder="./sentinel_hunting",
)
qry_prov = mp.QueryProvider("Sentinel_New", query_paths=["./sentinel_hunting"])

Since there are lots of queries, the import might take a little while in its current form.

Panel Tabulator now available as a DataViewer control.

HoloViz Panel is a powerful Bokeh-based data exploration & web app framework for Python. It has an immense amount of functionality that you can read about at the Panel documentation site. You need to have panel installed for the Tabulator-based viewer to run (pip install panel).

Unfortunately, the documentation for our Tabulator view never made it into this release but most of the functionality should be obvious from the UI. There are some useful load-time parameters that you can use at startup for things like:

  • selecting an initial column set.
  • adding columns to a per-row expando pane - useful for viewing long column values such as command-line.

We also kept the column chooser widget from the previous data viewer so that you can interactively select which columns to display. The Tabulator MSTICPy initialization parameters are documented in the code.

Most of the Tabulator init parameters are also passed through to the underlying control - which give you an immense amount of control over the viewer. These are described in the Panel Tabulator documentation

Big thanks to our contributors in this release!

@juju4
@jannieli
@ianhelle
@Tatsuya-hasegawa
@FlorianBracq
@danielyates2
@petebryan
@ashwin-patil

What's Changed PR Reference

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.4.0...v2.5.0

msticpy - v2.4.0

Published by ianhelle over 1 year ago

Main changes for this release

There are no huge changes in this release but a good variety of important updates and fixes.
We're also delighted to welcome 3 new contributors to the MSTICPy family:

  • @ZeArioch
  • @ctoma73
  • @jllangley

Thanks so much!

New Threat Intel provider for Pulsedive from @fr0gger #609

This includes a standard MSTICPy TI provider (so you can include it in you collection of providers used for
regular TI checks on IPs, URLs, etc. This provider also contain a few custom methods that let to query
some other facets of the Pulsedive data. For example, the explore function that allows you to use
the pulsedive query language

pddetail = pdlookup.explore(query="ioc=pulsedive.com or threat=AgentTesla")
pddetail

You can also request a can on a domain or URL

pdscan = pdlookup.scan(observable= "alvoportas.com.br")
pdscan

To use any of the Pulsedive features you'll need an account and API key from Pulsedive
See more details of the usage in the Pulsedive notebook

Process tree updates #637

  • @ZeArioch added Process Tree support for FireEye HX data so it should be automatically recognized and render correct
  • We also added the ability to export a process tree as a text object - which is useful if you want to copy and paste
    a tree or part of it into a non-HTML document. See the Process Tree docs for more details
+--  Process: C:Program FilesMicrosoft Monitoring AgentAgentMonitoringHost.exe
   PID: 0x888
   Time: 1970-01-01 00:00:00+00:00
   Cmdline: nan
   Account: nan  LoginID: 0x3e7
   +--  Process: C:WindowsSystem32cscript.exe  PID: 0x364
      Time: 2019-01-15 04:15:26+00:00
      Cmdline: "C:Windowssystem32cscript.exe" /nologo
         "MonitorKnowledgeDiscovery.vbs"
      Account: WORKGROUPMSTICAlertsWin1$  LoginID: 0x3e7
   +--  Process: C:Program FilesMicrosoft Monitoring AgentAgentHealth Service
      StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe  PID:
      0x1c4
      Time: 2019-01-15 04:16:24.007000+00:00
      Cmdline: "C:Program FilesMicrosoft Monitoring AgentAgentHealth
         Service
         StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe"
         GetInventory "C:Program FilesMicrosoft Monitoring
         AgentAgentHealth Service
         StateCT_602681692workServiceStateServiceState.mof" "C:Program
         FilesMicrosoft Monitoring AgentAgentHealth Service
         StateCT_602681692workServiceState"
      Account: WORKGROUPMSTICAlertsWin1$  LoginID: 0x3e7

Miscellaneous fixes #644

This sounds like a small item but contain several important fixes:

  • Azure authentication (az_connect) now avoids throwing exceptions if you ask it to use authentication types (e.g. clientsecret) where parameters are not passed (or available in environment variables). It will now just ignore those credential types and only throw an exception if no usable credential types remain.
  • Updates to API documentation
  • A new IPython magic "%save_to_cell" - this lets you save a Python object (e.g. a DataFrame to a base64-encoded blob in a new cell. The cell contains code to restore the original data. This is subject to the usual caveats about pickle - including the security ones. Do Not run a cell that unpickles some arbitrary data in notebooks that you do not trust.
  • A bunch of changes/fixes to the Sentinel APIs
    • Most of these are fixes related to the newly-supported Sentinel Dynamic Summaries feature
    • Some minor fixes also to Sentinel core

Python Logging support #640

We should have had this from the beginning but it's never too late to start correcting your mistakes.
We've implemented a central logging module and started to instrument some of the code that is especially complex
and where people often get stuck with cryptic errors. E.g. the init_notebook function.
We also enabled in in the authentication modules (az_connect) in #644
Most of the time, this will be invisible. However, if you need it you can just do the following:

# import msticpy as mp   # if not already imported
mp.set_logging_level("INFO")

Then re-run the function that you are having trouble with again.
You can also use the MSTICPYLOGLEVEL variable to control this. And, if you want to log to a file, set the env variable MSTICPYLOGFILE to the path of your log file. (You'll need to restart the kernel/python session and reload MSTICPy for this to take effect).

Support for Bokeh 3.0 #630 #642 and #650

@ctoma73 did some awesome work to track down problems with compatibility with Bokeh 3.0 and fix all of them (a lot were tedious mypy/linting fixes due to some of the more dynamic nature of the Bokeh 3.0 object model).
You'll notice in #650 that we still have Bokeh 2.4.3 in the MSTICPy requirements. We're not going to change that just yet since we want compatibility with PyViz/HoloViz panel - you will likely see some panel-related features in the next minor release.
Despite this (and assuming you can ignore some pip warning about MSTICPy not being compatible with Bokeh 3.x) you can install Bokeh 3.0 after MSTICPy and enjoy the delights of the new release. All of our code should be compatible (tested with 3.0.0 and 3.1.0).

That's all for this release.
We'll likely be doing a follow-on 2.5.0 release that will include several contributions from our 2023 Hackmonth (which turned into a HackNMonths event).

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.3.1...v2.4.0

msticpy - MSTICPy Feb 2023 Fixes

Published by ianhelle over 1 year ago

This is minor release with mostly fixes.

Some higlights from the #631 PR

#629 - You can now suppress progress bar for Threat Intel lookups (useful to avoid screen mess
when running multiple lookups from other code)

  tilookup.lookup_iocs(data, progress=False)

#572 - We've had a long-running issue in Azure Machine Learning where the UI does not correctly
handle javascript written by the notebook. This results in JS code in the output cells. While we're waiting
for AML to re-adopt the latest Azure Notebooks package and get rid of this bug altogether we've
added a fix to suppress javascript text for out Kqlmagic data provider

  • Fix to Azure ML use - automatic creation of msticpyconfig.yaml was writing the file to
    the wrong place, so users always got the message that no config file was found.

  • We had a request (again for batch jobs) to remove automatic display of license information in the geoip module.

  • Using MSTICPy offline or in isolated environment - it has always been our goal to support this but
    we recently discovered that we were running a check_version call from init_notebook. This function
    did not handle network failure and crashed the entire init_notebook process. This has been fixed
    so should be runnable offline or in air-gapped networks.

  • Related to this we've also cleaned up remaining units tests that make outbound network requests.

Full Changelist

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.2.3...v2.3.1

msticpy - v2.3.0

Published by ianhelle over 1 year ago

Some new data-related features in this release.

  • Support for the new (still in preview at time of writing) Dynamic Summaries feature of MS Sentinel
  • Added ability to create and use "ad-hoc" parameterized queries for data providers
  • Simple search mechanism for finding queries
  • Support for JSON queries for CyberReason

Support for Microsoft Sentinel Dynamic Summaries

Dynamic Summaries are a Sentinel feature that allow you to persist results of
query jobs in a summarized/serialized form. This might be useful for keeping
results of daily watch jobs, for example. We will be using it in MSTICPy notebooks
to publish more complex result sets from automated notebook runs.

MSTICPy operations available include:

  • Retrieve list of current dynamic Summaries
  • Retrieve a full dynamic summary
  • Create a dynamic summary
  • Delete a dynamic summary
  • Update an existing dynamic summary

Examples:

# list dynamic summaries
sentinel.list_dynamic_summaries()

# create a dynamic summary in Sentinel
sentinel.connect()
sentinel.create_dynamic_summary(
    name="My_XYZ_Summary",
    description="Summarizing the running of the XYZ job.",
    data=summary_df,
    tactics=["discovery", "exploitation"],
    techniques=["T1064", "T1286"],
    search_key="host.domain.dom",
)

The MSTICPy support also includes a DynamicSummary class that lets you
manipulate dynamic summary objects more easily

  # can also import the class directly
  # from msticpy.context.azure.sentinel_dynamic import DynamicSummary
  # dyn_summary = DynamicSummary(....)
  # This example shows using the "factory" method - new_dynamic_summary
  dyn_summary = sentinel.new_dynamic_summary(
      summary_name="My new summary",
      summary_description="Description of summary",
      source_info={"TI Records": "misc"},
      summary_items=ti_summary_df,
  )
  # Add the local summary object to add to the Sentinel dynamic summaries.
  sentinel.create_dynamic_summary(dyn_summary)

# Retrieve a dynamic summary from Sentinel
dyn_summary = sentinel.get_dynamic_summary(
      summary_id="cea27320-829c-4654-bbf0-b14367483418"
)
# the return value is a DynamicSummary object
dyn_summary

  DynamicSummary(id=cea27320-829c-4654-bbf0-b14367483418, name=test2, items=0)

By default get_dynamic_summary returns the header data for the summary.

The next example shows how you can also fetch full data for the dynamic
summary (by adding summary_items=True). From the returned object,
you can convert the summary items to a pandas DataFrame.

Note: fetching summary items is done via the Sentinel QueryProvider
since the APIs do not support retrieving these.

    dyn_summary = sentinel.get_dynamic_summary(
        summary_id="cea27320-829c-4654-bbf0-b14367483418",
        summary_items=True
    )

dyn_summary.to_df()
index Ioc IocType QuerySubtype Provider Result Severity Details TimeGenerated
OTX hXXp://38[.]75[.]37[.]1/static/encrypt.min.js url   OTX True 2 {‘pulse_count’: 3, ‘names’: [‘Underminer EK’ 2022-12-15 01:55:15.135136+00:00
VirusTotal hXXp://38[.]75[.]37[.]1/static/encrypt.min.js url   VirusTotal False 0 Request forbidden. Allowed query rate may ha 2022-12-15 01:55:15.135136+00:00
XForce hXXp://38[.]75[.]37[.]1/static/encrypt.min.js url   XForce

You can also create dynamic summaries from a DataFrame and append
DataFrame records to an existing dynamic summary.

Read the full documentation in MSTICPy Sentinel Dynamic Summaries doc

New QueryProvider API to dynamically add a parameterized query.

MSTICPy has always supported the ability to run ad hoc text queries for different providers
and return the results as a DataFrame. Using a static query string like this is quick and easy
if you only want to run a query once but what if you want to re-run with different time
range or host name? A lot of tedious editing or string search/replace!

Adding a full query template to MSTICPy, on the other hand, is overkill for this kind of thing.
Dynamic parameterized queries are especially suited for notebooks - you can create an
in-line parameterized query and have it update with the new parameters every time
you run the notebook.

To use dynamic queries - define the query with parameter placeholders (delimited
with curly braces "{" and "}"), then create parameter objects (these handle any special
formatting for datetimes, lists, etc.).
You add the list of parameter objects along with the replaceable parameter values
when you run the query, as shown below.

# intialize a query provider
qry_prov = mp.QueryProvider("MSSentinel")

# define a query
query = """
SecurityEvent
| where EventID == {event_id}
| where TimeGenerated between (datetime({start}) .. datetime({end}))
| where Computer has "{host_name}"
"""
# define the query parameters
qp_host = qry_prov.Param("host_name", "str", "Name of Host")
qp_start = qry_prov.Param("start", "datetime")
qp_end = qry_prov.Param("end", "datetime")
qp_evt = qry_prov.Param("event_id", "int", None, 4688)

# add the query
qry_prov.add_custom_query(
    name="get_host_events",
    query=query,
    family="Custom",
    parameters=[qp_host, qp_start, qp_end, qp_evt]
)

# query is now available as
qry_prov.Custom.get_host_events(host_name="MyPC"....)

See Dynamically Adding Queries in MSTICPy Docs

QueryProvider - Query Search

As the number of queries for some providers grows, it has become more difficult to quickly
find the right query. We've implemented a simple search capability that lets you search
over the names or properties of queries. It takes four parameters:

  • search - search terms to look for in the
    query name, description, parameter names, table and query text.
  • table - search terms to match on the target table of the query.
    (note: not all queries have the table parameter defined in their metadata)
  • param - search terms to match on a parameter name
  • case - boolean to force case-sensitive matching (default is case-sensitive).

The first three parameters can be a simple string or an iterable (e.g. list, tuple)
of search terms. The search terms are treated as regular expressions. This
means that a the search terms are treated as substrings (if no other
regular expression syntax is included).

Find all queries that have the term "syslog" in their properties

    qry_prov.search("syslog")
    # equivalent to qry_prov.search(search="syslog")

    ['LinuxSyslog.all_syslog',
    'LinuxSyslog.cron_activity',
    'LinuxSyslog.list_account_logon_failures',
    ...

See Search queries in MSTICPY Docs

Support for JSON queries in Data Providers

@FlorianBracq has updated the CyberReason data provider so that it supports JSON queries. The
mechanism that we used for KQL and SQL queries breaks JSON since it is a simple string substitution.
Other data sources that use JSON queries include Elastic - we are planning to leverage the same
mechanism to support parameterized Elastic queries in a future release.
Thanks @FlorianBracq!

What Else has Changed?

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.2.0...v2.3.0

msticpy - IoC Defanging, ServiceNow, GCC support for MDE, Python 3.11

Published by ianhelle almost 2 years ago

Highlights

Re-architected context and TI providers

The biggest feature of this release is not directly visible but has involved a huge amount of work by @FlorianBracq.
Florian spotted that our HTTP TI provider (used for several TI services such as VirusTotal, OTX, XForce) could be used more generically, specifically for non-TI sources that provided valuable context, such as ServiceNow. So, he re-worked the TI providers sub-package to pull out generic context provider capabilities used by both TI and non-TI sources.
The immediate benefit of this is the next highlight

ServiceNow context provider

This is yet to be full documented but if you have a ServiceNow instance and want to hook up MSTICPy to query it try the following.

  1. Add your ServiceNow configuration to msticpyconfig.yaml
ContextProviders:
  ServiceNow:
    Primary: True
    Args:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53
      AuthKey: "authkey"
      AuthId: "authid"
    Provider: "ServiceNow"

Note: you can store the secrets in KeyVault in the same way as TI and other Providers - see the Key Vault Secrets section of MSTICPy Settings Editor

Import and instantiate a ContextProvider and look things up

from msticpy.context.contextlookup import ContextLookup

context_lookup = ContextLookup()
result = context_lookup.lookup_observable("10.0.0.1", providers=["ServiceNow"])
result2 = context_lookup.lookup_observable("[email protected]", providers=["ServiceNow"])

Defanging support for IoCExtract and TI Providers

In threat reports, IoCs are often de-fanged to make IP addresses, URLs, etc, not clickable. An example
de-fanged IP address would look something like this 17[.]34[.]21[.]195

Previously these would not be matched by the IoCExtract patterns due to the "escaped" dots.
IoCExtract now supports common de-fanged markup such as

  • "[.]" to escape dots in IP addresses and domains,
  • "@" replaced by "AT"
  • "http(s)" and "(s)ftp(s)" replaced by "hXXp(s)" and "(s)fXp(s)" respectively.

We have also added support for email address patterns to IoCExtract.

TI providers will also accept de-fanged IoCs, removing the de-fanging before submitting them to the provider for lookup.

We've also supplied a couple of utility functions defang_ioc and refang_ioc in msticpy.common.utility. These are not yet added as Pivot functions to IpAddress, Url, Dns, Account but will be added in a future release.

Added GCC support to MDE/M365 data providers

This allows customers working with government clouds to query the correct Defender endpoints.

Python 3.11 officially supported

Although there wasn't anything in our code that was a Py 3.11 blocker, some of our dependencies took
a little while to publish 3.11-compatible wheels. That was all done with SciPy, Statsmodels and ScikitLearn
and our build pipeline now in includes a full test pass on Python 3.11. Many thanks to @tonybaloney for
pushing us through this.

What's Changed

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.5...v2.2.0

msticpy - Bokeh, ipywidgets version restrictions

Published by ianhelle almost 2 years ago

The main driver for this release is to restrict versions of bokeh, ipywidgets and pandas.

  • Version 3.0.0 of bokeh plots has some breaking changes that prevent it working with MSTICPy
  • Version 8.0.0 of ipywidgets has changes that prevent some of the MSTICPy compound widgets displaying correctly.

We also decided to start restricting versions of some of our other dependencies to the current major version - to prevent unexpected breaking changes stopping MSTICPy from working. We have included pandas in this list and will expand it to cover more packages in future. We will combine this with an automated build job that has no version restrictions so that we're aware of version changes that we need to address. The intent here is to have MSTICPy have as broad a version range as possible for its dependencies while still avoiding failures due to breaking changes.

Another small but important change is an update to the Process Tree viewer to allow process GUIDs as process IDs (rather than just hex or decimal format integers). Thanks to @nbareil for this change!

What's Changed

New Contributors

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.4...v2.1.5

msticpy - Fixes for MS Sentinel API and configuration

Published by ianhelle almost 2 years ago

Some minor fixes and improvements:

  • MicrosoftSentinel class now defaults to "Default" workspace or workspace name supplied as workspace parameter
    when connecting.
sentinel = MicrosoftSentinel()
sentinel.connect()  # connect to "Default" workspace
sentinel.connect(workspace="MyWorkspace")   # connect to named workspace
  • Sentinel create_* APIs now return ID of new item (incident, bookmark, analytic, watchlist)
  • init_notebook - now accepts config parameter to use custom msticpyconfig.yaml for notebook session (overrides enviromnent variable and other defaults
import msticpy as mp
mp.init_notebook(config="~/configs/all_ti_provs.yaml")   # use a custom msticpy config file.
  • Sentinel configuration editor no longer throws an exception if named control not found
  • Sentinel TI provider will not attempt lookups if ThreatIntelligenceIndicator table not found in the Sentinel data provider schema
  • Support for Kusto/Azure Data explorer settings in Settings editor
  • Added checked_kwargs decorator to utility/types.py

What's Changed

Full Changelog: https://github.com/microsoft/msticpy/compare/v2.1.3...v2.1.4

Package Rankings
Top 2.81% on Pypi.org
Top 6.75% on Proxy.golang.org
Badges
Extracted from project README
Azure Pipelines build Downloads BlackHat Arsenal 2020 Binder
Related Projects
jupyter-Kqlmagic

Extension (Magic) to Jupyter notebook and Jupyter lab, that enable notebook experience working wi...

07 Aug 2018 85
MLOps

MLOps examples

03 May 2019 1,814
cloud-advocate-workshops

Workshops created by the Cloud Advocacy team at Microsoft.

01 Aug 2023 35
infinibatch

Efficient, check-pointed data loading for deep learning with massive data sets.

27 Jun 2020 203
promptflow-rag-project-template

An end-to-end sample of RAG showcasing development, evaluation, experimentation, and deployment u...

14 Dec 2023 44
botbuilder-python

The Microsoft Bot Framework provides what you need to build and connect intelligent bots that int...

10 Jan 2018 700
pybryt

Python library for pedagogical auto-assessment

25 Jan 2021 60
dpu-utils

Utilities used by the Deep Program Understanding team

10 Oct 2018 102
azure-sdk-for-js

This repository is for active development of the Azure SDK for JavaScript (NodeJS & Browser). For...

25 Oct 2017 2,058
ms-rest-azure-js

Azure runtime for isomorphic javascript libraries generated by Autorest

30 Aug 2017 9
semanticworkbench

A versatile tool designed to help prototype intelligent assistants, agents and multi-agentic systems

29 Jul 2024 77
zero-shot-scfoundation

04 Oct 2023 46
BiDR

Repo for WWW 2022 paper: Progressively Optimized Bi-Granular Document Representation for Scalable...

28 Feb 2022 15
chat-copilot

17 Jul 2023 2,031
c9-python-getting-started

Sample code for Channel 9 Python for Beginners course

17 Sep 2019 7,752