Getting started notebook update

Adding Azure Sovereign cloud support
Adding numbering and better organization to headings and table of contents

Getting Started and Automated notebooks update

New Features

• Major updates to Getting Started Guide for Azure Sentinel ML Notebooks - this now includes automatic creation and assisted configuration of msticpyconfig.yaml. #95
• Added of "A Tour of Cybersec notebook features" notebook covering common features of MSTICPy and Azure Sentinel notebooks. #95
• Added notebook automation notebooks and supporting files #93
• Added MSTICPy Tour notebook #85
• Added KQL query creator notebook #82

MSTICPy v1.0.0 upgrade

Updates

New startup experience

The startup cell in notebooks using MSTICPy has been slimmed down to make
it easier to understand. Functionality moved to the utils/nb_check.py
module. The startup experience is cleaner (more readable, neater messages)
and with fewer prompts.

Nteract/Azure Machine Learning data explorer used by default

DataFrame output is now usually rendered in the nteract/AML viewer. This
allows filtering and ordering of columns in place. It also includes
some plotting functions (although these are often more suited to
data that is mostly numeric vs. security log data.)

Note: You can revert to static table plot by setting

pd.set_option("display.html.table_schema", False)

Single Sign-On experience with Azure CLI

Instead of authenticating at the start of each notebook, you can do
a single sign-on using Azure CLI. From a cell in a notebook or
from a terminal you can run:

az login

(in notebooks run !az login). MSTICPy and KqlMagic will use the CLI
authenticated session to obtain tokens for your Azure Sentinel workspace(s).

Support for Python 3.8 Kernels in AML Compute

Python 3.8 is rolling out as the default in Azure ML Compute. Python
3.6 is available but we encourage you to switch to 3.8.

Better detection and use of configuration files

The notebook setup cell will now find and use a msticpyconfig.yaml in
the root of your user folder in an Azure ML workspace. Previously
you had to set an environment variable to point to it.

KeyVault storage and reading of secrets (for API keys)

This is now working and supported on AML compute instances.

Lightweight install

MSTICPy and KqlMagic now use "extras". This means that the default
install results in few dependencies being installed. This means a
quicker installation and less likelihood of package conflicts. The
notebooks have been update to automatically install the correct "extra"
(most don't need any). If
you are running a notebook that is missing a dependency you should get
a friendly exception message telling you what to install.

The lightweight install should reduce the install/setup time by around
90% on Azure ML compute (i.e. what previously took 90 seconds will now
take 10 seconds or less).

KQL magic schema pop-up

Previously this did not work in AML unless you were prepared to do
some digging and set up environment variables. This should now
work automatically.

Getting Starting and Configuring notebooks

These have been updated to use the new MSTICPy settings editor.

MSTICPy v1.0.0

The notebooks release is happening at the same time as a new MSTICPy
release. This contains a lot of new and upgraded features such as

• Pivot functions
• Settings editor and management
• SQL -> KQL translator

Read more about the current MSTICPy release here

Transition to MSTIC Notebooklets

We will also shortly release an update to MSTIC Notebooklets (MSTICNB)
and updates to the Account Explorer and IP Explorer notebooks using
these. Using notebooklets allows a drastic reduction in notebook code (10% of an equivalent notebook not using notebooklets)
and a lot more flexibility about the flow of an
investigation.

Read more about MSTIC notebooklets here

Fixes

Fixed errors in many notebooks, added better explanations and clarifications.

MDATP Webshell alert notebook and improved notebook setup logic

New Features

• New notebook published #28 - Guided Investigation - MDATP Webshell Alerts
  This notebook takes you through triage and investigation of alerts from Microsoft Defender Webshell alerts.
• Update to setup section of notebooks #29
  Most of the setup logic is now in a msticpy module (nb_init) and some in a local module (utils/nb_check.py)
• Added nb_check.py to check kernel version and installed msticpy version (this can't be in the msticpy module since we don't yet know whether msticpy is installed).
  Note: the notebook will run fine without this file - it just skips the checks for python and msticpy version.
• Updated Notebook initialization markdown text and added links to run local versions of the Configuration and Troubleshooting notebooks (rather than just view static versions on GitHub).

Fixes

• Removed references to Setup section that no longer exists.
• Fixing a few errors in IP Explorer and Process alert