Alert based on Custom Metrics

The developer at Mystique Unicorn are looking for a way to alert them based on the logs their app is generating. Their app creates an log entry that has the following fields.

{
  "request_id": "f5570d0f-5389-4bce-98c3-7ed5380eefb2",
  "event_type": "inventory_event",
  "store_id": 6,
  "cust_id": 206,
  "category": "Camera",
  "sku": 100657,
  "price": 2.52,
  "qty": 18,
  "discount": 8.2,
  "gift_wrap": true,
  "variant": "red",
  "priority_shipping": true,
  "TimeGenerated": "2021-04-25T12:52:40.938937",
  "contact_me": "github.com/miztiik"
}

They would like to be alerted when there are suspicious activity. For example, consider this, If there is an deeply disocunted(>90%) order and the number of quantites ordered is higher then (>x) and priority shipping had been requested, then they would like to notified for further analysis.

Can you show them how the can get started?

🎯 Solutions

We will use Azure Monitor Agent¹ to collect the application logs from a specified location(for exampe: /var/log/miztiik*.json) and push them to a Log Analytics Workspace³. To receive the logs in the correct format we need to define a custom table^5,6 and a data collection rule. The rule will collect logs from our source(VM App), transform the logs using Kusto Query Language(KQL) to a structure defined in the custom table and store them in the sink(log analytics workspace). Later we will deploy a KQL query to setup an alert that meets the developer requirements.

This repository holds the bicep code to deploy the necessary infrastructure for this demo,

• Virtual Machine(RedHat) inside a Vnet with public access.
  • Bootstrapped with custom libs using customData script.
• Log Analytics Workspace ( Analytics tier)
  • Custom Table
  • Data Collection Endpoint - With public access
  • Data Collection Rule
    • Configured with the log location, KQL query and custom table
• Azure Monitort Alert Rule using KQL

1. 🧰 Prerequisites

   This demo, instructions, scripts and bicep template is designed to be run in westeurope. With few or no modifications you can try it out in other regions as well(Not covered here).

   • 🛠 Azure CLI Installed & Configured - Get help here
   • 🛠 Bicep Installed & Configured - Get help here
   • 🛠 VS Code & Bicep Extenstions - Get help here

2. ⚙️ Setting up the environment

   • Get the application code

     https://github.com/miztiik/azure-alert-on-custom-metrics
     cd azure-alert-on-custom-metrics

3. 🚀 Prepare the environment

   Let check you have Azure Cli working with

     # You should have azure cli preinstalled
     az account show

   You should see an output like this,

    {
      "environmentName": "AzureCloud",
      "homeTenantId": "16b30820b6d3",
      "id": "1ac6fdbff37cd9e3",
      "isDefault": true,
      "managedByTenants": [],
      "name": "YOUR-SUBS-NAME",
      "state": "Enabled",
      "tenantId": "16b30820b6d3",
      "user": {
        "name": "miztiik@",
        "type": "user"
      }
    }

4. 🚀 Deploying the application

   Let us walk through each of the stacks,

   • Stack: Main Bicep The params required for the modules are in params.json. Do modify them to suit your need.(Especially the adminPassword.secureString for the VM. You are strongly encouraged to Just-In-Time access or use SSH key instead of password based authentication). The helper deployment script deploy.sh will deploy the main.bicep file. This will inturn call the modules for Resource Group(RG), VNet & Virtual Machine, Log analytics workspace, Custom table, Data collection endpoint, Data collection rule(DCR) and finally attach the DCR to the VM.

     Alert Module We will create a KQL query to identify order events that match the given criteria -

     lumberyardTable001_CL | where discount > 90 and qty >1 and priority_shipping==true

     We can also refine it to aggregate only the events that had been received in the last 5 minutes like this,

     lumberyardTable001_CL
     | where discount > 90 and qty > 1 and priority_shipping == true
     | where Timestamp > ago(5m)

     But since have the option of choosing an evaluation frequency within the alert configuration, we will us that. Remember evaluationFrequency is smaller than the aggregation granularity, this will result in sliding window evaluation. The code is configured for evaluationFrequency of 1minute and aggregation windowSize of 5minutes, resulting in creating a sliding window. This may also mean we can trigger the alarm multiple times for the event. It is a conscious choice, to balance signal-to-noise towards more noise(aka false-positives) when we are trying to catch suspicious activity.

     sh deploy.sh

     This stack will create almost all of the resoureces. The RG is created as a deployment at the subscription level. The other resources are deployments at resource group level.

   • Virtual Machine A Redhat vm is bootstrapped using customData to install python3, git. We will use a bash script generate_data.sh. It will generate a json conformant log in this location /var/log/miztiik-yyyy-MM-dd.json for example /var/log/miztiik-2023-04-03.json. All new entries are appended to the file with new-line being the delimitter as expected by Azure Custom Logs.

     {"request_id": "08b6b258-c6d6-44ff-802e-69d52422095a", "event_type": "inventory_event", "store_id": 45, "cust_id": 4416, "category": "Camera", "sku": 17293, "price": 44.61, "qty": 15
     , "discount": 73.2, "gift_wrap": 0, "variant": "MystiqueAutomatoon", "priority_shipping": 1, "TimeGenerated": "2023-04-03T10:54:23", "contact_me": "github.com/miztiik" }
     {"request_id": "60051c39-6817-4f52-bd8f-59cc9a38be0f", "event_type": "inventory_event", "store_id": 64, "cust_id": 4081, "category": "Camera", "sku": 17815, "price": 50.92, "qty": 16
     , "discount": 29.2, "gift_wrap": 1, "variant": "MystiqueAutomatoon", "priority_shipping": 0, "TimeGenerated": "2023-04-03T10:54:24", "contact_me": "github.com/miztiik" }
     {"request_id": "f340f343-fc0f-40a1-b9f5-2abdd14b929d", "event_type": "inventory_event", "store_id": 35, "cust_id": 338, "category": "Camera", "sku": 16158, "price": 53.76, "qty": 13,
     "discount": 34.3, "gift_wrap": 1, "variant": "MystiqueAutomatoon", "priority_shipping": 0, "TimeGenerated": "2023-04-03T10:54:25", "contact_me": "github.com/miztiik" }

     Formatting a single log line entry properly, will look like this,

     {
       "request_id": "f5570d0f-5389-4bce-98c3-7ed5380eefb2",
       "event_type": "inventory_event",
       "store_id": 6,
       "cust_id": 206,
       "category": "Camera",
       "sku": 100657,
       "price": 2.52,
       "qty": 18,
       "discount": 8.2,
       "gift_wrap": true,
       "variant": "red",
       "priority_shipping": true,
       "TimeGenerated": "2021-04-25T12:52:40.938937",
       "contact_me": "github.com/miztiik"
     }

   After successfully deploying the stack, Check the Resource Groups/Deployments section for the resources.

5. 🔬 Testing the solution

   • Connect to the VM

     Connect to the VM using Just-In-Time access. This should allow your IP/IP list to securelly connect to your VM using private IP address. Use the password you updated in the params.json

     ssh miztiik@publicIP

   • Generate Log Data

     For some reason the customData, bootstrap script fails, will fix it later. Until then manually copy the generate_data.sh from this repo to the server. _You may need elevated privileges to create log file in /var/log.

     sudo su
     cd /var/log/
     # Copy generate_data.sh to the server
     sh generate_data.sh &

     The script by default is configured to write one log entry every 5 seconds and generate a total 100000 log lines to/var/log/miztiik*.json

   • Query in Azure Log Analytics Workspace

     Login to Azure Log Anaytics portal. You should be able to find it under /Miztiik_Enterprises_Custom_Logs_To_Monitor_xxx/providers/Microsoft.OperationalInsights/workspaces/lumberyard-payGTier-xxx",. Navigate to logs, select the custom table(lumberyardTablexxx_CL) & Run

     

6. Troubleshooting Azure Monitor Agent

   • In case if you are facing issues, log into the VM and check the logs /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/ for potential causes.

   • From the portal, you can query to find if the agent is reporting to Azure Monitor

     Heartbeat | where Category == "Azure Monitor Agent" and Computer == "<computer-name>" | take 10

     Check if the AMA is running,

     systemctl status azuremonitoragent

7. 📒 Conclusion

   Here we have demonstrated how to configure alerts based on custom logs. You can extend the solution to setup alerts actions groups & receive email notifications. An example of email notification is showin below,

   

8. 🧹 CleanUp

If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well

• Resources created during Deploying The Application
• Any other custom resources, you have created for this demo

# Delete from resource group
az group delete --name Miztiik_Enterprises_xxx --yes
# Follow any on-screen prompt

This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.

📌 Who is using this

This repository aims to show how to Bicep to new developers, Solution Architects & Ops Engineers in Azure.

💡 Help/Suggestions or 🐛 Bugs

Thank you for your interest in contributing to our project. Whether it is a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here

👋 Buy me a coffee

Buy me a coffee ☕.

📚 References

1. Azure Docs: Azure Monitor Agent Overview
2. Azure Docs: Azure Monitor Agents -vs- Other Agatnes Overview
3. Azure Docs: Collect text logs with Azure Monitor Agent
4. Azure Docs: Collect text logs with Log Analytics Agent
5. Azure Docs: Custom Table
6. Azure Docs: Workspace Table Creation
7. Azure Docs: Log Data Plan Basic-vs-Analytics Comparison
8. Azure Docs: Custom Log Ingestion using python SDK
9. Azure Docs: Custom Metrics to Azure Monitor
10. Azure Docs: Check Logs in Azure Logs
11. Azure Docs: Troubleshoot Azure Monitor Agent

🏷️ Metadata

Level: 200