Skip to content

Latest commit

 

History

History
176 lines (112 loc) · 8.91 KB

README.md

File metadata and controls

176 lines (112 loc) · 8.91 KB
Raw

Azure Stream Analytics

introductory diagram

Azure Stream Analytics is a serverless real-time analytics service. The goal of this sample is to demonstrate how to develop a streaming pipeline, with IaC and testability in mind.

Prerequisites

Setup

  1. Azure Cli Will be necessary for various tasks. Please follow the instructions found here.

  2. Bicep This project uses Bicep templates to setup Azure infrastructure. Please follow the steps under Install Bicep Tools to install the Azure Cli extension.

    For an introduction to Bicep, you can find more information in the Bicep repo under Get started with Bicep.

  3. Raspberry Pi Azure IoT Online Simulator To run through the Functional Test section, you can use this simulator. Makes sure to follow the step required to edit the js code.

  4. azure-streamanalytics-cicd Unit testing is possible using the npm package azure-streamanalytics-cicd. The tool provides additional features for working with Azure Stream Analytics locally. Check here for more info.

  5. Configure environment variables. We will assume these are customized and configured per your needs

    APP="tech-sample"
ENVIRONMENT="test"
LOCATION="westus"
STORAGE_ACCOUNT="st${APP/-/}${ENVIRONMENT}"

Build

# Run Unit Tests
azure-streamanalytics-cicd test -project ./asaproj.json -outputPath ./output/

# Create Resource Group

az group create -n rg-${APP} -l ${LOCATION}

# Validate Generated Template
az deployment group validate -f main.bicep -g rg-${APP} --parameters query='@./streamanalytics-tech-sample.asaql' name=${APP} env=${ENVIRONMENT}

# Show plan
az deployment group what-if -f main.bicep -g rg-${APP} --parameters query='@./streamanalytics-tech-sample.asaql' name=${APP} env=${ENVIRONMENT}

Deploy

# Create Azure Resources. This will also start the job

az deployment group create -f main.bicep -g rg-${APP} --parameters query='@./streamanalytics-tech-sample.asaql' name=${APP} env=${ENVIRONMENT}

# Add device

az iot hub device-identity create --hub-name iot-${APP}-${ENVIRONMENT} --device-id iot-${APP}-${ENVIRONMENT} --edge-enabled

Functional Test

# Use connection information with "Raspberry Pi Azure IoT Online Simulator": https://azure-samples.github.io/raspberry-pi-web-simulator/

az iot hub device-identity connection-string show --hub-name iot-${APP}-${ENVIRONMENT} --device-id iot-${APP}-${ENVIRONMENT} --output tsv

Check the blob storage container to ensure that a json file exists with expected temperature sensor data (exceeding 27 degrees) is present.

Automated End-to-End Test

export DEVICE_CONNECTION_STRING=$(az iot hub device-identity connection-string show --hub-name iot-${APP}-${ENVIRONMENT} --device-id iot-${APP}-${ENVIRONMENT} --output tsv)
export AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string -n ${STORAGE_ACCOUNT} --query connectionString -o tsv)

cd e2e
npm install 
npm test

Cleanup

az group delete -n rg-${APP}

Key concepts

Infrastructure as Code

This sample's infrastructure is created using Bicep templates. Bicep is a DSL that is transpiled into Azure's native ARM Templates. It's currently in preview but already has been integrated into the Azure Cli. Also ARM template reference documentation lists Bicep along side ARM Templates:

Bicep template screen capture

Alternatively, Terraform can also be used.

Key take aways for Bicep

  • Day 0 resource provider support. Any Azure resource — whether in private or public preview or GA — can be provisioned using Bicep.
  • No state or state files to manage. All state is stored in Azure, so makes it easy to collaborate and make changes to resources confidently.
  • VSCode extension for Bicep makes it extremely easy to author and get started with advanced type validation based on all Azure resource type API definitions.

Resource naming conventions attempt to follow examples from Define your naming convention where appropriate.

The structure of the IaC scripts allow you to setup as many versions or the infrastructure as you need easily. For example an individual developer may create a version to use within their Inner Dev Loop:

DEVELOPER="kaa"
ENVIRONMENT="inner"
LOCATION="japaneast"
STORAGE_ACCOUNT="st${DEVELOPER}${ENVIRONMENT}"

az group create -n rg-${DEVELOPER} -l $LOCATION

az deployment group create -f main.bicep -g rg-${DEVELOPER} --parameters query='@./streamanalytics-tech-sample.asaql' name=${DEVELOPER} env=${ENVIRONMENT}

az iot hub device-identity create --hub-name iot-${DEVELOPER}-${ENVIRONMENT} --device-id iot-${DEVELOPER}-${ENVIRONMENT} --edge-enabled

export DEVICE_CONNECTION_STRING=$(az iot hub device-identity connection-string show --hub-name iot-${DEVELOPER}-${ENVIRONMENT} --device-id iot-${DEVELOPER}-${ENVIRONMENT} --output tsv)
export AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string -g rg-${DEVELOPER} -n ${STORAGE_ACCOUNT} --query connectionString -o tsv)

# Cleanup
az group delete -n rg-${DEVELOPER}

Testing

Unit

Azure Stream Analytics can be unit tested locally via the azure-streamanalytics-cicd npm package.

Also this package can be used as part of an Azure DevOps CI\CD pipeline. More info can be found here. This should make integrating this into your existing CI\CD fairly straight forward.

Key take aways for azure-streamanalytics-cicd

  • Test configuration is found under the /test directory.
  • This is the default location for azure-streamanalytics-cicd.
  • test/testConfig.json defines the test cases.
  • test/temperature ... .json defines pipeline inputs and outputs (as JSON)

Functional

simulator site image

Functional testing can be done using the online tool Raspberry Pi Web Client Simulator. Source code can be found here. This is an easy way to interacte with your Azure Streaming Analytics pipeline.

You can manually check the blob storage to see that events are coming through correctly.

blob container screen cap

One aspect to consider, is that events will be batched on the scale of seconds due to the pathPattern defined in streamingjobs.bicep for bloboutput. This is done to aid automated testing. You can adjust during functional testing, and for production as required:

example

  pathPattern: '{date}'

Then you should be able to see many events per file.

Automated End-to-End

This sample combines Azure IoT device SDK and Microsoft Azure Storage SDK for JavaScript to create a Node.js (TypeScript) based End-to-End test solution. As mentioned under Functional by default blob output partitioning is done on the seconds resolution to prevent the automated test from waiting an impractical amount of time. This can be made configurable according to your requirements on deploy by altering the Bicep template.

test result output screen capture

Within the test file e2e/e2e.ts there is the EXPECTED_E2E_LATENCY_MS defined to be 1s. So this would also need to be adjusted for a real implementation.

CI/CD

A sample CI/CD Pipeline YAML file is present in this repo under "samplecicdpipeline.yml". This pipeline runs the tests present under the tests folder, sets up the IoTHub, and deploys the ASA job using the contents of the streamingjobs.bicep file. In order to add a new ASA job, please do the following:

  1. Include a new bicep file for the additional ASA job, and add it to the main.bicep file.
  2. Add the query into the inlineScript under the parameters of the yaml file where the deployment of main.bicep happens.