Memfault integration for Azure IoT Hub

Memfault integration for Azure IoT Hub developed in TypeScript.

Device information

Meta information about devices is populated by cloud when devices report their values

• board type (hardware_version) is inferred cloud side from dev.v.brdV Thing shadow property.
• nickname is inferred from the user's name setting for the device using the name Thing attribute.

Chunks

The Memfault SDK packages the data from all modules in chunks. They are received via MQTT and forwared to the Memfault chunks API.

Memfault embeds offset information in each chunk so they can re-assemble data that arrives out of order, so there is no need to buffer the chunks on the cloudside.

Devices publish the chunks via MQTT to a configurable topic. Right now the asset_tracker_v2 uses <deviceId>/memfault/<Memfault project key>, however deviceId and Memfault project key are superfluous because they can be inferred on the cloud side from the MQTT connection.

To support devices publishing to a MQTT topic named memfault directly, the Memfault project key needed for chunks API is stored on the cloud side in an SSM parameter.

This also allows for changing the project key on the fly if needed without needing deploy a new firmware to devices. However, note that the project key is not a secret and does not need rotation.

It is also useful to have the Memfault project key on the device to directly publish to the Memfault HTTP API to be able to debug problems with the MQTT connections.

This can however also be solved by setting up an API endpoint the accepts requests signed with JWT tokens created using the devices keypair. This would allow to remove the Memfault project key entirely from the firmware. Right now however this is not supported in nRF Connect SDK. Please reach out if you would like to see this feature.

Installation in your Azure account

Install dependencies

npm ci

Setup

Export these environment variables for the Memfault integration stack:

export STORAGE_ACCOUNT_NAME=...

Export these environment variables from your nRF Asset Tracker for Azure deployment:

export RESOURCE_GROUP=...
export APP_NAME=...
export KEY_VAULT_NAME=...

Deploy

Note
This adds the Memfault integration to the existing nRF Asset Tracker for Azure resources.

az deployment group create \
--mode Incremental \
--name memfault-integration-deployment \
--resource-group ${RESOURCE_GROUP:-nrfassettracker} \
--template-file memfault-integration.bicep \
--parameters \
    appName=${APP_NAME:-nrfassettracker} \
    keyVaultName=${KEY_VAULT_NAME:-assetTracker} \
    storageAccountName=${STORAGE_ACCOUNT_NAME:-nrfassettrackermemfault}

# Deploy the function app
npx tsc
npx tsx scripts/pack-app.ts
az functionapp deployment source config-zip -g ${RESOURCE_GROUP:-nrfassettracker} -n ${APP_NAME:-nrfassettracker}-memfault-integration --src dist/functionapp.zip

Configure memfault settings

You can retrieve the project settings from the settings page of the Memfault dashboard of your organization.

az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultProjectKey --value my-projectKey
az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultOrganization --value my-org
az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultProject --value my-project

The organization auth token can be accessed and managed by Administrators at Admin → Organization Auth Tokens in the Memfault UI.

az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultAuthToken --value my-authToken

End-to-end tests

Set up the mock API

# Create a new resource group
az group create -n ${MOCK_API_RESOURCE_GROUP:-memfault-mock-api} -l ${LOCATION:-northeurope}

# Create the resources
az deployment group create \
--mode Complete \
--name manual-deployment \
--resource-group ${MOCK_API_RESOURCE_GROUP:-memfault-mock-api} \
--template-file mock-http-api.bicep \
--parameters \
    storageAccountName=${MOCK_API_STORAGE_ACCOUNT_NAME:-memfaultmockapi} \
    appName=${MOCK_API_APP_NAME}

# Deploy the function app
export MOCK_HTTP_API_ENDPOINT=`az functionapp show -g ${MOCK_API_RESOURCE_GROUP:-memfault-mock-api} -n ${MOCK_API_APP_NAME} | jq -r '.defaultHostName'`
echo $MOCK_HTTP_API_ENDPOINT
npx tsc
npx tsx scripts/pack-mock-http-api-app.ts
az functionapp deployment source config-zip -g ${MOCK_API_RESOURCE_GROUP:-memfault-mock-api} -n ${MOCK_API_APP_NAME} --src dist/mock-http-api.zip

# Configure Memfault Key value parameters
USER_OBJECT_ID=`az ad signed-in-user show --query id -o tsv`
# Assign 'Key Vault Secrets Officer' permission
az role assignment create --role b86a8fe4-44ce-4948-aee5-eccb2c155cd7 \
        --assignee ${USER_OBJECT_ID} \
        --scope /subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP:-memfault}/providers/Microsoft.KeyVault/vaults/${KEY_VAULT_NAME:-assetTracker}

az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultProjectKey --value my-projectKey
az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultOrganization --value my-org
az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultProject --value my-project
az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultAuthToken --value my-authToken
az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultApiEndpoint --value "https://${MOCK_HTTP_API_ENDPOINT}/api/api.memfault.com/"
az keyvault secret set --vault-name ${KEY_VAULT_NAME:-assetTracker} --name memfaultChunksEndpoint --value "https://${MOCK_HTTP_API_ENDPOINT}/api/chunks.memfault.com/"

# Observe Mock API logs
az webapp log tail --resource-group ${MOCK_API_RESOURCE_GROUP:-memfault-mock-api} --name ${MOCK_API_APP_NAME}

# Observe integration logs
az webapp log tail --resource-group ${RESOURCE_GROUP:-nrfassettracker} --name ${APP_NAME:-nrfassettracker}-memfault-integration

# Run the end-to-end tests
npm run test:e2e

Continuous Integration

In order to continuously test this solution, authenticate GitHub Actions by follow the instructions to Configure a service principal with a Federated Credential to use OIDC based authentication. Use https://nrfassettracker.invalid/memfault-ci as the name.

From the command line this can be achieved using:

# Create application
az ad app create --display-name 'https://nrfassettracker.invalid/memfault-ci'
export APPLICATION_OBJECT_ID=`az ad app list | jq -r '.[] | select(.displayName=="https://nrfassettracker.invalid/memfault-ci") | .id' | tr -d '\n'`
# Create federated credentials
az rest --method POST --uri "https://graph.microsoft.com/beta/applications/${APPLICATION_OBJECT_ID}/federatedIdentityCredentials" --body '{"name":"GitHubActions","issuer":"https://token.actions.githubusercontent.com","subject":"repo:NordicSemiconductor/asset-tracker-cloud-memfault-azure-js:environment:ci","description":"Allow GitHub Actions to modify Azure resources","audiences":["api://AzureADTokenExchange"]}'
# Grant the application Owner permissions for subscription
export AZURE_CLIENT_ID=`az ad app list --display-name 'https://nrfassettracker.invalid/memfault-ci' | jq -r '.[].appId'`
export AZURE_SUBSCRIPTION_ID=`az account show | jq -r '.id'`
az ad sp create --id $AZURE_CLIENT_ID
az role assignment create --role Owner \
         --assignee ${AZURE_CLIENT_ID} \
         --scope /subscriptions/${AZURE_SUBSCRIPTION_ID}

Make sure to use the organization and repository name of your fork instead of NordicSemiconductor/asset-tracker-cloud-memfault-azure-js in the command above.

Then,

1. Store the application (client) ID of the service principal app registration created in step in the above step as a GitHub Actions secret

   gh secret set AZURE_CLIENT_ID --env ci --body `az ad app list --display-name 'https://nrfassettracker.invalid/memfault-ci' | jq -r '.[].appId'`

2. Store the directory (tenant) ID of the service principal app registration created in step in the above step as a GitHub Actions secret

   gh secret set AZURE_TENANT_ID --env ci --body `az account show | jq -r '.tenantId'`

3. Store the ID of the subscription which contains the resources as a GitHub Actions secret

   gh secret set AZURE_SUBSCRIPTION_ID --env ci --body `az account show | jq -r '.id'`