v1.9.0 (#197)

* Update smletsExchangeConnector.ps1 (#190) Fixed a small bug in the Set-AssignedToPerSupportGroup function that was preventing the "OOOVolume" DynamicWorkItemAssignment option from working as expected. * fix, Dynamic Analyst Assignment sort order (v1.x) (#191) Made a change to the Set-AssignedToPerSupportGroup function to address a small bug which selected the Analyst with the highest assigned Work Item count, rather than the lowest. * citation alignment (#194) Realigning the Apply Template code region with proper citation. * Azure Vision integration (v1.x) (#187) * Azure Vision functions Initial commit that declares the functions for Azure Vision Services that enable Azure to return words that describe the image (Get-AzureEmailImageAnalysis) and extract text from an image via OCR (Get-AzureEmailImageText). Will need to update both functions to work with binary streams instead of URLs so that as the connector parses attachments they can be uploaded and parsed by Azure within a processing loop of the connector. * Vision flow Introduced primary config variable for Azure Vision along with a variable placeholder for the API Key. Modified Attach-FileToWorkItem function with conditions to check for the file type to then decide if to send off to Azure Vision. Still need to update said Azure functions will support for memory stream/byte array instead of a local image. * File extension parsing When an attachment is retrieved through Exchange EWS, there does not appear to be a dedicated "file extension" property. As such, a small parsing mechanism has been introduced to Attach-FileToWorkItem so that the Extension property is now always filled out. This now makes the if statement that tests for common image types possible. * Including try/catch for file extension Wrapping previous statements in a try/catch just for the Extension property in the event a file is attached that doesn't end with a ".something" * byte array instead of memorystream Updating the call to the Azure Vision function to use the original Attachment Content's Byte Array of data as opposed to its Memory Stream. * logic for analysis and OCR Attach-FileToWorkItem now contains logic to decide how to define the Description property from Azure Vision. The top 5 image tags will always be written and if one of them is the word "text" then an OCR extraction attempt is made. If successful, it's appended to the Tags and then written into the File Attachment Description property * url to byte array input Azure Vision functions now take a byte array directly from the EWS Attachment * updating readme Adding information for Azure Vision integration * Azure Speech integration (v1.x) (#196) * Azure Speech services integration Adding support for transcribing wav/ogg files to text via Azure Speech services * Azure Speech integration screenshot for Azure Speech services * Azure Speech explanation Adding brief synopsis for Azure Speech services * HTTP Client issue (#201) Was previously receiving inconsistent results via PowerShell when contacting Azure Vision API. Adding the assembly call resolves this. * notes and documentation (#202) updating contributors and inline notes with github issues addressed Co-authored-by: bennyguk <benny_guk@hotmail.com>
AdhocAdam · Aug 23, 2020 · 8247650 · 8247650
1 parent 2d0457c
commit 8247650
Show file tree

Hide file tree

Showing 4 changed files with 192 additions and 11 deletions.
diff --git a/FeatureScreenshots/ACSVision.png b/FeatureScreenshots/ACSVision.png
diff --git a/FeatureScreenshots/speech-api.png b/FeatureScreenshots/speech-api.png
diff --git a/README.md b/README.md
@@ -13,6 +13,26 @@ The stock Exchange Connector is a seperate download for SCSM 2012+ that enables
 This is aimed at SCSM administrators looking to further push the automation limits of what their SCSM deployment can do with inbound email processing. As such, you should be comfortable with PowerShell and navigating SCSM via SMlets.
 
 ## What new things can it do?
+<table border="0">
+  <tr>
+    <td colspan="3"><i>Transcribe Audio Files to Text (v1.9)</i></td>
+  </tr>
+  <tr>
+    <td width="200"><img src ="/FeatureScreenshots/speech-api.png" /></td>
+    <td width="auto">Your SCSM environment can now hear! Translate incoming *.wav or *.ogg on emails to text with Azure Speech services. After enabling audio files will have their text transcriptions written to the File Attachment's Description. Get pricing details <a href="https://azure.microsoft.com/en-us/services/cognitive-services/speech-services/">here</a>.
+</td>
+  </tr>
+</table>
+<table border="0">
+  <tr>
+    <td colspan="3"><i>Describe and Extract Text from Images (v1.9)</i></td>
+  </tr>
+  <tr>
+    <td width="200"><img src ="/FeatureScreenshots/ACSVision.png" /></td>
+    <td width="auto">Give your SCSM environment eyes with integration into Azure Cognitive Services Vision API. Once enabled, any image attached to emails will be submitted to Azure and will be tagged with 5 words describing what the image is. In the event text is detected, Optical Character Recognition (OCR) will additionally be performed. These results are then written into the Description property of the attachment. Get pricing details <a href="https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/">here</a>.
+</td>
+  </tr>
+</table>
 <table border="0">
   <tr>
     <td colspan="3"><i>Predict Impacted Configuration Items with Azure Machine Learning (v1.8)</i></td>

diff --git a/smletsExchangeConnector.ps1 b/smletsExchangeConnector.ps1
@@ -10,7 +10,7 @@ enabling other organizational level processes via email
 
 .NOTES
 Author: Adam Dzyacky
-Contributors: Martin Blomgren, Leigh Kilday, Tom Hendricks, nradler2, Justin Workman, Brad Zima
+Contributors: Martin Blomgren, Leigh Kilday, Tom Hendricks, nradler2, Justin Workman, Brad Zima, bennyguk
 Reviewers: Tom Hendricks, Brian Weist
 Inspiration: The Cireson Community, Anders Asp, Stefan Roth, and (of course) Travis Wright for SMlets examples
 Requires: PowerShell 4+, SMlets, and Exchange Web Services API (already installed on SCSM workflow server by virtue of stock Exchange Connector).
@@ -20,6 +20,10 @@ Requires: PowerShell 4+, SMlets, and Exchange Web Services API (already installe
     Signed/Encrypted option: .NET 4.5 is required to use MimeKit.dll
 Misc: The Release Record functionality does not exist in this as no out of box (or 3rd party) Type Projection exists to serve this purpose.
     You would have to create your own Type Projection in order to leverage this.
+Version: 1.9.0 = #55 - Feature, Image Analysis (support for png, jpg, jpeg, bmp, and gif)
+                #5 - Feature, Optical Character Recognition (support for png, jpg, jpeg, bmp, and gif)
+                #54 - Feature, Speech to Text for Audio Files (support for wav and ogg)
+                #188 - Bug, Dynamic work item assignment not functioning as expected
 Version: 1.8.0 = #12 - Feature, Predict Affected/Impacted Configuration Item(s)
                 #175 - Bug, Get-TemplatesByMailbox: incorrect variable name
                 #88 - Bug, Verify-WorkItem does not handle null search results
@@ -439,6 +443,26 @@ $enableAzureTranslateForNewWI = $false
 $defaultAzureTranslateLanguage = "en"
 $azureCogSvcTranslateAPIKey = ""
 
+#optional, enable Azure Vision through Azure Cognitive Services
+#use Vision services from Azure in order to populate the Description of images attached to Work Items from email. By enabling, the image is first sent to
+#the Image Analysis API to attempt to describe the top 5 categories or Tags of the image. In the event one of these Tags is the word "text"
+#another call to the Optical Character Recognition (OCR) API will be made and attempt to extract text/words from the image.
+#Given the maximum length of the File Attachment's Description property is 255 characters, Tags will always be present but the OCR result could be chopped off.
+#For example a screenshot of an Outlook error message attached to an email would have these 5 Tags and the associated Description in the file's Description property.
+#Tags:screenshot,abstract,text,design,graphic;Desc:Microsoft Outlook Cannot start Microsoft Outlook. Cannot open the Outlook window.
+#pricing details can be found here: https://azure.microsoft.com/en-ca/pricing/details/cognitive-services/computer-vision/
+$enableAzureVision = $false
+$azureVisionRegion = ""
+$azureCogSvcVisionAPIKey = ""
+
+#optional, enable Azure Speech through Azure Cognitive Services
+#use Speech services from Azure in order to populate the Description of wav/ogg files attached to Work Items from email. By enabling, the audio file is first sent to
+#use the Speech to Text API in an attempt to convert the file to readable text.
+#pricing details can be found here: https://azure.microsoft.com/en-us/services/cognitive-services/speech-services/
+$enableAzureSpeech = $false
+$azureSpeechRegion = ""
+$azureCogSvcSpeechAPIKey = ""
+
 #optional, enable SCOM functionality
 #enableSCOMIntegration = set to $true or $false to enable this functionality
 #scomMGMTServer = set equal to the name of your scom management server
@@ -1808,8 +1832,49 @@ function Attach-FileToWorkItem ($message, $workItemId)
                 $NewFile = new-object Microsoft.EnterpriseManagement.Common.CreatableEnterpriseManagementObject($ManagementGroup, $fileAttachmentClass)
                 $NewFile.Item($fileAttachmentClass, "Id").Value = [Guid]::NewGuid().ToString()
                 $NewFile.Item($fileAttachmentClass, "DisplayName").Value = $attachment.FileName
-                #$NewFile.Item($fileAttachmentClass, "Description").Value = $attachment.Description
-                #$NewFile.Item($fileAttachmentClass, "Extension").Value =   $attachment.Extension
+                #optional, use Azure Cognitive Services Vision and OCR to set the Description property on the file
+                try
+                {
+                    $fileExtensionArrayPosition = $attachment.Name.Split(".").Length - 1
+                    $NewFile.Item($fileAttachmentClass, "Extension").Value = "." + $attachment.Name.Split(".")[$fileExtensionArrayPosition]
+                    if (((".png", ".jpg", ".jpeg", ".bmp", ".gif") -contains $NewFile.Item($fileAttachmentClass, "Extension").Value) -and ($enableAzureVision))
+                    {
+                        $azureVisionResult = Get-AzureEmailImageAnalysis -imageToEvalute $AttachmentContent
+                        $azureVisionTags = $azureVisionResult.tags.name | select-object -first 5
+                        $azureVisionTags = $azureVisionTags -join ','
+                        #if one of the Tags is "text" then attempt to extract text from the image through OCR as long as the confidence is greater than 90
+                        if ($azureVisionTags.contains("text"))
+                        {
+                            $AzureOCRConfidence = [math]::round((($azureVisionTags.tags | select-object name, confidence | ?{$_.name -eq "text"} | select-object confidence -ExpandProperty confidence) * 100), 2)
+                            if ($AzureOCRConfidence -ge 90)
+                            {
+                                $azureImageText = Get-AzureEmailImageText -imageToEvalute $AttachmentContent
+                                $ocrResult = $azureImageText.regions.Lines.words.text -join " "
+                                if ($ocrResult.length -ge 256){$ocrResult = $ocrResult.Substring(0,255)}
+                                #set the Description on the File Attachment with the Tags + OCR result
+                                $NewFile.Item($fileAttachmentClass, "Description").Value = "Tags:$($azureVisionTags);Desc:$ocrResult"
+                            }
+                            else
+                            {
+                                #The OCR confidence wasn't high enough to test
+                                $NewFile.Item($fileAttachmentClass, "Description").Value = "Tags:$($azureVisionTags)"
+                            }
+                        }
+                        else
+                        {
+                            #The returned Azure Vision Tags don't contain the word "text"
+                            $NewFile.Item($fileAttachmentClass, "Description").Value = "Tags:$($azureVisionTags)"
+                        }
+                    }
+                    if (((".wav", ".ogg") -contains $NewFile.Item($fileAttachmentClass, "Extension").Value) -and ($enableAzureSpeech))
+                    {
+                        $NewFile.Item($fileAttachmentClass, "Description").Value = (Get-AzureSpeechEmailAudioText -audioFileToEvaluate $attachmentContent).DisplayText
+                    }
+                }
+                catch
+                {
+                    #file doesn't have a parseable extension or the call to Azure Vision/Speech failed
+                }
                 $NewFile.Item($fileAttachmentClass, "Size").Value =        $MemoryStream.Length
                 $NewFile.Item($fileAttachmentClass, "AddedDate").Value =   [DateTime]::Now.ToUniversalTime()
                 $NewFile.Item($fileAttachmentClass, "Content").Value =     $MemoryStream
@@ -1846,8 +1911,51 @@ function Attach-FileToWorkItem ($message, $workItemId)
                 $NewFile = new-object Microsoft.EnterpriseManagement.Common.CreatableEnterpriseManagementObject($ManagementGroup, $fileAttachmentClass)
                 $NewFile.Item($fileAttachmentClass, "Id").Value = [Guid]::NewGuid().ToString()
                 $NewFile.Item($fileAttachmentClass, "DisplayName").Value = $attachment.Name
-                #$NewFile.Item($fileAttachmentClass, "Description").Value = $attachment.Description
-                #$NewFile.Item($fileAttachmentClass, "Extension").Value =   $attachment.Extension
+                #optional, use Azure Cognitive Services Vision, OCR, or Speech to set the Description property on the file
+                try
+                {
+                    $fileExtensionArrayPosition = $attachment.Name.Split(".").Length - 1
+                    $NewFile.Item($fileAttachmentClass, "Extension").Value = "." + $attachment.Name.Split(".")[$fileExtensionArrayPosition]
+                    if (((".png", ".jpg", ".jpeg", ".bmp", ".gif") -contains $NewFile.Item($fileAttachmentClass, "Extension").Value) -and ($enableAzureVision))
+                    {
+                        $azureVisionResult = Get-AzureEmailImageAnalysis -imageToEvalute $AttachmentContent
+                        $azureVisionTags = $azureVisionResult.tags.name | select-object -first 5
+                        $azureVisionTags = $azureVisionTags -join ','
+                        #if one of the Tags is "text" then attempt to extract text from the image through OCR as long as the confidence is greater than 90
+                        if ($azureVisionTags.contains("text"))
+                        {
+                            $AzureOCRConfidence = [math]::round((($azureVisionTags.tags | select-object name, confidence | ?{$_.name -eq "text"} | select-object confidence -ExpandProperty confidence) * 100), 2)
+                            if ($AzureOCRConfidence -ge 90)
+                            {
+                                $azureImageText = Get-AzureEmailImageText -imageToEvalute $AttachmentContent
+                                $ocrResult = $azureImageText.regions.Lines.words.text -join " "
+                                if ($ocrResult.length -ge 256){$ocrResult = $ocrResult.Substring(0,255)}
+                                #set the Description on the File Attachment with the Tags + OCR result
+                                $NewFile.Item($fileAttachmentClass, "Description").Value = "Tags:$($azureVisionTags);Desc:$ocrResult"
+                            }
+                            else
+                            {
+                                #The OCR confidence wasn't high enough to test
+                                $NewFile.Item($fileAttachmentClass, "Description").Value = "Tags:$($azureVisionTags)"
+                            }
+                        }
+                        else
+                        {
+                            #The returned Azure Vision Tags don't contain the word "text"
+                            $NewFile.Item($fileAttachmentClass, "Description").Value = "Tags:$($azureVisionTags)"
+                        }
+                    }
+                    if (((".wav", ".ogg") -contains $NewFile.Item($fileAttachmentClass, "Extension").Value) -and ($enableAzureSpeech))
+                    {
+                        $azureAudioTranscription = (Get-AzureSpeechEmailAudioText -audioFileToEvaluate $attachmentContent).DisplayText
+                        if ($azureAudioTranscription.Length -gt 255){$azureAudioTranscription = $azureAudioTranscription.Substring(0,255)}
+                        $NewFile.Item($fileAttachmentClass, "Description").Value = (Get-AzureSpeechEmailAudioText -audioFileToEvaluate $attachmentContent).DisplayText
+                    }
+                }
+                catch
+                {
+                    #file doesn't have a parseable extension or the call to Azure Vision failed
+                }
                 $NewFile.Item($fileAttachmentClass, "Size").Value =        $MemoryStream.Length
                 $NewFile.Item($fileAttachmentClass, "AddedDate").Value =   [DateTime]::Now.ToUniversalTime()
                 $NewFile.Item($fileAttachmentClass, "Content").Value =     $MemoryStream
@@ -1977,11 +2085,11 @@ function Set-AssignedToPerSupportGroup ($SupportGroupID, $WorkItem)
         #based on how Dynamic Work Item assignment was configured, set the Assigned To User
         if ($DynamicWorkItemAssignment -eq "volume")
         {
-            $userToAssign = $supportGroupMembers | foreach-object {Get-AssignedToWorkItemVolume -SCSMUser $_} | Sort-Object AssignedCount -Descending | Select-Object SCSMUser -ExpandProperty SCSMUser -first 1 
+            $userToAssign = $supportGroupMembers | foreach-object {Get-AssignedToWorkItemVolume -SCSMUser $_} | Sort-Object AssignedCount | Select-Object SCSMUser -ExpandProperty SCSMUser -first 1 
         }
         elseif ($DynamicWorkItemAssignment -eq "OOOvolume")
         {
-            $userToAssign = $supportGroupMembers | Where-Object {$supportGroupMembers.OutOfOffice -ne $true} | foreach-object {Get-AssignedToWorkItemVolume -SCSMUser $_} | Sort-Object AssignedCount -Descending | Select-Object SCSMUser -ExpandProperty SCSMUser -first 1
+            $userToAssign = $supportGroupMembers | Where-Object {$_.OutOfOffice -ne $true} | foreach-object {Get-AssignedToWorkItemVolume -SCSMUser $_} | Sort-Object AssignedCount | Select-Object SCSMUser -ExpandProperty SCSMUser -first 1
         }
         elseif ($DynamicWorkItemAssignment -eq "random")
         {
@@ -3089,9 +3197,62 @@ function Get-AzureEmailKeywords ($messageToEvaluate)
     #return the keywords
     return $keywordResult.documents.keyPhrases
 }
-#endregion
+function Get-AzureEmailImageAnalysis ($imageToEvalute)
+{
+    #azure cognitive services, vision URL
+    $imageAnalysisURI = "https://$azureVisionRegion.api.cognitive.microsoft.com/vision/v3.0/analyze?visualFeatures=Tags"
+
+    #adapted from C# per: https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts/csharp-print-text
+    Add-Type -AssemblyName "System.Net.Http"
+    $httpClient = New-Object -TypeName "System.Net.Http.Httpclient"
+    $httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "$azureCogSvcVisionAPIKey")
+    $content = New-Object "System.Net.Http.ByteArrayContent" -ArgumentList @(,$imageToEvalute)
+    $content.Headers.ContentType = "application/octet-stream"
+    $request = $httpClient.PostAsync($imageAnalysisURI,$content)
+    $request.wait();
+    if($request.IsCompleted) {$result = $request.Result.Content.ReadAsStringAsync().Result | ConvertFrom-Json}
+
+    #return the Vision API analysis
+    return $result
+} 
+function Get-AzureEmailImageText ($imageToEvalute)
+{
+    #azure cognitive services, vision URL
+    $imageTextURI = "https://$azureVisionRegion.api.cognitive.microsoft.com/vision/v3.0/ocr?detectOrientation=true"
+
+    #adapted from C# per: https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts/csharp-print-text
+    Add-Type -AssemblyName "System.Net.Http"
+    $httpClient = New-Object -TypeName "System.Net.Http.Httpclient"
+    $httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "$azureCogSvcVisionAPIKey")
+    $content = New-Object "System.Net.Http.ByteArrayContent" -ArgumentList @(,$imageToEvalute)
+    $content.Headers.ContentType = "application/octet-stream"
+    $request = $httpClient.PostAsync($imageTextURI,$content)
+    $request.wait();
+    if($request.IsCompleted) {$result = $request.Result.Content.ReadAsStringAsync().Result | ConvertFrom-Json}
+
+    #return the Vision API analysis
+    return $result
+}
 
-#region #### Modified version of Set-SCSMTemplateWithActivities from Morton Meisler seen here http://blog.ctglobalservices.com/service-manager-scsm/mme/set-scsmtemplatewithactivities-powershell-script/
+function Get-AzureSpeechEmailAudioText ($waveFileToEvaluate)
+{  
+    #build the request
+    $SpeechServiceURI = "https://$azureSpeechRegion.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-us"
+    $RecoRequestHeader = @{
+      'Ocp-Apim-Subscription-Key' = "$azureCogSvcSpeechAPIKey";
+      'Content-type' = "audio/wav; codecs=audio/pcm; samplerate=16000";
+      'Transfer-Encoding' = 'chunked'
+      'Except' = "100-continue"
+      'Accept' = "application/json"
+    }
+
+    #Pass the audio byte array into the body and submit the request
+    $RecoResponse = Invoke-RestMethod -Method POST -Uri $SpeechServiceURI -Headers $RecoRequestHeader -Body $waveFileToEvaluate
+
+    #return the result
+    return $RecoResponse
+}
+#endregion
 
 function Get-AMLWorkItemProbability ($EmailSubject, $EmailBody)
 {
@@ -3132,6 +3293,7 @@ function Get-AMLWorkItemProbability ($EmailSubject, $EmailBody)
     return ($probabilityMatrix)
 }
 
+#region #### Modified version of Set-SCSMTemplateWithActivities from Morton Meisler seen here http://blog.ctglobalservices.com/service-manager-scsm/mme/set-scsmtemplatewithactivities-powershell-script/
 function Update-SCSMPropertyCollection
 {
     Param ([Microsoft.EnterpriseManagement.Configuration.ManagementPackObjectTemplateObject]$Object =$(throw "Please provide a valid template object")) 
@@ -3182,6 +3344,7 @@ function Apply-SCSMTemplate
     #Apply update template
     Set-SCSMObjectTemplate -Projection $Projection -Template $Template -ErrorAction Stop @scsmMGMTParams
 }
+#endregion
 
 function Remove-PII ($body)
 {
@@ -3203,8 +3366,6 @@ function Remove-PII ($body)
     return $body
 }
 
-#endregion 
-
 #region #### SCOM Request Functions ####
 function Get-SCOMAuthorizedRequester ($sender)
 {