Dialog context not showing up in Power Bi report

[tomSauret847]

What project is affected?

Power Bi report for Virtual Assistant and skills

What language is this in?

TypeScript

What happens?

We are unable to get any dialog context to show up in the power Bi dashboard. We have updated the telemetry logger to record PII but are still not getting any dialog context in the report. All other tabs are populating correctly we are just unable to see any output on the dialog tabs

What are the steps to reproduce this issue?

Create Virtual Assistant,
Add the Boolean value "true" to the telemetry logger
const telemetryLoggerMiddleware = new TelemetryLoggerMiddleware(telemetryClient, true);
const telemetryInitializerMiddleware = new TelemetryInitializerMiddleware(telemetryLoggerMiddleware, true);
check the "All Dialogs Overview" and "Dialog Overview" tabs in the power Bi report

What were you expecting to happen?

Dialog information to be populated in the report

Can you share any logs, error output, etc.?

No errors are present

Any screenshots or additional context?

[Batta32]

Thanks @tomSauret847 - we will be reviewing this issue and we will back to you asap!

[peterinnesmsft]

Hi @tomSauret847, just to provide an update, we are actively working on investigating this issue and working through reproducing this issue. Once we have been able to do so, we can provide next steps from there. Thank you for your patience and apologies for the delays!

[Batta32]

Hi @tomSauret847 - we successfully reproduced the issue.

Also, we tried to reproduce this issue in the TypeScript Skill as well as in a C# Virtual Assistant/Skill but they correctly worked, so this seems to be related to the TypeScript Virtual Assistant only.

We would like to validate with you the repro steps that we followed to accomplish this:

Repro steps

1. Install Power BI Desktop

2. Download the Virtual Assistant analytics Power BI template

3. Deploy a TypeScript Virtual Assistant

4. Modify the index file adding true the following two constructors, as seen in the image:
   

5. Open the Virtual Assistant analytics template file in Power BI application

6. Obtain from your Application Insights resources, the Application Insights id

7. Add the previous data in the Power BI configuration

8. Start the bot

9. Check that the Power BI is correctly receiving the metrics

10. Verify that the all dialogs overview page it's not receiving any kind of information

We used as a guide the Power BI documentation to configure and obtain the Virtual Assistant Insights metrics.

As soon as we have any update, we will back to you 😊.

Issue reproduced using TypeScript Virtual Assistant

[tomSauret847]

Thank you @Batta32 for looking into this. I did follow the steps you listed above and received the same result. it seems that this started after the upgrade to the published NPM package as the first version we had built before it went GA was populating the the dialog context in the Power BI report.

[Batta32]

Hi @tomSauret847 - we successfully fixed the issue in the TypeScript Virtual Assistant upcoming 1.0 release adding those changes in the PR #3584, specifically this commit.

As a workaround to get your TypeScript Virtual Assistant working, do the following changes in these files:

• mainDialog.ts
• onBoardingDialog.ts
• index.ts

1. Update the constructor of the dialogs adding the telemetryClient object

   //Add the BotTelemetryClient dependency from botbuilder
   import {
       ...
       BotTelemetryClient } from 'botbuilder';
   
   // At the constructor parameter list, add telemetryClient as the final parameter
   public constructor(
       ...,
       telemetryClient: BotTelemetryClient
   ) {
       // Inside the constructor assign the incoming telemetryClient to the local property
       this.telemetryClient = telemetryClient;
       });
   }

2. Send the telemetryClient object in the dialog initialization of the index.ts

   // Add the telemetryClient as the final parameter in the OnboardingDialog constructor
   const onboardingDialog: OnboardingDialog = new OnboardingDialog(..., telemetryClient);
   
   // Add the telemetryClient as the final parameter in the MainDialog constructor
   const mainDialog: MainDialog = new MainDialog(
        ...
        telemetryClient
   );

3. Once this is done, follow up these steps to setup the metrics in your Virtual Assistant using Power BI

4. Check that the Power BI is correctly receiving the metrics

5. Verify that the all dialogs overview page it's receiving the bot information

If you have any questions or changes, don't hesitate to ask us.

Virtual Assistant metrics using the changes

[tomSauret847]

Thank you @Batta32! This did resolve the issue and we are now seeing the dialog context show up in the Power Bi report.

[tomSauret847]

@Batta32 I did have a question. We are seeing the dialogs in the reporting now but all the skills are showing up under the "Switch Skill Dialog" and not the skill. We do have the telemetry enabled on all the skills and are passing it into the constructor the same as the VA. Am I missing a setting in the skill dialog constructor?

[Batta32]

Thanks @tomSauret847 for your reply! We will be reviewing this scenario and we will back to you as soon as we have any update 😊!

[tomSauret847]

Thank you again @Batta32, I also noticed that the QNAMakerDialog that is used for the QNA makers, has no telemetry client passed into it. This prevents the QNA maker from populating in the report as well.

[VictorGrycuk]

Thank you @tomSauret847 bringing to our attention that the QnAMakerDialog issue. A fix we found is to modify the tryCreateQnADialog method and set the telemetryClient property in the QnAMaker Dialog (see image line # 201 & # 214).

This is because QnAMakerDialog extends from WaterfallDialog, which in turns extends from dialogthat does have a telemetry client property but set as NullTelemetryClient.

We tested this scenario on the TypeScript Virtual Assistant 1.0 branch and it is fixed there without the need to pass the telemetry client as parameter. Also using the C# bots of master this is working as expected.

Please let us know if this fixes the telemetry registration for QnAMakerDialog.

We will continue reviewing the issue related to SwitchSkillDialog.

[tomSauret847]

Thank you @VictorGrycuk for this information. I have made the changes and we are seeing the QNA maker dialogs populating in the reporting now as shown below. But the QNA performance is still not populating. We are hitting the QNA makers as shown by the dialogs, but the performance is not being populated in the reporting.

[Batta32]

Thanks @tomSauret847 we will back to you as soon as possible!

[VictorGrycuk]

Hi @tomSauret847, we reproduced the issue you mentioned about the SwitchSkillDialog and determined that this happens by design as in C# the behavior is the same.

An alternative is to set the Instrumentation Key for the Virtual Assistant Application Insights in the appsettings.json of the Skills.

1. Follow the steps to activate the telemetry
2. After deploying the Virtual Assistant, copy the instrumentationKey value from its appsettings file
3. After deploying the skill, replace skills, replace their instrumentationKey value with the one copied in step 2
4. Start the Virtual Assistant and call the skills
5. The calls to the skill dialogs will start to being registered in Power BI

Note: Have in mind that dialogs that have the same name in multiple bots, such as MainDialog, will be grouped together.

We will be attentive to your reply, and we will be reviewing the issue of QnA Performance 😊.

More details

Since at the moment of deployment each bot create their own application insight resource, the telemetry of each bot is isolated from each other, so to work around this and see the telemetry of all the bots in the same Power BI dashboard it is necessary to set the same instrumentation key in their settings.

We investigated why SwitchSkillDialog was being registered in the Power BI dashboard, and found it registers the id property of the dialogs.

With this in mind we checked what was the id of the SwitchSkillDialog at the moment of switching, and found that it will always have its own name as id, no matter the skill it is switching to.

[tomSauret847]

Thank you @VictorGrycuk I made the changes you outlined above and now all the skill dialogs are showing in the Power Bi reporting. After setting the app insights key in the skills the same as the VA we are populating all the dialogs used in the Skills as well as in the VA.

[VictorGrycuk]

Excellent @tomSauret847, good to know it worked.
We will now be focusing on why the QnA performance is not populated, as soon as we have any update we will let you know.

[matiasroldan6]

Hi @tomSauret847, we just reproduced the issue you reported about QnA in TypeScript not displaying performance.

We tested this with an Assistant using master branch as well as the 1.0 release version, with equal results in both cases.

Environment used:

• TS Virtual Assistant with the telemetry lines enabled in index.ts. Deployed to Azure
• App Insights connected to the Assistant
• Power BI with Virtual Assistant analytics template

Steps used:

1. Talk to the Virtual Assistant using an utterance that would trigger QnA (e.g. "what is a virtual assistant")
2. Open the Virtual Assistant analytics template from Power BI, using the Application Id from the App Insights that's connected to it
3. Check that the QnA is not showing up performance

We will work on a fix for this both in master and 1.0. Thanks again for reporting.

[matiasroldan6]

Hi @tomSauret847, we could reproduce the issue with the QnA Maker dialogs not displaying performance in Power BI.
Unfortunately, it seems to be an issue with the BotBuilder SDK so we raised the issue microsoft/botbuilder-js#2779. You can track that issue in order to have the news of the problem.

In order to display the QnA performance information it's necessary to, among other things, enable the transmission of personally identifiable information (PII) as mentioned in this comment of the issue #3447.

We observed though, that the referenced flag logPersonalInformation is not present in the QnAMakerDialog class in BotBuilder-JS, as opposed to the same class from BotBuilder-Dotnet.
On top of that, the creation of the QnAMaker object at the method getQnAClient is omitting once again passing the flag through constructor for logging personal information, as seen below.

We tested doing those changes by manually modifying the QnAMakerDialog class in node_modules, in our local bot, adding the logPersonalInformation flag forced to true:

public logPersonalInformation: BoolExpression = new BoolExpression(true);

Then handing over the flag to QnAMaker at the getQnAClient method:

    /**
     * Creates and returns an instance of the QnAMaker class used to query the knowledgebase.
    **/
    private getQnAClient(): QnAMaker {
        const endpoint = {
            knowledgeBaseId: this.knowledgeBaseId,
            endpointKey: this.endpointKey,
            host: this.getHost()
        };
        const options = this.getQnAMakerOptions();
        return new QnAMaker(endpoint, options, this.telemetryClient, this.logPersonalInformation.value);
    }

And we could see the QnA information in Power BI

[tomSauret847]

Thank you @matiasroldan6 for looking into this. I have subscribed to that issue so I will stay updated as they implement a fix for it.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[tomSauret847]

@matiasroldan6 Are we still waiting for this to be fixed in the Botbuilder-js repo?

[Batta32]

Yes @tomSauret847, this last issue was a disparity in the SDK that we raised in microsoft/botbuilder-js#2779. The SDK team confirmed the disparity and they are currently working to solve this.

If you agree, we can close this issue and as soon as we any update about the SDK team, we will back to you here 😊.

[tomSauret847]

Thank you @Batta32 I will close this issue and watch the one you provided.