[Typescript] Add GA SDK Skill support to VA/Skill Generator

[darrenj]

Following a Skill protocol change to WebSockets we have been waiting on a new Streaming Extensions library to be available for Typescript, previously only available for C#. This will unblock Skill invocation scenarios for Virtual Assistant when using Typescript.

User Story

As ... a Developer
I want ...to be able to connect skills to a TypeScript version of the Virtual Assistant
so that ...there is parity between C# and Typescript

Acceptance Criteria

• Update VA and Skill generator to incorporate new GA SDK Skills
• Deploy a typescript VA and add a C# Skill (e.g. Calendar)
• Provide documentation on the changes required to existing VA solutions and Skills created by the generator as needed

[Kumar3010]

hi Darrenj,
Is there any ETA on the issue to be fixed?
thanks,
Kumar

[darrenj]

Hoping to have end to end working for testing this week all being well.

[Kumar3010]

Thank you @darrenj for the update.

[Kumar3010]

is there any update on this issue @darrenj

[Kumar3010]

Hi Darren,

@darrenj could you please provide an update on this work item. we need this critical feature to be developed for our use case.

appreciate your help.

thanks,
Kumar

[tommyJimmy87]

Hi @darrenj ,

Sorry to bother you again, there are any update on this?
I saw that the library has been released for the version 4.6.0 (https://botbuilder.myget.org/feed/botbuilder-v4-js-daily/package/npm/botframework-streaming), is this compatible with the Botframework 4.5.0 and the Virtual Assistant?

[darrenj]

There are a few things in flight here, one is the streaming extension availability which you've spotted and the other is the typescript availability of the new GA version of the Skills library. I am expecting the first packaged build of the Skills library next week at which point we can start updates of both the C# and Typescript templates to test and provide the (simple) migration steps.

We are targeting having this work ready for use in January and will keep this issue updated with progress including when daily builds are available.

In the meantime a Javascript/Typescript VA Bot can call a Skill created in JavaScript/Typescript as this uses HTTP due to the lack of the streaming extension dependency. The issue comes in when you wish to call a C# Skill like Calendar which the above changes will align across all languages.

[amistry79]

@darrenj is this fixed and can I connect C# Skill from typescript VA? I am confused about your last comment.

[rjgmail88]

Hello @darrenj ,

Once this resolved, would it be possible to invoke cSharp skills like ToDo from a TypeScript version on VA ?

Currently, I am getting an error related to WebSocketTransport used on C# skill. But my VA is typescript.

[darrenj]

Yes, migration to the GA version of Skills removes the dependency on Websockets and thus will enable seamless mix/match of skills across platforms. We have the latest bits for C# and JS and working on initial integration, goal is to get changes into a daily build within the next couple of weeks to unblock scenarios.

[tommyJimmy87]

Hi @darrenj,

I have another use case which is Typescript VA and Typescript Skill.

My problem is that the updateActivity is not available across the VA to the Skill and I need the streaming library for this (as far as I understood). Now I actually noticed that the WebSocket dependencies have been removed from the latest skills library (4.4.9).
Don't understand why :) Are going to be moved somewhere else? Or?

[darrenj]

OK, I have an update on progress with regard to the GA of the Skill capabilities within the Bot Framework SDK. This has transitioned the work incubated by Virtual Assistant into a formal GA version.

A sample is available that shows how a Bot can invoke another Bot (Skill). Key change is we now have a unified adapter meaning there is no need to utilise a different adapter or Streaming extensions. C# and JS.

We are working on updates to the Virtual Assistant / Skill template to provide the (small) changes out of the box and provide documentation on the few steps required to update existing Skills. Targeting end of January, with builds on our daily builds soon.

If you are keen to make progress ahead of this the above samples should give you everything you need. I'll update this issue with pointers to daily builds/changes as soon as they are ready.

[rjgmail88]

@darrenj thank you for the this awesome update. I am just curious to know about unified adapter before we start leveraging that in our production Virtual assistant.

Does this mean we can use unified adapter in Typescript VA and can still invoke c# skill ?

[darrenj]

A long awaited update!

We posted our 0.8 release yesterday which include an update to all of our sample Skills to support the new GA Skills capability and no longer preview. Websockets has been retired in favour of Web-Service communication between skills, Skills communicate back to the Bot asynchronously using a "callback" HTTP connection.

As a result there are some straight forward updates to existing Virtual Assistant and Skills to be made. C# VA updates are documented here and C# Skill updates are documented here.

The equivalent JS/Typescript generator changes are being finalised as we speak and targeting publishing of a new Generator and docs for existing VA/Skills by 14th February. For background reference, the high level steps to support invoking skills can be found here

Once these changes are complete it means a JS/typescript VA can invoke C# Skills, vice versa and indeed Python too.

[darrenj]

@tomSauret847 let me know if you see any other issues. We are completing final validation and will push new builds to npm once complete.

[tommyJimmy87]

Hi @darrenj , I got an error when trying to resolve the lg files with the new template manager, basically what happens is that I'm receiving the locale en-US from MS Teams but I set only en-us ( lower case ) and is not going to be recognized. Maybe adding a lower case here if (this.templateEnginesPerLocale.has(locale))would resolve the problem (LocaleTemplateEngineManager.ts line 61) . To circumvent this problem for the moment I had to add this in my index.ts :

const supportedLocales: string[] = ['en-us', 'en-US','de-de'];

supportedLocales.forEach((locale: string) => {
    const localeTemplateFiles: string[] = [];
    templateFiles.forEach(template => {
        // LG template for en-us does not include locale in file extension.
        if (locale === 'en-us' || locale === 'en-US') {
            localeTemplateFiles.push(path.join(__dirname, 'responses', `${ template }.lg`));
        }
        else {
            localeTemplateFiles.push(path.join(__dirname, 'responses', `${ template }.${ locale }.lg`));
        }
    });

    localizedTemplates.set(locale, localeTemplateFiles);
});

This works well but I don't know if can be done in a better way.

[darrenj]

Thanks. @Batta32 - can you check to see if you have the change to LocaleTemplateEngine in the solutions Lib which uses the SDK provided MultiLanguageLG ? I would hope this would elegantly handle mixed case but some tests would be good to validate.

@lzc850612 Can we run a quick validation with the csharp GA RC to double check it works as expected with mixed case locales coming into the Bot on an Activity?

[Batta32]

Sure @darrenj, we will be checking that change!

[tomSauret847]

@darrenj thank you for the update on the progress! I had the same locale issue that tommyJimmy pointed but got past it by add locale.toLowerCase in the localeTemplateManager. I am currently attaching a custom skill to the VA but still receiving an error I am trying to track down. The skill is returning a 501 error to the VA but it is processing the message in the onMessage function in the defaultActitivityHandler.ts, it is just not passing the activity over to the main dialog for the skill to process. I added the skill by updating the appSettings.json file, and the skill is receiving the activity.

[tomSauret847]

@Batta32 I was wanting to check if the path forward still includes the localTemplateEngineManager. I noticed it is missing from the preview builds and only responseManager is present. Moving forward are we going to need to use the Response manager or is the Locale template going to be added back?

[tomSauret847]

@Batta32 I was able to track down the error that I was receiving on communicating with a skill from the VA. In the dialogEx module there is a send trace that is sending the trace back to the VA that the skill was started. This sendActivity is throwing the following error when called:
Error: /api/messages/v3/conversations/5659a380-841b-11ea-bed3-831b5a6aa1eb%7Clivechat-dlpskill-emulator-skillconvo/activities/5b61c920-841b-11ea-b148-8581dc55a9bb does not exist
When the conversation ID is appended in SkillConversationIdFactory the skill cannot post back to the new conversation ID.

Update:
I had a typo in the skillHostEndpoint that was causing this issue. Once I corrected that I am now able to attach skills and have them interact with the VA.

[tommyJimmy87]

@Batta32 @darrenj I'm facing an unexpected behavior when trying to use the chitchat with the VA: basically every time the user asks something to the Bot the introstep will reprompt the initial message, which in the case of chitchat is not great as UX. Is this an expected or I'm missing something?

(the please hold on message is a custom message don't worry about that
)

[tommyJimmy87]

@Batta32 I was able to track down the error that I was receiving on communicating with a skill from the VA. In the dialogEx module there is a send trace that is sending the trace back to the VA that the skill was started. This sendActivity is throwing the following error when called:
Error: /api/messages/v3/conversations/5659a380-841b-11ea-bed3-831b5a6aa1eb%7Clivechat-dlpskill-emulator-skillconvo/activities/5b61c920-841b-11ea-b148-8581dc55a9bb does not exist
When the conversation ID is appended in SkillConversationIdFactory the skill cannot post back to the new conversation ID.

Update:
I had a typo in the skillHostEndpoint that was causing this issue. Once I corrected that I am now able to attach skills and have them interact with the VA.

@tomSauret847 What was your problem? because I'm having the same error and don't understand what is it. In my Virtual Assistant I have this :
"skillHostEndpoint": "https://1c34e472.ngrok.io/api/skills"
Which is the endpoint of the VA.

Error: /api/skills/v3/conversations/a%3A1QIRXOg6VwVn2aGC-eTj9sj3YYd6YUVrRKf8J7wLrjH1fPYcpn2dYpMyQpr_aHLHSp9Z2SHVzu8lKbdj_ArEh06kvjyF48IG1tgE1ctKTWdIASDloNgy51_eBFoGzwwIN-waSearchSkill-msteams-skillconvo/activities/1588679971133 does not exist
    at new RestError (/Users/em.tomaselli/Desktop/repository/BMW/wa-search-skill/node_modules/@azure/ms-rest-js/dist/msRest.node.js:1397:28)
    at /Users/em.tomaselli/Desktop/repository/BMW/wa-search-skill/node_modules/@azure/ms-rest-js/dist/msRest.node.js:1849:37
    at process._tickCallback (internal/process/next_tick.js:68:7)

[tomSauret847]

@Batta32 I was able to deploy my VA and 1 skill to Azure. When I use the web chat everything works as expected. When I try to use the teams channel I receive the below error. This seems to be isolated to the Teams channel since the web chat is routing correctly.
Error: Error invoking the skill id: "dlpskill" at "https://{app name}.azurewebsites.net/api/skill/messages" (status is 500).
Error: /api/skills/v3/conversations/a:1lFWWXG9CVtdrkA76VLFeS9CbYWzsHVr9mQ4S7BqDpL4K5WNIc0ND6mqaaj_QsSqH-qUU-Gb2vLX3HQ4CpG3ZDPCbR0XKLtJGOqW0kwKUbe2HKv5C9fs1sjLSO4EvoE5h-dlpskill-msteams-skillconvo/activities/1588712064379 does not exist
at SkillDialog. (D:\home\site\wwwroot\node_modules\botbuilder-dialogs\lib\skillDialog.js:164:23)
at Generator.next ()
at fulfilled (D:\home\site\wwwroot\node_modules\botbuilder-dialogs\lib\skillDialog.js:11:58)
at process._tickCallback (internal/process/next_tick.js:68:7)

@tommyJimmy87 I was not using ngrok to communicate from VA to skill locally so I was able to use the localhost:3979/api/skills for the skillHostEndpoint and was able to reach the skill. Since you are using the Teams channel as well I will say this is a issue with the Teams channel. You can try reaching the skill using the web chat and see if you can communicate with the skill from the VA in azure.

[tommyJimmy87]

Hi @Batta32, I agree with @tomSauret847 that most likely there's some kind of problem whit the Teams Channel.

After a bit of debugging I might try to explain what I have found out:

1. In the ChannelServiceHandlerall the methods are not implemented, meaning that when one of the /api/skills is called will everytime return an error code;

i.e. File : ChannelServiceHandler.ts (botbuilder) , line : 219

    protected async onGetActivityMembers(claimsIdentity: ClaimsIdentity, conversationId: string, activityId: string): Promise<ChannelAccount[]> {
        throw new StatusCodeError(StatusCodes.NOT_IMPLEMENTED, `ChannelServiceHandler.onGetActivityMembers(): ${StatusCodes.NOT_IMPLEMENTED}: ${STATUS_CODES[StatusCodes.NOT_IMPLEMENTED]}`);
    }

2. /api/skills/v3/conversations/a%3A1QIRXOg6VwVn2aGC-eTj9sj3YYd6YUVrRKf8J7wLrjH1fPYcpn2dYpMyQpr_aHLHSp9Z2SHVzu8lKbdj_ArEh06kvjyF48IG1tgE1ctKTWdIASDloNgy51_eBFoGzwwIN-waSearchSkill-msteams-skillconvo/activities/1588761991793 In this case instead there's some problem in parsing the conversation ID and the error result is always that the resource doesn't exists, but if you try to call the same URL changing the conversationId with a simpler string it will work ( but the result will be still that the method is not implemented because of the previous point).

[Batta32]

Hi @tommyJimmy87, @tomSauret847 - sorry for the delay. We will be reviewing the Teams Channel's scenario using the GA Skills SDK changes and we will back to this thread later 😊.

[Batta32]

@tommyJimmy87 - we successfully reproduced the issue using Microsoft Teams Channel. We will back to this thread as soon as we fix the problem.

Thanks!

[Batta32]

Hi @tommyJimmy87, @tomSauret847, we successfully found the reason of the problem.

The problem is that the ':' character is being replaced to '%3A' when the activities are sent back to the Virtual Assistant from the Skill, generating the mismatching of conversationId between both bots.
Apparently, this behavior is executed in axios dependency replacing the characters as mentioned.

Last but not least, there is an issue in microsoft/botbuilder-js#2182 which replaces axios in botbuilder and it's tagged to be ready for R10.

We are going to update the Known Issues document explaining the problem

Finally, we will be back to this thread as soon as we have new updates for you guys 😊

Mismatching of conversationIds

axios dependency replaces ':' to '%3A'

[tommyJimmy87]

@Batta32 Do you think will be fixed soon or will take a bit of time?

[Batta32]

@tommyJimmy87, @tomSauret847 - we noticed that the problem is in ms-rest-js instead of axios.

The problem is a mismatch of conversationId between the Skill and the Virtual Assistant. This issue is raised because the ':' character of the conversationId from Teams is being replaced to '%3A' which should be replaced back to ':'. The last conversion is not implemented in sendOperationRequest as axios does.

Sorry for the mistake 😊.

encodeURIComponent doesn't replace the '%3A' back to ':'

Differences of encoding between axios and ms-rest-js

[EricDahlvang]

@Batta32 the default agent for ms-rest-js versions < 2.0 is axios https://github.com/Azure/ms-rest-js/blob/1.x/lib/axiosHttpClient.ts

I think one way to resolve this issue would be to use encodeURI within the SkillConversationIdFactory implementation. This way, the underlying agent will not alter the conversation.id from what is stored within the factory.

[Batta32]

Thanks @EricDahlvang, we will be reviewing the alternative of use encodeURI within SkillConversationIdFactory!

[Batta32]

@EricDahlvang - adding the encodeURI in the SkillConversationIdFactory doesn't fix the issue as the Skill is already encoding the conversationId before sending the activities back to the Virtual Assistant.

We realized that the issue might be related to the conversationId's length when it's received from Microsoft Teams. We noticed that using the original conversationId the VA's endpoint is not hitted, however, trimming some characters from the conversationId is correctly hitted.

Last but not least, this issue is not present in C#.

The endpoint is not hitted with the original conversationId

The endpoint is hitted with the trimmed conversationId

The issue is not present in C#

[EricDahlvang]

@Batta32 If the SkillConversationIdFactory uses encodeURI, then the skill will receive that conversation.id and respond to it accordingly. I'm pretty sure this is required.

Also yes, there is an issue with long conversation ids while using restify. We make mention of how to resolve the issue in this sample: https://github.com/microsoft/BotBuilder-Samples/blob/master/samples/javascript_nodejs/80.skills-simple-bot-to-bot/simple-root-bot/skillConversationIdFactory.js#L21

// This key has a 100 character limit by default. Increase with restify.createServer({ maxParamLength: 1000 }); in index.js.

[tomSauret847]

@EricDahlvang Thank you for this information. I added the update to both my VA and skill. I am now able to get the VA to communicate with the skill in the Teams channel without error.

[Batta32]

Thanks @EricDahlvang for the provided information! We created the PR #3359 improving the maxParamLength and also updating the version of @types/restify@^8.4.2.

Thanks guys @tommyJimmy87, @tomSauret847, as soon as these changes are merged, you will be able to test it 😊.

[tommyJimmy87]

@Batta32 sorry to bother you again, first of all, I applied the changes and now it is working. Now I have a problem with the update activity of skill through the VA, within the SkillHandler there's no implementation for that, could you provide one? Thx

[Batta32]

Hi @tommyJimmy87, we noticed that the SkillHandler in C# and JS doesn't contain the onUpdateActivity() as you mentioned, meanwhile you can add your own handler extending from SkillHandler or ChannelServiceHandler.

@darrenj - can you confirm that the absence of onUpdateActivity() method in the SkillHandler is correct or is there any plan to implement it?

[tommyJimmy87]

@Batta32 I actually tried this, the implementation looks like this :

    protected async onUpdateActivity(claimsIdentity: ClaimsIdentity, conversationId: string, activityId: string, activity: Activity): Promise<ResourceResponse> {
        return await this.updateActivity(claimsIdentity, conversationId, activityId, activity);
    }

And the updateActivity function is the following:

private async updateActivity(claimsIdentity: ClaimsIdentity, conversationId: string, replyToActivityId: string, activity: Activity): Promise<ResourceResponse> {

        let skillConversationReference: SkillConversationReference;
        try {
            skillConversationReference = await this.conversationIdFactory.getSkillConversationReference(conversationId);
        } catch (err) {
            // If the factory has overridden getSkillConversationReference, call the deprecated getConversationReference().
            // In this scenario, the oAuthScope paired with the ConversationReference can only be used for talking with
            // an official channel, not another bot.
            if (err.message === 'Not Implemented') {
                const conversationReference = await this.conversationIdFactory.getConversationReference(conversationId);
                skillConversationReference = {
                    conversationReference,
                    oAuthScope: JwtTokenValidation.isGovernment(this.channelService) ?
                        GovernmentConstants.ToChannelFromBotOAuthScope :
                        AuthenticationConstants.ToChannelFromBotOAuthScope
                };
            } else {
                // Re-throw all other errors. 
                throw err;
            }
        }

        if (!skillConversationReference) {
            throw new Error('skillConversationReference not found');
        }
        if (!skillConversationReference.conversationReference) {
            throw new Error('conversationReference not found.');
        }

        const activityConversationReference = TurnContext.getConversationReference(activity);

        /**
         * Callback passed to the BotFrameworkAdapter.createConversation() call.
         * This function does the following:
         *  - Caches the ClaimsIdentity on the TurnContext.turnState
         *  - Applies the correct ConversationReference to the Activity for sending to the user-router conversation.
         *  - For EndOfConversation Activities received from the Skill, removes the ConversationReference from the
         *    ConversationIdFactory
         */
        const callback = async (context: TurnContext): Promise<void> => {
            const adapter: BotFrameworkAdapter = (context.adapter as BotFrameworkAdapter);
            // Cache the ClaimsIdentity and ConnectorClient on the context so that it's available inside of the bot's logic.
            context.turnState.set(adapter.BotIdentityKey, claimsIdentity);
            context.turnState.set(this.SkillConversationReferenceKey, activityConversationReference);
            activity = TurnContext.applyConversationReference(activity, skillConversationReference.conversationReference) as Activity;
            const client = adapter.createConnectorClient(activity.serviceUrl);
            context.turnState.set(adapter.ConnectorClientKey, client);

            context.activity.id = replyToActivityId;
            await context.updateActivity(context.activity);
            return;
        };

        // Add the channel service URL to the trusted services list so we can send messages back.
        // the service URL for skills is trusted because it is applied based on the original request
        // received by the root bot.
        AppCredentials.trustServiceUrl(skillConversationReference.conversationReference.serviceUrl);

        await (this.adapter as BotFrameworkAdapter).continueConversation(skillConversationReference.conversationReference, skillConversationReference.oAuthScope, callback);
        return { id: uuid() };
    }

It's basically a copy of the processActivity function that is already implemented in the SkillHandler but calling the context.updateActivity(context.activity) at the end.

Said that I'm getting this error :

(node:68325) UnhandledPromiseRejectionWarning: Error: Failed to decrypt conversation id 

Probably there's something that I'm missing or I'm doing wrong.

[Batta32]

@tommyJimmy87 - we will try to reproduce your scenario and we will back to you later 😊.

[tomSauret847]

@Batta32 I noticed on the "Help card" with the Virtual assistant that it is not displaying the help card for the active skill. When you hit the "help" interrupt while in the skill dialog it is still presenting the help card for the VA and not the skill. I have updated to the last release that included setting the activeSkillProperty state with the skill that is active. Any ideas on how to fix this issue? Logging out the state the active dialog is still registered as a dialog I have in the VA and not the active skill.

[Batta32]

Hi @tommyJimmy87, @tomSauret847, thanks for reporting these issues.

As generator-bot-virtualassistant@1.0.0, bot-solutions@1.0.0 and botskills@1.0.15 are published in npmjs and merged in branch master, you can create the issues in the repository in order to explain better the problem and we will follow up them 😊.