Export Cognitive Service powershell script (qnamaker:kb:export) qna export mangles special characters

[edemehmetcan]

What project is affected?
Deployment script

What language is this in?
What happens?
Special characters :
Prompts:

Plus d'information sur la lettre reçu des éléctions ?
are exported like this

it needs to be - Plus d'information sur la lettre reçu des éléctions ?

What are the steps to reproduce this issue?
Steps to reproduce the behavior:

Create a knowledge base
Use the special characters in the Question or Answer field
Export using the update_cognitive_services.ps1 script
Inspect the resultant in Deployment/Resources/Qna/{language}/nameofknowledgebase.qna

What were you expecting to happen?
Support special characters such as é, ç, ë, à, etc.

Can you share any logs, error output, etc.?
Any screenshots or additional context?

I am reopening this issue, I have the same problem right now even if I have put "-encoding oem $($outJson)"

[Batta32]

Thanks @edemehmetcan for reporting this issue. We will review this issue following the repro steps you mentioned.

We will back to you as soon as we have any update for you 😊.

[edemehmetcan]

Thank you so much @Batta32 . I am waiting your answer.

Please also check this ? During export, there is an option "qnaFormat" it's a boolean that I put to true in order to have a .qna file in my repository but I have an error "Unexpected Argument: true"

https://www.npmjs.com/package/@microsoft/bf-qnamaker#bf-qnamakerkbexport

Why use that ? I want to export directly the .qna file and not pass by bf qnamaker:convert after to convert json to QnA file.

[Batta32]

@edemehmetcan how are you executing the bf qnamaker:kb:export command with the --qnaFormat argument?
It should be executed as follows:

bf qnamaker:kb:export --endpoint $qnaEndpoint --environment Prod --kbId $kb.kbId --subscriptionKey $kb.subscriptionKey --qnaFormat

See this example of the unit tests of the botframework-cli repository.

[edemehmetcan]

I have tried to export it like this and this is now working, is the syntax correct ?

        bf qnamaker:kb:export `
            --endpoint $qnaEndpoint `
            --environment Prod `
            --kbId $kb.kbId `
            --subscriptionKey $kb.subscriptionKey `
            --qnaFormat | Out-File -Force $($fileDestination)

[Batta32]

Yes! That syntax is correct too.

We will be reviewing the special characters issue now, and as soon as we have any update for you, we will back to this thread 😊.

[edemehmetcan]

Super thanks!

When I am trying without "Out-File", the export is correct (in the Terminal) with correct characters. Once exporting to a file.. The characters are again changed strangely...

[axelsrz]

Thanks @edemehmetcan for submiting this. Could you confirm if this is the same issue as in this one ?
And if it is, this PR should fix it

[edemehmetcan]

I have updated the botframework-cli and the same script has been launched in Azure DevOps (Azure CLI step), the characters are still changed... @axelsrz @Batta32

[edemehmetcan]

After reviewing the file, it's better than before. Now, I just have the problem with "é" and "ï"

[Batta32]

@axelsrz - should we transfer this issue to botframework-cli repository? As this appears to be an issue related to the bf-cli tool (bf qnamaker:kb:export specifically)

[edemehmetcan]

@Batta32 Even if I execute the qnamaker:kb:export locally with powershell. I still have the issue of characters. This is not something related to Azure CLI I would say...

[Batta32]

@edemehmetcan - appears not be related to the Azure CLI but probably related to the botframework-cli which is the interftace used to manage the Bot Framework bots and related services (see botframework-cli repository), this CLI contains the bf qna:maker:kb:export command that you are using 😊.

[hcyang]

Hi @edemehmetcan,

Can you try changing your PS config to use UTF? See a relevant link below.
https://stackoverflow.com/questions/40098771/changing-powershells-default-output-encoding-to-utf-8

[edemehmetcan]

Hello @hcyang thanks for your answer. The powershell is using UTF8.

I have noticed something, after exporting the qna and opened on Notepad++ the encoding is "UCS 2 le BOM" which is very strange...

And it's very ennoying to not have like "luis:export" a property --out for qnamaker:export...

Regards

[feich-ms]

Hi folks, I did some investigations and this seems not related with the bf-cli. It's more related with the powershell output encoding format issue. I use the cmd to call above export command and the output returns normal result for special chars, but when I use powershell to call the same command, then the output returns unexptected result. So this issue only happens to powershell with Out-File set. I tried some ways to set the default encoding to UTF in powershell, but seems not working. This is same issue with the one microsoft/botframework-cli#1057 @axelsrz pointed above.

As @edemehmetcan suggested, the optional fix is to add --out for kb export command so that it doesn't depend on Out-File or >> or > to output files

Add @munozemilio for more insights.

[feich-ms]

Created a PR in cli microsoft/botframework-cli#1065 to support output exported qna content to a file with --out option. With this changes in, users can use --out to specify the output file path. Tested that works for powershell.

[edemehmetcan]

@feich-ms Thanks for you answer. It's what I was looking for! I am waiting that the PR is merged and I will try.

[hcyang]

@munozemilio, can you review @feich-ms PR?

[edemehmetcan]

Hello, I just had a try after updated the botframework-cli to version 4.11.1

            bf qnamaker:kb:export `
            --endpoint $qnaEndpoint `
            --environment Prod `
            --kbId $kb.kbId `
            --qnaFormat `
            --out $outQnA `
            --subscriptionKey $kb.subscriptionKey `
            --force

I have this error: ".Unexpected arguments: --out"

[Batta32]

Hi @edemehmetcan, botframework-cli@4.11.1 was released 8 days ago and doesn't contain the PR microsoft/botframework-cli#1065 as it was merged 3 days ago.

Those changes will be integrated in the next release of the botframework-cli tool 😊.

[edemehmetcan]

Hi @Batta32 it's what I have thought about. Thanks for the clarification. When the next release will be scheduled ? :)

[Batta32]

@munozemilio & @feich-ms - is there any ETA for the new release of botframework-cli tool?

[edemehmetcan]

an update ? @munozemilio & @feich-ms @Batta32

[feich-ms]

@edemehmetcan sorry for the late response, as far as I know, next release will not be too early. To unblock you, you can install the latest nightly build version which already contains the fix. Just use npm install -g @microsoft/botframework-cli@next. It will install the latest one 4.12.0-dev.20201208.3315f42. Thank you.

[edemehmetcan]

@feich-ms thanks a lot, it works with the next release :) waiting for the real release then...

[edemehmetcan]

@feich-ms @Batta32 When the next release will be available ?

[feich-ms]

Hi @edemehmetcan, sorry for the late response. As far as I know, the next release will be mid of Feb. 2/12 is our code completion day and we may push the release one week later after that.