Skip to content
This repository has been archived by the owner on Feb 27, 2024. It is now read-only.

Action not work via Assistant #8

Open
nimish17 opened this issue Jul 1, 2019 · 53 comments
Open

Action not work via Assistant #8

nimish17 opened this issue Jul 1, 2019 · 53 comments

Comments

@nimish17
Copy link

nimish17 commented Jul 1, 2019 •
edited

action is running in the device via App Actions Test Tool but not working via voice or written commands in the Assistant.

below all steps are perfect in my case
The plugin loaded the actions preview at least one time.
The account in the device is the same as the one in Android Studio.
The device has access to an internet connection.
You are using the latest version of the Assistant or Google app.
@donkii
Copy link

donkii commented Jul 2, 2019

I have a similar issue when I click run on App Actions Test Tool, I get Sorry, I couldn't find that from the assistant

@NoluckJustSkills
Copy link

@nimish17 same here.

@marcelpinto
Copy link
Contributor

Could you share which action are you trying and your actions xml file?

@nimish17
Copy link
Author

nimish17 commented Jul 4, 2019 

<!-- This file describes the supported actions by this app -->

<action intentName="actions.intent.START_EXERCISE">

    <!-- Each action requires at least one fulfillment that defines how the app will handle this action -->
    <!-- Define the urlTemplate in the format you define your deeplinks in AndroidManifest.xml -->

    <fulfillment urlTemplate="https://fit-actions.firebaseapp.com/start{?exerciseType}">

        <!-- Define how the actions parameters (intentParameter) is mapped in the urlTemplate above -->

        <parameter-mapping
            intentParameter="exercise.name"
            urlParameter="exerciseType" />

    </fulfillment>

    <!-- We can define our custom inline inventory, mapping a parameter to an entity set reference -->

    <parameter name="exercise.name">
        <entity-set-reference entitySetId="ExerciseEntitySet" />
    </parameter>

</action>

<action intentName="actions.intent.STOP_EXERCISE">
    <fulfillment urlTemplate="https://fit-actions.firebaseapp.com/stop" />
</action>

<action intentName="actions.intent.GET_EXERCISE_OBSERVATION">

    <!-- You can define the fulfillment mode, it can be SLICE or DEEPLINK -->
    <!-- When slice is used, make sure you are supporting slices in your app -->
    <!-- Also, not that the urlTemplate must be of the style content://{slice_provider_authority}/... -->

    <fulfillment
        fulfillmentMode="actions.fulfillment.SLICE"
        urlTemplate="content://com.devrel.android.fitactions.FitSliceProvider/stats{?exerciseType}">

        <!-- If a parameter is set as required, the action will only be fulfilled if the parameter is found -->
        <!-- That's why a fallback urlTemplate needs to be provided for such case. -->

        <parameter-mapping
            entityMatchRequired="true"
            intentParameter="exerciseObservation.aboutExercise.name"
            required="true"
            urlParameter="exerciseType" />

        <!-- Note, that for the parameter above we are setting entityMatchRequired="true" -->
        <!-- This tells the Assistant to only use the entity set values to map this parameter -->
        <!-- Meaning that even if the assistant know how to identify the exercise (i.e "Climbing") -->
        <!-- if it's not defined in our entity set, the parameter won't be use. -->

    </fulfillment>

    <!-- In case the exercise name is not found we fallback to the stats deep-link inside the app -->

    <fulfillment
        fulfillmentMode="actions.fulfillment.DEEPLINK"
        urlTemplate="https://fit-actions.firebaseapp.com/stats" />

    <!-- Same as the first action, we map the parameter name with out supported entity set. -->

    <parameter name="exerciseObservation.aboutExercise.name">
        <entity-set-reference entitySetId="ExerciseEntitySet" />
    </parameter>

</action>

<!-- Defines an entity set with our supported entities -->

<entity-set entitySetId="ExerciseEntitySet">

    <!-- For each entity you can specify the name, alternate names and the identifier -->
    <!-- The identifier is the value that will be added to the action uri. -->
    <!-- For our sample we map the supported entities with the class FitActivity.Type  -->

    <entity
        name="@string/activity_running"
        alternateName="@array/runningSynonyms"
        identifier="RUNNING" />
    <entity
        name="@string/activity_walking"
        alternateName="@array/walkingSynonyms"
        identifier="WALKING" />
    <entity
        name="@string/activity_cycling"
        alternateName="@array/cyclingSynonyms"
        identifier="CYCLING" />
</entity-set>

@AnilKolla
Copy link

I too have faced same issue in nexus 5 mobile but it is working in oneplus mobile. anyways issue in some mobiles.

@marcelpinto
Copy link
Contributor

Sorry for the delayed answer. Did you manage to make it run?

If not make sure:

  • All the above you mentioned is correct and you can run the aciton via plugin
  • When uploading the preview check the name you are using to upload, this is the one that you have to say
  • Try this sentence "Start running in Fit Actions"
  • Try modifying the acitons.xml and remove the in-line inventory (aka entity-set)

@harshfast
Copy link

@skimarxall
Same issue I am also facing if anyone has fixed please reply to the same thread.
Steps:

Uploaded the APK to play store internal test track in draft mode
Logged in with same account to (Device, PlayConsole and Android Studio)
Changed the packageId to the app which I have uploaded to playstore
Running App Actions Test Tool -> Preview uploaded successfully
Run -> Opens Assistant in device with message 'Sorry, Couldn't find that'
Let me know if I have missed any step. Thank you

@marcelpinto
Copy link
Contributor

@harshfast are you using the account that you use to publish the apk in the device?
Open google search, and make sure the account selected there is the same.

Also, can you tell us which Google app version are you using?

@harshfast
Copy link

@skimarxall
Thanks for quick response Now I am able to see the Android Slices and App Actions via google assistant. It started working I just checked all me configuration and everything looks good.

Steps I performed:
Checked all my logins (Test Device, Play Console and Android Studio)
Freshly installed the application with the package name which was uploaded to playstore.
Deleted actions preview and uploaded again from AATT.(App Actions Test Tool)
Run the command it started launching the app via google assistant and showing the slice from app inside google assistant window.

Just a quick query how can i trigger the voice commands for the same instead of adb command generated by AATT?

Thank you again for your help.

@adamkoch
Copy link

Just a quick query how can i trigger the voice commands for the same instead of adb command generated by AATT?

Go here:
https://developers.google.com/actions/reference/built-in-intents/

Find your the Assistant Intent you want to test, and look under "example queries" that you can use to try (you should be able to trigger via voice or by manually typing into Google Assistant using the keyboard).

Note that actions.intent.OPEN_APP_FEATURE doesn't currently have any samples right now and is broken outside of the Test Tool (https://issuetracker.google.com/issues/135714718).

@harshfast
Copy link

@adamkoch
I am using actions.intent.GET_TAXI_RESERVATION

Voice Command: "Call my driver with "

Google assistant shows generic search results rather showing my app slice. Also it has mentioned here "The App Actions feature is currently in developer preview. You can build and test App Actions (en-US locale only), but user triggering from the Assistant is not available yet. "

is it possible to trigger voice command or for now adb command will work?

@adamkoch
Copy link

adamkoch commented Sep 16, 2019 •
edited

As per the Built-in intents page, GET_TAXI_RESERVATION should trigger with a test voice query of "Call my driver with [app name]". If it's not working, make sure (once again) that you are logged in with the right Google account that has the preview published via the Test Tool and that you have the right APK installed.

And yes, you should switch to en-US locale on your device and in Assistant settings for now to test as it won't trigger in other locales.

If it's still not working, you can file an issue here.

You may also want to try the fitness actions via voice query first (using the sample app) because those definitely work fine.

@harshfast
Copy link

@adamkoch
I have few queries before proceeding with the actual implementation, your help will be appreciated

  • Can we create our own built-in-intent as I am not able to find the matching BII in current list. If yes, please let me know the procedure and the time it will take to be available for POC.
  • Will this newly created intent will be exclusive to my application?
  • Will google assistant display multiple slices if more than one app is able to fulfill the user query or multiple apps has the same intents?

Thank you.

@adamkoch
Copy link

Can we create our own built-in-intent as I am not able to find the matching BII in current list. If yes, please let me know the procedure and the time it will take to be available for POC.

No, you can't. You can request additional intents here:
https://issuetracker.google.com/issues/new?component=617864&template=1261453

Will google assistant display multiple slices if more than one app is able to fulfill the user query or multiple apps has the same intents?

Right now, the user specifies the app name in their query, so only one app will ever be able to respond to the query (Slices or otherwise).

@harshfast
Copy link

Thank you @adamkoch for your quick response. It gives us direction to build the app experience accordingly.

@harshfast
Copy link

harshfast commented Sep 18, 2019 •
edited

As per the Built-in intents page, GET_TAXI_RESERVATION should trigger with a test voice query of "Call my driver with [app name]". If it's not working, make sure (once again) that you are logged in with the right Google account that has the preview published via the Test Tool and that you have the right APK installed.

And yes, you should switch to en-US locale on your device and in Assistant settings for now to test as it won't trigger in other locales.

If it's still not working, you can file an issue here.

You may also want to try the fitness actions via voice query first (using the sample app) because those definitely work fine.

I tried triggering voice queries for FitActions sample as suggested in above answer but it didn't work. I have raised below issue with screenshots for your reference.
https://issuetracker.google.com/issues/141225979

@vpulpatil
Copy link

Is there any specific region that you need to choose in Google Assistant? Like only en-US is supported, or en-IN is also supported?

@adamkoch
Copy link

Right now, App Actions is targeting only en-US.

@harshfast
Copy link

Is there any specific region that you need to choose in Google Assistant? Like only en-US is supported, or en-IN is also supported?

@vpulpatil
The App Actions feature is currently in developer preview. You can build and test App Actions (en-US locale only), but user triggering from the Assistant is not available yet. Refer Here

@Grandolf49
Copy link

action is running in the device via App Actions Test Tool but not working via voice or written commands in the Assistant.

below all steps are perfect in my case
The plugin loaded the actions preview at least one time.
The account in the device is the same as the one in Android Studio.
The device has access to an internet connection.
You are using the latest version of the Assistant or Google app.

Even I'm having the same issue. The action is getting triggered from the App Actions Test Tool but when I say "Start Running in Fit Actions" it just sends me some responses from Google.
If you were able to make it work, kindly post the steps

@adamkoch
Copy link

Also make sure you have fully set up Google Assistant. Open Google Assistant and ensure you don't see anything indicating you need to finish setting it up. Usually this looks like a blue bar at the bottom of the screen but it may look different in some cases.

Screen Shot 2019-11-01 at 10 22 16 AM

@Grandolf49
Copy link

Also make sure you have fully set up Google Assistant. Open Google Assistant and ensure you don't see anything indicating you need to finish setting it up. Usually this looks like a blue bar at the bottom of the screen but it may look different in some cases.

Screen Shot 2019-11-01 at 10 22 16 AM

Yes I checked this as well. I think Google Assistant is not being able to match the query "Start running in Fit Actions" with the START_EXERCISE intent.

@adamkoch
Copy link

adamkoch commented Nov 1, 2019

Also make sure the invocation name (that is shown in the Studio Plugin) matches the query you are using ("Fit Actions" in your query). You can update the invocation name if needed by deleting the preview and then before creating it again there will be a field to set it.

@Grandolf49
Copy link

Also make sure the invocation name (that is shown in the Studio Plugin) matches the query you are using ("Fit Actions" in your query). You can update the invocation name if needed by deleting the preview and then before creating it again there will be a field to set it.

I tried deleting the preview and create it. Gave the name "Fit Actions". It works like a charm from the tool but it doesn't work for the same voice command.

Is there any other way to trigger the built-in intent actions.intent.START_EXERCISE?

@Grandolf49
Copy link

Grandolf49 commented Nov 1, 2019 •
edited

Also make sure the invocation name (that is shown in the Studio Plugin) matches the query you are using ("Fit Actions" in your query). You can update the invocation name if needed by deleting the preview and then before creating it again there will be a field to set it.

It's working now!

  1. I deleted the preview and named it "Fit Actions"
  2. Tested with App Actions Tool - working perfectly
  3. Gave this voice command - "Start running in Fit Actions"
    Working perfectly now!
    Thanks @adamkoch for the help. Very much appreciated!

@NNirosan02
Copy link

No luck for me as of yet! Followed all steps here.
I deleted the preview and named it "Fit Actions", logged into to everything with the relevant account. checked to see if I could execute Deeplink with adb (I can). just can't get it running with the actions tool nor with assistant invocation

@Grandolf49
Copy link

No luck for me as of yet! Followed all steps here.
I deleted the preview and named it "Fit Actions", logged into to everything with the relevant account. checked to see if I could execute Deeplink with adb (I can). just can't get it running with the actions tool nor with assistant invocation

Are you getting any errors while testing on App Actions Test Tool?
Because all the steps that you mentioned should enable you to test it successfully with at least App Actions Test Tool.

@NNirosan02
Copy link

NNirosan02 commented Nov 4, 2019 via email

@harshfast
Copy link

harshfast commented Nov 5, 2019 •
edited

@SoftwareMate02
Please try to verify below steps:

  • Change the device and Google Assistant language as en-US.
  • Check your login to all the three places: Android Studio, Playstore and Test Device.
  • Check the package name and version code of the application at playstore and test application.
  • Review your AndroidManifest and actions.xml.
  • Remove and install fresh application to device.
  • Delete the preview from AATT and create the fresh preview again.
  • Sometimes Assistant always triggers "Sorry couldn't find that". It worked for me after multiple tries.
  • If possible grab the stock android device(Pixel) and try with that.(This is optional it should work in every device)
  • Check the logcat without and filter and capture the logs if it shows any failure regarding actions fulfillment.

@Grandolf49 Still I haven't verified with voice command but it is working with AATT for both Actions and Slices. Also have you verified with voice command using inline inventory words?

@adamkoch
Copy link

adamkoch commented Nov 6, 2019

@Grandolf49 glad you got it working!

@SoftwareMate02 good advice from @harshfast, also make sure it all triggers fine using the Studio Test Tool plugin first, once that is working (but it's not triggering directly from Assistant) double check your Google accounts are matching up, that Assistant is fully setup (as per my comment above) and that you are using the right invocation name.

@mmaSunshine
Copy link

@skimarxall same here ,I follow all this steps,but no luck for me ,It's remind me sorry and can't found it.
I checked:
1.make sure my login in three places:Android Studio, Play console and Test Device.
2.make sure the package name is my own.
3.remove and install fresh application to device.
4.Delete the preview from AATT and create the fresh preview again.
5.test the sample via adb in terminal ,not AAPT,the code is
adb shell am start -a android.intent.action.VIEW -d 'https://fit-actions.firebaseapp.com/start',the sample can run
but it still can't run on AATT.

@dzuluaga
Copy link
Contributor

dzuluaga commented Nov 7, 2019

Are you using Android Emulator or a physical device? Physical devices work at the moment.

@harshfast
Copy link

Also make sure the invocation name (that is shown in the Studio Plugin) matches the query you are using ("Fit Actions" in your query). You can update the invocation name if needed by deleting the preview and then before creating it again there will be a field to set it.

It's working now!

  1. I deleted the preview and named it "Fit Actions"
  2. Tested with App Actions Tool - working perfectly
  3. Gave this voice command - "Start running in Fit Actions"
    Working perfectly now!
    Thanks @adamkoch for the help. Very much appreciated!

@adamkoch @Grandolf49 Thanks for your responses. It worked with voice command as well.
"Start running in Fit Actions". Still want to understand the logic behind the Invocation name and voice command as if we have our own Action defined with different app name. what invocation name should i provide for action let's say "actions.intent.CREATE_TAXI_RESERVATION" ?

@adamkoch
Copy link

@harshfast the invocation name is only configured/set for preview mode (via the studio test tool). in production, when you launch, the invocation name will simply be your app name. whatever invocation name you choose, it will be used for all the actions you have defined in your actions.xml. likewise in production, your app name will be used for all the actions you have defined.

@aab0y07
Copy link

aab0y07 commented Apr 19, 2020

Just use your physical device as an emulator, not virtual one!!!

@Stalker11
Copy link

Hello everyone!
I have similar problem.
I have cloned fit actions app example, changed applicationId and upload app bundle to developers console.
Снимок экрана 2020-06-04 в 15 22 03
I can debug app using App Actions Test Tool plugin, but when I try to start running Test App Action Google assistant write message "Sorry, I cold not find that"
GAExample
Account of developers console, Android Studio and device is the same.

What can be wrong?
Why it does not work?

@aab0y07
Copy link

aab0y07 commented Jun 4, 2020 via email

@aab0y07
Copy link

aab0y07 commented Jun 4, 2020 via email

@Stalker11
Copy link

No, I am using real devices for testing. Huawei and Samsung

@JinalAwaiya410
Copy link

JinalAwaiya410 commented Jul 8, 2020 •
edited

@mmaSunshine Have you found any solution?I have same issue

I follow all this steps,but no luck for me ,It's remind me sorry and can't found it.
I checked:
1.make sure my login in three places:Android Studio, Play console and Test Device.
2.make sure the package name is my own.
3.remove and install fresh application to device.
4.Delete the preview from AATT and create the fresh preview again.
5.test the sample via adb in terminal ,not AAPT,the code is
adb shell am start -a android.intent.action.VIEW -d 'https://fit-actions.firebaseapp.com/start ',the sample can run
but when run in Test tool its not working.

It gives error "Sorry i couln't find that"
I am using real devices for testing. Samsung.
Please help me for solve this issue.

@dzuluaga
Copy link
Contributor

dzuluaga commented Jul 8, 2020

I've compiled a living and breathing list here list. Try these options and let me know how it goes.

@nicolasjafelle
Copy link

nicolasjafelle commented Sep 18, 2020 •
edited

@dzuluaga @adamkoch Hey! I know that maybe this is not the plase to ask but probably it is also a good place to understand what is going on with my integration.
So I managed to integrate my app with Google assistant and the AATT works excellent with my local device and my local environment.
I have an internal test channel in Google Play for QA things so our testers can do their work by simple downloading our test builds in Google Play. I already uploaded the build with the actions.xml and checked the Actions on Google on the Play Store.

The problem is that our QA testers cannot trigger by any means the "Hey Google, show info about this on xxx app". That means that we need to wait till release the build for production (going live massively) to allow everyone else use the Google Assistant capabilities in our app? Is there a way that other people with their phones and their personal account (gmail account) be able to test it before release?

There is also a form we already submitted but still do not receive any feedback from Google Team. Thanks in advance!

@dzuluaga
Copy link
Contributor

Google Assistant should work once the preview gets created for that particular account. Could you verify that your tester is using the same account? If not, you need to create a preview using AATT with the tester account. What's your package id? Please send us your package id through the support form. Thanks!

@nicolasjafelle
Copy link

nicolasjafelle commented Sep 18, 2020 •
edited

hey! @dzuluaga how fast to reply, thank you so much. I can confirm our testers are not using the same account. And they are not using the phones connected with ADB. So this is something maybe it is not supported the way we thought..
We have a Google play account and with this account we have other google accounts linked to allow testers and developers login with their accounts and check some insights of the build. For example:
Root email: aaa@gmail.com
Linked email: xxx@gmail.com - yyy@gmail.com

Do you think that it will only work if the phone is connected with aaa@gmail.com but not with xxx@gmail.com and yyy@gmail.com ??

BTW already submitted the form, couple of minutes ago. Thanks again.

@dzuluaga
Copy link
Contributor

Hi, nicolasjafelle@ You can grant access to multiple testers. https://developers.google.com/assistant/app/test-tool#additional-testers. However, previews are linked to each Gmail account. So, make sure you create them from Android Studio.

Someone from the Support team will be in touch with you. Let know if you don't get a response in about a week.

@nicolasjafelle
Copy link

@dzuluaga aahh! good to know!! thank you so much for all your support. Last thing: When we send the build live to production we also need to link accounts? Or it just works because is a release build ? Thanks in advance!

@dzuluaga
Copy link
Contributor

It just works as it's in prod. Do you still want to release this build in prod and then test? Or want to try in QA first using the above recommendation. Please send a request to the same support form if you're going to wait.

@Rathish100
Copy link

@dzuluaga I could able to run the command successfully via "App Action Tool Plugin", But when I try to use the same command via voice from the google assistant screen, it's just showing web results.

  1. Logged in using the same credentials (Android Studio, Google Console, Real Device - Samsung s10+) and Device language set to EN-US
  2. Selected App Action tool plugin from the tools menu
  3. Gave invocation name (todo app) and locale to En-US
  4. Changed default feature "History" with my command (In my case "Open active tasks in todo app")
  5. Tapped on Update Preview.
  6. Once the preview updated, Run the command via AATT tool, it's working perfectly.
  7. Opened Google Assistant screen in a Real device and gave voice command (Open active tasks in todo app), but its showing web results in Assistant screen instead of showing my App.

Please let me know if I'm missing anything here.

@rajpalrohit7
Copy link

I could able to run the command successfully via "App Action Tool Plugin", But when I try to use the same command via voice from the google assistant screen, it's just showing web results.

{
"@type": "Call",
"callFormat": "Audio",
"@context": "http://schema.googleapis.com",
"participant": {
"@type": "Person",
"name": "John Doe",
"telephone": "012-345-6789"
}
}

Above json is updated with call fetaure which is working from App Action Tool Plugin not from voice cmd: "call jhon doe on myapp"

@dzuluaga
Copy link
Contributor

Please try to narrow down your issue by running this checklist.

@VSKarthi
Copy link

VSKarthi commented Oct 8, 2021

action is running in the device via App Actions Test Tool but not working via voice or written commands in the Assistant.

below all steps are perfect in my case The plugin loaded the actions preview at least one time. The account in the device is the same as the one in Android Studio. The device has access to an internet connection. You are using the latest version of the Assistant or Google app.

im also facing the same issue ,did you find any solution?

@mykola-tarchanyn
Copy link

hey @dzuluaga , I have the app actions working with START_EXERCISE intent, and Google Assistant picks up the App name when I say - "Hey Google, Start exercise on ExampleApp" and works just fine, but it doesn't pick up the App name for CREATE_CALL intent, my query is - "Hey Google, Call *123 on ExampleApp", it either picks up default Phone app or Duo. Please advise.

@dvghule121
Copy link

hey @nimish17 @mykola-tarchanyn @dzuluaga @karthi1307 @adamkoch @Rathish100 @marcelpinto @nicolasjafelle

has anyone tried custom intent with app actions
https://developer.android.com/guide/app-actions/custom-intents

my problem is i am not able to frame query it works fine when called from App actions assistant
my problem is exactly similer to
https://stackoverflow.com/questions/73454186/unable-to-open-custom-intent-from-google-assistant-in-app-actions

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests