QA for 4/17/2024 production deploy

[ccostino]

Log In

• notify-staging.app.cloud.gov resolves
• Clicking sign in takes you to login page
• Signing in with Login.gov works as expected
• After logging in, you are brought to the “choose service” screen
• Choosing a service brings you to the dashboard for that service

Create a template and send

• Clicking “Send messages” brings you to templates page
• When “Create a template” button is clicked, the option to create new or copy existing template appears
• Uploading a csv is quick and automatically directs you to the preview/send page
• Formatting looks clean and in place
• Message dashboard populates and shows “pending” until call is made to AWS for delivery receipt

Send a message from copied template

• When “Create a template” button is clicked, the option to create new or copy existing template appears
• Clicking “copy an existing template” allows you to choose from pre-existing template
• After editing, the new template saves and you prepare to send

Send a message from existing template

• Clicking on existing template brings you to preview screen with prepare to send button
• All else above persists as true (formatting, send, dashboard, etc)

Download the csv delivery report

• Clicking on the csv download directly after a message send downloads a report that contains the data we expect
• The batch report appears on the main dashboard batch table with accurate information
• Clicking on the csv download link from the main dashboard downloads a report that contains the data we expect

Invite a team member

• Clicking on “Team” in settings brings you to team member screen
• Clicking “Invite a team member” brings you to email and permissions screen
• Clicking invite team member sends an invite and shows confirmation
• Invite email provides live link that works to register new team member

Delete a team member

• From team member screen, clicking “Change Details” on a specific profile brings you to that person’s profile
• Clicking on “Remove this team member” prompts a confirmation
• Clicking the confirmation removes the team member

Sign Out

• Hitting “Sign Out” takes you back to the homepage and logs you out

Specific Issues to review

We have a handful of issues that are ready to be reviewed or could use verification whether or not they're fixed or need more follow up:

• Update the default free message limit when service goes live notifications-api#645
• Bug: SMS message fragment counts in platform do not match calculated value #1125
• Add end-to-end test: Download the CSV delivery report #1338
• Add a test or additional parameter that covers mixed English-language texts #952
• Update service invitation email template to include Login.gov information #1270
• User Story: Update 'change email' flow in Settings to include Login.gov #1269
• User Story: Remove email and password in user settings after login.gov integration #1427
• Investigate API error for the Demo site Federal Test Service notifications-api#798
• variable data is "hidden" in some views, but shown in others notifications-api#903
• convert nav templates to components #1350

TODO

We're waiting on a couple of things to still be completed and merged to staging:

• Changes to the user settings page to remove sign-in method option
• Changes to the user profile page to remove the option to change password and email (but still show the email address)
• Changes to the "Set up your profile" template to not mention sending a text code
• Adding the Login.gov button to the "You need to sign in again" template

[ccostino]

I think we've already captured this, but noting here again that when you go to create a new template and copy from an existing template, you don't see a list of any of the existing templates and can only search by name, which means you have to remember which templates exist.

[heyitsmebev]

@ccostino Everything on the QA list worked for me except the items listed below.

Send a message from copied template

• When “Create a template” button is clicked, the option to create new or copy existing template appears
  My button says "New template " and not Create a tempalte

• Clicking “copy an existing template” allows you to choose from pre-existing template
  I have to search for the template first
• After editing, the new template saves and you prepare to send

Send a message from existing template

• Clicking on existing template brings you to preview screen with prepare to send button
  After click on existing template, it brings me to an edit/save screen. There's no "prepare to send button"

[heyitsmebev]

If I'm signed out and I somehow land on "You need to sign in again" page...There's no sign in inputs.

[ccostino]

Awesome, thanks for capturing these @heyitsmebev! We'll follow up to see if we're dealing with known issues or need to create new ones, and if any of these are actual show-stoppers.

As for the sign-in again form there, that's a known issue that @em-herrick has already flagged that still needs to be taken care of - it's on our list of TODOs above that @jonathanbobel is working on! 🙂

[ccostino]

Something else that we might need to account for: when I invited myself to another organization (the TTS organization), I got the email as expected and then clicked on the link.

From there, it prompted me to sign in with Login.gov, which is what I believe we want now, except in this case I was already signed in to the site...

I still signed back in, and was then taken to the set up my profile page, where it said a security code would be texted to me (that never happened):

I filled things in anyway, and then took me to the usage page for the org:

And finally I clicked on Team Members, where I see me listed as an invited user with the option to still cancel, even though I had accepted:

It's entirely possible I broke the flow here by being a platform admin and signed in, inviting myself to an org with the exact same email and going from there.

EXTRA NOTE

This is with an organization invite flow, which is different than a service invite flow!

[em-herrick]

A small fix here is that on the Set up your profile page under the Mobile number the copy here is out-of date, we no longer send security codes.

We should replace with: We need your number so you can send yourself test texts.

[ccostino]

Ah yes, this is what I was seeing yesterday too in the set up your profile form, thanks @em-herrick!

Update: now it's own issue

[A-Shumway42]

@ccostino The allowance updates to 250,000 when switching the service to live. The values aren't updating now? I sent a few text to verify.

I tested locally in dev and the values update. Possibly a staging issue?

[ccostino]

🤔 I'm not sure, @A-Shumway42, so once the service was switched to live the updates to the counts stopped working?

Is this an existing service that was creating a while back, or a new one you just set up for testing?

[jskinne3]

While investigating #1427 I encountered two broken links:

Both lead to a "Page not found" at these two URLs:

• https://notify-staging.app.cloud.gov/services//service-settings
• https://notify-staging.app.cloud.gov/services//users

The double slash (//) in the path looks like something is missing.

As a result of this I can't say for sure if user story 1427 is complete or not

[ccostino]

Added an issue for the "you need to sign back in" template - the Login.gov button is missing.

[A-Shumway42]

@ccostino this was a new service in staging. I logged back into staging and it appears it is working now. I'm not sure what the delay was on the count.

[ccostino]

Oh strange, @A-Shumway42! Two things come to mind that we could look into further to see:

• Was it just because of cached data with your user session - doe they update properly now if you send more messages?
• Is it waiting for the delivery status to update to be successful (and/or failed)?

[jskinne3]

When investigating issue #1270 I found that there is some text in the google doc draft of email language which does not appear in the email:

Notify.gov uses Login.gov to allow you to sign in safely. Login.gov is a secure sign in service to help you access participating government agencies’ digital touchpoints.

Unsure if this was actually supposed to be included, or if it was old and discarded text

[A-Shumway42]

@ccostino Hard to verify now. I'm not receiving messages to my phone and I've sent a few more in staging. The dashboard still has a count of 2 in the count at the bottom and the top. I also can't access logs in staging because my cf cli can't login...oof.

[ccostino]

@A-Shumway42 okay - I noticed some delays with messages going out yesterday myself - you might be able to see some activity (or not!) in AWS in the staging account, but I think we're okay for now and we can follow up on this more if need be.

@heyitsmebev, thanks for all of your reports there! It looks like the discrepancies with the templates are more so with the QA checklist and not the site itself, so we'll have to update that issue template. 🙂 As for the sign in again page, that's tracked here and nearly wrapped up!

@jskinne3, thanks for reporting these! Sure enough those are bugs caused by the current information architecture of all things. Those two links are tied to a service and only work if you've clicked into a service first - they are settings page for a service, not global/universal for just your user account. So thank you for flagging, @em-herrick and I confirmed this just a bit ago! As for the template, thanks for flagging that too; it looks like it didn't get fully updated for some reason, and highlights that the template updating process is still a bit wonky.

@em-herrick and I will write up separate issues to track the things that we do need to follow up on from all of this. Thank you all!

[ccostino]

Our production release has gone out and new issues have been created. Thanks, everyone!