Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Record Playwright tests with Replay #348

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Record Playwright tests with Replay #348

wants to merge 14 commits into from
+837 −24

Conversation

bvaughn
Copy link
Owner

@bvaughn bvaughn commented May 1, 2024

No description provided.

@vercel Vercel
Copy link

vercel bot commented May 1, 2024 •
edited

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
react-resizable-panels ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 19, 2024 1:50pm

Andarist
Andarist reviewed May 2, 2024
View reviewed changes
package.json Outdated
@@ -23,6 +23,7 @@
"devDependencies": {
"@babel/preset-typescript": "^7.22.5",
"@playwright/test": "^1.37.0",
"@replayio/playwright": "3.0.0-alpha.3",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this setup work for you with this version now?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. The latest comment I left in Linear mentioned that the steps are all missing. I'll add you to the workspace so you can look at them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's see what happens with the newest version ( #349 ). I'm simultaneously testing this release in the overboard's repo

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the latest recording: https://app.replay.io/recording/68393c17-bc53-4c80-a972-f29c9e0fa579

No steps.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm able to repro this locally using ur tests here... diving in! 🤿

@bvaughn
Copy link
Owner Author

bvaughn commented May 3, 2024

@Andarist The latest v6 alpha looks to have both steps and metadata. Great!

@bvaughn
Copy link
Owner Author

bvaughn commented May 3, 2024

Removing the test.only so that all of my tests run shows a new failure though:

RecordingUploadError: Failed to set recording metadata Error: Protocol client is not initialized

@Andarist
Copy link
Contributor

Andarist commented May 3, 2024

Interesting, gonna investigate soon!

Andarist
Andarist reviewed May 6, 2024
View reviewed changes
package.json Outdated
@@ -23,6 +23,7 @@
"devDependencies": {
"@babel/preset-typescript": "^7.22.5",
"@playwright/test": "^1.37.0",
"@replayio/playwright": "3.0.0-alpha.6",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could now be bumped to .7

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha. Just bumped the version.

Noticed that I can't run e2e tests locally though:

$ p test:e2e        

> react-resizable-panels-website@0.0.1 test:e2e /Users/bvaughn/Documents/git/oss/react-resizable-panels/packages/react-resizable-panels-website
> playwright test

[replay.io]: Failed to send test metrics
[replay.io]: Error: request to https://webhooks.replay.io/api/metrics failed, reason: socket hang up
[replay.io]: 🕑 Completing some outstanding work ...
[replay.io]: 
[replay.io]: ❌ We encountered some unexpected errors processing your recordings and was unable to upload them.
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io
[replay.io]:    - Failed to upload recording: Cannot connect to server wss://dispatch.replay.io

Maybe this is because I didn't pass the api token env var? But the error message isn't great if that's the case, and this would mean that no contributors could run these tests either.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the terminal was totally silent for several seconds before printing this. I'm not sure if that meant tests weren't running, or if it was just swallowing the stdout output. (That output is important to debug.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you have this error, CI reports "RecordingUploadError: Failed to set recording metadata Error: Protocol client is not initialized" and I have "RecordingUploadError: Failed to set recording metadata Protocol error 1: Internal error" locally now 😂

Well, I'm investigating all of this now ;p

@Andarist
Update @replayio/playwright (#353)
8f63b71 
This one makes sure that the `@replayio/reply` (transitive dependency)
gets updated too (u can see the change in the lockfile).

With this one, I can record this repo and I get recordings. Some fail at
times when the metadata's size gets exceeded. We know about the issue
and we are in the process of figuring out the solution.
@Andarist
Copy link
Contributor

Andarist commented May 9, 2024

We have a partial success here! It still suffers from the metadata size problem - but most tests don't exceed that size and recordings are nicely created. Could you also confirm that it works for you locally?

@bvaughn
Copy link
Owner Author

bvaughn commented May 9, 2024

Could you also confirm that it works for you locally?

Yes, tests do run (although tests/StackingOrder.spec seems perhaps a bit flakier? could be a coincidence or could be a difference caused by the older Chromium version).

I think our runner's output is less helpful as you go. For instance, running these tests on main gives feedback as you go:
Screenshot 2024-05-09 at 10 26 40 AM

Running them on this branch is more quiet:
Screenshot 2024-05-09 at 10 27 16 AM

But that's a nit.

@Andarist Andarist mentioned this pull request May 10, 2024
Merged
@Andarist
Use a nicer list reporter locally (#356)
e786040 
This addresses the complaint from
#348 (comment)
. I didn't realize that the default is `'list'` locally, I added
`'line'` reporter since I've seen that being used in a couple of other
projects. I totally agree that `'list'` is quite nice and I prefer it to
`'line'` - especially if we consider smaller test suites.

⚠️ `@replayio/playwright` still spits out a bunch of junk into the
stdout. That makes the current output very cluttered - even with this
improvement. This is going to be addressed with PRO-315
@replay-io replay.io
Copy link

replay-io bot commented May 19, 2024 •
edited

Status Complete ↗︎
Commit eaa041f
Results
❌ 1 Failed
  • should update cursor when dragged past the min/max size (like VS Code)
    • ✅ 23 Passed
  • earlier panels should be springy when expanding then collapsing the last panel
  • Enter key (collapsible)
  • Enter key (not collapsible)
  • Home and End keys
  • horizontal lists
  • implements optional F6 key functionality
  • later panels should be springy when expanding then collapsing the first panel
  • panels should remember a max spring point per drag
  • scenario: other maxSize(s)
  • scenario: other minSize(s)
  • scenario: own maxSize
  • scenario: own minSize
  • should call onResize when panels are resized
  • should not update cursor or start dragging if a resize handle is underneath another element
  • should not update cursor style for resizes caused by keyboard events
  • should remember the most recent expanded size for collapsed panels
  • should resize and maintain layouts independently
  • should restore previous layout if autoSaveId prop has been provided
  • should set 'data-resize-handle-active' attribute when active
  • should store layouts separately per panel combination
  • should support custom collapsedSize values
  • should update cursor when dragging intersecting panels (like tmux)
  • vertical lists

    • Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Reviewers

    @Andarist Andarist Andarist left review comments

    Assignees
    No one assigned
    Labels
    None yet
    Projects
    None yet
    Milestone
    No milestone
    Development

    Successfully merging this pull request may close these issues.

    None yet
    2 participants
    @bvaughn @Andarist