Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: framework detection test race condition #6564

Merged
merged 9 commits into from May 9, 2024

Conversation

mrstork
Copy link
Contributor

@mrstork mrstork commented May 8, 2024

Summary

Fixes CT-1094

This solves the race condition and flakiness of test frameworks/framework-detection > should log the command if using static server and command is configured by removing it (justifiably, see explanation below). This also presents as Integration Tests / Integration (macOS-latest, 22, 4/4) failing in the test suite.

Explanation:

This test runs ntl dev --dir public --command npm run start in the test directory. When I run this command manually, I get the following error page: This site can’t be reached. localhost refused to connect.. Interestingly, I can sometimes get the page to show up if I close and re-open the tab, however I believe this is due to the browser caching a working copy.

The root of the problem is that npm run start relies on a package.json file, and a scripts declaration, which this test is not setting up. From there, npm throws an error (Command failed with exit code 1: npm. Shutting down Netlify Dev server), shutting down the static server before it can be queried (hence socket hang up), and thus failing the test assertion around site content.

When the test works, I believe it's either due to the browser caching a functioning site (perhaps of another site running in parallel, or one that ran just before), or there is actually a tiny window of time in which the static server is up, before npm run start fails.

Fix:

I've opted to remove the test case as it is testing a scenario that doesn't actually work in practice.

Alternate Solutions Considered:

I am happy to switch to any of these options (or something else entirely) if that's preferred.

  1. Adjust the test to not fetch, and simply look at console output
    • This may also be flaky at times if npm fails before the test finishes
  2. Adjust the command in the test npm run start -> echo hello, and simply verify the output of the cli matches what's expected.
    • Requires us to remove fetch from the test as well
  3. Generate a package.json file with a valid start script
    • Although, this is captured in some of the framework detection tests in the same file
Bonus Changes:

Extra changes in this pull request:

  • Use non-depracated method signature for withSiteBuilder (and update associated snapshots)
  • Update buildAsync -> build in the test file to remove deprecation warnings
  • Remove final reference to cleanupAsync and remove the method
  • Add type definitions to acquirePort,
  • Avoid mixing await and .then() in a single statement (code style preference)

For us to review and ship your PR efficiently, please perform the following steps:

  • Open a bug/issue before writing your code 🧑‍💻. This ensures we can discuss the changes and get feedback from everyone that should be involved. If you`re fixing a typo or something that`s on fire 🔥 (e.g. incident related), you can skip this step.
  • Read the contribution guidelines 📖. This ensures your code follows our style guide and
    passes our tests.
  • Update or add tests (if any source code was changed or added) 🧪
  • Update or add documentation (if features were changed or added) 📝
  • Make sure the status checks below are successful ✅

@mrstork mrstork requested a review from a team as a code owner May 8, 2024 02:29
@mrstork mrstork force-pushed the fix/framework-detection-test-race-condition branch from 1bc1f52 to 3b155b3 Compare May 8, 2024 02:31
Copy link

github-actions bot commented May 8, 2024

📊 Benchmark results

Comparing with f2b0a5f

  • Dependency count: 1,347 (no change)
  • Package size: 312 MB ⬆️ 0.00% increase vs. f2b0a5f
  • Number of ts-expect-error directives: 996 ⬇️ 0.10% decrease vs. f2b0a5f

@mrstork mrstork merged commit a79430f into main May 9, 2024
48 checks passed
@mrstork mrstork deleted the fix/framework-detection-test-race-condition branch May 9, 2024 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants