Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better tracking of failed requests + logging context exclude #485

Merged
merged 5 commits into from Mar 7, 2024
Merged

Better tracking of failed requests + logging context exclude #485

merged 5 commits into from Mar 7, 2024

Commits on Mar 7, 2024

  1. logging: exclude log contexts 

    - add --logExcludeContext for log contexts that should be excluded
- enable 'recorderNetwork' logging
- create default log context exclude list (containing: screencast, recorderNetwork, jsErrors), customizable via --logExcludeContext

recorder: track failed requests
- cleanup cdp handler methods
- intercept requestWillBeSent to track requests that started, but may not complete
- fix shouldSkip() still working if no url is provided (eg. check only headers)
- set status to 0 for async fetch failures
- remove responseServedFromCache interception, as response data generally not available then, and responseReceive is still called
- pageinfo: include page requests that failed with status code 0
- ensure page is closed on failure
- ensure pageinfo still written even if nothing else is crawled for a page

tests: add pageinfo test for crawling invalid URL, which should still result in pageinfo record with status code 0

bump to 1.0.0-beta.7
    @ikreymer
    ikreymer committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    ce31d1d View commit details
    Browse the repository at this point in the history

  2. remove commented out

    @ikreymer
    ikreymer committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    e85c78a View commit details
    Browse the repository at this point in the history

  3. ensure added to pageinfo even if dupe via async fetcher

    @ikreymer
    ikreymer committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    811dafb View commit details
    Browse the repository at this point in the history

  4. pageinfo: also include 'error' field when status is 0 

    cache status: track if a resource is loaded from browser cache, include in debug logging
(though not yet in pageinfo, to revisit later)
    @ikreymer
    ikreymer committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    2e5d766 View commit details
    Browse the repository at this point in the history

  5. minor tweaks, ignore direct fetch reqresp

    @ikreymer
    ikreymer committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    30b6b63 View commit details
    Browse the repository at this point in the history