Skip to content

Commit

Permalink
skip empty url frames: likely frame is not initialized yet and runnin…
Browse files Browse the repository at this point in the history
…g frame.evaluate() never returns

skip such frames altogether for now
  • Loading branch information
ikreymer committed May 7, 2024
1 parent ddc3e10 commit 66395da
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions src/crawler.ts
Expand Up @@ -1055,6 +1055,10 @@ self.__bx_behaviors.selectMainBehavior();

const frameUrl = frame.url();

if (!frameUrl) {
return null;
}

// this is all designed to detect and skip PDFs, and other frames that are actually EMBEDs
// if there's no tag or an iframe tag, then assume its a regular frame
const tagName = await frame.evaluate(
Expand Down

0 comments on commit 66395da

Please sign in to comment.