Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sitemap improvements: gz support + application/xml + extraHops fix #511

Merged
merged 3 commits into from Mar 26, 2024
Merged

sitemap improvements: gz support + application/xml + extraHops fix #511

merged 3 commits into from Mar 26, 2024

Conversation

ikreymer
Copy link
Member

@ikreymer ikreymer commented Mar 26, 2024 •
edited

sitemap fixes, follow up to #496

  • support parsing sitemap urls that end in .gz with gzip decompression
  • support both application/xml and text/xml as valid sitemap content-types (add test for both)
  • ignore extraHops for sitemap found URLs by setting to past extraHops limit (otherwise, all sitemap URLs would be treated as links from seed page)

@ikreymer
sitemap fix, follow up to #496
f174658 
- support parsing sitemap urls that end in .gz with gzip decompression
- ignore extraHops for sitemap found URLs by setting to past extraHops limit
(otherwise, all sitemap URLs would be treated as links from seed page)
@ikreymer ikreymer changed the base branch from main to 1.0.3-release March 26, 2024 00:48
@ikreymer ikreymer requested a review from tw4l March 26, 2024 02:30
tw4l
tw4l reviewed Mar 26, 2024
View reviewed changes
src/crawler.ts Show resolved Hide resolved
@tw4l tw4l mentioned this pull request Mar 26, 2024
Merged
@tw4l
Copy link
Contributor

tw4l commented Mar 26, 2024

We may also want to add a test for a gzipped sitemap, if we can find a stable one to test against?

@ikreymer
Copy link
Member Author

We may also want to add a test for a gzipped sitemap, if we can find a stable one to test against?

Only one I found is for www.pinterest.com - that's where it occurred, could add as a test, or maybe wait till we find something else..

@tw4l
Copy link
Contributor

tw4l commented Mar 26, 2024

We may also want to add a test for a gzipped sitemap, if we can find a stable one to test against?

Only one I found is for www.pinterest.com - that's where it occurred, could add as a test, or maybe wait till we find something else..

Hm that's fair, maybe we wait, or could host our own on a domain we own at some point for tests? If it's been testing manually for Pinterest (I will do that now too), then can probably merge without adding that test now to keep tests from getting too brittle.

Doing a bit more testing now and then will approve if all looks good.

tw4l
tw4l approved these changes Mar 26, 2024
View reviewed changes
Copy link
Contributor

@tw4l tw4l left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and working well, other than failing exclusions test in 1.0.3-release

@ikreymer
also support application/xml as valid xml content-type
c0bfc76 
add test with application/xml sitemap
@ikreymer ikreymer changed the title .gz sitemap support + extraHops fix sitemap improvements: gz support + application/xml + extraHops fix Mar 26, 2024
@ikreymer ikreymer merged commit b707d00 into 1.0.3-release Mar 26, 2024
2 of 4 checks passed
@ikreymer ikreymer deleted the gz-sitemaps branch March 26, 2024 20:55
@ikreymer ikreymer mentioned this pull request Mar 26, 2024
Merged
ikreymer added a commit that referenced this pull request Mar 26, 2024
@ikreymer
Fixes from 1.0.3 release -> main (#517)
01c4139 
sitemap improvements: gz support + application/xml + extraHops fix #511
- follow up to
#496
- support parsing sitemap urls that end in .gz with gzip decompression
- support both `application/xml` and `text/xml` as valid sitemap
content-types (add test for both)
- ignore extraHops for sitemap found URLs by setting to past extraHops
limit (otherwise, all sitemap URLs would be treated as links from seed
page)

fixes redirected seed (from #476) being counted against page limit: #509
- subtract extraSeeds when computing limit
- don't include redirect seeds in seen list when serializing
- tests: adjust saved-state-test to also check total pages when crawl is
done

fixes #508
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tw4l tw4l tw4l approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@ikreymer @tw4l