Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow consumer control of crawled paths #141

Open
jahredhope opened this issue Nov 20, 2018 · 0 comments
Open

Allow consumer control of crawled paths #141

jahredhope opened this issue Nov 20, 2018 · 0 comments

Comments

@jahredhope
Copy link
Collaborator

jahredhope commented Nov 20, 2018

Background

Currently there is a feature to crawl through rendered html for additional links.
This is passed in as a boolean.
An issue #127 was opened asking for the ability to disallow some domains to be rendered.

Change

We could change the crawl behaviour from a boolean to an optional function the consumer can pass in to decide whether a link should be rendered.

Option 1: On all HTML
Crawl function could get called once a render is complete.
It would be responsible for looking for all links on the page and returning an array of new pages to render.

Optionally we could add a getHrefsFromHtml convenience function to save each consumer writing this parser.

import StaticSiteGeneratorPlugin, { getHrefsFromHtml } from 'static-site-generator-webpack-plugin';

new StaticSiteGeneratorPlugin({
  crawl = ({html}) => getHrefsFromHtml(html)
    .filter(href => !href.includes('bad.com')
});

Option 2: On each link
Crawl function could get called after we've parsed the rendered HTML for links.

import StaticSiteGeneratorPlugin from 'static-site-generator-webpack-plugin';

new StaticSiteGeneratorPlugin({
  crawl = ({href, html, index}) => !href.includes('bad.com')
});

Feedback

Feedback is welcome. Please comment below with your thoughts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant