Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to link fetch requests with their results in bundled calls #417

Open
Vacilando opened this issue Jan 5, 2021 · 0 comments
Open

How to link fetch requests with their results in bundled calls #417

Vacilando opened this issue Jan 5, 2021 · 0 comments

Comments

@Vacilando
Copy link

Vacilando commented Jan 5, 2021

Consider 5 web pages that need to be enhanced with 5 images from these Wikipedia articles:

wikiArticles = [
    'Richard P. Feynman',
    'Thor Heyerdahl',
    'Interstellar travel',
    'Religion',
    'Cosmology',
]

Then fetch all of these asynchronously:

var [page] = await Promise.all([
  //wtf.fetch(wikiArticle, wikiLang, { 'Api-User-Agent': wikiUserAgentMail, }).then((doc) => doc.summary()),
  wtf
    .fetch(wikiArticles, wikiLang, { 'Api-User-Agent': wikiUserAgentMail })
    .then(docList => {
      return docList.map((doc, i) => {
        return {
          requestArticle: wikiArticles[i],
          title: doc.title(),
          firstImage: doc.image(0).url(), 
        }
      })
    }),
])

As you see we use requestArticle to track which request fetched which image.

The problem is that the returned list of fetches is in a different order, e.g.

  • Interstellar travel
  • Religion
  • Richard Feynman
  • Thor Heyerdahl
  • Cosmology

Matching the original article name with doc.title() would not be a solution because of redirects (e.g. Richard P. Feynman correctly fetches the redirected page Richard Feynman).

Any ideas how to solve this in a safe way?

NB Due to this, and #414 and #416 we are more and more inclined to abandon the bundling and do the fetches sequentially. But that of course increases the risk of being blocked by Wikipedia's bots.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant