Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wrong metadata charset tags #301

Open
yuval-herman opened this issue Feb 28, 2023 · 0 comments
Open

wrong metadata charset tags #301

yuval-herman opened this issue Feb 28, 2023 · 0 comments

Comments

@yuval-herman
Copy link

Firstly thank you for working on this library! It's a big help!

I was using your library to scrap some old sites with node fetch and came across a strange issue.
While scraping this site specifically, I got this error out of iconv:

Error: Encoding not recognized: 'visual' (searched as: 'visual')

This was caused by this tag in one the subframes(which I also scrap) in the page:

<meta http-equiv="Content-Type" content="text/html; charset=visual">

A solution would be to check if the meta content tag hold garbage data before committing to it.

Sample code to reproduce:

import fetch from "node-fetch"
import convertBody from "fetch-charset-detection"

fetch(
	"https://www.gov.il/apps/elections/Elections-knesset-15/heb/banner.html"
).then((res) =>
	res
		.arrayBuffer()
		.then((buf) => convertBody(buf))
		.then(console.log)
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant