Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spoof/Change User-Agent #28

Open
jonzlin95 opened this issue Feb 8, 2017 · 2 comments
Open

Spoof/Change User-Agent #28

jonzlin95 opened this issue Feb 8, 2017 · 2 comments

Comments

@jonzlin95
Copy link
Contributor

jonzlin95 commented Feb 8, 2017

Some sites don't play well without User-Agents (e.g. ESPN).

HTTPoison by default sends hackney which is blocked, so it could be useful to set it to

["User-Agent": "Elixir"] or some similar string to bypass these problems.

@Anonyfox
Copy link
Owner

Anonyfox commented Feb 9, 2017

Why not faking a typical iPhone signature? I think some sites whitelist on UA strings

@jonzlin95
Copy link
Contributor Author

That could work too. Maybe it would be useful to add a way to pass in a User Agent to the Scrape.X function calls. I notice that you already have fetch(url, html_headers \ []), and that could just be passed onto the next set of functions.

I can make a simple PR for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants