Skip to content

Commit

Permalink
feat(lookup): use random user agents (#1335)
Browse files Browse the repository at this point in the history
  • Loading branch information
@jef
jef committed Dec 11, 2020
1 parent 2115c0c commit b599c23
Show file tree
Hide file tree
Showing 9 changed files with 116 additions and 71 deletions.
1 change: 0 additions & 1 deletion .env-example
View file
Expand Up @@ -111,5 +111,4 @@ TWITTER_ACCESS_TOKEN_SECRET=""
TWITTER_CONSUMER_KEY=""
TWITTER_CONSUMER_SECRET=""
TWITTER_TWEET_TAGS=""
USER_AGENT=""
WEB_PORT=""
12 changes: 2 additions & 10 deletions docs/help/troubleshoot.md
View file
Expand Up @@ -4,19 +4,11 @@

### Option 1

If you're running into _a lot_ of captcha problems, be sure to update your user agent by searching ["what's my user agent" on Google](https://www.google.com/search?q=whats+my+user+agent).

![image](https://user-images.githubusercontent.com/12074633/101272427-07a88100-375a-11eb-9cb3-4e8783db6ae5.png)

You can update your user agent by using `USER_AGENT="your-result"`.

### Option 2

If you're _still_ running into problems, try running in headful mode: `HEADLESS="false"`.
If you're running into problems, try running in headful mode: `HEADLESS="false"`.

This will open a browser and run streetmerchant. Note that this isn't a great solution for those running in a headless environment, i.e.: VPS, cloud, docker. Instead, it would be a good solution for those running on separate computer that won't be blocked by running in the background.

### Option 3
### Option 2

As a last case scenario, use `PUPPETEER_EXECUTABLE_PATH`. This will use your computer's Chrome browser. You can run this is headless or headful mode.

Expand Down
1 change: 0 additions & 1 deletion docs/reference/application.md
View file
Expand Up @@ -19,7 +19,6 @@
| `PROXY_ADDRESS` | IP Address or fqdn of proxy server |
| `PROXY_PORT` | TCP Port number on which the proxy is listening for connections. Default: `80` |
| `SCREENSHOT` | Capture screenshot of page if a card is found. Default: `true` |
| `USER_AGENT` | Custom User-Agents headers for HTTP requests. Newline separated, e.g.: `USER_AGENT_STRING1 \n USER_AGENT_STRING2`. Default: `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36` |
| `WEB_PORT` | Starts a webserver to be able to control the bot while it is running. Setting this value starts this service. |

???+ info
Expand Down

5 comments on commit b599c23

@gigi2006
Copy link

@gigi2006 gigi2006 commented on b599c23 Dec 11, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jef sorry quick question, but that I can no longer use the user agent as I want? so you have taken out the whole and put random in?

because since i read your readme and found this USER_AGENT_STRING1 \n USER_AGENT_STRING2 and understood it, i don't get captchas anymore....

was a little tedious to enter something over 100 user agents but I did it... Yes by hand :D

@jef
Copy link
Owner Author

@jef jef commented on b599c23 Dec 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gigi2006 I'm sorry. I'll actually revert some of these changes so you can use your custom user agents.

Let me know how it goes in the meantime though. I'd be curious.

@jef
Copy link
Owner Author

@jef jef commented on b599c23 Dec 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3386e8f

This should get you going again :)

@gigi2006
Copy link

@gigi2006 gigi2006 commented on b599c23 Dec 11, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jef i can not start anymore :/

//edit: sorry my mistake, had not seen that it is no longer the .env but the dotenv... everything fits perfectly :)

@jef
Copy link
Owner Author

@jef jef commented on b599c23 Dec 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jef i can not start anymore :/

//edit: sorry my mistake, had not seen that it is no longer the .env but the dotenv... everything fits perfectly :)

I also goofed this up... Shouldn't code late night 馃槗 You can use .env or dotenv now.

Thanks for your feedback 馃榿

Please sign in to comment.