Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add support for multiple browser user agents #547

Merged
merged 3 commits into from
Oct 20, 2020
Merged

feat: add support for multiple browser user agents #547

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5feb63f
Added support for multiple browser user agents that get used randomly…
Oct 19, 2020
ec0d6f9
Update config.ts
SesioN Oct 20, 2020
2cd02a1
revert: USER_AGENTS to non-plural USER_AGENT
Oct 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ Here is a list of variables that you can use to customize your newly copied `.en
| `TWITTER_CONSUMER_KEY` | Twitter Consumer Key | Generate all Twitter keys at: https://developer.twitter.com/ |
| `TWITTER_CONSUMER_SECRET` | Twitter Consumer Secret | |
| `TWITTER_TWEET_TAGS` | Optional list of hashtags to append to the tweet message | E.g.: `#nvidia #nvidiastock` |
| `USER_AGENT` | Custom User-Agent header for HTTP requests | Default: `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36` |
| `USER_AGENT` | Custom User-Agents headers for HTTP requests | Newline separated, E.g.: `USER_AGENT_STRING1 \n USER_AGENT_STRING2` | | Default: `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36` |

> :point_right: If you have multi-factor authentication (MFA), you will need to create an [app password](https://myaccount.google.com/apppasswords) and use this instead of your Gmail password.

Expand Down
6 changes: 4 additions & 2 deletions src/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ config_({path: path.resolve(__dirname, '../.env')});
* @param array Default array. If not set, is `[]`.
*/
function envOrArray(environment: string | undefined, array?: string[]): string[] {
return (environment ? environment.split(',') : (array ?? [])).map(s => s.trim());
return (environment ? (
environment.includes('\n') ? environment.split('\n') : environment.split(',')
) : (array ?? [])).map(s => s.trim());
}

/**
Expand Down Expand Up @@ -195,7 +197,7 @@ const page = {
inStockWaitTime: envOrNumber(process.env.IN_STOCK_WAIT_TIME),
screenshot: envOrBoolean(process.env.SCREENSHOT),
timeout: envOrNumber(process.env.PAGE_TIMEOUT, 30000),
userAgent: envOrString(process.env.USER_AGENT, 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'),
userAgents: envOrArray(process.env.USER_AGENT, ['Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36']),
width: 1920
};

Expand Down
4 changes: 2 additions & 2 deletions src/store/lookup.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import {Browser, Page, Response} from 'puppeteer';
import {Link, Store} from './model';
import {Print, logger} from '../logger';
import {Selector, cardPrice, pageIncludesLabels} from './includes-labels';
import {closePage, delay, getSleepTime, isStatusCodeInRange} from '../util';
import {closePage, delay, getRandomUserAgent, getSleepTime, isStatusCodeInRange} from '../util';
import {config} from '../config';
import {disableBlockerInPage} from '../adblocker';
import {fetchLinks} from './fetch-links';
Expand Down Expand Up @@ -37,7 +37,7 @@ async function lookup(browser: Browser, store: Store) {

const page = await browser.newPage();
page.setDefaultNavigationTimeout(config.page.timeout);
await page.setUserAgent(config.page.userAgent);
await page.setUserAgent(getRandomUserAgent());

if (store.disableAdBlocker) {
try {
Expand Down
6 changes: 5 additions & 1 deletion src/util.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ export async function usingResponse<T>(
export async function usingPage<T>(browser: Browser, cb: (page: Page, browser: Browser) => Promise<T>): Promise<T> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(config.page.timeout);
await page.setUserAgent(config.page.userAgent);
await page.setUserAgent(getRandomUserAgent());

try {
return await cb(page, browser);
Expand All @@ -68,3 +68,7 @@ export async function closePage(page: Page) {

await page.close();
}

export function getRandomUserAgent(): string {
return config.page.userAgents[Math.floor(Math.random() * config.page.userAgents.length)];
}