Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: set country in config, login to nvidia when starting #162

Merged
merged 23 commits into from
Sep 22, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
f4a0776
Break out Nvidia setup
fuckingrobot Sep 21, 2020
6dc2e52
Add ID and locale documentation to comment
fuckingrobot Sep 21, 2020
70f5077
Add series to card name in log
fuckingrobot Sep 21, 2020
96d6c88
Move ID and locale constants to Nvidia store
fuckingrobot Sep 21, 2020
6a6197c
Set everything based on country
fuckingrobot Sep 21, 2020
6002790
Add locale to stock URL
fuckingrobot Sep 21, 2020
b60e9d3
Split out nvidia and digital river locales
fuckingrobot Sep 21, 2020
475e862
Load from config, populate values from #161
fuckingrobot Sep 21, 2020
d5052f0
Update readme for countries
fuckingrobot Sep 21, 2020
27dae92
Add missing semicolons
fuckingrobot Sep 21, 2020
15bc8dc
Remove Russia, no ID
fuckingrobot Sep 21, 2020
2fbb999
Spacing fixes for linter
fuckingrobot Sep 21, 2020
4bb6de1
Linter fixes
fuckingrobot Sep 21, 2020
ea4dd37
Add 3090 support despite missing all IDs
fuckingrobot Sep 21, 2020
c982efd
Add great_britain, alphabetize countries
fuckingrobot Sep 21, 2020
b6aa057
Create cart when starting run to manually log in early
fuckingrobot Sep 21, 2020
b0fbfe2
Open cart for login automatically
fuckingrobot Sep 21, 2020
77be870
Linter fixes
fuckingrobot Sep 21, 2020
408c87f
Merge branch 'main' into nvidia-add-to-cart-trois
fuckingrobot Sep 21, 2020
8bf60ce
Add Ireland
fuckingrobot Sep 21, 2020
705550c
Merge branch 'main' into nvidia-add-to-cart-trois
fuckingrobot Sep 21, 2020
5a1de79
Merge branch 'main' into nvidia-add-to-cart-trois
fuckingrobot Sep 21, 2020
9ea62f9
Merge branch 'main' into nvidia-add-to-cart-trois
fuckingrobot Sep 22, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .env-example
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ SHOW_ONLY_SERIES="3080"
SLACK_CHANNEL="SlackChannelName"
SLACK_TOKEN="slack-token"
STORES="bestbuy,bandh,nvidia"
COUNTRY="usa"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we plan to allow multiple countries? e.g: I am in Canada, and would like to scan both US and CA stores

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the first step in localization. We don't want to localize every store all at once in case there are issues. Nvidia is the first and proof of concept, assuming all goes well we can work on implementing localization across stores.

Copy link

@kirbdee kirbdee Sep 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Work around would be to just run two instances with different configs, but also depends on the stores. IE FE's from nvidia would be the same stock

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a priority for me, but yes, this is the first step to getting there (while keeping the code clean-ish/manageable).

SCREENSHOT="true"
TELEGRAM_ACCESS_TOKEN=""
TELEGRAM_CHAT_ID="1234"
Expand Down
25 changes: 25 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ Here is a list of variables that you can use to customize your newly copied `.en
| `SLACK_CHANNEL` | Slack channel for posting | E.g., `update`, no need for `#` |
| `SLACK_TOKEN` | Slack API token |
| `STORES` | [Supported stores](#supported-stores) you want to be scraped | Comma separated, default: `nvidia` |
| `COUNTRY` | [Supported country](#supported-countries) you want to be scraped, currently only used by Nvidia | default: `usa` |
| `SCREENSHOT` | Capture screenshot of page if a card is found | Default: `true` |
| `TELEGRAM_ACCESS_TOKEN` | Telegram access token |
| `TELEGRAM_CHAT_ID` | Telegram chat ID |
| `USER_AGENT` | Custom User-Agent header for HTTP requests | Default: `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36` |
Expand Down Expand Up @@ -124,6 +126,29 @@ Here is a list of variables that you can use to customize your newly copied `.en
| T-Mobile | `tmobile`| |
| Verizon | `verizon`| Works with Visible |

#### Supported countries

| **Country** | **Nvidia.com (3080 FE)** | **Nvidia.com (3090 FE)** | **Notes** |
|:---:|:---:|:---:|:---:|
| austria | `✔` | | |
| belgium | `✔` | | Nvidia supports debug |
| canada | `✔` | | |
| czechia | `✔` | | |
| denmark | `✔` | | |
| finland | `✔` | | |
| france | `✔` | | |
| germany | `✔` | | |
| great_britain | `✔` | | |
| ireland | `✔` | | |
| italy | `✔` | | |
| luxembourg | `✔` | | Nvidia supports debug |
| poland | `✔` | | |
| portugal | `✔` | | |
| russia | | | Missing all IDs |
| spain | `✔` | | |
| sweden | `✔` | | |
| usa | `✔` | | Nvidia supports debug |

## FAQ

**Q: What's Node.js and how do I install it?** Visit [their website](https://nodejs.org/en/) and download and install it. Very straight forward. Otherwise, Google more information related to your system needs.
Expand Down
3 changes: 2 additions & 1 deletion src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,8 @@ const page = {
const store = {
showOnlySeries: process.env.SHOW_ONLY_SERIES ? process.env.SHOW_ONLY_SERIES.split(',') : ['3070', '3080', '3090'],
showOnlyBrands: process.env.SHOW_ONLY_BRANDS ? process.env.SHOW_ONLY_BRANDS.split(',') : [],
stores: process.env.STORES ? process.env.STORES.split(',') : ['nvidia']
stores: process.env.STORES ? process.env.STORES.split(',') : ['nvidia'],
country: process.env.COUNTRY ?? 'usa'
};

export const Config = {
Expand Down
4 changes: 4 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ async function main() {

for (const store of Stores) {
Logger.debug(store.links);
if (store.setupAction !== undefined) {
store.setupAction(browser);
}

setTimeout(tryLookupAndLoop, getSleepTime(), browser, store);
}
}
Expand Down
2 changes: 1 addition & 1 deletion src/store/lookup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ async function lookup(browser: Browser, store: Store) {
page.setDefaultNavigationTimeout(Config.page.navigationTimeout);
await page.setUserAgent(Config.page.userAgent);

const graphicsCard = `${link.brand} ${link.model}`;
const graphicsCard = `${link.brand} ${link.model} ${link.series}`;

let response: Response | null;
try {
Expand Down
148 changes: 148 additions & 0 deletions src/store/model/helpers/nvidia.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
import {timestampUrlParameter} from '../../timestamp-url-parameter';
import {Browser, Response} from 'puppeteer';
import {Logger} from '../../../logger';
import open from 'open';
import {Link} from '../store';
import {Config} from '../../../config';
import {NvidiaRegionInfo, regionInfos} from '../nvidia';

const nvidiaApiKey = '9485fa7b159e42edb08a83bde0d83dia';

function getRegionInfo(): NvidiaRegionInfo {
const country = Array.from(regionInfos.keys()).includes(Config.store.country) ? Config.store.country : 'usa';

const defaultRegionInfo: NvidiaRegionInfo = {drLocale: 'en_us', nvidiaLocale: 'en_us', fe3080Id: 5438481700, fe3090Id: null, fe2060SuperId: 5379432500};
return regionInfos.get(country) ?? defaultRegionInfo;
}

function digitalRiverStockUrl(id: number, drLocale: string): string {
return `https://api.digitalriver.com/v1/shoppers/me/products/${id}/inventory-status?` +
`&apiKey=${nvidiaApiKey}` +
`&locale=${drLocale}` +
timestampUrlParameter();
}

interface NvidiaSessionTokenJSON {
access_token: string;
}

function nvidiaSessionUrl(nvidiaLocale: string): string {
return `https://store.nvidia.com/store/nvidia/SessionToken?format=json&locale=${nvidiaLocale}` +
`&apiKey=${nvidiaApiKey}` +
timestampUrlParameter();
}

function addToCartUrl(id: number, drLocale: string, token: string): string {
return 'https://api.digitalriver.com/v1/shoppers/me/carts/active/line-items?format=json&method=post' +
`&productId=${id}` +
`&token=${token}` +
'&quantity=1' +
`&locale=${drLocale}` +
timestampUrlParameter();
}

function checkoutUrl(drLocale: string, token: string): string {
return `https://api.digitalriver.com/v1/shoppers/me/carts/active/web-checkout?token=${token}&locale=${drLocale}`;
}

function fallbackCartUrl(nvidiaLocale: string): string {
return `https://www.nvidia.com/${nvidiaLocale}/shop/geforce?${timestampUrlParameter()}`;
}

export function generateSetupAction() {
return async (browser: Browser) => {
const {drLocale, nvidiaLocale} = getRegionInfo();

const page = await browser.newPage();

Logger.info('[nvidia] creating cart/session token...');
let response: Response | null;
try {
response = await page.goto(nvidiaSessionUrl(nvidiaLocale), {waitUntil: 'networkidle0'});
if (response === null) {
throw new Error('NvidiaAccessTokenUnavailable');
}

const data = await response.json() as NvidiaSessionTokenJSON;
const accessToken = data.access_token;

Logger.info('[nvidia] you can log into your cart now...');
Logger.info(checkoutUrl(drLocale, accessToken));
await open(checkoutUrl(drLocale, accessToken));
} catch (error) {
Logger.debug(error);
Logger.error('✖ [nvidia] cannot generate cart/session token, continuing without, auto-"add to cart" may not work...');
}

await page.close();
};
}

export function generateOpenCartAction(id: number, nvidiaLocale: string, drLocale: string, cardName: string) {
return async (browser: Browser) => {
const page = await browser.newPage();
Logger.info(`🚀🚀🚀 [nvidia] ${cardName}, starting auto add to cart... 🚀🚀🚀`);
let response: Response | null;
try {
Logger.info(`🚀🚀🚀 [nvidia] ${cardName}, getting access token... 🚀🚀🚀`);
response = await page.goto(nvidiaSessionUrl(nvidiaLocale), {waitUntil: 'networkidle0'});
if (response === null) {
throw new Error('NvidiaAccessTokenUnavailable');
}

const data = await response.json() as NvidiaSessionTokenJSON;
const accessToken = data.access_token;

Logger.info(`🚀🚀🚀 [nvidia] ${cardName}, adding to cart... 🚀🚀🚀`);
response = await page.goto(addToCartUrl(id, drLocale, accessToken), {waitUntil: 'networkidle0'});

Logger.info(`🚀🚀🚀 [nvidia] ${cardName}, opening checkout page... 🚀🚀🚀`);
Logger.info(checkoutUrl(drLocale, accessToken));
await open(checkoutUrl(drLocale, accessToken));
} catch (error) {
Logger.debug(error);
Logger.error(`✖ [nvidia] ${cardName} could not automatically add to cart, opening page`);
await open(fallbackCartUrl(nvidiaLocale));
}

await page.close();
};
}

export function generateLinks(): Link[] {
const {drLocale, nvidiaLocale, fe3080Id, fe3090Id, fe2060SuperId} = getRegionInfo();

const links: Link[] = [];

if (fe2060SuperId) {
links.push({
series: 'debug',
brand: 'TEST',
model: 'CARD',
url: digitalRiverStockUrl(fe2060SuperId, drLocale),
openCartAction: generateOpenCartAction(fe2060SuperId, nvidiaLocale, drLocale, 'TEST CARD debug')
});
}

if (fe3080Id) {
links.push({
series: '3080',
brand: 'nvidia',
model: 'founders edition',
url: digitalRiverStockUrl(fe3080Id, drLocale),
openCartAction: generateOpenCartAction(fe3080Id, nvidiaLocale, drLocale, 'nvidia founders edition 3080')
});
}

if (fe3090Id) {
links.push({
series: '3090',
brand: 'nvidia',
model: 'founders edition',
url: digitalRiverStockUrl(fe3090Id, drLocale),
openCartAction: generateOpenCartAction(fe3090Id, nvidiaLocale, drLocale, 'nvidia founders edition 3090')
});
}

return links;
}
116 changes: 31 additions & 85 deletions src/store/model/nvidia.ts
Original file line number Diff line number Diff line change
@@ -1,96 +1,42 @@
import {Store} from './store';
import {Browser, Response} from 'puppeteer';
import {timestampUrlParameter} from '../timestamp-url-parameter';
import {Logger} from '../../logger';
import open from 'open';
import {generateLinks, generateSetupAction} from './helpers/nvidia';

const fe2060SuperId = 5379432500;
const fe3080Id = 5438481700;
const locale = 'en_us';
// Region/country set by config file, silently ignores null / missing values and defaults to usa

const nvidiaApiKey = '9485fa7b159e42edb08a83bde0d83dia';

function digitalRiverStockUrl(id: number): string {
return `https://api.digitalriver.com/v1/shoppers/me/products/${id}/inventory-status?` +
`&apiKey=${nvidiaApiKey}` +
timestampUrlParameter();
}

interface NvidiaSessionTokenJSON {
access_token: string;
}

function nvidiaSessionUrl(): string {
return `https://store.nvidia.com/store/nvidia/SessionToken?format=json&locale=${locale}` +
`&apiKey=${nvidiaApiKey}` +
timestampUrlParameter();
}

function addToCartUrl(id: number, token: string): string {
return 'https://api.digitalriver.com/v1/shoppers/me/carts/active/line-items?format=json&method=post' +
`&productId=${id}` +
`&token=${token}` +
'&quantity=1' +
timestampUrlParameter();
}

function checkoutUrl(token: string): string {
return `https://api.digitalriver.com/v1/shoppers/me/carts/active/web-checkout?token=${token}`;
}

function fallbackCartUrl(): string {
return `https://www.nvidia.com/en-us/shop/geforce?${timestampUrlParameter()}`;
export interface NvidiaRegionInfo {
drLocale: string;
nvidiaLocale: string;
fe3080Id: number | null;
fe3090Id: number | null;
fe2060SuperId: number | null;
}

function generateCartAction(id: number, cardName: string) {
return async (browser: Browser) => {
const page = await browser.newPage();
Logger.info(`🚀🚀🚀 [nvidia] ${cardName}, starting auto add to cart... 🚀🚀🚀`);
let response: Response | null;
try {
Logger.info(`🚀🚀🚀 [nvidia] ${cardName}, getting access token... 🚀🚀🚀`);
response = await page.goto(nvidiaSessionUrl(), {waitUntil: 'networkidle0'});
if (response === null) {
throw new Error('NvidiaAccessTokenUnavailable');
}

const data = await response.json() as NvidiaSessionTokenJSON;
const accessToken = data.access_token;

Logger.info(`🚀🚀🚀 [nvidia] ${cardName}, adding to cart... 🚀🚀🚀`);
response = await page.goto(addToCartUrl(id, accessToken), {waitUntil: 'networkidle0'});

Logger.info(`🚀🚀🚀 [nvidia] ${cardName}, opening checkout page... 🚀🚀🚀`);
Logger.info(checkoutUrl(accessToken));
await open(checkoutUrl(accessToken));
} catch {
Logger.error(`✖ [nvidia] ${cardName} could not automatically add to cart, opening page`);
await open(fallbackCartUrl());
}

await page.close();
};
}
export const regionInfos = new Map<string, NvidiaRegionInfo>([
['austria', {drLocale: 'de_de', nvidiaLocale: 'de_de', fe3080Id: 5440853700, fe3090Id: null, fe2060SuperId: null}],
['belgium', {drLocale: 'fr_fr', nvidiaLocale: 'fr_fr', fe3080Id: 5438795700, fe3090Id: null, fe2060SuperId: 5394902700}],
['canada', {drLocale: 'en_us', nvidiaLocale: 'en_ca', fe3080Id: 5438481700, fe3090Id: null, fe2060SuperId: null}],
['czechia', {drLocale: 'en_gb', nvidiaLocale: 'en_gb', fe3080Id: 5438793800, fe3090Id: null, fe2060SuperId: null}],
['denmark', {drLocale: 'en_gb', nvidiaLocale: 'en_gb', fe3080Id: 5438793300, fe3090Id: null, fe2060SuperId: null}],
['finland', {drLocale: 'en_gb', nvidiaLocale: 'en_gb', fe3080Id: 5438793300, fe3090Id: null, fe2060SuperId: null}],
['france', {drLocale: 'fr_fr', nvidiaLocale: 'fr_fr', fe3080Id: 5438795200, fe3090Id: null, fe2060SuperId: null}],
['germany', {drLocale: 'de_de', nvidiaLocale: 'de_de', fe3080Id: 5438792300, fe3090Id: null, fe2060SuperId: null}],
['great_britain', {drLocale: 'en_gb', nvidiaLocale: 'en_gb', fe3080Id: 5438792800, fe3090Id: null, fe2060SuperId: null}],
['ireland', {drLocale: 'en_gb', nvidiaLocale: 'en_gb', fe3080Id: 5438792800, fe3090Id: null, fe2060SuperId: null}],
['italy', {drLocale: 'it_it', nvidiaLocale: 'it_it', fe3080Id: 5438796200, fe3090Id: null, fe2060SuperId: null}],
['luxembourg', {drLocale: 'fr_fr', nvidiaLocale: 'fr_fr', fe3080Id: 5438795700, fe3090Id: null, fe2060SuperId: 5394902700}],
['poland', {drLocale: 'pl_pl', nvidiaLocale: 'pl_pl', fe3080Id: 5438797700, fe3090Id: null, fe2060SuperId: null}],
['portugal', {drLocale: 'en_gb', nvidiaLocale: 'en_gb', fe3080Id: 5438794300, fe3090Id: null, fe2060SuperId: null}],
['russia', {drLocale: 'ru_ru', nvidiaLocale: 'ru_ru', fe3080Id: null, fe3090Id: null, fe2060SuperId: null}],
['spain', {drLocale: 'es_es', nvidiaLocale: 'es_es', fe3080Id: 5438794800, fe3090Id: null, fe2060SuperId: null}],
['sweden', {drLocale: 'sv_SE', nvidiaLocale: 'sv_se', fe3080Id: 5438798100, fe3090Id: null, fe2060SuperId: null}],
['usa', {drLocale: 'en_us', nvidiaLocale: 'en_us', fe3080Id: 5438481700, fe3090Id: null, fe2060SuperId: 5379432500}]
]);

export const Nvidia: Store = {
links: [
{
series: 'debug',
brand: 'TEST',
model: 'CARD',
url: digitalRiverStockUrl(fe2060SuperId),
openCartAction: generateCartAction(fe2060SuperId, 'TEST CARD')
},
{
series: '3080',
brand: 'nvidia',
model: 'founders edition',
url: digitalRiverStockUrl(fe3080Id),
openCartAction: generateCartAction(fe3080Id, 'nvidia founders edition 3080')
}
],
links: generateLinks(),
labels: {
outOfStock: ['product_inventory_out_of_stock', 'rate limit exceeded', 'request timeout']
},
name: 'nvidia'
name: 'nvidia',
setupAction: generateSetupAction()
};
1 change: 1 addition & 0 deletions src/store/model/store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,5 @@ export interface Store {
links: Link[];
labels: Labels;
name: string;
setupAction?: (browser: Browser) => void;
}