-
-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add uk stores #455
Merged
Merged
feat: add uk stores #455
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
64647c9
Add helper function to extract card names and links from a page
andrewmackrodt 5c4075a
Add ttl for checking how often links should be refreshed
andrewmackrodt 9888e43
Allow specifying more than one url per series when fetching product l…
andrewmackrodt 2d2942e
Modify addNewLinks log verbosity
andrewmackrodt 4eba4bb
Update card model detection
andrewmackrodt 6430237
Add eBuyer UK store
andrewmackrodt f3cbbe0
Add Box UK store
andrewmackrodt a30dc9a
Add Scan UK store
andrewmackrodt 90082c7
Add Novatech UK store
andrewmackrodt aa3d530
Add Overclockers UK store
andrewmackrodt be12c3b
Add Currys UK Store
andrewmackrodt bc1482c
Add CCL Online UK Store
andrewmackrodt 8a86071
Add Aria PC UK Store
andrewmackrodt da9e90b
Add Amazon UK Store
andrewmackrodt 28cb629
Add Very UK Store
andrewmackrodt File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
import {Link, Store} from './store'; | ||
import {logger} from '../../logger'; | ||
import {parseCard} from './helpers/card'; | ||
|
||
export const AmazonUk: Store = { | ||
backoffStatusCodes: [403, 429, 503], | ||
labels: { | ||
captcha: { | ||
container: 'body', | ||
text: ['enter the characters you see below'] | ||
}, | ||
inStock: { | ||
container: '#availability', | ||
text: ['in stock'] | ||
}, | ||
maxPrice: { | ||
container: 'span[class*="PriceString"]' | ||
}, | ||
outOfStock: [ | ||
{ | ||
container: '#availability', | ||
text: ['out of stock', 'unavailable'] | ||
}, | ||
{ | ||
container: '#backInStock', | ||
text: ['unavailable'] | ||
} | ||
] | ||
}, | ||
links: [ | ||
{ | ||
brand: 'test:brand', | ||
cartUrl: 'https://www.amazon.co.uk/gp/aws/cart/add.html?ASIN.1=B081265T5Z&Quantity.1=1', | ||
model: 'test:model', | ||
series: 'test:series', | ||
url: 'https://www.amazon.co.uk/dp/B081265T5Z/' | ||
} | ||
], | ||
linksBuilder: { | ||
builder: (docElement, series) => { | ||
const productElements = docElement.find('.s-result-list .s-result-item[data-asin]'); | ||
const links: Link[] = []; | ||
for (let i = 0; i < productElements.length; i++) { | ||
const productElement = productElements.eq(i); | ||
const asin = productElement.attr()['data-asin']; | ||
|
||
if (!asin) { | ||
continue; | ||
} | ||
|
||
const url = `https://www.amazon.co.uk/dp/${asin}/`; | ||
const titleElement = productElement.find('.sg-col-inner h2 a.a-text-normal[href] span').first(); | ||
const title = titleElement.text().trim(); | ||
|
||
if (!title || !new RegExp(`RTX.*${series}`, 'i').exec(title)) { | ||
continue; | ||
} | ||
|
||
const card = parseCard(title); | ||
|
||
if (card) { | ||
links.push({ | ||
brand: card.brand as any, | ||
cartUrl: `https://www.amazon.co.uk/gp/aws/cart/add.html?ASIN.1=${asin}&Quantity.1=1`, | ||
model: card.model, | ||
series, | ||
url | ||
}); | ||
} else { | ||
logger.error(`Failed to parse card: ${title}`); | ||
} | ||
} | ||
|
||
return links; | ||
}, | ||
ttl: 300000, | ||
urls: [ | ||
{ | ||
series: '3080', | ||
url: [ | ||
'https://www.amazon.co.uk/s?k=%2B%22RTX+3080%22+-2080+-GTX&i=computers&rh=n%3A430500031%2Cp_n_availability%3A419162031&s=relevancerank&dc&qid=1601675291', | ||
'https://www.amazon.co.uk/s?k=%2B%22RTX+3080%22+-2080+-GTX&i=computers&rh=n%3A430500031%2Cp_n_availability%3A419162031&s=relevancerank&dc&qid=1601675594&page=2' | ||
] | ||
}, | ||
{ | ||
series: '3090', | ||
url: [ | ||
'https://www.amazon.co.uk/s?k=%2B%22RTX+3090%22+-3080+-GTX&i=computers&rh=n%3A430500031%2Cp_n_availability%3A419162031&s=relevancerank&dc&qid=1601675291', | ||
'https://www.amazon.co.uk/s?k=%2B%22RTX+3090%22+-3080+-GTX&i=computers&rh=n%3A430500031%2Cp_n_availability%3A419162031&s=relevancerank&dc&qid=1601675594&page=2' | ||
] | ||
} | ||
] | ||
}, | ||
name: 'amazon-uk' | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
import {Store} from './store'; | ||
import {getProductLinksBuilder} from './helpers/card'; | ||
|
||
export const Aria: Store = { | ||
labels: { | ||
inStock: { | ||
container: '#addQuantity', | ||
text: ['add to shopping basket'] | ||
}, | ||
outOfStock: { | ||
container: '.fBox', | ||
text: ['out of stock', 'there is currently no stock of this item'] | ||
} | ||
}, | ||
links: [ | ||
{ | ||
brand: 'test:brand', | ||
model: 'CARD', | ||
series: 'test:series', | ||
url: 'https://www.aria.co.uk/Products/Components/Graphics+Cards/NVIDIA+GeForce/GeForce+RTX+2060+Super/Gigabyte+NVIDIA+GeForce+RTX+2060+SUPER+8GB+WINDFORCE+OC+Turing+Graphics+Card+%2B+RTX+Bundle%21?productId=71541' | ||
} | ||
], | ||
linksBuilder: { | ||
builder: getProductLinksBuilder({ | ||
productsSelector: '#productListingInner .listTable .listTableTr', | ||
sitePrefix: 'https://www.aria.co.uk', | ||
titleSelector: 'strong > a[href]' | ||
}), | ||
urls: [ | ||
{ | ||
series: '3080', | ||
url: 'https://www.aria.co.uk/Products/Components/Graphics+Cards/NVIDIA+GeForce/GeForce+RTX+3080' | ||
}, | ||
{ | ||
series: '3090', | ||
url: 'https://www.aria.co.uk/Products/Components/Graphics+Cards/NVIDIA+GeForce/GeForce+RTX+3090' | ||
} | ||
] | ||
}, | ||
name: 'aria', | ||
waitUntil: 'domcontentloaded' | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
import {Store} from './store'; | ||
import {getProductLinksBuilder} from './helpers/card'; | ||
|
||
export const Box: Store = { | ||
labels: { | ||
inStock: { | ||
container: '#divBuyButton', | ||
text: ['add to basket'] | ||
}, | ||
outOfStock: { | ||
text: ['request stock alert', 'coming soon'] | ||
} | ||
}, | ||
links: [ | ||
{ | ||
brand: 'test:brand', | ||
model: 'CARD', | ||
series: 'test:series', | ||
url: 'https://www.box.co.uk/ASUS-TUF-GeForce-RTX-2060-6GB-Gaming-Gra_2669497.html' | ||
} | ||
], | ||
linksBuilder: { | ||
builder: getProductLinksBuilder({ | ||
productsSelector: '.products-right .p-list', | ||
sitePrefix: 'https://www.box.co.uk', | ||
titleSelector: '.p-list-section > h3 > a[href]' | ||
}), | ||
urls: [ | ||
{ | ||
series: '3070', | ||
url: 'https://www.box.co.uk/rtx-3070-graphics-cards' | ||
}, | ||
{ | ||
series: '3080', | ||
url: 'https://www.box.co.uk/rtx-3080-graphics-cards' | ||
}, | ||
{ | ||
series: '3090', | ||
url: 'https://www.box.co.uk/rtx-3090-graphics-cards' | ||
} | ||
] | ||
}, | ||
name: 'box', | ||
waitUntil: 'domcontentloaded' | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
import {Store} from './store'; | ||
import {getProductLinksBuilder} from './helpers/card'; | ||
|
||
export const Ccl: Store = { | ||
labels: { | ||
inStock: { | ||
container: '#pnlAddToBasket', | ||
text: ['add to basket'] | ||
}, | ||
outOfStock: { | ||
container: '#pnlSoldOut', | ||
text: ['sold out', 'coming soon'] | ||
} | ||
}, | ||
links: [ | ||
{ | ||
brand: 'test:brand', | ||
model: 'CARD', | ||
series: 'test:series', | ||
url: 'https://www.cclonline.com/product/296443/RTX-2060-SUPER-VENTUS-GP-OC/Graphics-Cards/MSI-GeForce-RTX-2060-SUPER-VENTUS-GP-OC-8GB-Overclocked-Graphics-Card/VGA5671/' | ||
} | ||
], | ||
linksBuilder: { | ||
builder: getProductLinksBuilder({ | ||
productsSelector: '.productListingContainerOuter .productList', | ||
sitePrefix: 'https://www.cclonline.com', | ||
titleAttribute: 'title', | ||
titleSelector: '.productList_Detail a[title]' | ||
}), | ||
urls: [ | ||
{ | ||
series: '3070', | ||
url: 'https://www.cclonline.com/category/430/PC-Components/Graphics-Cards/GeForce-RTX-3070-Graphics-Cards/' | ||
}, | ||
{ | ||
series: '3080', | ||
url: 'https://www.cclonline.com/category/430/PC-Components/Graphics-Cards/GeForce-RTX-3080-Graphics-Cards/' | ||
}, | ||
{ | ||
series: '3090', | ||
url: 'https://www.cclonline.com/category/430/PC-Components/Graphics-Cards/GeForce-RTX-3090-Graphics-Cards/' | ||
} | ||
] | ||
}, | ||
name: 'ccl', | ||
waitUntil: 'domcontentloaded' | ||
}; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I imagine we could use this for the other Amazon stores, but could be very wrong there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I imagine the
builder
function would work between stores, but determining theurls[].url[]
would require a manual map per region. At a minimum, the following may work (not tested it compiles let alone works):There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes sense to me. Maybe we can bring this in at a later point. Thanks for checking this out.