usage of proxies is broken in nightly release (net::ERR_FAILED)

[richie256]

Expected Behavior

Using proxies with the docker nightly build or nodejs (using the latest code from github) should works fine without net::ERR_FAILED error.

Current Behavior

Using proxies with the nightly build or nodejs (using the latest code from github) return net::ERR_FAILED error. It works as expected using latest docker image.

Steps to Reproduce

1. Configure the dotenv and global.proxies file with valid proxies
2. Launch docker run -it --rm --env-file ./dotenv -v $(pwd)/global.proxies:/app/global.proxies ghcr.io/jef/streetmerchant:nightly

OR

1. Clone the streetmerchant github.
2. Configure the dotenv and global.proxies file with valid proxies
3. Install and run: npm run start

Environment

• OS: I've tried Windows 10 and MacOS.
• NodeJS: I've tried NodeJS 16, and NodeJS 15.

dotenv

Nothing Particular dotenv. Any should reproduce the same result.

global.proxies

socks5://<my_username>:<my_password>@<my_secret_ip_address_1>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_2>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_3>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_4>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_5>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_6>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_7>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_8>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_9>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_10>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_11>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_12>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_13>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_14>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_15>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_16>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_17>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_18>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_19>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_20>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_21>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_22>:1085
socks5://<my_username>:<my_password>@<my_secret_ip_address_23>:1085

Logs

Using proxies with the nightly build we got net::ERR_FAILED error.

[1:38:42 AM] error :: ✖ [2/23] [newegg-ca] evga 3080 ftw3 - net::ERR_FAILED at https://www.newegg.ca/evga-geforce-rtx-3080-10g-p5-3895-kr/p/N82E16814487519
[1:38:42 AM] error :: ✖ [3/23] [canadacomputers] msi 3080 gaming x trio - net::ERR_FAILED at https://www.canadacomputers.com/product_info.php?cPath=43_557_559&item_id=181348&language=en
[1:38:42 AM] error :: ✖ [4/23] [memoryexpress] evga 3080 xc3 black - net::ERR_FAILED at https://www.memoryexpress.com/Products/MX00114094

Exactly the same config works as expected with the latest docker image

[1:47:52 AM] info :: ✖ [2/23] [memoryexpress] [gigabyte (3080)] eagle oc :: OUT OF STOCK
[1:47:52 AM] info :: ✖ [1/23] [canadacomputers] [evga (3080)] ftw3 ultra :: OUT OF STOCK

[jef]

Hi there, thanks for reporting. Is this still the case?

[richie256]

hello @jef ,

Yes it is still Broken.

I've done some research: I've list all the commit between HEAD and Tag v3.5.0 and tested the versions.
Based on my test, it doesn't work since the commit: 1d08d4c12292e1b98ad952fa4f88c1965fa7bd6a chore: update dependencies

Proxy Test Results

00b468a73832b959e0c6ac1ed67b0b23856db76b (HEAD -> main, origin/main, origin/HEAD) docs: make templates required
74d71e457956743566796404458bd1d5d84598db docs: update issue templates
9aa3f164a59cd521701f55bfe4a54872b1c5fc4d docs: update issue templates
71711131ff4c57abe3016e543b7d0ce994e0ecd0 docs: update issue templates
9c346e70fac17854d347dfbdbc238bfb0056c6a8 fix(terraform): update IAM configuration for ECS deployment (#2675)
61af2a4325733463c918396868d313b7632bb310 fix: partial url regexp (#2382)
f14a1ca76e41943e639e08a9c8a6299dfa362d9d chore: update dependencies       ==> TEST: FAILED
0c589a71f259c80ff21a4bb623f9316cb7702a61 fix(bestbuy-ca): avoid false positive (#2694)
0b060e8f6aca1793d04474210aeec98212172014 feat: add a few stores for spain (#2646)
4bb58ac26192354ccbcd0c416aba78df2234cf19 chore(coolblue): add ps5 consoles (#2597)
e0303106f725ee9726c0b7e2f29d97fe5137ca90 chore(deps): bump actions/stale from 3 to 4 (#2671) ==> TEST: FAILED
5e9b3db6f1630f2000da7f9a8b0dd0dc34c8341f chore: revert puppeteer, run linter
1d08d4c12292e1b98ad952fa4f88c1965fa7bd6a chore: update dependencies	 ==> TEST: FAILED
4e3eadea083071eba563dffc70d749db334dce69 chore: bump node 16.5.0 	 ==> TEST: OK
8c7b8aabe4633757dea836b7d11af2a407869e90 feat: add alternate-fr, 3070ti, change alternate to alternate.de (#2653). ==> TEST: OK
c0f46d77cf9d4a3f7481aec5b0ddf33883b47d6e feat(stores-nz): update labels (#2486)
cc0aaeca77dd18a2c9797055f26edf6935fa755b chore(bestbuy): add 3080ti models (#2670)
b33f678090396b77a5765115d7a3fa42f6de06c8 chore: prune invalid links (#2592)
51c9b3ab2ac251c9c3abebf64025e1a03ecf4944 feat: add 3080ti (#2600)
4cfbd37ae0d28d1c2b8cf596b5dcb941daa7d9bc feat: add 3080ti (#2598)
52b5e5e79ac381b585ec1dd63288aefe4f9da9d1 ci: fix nightly
812dc5ea8d9830c251a526c02de05a15bfe20602 ci: change check-date name
3b43732f291d263792585fa05264dcf322b55620 chore: update dependencies and nightly workflow
e4a267f675800b04d06ddb4212de1373a767a113 chore: update stale workflow
da635f41dd3c3e949f3d77fc2b1bcbe860639179 build: remove smartthings
2e5b07c93fb02677596eb5a2f2ea857b99c2d4d9 chore(deps): bump nodemailer from 6.6.0 to 6.6.1 (#2568)
a751cb11c53c7cf6b7883e69eb5e8e2fe1e57adf chore(deps): bump node-notifier from 9.0.1 to 10.0.0 (#2583)
e4ebdc4719afdf390f7060bbc755229dcaba1410 chore(deps-dev): bump @types/cheerio from 0.22.28 to 0.22.29 (#2580)
1cb1787c27b2e7e37e3e989be686b635f98442e9 chore(deps-dev): bump @types/nodemailer from 6.4.1 to 6.4.2 (#2582)
9fc5ec06ff92e458ea9649f22af9df1ae5ba1d95 chore(deps-dev): bump webpack from 5.37.0 to 5.38.0 (#2579)  ==> TEST: OK
9ba964a587158b551d7217764eb9799073aea21f chore: bump node version
9e13cd7dbb8026c53da4499309ebef03d51c3e92 chore(deps-dev): bump webpack from 5.37.0 to 5.37.1 (#2558)
3e74a4c7c41c939b9ce0952e089f36269f3fb46f chore(deps-dev): bump @types/node from 15.3.0 to 15.6.1 (#2571)
7d35c2b8f0620501061220baefeb499d6e66e2da chore(deps): bump twilio from 3.62.0 to 3.63.0 (#2561)
36e188fd36d31a915d9ac59016710b6f3fa4436d chore(deps): bump open from 8.0.2 to 8.2.0 (#2572)
11220750143df7ebdc011aafccb8e035bfd2bc5d chore(alternate): add PS5 bundle (#2562)
8854ce97bba7858dfc9385530e0c090200944ede fix: clear cookies when rotating proxies (#2575)
ebf8b20071dc628b1f47b96ee2b03f888c056dd9 ci: fix documentation
729afd0ef3f4ceea1ae38b04e1a25bc4c4dcd3ec chore(deps): bump top-user-agents from 1.0.25 to 1.0.26 (#2556)
42a24a633c537941d63a6b76ed9598179dda9ea4 chore(computeruniverse): add backoff code (#2542)
32fe440049ce97c5683ca03276212899b2211af5 feat(gotify): add priority option (#2546)
0378a7cf0aa673933af35c3a3605f147615f6e96 chore(netonnet-no): update selectors (#2547)
1e62878c8d1b3c0b49189b05f846d555cfc205b3 chore(deps): bump node 16.1.0
9f58c22c0c6d14d0f0e59782bbe71a18b8329f6f style(gotify): linting
d12639ee8561240d99ef5c2c364255ff808f78b0 chore(notification): add better logging, remove debug
8f2de5dc68667f6b9f0033bed129fd6cdbd9d8a7 feat(notification): add gotify
1ab0af234ca76b5db8602ffe04a1fc80b3c6515b chore(deps): bump dependencies
6e614a24918f38d189a3d3b70174fbd1590f78cc feat(captcha): add support for configurable capture type (#2468)
b6af582f70843c5608249c5d629fdec8f10630dd chore: remove hooks
d0a6c7e85950fe3cffbcb30662967a687aaf7347 chore(otto): add labels for xbox series x & s (#2515)
c9049a3054d35f8a37340e552e19841e3ecaf631 chore(elkjop): update selectors (#2505)
64120e73d8f3a01dd5852044ffa0ef6b0ce31730 chore(mediamarkt): update captcha selector (#2496)
4d8930616d2296fcdd6c0d79bac2109f1388a0b0 chore(xbox): update oos selector
164902a587d80f0421b44079e0616ba37a7a743f fix(captcha): use Slack captcha handler token when provided (#2460)
790fe9b6cf4f9627e7f3dc7acc572737a739e4cc fix(amazon): false positives for xbox (#2497)
7c4c6ed79ecd89767ffbb3577716be28ce6c1bdc chore(multicom): update selectors (#2499)
e397306f1a3b1576c0595dec5fae7be277e88f25 chore(deps-dev): bump webpack from 5.35.0 to 5.36.2 (#2480)
b57905291df85ee2f722e5e3a81d1cecd0ab6ee4 feat: add INCOGNITO to dotenv-example and volta support (#2484)
441617f6729ab61174f7bcfe287c41a34a78728f feat(store): add amd-at (#2462)
5a744b1add24dd4d3c250f41382a034212d9c5b0 feat(store): add cyberport-at (#2463)
eb76f0d2b78726c4380853bd47edcf9b1302e7bb fix: remove spaces from price (#2482)
41305f1ed5a81240ba7aa0dda1619d635d917371 feat(store): add multicom.no and more products to NO stores (#2483)
39fac90ba4feadb7e5c681db60e60a83749f5e62 fix(lookup): prevent terminal from being hidden (#2461)
cac7c20193caaec5d41291104e66aa01a65adceb feat: add waitUntil and waitForSelctor to linksBuilder (#2437)
35f9eb43b6cca8a92f60ba2870e60a225b4c1da6 fix(alternate): use button selector instead of text (#2419)
b6ac6c53ddc34ac35b537dd99415bbcb9eec0de7 fix(nodemon): use npx with ts-node (#2423)
092919ab349edac6a8b5b4747917a1a232235da7 fix(microcenter): add outOfStock label (#2432)
d652b4936435377d36268141a2b63c8798f7f95b refactor(captcha): use seconds instead of ms (#2424)
2af153192c2cf3ac2bf9ba0d1d3201f250363c9b chore: change urls.  ==> TEST: OK
5937d0141d1c34fdb90475bffbbe32a2977c789b chore(alternate): add new oos label (#2409)
3c00eca22378afb2a18aa76c1d4204107e6364c2 chore: bump dependencies (#2413)
409769e6dbd3add614fcd985731bb244297a2c2d feat: add interactive captcha handler with Slack and Discord (#2385)
8c39d0d30a9f8db7eda8665eb08d350fadf157e4 chore(materiel): add 3070 and 3080 cards (#2393)
4eb702b9a1187c8dcf159ea991ab1049c3c44bb2 fix(amazon-es): use euroFormat (#2371)
2be33ac6147f8f7460d108f361d954b17a335bd9 fix(web): add favicon to prevent missing icon console error (#2375)
d4dc397e9a848d6d2ba1ed56188fb0ddda98e795 chore(store): add some PS5 bundles (#2021)
d13f7f1ff91ada223e8ce7eb50d9a80030cbe668 fix(docker): disable opening browser
f0e5c5d17bad282480da30cc68e5bd8eb9d80844 chore: update pr template
6e9b1a9640ca93f620623d53624230514fac6d40 feat(store): add amd-be store (#2338)
49c9e8c0dc5c3b25ce2aa20265ccd2865f229385 feat(store): add amd-nl store (#2072)
bc59da11d512f6f55449781cb8f7f4bd1d3e7e57 fix(discord): properly wait for webhooks (#2352)
aed9cfe98919e6524bfde0156a308c9a56d4dfc0 chore(deps): bump dependencies
e2d804027a3ec6a19201ef9ce35dcb4c4dee7025 feat(store): add MightyApe, NoelLeeming and TheWarehouse (NZ) (#2133)
c1c92bd25b7b744a05f21664f2029a0cb0683d35 chore(gamestop): update inventory (#2239)
ae8e72a73044b8311add15d2b5c10936bae16e82 feat(store): add mediamarkt austria (#1836)
211e3e2692c1c97f607b39b73623a09958c69d81 chore(currys): add rx6800, rx6800xt and rx6900xt (#2236)
7ec8a160cf9844df34578fe9f413b88d6141178c refactor(materiel): update docs, change name
0ebe7451e4e4c6e46c64e3782bd51a92ce594dca feat(store): add materiel.net (french), 35 3060ti references (#2287)
5b2fe88a83f17ff98ebe5c91806c7eb5c1dc3bdc chore(amazon-fr): update evga 3080 ftw3 link (#2286)
bf8f242f4b33796fcee91a863f0e086fcec29537 refactor(freemobile): remove local throw
6be2e0d491646d6e88f72f67c694874b11cda03a feat(notification): add support for Free mobile (#2285)
a25b7f19cc8e403ffd7856e631cde7e9df63f1cc refactor: simplify maxPrice logic
e006c48795b8fa104f0cee5584517a5f164d6512 chore: change verbiage to be more generic
9b02c36370aaf7fef20a90b2427273c2f7b30f00 feat: change max price behavior
b4121463ed7f25566cfa4ff508fa0c4d03237a9d ci: update stale labels
d03aea05a7783c9cc6a18c8baf0381050c105357 feat(pushover): ability to specify device (#2101)
ea095a364c250f805704c1189c0f3e697e09673d chore(alternate): update links and selectors (#2282)
c7e24e77d415819d27043d5814b2133b46549e57 chore(amd): add rx6800xt midnight black (#2321)
3c1419f170b1d00aa6ad08ad89752fc8f9134a93 chore: bump stale workflow to 60 days
cd6e1f614bcaf00bc468c32a1b5a98cb51c7730f fix: add missing tests in Dockerfile (#2323)
91a0ff5e1ba83efabe5ef45c6a3e17e1c564f16b chore(deps): bump dependencies, node 15.14.0
bbfa808b820ee2d16c65dfdcecce58645f4899d0 feat: color empty stores (#2312)
c2c8531ea26e2b75f50ed384f5128e5df54053e9 docs: update badges
113c9ce19f3ed1f140eec7cd555ec373f25d256f feat(series): support for rx6700xt (#2290)
ba35706cca84a425da8d5a5bd6b86dbecbadffa1 docs: update Xbox Series S and X price limit desc (#2098)
2ebbf55004da325c38c1eb12bcd5dc3264791604 chore(deps): bump node from 15.11.0-alpine3.13 to 15.12.0-alpine3.13 (#2185)
1714e89af7964d06b01120fb75595ef8ec8177ba feat(store): Add AO and BT (UK) (#2089)
983cc525164744e4d3bff1475d12b5563d23200e chore(amazon-uk): add 3060 cards (#2229)
a7a5e6954b3568b40d360982d5f11287c1b7f440 chore(deps-dev): bump @types/node from 14.14.35 to 14.14.36 (#2233)
2a1734a71d20e9a65c04edcd4318eaa3b5872a7d chore(deps): bump dependencies
83ce922761f85a13424ed43a1961668ffc870485 feat(pushover): add support for sound (#2203)
0a9ce0dd4c7681cba7681d570db44f65359d83cf chore(store): add Nvidia RTX Cards (#2204)
b2ce7157d9631d6edd45debcb9ab85bc79241133 chore(walmart): update 5900x url (#2184)
b0efc837bc554f5c8635e125b67775d8b1f8f6b3 fix(playstation): fix inStock selector (#2187)
54883f0ad7cf86052035d88ed2ffbc612c439725 chore(deps): bump open from 8.0.2 to 8.0.3 (#2186)
4bd048ff7a496e82e66293a9706581b8d10af6b3 chore: add APNS to dotenv-example (#2174)
99a274dbea088a6bcb053b1c1bf7812d0796442a fix(apns): use when specified
10f7e0cf5d8f633aa0195a1f9403965b4347e017 chore: bump dependencies
2775124d80f86466dbc0c3bcb227f489f35aab62 chore(deps-dev): bump webpack from 5.24.2 to 5.26.3 (#2164)
a575143b244525ff87579b689ed90bf6ae251847 chore(deps-dev): bump mocha from 8.3.0 to 8.3.2 (#2167)
4cdda9890cb30443826e6df02b99d41dbbea026d chore(deps-dev): bump @types/sinon from 9.0.10 to 9.0.11 (#2166)
53e7a8d692b76155699b371e926a2b9687d4a081 chore: remove unused dependencies
b516e268428219de70358821683e2dd74df59943 chore(deps-dev): bump typescript from 4.2.2 to 4.2.3 (#2092)
00e4ddce162877a4a08747955859508f5af0a802 chore(deps): bump open from 7.4.2 to 8.0.2 (#2078)
eeab4b112a6482d4fd381014e44fd0d6e071c494 chore(deps): bump twitch-chat-client from 4.4.13 to 4.5.1 (#2127)
0ff8158511f4f90dc26407fffa22ab0a4ad3aa0b revert: replace euroFormat with regex (#2131)
91c4f12308905f4e6d65341ba54ae814c8ad94d6 revert: parsing prices with thousands separator and European format (#2153)
6b78b2ee4445ad2b868b6f848730a474ca12d657 feat(notifications): add apple push notifications (#2077)
d46c6366b70ca668437bd2aab87c3c529f4c569b chore(microcenter): add 3090 cards
7fcce586bf03a573a1bba143c7d56e046418d31f chore(amd): update rx6900xt urls (#2094)
245d110d0a52df8b2d05f020aaaa1bfa8be7cab4 chore(asus): update urls (#2155)
61f8e3a9a03136682035881fce6d14356717d1ac fix: parsing prices with thousands separator and European format (#2153)
795c7230ad4632c81b17f5c2aeb3b0e88b2b04e3 refactor(store): replace euroFormat with regex (#2131)
afe7a709009f43fba1cef22aa6c136976507c7bc chore(nodejs): bump version (#2088)
f65aab87dade27c9f271ca816f83fe5a57ff7da6 docs: update node references
c2a22716cce0de6cbdbfdf7ebdceb97844a22e0f ci: fix docker release	   ==> TEST: OK

thanks!

[jef]

I reverted that puppeteer change, which may have done some damage. Do you mind testing with this commit?

5e9b3db chore: revert puppeteer, run linter

And to clarify, it works with the commit just prior to the one you posted?

4e3eade chore: bump node 16.5.0

[richie256]

Hello @jef ,

And to clarify, it works with the commit just prior to the one you posted?

Exactly, it work with the commit just prior the one posted:

• I got net::ERR_FAILED with (1d08d4c12292e1b98ad952fa4f88c1965fa7bd6a chore: update dependencies) or any newer commit.
• Works fine with commit just before: 4e3eadea083071eba563dffc70d749db334dce69 chore: bump node 16.5.0

Same failure net::ERR_FAILED with the docker nightly build, as of now.

docker pull ghcr.io/jef/streetmerchant:nightly
docker run -it --rm --env-file ./dotenv -v $(pwd)/global.proxies:/app/global.proxies ghcr.io/jef/streetmerchant:nightly

[richie256]

@jef

Do you mind testing with this commit?

5e9b3db chore: revert puppeteer, run linter

This fails net::ERR_FAILED as well using proxies.

[Fmstrat]

If it's of help, I had to roll back to https://github.com/jef/streetmerchant/pkgs/container/streetmerchant/3431703 to get things working.

[Tiippex]

Oh, I came across the same issue. It is present in 0906f64 (nightly).

I'm happy to help :) Anything we can do here to help you solve the issue?