Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't work well on a flaky connection #746

Closed
appsforartists opened this issue Oct 11, 2016 · 37 comments
Closed

Doesn't work well on a flaky connection #746

appsforartists opened this issue Oct 11, 2016 · 37 comments
Assignees
Labels

Comments

@appsforartists
Copy link

Do you want to request a feature or report a bug?
Not sure.

What is the current behavior?
I'm at an airport today, so I've been tethering on my phone and using in-flight WiFi. Every time I try to run yarn, I eventually get a socket hang up error on an arbitrary package. Presumably, it doesn't handle mobile connectivity very well. npm install works in these same scenarios.

If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior?

  • Be able to install a particular module without worrying about the rest of the installed modules.
  • Retry requests if one times out.

Please mention your node.js, yarn and operating system version.
yarn v0.15.1
node v6.2.1
macOS v10.11.6

@Daniel15
Copy link
Member

Somewhat similar to #627, in that we're not retrying in a case where we probably should.

@designorant
Copy link

designorant commented Oct 11, 2016

Interesting. I often work while traveling through areas with little or no coverage, and found using NPM almost impossible then. Yarn seemed very promising, and some quick tests showed it does behave just fine having most/all packages cached, but it does appear equally ineffective when starting from scratch indeed. It would be great if it was more proactive on flaky networks.


Context:

I currently work on a project with 803 packages in total.

On a 1mbps, 0% packets drop rate, 5ms delay connection, the install times with cleaned caches are as follows:

  • npm install => 8m 0s
  • yarn => 3m 38s

On a 1mbps, 10% packets drop rate, 500ms delay connection, the rather unscientific results are as follows:

  • npm install just hangs without any feedback at some point.
  • yarn trips up either with a socket hang up error, or just hangs.

--verbose was of no use in both cases.

@th3fallen
Copy link

can confirm yarn will trip up on a slow/unstable connection with socket hung up regardless of what you do

@deepu105
Copy link

I'm facing the same as well. On windows with a tethered mobile connection (not very fast but npm works)

yarn install v0.15.1
info No lockfile found.
[1/4] Resolving packages...
error https://registry.yarnpkg.com/progress: socket hang up
    at createHangUpError (_http_client.js:206:15)
    at TLSSocket.socketOnEnd (_http_client.js:298:23)
    at emitNone (events.js:85:20)
    at TLSSocket.emit (events.js:179:7)
    at endReadableNT (_stream_readable.js:906:12)
    at nextTickCallbackWith2Args (node.js:474:9)
    at process._tickCallback (node.js:388:17)
info Visit http://yarnpkg.com/en/docs/cli/install for documentation about this command.

@juangl
Copy link

juangl commented Oct 12, 2016

Same in Archlinux with my 1mbp connection. npm i works.

@gitnik
Copy link

gitnik commented Oct 12, 2016

This happens to me despite having a very stable connection

@jromero2k
Copy link

Windows 10, 512kbps... I can't finish any install. Lots of (random) socket hang ups. NPM works properly most of the time

Example:

yarn add v0.15.1
info No lockfile found.
[1/4] Resolving packages...
error https://registry.yarnpkg.com/babel-preset-es2015-node: socket hang up
at createHangUpError (_http_client.js:252:15)
at TLSSocket.socketOnEnd (_http_client.js:344:23)
at emitNone (events.js:91:20)
at TLSSocket.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)

@wx-chevalier
Copy link

i run into the same problem,in MacOS 10.11.5, node 5.3.0, npm 3.9.5:

 yarn add material-ui
yarn add v0.15.1
info No lockfile found.
warning webpack-react-redux-boilerplate@1.0.2: "dependencies" has dependency "react-komposer" with range "^1.11.0" that collides with a dependency in "devDependencies" of the same name with version "^1.13.1"
[1/4] 🔍  Resolving packages...
error socket hang up
    at TLSSocket.onHangUp (_tls_wrap.js:1090:19)
    at TLSSocket.g (events.js:261:16)
    at emitNone (events.js:73:20)
    at TLSSocket.emit (events.js:167:7)
    at endReadableNT (_stream_readable.js:906:12)
    at nextTickCallbackWith2Args (node.js:455:9)
    at process._tickCallback (node.js:369:17)
info Visit http://yarnpkg.com/en/docs/cli/add for documentation about this command.

@stanonyime
Copy link

Having the same issues on Windows 8, yarn 0.15.1, third-world connectivity ( flaky networks). npm works fine

@henri-hulski
Copy link

Here exactly the same on Debian Jessie with mobile connection (EDGE about 12kbps).

@henri-hulski
Copy link

If I try yarn upgrade I get something like this:

yarn upgrade v0.15.1
[1/4] Resolving packages...
error socket hang up
    at TLSSocket.onHangUp (_tls_wrap.js:1097:19)
    at TLSSocket.g (events.js:273:16)
    at emitNone (events.js:85:20)
    at TLSSocket.emit (events.js:179:7)
    at endReadableNT (_stream_readable.js:913:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
info Visit http://yarnpkg.com/en/docs/cli/upgrade for documentation about this command.

No chance to get it working.

npm works fine.

@yunxing yunxing added this to the Reason Support milestone Oct 20, 2016
@hustcer
Copy link

hustcer commented Oct 21, 2016

yarn v0.16.0
node v6.9.1
macOS v10.12

warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
...

however, npm works well.

@mmartinez
Copy link

I don't have a flaky connection and I'm having issues as well.

yarn v0.15.1
node v6.9.0
macOS v10.12

> $ yarn add dotenv                                                             [±dev ✓]
yarn add v0.15.1
[1/4] 🔍  Resolving packages...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
error getaddrinfo ENOTFOUND registry.yarnpkg.com registry.yarnpkg.com:443
    at errnoException (dns.js:28:10)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)
info Visit http://yarnpkg.com/en/docs/cli/add for documentation about this command.

@RGPaul
Copy link

RGPaul commented Oct 22, 2016

I have the same issue and just found out that Kaspersky Internet Security is blocking the connection from yarn. I don't have problems installing with npm.

@yunxing
Copy link
Contributor

yunxing commented Oct 27, 2016

I ran into this issue the other day on VPN, and can reliably repro it until I remove the cache. Maybe it also has something to do with cache.

@PokimLee
Copy link

PokimLee commented Nov 9, 2016

I got the same issue as @mmartinez

@gitnik
Copy link

gitnik commented Nov 9, 2016

Just investigated this again and for me it's also Kaspersky (on OSX) causing the issue

@jromero2k3
Copy link

jromero2k3 commented Nov 14, 2016

The same happens to me on 2 mbps line! Npm works fine...

@kvz
Copy link

kvz commented Dec 1, 2016

Just hit this but my connection is stable otherwise (I'm posting this without problems - yet consistent failures towards the registry). Also not a kaspersy or other antivirus/proxy user.

@Kitanga
Copy link

Kitanga commented Dec 5, 2016

@gaearon this is the issue I mentioned on twitter

@gaearon
Copy link
Contributor

gaearon commented Dec 7, 2016

@Kitanga Thanks. I’m compiling a list in facebook/create-react-app#1189, let me know if you had any other issues.

@caesarsol
Copy link

+1, I'm experiencing this as well when the packet drop os high.
Is it possible at least to install as many packages as possible before failing?
(Beside a reasonable number of retries)

@atran
Copy link

atran commented Feb 7, 2017

I fixed it with this local NPM registry DNS comment: npm/npm#6686 (comment)

@iraklisg
Copy link

iraklisg commented Mar 9, 2017

Same problem here using Debian Jessie....

warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "https://registry.yarnpkg.com/babel-core/-/babel-core-6.23.1.tgz: read ECONNRESET".
info If you think this is a bug, please open a bug report with the information provided in "/var/www/laravel/apptree2/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

npm works fine with the same setup

@Dakuan
Copy link

Dakuan commented Mar 16, 2017

happening on Heroku this am

@notrab
Copy link

notrab commented Mar 16, 2017

Having this problem also on Heroku right now. Deleting yarn.lock would work for those who really need to deploy.

@Dakuan
Copy link

Dakuan commented Mar 16, 2017

magically fixed itself - also had issues with a buildserver failing into install nodejs, glitch in a cdn somewhere?

@drdaeman
Copy link

drdaeman commented Jun 24, 2017

Using vanilla node:7 Docker base image, with Yarn 0.24.4 preinstalled. docker build has RUN yarn install as one of the steps.

I encounter a semi-persistent "There appears to be trouble with your network connection" issue. The actual exception is ESOCKETTIMEDOUT, like this (I've put console.log in onError):

Error:  { Error: ESOCKETTIMEDOUT
    at ClientRequest.<anonymous> (/usr/local/bin/yarn:108470:19)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at ClientRequest.emit (events.js:188:7)
    at TLSSocket.emitTimeout (_http_client.js:679:10)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at TLSSocket.emit (events.js:188:7)
    at TLSSocket.Socket._onTimeout (net.js:352:8)
    at ontimeout (timers.js:386:14) code: 'ESOCKETTIMEDOUT', connect: false }

I haven't figured out why exactly those fail and why things work fine without Docker. The container has no resource limits (and there are no kernel issues reported) and the network connectivity from the container looked pretty healthy to my probing. For example the latency is low and there is no packet loss neither on small, nor large packets:

--- registry.yarnpkg.com ping statistics ---
1000 packets transmitted, 1000 packets received, 0% packet loss
round-trip min/avg/max/stddev = 11.886/12.664/17.542/0.453 ms

What I've tried and what doesn't seem to help:

  1. Forcing isOffline to always return false. Probably irrelevant but I tried this because of this SO answer.

  2. Bumping up libuv threadpool size to a maximum with UV_THREADPOOL_SIZE=128. I hoped that would work, because failures could be related to DNS, but it hadn't helped.

  3. Passing agent: false and pool: {maxSockets: Infinity} to request module.

Still, "[2/4] Fetching packages..." step retries randomly. Sometimes it passes just fine, sometimes it retries once, sometimes it retries for very long. With --verbose it shows that it downloads most files OK, but sometimes hangs on "Performing "GET" request to ..."

(One of our team members got really unlucky yesterday, and docker build took an hour instead of the usual 2-3 minutes. Could be the case because he ran build on somewhat dated laptop.)

@saravvij
Copy link

saravvij commented Jul 7, 2017

I am keep getting this error. I have tried at my work, and home network, but no luck.

Environment: Windows 7, yarn 0.27.5, node v7.7.2

Trace: 
  Error: https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30000697.tgz: ESOCKETTIMEDOUT
      at ClientRequest.<anonymous> (C:\Program Files (x86)\Yarn\lib\yarn-cli.js:114873:19)
      at Object.onceWrapper (events.js:293:19)
      at emitNone (events.js:86:13)
      at ClientRequest.emit (events.js:188:7)
      at TLSSocket.emitTimeout (_http_client.js:679:10)
      at Object.onceWrapper (events.js:293:19)
      at emitNone (events.js:86:13)
      at TLSSocket.emit (events.js:188:7)
      at TLSSocket.Socket._onTimeout (net.js:354:8)
      at ontimeout (timers.js:380:14)

Any solution to this problem?

@temp3l
Copy link

temp3l commented Aug 9, 2017

adding registry.npmjs.org to /etc/hosts file helped me

in my case:
echo "151.101.12.162 registry.npmjs.org" >> /etc/hosts

@hponiang
Copy link

very lucking,thank you @stephanMettler
adding registry.npmjs.org to /etc/hosts file helped me

in my case:
echo "151.101.12.162 registry.npmjs.org" >> /etc/hosts

@BYK
Copy link
Member

BYK commented Aug 11, 2017

Is there anything you guys want yarn to do differently? May be add the option to increase retries or something?

Otherwise, I don't see anything actionable here since we cannot fix everyone's internet connection (not yet, anyways).

@dannylewis-sheffield
Copy link

OK I had the same issues when building create-react-app inside docker, never managed to install.

Backed out one step to the vagrant box, running on VirtualBox, installed node yarn etc and then again ran create-react-app. Same issue but it did get further, downloaded more packages, than under docker.

Changed the network settings on vagrant from NAT to Bridged:

Old:
config.vm.network :forwarded_port, guest: 80, host: 1026, auto_correct: true

New:
config.vm.network :public_network, :public_network => "wlan0"

You can switch back the configs once you have successfully done what you needed to.

Ran create-react-app again and it looked like it it would complete, despite the odd "warning There appears to be trouble with your network connection. Retrying..." Nope failed on ESOCKETTIMEDOUT, but just about there!

Ran it again, success!

Back to docker version, long story short it eventually successfully built. Good job I'm doing intermediate builds...

Networks can be flaky, some more than others.

I would like to see the following be able to be set in config:

download_max_retry - does what it says on the tin and defaults to 20
download_clear_failed_count_on_success - true or false, default false, if true would reset the failed download count to 0 on the next successful download, this would mean it would only fail to complete if it failed to download the last file in the queue (assuming downloads are parallel, some fail and are then put back in the queue)

Hope this helps and thanks for the tools!

@davidcalhoun
Copy link

davidcalhoun commented Sep 12, 2017

Getting a similar situation today. I'm running a yarn upgrade command and got this message multiple times:

warning There appears to be trouble with your network connection. Retrying...

Then got

error An unexpected error occurred: "https://registry.yarnpkg.com/material-ui: getaddrinfo ENOTFOUND registry.yarnpkg.com registry.yarnpkg.com:443".

info If you think this is a bug, please open a bug report with the information provided in "/Users/davidcalhoun/dev/curator/yarn-error.

These were caused by connection issues, but once they resolved themselves I found myself unable to run the command for a few minutes. Running the command from a new terminal window instantly worked.

@BYK
Copy link
Member

BYK commented Sep 12, 2017

These all look like DNS issues. I do not know why yarn saturates the DNS resolving though. I'll try to investigate this.

@BYK BYK self-assigned this Sep 12, 2017
BYK added a commit that referenced this issue Sep 13, 2017
**Summary**

Fixes #746. Unfortunately neither Node, nor many systems come with
built-in DNS caching so the many parallel requests that Yarn makes
sometimes overwhelm the DNS servers, and most of the time, for the
very same domain(s). Even worse, we pay the DNS look up cost for
each request, which is quite sad at best. This patch introduces
the `dnscache` module which intercepts all DNS look ups and answers
them from an in-memory cache when possible. This applies to the
built-in `http` and `https` modules, used by `request`.

**Test plan**

Exiting tests should pass, and hopefully be faster. Total number of
DNS look ups should decrease dramatically.
@BYK BYK closed this as completed in #4447 Sep 14, 2017
BYK added a commit that referenced this issue Sep 14, 2017
**Summary**

Fixes #746. Unfortunately, neither Node, nor many systems come with
built-in DNS caching so the many parallel requests that Yarn makes
sometimes overwhelm the DNS servers, and most of the time, for the
very same domain(s). Even worse, we pay the DNS look up cost for
each request, which is quite sad at best. This patch introduces
the `dnscache` module which intercepts all DNS look ups and answers
them from an in-memory cache when possible. This applies to the
built-in `http` and `https` modules, used by `request`.

**Test plan**

Existing tests should pass, and hopefully be faster. Total number of
DNS look ups should decrease dramatically.
joaolucasl pushed a commit to joaolucasl/yarn that referenced this issue Oct 27, 2017
**Summary**

Fixes yarnpkg#746. Unfortunately, neither Node, nor many systems come with
built-in DNS caching so the many parallel requests that Yarn makes
sometimes overwhelm the DNS servers, and most of the time, for the
very same domain(s). Even worse, we pay the DNS look up cost for
each request, which is quite sad at best. This patch introduces
the `dnscache` module which intercepts all DNS look ups and answers
them from an in-memory cache when possible. This applies to the
built-in `http` and `https` modules, used by `request`.

**Test plan**

Existing tests should pass, and hopefully be faster. Total number of
DNS look ups should decrease dramatically.
@MichelDiz
Copy link

MichelDiz commented Jan 2, 2018

Does look like a DNS issue. I have this issue using Rancher (that has complicated workarounds with DNS).

UPDATE: Solved putting the stacks to use network the "bridge" - For me it is a temporary solution, because my environment is for tests.

Inside the app that is inside the the Rancher environment

/usr/src/app # node -e 'const os = require("os"); const interfaces = os.networkInterfaces(); for (const interface in interfaces) {console.log(interface); const addrs = interfaces[interfac
e]; for (const addr of addrs) {console.log(addr.address)}}'
lo
127.0.0.1
eth0
10.42.222.77

cat hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.42.222.77    441206ad059d

Inside the app that is inside the the Rancher environment

/usr/src/app # ping yarn.org
PING yarn.org (184.168.221.96): 56 data bytes
64 bytes from 184.168.221.96: seq=0 ttl=61 time=222.452 ms
64 bytes from 184.168.221.96: seq=1 ttl=61 time=225.904 ms
64 bytes from 184.168.221.96: seq=2 ttl=61 time=223.484 ms
64 bytes from 184.168.221.96: seq=3 ttl=61 time=222.596 ms
64 bytes from 184.168.221.96: seq=4 ttl=61 time=222.305 ms
64 bytes from 184.168.221.96: seq=5 ttl=61 time=221.858 ms
64 bytes from 184.168.221.96: seq=6 ttl=61 time=222.100 ms
64 bytes from 184.168.221.96: seq=7 ttl=61 time=222.088 ms
64 bytes from 184.168.221.96: seq=8 ttl=61 time=221.959 ms

Inside the app that is inside the the Rancher environment

/usr/src/app # nslookup registry.npmjs.org
nslookup: can't resolve '(null)': Name does not resolve

Name:      registry.npmjs.org
Address 1: 151.101.176.162

Outside of Rancher (in docker machine)
captura de tela 2018-01-01 as 23 02 53

@margusha
Copy link

margusha commented Oct 5, 2018

Is there anything you guys want yarn to do differently? May be add the option to increase retries or something?

Otherwise, I don't see anything actionable here since we cannot fix everyone's internet connection (not yet, anyways).

100mbps down, 26mbps up current internet speed test. ping 7ms. doesn't seem like a network issue. but yet again, yarn install and yarn start fails with ETIMEDOUT and it has done it randomly for a past year. running npm install starts up the app in ~10 seconds. Guess it was my last time to use yarn.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests