Error: watch ENOSPC on 0.15 (running inside Docker) #739

siboulet · 2015-07-29T18:31:07Z

I can no longer use forever inside Docker container after updating to 0.15. I have plenty of space left, not sure why I'm getting ENOSPC.

Rolling back to 0.14.2 works.

Jul 29 14:23:34 coreos docker[12137]: events.js:85
Jul 29 14:23:34 coreos docker[12137]: throw er; // Unhandled 'error' event
Jul 29 14:23:34 coreos docker[12137]: ^
Jul 29 14:23:34 coreos docker[12137]: Error: watch ENOSPC
Jul 29 14:23:34 coreos docker[12137]: at exports._errnoException (util.js:746:11)
Jul 29 14:23:34 coreos docker[12137]: at FSWatcher.start (fs.js:1172:11)
Jul 29 14:23:34 coreos docker[12137]: at Object.fs.watch (fs.js:1198:11)
Jul 29 14:23:34 coreos docker[12137]: at createFsWatchInstance (/usr/src/app/node_modules/forever/node_modules/forever-monitor/node_modules/chokidar/lib/nodefs-handler.js:37:15)
Jul 29 14:23:34 coreos docker[12137]: at setFsWatchListener (/usr/src/app/node_modules/forever/node_modules/forever-monitor/node_modules/chokidar/lib/nodefs-handler.js:80:15)
Jul 29 14:23:34 coreos docker[12137]: at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/src/app/node_modules/forever/node_modules/forever-monitor/node_modules/chokidar/lib/nodefs-handler.js:228:14)
Jul 29 14:23:34 coreos docker[12137]: at EventEmitter.NodeFsHandler._handleFile (/usr/src/app/node_modules/forever/node_modules/forever-monitor/node_modules/chokidar/lib/nodefs-handler.js:255:21)
Jul 29 14:23:34 coreos docker[12137]: at EventEmitter.<anonymous> (/usr/src/app/node_modules/forever/node_modules/forever-monitor/node_modules/chokidar/lib/nodefs-handler.js:468:21)
Jul 29 14:23:34 coreos docker[12137]: at FSReqWrap.oncomplete (fs.js:95:15)

The text was updated successfully, but these errors were encountered:

indexzero · 2015-07-29T18:44:03Z

This feels like a chokidar issue. I'm going to reach out there and see what's up.

indexzero · 2015-07-29T19:07:38Z

Also @siboulet are you sure the container has enough space as well?

siboulet · 2015-07-29T19:13:58Z

@indexzero thanks for the follow up. Absolutely sure I have sufficient disk space. I initially though it was a space issue but it really isn't. Simply changing the forever version and restarting (the same container) works.

indexzero · 2015-07-29T19:14:45Z

Can you show me a df -h from inside the container? As I understand it the disk allocated to the container is different than the disk on the machine itself. Trying to figure this out accordingly.

siboulet · 2015-07-29T19:16:58Z

@indexzero I checked multiple time already, trust me, it's not a disk space issue. Also checked df -i for the inodes. I have plenty of space left (several GB). I don't have time to set a test for this right now, but I'm using the official Node image here: https://registry.hub.docker.com/_/node/

indexzero · 2015-07-29T19:20:27Z

Ok. Seems like it could be a ulimit issue: paulmillr/chokidar#322 (comment)

210ToNy210 · 2015-08-03T23:04:53Z

https://github.com/210ToNy210/forever

brycereynolds · 2015-08-12T00:45:27Z

@210ToNy210 does that version contain a patch? It looks like that has been merged with main right? Anyhow I'm using the latest forever (0.15.1) and am also running into the same error as OP. I also confirm enough space is available. In my case though forever works if I am not watching.

This is my full command which works fine locally but inside docker blows up.

forever -w --watchDirectory $PWD -c 'node --harmony' app/server.development.js --seneca.log=plugin:collection --seneca.print.tree --seneca.log=level:info,level:fatal --seneca.log=plugin:transport,regex:cmd.listen

Without watch forever works as expected. With watch after about 10 seconds it dies.

brycereynolds · 2015-08-12T00:47:43Z

And somehow I didn't follow that link by @indexzero before posting. Looks like watch has already been picked up on...

pappacena · 2015-08-12T20:24:32Z

+1

FelikZ · 2015-09-07T14:10:28Z

Rollback to @0.14.2 helps

poldridge · 2015-09-20T03:43:06Z

+1

abenevaut · 2016-10-22T19:36:03Z

I got an error, maybe related to this one, when i use the watcher :

events.js:154
      throw er; // Unhandled 'error' event
      ^

Error: watch /project/current/node_modules/nyc/node_modules/lodash/fp/split.js ENOSPC
    at exports._errnoException (util.js:893:11)
    at FSWatcher.start (fs.js:1313:19)
    at Object.fs.watch (fs.js:1341:11)
    at createFsWatchInstance (/usr/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:37:15)
    at setFsWatchListener (/usr/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:80:15)
    at FSWatcher.NodeFsHandler._watchWithNodeFs (/usr/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:228:14)
    at FSWatcher.NodeFsHandler._handleFile (/usr/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:255:21)
    at FSWatcher.<anonymous> (/usr/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:473:21)
    at FSReqWrap.oncomplete (fs.js:82:15)

DeekshaPandit · 2017-01-08T17:33:33Z

@42antoine
Did you find any fix for this?
I am getting the same error when compiling Angular2 Application.

Jazzepi · 2018-03-02T00:23:19Z

Has anyone here tried upping the number of inotify watchers? You have to do it inside of the host, and then the docker container will have the same number. You can't modify this value from inside the docker container for security reasons AFAIK.

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

In jestjs/jest#3254 @maraisr did some research for another problem.

There's also a great post here if you want to know how to see your current inotify watcher value, and see how many are being used at any given time.

https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached

chpio · 2019-07-17T10:49:40Z

im getting Error: EACCES: permission denied, watch '/proc/kmsg'

forever --watch /app/server.js

warn:    --minUptime not set. Defaulting to: 1000ms
warn:    --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
error: Could not read .foreverignore file.
error: ENOENT: no such file or directory, open '/.foreverignore'
error: restarting script because unlinkDir changed
error: Forever detected script was killed by signal: SIGKILL
error: Script restart attempt #1
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: EACCES: permission denied, watch '/proc/kmsg'
    at FSWatcher.start (internal/fs/watchers.js:165:26)
    at Object.watch (fs.js:1258:11)
    at createFsWatchInstance (/usr/local/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:37:15)
    at setFsWatchListener (/usr/local/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:80:15)
    at FSWatcher.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:228:14)
    at FSWatcher.NodeFsHandler._handleFile (/usr/local/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:255:21)
    at FSWatcher.<anonymous> (/usr/local/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:473:21)
    at FSReqWrap.oncomplete (fs.js:154:5)
Emitted 'error' event at:
    at FSWatcher._handleError (/usr/local/lib/node_modules/forever/node_modules/chokidar/index.js:257:10)
    at createFsWatchInstance (/usr/local/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:39:5)
    at setFsWatchListener (/usr/local/lib/node_modules/forever/node_modules/chokidar/lib/nodefs-handler.js:80:15)
    [... lines matching original stack trace ...]
    at FSReqWrap.oncomplete (fs.js:154:5)

and also some unintended behavior restarting script because unlinkDir changed

chpio · 2019-07-17T10:54:47Z

WTF... it's watching everything, even without --watchDirectory, are you kidding?

indexzero mentioned this issue Jul 29, 2015

ENOSPC in docker containers? paulmillr/chokidar#322

Closed

gregorskii mentioned this issue Nov 6, 2015

chokidar requirement from nunjucks requests fs.watch access alaimo/kama#2

Open

clevernyyyy mentioned this issue Apr 19, 2016

ember: fix ember serve issues reverse0x90/edCTF#3

Open

gattazr mentioned this issue Feb 8, 2017

Sails container cannot use forever --watch Packebian/Backend-Controller-env#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: watch ENOSPC on 0.15 (running inside Docker) #739

Error: watch ENOSPC on 0.15 (running inside Docker) #739

siboulet commented Jul 29, 2015

indexzero commented Jul 29, 2015

indexzero commented Jul 29, 2015

siboulet commented Jul 29, 2015

indexzero commented Jul 29, 2015

siboulet commented Jul 29, 2015

indexzero commented Jul 29, 2015

210ToNy210 commented Aug 3, 2015

brycereynolds commented Aug 12, 2015

brycereynolds commented Aug 12, 2015

pappacena commented Aug 12, 2015

FelikZ commented Sep 7, 2015

poldridge commented Sep 20, 2015

abenevaut commented Oct 22, 2016 •

edited

DeekshaPandit commented Jan 8, 2017

Jazzepi commented Mar 2, 2018

chpio commented Jul 17, 2019

chpio commented Jul 17, 2019 •

edited

Error: watch ENOSPC on 0.15 (running inside Docker) #739

Error: watch ENOSPC on 0.15 (running inside Docker) #739

Comments

siboulet commented Jul 29, 2015

indexzero commented Jul 29, 2015

indexzero commented Jul 29, 2015

siboulet commented Jul 29, 2015

indexzero commented Jul 29, 2015

siboulet commented Jul 29, 2015

indexzero commented Jul 29, 2015

210ToNy210 commented Aug 3, 2015

brycereynolds commented Aug 12, 2015

brycereynolds commented Aug 12, 2015

pappacena commented Aug 12, 2015

FelikZ commented Sep 7, 2015

poldridge commented Sep 20, 2015

abenevaut commented Oct 22, 2016 • edited

DeekshaPandit commented Jan 8, 2017

Jazzepi commented Mar 2, 2018

chpio commented Jul 17, 2019

chpio commented Jul 17, 2019 • edited

abenevaut commented Oct 22, 2016 •

edited

chpio commented Jul 17, 2019 •

edited