Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race condition when watching dirs leads to missed files #1112

Open
jimparis opened this issue Jun 3, 2021 · 1 comment · May be fixed by #1228
Open

Race condition when watching dirs leads to missed files #1112

jimparis opened this issue Jun 3, 2021 · 1 comment · May be fixed by #1228
Labels
has test case linux

Comments

@jimparis
Copy link

jimparis commented Jun 3, 2021

Describe the bug

There is a basic race condition when setting up a watch on a directory, in nodefs-handler.js: First the directory contents are scanned, and then a fs.watch watcher is set up. Any file created between those two operations won't be seen, until there's another operation that causes the fs.watch to trigger and the directory is re-scanned.

Versions (please complete the following information):

  • 3.5.1
  • 10.21.0
  • Linux

To Reproduce:

First add this delay, which doesn't cause the race, just makes it a lot easier to trigger.

--- a/lib/nodefs-handler.js
+++ b/lib/nodefs-handler.js
@@ -556,6 +556,9 @@ async _handleDir(dir, stats, initialAdd, depth, target, wh, realpath) {
       if (this.fsw.closed) return;
     }
 
+    console.error("race!");
+    await new Promise(res => setTimeout(res, 5000));
+
     closer = this._watchWithNodeFs(dir, (dirPath, stats) => {
       // if current directory is removed, do nothing
       if (stats && stats.mtimeMs === 0) return;

Then use something like this

const chokidar = require('chokidar');
const fs = require('fs').promises;

function sleep(n) {
    return new Promise(res => setTimeout(res, n * 1000));
}

(async () => {
    try {
        const dir = await fs.mkdtemp('/tmp/foo-');

        const w = chokidar.watch(dir);
        w.on('all', (event, path) => {
            console.log("-- EVENT:", event, path);
        });
        w.on('ready', () => {
            console.log("-- EVENT: ready");
        });

        await sleep(2);
        console.log(`creating ${dir}/1`);
        await fs.writeFile(`${dir}/1`, "hi");

        await sleep(2);
        console.log(`creating ${dir}/2`);
        await fs.writeFile(`${dir}/2`, "hi");

        await sleep(2);
        console.log("nothing?");

        await sleep(10);
        console.log(`creating ${dir}/3`);
        await fs.writeFile(`${dir}/3`, "hi");

        await sleep(2);
        process.exit(0);
    } catch (e) {
        console.error(e);
    }
})();

Actual output

Events for adding files 1 and 2 don't happen until file 3 is created much later:

-- EVENT: addDir /tmp/foo-A2kUIi
race!
creating /tmp/foo-A2kUIi/1
creating /tmp/foo-A2kUIi/2
-- EVENT: ready
nothing?
creating /tmp/foo-A2kUIi/3
-- EVENT: add /tmp/foo-A2kUIi/1
-- EVENT: add /tmp/foo-A2kUIi/2
-- EVENT: add /tmp/foo-A2kUIi/3

Expected output

Events for adding files 1 and 2 should happen right before or after
ready:

-- EVENT: addDir /tmp/foo-A2kUIi
race!
creating /tmp/foo-A2kUIi/1
creating /tmp/foo-oh0HPl/2
-- EVENT: ready
-- EVENT: add /tmp/foo-A2kUIi/1
-- EVENT: add /tmp/foo-A2kUIi/2
nothing?
creating /tmp/foo-A2kUIi/3
-- EVENT: add /tmp/foo-A2kUIi/3
@dividedmind
Copy link

Huh, I just spent a couple of hours tracking down the same exact issue. Serves me right for not searching first :)

I think I have a fix, a test case is more difficult however due to this being a race. The only sure way to cause the problem is to add that timeout in chokidar code.

dividedmind added a commit to getappmap/chokidar that referenced this issue Jul 8, 2022
@dividedmind
fix: Rescan directory after setting up watch to prevent a race
a2ea88c 
Sometimes, when another process just happened to create a file
when we were setting up a watch, the new file wouldn't be noticed.

To prevent the event not being generated in this case, schedule
a directory rescan immediately after setting up a watch.

Fixes paulmillr#1112
@dividedmind dividedmind linked a pull request Jul 8, 2022 that will close this issue
Open
dividedmind added a commit to getappmap/chokidar that referenced this issue Jul 10, 2022
@dividedmind
fix: Rescan directory after setting up watch to prevent a race
4f54121 
Sometimes, when another process just happened to create a file
when we were setting up a watch, the new file wouldn't be noticed.

To prevent the event not being generated in this case, schedule
a directory rescan immediately after setting up a watch.

Fixes paulmillr#1112
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
has test case linux
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix for a race condition on Linux getappmap/chokidar
3 participants
@jimparis @paulmillr @dividedmind