Native packages rebuilt every time #932
Comments
|
Can't reproduce this in Mac OSX (10.11.6) so might be a Ubuntu specific issue? |
|
I could repro on Windows 10, but only once. The second time I ran "yarn", it didn't rebuild the native libraries. Strange. |
|
I was playing with it some more and came up with a few more details:
I'm not at all familiar with the internals but it seems to me that the order in which the packages are compiled matter and they are simply not being sorted when first installed (and they are sorted when later invoking |
|
Thank you for investigating! That sounds like a good lead. The hash is written here: yarn/src/cli/commands/install.js Lines 581 to 583 in f04b157 .yarn-integrity is based off the lockfile.
|
I suspected that but what threw me off was the fact that the lock file doesn't change, it's always the same. It's just the hash in .yarn-integrity that changes. |
|
I have the same issue: I use bcrypt too. Every time when I install some new modules or upgrade exist ones, I have to run macOS 10.12 && node v7.0.0 && yarn v0.16.1 |
|
@jiripospisil Thanks, It's okay now after upgrade to yarn v0.17.4. |
|
I'm still seeing this, or something very similar, on 0.18.1. Incidentally, it's also leveldown that keeps getting rebuilt repeatedly. Using leftpad as a package with no dependencies (and that is not depended on by leveldown) for demonstrative purposes, repro steps are as follows: mkdir test && cd test
echo '{ "name": "test", "version": "1.0.0" }' > package.json
yarn add leveldown
yarn add leftpad
yarn remove leftpadMy output when I run this follows. Note that removing leftpad takes almost 40 seconds, the majority of which is rebuilding leveldown. This happens consistently, with and without leveldown or leftpad in the Yarn cache, though only during Versions: Node v7.3.0 |
|
Please reopen as this still happens. Node v6.9.4 |
|
@seansfkelley I followed your steps with the latest version and it worked. Could you try again? @nexxado Could you please add a few reproduction steps? I tried a few combinations but it worked. Node 7.3.0 |
|
@jiripospisil I have no reproduction steps to provide, simply installing an additional package causes yarn to link and rebuild everything. Here's the output of adding one package (lock file already exists): Linking dependencies: Rebuilding: |
|
@jiripospisil I am also still seeing this, but during my repro I got tripped up because it looks like leveldown (or a dependency thereof) may have started shipping an OS X-compatible binary, so the install times dropped suspiciously from 50s to 3s. If you're on OS X and you specifically |
|
I have an indirect dependency on However, it only rebuilds every time when there is a change to |
|
I'm also seeing this issue, though I could not reproduce it with the steps above. However I can reproduce it with these steps: which builds leveldown. Then if I add another package: then again it builds leveldown. It doesn't seem to matter what package I add, it always rebuilds leveldown. I'm using Yarn v0.21.3, Windows 10, and Node v7.7.1 |
|
I'm seeing this as well. I'm using BuckleScript (bs-platform).... |
|
I'm encountering this issue, too, with Tested in yarn v0.21.3, Node 7.0.0, under Windows 10 and Ubuntu Linux 16.04. Here is {
"devDependencies": {
"auto-reload-brunch": "^2.7.1",
"babel-brunch": "^6.1.1",
"babel-preset-env": "^1.2.1",
"brunch": "^2.10.8",
"chai": "^3.5.0",
"clean-css-brunch": "^2.10.0",
"css-brunch": "^2.10.0",
"express-mysql-session": "^1.2.0",
"javascript-brunch": "^2.10.0",
"jquery": "^3.1.1",
"less-brunch": "^2.10.0",
"mocha": "^3.2.0",
"nodemon": "^1.11.0",
"npm-run-all": "^4.0.2",
"postcss-brunch": "^2.0.5",
"postcss-cssnext": "^2.9.0",
"postcss-font-magician": "^1.6.1",
"uglify-js-brunch": "^2.10.0"
},
"dependencies": {
"body-parser": "^1.17.1",
"connect-redis": "^3.2.0",
"cookie-parser": "^1.4.3",
"debug": "^2.6.1",
"express": "^4.15.2",
"express-session": "^1.15.1",
"jstransformer-marked": "^1.0.2",
"md5": "^2.2.1",
"morgan": "^1.8.1",
"multer": "^1.3.0",
"node-mysql": "^0.4.2",
"passport": "^0.3.2",
"passport-local": "^1.0.0",
"pug": "^2.0.0-beta11",
"serve-favicon": "^2.4.1",
"sharp": "^0.17.2"
}
} |
|
also seeing this with |
|
Also experiencing this with |
|
I have this with couchbase as well edit: seems to be fixed in 0.23.2 |
|
It still happens to me on 0.23.2 ( OS: Windows 10 |
|
To add some more color, my perception of this happening on Certainly convenient to reuse the install logic in remove to generate the lockfile, but it'd be nice if it didn't come with all the baggage of a forced install :) |
|
For me this started happening again when I upgraded to 0.23.x. I reverted to 0.21.3 and it no longer builds each time. Also this led to this issue where my IP was blocked by unicode.org after upgrading a few packages in a row dodo/node-unicodetable#16 |
|
While 0.21.3 does not rebuild all the packages on add a new package, it rebuilds all the packages on remove. And it seems yarn doesn't regard it a failure if rebuilding fails. |
|
For me, downgrading to |
|
Seeing this on macOS with Yarn 0.23.4. Rebuilds $ yarn add gulp-rename gulp-notify gulp-sass -D 1 ↵
yarn add v0.23.4
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
warning 7zip-bin-linux@1.0.3: The platform "darwin" is incompatible with this module.
info "7zip-bin-linux@1.0.3" is an optional dependency and failed compatibility check. Excluding it from installation.
warning 7zip-bin-win@2.0.2: The platform "darwin" is incompatible with this module.
info "7zip-bin-win@2.0.2" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] 🔗 Linking dependencies...
[4/4] 📃 Building fresh packages...
success Saved lockfile.
success Saved 42 new dependencies.
├─ amdefine@1.0.1
├─ async-foreach@0.1.3
├─ cross-spawn@3.0.1
├─ gaze@1.1.2
├─ glob@7.1.1
├─ globule@1.1.0
├─ growly@1.3.0
├─ gulp-notify@3.0.0
├─ gulp-rename@1.2.2
├─ gulp-sass@3.1.0
├─ gulp-util@3.0.8
├─ in-publish@2.0.0
├─ is@3.2.1
├─ js-base64@2.1.9
├─ lodash._reinterpolate@3.0.0
├─ lodash.assign@4.2.0
├─ lodash.clonedeep@4.5.0
├─ lodash.mergewith@4.6.0
├─ lodash.template@4.4.0
├─ lodash.templatesettings@4.1.0
├─ meow@3.7.0
├─ minimatch@3.0.3
├─ nan@2.4.0
├─ node-gyp@3.6.1
├─ node-notifier@5.1.2
├─ node-sass@4.5.2
├─ node.extend@1.1.6
├─ npmlog@4.0.2
├─ osenv@0.1.4
├─ readable-stream@2.2.9
├─ request@2.81.0
├─ sass-graph@2.2.3
├─ scss-tokenizer@0.2.3
├─ semver@5.3.0
├─ shellwords@0.1.0
├─ source-map@0.5.6
├─ stdout-stream@1.4.0
├─ string-width@1.0.2
├─ tar@2.2.1
├─ through2@2.0.3
├─ vinyl-sourcemaps-apply@0.2.1
└─ which@1.2.14
$ install-app-deps
Rebuilding native production dependencies for darwin:x64
Rebuilding native dependency sqlite3
✨ Done in 56.61s. |
|
@snowyu did you delete yarn.lock, node_modules and |
|
It's a new project in temp. Yes I can see the > yarn init -y
> yarn add node-sass
yarn add v1.6.0-20180327.1507
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...Downloading binary from https://github.com/sass/node-sass/releases/download/v4.8.3/linux-x64-57_binding.node
success Saved lockfile.
success Saved 152 new dependencies.
Done in 11.98s.
> yarn add node-sass
yarn add v1.6.0-20180327.1507
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...Downloading binary from https://github.com/sass/node-sass/releases/download/v4.8.3/linux-x64-57_binding.node
success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ node-sass@4.8.3
info All dependencies
└─ node-sass@4.8.3
Done in 13.45s.
``
|
|
OK, i did simple git repo to reproduce this bug. https://github.com/vlmonk/yarn-bug-test yarn perform unnecessary rebuild git clone https://github.com/vlmonk/yarn-bug-test
cd yarn-bug-test
yarn
[ ... building binary package here ... ]
yarn add left-pad
[ ... rebuilding binary packages here ... ]I can reproduce this in both host OSX system and in docker container with latest NPM works correctly in this case: git clone https://github.com/vlmonk/yarn-bug-test
cd yarn-bug-test
npm i
[ ... building binary package here ... ]
npm i left-pad
[ ... don't rebuild binary packages here ... ]My yarn version |
|
@vlmonk does this still happen if you clone https://github.com/rally25rs/yarn from @rally25rs and use the code in #5470 (branch |
|
@karlhorky yes, yarn still rebuild |
|
The Yarn should handle this situation better: It should see that those files changed during the build step and it should accept those changed files as the "correct" files, not treat them as a reason for a reinstall. I checked this with additional logging (https://github.com/sth/yarn/tree/trace-rebuild). On first install it shows: The package file https://registry.npmjs.org/ttf2woff2/-/ttf2woff2-2.0.3.tgz indeed contains those files. |
* fix(linking): Dont integrity check build artifacts If an install script modified a file that existed in the package, then yarn would view the file as stale on every command, mark the package as "fresh", and re-run the install scripts. With this change, known build artifacts from the integrity file are checked, and if this package file is a build atrifact, it's size and timestamp are not checked. #932 * WIP log message for debugging on CI * WIP adding logging for debugging CI * WIP adding logging for debugging CI * remove debugging console statements. * fix file timestamps for node 8 copyFile. fixes issue on linux. * change file mode to what works on windows * add file timestamp comparison and skip if correction not needed * fix lint errors * add error handling to fixTimes for some edge cases * pass undefined instead of 0 as a file id * refactor fs code, move OS specific edge cases to fs-normalized.js * empty commit to rerun CI * fix logic error when closing file, and fix Flow error
|
We see this with OS X as well, adding any package with We are using Please let me know if there are any workarounds, or when it will be fixed. |
This issue was fixed in 1.6.0 which is out quite recently. |
|
I still see this with 1.6. Since moving from Steps to reproduce:
|
|
@grantila can you provide a complete |
|
this is still happening on 1.6.0 you can repro this using this #932 (comment) |
|
I have a simple project and I'm seeing this too. Adding or removing a package seems to trigger a complete rebuild of at least one package every time. After these were installed, I added the EDIT: using Node 8.11.1 and yarn 1.6.0 on Debian Stretch. |
|
@arcanis @rally25rs pleaase reopen the issue, multiple reports of this still happening, even with the merged fix. |
|
I think this is more of a @rally25rs issue :) |
|
@grantila an Everyone else: In #5680 I point out that this still happens if a package deletes it's own files (why oh why do they do these things 😿) and yarn doesn't track that anywhere (we track what files are created or modified), so it just thinks the package is missing files and rebuilds it. I suppose we can reopen this, but this has been fixed for most packages. If anyone wants to add "me too" to this, please either provide your package.json, or mention specifically which package is continually rebuilding, since you may have some dependencies that do rebuild and some that do not. Anyone is welcome to make a PR for this! (see my debugging comments in #5680 ) |
|
Sorry for adding more noise, but I'd like to suggest locking this issue and pointing to a new one with this latest information at the top. I think the issue here has shifted quite a bit and is at least partially resolved. With all the posts here it's difficult for someone new to get up-to-date with what has been fixed and what is still an issue. |
|
I agree with @james-rabbit |
|
Yep, you're right. I'm going to lock this one so that @rally25rs' answer stays visible. If you have an issue with native packages:
|
and others
Do you want to request a feature or report a bug?
Bug.
What is the current behavior?
It seems that all native packages are rebuilt every time yarn is asked to either add a new package or just install the currently locked.
If the current behavior is a bug, please provide the steps to reproduce.
The same happens when adding a completely unrelated packages which, as far as I can tell, cannot affect the native packages in any way.
What is the expected behavior?
Native packages should not be rebuilt if there's no reason to do that. Note that #248 seems pretty similar.
Please mention your node.js, yarn and operating system version.
Node.js 6.7.0
Yarn 0.15.1
Ubuntu 12.04
The text was updated successfully, but these errors were encountered: