Composer very slow to read & write cache files #6263
Comments
|
Looks like an environment issue, not Composer related. Nothing I can further add to this without knowing more about your system setup (operating system, partitions, usage of VM yes/no, any symlinks in your path or not, encrypted home directory or not, etc). |
|
For the record, I just tried your Note the 5 seconds, while that's fast on the other end of the scale, your 270 seconds are definitely a local issue. Also: are you running the latest version of Composer, on PHP 7.x? You should realize that PHP 5.6 with Xdebug on an old Composer version may on its own already be 5+ times slower than a recent build that disables Xdebug, on PHP 7.1 which is frequently 2~3 times faster than PHP 5.6 on the kind of operations that happen during cache unserialization. |
|
@curry684 thanks for the comparison. I am running Composer version @alcohol sorry, I should have thought to add more info. This is on:
My I do have symlinks in my home directory, but they point to a directory in my path. The (maybe?) relevant one: I did try a separate/fresh install of Composer from source following the directions in CONTRIBUTING.md, but that produced substantially identical times. Any other suggestions on how to debug further? |
|
Does FileVault not encrypt your home directory by default? |
|
I believe it does, but was under the impression that while logged in, it doesn't affect performance. I'm turning it off now to test. Edit: some quick searching reveals FileVault probably isn't the culprit, but I'll test anyway. |
|
You could also try moving the repository to a directory that is outside of any existing path, e.g. |
|
Do you mean the project I'm working on, or the Composer install? |
|
The project you are working on. You can also try creating |
|
@tnorthcutt Any chance you're running the example application in a container service? |
|
@davidjeddy Do you mean something like Docker? If so, no. |
|
Ok, new info to report. I:
Output from 4: https://gist.github.com/tnorthcutt/6b88fb29713baf9d425b626afafb73aa (528.28 seconds) Am I correct that this seems very slow? |
|
Now it seems a lot of the slowness is in downloading files from packagist; others are reporting issues as well. Anecdata: If I If I I've confirmed similar ratios with people in other geographic locations. What seems odd to me is that my connection to Packagist is an order of magnitude slower than to Github, and it's the same case for others - but theirs are on the order of ~900KB/s from Packagist and ~3MB/s from Github. So it's not as if I'm maxing out the possible connection speed to Packagist right now. |
|
Yup it seems like we have some bandwidth crunch at peak time when the US starts working. Working on improving things. Will get back to you later. |
|
OK we'll see how things progress, but we have an mirror live now in North America. DNS propagation should kick in slowly, so I guess in a few hours once US is at work we can assess the situation a bit more in details. Please report any weirdness. |
|
I'm not sure why, but sometimes I'm completely baffled as to why file reads & writes are taking so long. I'd love to be able to debug this more, but I'm not sure how at the moment. Any pointers would be very much appreciated. |
|
@tnorthcutt I see you're using Loading plugin Hirak\Prestissimo\Plugin in there, can you profile the same without the plugin? Also can you tell us what IP your DNS resolves packagist.org to and where you're located? |
|
One question is whether it goes better today than yesterday, but given you seem to be in Australia I guess it's not gonna help as much as it will people in the US and around. |
|
I'm in Oklahoma, in the US.
|
|
@tnorthcutt ok you'll hopefully get better results once your dns cache is reloaded, that's the french packagist.org you're connecting to. |
|
You can try to hardcode packagist.org to 144.217.203.53 in your hosts file to try it out now, but please make sure to remove it again later as the IP may change without warning. |
|
Got it, I'll give it a shot hardcoded. Any ideas on the super slow file reads & writes, though? An update is taking nearly 10 minutes without any downloads 😕 |
|
@tnorthcutt like i said, please try without the plugin, since I'm not sure it's accurately reporting what is taking long with the downloads happening in parallel. |
|
@naderman oh sorry, I meant to mention that I'm trying it without that - it's just taking a while 😬 |
|
The slow cache reads/writes, most likely that's just an impression you have from the output, because it doesn't output when the CPU is crunching stuff. |
|
@Seldaek Got it. Does that seem like a normal amount of time to you? |
|
No 10min is definitely on the long side of things. I'm investigating other factors than network (which was definitely an issue, and I hope things will improve, but it may not be the only cause for you). |
|
@naderman here's a re-run without Prestissimo: https://gist.github.com/tnorthcutt/2f41f421198b88329c64a52d0730a8b1 Total time 138.15s so that's a huge improvement. I also re-ran a traceroute right after and I'm still hitting |
|
@tnorthcutt TTL on that DNS record was a day, so may take a few hours |
|
I just flushed my dns and now I'm getting Doesn't seem to be drastically improving download speeds, though (currently running Besides the slow network speeds, is there some way I can further diagnose my local-only issues? I'm not sure how to proceed since I'm on a fresh install of Composer, new |
|
@tnorthcutt I don't see any local-only issues per-se in that log file anymore? Can you send another traceroute? |
|
|
You're right, no local-only issues on the latest run: https://gist.github.com/tnorthcutt/538b77aa8bde471114686f074762a012 I may lack a fundamental understanding of how Composer works, but sometimes it downloads many files (as on that run) and other times it downloads almost nothing. I think the pattern is that on the runs when it doesn't download much, the local reads/writes are very slow. I can't say that for sure though. |
|
@tnorthcutt did some cleanup on the illuminate stuff.. won't go too much into details here but anyway that should speed things up further. |
|
Seems everything is going well so far so closing this https://twitter.com/packagist/status/842441015839608833 |
|
@Seldaek thank you! Mirror is much, much faster. |
|
For me it's always very slow, I tried to search everywhere without much success. I can't pinpoint what's the culprit, the download is most of the time slow, but sometimes it doesn't seem to be any network activity or CPU for that matters and composer just stays there. It's normally slow, today is super super slow.. maybe it's a temporary thing.
I am running 1.4.2 and PHP 7.0.18, no container, OSX 10.12.5, SSD drive, FileVault enabled. Any help here? |
|
This problem still exists for me, and none of the issues raised in this issue, does not resolve my problem. |
|
Yeah composer has been super super slow nowadays :( what is actually happened here? here's some log when I tried to install a single package:
EDIT: nevermind this was a connection issue. I changed my connection and it's working fine although still a bit slow. |
|
same problem in docker arquitecture (php container 1GB ram and 4 cores) |
|
@juanantoniomosq you on OSX? If so the file read/write performance is a Docker issue. |
|
@davidjeddy no, linux env. |
|
Ah, interesting; whats the output from a traceroute? |
|
I have been struggling with Composer's speed since a couple of weeks now. First I thought it had something to do with my environment, but today I've had speed issues on a completely different environment too. Running Composer using Docker or running it on Linux host makes no differences in terms of speed. If I run https://gist.github.com/pascal08/c9cdfc476b09a468989e9b95f4831671 |
|
My problem solved using latest kernel of CentOS (before use elrepo kernel). |
|
Slowness for me on docker also I am not sure how much memory composer should eat but 500MB sounds too mutch to install one dependancy. |
kernel host problem probably.. |
|
restarted pc and rebuild docker image now so maybe you were right |
|
@juanantoniomosquera @svycka thanks! you save my day! |
When I run
I get the following output:
https://gist.github.com/tnorthcutt/770ba18a3f3f182354d85b7c80a91fe1
The dependency resolution itself is very fast, and (in this case) no packages are installed, updated, or removed (because I previously ran
composer update). However, reading and writing the cache files seems extremely slow.Is there something I can do to speed this up, or to further debug?
My
composer.json:{ "name": "laravel/laravel", "description": "The Laravel Framework.", "keywords": [ "framework", "laravel" ], "license": "MIT", "type": "project", "require": { "php": ">=5.6.4", "laravel/framework": "5.3.*", "laravel/cashier": "~7.0", "laravel/spark": "*@dev", "yab/laracogs": "^1.9", "hashids/hashids": "^1.0", "barryvdh/laravel-cors": "^0.8.0", "barryvdh/laravel-debugbar": "^2.2", "barryvdh/laravel-ide-helper": "^2.1", "doctrine/dbal": "^2.5", "spatie/laravel-collection-macros": "^1.4", "league/csv": "^8.1", "guzzlehttp/guzzle": "^6.2" }, "require-dev": { "fzaninotto/faker": "~1.4", "mockery/mockery": "0.9.*", "phpunit/phpunit": "~5.0", "symfony/css-selector": "3.1.*", "symfony/dom-crawler": "3.1.*", "phpmd/phpmd": "@stable", "spatie/laravel-migrate-fresh": "^1.3" }, "autoload": { "classmap": [ "database" ], "psr-4": { "App\\": "app/" } }, "autoload-dev": { "classmap": [ "tests/TestCase.php" ] }, "scripts": { "post-root-package-install": [ "php -r \"copy('.env.example', '.env');\"" ], "post-create-project-cmd": [ "php artisan key:generate" ], "post-install-cmd": [ "Illuminate\\Foundation\\ComposerScripts::postInstall", "php artisan optimize" ], "post-update-cmd": [ "Illuminate\\Foundation\\ComposerScripts::postUpdate", "php artisan ide-helper:generate", "php artisan optimize" ] }, "config": { "preferred-install": "dist" }, "repositories": [ { "type": "path", "url": "./spark" } ] }Output of
composer diagnose:The text was updated successfully, but these errors were encountered: