Allow inspecting the sizes of configuration cache entries #28327
Comments
mgroth0
changed the title
|
Using Why is |
|
This feature request is in the backlog of the relevant team and is prioritized by them. |
This tool might help in the meantime: https://github.com/gradle/gcc2speedscope |
|
@bamboo while it doesn't seem like a perfect solution for inspecting the size of the configuration cache entries, the tool you shared still seems it will be very helpful. I just tried it, and the rendered graph contains a lot of useful-looking data for me to look through. It will definitely help in the meantime, thank you! |
|
I have made my first important discovery using speedscope. It seems kotlin BuildFusService was about 25% of my speedscope graph. So I created a YouTrack issue and PR requesting an ability to disable BuildFusService completely. I will continue to update this issue if I find anything important such as this. |
|
@bamboo I think I have done all I can with this tool, and I would be curious to hear what advice you would have for what I should do next in my troubleshooting of my configuration cache size. As of now, my "work.bin" file is above 1 gigabyte. The Sppedscope seems promosing, but I think the main problem is that it seems to only provide the class name and not any details about the instances. For example, it will list many Here is a screenshot of my speedscope. This is just from running 
As you can see there are 2 "sections":
Here is a zoomed in view of the first part of the Gradle section It is very uniform. Most of this first part looks like this. And here is a zoomed in view of the Cleanup registrations section. 
This part is also very uniform. Here is what I can gather from this analysis:
I am eager to break this down more to see how it ammounts to over 1 gigabyte. I am wondering if there might be some sort of bug or issue. For example, do you think it might be possible that something in my build logic is causing the same file path entry to be cached many times? If there was some sort of multiplicitive effect like this, where say duplicates of file path entries were being cached, that might start to explain the large size and some of the performance issues I am facing. Seeing the actual file paths seems like it would be critical for this analysis. Also, maybe breaking down the huge "RegisteredBuildServiceProivder" chunk into smaller more detailed entries would help as well. |
|
I realize there are two more views. Here they are for the same build 
|
|
I also notice that in the charts, the numbers are going up to a few hundred kilobytes, which is nowhere near the over 1 gigabyte |
Expected Behavior
The title says it all.
Current Behavior (optional)
There is no way to inspect the size of configuration cache interies.
The configuration cache is stored in
.binfiles (binary, with a custom, internal encoding?) inside.gradle/confgiuration-cache. I'm sure that the encoding is designed and optimized for performance, but currently there is no direct way for a developer to answer the question "what entries in my configuration cache is making it so large"?The workarounds I can think of are:
Context
This is the third in my series of issues requesting features to allow developers to debug and better understand their own configuration cache. The previous issues (which as of now are still open):
I have a very large build, and the time it takes to read and write the configuration cache is enormous (in the minutes sometimes) and growing. Now, for the first time, I got an OOM error while gradle was doing some work related to the configuration cache:
OutOfMemoryError Stack Trace
Currently, there is no reasonable way for me to fix my error. My only options are to basically to delete the cache and try again. My build is large and complex and I have no idea what subprojects/tasks/plugins are making the cache so large. My machine has more than enough memory, and this should not be an issue. Even if I didn't get an OOM error, there is still the issue of configuration-cache work taking minutes at a time.
The whole Gradle community would benefit greatly if developers had the tools to start inspecting their own configuration cache.
One way that could possibly be the easiest way to resolve this issue would be a simple command line tool or gradle task that reads a configuration cache folder (with the .bin files) and writes the entries to a json file. Each entry would just need a human-readable name and a byte size. This likely requires a solution to #24757 as well; we need a way to identify which build is associated with which configuration cache folder.
Here is a proposed json format:
[ { "build": "<a build identifier>", "configuration-cache": { "entries": [ { "name": "FileContents[/some/jar]", "size": 293555 }, { "name": "Property[somePropertyName]", "size": 53 } ] } } ]The text was updated successfully, but these errors were encountered: