Having problems with too long compilation command lines

[ewantowie]

Hi there,

I'm having problems using B2 with a very large software project and compilation command lines that are too long. The issue we have is the number of includes '-I' which makes the command line length very long. In our case, we do use the '--abbreviate-paths' b2 option but this helps with linking and not compilation command line lengths.

At first glance, an ideal solution would be to switch from absolute paths for the includes, but I'm not clear if this is easily possible from the B2 manual.

Does anyone have any suggestions on how to resolve this issue?

Edited to add that I'm on a Linux system and use a combination of GCC, PGI, and Intel compilers.

Thanks,
Ewan

[grafikrobot]

Dealing with this will likely require changes to the B2 toolsets. There's a built-in feature to automatically switch to response files when command lines get too long. But unfortunately that's only used on Windows commands as that's where the limitations are most restrictive. It could be fairly easily get used universally though. As for what to do to deal with it now.. In addition to --abbreviate-paths there's the --hash option. The --hash option (https://www.bfgroup.xyz/b2/manual/release/index.html#bbv2.overview.invocation.options) will shorten the paths to a consistent absolute minimum. Perhaps that will work for your case sufficiently until we can change the toolset code.

Could you post the length of the lines you are having problems with?

[ewantowie]

Both the '--hash' and '--abbreviate-paths' options help to reduce the link command line where you are referring to the files within the b2 output directory structure. What they don't help with is reducing the compile command line where you are including many paths from the workspace. This is fundamentally the issue I have.

In the problem case I'm having, the compile command line is 3500 characters long. The majority of this is the many, many -I<include-path> statements needed.

The setup we have for b2 uses an absolute path for every <include-path> when all of them are actually within the workspace root. I.e. could be switched to a relative path from the workspace root. If I do this manually, the compile command line is shrunk to 1520 characters long.

As we are having a failing build at the moment, any suggestions on how to resolve ASAP would be much appreciated.

[grafikrobot]

Lets go from the top by using Boost as an example.. When I do:

cd boost-root/libs/thread/build
b2 -an

I see include options that use relative paths as such:

"g++"   -fvisibility-inlines-hidden -fPIC -m64 -pthread -O0 -fno-inline -Wall -g -fvisibility=hidden -Wextra -Wno-long-long -Wno-unused-parameter -Wunused-function -pedantic -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_THREAD_DONT_USE_CHRONO -DBOOST_THREAD_POSIX  -I"../../.."  -c -o "../../../bin.v2/libs/thread/build/gcc-12/debug/threadapi-pthread/threading-multi/visibility-hidden/future.o" "../../../libs/thread/src/future.cpp"

For this case the one include path is specified at the root of the project as <include>.. I.e. using a relative path (https://github.com/boostorg/boost/blob/master/Jamroot#L172).

How are you specifying the include paths that you are getting absolute paths? Are they from external projects? From a package manager?

[ewantowie]

We specify the include paths as relative within each project, but we have many different projects. All the projects are defined within the workspace root.

Feel free to ask more questions, I'm not a B2 expert but I'm more than happy to elaborate where I can!

[ewantowie]

Just to try and clarify what we are doing. We have one parent project that contains many sub-projects. The sub-projects are often included in other sub-projects.

Within each project we define the includes as being a relative path.

It is the case that when we include another sub-project, the include path on the compile line is given as an absolute path.

[grafikrobot]

I can try and replicate that arrangement. But it would be really helpful if you can create a simple example that I can use to test with. As I'm think we are straying into it's a bug territory.

[ewantowie]

Let me see what I can do. It will at least be interesting to see if the problem re-appears on a smaller scale.

[ewantowie]

After some further investigation on my side, I realised that it isn't quite the way I had it pictured.

We actually have a workspace containing many parent projects, that is each project defines its own Jamroot. To 'include' another project within a project we can call the use-project function and construct the path using our own functions. It is our code that constructs the absolute path, and therefore not a bug!

This all works as we have a single compilation script that gathers all the projects together and passes them in a single call to b2. Hence the impression of their being a single parent project.

I guess I'll need to do more work on my side to resolve this issue, but thanks for the help.