Introduced SCALANATIVE_CC, SCALANATIVE_CXX and SCALANATIVE_LLVM_CONFIG

[catap]

An idea of this changes is simple: let user to control which clang or llvm-config is used to build an application.

Right now it picks some clang from PATH which might be quite wrong.

[sjrd]

This can already be configured with the withClang() method of nativeConfig. There's no need to introduce more environment variables.

[WojciechMazur]

For *_CC and *_CXX there is already LLVM_BIN (previously we used CLANG_PATH, CLANGPP_PATH). At this point, I see no reason to include these new env variables. However, we cannot redirect llvm-config to a specific version. In my opinion, we should reuse LLVM_BIN for this task as well.

[catap]

@sjrd it required to change sbt file. Here I need a way to specify which clang I should use.

@WojciechMazur at machine which I'm using to build, to be hones, I haven't got any llvm-config. Only clang. Tricky part a few of them and the first one in PATH isn't the right one.

Usually any build system support CC or anything like that to specified exact path to compiller that should be used. As far as I can see you broke it at c73f92b

Different packeting system uses different way to detect the right compiler. For example MacPorts has ${configure.cc} and ${configure.cxx} which points to the right clang and clang++. WIthout this env it is quite tricky because you need to edit the path, and achive that which returns that you needed.

My idea => if user know exact path to clang, let allow him to use it.

[catap]

@sjrd about your way.

Let assume that I have /opt/local/bin/no_default_gcc/clang which is simple bash script which exits with code 1. /opt/local/bin/no_default_gcc is the first path in PATH.

I've added to build.sbt:

nativeConfig ~= {
  import java.nio.file.Paths
  println(s"CC: ${sys.env("CC")}")
  println(s"CXX: ${sys.env("CXX")}")
  _.withClang(Paths.get(sys.env("CC")))
    .withClangPP(Paths.get(sys.env("CXX")))
}

and run env PATH=/opt/local/bin/no_default_gcc:$PATH CC=/usr/bin/clang CXX=/usr/bin/clang++ sbt cliNative/nativeLink , and it fails like:

error] Problem running '/opt/local/bin/no_default_gcc/clang --version'. Please check clang setup.
[error] Refer to (http://www.scala-native.org/en/latest/user/setup.html)
[error] (Global / nativeConfig) Problem running '/opt/local/bin/no_default_gcc/clang --version'. Please check clang setup.
error] Refer to (http://www.scala-native.org/en/latest/user/setup.html)
[error] Total time: 2 s, completed Sep 9, 2021 10:36:37 AM

How should I ask scala-native to use the right compiller?

[catap]

@WojciechMazur FYI the old way allows to adjust clang or clang++ path via CLANG_PATH and CLANGPP_PATH, maybe recover it?

[WojciechMazur]

Can't you just pass LLVM_BIN instead? In /usr/bin, etc. you only have symlinks, in my case I could see following:

➜   readlink -f /bin/clang
/usr/lib/llvm-10/bin/clang

Then when running sbt with export LLVM_BIN=/usr/lib/llvm-10/bin clang would be resolved to that LLVM_BIN directory. I can easily switch between this one and eg. clang-6.0.
I belive that in the other places you can find symlinks as well. If actual binaries are not within some common directory I would say that's non-canonical setup. /usr/lib/llvm-10/bin

[catap]

@WojciechMazur my point that CLANG_PATH, CLANGPP_PATH or similar variables is standard de facto. And scala native before your changes supported it.

I just would like to recover this environment some how.

[sjrd]

@sjrd it required to change sbt file. Here I need a way to specify which clang I should use.

You can use the set command of sbt to specify sbt settings on the command line.

[catap]

@sjrd anyway, env CLANG_PATH=/usr/bin/clang CLANGPP_PATH=/usr/bin/clang++ sbt cliNative/nativeLink works as expected.

=> I just would like to save this way for the next releases.

[WojciechMazur]

I understand this concern. At the same time setting LLVM directory instead of it's separate binaries will require less maintance in the future and should be easier for the users, especially since you should set both clang and clang++ to point to the same version. Also Graal Native Image uses just single env 'LLVM_TOOLCHAIN' or 'llvm.home' system property.
Also when it comes to env variables we would opt to remain only minimal needed amount of them in the future.

[ekrich]

It does seem that Discover.clang() gets called in the plugin so it may be too eager for using nativeConfig as you tried above. Perhaps that should be changed somehow?

I did a lot of searching and could not find any reason for the prior setup and did find a usage of LLVM_BIN online. The current change really simplifies the setup and makes it more inline with other system setups that require the system in the path or an environment variable set for some common root. I also found systems where clang++ is just a symlink to clang (brew install on macOS).

lrwxr-xr-x    1 eric  staff         7 Dec 11  2019 clang -> clang-9
lrwxr-xr-x    1 eric  staff         5 Dec 11  2019 clang++ -> clang
-rwxr-xr-x    1 eric  staff  89035740 Dec 11  2019 clang-9

You should be able to do the following as Discover will use LLVM_BIN as a path to clang and clang++:

env LLVM_BIN=/usr/bin sbt cliNative/nativeLink

[david-bouyssie]

I understand this concern. At the same time setting LLVM directory instead of it's separate binaries will require less maintance in the future and should be easier for the users, especially since you should set both clang and clang++ to point to the same version. Also Graal Native Image uses just single env 'LLVM_TOOLCHAIN' or 'llvm.home' system property.
Also when it comes to env variables we would opt to remain only minimal needed amount of them in the future.

If GraalVM is already using LLVM_TOOLCHAIN similarly to LLVM_BIN, don't you think it could be simpler to use the same variable name?
I'm thinking about contexts where people would work both with Graal and SN.
Might not be a huge use case, though.

[catap]

@WojciechMazur I don't know GraalVM code base but it seems for me that it can use CLANG and CLANGXX. But I might be mistaken.

[catap]

@ekrich this is still different behaviour with 0.4.0 which allows to define CLANG_PATH and CLANGPP_PATH.

Thus, if you haven't define LLVM_BIN and the first clang in PATH exits with code 1, you haven't got any options to specify which clang should be used because sbt fails on Discover.clang() stage when it's looking a standard clang in PATH :)

[ekrich]

@catap I am looking into changing the discovery. I believe your withClang should have worked. I have a good idea on how to improve this.

[catap]

Thus, addung LDFLAGS or CFLAGS is also a night mare, special that you must add it to exact project and hacks which is required for cross-project and matrix are different.

I still think that supporting: CC, CXX, CFLAGS, CXXFLAGS and LDFLAGS is worth to add.

Because otherwise any maintanier who would like to pack SN application will have a nightmare.

[ekrich]

Yes, I don't believe the configuration is all figured out yet. Do you have any specifics on what you are doing, have in your sbt or build files etc. It will be really hard to find better solutions without more specific examples. I think Scala.js projects can have similar complex setups too but their setup is working.

I also think that published projects need to be allowed to setup flags because there is no way that the users can know what flags to pass and that is for the whole build not dependency by dependency.

[catap]

@ekrich I've try to create a few simple ports for macports for SN application.

• https://github.com/macports/macports-ports/blob/master/devel/sn-demangler/Portfile
• https://github.com/macports/macports-ports/blob/master/math/curves/Portfile

Any SN application requires MAP_ANONYMOUS, linkat and symlinkat calls that was added not so long ago to macOS => it needed to use legacysupport which is adding https://github.com/macports/macports-legacy-support to include dirs (for MAP_ANONYMOUS) and dynamic library (for linkat and symlinkat). Thus, right now the second port expects that sdl2 is installed at one of standard path and adjust it for system specified settings => night mare.

Also, MacPorts provides different compilers and it requires that specified compiler are used, and not cc or clang from PATH.

I feel that any packaging manager (deb, rpm, you name it) => may need to add this flags like any another build system allows.

Right now the only way is patching build.dbt which might be quite complicated.