Prevent extern "C" propagation

[Jan200101]

extern "C" is C++ specific and if we enable it our consumers are likely not C++.

[Jan200101]

I was unsure about if this should also be made private.

There doesn't appear to be anything in the header that makes use of it and a consumer may want to know about this.

[Jan200101]

Same as the other

[vegorov-rbx]

Sorry for the long response, we had to find someone who knows how CMake works.

This change will result in anything downstream from Luau.CodeGen that relies on LUACODEGEN_API having to also define it themselves, if they don't want to get linker errors.
If someone is linking against Luau.CodeGen from CMake and they're not C++, they do get the problem you are trying to solve, but this use case is rare - for a C library in CMake to link against a C++ library.

The better solution would have been to use a generator expression like $<$<COMPILE_LANGUAGE:CXX>:LUACODE_API=extern\"C\"), but that requires CMake 3.15+.
I believe we can do the same using an additional if statement to check for COMPILE_LANGUAGE.

[Jan200101]

The better solution would have been to use a generator expression like $<$<COMPILE_LANGUAGE:CXX>:LUACODE_API=extern\"C\"), but that requires CMake 3.15+. I believe we can do the same using an additional if statement to check for COMPILE_LANGUAGE.

The generator expressions for the language appears to have been introduced with cmake 3.3 not 3.15, but that is still above the minimum version luau requires.

I don't think a single if would do the job, the if statement would be evaluated while luau is being configured, at which point CXX would be part of COMPILE_LANGUAGE.

I'm still looking around for other alternatives, but a massive workaround would be setting CMAKE_CXX_FLAGS

[TheGreatSageEqualToHeaven]

This change breaks exporting to a static and dynamic library which breaks embedding Luau in e.g CSharp. This is not a useful change and there should be specific settings added for exporting to a DLL, static library and basic extern "C" rather than one single setting

@Jan200101 @vegorov-rbx

[Jan200101]

This change breaks exporting to a static and dynamic library which breaks embedding Luau in e.g CSharp.

As far as I can tell the cmake setup does not currently support building a dyanamic library, so this appears to a moot point, though it does affect how Luau would be consumed by C++ projects, breaking compatibility there.

This is not a useful change and there should be specific settings added for exporting to a DLL, static library and basic extern "C" rather than one single setting

This change needs to be done somehow regardless.

Most languages rely on the C ABI for their FFI and those that can consume the header files (e.g. C, Zig, etc.) may not understand extern "C".

An alternative (that I think would be best overall) would be dropping the build option and forcing extern C on the entire header when C++ is used.

[vegorov-rbx]

@zeux I wonder if we should always use "C" linkage without relying on CMake and place them in a block like:

#ifndef __cplusplus
extern "C" {
#endif

[zeux]

Unfortunately we can't use extern "C" unconditionally when LUA_USE_LONGJMP is set to 0 -- this breaks exception handling because MSVC doesn't generate unwind tables, so calling functions like luaL_checknumber will abort instead of throwing a catchable exception.

It does seem like we'd need to have a separate LUA_EXTERN_C define :-/ I think the DLL export/import can still be handled as we do now (with CMake configuration without explicit support)?