How the module system work? Can you make a "fast documentation"?

[rempas]

You know I'm trying the language when I'm making two post in the same day. I hope you don't mind. Now first of all I want to ask how the modules work. I have the following structure:

voxlib
    | io.vx
test.vx

I want to import the "io.vx" module from the "voxlib" package (using D words). I'm trying the following statement in the "test.vx" module: import voxlib.io. And I'm getting the following error message: Error: Cannot find module voxlib.io`. So I suppose the system is different. What should I do?

The second thing I wanted to say (didn't want to make a separate post) is that I would like some sort of documentation. Now I said it before and I know that making a documentation is hard and takes time so I have an idea! What do you say to make something simple and fast that will only show the language features without explaining match? Just for the sake of knowing what's available and how things work (take the module system for example). For example something like that.

# Variables
u8 age; # Declare a variable
u8[] name = "John" # Declare and define a variable

# Types
u8 # Unsigned 8byte type
i8 # Signed 8byte type
float # 4byte floating point type
double # 8byte floating point type
...... # Rest goes here

# Functions
# Synntax = return_type name(type paremater1 = defalut_val, type parameter2) { // Body code here }

# Control flow
if (condition) { // If the condition is true then do something }
else if (condition) { // If the previous condition is false then check this one }
else { // If none previous branch matches then execute the code here }

How does this sound? This will probably take 3-4 hours (You don't have to explain everything in super detail) and you can have it ready in 3-4 days depending on how much free time you have. People will ask questions (like I do now) anyway so I think that's a nice to have for now until we get a big full documentation!

Alternatively, What about writing the changes from D?? That can also be a great idea. From example the changes in the name of the types! In the module system, etc. That can be even faster! Man I'm so exited!!!

[MrSmith33]

a) io.vx should have module voxlib.io; at the top
b) you need to pass every source file to the compiler (I want to implement -i switch, like in D, so compiler would automatically compiler all imported files for you)

Let me see what I can cobble together in an hour. See this in the meantime.

[MrSmith33]

Is this planned to get implemented in the future?

TL;DR Vox will support emitting and consuming .obj/.lib files for the purpose of integration with other languages.

The pros of object files and separate compilation:

• compile different compilation units in parallel
• recompile only parts that changed
• distribute static libraries (which are just archives of object files)
• other programming languages can statically link with you objects and vice versa (this is the use case I'm interested in)

Cons:

• you need to write object file / static libs to the file system
• you depend on linker being installed in the system or implement it yourself (Vox only links internally + links with dlls atm)
• linker needs to read the files from disk
• when compiling separately you spend time not only compiling current module, but also you need to type check all the imported symbols and generate code for all templates (all of which is pretty non-trivial)
• detecting which parts changed is non-trivial too, and requires either generating some metadata, having compilation server that is always online or rely on very coarse info such as file change times.
• classical linking resolves references by string match, which requires generating mangled identifiers
• linker errors are usually pretty garbage because linker in most cases is external program which has no idea about the programming languages
• linkers are actually slow

As you can imagine any extra bookkeeping needed for separate compilation eats into the gains of separate compilation if we are talking about compile times. I'm not even counting numerous bugs that happen with current systems. Ergonomics of integrated solution is IMO substantially better.

One positive example is Go infrastructure, it is a completely independent, vertically integrated system.

Linker is built-in in Vox. The only symbols that require actual lookup are external ones. This is usually used to link with dlls or with host symbols. Then the only work linker does is calculate positions of all symbols and fixup of references.

Current strategy for Vox is to compile very quickly. Crude benchmark shows > 1M lines of code / second, in single thread. This will plenty for a long time.

Future paths may include multi-threading (for speed) and incremental compilation for speed and Language Server Protocol (LSP) support.

[rempas]

Fun fact (well not so much), this is the third time I'm writing this! The first time, I was in my phone and while I was typing the text got deleted for no reason and the second time I was in the work laptop (3 minutes ago), and I forget to plug the power cable and it run out of battery VERY FAST! F my life....

Ok, so 1M LOC in 1 second you say? That's UNREAL!!! Where did this test took place and in what hardware? Also does this include templates/macros and meta-programming in general (Even tho I suppose code generation is the most time consuming thing)?

Man I'm thinking what I have to write again and I get crazy.... I'll make it quick. If I'm right, supporting libraries from other languages means that you have to know the name mangling of every languages so you know how to call symbols. This means either support the language mangling directly (like D does for C AND C++) or only support C and have the other languages to create binding to C and then bind from them (Which will be time consuming and error prone probably). Another thing to consider is that, from what I know, C has a library for pretty much anything you want right. C++ (or even some other language) may have a better one is some cases but this will probably be rare and the differences will not be so huge and worth going through all this trouble. So just targeting C will be a good idea that will also not piss you off...

Another thing to consider is the people that will create and maintain those bindings. Based on my experience, only language like C, C++, Rust and Python (and maybe Go in some cases?) will be popular enough that they will have binding for the most important (still not for everything) libraries and this is what I found trying to find libraries for language like D, Vala, Nim, Vlang etc. This is also why it is hard for a new language to become widely used. When I say to someone that I found a new cool language they always reply: "Yeah and are there any libraries yet? What can you do with it?". So I really believe that the best advertising for the languages are native libraries written in this language. And this will be even better in Vox if it compiles that fast! I want to also point out that the "famous" languages that I mentioned, also have native libraries written in the language so they don't need binding either way! If Vox becomes famous, people will start creating libraries in it, I'm pretty sure!

The object system is also very interesting. I didn't knew about all the disadvantages and I'm wondering. How much the loss in the compilation speed will be? Also would people still have the choice to use the old system and don't create object files, link etc. and keep those fast compilation times? Another idea that I have is creating a tool that converts C source code to the equivalent in Vox. A C-to-Vox translator. This is an amazing idea as:

1. You won't have to change Vox itself and lose all those advantages it has
2. This will also allow people to convert their project and continue the development in Vox which they may prefer anyway so It's a good tool to have in general.

However, This tool will probably take a lot of time as we must first write a library on Vox (I'm already trying to write something about that) which will be the equivalent of Libc. Thankfully Libc is not huge so it won't take years. I don't know how much time the tool will take. I suppose you don't have to check if the C syntax is right, supposing that it has been checked by a C compiler so you can skip all this part and this will probably save some time.

Anyway the funny thing is that this was not quick, this was probably even more than what I wrote in the last 2 times. I would like to see what you think about all I've said. You are probably 100x times more experienced that I am so you probably have thought about all that anyway...

[MrSmith33]

Where did this test took place and in what hardware?

Win10 + 3950x, should be a bit faster on linux (on the order of 5-7% faster)

This is super simple benchmark where lots of fibonacci functions are compiled in a single file. Doesn't include templates or any metaprogramming. Templates require a copy of the AST for each instance, which is equivalent to compiling template body this number of time + time to copy the body. Yesterday I came up with an idea of benchmark that instead of compiling N fibonacci functions generates them at compile-time. Would be interesting to compare with current test. This is not possible with current compiler though.

How much the loss in the compilation speed will be?

You don't know until you implement it.

You can already interact with other languages through C ABI. Currently you can link with dlls (only on Windows), use syscalls (on Linux), call host functions and be called by host in case vox compiler is embedded into host program (supported on win/mac/linux).

My ideal is to have all compilers support emitting some machine readable description of the compiled program, so that other compilers can consume it. See vulkan xml description for example. It is not perfect, but is pretty close to what I imagine. Because right now people write bindings by hand or write some binding generator that is target language specific, which means M x N effort. Where if we had common IR, then it would be M + N instead.

The only sane way of parsing C is to use clang as a frontend, which is what tools such as DStep or dpp do.

The basic universally useful thing is the ability to call OS APIs, everything else can be built on top of that.

[rempas]

Got that! You really know what you are doing! So, I'll let you do your thing, I'll try to work in the library in the meanwhile! Have fun!

[MrSmith33]

Now that I think of it, it may be a good idea to make all of our bindings be in machine readable format + a library for generation of Vox source. This way it will be useful for other people, using other languages (which means more potential community), and allows us to change Vox syntax/semantics much more drastically in the future. Plus, if that format turns out to be flawed, one can always transform it into another, since it is easily consumable.