Loader

[Roxxik]

I was playing with the idea about writing a simple loader for BareMetal OS so that it is possible to start another app while running (i.e. a shell starting a application or a application and a network stack running in paralell)

I have some ideas but i don't know which to try out first:

1. making apps PIE (position independent executables)
   • the loader is easy just read from memory and jump right into it, just like the cli currently does, bot can be loaded to any location
   • this needs support from the build tools i haven't found much about compilers supporting it
2. having a relocator
   • the relocator would disassemble the binary and translate all address that point into its own memory
   • for larger application the load time needs to be considered
3. don't build flat binaries, build some relocateable format (i.e. elf)
   • requires to build or port a tool that can read this format
4. use paging to let the applications appear all in the same location
   • i think this makes other things harder, ipc in a flat memory model could be done by just passing pointers with paging this isn't possible

If anyone has some advices or suggestion, feel free to post

[scherrey]

Probably the "easiest" thing would actually be to have each executable run
under it's own VMX environment and have BareMetal act as a hypervisor. The
model is not unlike the current smp concept except now enforced by the CPU.
Naturally every VMM would be a root VMM with full access although the first
one would have to negotiate sharing of devices. I think the only overhead
involved would be when sensitive instructions are intercepted and memory
pages come in and out needing remapping. Avoiding such instructions and
mapping larger pages should make the performance impact too small to be
measured. I've not looked at whether there's an impact on memory mapping in
flat mode for VMM or not. I would suggest some range of memory be globally
mapped to support sharing amongst processors and the rest be treated as
private memory to the local VMM. In this manner each application would
effectively have control of the entire computer but could still co-exist
with other apps who (preferably) have their own dedicated cores.

This probably isn't much more difficult than supporting elf or other
relocatable systems and offers a great deal more power and flexibility.

-- Ben

On Wed, Jul 8, 2015 at 2:32 AM, Mason X notifications@github.com wrote:

I agree with this. +1.

—
Reply to this email directly or view it on GitHub
https://github.com/ReturnInfinity/BareMetal-OS/issues/98#issuecomment-119310789
.

[ohnx]

However, I fell like this loader should not be a part of the OS, but instead a program that you can run.

[Roxxik]

My idea was that it could be part of a shell
Am 10.07.2015 00:13 schrieb "Mason X" notifications@github.com:

However, I fell like this loader should not be a part of the OS, but
instead a program that you can run.

—
Reply to this email directly or view it on GitHub
https://github.com/ReturnInfinity/BareMetal-OS/issues/98#issuecomment-120163075
.

[benaryorg]

I think the shell should be "outsourced" in it's own repo where it can
be developed as a general purpose app.

This has been discussed at #40 and @primoz made some pretty good progress.

This would ensure BareMetal stays bare metal.

[scherrey]

So the intent is for the shell should provide a relocatable loader? Elf or
some kind of special parsing?

-- Ben

On Fri, Jul 10, 2015 at 5:53 PM, Katze notifications@github.com wrote:

I think the shell should be "outsourced" in it's own repo where it can
be developed as a general purpose app.

This has been discussed at #40 and @primoz made some pretty good progress.

This would ensure BareMetal stays bare metal.

—
Reply to this email directly or view it on GitHub
https://github.com/ReturnInfinity/BareMetal-OS/issues/98#issuecomment-120382091
.

[benaryorg]

My intent was to have a shell, which is a shell, and a loader, which
is a loader, both completely indepentently developed.

So we would have the kernel being minimalistic.
The kernel could be configured to start the loader.
The loader would prepare an environment capable of running multiple independent programs, by for example exposing functionality to load programs.
These programs could be either drivers or applications.
The loader would start an unix-like init (or whatever you specified),
which then starts the shell.

But that's only one possibility.

[scherrey]

The thread started out as a discussion for how to handle multiple apps
loaded at once. I'm not sure how you're defining your choice of words but
multi-tasking, to me, generally implies multithreading (more than one
thread of execution on a cpu probably exclusive of hyper threading) whereas
multiprocessing would mean supporting as many processes as there are
available cpus (hyper threading potentially included). I'm not sure which
one you're suggesting. Outside of this - the level of isolation that is
intended is another key point. The word's ambiguous enough to cover both
but I don't think the multi-processing meaning is common usage for
multi-tasking.

My thoughts so far, that would keep the kernel minimalistic but capable of
fully utilizing the CPU(s), would be either go with a VTX style isolation
amongst one or more CPUs per VMM enforced by the chip set (possibly with a
shared memory map for IPC) , or require applications to be written in a
relocatable format and everything goes into a big single memory space but
CPUs are assigned to apps and concurrency is likely limited to however
number of logical cores are available. The former seems like the next
logical "level up" step of BareOS but I'm not sure if there's an intent to
ever go that far. The latter is the ultimate flat model and pushes most of
the effort over to the loader/software development tools which could
possibly add friction for software development. If we're really serious
about the exokernel model then we could do the latter at first and offer
another kernel module that adds the VTX option later with probably little
impact to the relocatable loader option (although it may obsolete it for
many use cases).

Care to clarify which intent (or some other) you have in this regard? I
presume the latter but it's still not clear to me. I'd be very curious to
hear what everyone elses' thoughts are regarding this, especially Ian's.

-- Ben

On Fri, Jul 10, 2015 at 6:15 PM, Katze notifications@github.com wrote:

My intent was to have a shell, which is a shell, and a loader, which
is a loader, both completely indepentently developed.

So we would have the kernel being minimalistic.
The kernel could be configured to start the loader.
The loader would initialize the multi-tasking.
The loader would start an unix-like init (or whatever you specified),
which then starts the shell.

But that's only one possibility.

—
Reply to this email directly or view it on GitHub
https://github.com/ReturnInfinity/BareMetal-OS/issues/98#issuecomment-120387246
.

[Roxxik]

The loader I'm currently thinking of is just an app which can load(and relocate) another binary into memory and jump to the start of it.
Keeping it very basic right now.

This could be used by a shell to start another binary.
The binary could be started with smp_enqueue instead of a jmp which is essentialy multiprocessing.

But first of we need to be able to load an executable.

If this kernel is 'just' starting VMMs its basically a Hypervisor Hypervisor and not an exokernel.

And if you want to load multiple VMMs you'd need to load them to different locations, too.

[benaryorg]

@scherrey Sorry for making that ambiguous.

I edited that comment.

[benaryorg]

We can already load an executable. Only a binary, but an executable one.

So if someone make a program called "loader" that reads some e.g. elf file, parses and loads it (via paging so multiple instances can be run simultaneous) and provides that functionality to the program it just loaded, the underlying program could set up a fully working environment with process isolation and the usual stuff while keeping the kernel a simple exokernel.

[ohnx]

I've actually started working on what was described in #40 here, you can take a look if you want.

[IanSeyler]

A part of this functionality was built into the CLI in the past. The CLI would begin and look for an app on the disk called 'start.app'. If it existed then it would be loaded and executed automatically. This function was removed for some reason I can't recall at the moment. Probably because it was limited to a specific app name to work properly.

I guess we sort of need an autoexec.bat file :P

[Roxxik]

I think in the long term it would be best to choose what application is started first by the OS (currently hardcoded to the cli) by calling it a certain name (start.app, init.app ... the name is not a matter of this discussion)

where the default would be to start a shell reading and executing another file (start, init ...) as shell script

this hypothetical shell can then be build in another repository