Document build dependencies #2

spencerking · 2019-08-30T01:43:02Z

I've found that a clean install of Fedora 30 is pretty lacking when it comes to building the project. I'm using this issue to document all of the packages I need to install as I work towards a build. Going forward it may make sense to create a nix or guix configuration, or perhaps split the custom components into their own repositories with clear documentation.

Packages:
llvm-devel
pcre-devel
libevent-devel
patch
texinfo
zlib-devel

drhuffman12 · 2019-10-25T03:46:38Z

Maybe wrap the build in some docker/compose files?

ffwff · 2019-10-25T10:13:29Z

Yeah I don't use Docker.

Maybe I'll document it in depth later but for now please just view Crystal and GCC's manual for required build dependencies.

spencerking · 2019-12-29T06:14:34Z

In response to #15:

I never found out what package contains x86_64-elf-as, which is needed to build the kernel.

When trying to build binutils I get the following:

make[1]: *** [Makefile:849: all] Error 2
make[1]: Leaving directory '/home/spencerking/Documents/lilith/userspace/toolchain/binutils'
make: *** [Makefile:13: build_binutils] Error 2

ffwff · 2019-12-29T06:35:15Z

If you're just building the kernel, you'll only need x86_64-elf binutils, GCC and Crystal; userspace/toolchain/binutils is for userspace applications.

ghost · 2019-12-29T17:13:46Z

Building patched Crystal fails despite various attempts, ~~probably due to different (and constantly updating) LLVM and Crystal versions.~~ (Edit: I didn't make clean after the initial build failed, screwing up future builds.)
@ffwff Could you tell us which version of LLVM and Crystal you're using to build this? ~~I can't sleep I think I'm gonna cry~~
Edit: Compiler building works with LLVM 9.0.0 (installed with linuxbrew) + Crystal 0.32.1 (From crystal github releases)

spencerking · 2019-12-29T17:41:57Z

Building the patched Crystal consistently works for me.

The bigger issue for me is it's really not clear what packages are required for the entirety of the build process (kernel, userspace, the whole deal). This is why I had originally mentioned creating a nix shell. Everyone would be using the same versions of the same packages, making this a non-issue.

ghost · 2019-12-29T18:27:51Z

I would definitely appreciate that, but I'm sadly not well acquainted at nix to approach that :(
I'll at least try to document the process thoroughly after I get it to work :)

spencerking · 2019-12-29T21:28:25Z

@Scorbie I would be happy to do it once I can actually build the project.

ghost · 2019-12-30T09:45:07Z

Okay, I got all the specified requirements ready. I can build the kernel (which doesn't run but that's a seperate issue AFAIK; origin/master is shaky, captain!)

@ffwff Present for your 1000th commit, the text below is yours. Congrats.

The requirements

LLVM 8.0 (recommended by Crystal team at the time of writing)
or 9.0 (worked for me)

Get it with your package manager or LinuxBrew or nix or compile it.

Crystal
(Note. currently it has to be installed as `/usr/bin/crystal`)

You may get it with your package manager; see https://crystal-lang.org/install/ for details.
There are linux binary releases at https://github.com/crystal-lang/crystal/releases

v0.32.1 works with LLVM 9.0
I'm not sure which version goes with 8.0, so you need to experiment.
There may be more hidden dependencies (see first post) that I installed in the process of trying with LinuxBrew (which failed)

Cross-Binutils (amazingly the easiest step for me)

Edit(2020.11): Welp... I was careless and misread the instructions and this tripped me.
Updating the instructions to be clearer (and hopefully more foolproof).

I don't think there's prebuilt packages available, even for NixOS. Only Arch seems to have it. Source
Download & Compile Binutils yourself! Recommended instructions here.
You probably should prepare two copies of the same source; one for each arch.
(i.e. one for x86_64-elf and the other for i686-elf)
- configure --prefix ... --target x86_64-elf && make && make install in the first copy
- configure --prefix ... --target i686-elf && make && make install in the other one
- Even make distclean doesn't allow you to reuse the same source directory.
  (There are build caches not cleaned) I had to learn this the hard way. I hope you don't.
Also, when running configure, you have to
- Either not set --prefix if you have root (root is needed for make install)
  and don't mind the tools getting in your /usr/local/bin
- Or set it, then Add PATH="<prefix-path>:$PATH" in your sh startup script.
  - The tools are used during make and make launches sh for executing commands.
    That sh has to see the tools in $PATH, regardless of which shell you use.
  - According to POSIX the startup script is specified in the $ENV variable;
    So this is guaranteed to work on all POSIX-compliant systems unless I am mistaken:
    
    export ENV=<sh-startup-script-path> in your ~/.profile if it's not set
    PATH="<prefix-path>:$PATH" in your <sh-startup-script>
    logout and login to make sure changes are applied.
  - Although on my Ubuntu 20.04, adding the paths to my $PATH in ~/.profile did work for some reason I don't understand.

Building

Of course git clone https://github.com/ffwff/lilith and cd lilith first.

Building the kernel

Actually I'm just reiterating the pretty well documented readme, ~~it just doesn't work~~ It may not build when something screwed up in the process so good luck.

Build the toolchain: make toolchain/crystal/.build/crystal
- Note: if the build fails make sure to at least cd toolchain/crystal && make clean && rm -rf ~/.cache/crystal so that the patched Crystal is not littered with your past failed build artifacts.
Build the kernel: make build/kernel

Building the userspace (optional)

Make sure to install bison and flex beforehand, then see my post below for instructions.

Running

make run - Qemu hangs with no disk found but it runs, at least.
make disk.img => make run_img - Fails with error, will raise an issue if needed.

ghost · 2019-12-30T10:17:05Z

I'll try with docker if there's much request (I haven't used docker much so it's quite time consuming)

spencerking · 2019-12-31T00:57:23Z

@Scorbie I had to edit the Makefile to hardcode the paths to the binutils binaries on my system. Possibly a fish issue, but I don't know yet.

I have gotten the kernel to build, but when I try to make run I get a black screen in qemu and an error in my terminal stating that no disk was found.

Edit: Nevermind, I see you ran into the same issue.

ffwff · 2019-12-31T02:05:53Z

I'm busy at the moment, but I usually use this command for debugging:

make rungdb_img RELEASE=1

You can also pass NO_CPUEX=1 to trigger a breakpoint on CPU exceptions.

spencerking · 2019-12-31T03:55:33Z

qemu-system-x86_64 -kernel build/kernel -monitor telnet:127.0.0.1:7777,server,nowait -m 512M -serial stdio -no-shutdown -no-reboot -vga std -cpu SandyBridge,+pdpe1gb -hda disk.img -S -gdb tcp::9000 &
sleep 0.1s && /usr/bin/gdb -quiet \
        -ex 'target remote localhost:9000' \
        -ex 'hb kmain' \
        -ex 'continue' \
        -ex 'disconnect' \
        -ex 'set arch i386:x86-64:intel' \
        -ex 'target remote localhost:9000' \
        build/kernel64
qemu-system-x86_64: -hda disk.img: Could not open 'disk.img': No such file or directory
Reading symbols from build/kernel64...
localhost:9000: Connection timed out.
Hardware assisted breakpoint 1 at 0xffff808000124470: file /home/spencerking/Documents/lilith/src/main.cr, line 29.
The program is not being run.
You can't do that when your target is `exec'
The target architecture is assumed to be i386:x86-64:intel
localhost:9000: Connection timed out.

ghost · 2019-12-31T03:58:03Z

@spencerking You probably shouldn't hardcode the binutils path, but rather add them to "$PATH" as (edit) I mentioned in the post. Or did you mean that it couldn't find cross-binutils when you configured without setting --prefix?
@ffwff Yep seems like you're busy. I'll try that after I get back home. Thanks.

spencerking · 2019-12-31T04:10:09Z

@Scorbie I mean I added them to my path and make still isn't picking them up, but they work fine otherwise. I imagine it's fish (my default shell) not playing nice with make, but I haven't tested with anything else.

ghost · 2019-12-31T12:06:06Z

@spencerking Ah, strange indeed, I'm out of clues and can only random guess... Wait! make (if it's POSIX-compatible) by default invokes sh, not fish. So I think you should update the ~~whatever-sh-source directory~~ ~/.profile (Updated my post above as well).
Edit 2020.11: Updated the post again for complete information, ping me if there's anything wrong.

ffwff · 2020-01-01T11:10:52Z

I added additional build instructions (thanks @Scorbie!) Tell me what yall think.

ghost · 2020-01-01T17:32:35Z

Oh you said "tell"!
Love it! Newbie friendly, on-topic, I think I can see you put in some thoughts there. Thanks.

spencerking · 2020-01-01T19:09:28Z

Hopefully the last question from me on this one: Any idea what package contains grub-install? I'm not seeing it in dnf.

ghost · 2020-01-01T23:35:22Z

If you use a linux system, you should probably have it in /usr/sbin. (Thus, in your path.)
Otherwise, I don't have any idea except from building from source (which should be possible, I hope)

spencerking · 2020-01-02T05:42:52Z

@Scorbie Turns out grub2-install is not necessarily aliased as grub-install.

ghost · 2020-01-02T05:44:18Z

Eck, kudos to you for finding that out.

…

On Thu, Jan 2, 2020, 2:42 PM Spencer King ***@***.***> wrote: @Scorbie <https://github.com/Scorbie> Turns out grub2-install is not necessarily aliased as grub-install. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=ABZTKWOMDODQRGP5GXWS7JTQ3V5F3A5CNFSM4ISIKSJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5WENI#issuecomment-570122805>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZTKWK7U76RPPCKMC6FRADQ3V5F3ANCNFSM4ISIKSJA> .

ffwff · 2020-01-02T06:21:32Z

I'm only using legacy grub, grub2 is currently not supported.

ghost · 2020-01-02T14:48:56Z

You mean legacy multiboot? I think GRUB2 supports old multiboot just fine, guessing from the fact that GRUB 2.04 manual points to multiboot 0.6.96 spec. So no worries?

ffwff · 2020-01-03T02:12:08Z

I'm only using legacy grub for the image's bootloader. I might be switching to grub2 at some point but it really isn't worth it (apart from maybe removing the i686-elf toolchain for building bootstrap code).

I haven't tried using grub2 yet, however it should work if you load it from the multiboot command.

ghost · 2020-01-03T02:36:17Z

I see, thank you :)

ghost · 2020-01-05T06:21:34Z

Okay, I am building the userspace and the following words would be VERY helpful for others seeing the issue.
(BTW, assume every doc I write in this repo as Public Domain or CC0)
For this post, I'm assuming the working directory is <path-to-lilith-repo>/userspace/toolchain

You need flex and bison installed, in ADVANCE. (maybe texinfo too)
- Or else you'll see some error like syslex_wrap: missing syslex.c that keeps haunting you whatever you do.
When the build failed like the above, install flex and bison, then rm -rf binutils and run make again.
- Yes, you really need to wipe the whole thing away.
- No, make clean doesn't work if you didn't have flex or bison in the first place. (After googling, my guess is that this affects the configure stage)

The build is still ongoing; Will update if there's any changes.

ghost · 2020-01-05T12:31:50Z

For people reading this issue thread, I documented my trial and error on another thread (...) #22 (comment)

jwoffenden · 2020-05-02T00:51:32Z

Yes, you really need to wipe the whole thing away.

No, make clean doesn't work if you didn't have flex or bison in the first place. (After googling, my guess is that this affects the configure stage)
I think you're totally right about the configure stage - I wasted a lot of time on this and your post helped tremendously. Thank you!

ghost · 2020-11-24T04:09:48Z

Yes, you really need to wipe the whole thing away.

No, make clean doesn't work if you didn't have flex or bison in the first place.
(After googling, my guess is that this affects the configure stage)

I think you're totally right about the configure stage - I wasted a lot of time on this and your post helped tremendously. Thank you!

Thank you, I've updated the instructions to emphasize this once more.

ffwff mentioned this issue Dec 28, 2019

Invalid option: --mcmodel when running 'make build/kernel' #3

Closed

ffwff added the os label Dec 28, 2019

ffwff added the build system label Dec 30, 2019

ghost mentioned this issue Dec 31, 2019

error build #19

Open

ffwff added a commit that referenced this issue Jan 1, 2020

os: better build instructions (#2)

76070a3

ghost mentioned this issue Jan 5, 2020

Kernel: Build success? #22

Closed

ffwff added a commit that referenced this issue Jan 5, 2020

os: better build instructions (#2)

eab1395

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document build dependencies #2

Document build dependencies #2

spencerking commented Aug 30, 2019

drhuffman12 commented Oct 25, 2019

ffwff commented Oct 25, 2019

spencerking commented Dec 29, 2019

ffwff commented Dec 29, 2019

ghost commented Dec 29, 2019 •

edited by ghost

spencerking commented Dec 29, 2019

ghost commented Dec 29, 2019 •

edited by ghost

spencerking commented Dec 29, 2019

ghost commented Dec 30, 2019 •

edited by ghost

ghost commented Dec 30, 2019

spencerking commented Dec 31, 2019 •

edited

ffwff commented Dec 31, 2019

spencerking commented Dec 31, 2019 •

edited

ghost commented Dec 31, 2019 •

edited by ghost

spencerking commented Dec 31, 2019

ghost commented Dec 31, 2019 •

edited by ghost

ffwff commented Jan 1, 2020

ghost commented Jan 1, 2020

spencerking commented Jan 1, 2020

ghost commented Jan 1, 2020 •

edited by ghost

spencerking commented Jan 2, 2020

ghost commented Jan 2, 2020 via email

ffwff commented Jan 2, 2020

ghost commented Jan 2, 2020

ffwff commented Jan 3, 2020

ghost commented Jan 3, 2020

ghost commented Jan 5, 2020 •

edited by ghost

ghost commented Jan 5, 2020

jwoffenden commented May 2, 2020

ghost commented Nov 24, 2020

Document build dependencies #2

Document build dependencies #2

Comments

spencerking commented Aug 30, 2019

drhuffman12 commented Oct 25, 2019

ffwff commented Oct 25, 2019

spencerking commented Dec 29, 2019

ffwff commented Dec 29, 2019

ghost commented Dec 29, 2019 • edited by ghost

spencerking commented Dec 29, 2019

ghost commented Dec 29, 2019 • edited by ghost

spencerking commented Dec 29, 2019

ghost commented Dec 30, 2019 • edited by ghost

The requirements

LLVM 8.0 (recommended by Crystal team at the time of writing)or 9.0 (worked for me)

Crystal(Note. currently it has to be installed as /usr/bin/crystal)

Cross-Binutils (amazingly the easiest step for me)

Building

Building the kernel

Building the userspace (optional)

Running

ghost commented Dec 30, 2019

spencerking commented Dec 31, 2019 • edited

ffwff commented Dec 31, 2019

spencerking commented Dec 31, 2019 • edited

ghost commented Dec 31, 2019 • edited by ghost

spencerking commented Dec 31, 2019

ghost commented Dec 31, 2019 • edited by ghost

ffwff commented Jan 1, 2020

ghost commented Jan 1, 2020

spencerking commented Jan 1, 2020

ghost commented Jan 1, 2020 • edited by ghost

spencerking commented Jan 2, 2020

ghost commented Jan 2, 2020 via email

ffwff commented Jan 2, 2020

ghost commented Jan 2, 2020

ffwff commented Jan 3, 2020

ghost commented Jan 3, 2020

ghost commented Jan 5, 2020 • edited by ghost

ghost commented Jan 5, 2020

jwoffenden commented May 2, 2020

ghost commented Nov 24, 2020

ghost commented Dec 29, 2019 •

edited by ghost

ghost commented Dec 29, 2019 •

edited by ghost

ghost commented Dec 30, 2019 •

edited by ghost

LLVM 8.0 (recommended by Crystal team at the time of writing)
or 9.0 (worked for me)

Crystal
(Note. currently it has to be installed as `/usr/bin/crystal`)

spencerking commented Dec 31, 2019 •

edited

spencerking commented Dec 31, 2019 •

edited

ghost commented Dec 31, 2019 •

edited by ghost

ghost commented Dec 31, 2019 •

edited by ghost

ghost commented Jan 1, 2020 •

edited by ghost

ghost commented Jan 5, 2020 •

edited by ghost