Async/Await

[utterances-bot]

This is a general purpose comment thread for the Async/Await post.

[Cerber-Ursi]

The thing I was trying to wrap my head around for quite some time, thanks for so detailed explanation!

Two minor points:

1. The following code in run_ready_tasks:

if !self.waker_cache.contains_key(&task_id) {
    self.waker_cache.insert(task_id, self.create_waker(task_id));
}
let waker = self.waker_cache.get(&task_id).expect("should exist");

- why not use the Entry API instead?
2. "it's both simpler and safer to instead implement the Arc-based Wake trait" - the link on "Wake" leads to wake method on Waker - probably it should be this trait instead?

[jounathaen]

The statemachine in https://os.phil-opp.com/async-await/#the-async-await-pattern

    fn poll(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Self::Output> {
        loop {
            match self { // TODO: handle pinning
                ExampleStateMachine::Start(state) => {…}
                ExampleStateMachine::WaitingOnFooTxt(state) => {…}
                ExampleStateMachine::WaitingOnFooTxt(state) => {…}
                ExampleStateMachine::End(state) => {…}
            }
        }
    }

Contains the state ExampleStateMachine::WaitingOnFooTxt twice. Should probably be ExampleStateMachine::WaitingOnBarTxt 😉

edit: Fix in #769

[senseiod]

Excellent! Looking forward to the next article

[phil-opp]

@Cerberuser Great to hear that the post is useful to you!

• why not use the Entry API instead?

Good question! I tried using it, but it leads to borrowing errors here because we need to borrow self again for the create_waker method. Of course we could work around this, for example by destructuring self and defining the create_waker methods in a way that it only needs access to the wake_queue field. However, the variant with expect was shorter and easier to explain, so I found it the better choice for this post.

the link on "Wake" leads to wake method on Waker - probably it should be this trait instead?

Thanks for reporting! Fixed in f32ee7f.

[phil-opp]

@jounathaen Thanks for reporting and providing the fix!

[phil-opp]

@senseiod I already started working on it :).

[p2mate]

I'm surprised that on x86 you need to re-enable interrupts before doing a hlt. On cortex m you can do:
cortex_m::interrupt::free(|_| {
if self.wake_queue.is_empty() {
cortex_m::asm::wfi()
}
})

So the CPU will wake from wfi even with the interrupts disabled. Ofcourse the interrupt handler will only be executed when the interrupts are re-enabled.

[phil-opp]

@p2mate Yeah, I think the approach of ARM is definitely better. As far as I know, there is no equivalent to the wfi instruction on x86 unfortunately.

[andyx719]

thank you , it's very detailed/clear and very useful.

[AlphaHot]

Thank you very much for the detailed lesson, and I look forward to continuing.

[LordSparhawk]

There is a repetition typo: "have to have to" in the first paragraph of
Executors and Wakers

Thanks for a fascinating article.

[ZoeyR]

Isn't the uniqueness of TaskId not be guaranteed for zero-sized futures? Box would give the same address for multiple "instances" of the zero sized type.

Edit: See this playground link for an example: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=72f510e1edcb7657ce46becab9179977

[camorimd]

Great article! Thank you so much :)

[phil-opp]

Thank you all for the nice feedback :).

@LordSparhawk

There is a repetition typo: "have to have to" in the first paragraph of Executors and Wakers

Thanks for reporting! Fixed in b6d09c8.

[phil-opp]

@ZoeyR

Isn't the uniqueness of TaskId not be guaranteed for zero-sized futures? Box would give the same address for multiple "instances" of the zero sized type.

Very good catch! While zero-sized futures are rare in practice, we should still ensure that they can be used without problems. I opened #782 and #783 to create task IDs using a static counter instead.

[kennystrawnmusic]

Using the aforementioned shell script also allows for the creation of the following .vscode/tasks.json file to simplify the build process still further:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Build Kernel Binary",
            "type": "shell",
            "command": "cargo",
            "args": [
                "build",
                "--target=x86_64-unknown-blog_os.json",
                "-Zbuild-std=core,compiler_builtins",
                "-Zbuild-std-features=compiler-builtins-mem"
            ],
            "group": {
                "kind": "build",
                "isDefault": true
            }
        },
        {
            "label": "Build Boot Image",
            "type": "shell",
            "command": "./build.sh",
            "args": [
                "x86_64-unknown-blog_os.json"
            ],
            "group": {
                "kind": "build",
                "isDefault": true
            }
        },
        {
            "label": "Run Legacy Boot Image In QEMU",
            "type": "shell",
            "command": "qemu-system-x86_64",
            "args": [
                "-drive",
                "format=raw,file=target/release/boot-bios-blog_os.img"
            ],
            "group": {
                "kind": "build",
                "isDefault": true
            }
        },
        {
            "label": "Run UEFI Boot Image In QEMU",
            "type": "shell",
            "command": "qemu-system-x86_64",
            "args": [
                "-bios",
                "/usr/share/edk2-ovmf/x64/OVMF.fd",
                "-drive",
                "format=raw,file=target/release/boot-uefi-blog_os.img"
            ],
            "group": {
                "kind": "build",
                "isDefault": true
            }
        }
    ]
}

With this configured, there's no manual command-typing at all ― just a simple Ctrl-Shift-B keystroke and the use of either the mouse or the arrow keys to select which build task you want to run (and this can also be added to in order to allow the automation of custom cargo test, etc). Another major benefit of this approach is that it can completely replace aliases.

[kennystrawnmusic]

Well, guys, at least we now can see the progress on the edition-3 branch. It’s still a major WIP though — all posts except the first and second are nothing but “lorem ipsum” at the moment, but now it can at least be contributed to through PRs.

[kennystrawnmusic]

And just like that, a new pull request to add useful information to edition-3 has now been submitted.

[kuqgp]

What modifications are needed to divide the virtual memory into kernel space and user space and how context switch can be done.

[Merlin1846]

We use destructuring to split self into its three fields to avoid some borrow checker errors. Namely, our implementation needs to access the self.task_queue from within a closure, which currently tries to borrow self completely. This is a fundamental borrow checker issue that will be resolved when RFC 2229 is implemented.

This RFC was implemented.

[Merlin1846]

Sad to not see user space and file system here as those were the two I was looking forwards for.

[kennystrawnmusic]

There's a third edition of the entire tutorial that @phil-opp is working on which will include them if I'm not mistaken.

[Merlin1846]

Oh good 😄

[tsatke]

Additionally, there's a few rust projects out there that have one or even both already implemented.

[Merlin1846]

Yeah, I was looking at MOROS but was having trouble figuring it out.

[tsatke]

@Merlin1846 you can figure out an own implementation of a filesystem. There's a lot of good documentation on linux's node graph concept e.g., and I find it somewhat intuitive. Another approach would be windows' mount list-ish approach.
Or you check out Redox for their file system implementation.
I personally would recommend to go for a basic ext2 implementation using a node graph.

Regarding user space, you need to do a few things to be able to make it work.
Not saying that I managed to do it, but for example, you need to set up new segments in your GDT. https://github.com/vinc/moros/blob/da9fca121bb87d69e908b207acd0469dc2cdd9b9/src/sys/gdt.rs#L43-L44
(assuming you have multitasking) On a task switch (context switch), you would have to determine whether you need to switch to user space.
I don't think MOROS has preemptive multitasking, so you won't find anything there (but maybe in eduOS-rs).

On the general topic of user space, there's some amount of information on the osdev wiki. https://wiki.osdev.org/Getting_to_Ring_3

For a basic setup, you probably don't need separate address spaces, but that sure would make sense eventually. You could get into that by reading about the x86 CR3 register (the basic idea being that you create separate memory mappings for different processes or process groups, with the kernel memory being shared between all of them).

That's only what I understood from my own research and kernel, and might be wrong.

[sytzemeijer]

Please write on

[sinjenda]

will there be a new post?

[phil-opp]

I'm currently working on a new third edition of the blog with UEFI and framebuffer support. So yes, there will be new content, but I don't know when it will be ready.

[Joelcgx]

How do I remove the dots that appear when launching the kernel?

[phil-opp]

These are from the timer interrupt handler. Just remove this print! call:

blog_os/src/interrupts.rs

Line 75 in c9cdf56

print!(".");

[Portgas-D-Asce]

ddd

[Croug]

Just a quick note, the OnceCell struct is now included at core::cell::OnceCell

[Croug]

Actually looks like they've split it into thread safe and non thread safe, the thread safe version is called OnceLock

[Croug]

and I suppose I should've tried it before commenting because it seems that OnceLock isn't available in core, only in std.

[ebkalderon]

Yeah, it seems that conquer_once::OnceCell is more akin to std::sync::OnceLock, except that conquer_once::OnceCell works with #![no_std]. On the other hand, std::sync::OnceLock relies on std::sync::Once, which in turn depends on an operating system.

[sampathsris]

Not sure if anyone's still following this post, but turns out after making the changes in this post, cargo test would hang. The reason obviously is that executor.run() never returns, so test_main() which is the line immediately after that is unreachable code (So is the println and the hlt_loop).

To fix this what I did was:

// main.rs
#[cfg(test)]
async fn invoke_test_main() {
    test_main();
}

pub fn kernel_main(boot_info: &'static BootInfo) -> ! {
    // ....
    #[cfg(test)] // new
    executor.spawn(Task::new(invoke_test_main())); // new

    // this will be the returning expression in kernel_main (note the missing semicolon)
    executor.run()
}

[yijun-tang]

// in src/main.rs

use blog_os::task::executor::Executor; // new

fn kernel_main(boot_info: &'static BootInfo) -> ! {
    // […] initialization routines, including init_heap, test_main

    let mut executor = Executor::new(); // new
    executor.spawn(Task::new(example_task()));
    executor.spawn(Task::new(keyboard::print_keypresses()));
    executor.run();
}

Actually, author moved the test_main above the executor though.

[ebkalderon]

Small correction:

this would never happen since our keyboard_task never finishes

No such async keyboard_task was ever mentioned before. I presume you instead meant to write here:

this would never happen since our keyboard::print_keypresses() task never finishes

[phil-opp]

Yes, should be keyboard::print_keypresses(). Do you want to create a PR for this? The post is rendered from this file: https://github.com/phil-opp/blog_os/blob/main/blog/content/edition-2/posts/12-async-await/index.md

[ebkalderon]

Sure, I can open a PR! Thanks for the guidance, and awesome blog BTW. ❤️

[phil-opp]

Thanks!

[yijun-tang]

Since our ScancodeStream implementation registers wakers before checking whether a task needs to be put to sleep, it might happen that a wake-up occurs for a task that no longer exists.

Is there anyone can help me understand this sentence?

[phil-opp]

We call WAKER.register(&cx.waker()) before calling queue.pop(). If the queue isn't empty, the task can continue and doesn't need to be woken. So we call WAKER.take() in that case to remove the registered waker again. However, when the user presses a key in between the register and take calls, we instruct the executor to wake up the task, which was never paused. In that case, the executor doesn't find the corresponding task because it's still running. In that case, we simply ignore the wake up signal in the executor.

[yijun-tang]

Thanks for your detailed explanation!

[MatthewG4mz3r]

are you going to continue the blog?

[tsatke]

Yes, see #979 #1025 #1063 #1279 (and probably more).

[GCPanic]

Considering we now using async, is it worthy to use tracing for logging? Or is tracing is too hard to implement or too expensive to using in kernel space?

[AtomicGamer9523]

You may, but I would refrain from using it. I prefer something lighter.