Disable the Red Zone (Extra Post)

[utterances-bot]

This is a general purpose comment thread for the “Disable the Red Zone” post.

[luojia65]

There is one small thing that I cannot understand: what does leaf function mean? Does it implies that the two instructions can be optimized when it comes to tail recursive optimization? Thank you!

[phil-opp]

A leaf function is a function that does not call other functions, i.e. a leaf node in the call graph of the program. A tail recursive function that doesn't call other functions can be turned into a leaf function, because tail recursion can be turned into a loop. Hope this helps!

[helinwang]

Thanks for the article.
As you mentioned:

this optimization leads to huge problems with exceptions or hardware interrupts

Why this optimization exists at all when it leads problem with exceptions or hardware interrupts? Are there situations that using red zone is absolutely safe? I thought "normal" code can have exceptions too.

[bjorn3]

When you are running in userspace, the kernel adjusts the stack pointer to take the red zone into account before calling the signal handler. The kernel itself tells the processor to switch to a different stack for the exception handler when calling the exception happened in userspace. Because this needs some extra setup, it isn't done in this OS. This means you still need to disable the red zone for the kernel.

[phil-opp]

For normal userspace applications this optimization is not a problem because the kernel usually switches to a completely different stack through a hardware mechanism whenever an interrupt or an exception occurs. Thus, no data of the userspace stack is overwritten, even with the red zone optimization.

Edit: Seems like bjorn3 beat me to it :).

[helinwang]

Got it, thanks for the explanations!

[Gnarus-G]

Is the System V ABI unrelated to the AMD64 ABI reference the first "red zone" link in the first paragraph? I ask because the claim there about the red-zone not being modified by interrupt handlers seems to contradict this one. That article reads,

Code can assume that the 128 bytes below rsp will not be asynchronously clobbered by signals or interrupt handlers"

[phil-opp]

Where do you see a contradiction? It's not a claim, it's a requirement.
The red zone optimization requires that interrupt handlers don't clobber the red zone. To ensure this, we need to disable the red zone usage for the interrupt handler code.
Userspace code can only use the red zone optimization because we guarantee that the interrupt handlers in our kernel won't clobber the red zone data.