Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fastboot for fusee: a workflow improvement #1209

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

misson20000
Copy link
Contributor

This PR adds a USB gadget to fusee-primary to allow users to download files to the SD card and proceed to boot in a way that is robust and scriptable (compare to USB mass storage; it is hard to script to wait for device, then proceed with filesystem mount/copy/unmount, etc.). The USB device stack is based on the XUSB controller, so it should work on Mariko in theory.

It is intended to improve Atmosphere development workflow. Instead of ejecting SD card and copying files over every time you make a change, you can instead set your configuration file to enable fastboot. Next time you boot, you can run make flash and the device will wait until your build completes and flashes a fresh filesystem onto the SD card, and then it can proceed to boot as normal.

Isn't fastboot an Android thing? This isn't Android! What's it doing in Atmosphere?!

It's a simple USB protocol that does what we need it to do, and it comes with a client for all three major desktop operating systems. This means we get to avoid writing, debugging, maintaining, and teaching people how to install our USB code for three different platforms. Plus, fastboot has seen some limited use outside of Android in e.g. Tizen. (admittedly, not all that different from Android)

That being said, the client-side tool does have a lot of Android-isms in it. It would be great to be able to extract a zip file over the filesystem with fastboot update, but the tool tries to verify that the zip file is a proper Android update. The fastboot boot command tries to create a "boot image", which is supposed to include kernel and ramdisk and other such things. It's not a great match, but it works and imo beats maintaining desktop USB code.

Isn't this bloat in fusee-primary?

Yes, but I wanted to be able to overwrite and reload fusee-secondary in place, which I wouldn't have been able to do if I implemented this in fusee-secondary. If we really want it in fusee-secondary, that can be arranged.

Usage

There is a new BCT configuration section:

# atmosphere/config/BCT.ini
[fastboot]
force_enable = 0
button_timeout_ms = 3000

The force_enable key will cause fusee-primary to always enter Fastboot. The button_timeout_ms key will offer a period of time for the user to press the volume up button to enter fastboot on each boot. If it is set to zero (the default), it will not prompt to enter fastboot.

From there, it enables the USB gadget and it can be accessed from the host. These are the supported fastboot usages:

# testing
$ fastboot getvar version
$ fastboot stage <file> # downloads a file to RAM
$ fastboot oem crc32 # returns crc32 of downloaded file, for testing transfer integrity

# flashing
$ fastboot -S 2G flash sd <image> # writes image directly to SD card
$ fastboot -S 2G flash ams <dist>.zip # extracts zip file to SD filesystem

# booting
$ fastboot boot --base 0xf0000000 --kernel-offset 0 --page-size 4096 --header-version 2 fusee-secondary.bin # sideloads fusee-secondary
$ fastboot reboot # reloads fusee-primary

I've also added a Makefile rule (make flash) to prepare and flash an image directly to the device. It uses GNU mtools to create a 64 MiB FAT filesystem in userspace. This size downloads pretty quickly and is enough for a base atmosphere install. It will also include any files under the extras/ directory, which can be used to override the default config templates or symlink in custom sysmodules.

There is currently some issue with Mac hosts where it will hang after one fastboot command. I don't have consistent enough access to a Mac device to debug this myself.

Review

This PR is meant to be reviewed one commit at a time. There are a lot of small changes to fusee-primary, before the main commit that includes the bulk of the USB stack and gadget. After that are various commits adding more features, which should each be able to be easily removed.

Apologies to hexkyz for the conflicts if this gets merged before the mariko_fusee branch.

…rgets to get rebuild when they did not need to be rebuilt
The SDMMC driver is currently the only user of the AHB redirect, but the
upcoming XUSB driver will also need to use the AHB redirect, potentially at the
same time. This ensures that the AHB redirect will not be disabled unexpectedly
if one driver is deinitialized without the other.
Add dram section to link script, and also creates a framebuffer section so we
can start to use the link script as a single source of truth for how
fusee-primary uses dram.
BCT0 is now only parsed once into a struct this is shared between log
level logic and stage2 logic. This has been done in anticipation of
needing config items for fastboot gadget.
This code is essentially a giant unrolled loop over an array of register
copies. I've manually re-rolled the loop so that rather than emitting code for
each parameter to read it, perform bit arithmetic on it, and write it, there is
instead a static list of parameters which can be encoded using much less memory.
@SciresM
Copy link
Collaborator

SciresM commented Nov 21, 2020

Will review in more detail over the next few days.

Few things off the top of my head before then:

  • Instead of ejecting SD card and copying files over every time you make a change heh, I'm willing to accept this anyway, but this doesn't work very well with my experience of the traditional atmosphere development workflow :)
  • How big is fusee-primary with this included? What's our margin versus the maximum?
  • You mention "reloading" -- how does this work? Reminder that we cannot perform a reboot to payload on mariko.
  • extras rename this to your choice of user_config, user_overrides, overrides, user_content, or something similar that you ask me about on discord.
  • "android-isms", not an issue so long as they're kept out of core logic.

I guess the other big thing I have to ask is this:

  • How do you see migration to a future C++ fusee rewrite which uses libexosphere going? Easy to do, difficult to do? Assuming that fusee's style was migrated to be like the rest of atmosphere (and that fusee basically became nx-bootloader-like, internally), how much work do you think it would be to get this stuff migrated?

If this gets merged before the mariko_fusee branch -- it'll be merged after.

Basically no big features are getting in of any kind before mariko is merged/in, because of the risks of the mariko stuff becoming broken/hard to merge especially with possible sysupdates in the near future.

SciresM
SciresM previously approved these changes Nov 21, 2020
Copy link
Collaborator

@SciresM SciresM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming this all works, this commit/build changes seems fine to me.

@SciresM
Copy link
Collaborator

SciresM commented Nov 21, 2020

Ugh @ approval for specific commit approving whole thing. I'll undo that, why is GitHub's UI such ass

@SciresM SciresM self-requested a review November 21, 2020 02:36
@SciresM SciresM dismissed their stale review November 21, 2020 02:36

Intended for specific commit.

@misson20000
Copy link
Contributor Author

How big is fusee-primary with this included? What's our margin versus the maximum?

Memory region         Used Size  Region Size  %age Used
            NULL:          0 GB         4 KB      0.00%
            main:      123456 B       128 KB     94.19%
        low_iram:       27424 B        32 KB     83.69%
            dram:    12345984 B         1 GB      1.15%

You mention "reloading" -- how does this work? Reminder that we cannot perform a reboot to payload on mariko.

The make flash rule runs fastboot reboot, which just reboots to payload. It can be trivially changed to exit back to main loop and continue normal boot flow instead.

extras rename this to your choice of user_config, user_overrides, overrides, user_content, or something similar that you ask me about on discord.

Ok.

"android-isms", not an issue so long as they're kept out of core logic.

They only really appear here, where I have to unpack the boot image format that fastboot boot uses.

How do you see migration to a future C++ fusee rewrite which uses libexosphere going? Easy to do, difficult to do? Assuming that fusee's style was migrated to be like the rest of atmosphere (and that fusee basically became nx-bootloader-like, internally), how much work do you think it would be to get this stuff migrated?

Easy. Change the register access patterns to whatever libexosphere has settled on, and the rest of it I've tried to keep pretty close to the rest of Atmosphere's C++ style already.

@misson20000
Copy link
Contributor Author

Documenting some conversation with SciresM and some ideas I had:

  • Mtools isn't going to work on Windows and working on Windows is really important
    • Maybe we could have a python script for building images?
  • We will need more space for Mariko support in fusee.
    • Compress sdram tables?
    • Move some xusb structures to dram? I had some issues with this earlier but it might be possible.
    • Remove zip file support?
  • Paving over Nintendo folder is not great, but I want to enforce clean slate for consistent testing environment. Maybe delete everything except Nintendo folder (or only clean atmosphere/,sept/ folders? make it an option?) and copy over files from a different kind of image. Combined with the zip file note from earlier, maybe uncompressed tarballs are a better way to go?
  • Image creation/flashing should maybe be a shell function instead of a makefile target so you can run it from in-tree without Make having to rescan all the dependencies, since apparently this takes a while on Windows with recursive make.

The intent behind paving over the entire SD card in a single image and having flash be a makefile target was to ensure that the testing environment is consistent and predictable; we can be reasonably sure that there are no stray files left over from a previous build, that the filesystem has not been corrupted by some rogue or driver, and that the entire CFW was built from the same revision of the code, without having components mixed and matched. This comes from some paranoia I have over having been bitten by missing header file dependencies or the like too many times.

In practice, the overhead that comes from ensuring these guarantees may not be worth it.

@urherenow
Copy link

FWIW, I think the the hot key should be volume down. People use volume up to enter RCM. Many have dongles for convenience, causing the boot process to start almost immediately after power on, so using volume up could send people into fastboot when it was not intended (not letting go of volume up fast enough).

@misson20000
Copy link
Contributor Author

FWIW, I think the the hot key should be volume down. People use volume up to enter RCM. Many have dongles for convenience, causing the boot process to start almost immediately after power on, so using volume up could send people into fastboot when it was not intended (not letting go of volume up fast enough).

It checks for a not-pressed -> pressed transition instead of just whether the button is pressed. ^^

@ghost
Copy link

ghost commented Apr 12, 2021

It would be great not only have fastboot but also search on start for update.zip in root folder of sdcard and auto apply it.

@misson20000
Copy link
Contributor Author

It would be great not only have fastboot but also search on start for update.zip in root folder of sdcard and auto apply it.

@SciresM solution to the live updating from userland problem? ^

@SciresM
Copy link
Collaborator

SciresM commented Apr 12, 2021

I don't want on-device updating in the main atmosphere repo's codebase, still.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants