pine64-rock64: Add support for ROCK64

[moosingin3space]

As mainline U-Boot contains support for the ROCK64, I added support for it to Tow-Boot.

Fixes #151

[samueldr]

Hi! 👋

Thank you for your contribution!

I know there's no procedure to follow for contributions (yet), so please bear with these few questions:

Can you tell me what was tested?

• shared storage [booting the SPI installer from SD from a clean slate would test this]
• dedicated storage
• SPI installer

cc @Absolucy who asked about ROCK64 support previously.

[samueldr]

Right, there's a few commits that needs to be merged back into development, that's why changing the target branch to development floated them up.

Updated development.

New changes happens in the development branch, and ends-up in released on new release. So it's why I messed a bit with the target branch.

[moosingin3space]

I actually have to admit I'm having some trouble compiling it. Wanted to open the PR so that people watching #151 would be aware it's in progress, and to ask for help.

It's failing with the following config messages:

ERROR: CONFIG_BMP_16BPP is expected to be set to «CONFIG_BMP_16BPP=y», but is not present in config file.
ERROR: CONFIG_BMP_24BPP is expected to be set to «CONFIG_BMP_24BPP=y», but is not present in config file.
ERROR: CONFIG_BMP_32BPP is expected to be set to «CONFIG_BMP_32BPP=y», but is not present in config file.
ERROR: CONFIG_CMD_BMP is expected to be set to «CONFIG_CMD_BMP=y», but is not present in config file.
Warning: CONFIG_SPL_ENV_IS_NOWHERE is expected to be set to «CONFIG_SPL_ENV_IS_NOWHERE=y», but is not present in config file.
ERROR: CONFIG_SPL_SPI_FLASH_SFDP_SUPPORT is expected to be set to «CONFIG_SPL_SPI_FLASH_SFDP_SUPPORT=y», but is not present in config file.
ERROR: CONFIG_SYS_SPI_U_BOOT_OFFS is expected to be set to «CONFIG_SYS_SPI_U_BOOT_OFFS=0x80000», but is not present in config file.
Warning: CONFIG_TPL_ENV_IS_NOWHERE is expected to be set to «CONFIG_TPL_ENV_IS_NOWHERE=y», but is not present in config file.      

[samueldr]

I see two general issues in there, for which I need to ask questions:

• Does upstream U-Boot support HDMI output on the ROCK64 (or rk3328)?
• I assume there's no real support for SPI flash with the ROCK64 yet, given this other somewhat related PR

cc @mirrexagon, I kinda forget what the actual state of what you were doing is.

[moosingin3space]

• HDMI output: looks like it's in the device tree for u-boot: https://github.com/u-boot/u-boot/blob/07043055280a91d769ffe5e30d214241bd2e9e78/arch/arm/dts/rk3328.dtsi#L710
• SPI flash: I followed https://github.com/ayufan-rock64/linux-build/blob/master/recipes/flash-spi.md a long time ago when I first got my ROCK64. That said, reading what I see in your link suggests that I might be misinformed here.

[mirrexagon]

Regarding SPI flash:

• Currently, upstream U-Boot doesn't build SPI images for any Rockchip target basically all Rockchip targets (there's some maybe broken SPI image in there that comes out as u-boot.rom for RK3399), you have to do it manually.
• ROCKPro64 SPI images can be built using the method documented upstream, using mkimage, and they work.
• ROCK64 SPI boot support is not in upstream currently AFAIK, though I only recently found out the RK3328 boots rksd-format images from SPI, not rkspi format [1], and I haven't retested upstream to see if SPI images built the same way as ROCKPro64 but rksd format work. I suspect that it still needs some changes that Ayufan did on his fork, more on that below.
• My patches to get general SPI flash images building in upstream U-Boot [2] are going pretty slowly, and won't have support for the ROCK64 initially (just ROCKPro64 as a known working board).
• After that, I'm planning to upstream ROCK64 SPI image support, for which I applied what Ayufan did and tried that and it worked for me - see last commit of https://github.com/mirrexagon/u-boot/tree/rock64-spi which combines Ayufan's changes and my SPI image building changes to build for ROCK64.

[1] I don't remember exactly where I saw it, but maybe one of the patches referenced in a review someone did of my patches notes that the rkspi format that puts data in only the first 2K out of every 4K block thing is to work around a boot ROM bug, so perhaps the bug exists on RK3399 but not RK3328. EDIT: Found it: https://lore.kernel.org/u-boot/CAMdYzYqAdYTOOwjXsYJ_wUm7P71V-EDfqRabSMcsKgHEPTPLsg@mail.gmail.com/
[2] https://lists.denx.de/pipermail/u-boot/2022-May/484193.html

[samueldr]

Thank you for the brain dump.

About [1], The documentation bit was previously in README.rockchip, blame'd to its introduction in 2015:

• https://source.denx.de/u-boot/u-boot/-/commit/adfb2bfe50bba29e59fd04a81f0a9bbebdd203ce#5c5212e8f472c00c85d2d14f744d5e0e2b4314de_0_130

Which added support for RK3288.

[samueldr]

Tentatively putting in the milestone for the next release. It might be possible to do this as I was provided with a ROCK64 by someone (not sure if they wanted to stay anonymous) to support the platform.

I'll try poking at it soon, once I have the Tow-Boot patches rebased on top of 2022.07-rc_ and once the other PRs are handled.

[samueldr]

About HDMI: there's no rk3328 HDMI driver in U-Boot. So this won't be a trivial thing to get going.

I don't really know the platform yet, but if the subsystems on the 3328 are close enough to 3399, it might be possible to handle. Though a quick glance at the Linux driver makes me think it's going to take some work or someone that already is familiar with it.

[Absolucy]

Note that I'm getting the same build failure as @moosingin3space was, and I can't really test this as a result.

[samueldr]

I'll post an update soon once I better get the subtleties of SPI on the Rock64. I have the shared image going already.

[samueldr]

@moosingin3space I went ahead and:

• split your commit in three distinct commits
• finished Rock64 support as co-author
• made some necessary changes around the rockchip builder

I took the liberty of force-pushing on this PR branch, your changes (except for the rock64 commit) are untouched. The rock64 commit only has relevant additions.

If you have any concern with the changes, don't hesitate to ask.

[samueldr]

What works

• Tow-Boot starts from SD
• Tow-Boot starts from SPI
• Tow-Boot can be installed from the installer (using the serial console)
• Saving environment to SPI (with SPI builds)
• USB mass storage (on any of the three ports) [fix incoming]
• Booting an OS from USB

What is untested

• Booting an OS from SD (assumed working)
• Booting an OS from eMMC (assumed working)

What doesn't work

• Display output (expected, no upstream driver / in Tow-Boot / verified working in Linux)

[samueldr]

I've looked quickly at HDMI, and reading the Linux drivers, it differs too much from the existing RK3288 and RK3399 implementations to be a trivial thing to implement.

So for the time being, it'll have to be used through serial :/.

---

How does it look in the vendor BSP?

They have a totally different driver model for video in the Rockchip BSP. It's named drm, so I suspect they're sharing some of the implementation with their BSP-flavoured Linux drivers.

[moosingin3space]

TLDR, the main issue is that there's no power to the USB ports. Using a USB power monitor, the power monitor doesn't even turn on. With the appropriate changes it turns on.

Okay, that's pretty interesting. With this fix, I was able to boot the Fedora Server 36 netinstall image when plugged into one of the USB 2 ports. USB 3, indeed, does not work, but my drive is powered on this time. Looks like it's a bug in the u-boot XHCI driver.

Thankfully, it appears as though someone has already found a solution to the Linux bug: https://artemis.sh/2022/05/16/everfrees-armferno.html (mentioned under the Device Trees header, looks like it's slated for 5.19). Based on hearing this, I might just build a minimal image with Buildroot or Yocto that uses Linux 5.19 for the sake of testing USB on this platform.

[moosingin3space]

Aside, it appears as though JTAG headers are exposed on the Rock64: https://crwulff.blogspot.com/p/rock64.html

This could help significantly with debugging some of these drivers.

[samueldr]

The underlying issue is a bit more subtle than implied in that post.

The problem is that the innate FDT (the built-in device tree) is built for U-Boot, with quirks added. There is a quirk added by U-Boot disabling the always-on USB power. This is what the patch from ayufan works around within U-Boot.

So a workaround is to load the device tree from a mainline Linux dtbs folder for the rock64 before booting Linux. How to do so differs depending on the boot method used. Some methods are documented in distro.rst.

A solution would be to implement a generic method of "unquirking" a for-U-Boot device tree before booting an operating system. Doing so would be pretty much equivalent to loading the mainline Linux device trees as they are regularly synced in U-Boot with the mainline Linux trees. This would be helpful to keep the concerns separate about the U-Boot-specific device tree, and the generic innate device tree.

[Absolucy]

I may be dumb, but I can't seem to get any serial output?

Tried flashing both spi.installer.img and shared.disk-image.img to a microSD. ROCK64 turns on, but no output over UART. 112500 baud rate and all.

edit: i think my cables are just too long for uart?

[samueldr]

It's 115200, if that wasn't a typo. Though it's plausible that length can be an issue for UART, too. Also check if RX/TX weren't swapped, it's IMO always weird with the labeling.

[moosingin3space]

Thanks for the explanation. I have a number of questions, as my experience with U-Boot isn't that extensive.

The problem is that the innate FDT (the built-in device tree) is build for U-Boot, with quirks added.

So basically, U-Boot loads a device tree that has certain quirks, ostensibly to account for differences between U-Boot and Linux drivers, and then Linux loads the same one from some specified address?

So a workaround is to load the device tree from a mainline Linux dtbs folder for the rock64 before booting Linux. How to do so differs depending on the boot method used. Some methods are documented in distro.rst.

To clarify: in the case where I have a UEFI-booting distribution image (i.e. Fedora), this implies that U-boot must load the device tree? I was under the impression that in the case of something like the Fedora AArch64 generic image, GRUB (the bootloader it ships with) would handle booting Linux with the mainline device tree.

Perhaps I just need to build a minimal distro with Yocto (or even Nix -- I noticed your work on linux-3ds.nix, that's really cool 😄) and drop into an interactive console to boot it, just so I can see how that's supposed to work.

A solution would be to implement a generic method of "unquirking" a for-U-Boot device tree before booting an operating system. Doing so would be pretty much equivalent to loading the mainline Linux device trees as they are regularly synced in U-Boot with the mainline Linux trees. This would be helpful to keep the concerns separate about the U-Boot-specific device tree, and the generic innate device tree.

This would be something done upstream, I assume. Would it basically result in the same effect (loading the mainline device tree when booting Linux)? Or would it move the responsibility into the OS?

[samueldr]

Those are "heavy" questions, and this will derail the discussion a bit, and possibly exponentially grow. Though let's try and answer them. Some of them will stretch my verified knowledge and might slip into assumptions.

A lot of the answers will be about my opinionated ideals that are currently mostly fiction in the ecosystem.

---

The problem is that the innate FDT (the built-in device tree) is build for U-Boot, with quirks added.

So basically, U-Boot loads a device tree that has certain quirks, ostensibly to account for differences between U-Boot and Linux drivers, and then Linux loads the same one from some specified address?

Yeah~ish. Here U-Boot does the loading, and tells the OS (Linux here) where the FDT is in memory. When there is no "loading" done, e.g. with UEFI by default, U-Boot will re-use what I call the "innate" FDT, the one that U-Boot was built with.

We'll see that Linux completely absolving itself from the task of loading the correct dtb file is a recurring theme, and a problem.

So a workaround is to load the device tree from a mainline Linux dtbs folder for the rock64 before booting Linux. How to do so differs depending on the boot method used. Some methods are documented in distro.rst.

To clarify: in the case where I have a UEFI-booting distribution image (i.e. Fedora), this implies that U-boot must load the device tree? I was under the impression that in the case of something like the Fedora AArch64 generic image, GRUB (the bootloader it ships with) would handle booting Linux with the mainline device tree.

Any component before the OS can load the FDT in memory. Though there is a severe dysfunction in that there is no way to know which dtb file to load. So when targeting generic boot, you can't get GRUB to e.g. load the Pinebook Pro dtb on the Pinebook Pro, GRUB (and pretty much all tools except U-Boot) have no way to ask "who am I loading this for?".

I have not verified what the actual Fedora iso does, I suspect either it does nothing special, or it adds the "ESP:/dtb" folder with the kernel-built device tree binaries. The latter is a sort of workaround that U-Boot people implemented to allow a "more correct" dtb file to be generally loaded. This is possible since it's U-Boot that will load the "correct" dtb file before starting the EFI program.

So, how does U-Boot does it? "Heuristics"!

Generally speaking, the platform name is either entirely hardcoded, or there is a small detection routine in a board-specific file that will distinguish between few compatible platforms (e.g. Pinephone A64).

[...]

A solution would be to implement a generic method of "unquirking" a for-U-Boot device tree before booting an operating system. Doing so would be pretty much equivalent to loading the mainline Linux device trees as they are regularly synced in U-Boot with the mainline Linux trees. This would be helpful to keep the concerns separate about the U-Boot-specific device tree, and the generic innate device tree.

This would be something done upstream, I assume. Would it basically result in the same effect (loading the mainline device tree when booting Linux)?

Development could happen here, but it should probably be coordinated with upstream so that it is acceptable to them.

The effect wouldn't exactly be "loading the mainline device tree when booting Linux", but it sure would look similar to it. The effect would be that we'd undo the "quirky" changes U-Boot applies (e.g. see rk3328-rock64-u-boot.dtsi). In turn, since U-Boot frequently syncs the device tree definitions with mainline Linux, it would in practice give the same ballpark result you were describing.

Or would it move the responsibility into the OS?

You hit the nail on the head. We're now diving into my opinion on the subject matter.

I think there is one deep flaw that needs to be solved with the trifecta Device Tree, Linux, Generic Boot: the kernel wants its own device tree representations, but does absolutely nothing to make it easy or possible. Everything has been authored with an extremely embedded mindset, where the platform is tightly vertically integrated.

I might be missing some things, but basically I think that dtb files shouldn't be handled by the bootloader stages, since they have no way to know which file to load (or a protocol devised to be implemented by the bootloaders).

One way I've thought up to handle this would be to bundle all dtb files from a kernel build in a cpio archive, and load it with the kernel "before" the initramfs.

The kernel in its earliest init would look at the current "facts" available about the platform, and somehow select the right dtb file. That somehow is load bearing. Note that the same exact system could be used for ACPI platforms to implement quirks with an "overlay" instead of peppering the whole tree with dmi_*_match quirks (or maybe Linux should treat DT equal to ACPI, and use dt_*_match to apply quirks in drivers instead of "fixing" the device tree).

What facts are there? I don't exactly know, but I have a few in mind. First, a new DT property could be added to short-circuit facts-checking e.g. /linux,dtbname, when present the kernel knows this is the "truth" and loads that file. That won't help for existing platforms, or platforms that don't implement that extension to DT. In these cases, looking at /compatible should and /model and fudge around hashes could be done to look up which file to load. But that isn't enough as many platforms ship with inadequate names here. For these it might be best to look at "unique enough" properties e.g. /compatible × &panel/compatible, add a couple other data point that uniquely identify the platform and there you have it.

With such a scheme, you would resolve I believe all Linux-side pains with ARM and Generic Boot. It would be possible to implement the same scheme in a bootloader step, the same strategy is used for WoA laptops [citation needed], but the solution needs to be applied as close as possible to the kernel or else it continues being an issue to any platforms not using the special solution. Another issue with a bootloader-y step is that for OS with generations like NixOS, you might face an issue where you need to first load the "correct" dtb file, but the "correct" dtb file will vary depending on which kernel/system generation you boot next. This would also affect other similar strategies, like I think SUSE does with Snapper.

---

I moved this from the response, as I don't know how to interpret this:

Perhaps I just need to build a minimal distro with Yocto (or even Nix -- I noticed your work on linux-3ds.nix, that's really cool smile) and drop into an interactive console to boot it, just so I can see how that's supposed to work.

This seems unrelated to the FDT/boot questions. Or do you mean that you want to know how to manually boot an OS using the U-Boot console? It won't be as helpful as you'd hope if you want to boot with EFI, as you'd basically load the EFI program, and run it. And otherwise the boot process is different enough when booted through U-Boot native methods that it won't exactly map to EFI boot semantics.

Note that the knowledge learned with the linux-3ds.nix shenanigans is part of celun, which you can see a use of with the embedded Linux OS build for the touch installer for Tow-Boot.

---

Sorry for the long rambling! If anything needs to be discussed still... I don't know where it should be... But there's the matrix room (#Tow-Boot:matrix.org), where it might be easier to bounce smaller bite-sized questions/answers instead of huge essays :).

[moosingin3space]

Thank you for the incredibly detailed response, you are teaching me quite a bit about the way that these SBCs work. My experience in this area has been heavily focused on the embedded Linux side, after U-Boot loads an ITB-formatted image with kernel, initramfs, and device tree. That's a very... embedded systems way to do it, versus the generic boot mechanism that you're describing, and that Tow-Boot aims to provide.

---

Bringing this conversation back to this specific PR's goal, bringing up Tow-Boot on a ROCK64, and the questions regarding booting a Fedora Server ISO, I opened the ISO with guestfish and managed to find this:

><fs> cat /EFI/BOOT/grub.cfg
set default="1"

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2

set timeout=60
### END /etc/grub.d/00_header ###

search --no-floppy --set=root -l 'Fedora-S-dvd-aarch64-36'

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Install Fedora 36' --class red --class gnu-linux --class gnu --class os {
	linux /images/pxeboot/vmlinuz inst.stage2=hd:LABEL=Fedora-S-dvd-aarch64-36 ro
	initrd /images/pxeboot/initrd.img
}
menuentry 'Test this media & install Fedora 36' --class red --class gnu-linux --class gnu --class os {
	linux /images/pxeboot/vmlinuz inst.stage2=hd:LABEL=Fedora-S-dvd-aarch64-36 rd.live.check
	initrd /images/pxeboot/initrd.img
}
submenu 'Troubleshooting -->' {
	menuentry 'Install Fedora 36 in basic graphics mode' --class red --class gnu-linux --class gnu --class os {
		linux /images/pxeboot/vmlinuz inst.stage2=hd:LABEL=Fedora-S-dvd-aarch64-36 nomodeset
		initrd /images/pxeboot/initrd.img
	}
	menuentry 'Rescue a Fedora system' --class red --class gnu-linux --class gnu --class os {
		linux /images/pxeboot/vmlinuz inst.stage2=hd:LABEL=Fedora-S-dvd-aarch64-36 inst.rescue
		initrd /images/pxeboot/initrd.img
	}
}

I can't tell, from this, how it decides what device tree to load. Decided to look at the boot loader for the raw image:

><fs> cat /boot/loader/entries/e18134ba4d124da581bbd0eb9114b13f-5.17.5-300.fc36.aarch64.conf
title Fedora Linux (5.17.5-300.fc36.aarch64) 36 (Server Edition)
version 5.17.5-300.fc36.aarch64
linux /vmlinuz-5.17.5-300.fc36.aarch64
initrd /initramfs-5.17.5-300.fc36.aarch64.img
options root=/dev/mapper/fedora_fedora-root ro rd.lvm.lv=fedora_fedora/root rhgb quiet
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora

This doesn't appear to specify a device tree either. It looks to me as if Fedora, when booted with UEFI on AArch64, expects to be able to find ACPI tables. There's definitely a device tree binary located on the boot partition at /boot/dtb/rockchip/rk3328-rock64.dtb, but it doesn't look like Grub passes that in.

So I guess the next question is: what's next for this PR?

[samueldr]

So I guess the next question is: what's next for this PR?

I'm not sure. I don't like the fact that the innate FDT is "wrong" as far as Linux is concerned, but it's not like it's something that's caused by changes from Tow-Boot. So I should ignore that detail since it's the same with upstream.

I don't like that HDMI out doesn't work... but same story, upstream problem.

So really, it's probably as ready as can be.

[samueldr]

[cut down fedora aarch64 exposition]

This doesn't appear to specify a device tree either. It looks to me as if Fedora, when booted with UEFI on AArch64, expects to be able to find ACPI tables. There's definitely a device tree binary located on the boot partition at /boot/dtb/rockchip/rk3328-rock64.dtb, but it doesn't look like Grub passes that in.

Fedora AFAIK on AArch64 "expects" ACPI tables just as much as other "standards-based boot" distros do, really DT is fine too.

Is the boot partition you're speaking of the ESP? If so, U-Boot itself (and thus Tow-Boot) should load the dtb file according to its heuristics before starting the UEFI program.

• https://source.denx.de/u-boot/u-boot/-/blob/v2022.01/include/config_distro_bootcmd.h#L150-164

There are no mechanisms available after U-Boot relinquishes control to determine the "correct" dtb file name to load. So any UEFI program loading a dtb file would need to hardcode the platform, so it's not an option for generic boot images (and should be seen as a workaround for device-specific systems).

[moosingin3space]

Is the boot partition you're speaking of the ESP? If so, U-Boot itself (and thus Tow-Boot) should load the dtb file according to its heuristics before starting the UEFI program.

It is not - the ESP contains grub with a grub config that mounts a different (XFS) partition. That XFS partition contains the dtb. I am not sure why it doesn't just use the ESP for storing all the boot resources.

Does this imply that if I were booting a distribution that placed the dtbs on the ESP, U-Boot should load Linux's device tree according to heuristics?

[samueldr]

Yes, this is precisely what the code aims to do.

Note that the dtb folder (or dtb/current folder) has to be structured like the dtb folder from a Linux kernel build, so it will look something like ESP:/dtb/rockchip/rk3328-[...].dtb.

[samueldr]

❌ This can't be merged as-is.

A discussion with @mirrexagon and other peeps in the NixOS on ARM room floated up an issue I had forgotten about: there are variants that needs slower RAM init, or else the board might exhibit issues.

See this discussion:

• use unmodified (blob-free) U-Boot for the Rock64 board NixOS/nixpkgs#145506

So this will require fishing-up the proprietary ram training bits from other WIP branches.

[moosingin3space]

Hi, sorry for the silence. I will have to look through ayufan's commits to see if there are any patches relating to RAM init across variants, or even detection of the variants with this requirement.

[samueldr]

Moving this in 007 to expedite 006 with finished work.

Also noting that downstream-upstream there are further changes related to RAM init that we might want to look at

• u-boot: ROCK64 RAM init improvements NixOS/nixpkgs#191538

Namely, I wasn't aware of it for lpddr3, mainline U-Boot already has multiple blobs for RAM init with different speeds available. I assumed there wasn't from experience with #54 where it's not the case for lpddr4.

So I want to take proper care to verify this.

We'll also need to figure out how we handle differing RAM init...

Do we build multiple builds? For now I guess we can build with the single slower RAM init and push that question as something to be answered later. Maybe an "applet" within the future configuration menu that can be used to replace the blob in-situ? Though that would be hard and prone to errors.

[moosingin3space]

We'll also need to figure out how we handle differing RAM init...

Is there documentation from Pine64 (or Rockchip) about how the different variants require different raminit?

[samueldr]

I don't know of any. Tried looking, and no official word AFAIK. It would be from Pine64.

There are multiple locations stating v2 is the problematic revision.

¯\_(ツ)_/¯

[matdibu]

At this time, what are the remaining issues before this is in an acceptable state?

From what I understood:

1. SPI does not work (yet? https://github.com/mirrexagon/u-boot/tree/rock64-spi isn't merged) - comment
2. there needs to be a way to do RAM training for multiple board variants - comment
3. HDMI is in the dt, but it's missing the actual driver in u-boot - comment

Looking at https://wiki.pine64.org/wiki/ROCK64#Information,_Schematics_and_Certifications, I can see two board variants (2.0 and 3.0) and 3 RAM variants (Hynix, Samsung, Spectek)

I have a ROCK64 2GB, but I'm not sure which variant it is; is there anything I can help with?