WIP: Update buildroot #401

AaronDewes · 2020-07-04T09:06:42Z

This is not tested yet. This updates the included buildroot to version 2018.11, the last version with QT4.
Differences to the default buildroot 2018.11:

Add kernel 4.19 support
Update ca-certificates
Include PINN packages
Fixes for the RPi kernel

lurch · 2020-07-04T10:55:14Z

11,000 changed files! 👀 Holy moly, that's going to be fun to try and review 😉

Top tip: When making several changes, as you've done in your list above, it's easiest if each of those changes is in a separate commit, rather than squashing everything into a single commit. That way it's easier to pick out exactly what changes e.g. "Fixes for the RPi kernel" involves.

EDIT: But that's just my opinion of course, @procount may see things differently.

AaronDewes · 2020-07-04T12:27:37Z

I rebased it, I hope it is okay like this (I can also squash them again if that's too many commits).

buildroot/package/linux-headers/Config.in.host

AaronDewes · 2020-08-15T13:18:59Z

I think I found a way to get QT5, I'm still testing it, and many things still need to be done, but I got a working window system:

AaronDewes · 2020-08-21T18:26:45Z

I got QT5 working, there are still many tweaks needed, but I now got a working build system, and there is one issue:
It needs more resources, and can't use the cd (cut-down) firmware. As NOOBS and PINN use the name recovery, I'm not sure if this is okay for you. Also, the RAM usage increases, and PINN is about 20 MB bigger with QT5 (which also uses even more RAM).

@procount @lurch What are your opinions on this?

Note: This works in parts, but there are many things that still need to be done, but before putting even more work into this, I thought I should ask if you even accept these changes.

procount · 2020-08-21T23:26:25Z

What's the issue with the cutdown firmware? I've never really studied what the differences were between the different versions.

As I said before, I'd like to move PINN onto a later buildroot and QT5 so that I can use a more up to date version of Ubuntu to develop it on. It's a BIG job and I appreciate your work on this. Hopefully it won't be at the expense of too much bloating, because it is supposed to be small and nimble. Unfortunately, I've not had the time to put any effort into this, but at some point I need to, because I can't/won't integrate anything I don't understand or couldn't support. I therefore need to be able to follow along with what you've been doing when I can devote more time to it. So I would ask you to keep your PRs well documented and clear, with each one performing a specific change, and each change being integratable, so there is always a working version, as far as possible.

I can't guarantee when or if I can integrate these changes - it depends on my other commitments, and how well you progress. So if you want to continue, please do so primarily because YOU want to. I hope PINN continues to be useful, but with other advances like UEFI and devs wanting to use more of UBoot, etc, I do wonder how long it will last. Hopefully long enough to complete my roadmap of new features 😄

AaronDewes · 2020-08-22T06:02:53Z

Okay, I'll push more changes soon, I currently have about 20 commits just named "Fix". I currently got a working Window system. Here's a list of all changes to PINN packages I did and I still have to do:

Use a custom arora with bugfixes like in Fix certificate errors #411 and QT5 support
Remove QWS from PINN
Replace some QT4 features with QT5 features (toAscii was replaced by toLatin1)
Add a custom compositor
Create a way for recovery to change compositor settings

Because the Compositor uses later QT features, but the buildroot currently used here has an older qt5 version, so I need to do the buildroot update & QT5 port in one PR. I can split this into multiple commits, but not create multiple PRs.

procount · 2020-08-22T06:22:55Z

Sorry, yes, I meant commits not PRs.

AaronDewes · 2020-08-22T09:28:17Z

I uploaded the version I'm currently working on here.
The current version does not boot, there is still something needed, but I need to do a test build first.
Also, it currently starts a shell on boot, I will change that later, but this makes debugging easier.

The first commit is just what happens after deleting the buildroot folder and replacing it with buildroot 2020.08-rc1.
The commits after that are changes to buildroot which are already done here, and changes that needed to be done to get the latest QT working.

AaronDewes

I'm documenting the commits in more detail here.
I'll start with commit 137bca4

Update build script for latest buildroot

AaronDewes · 2020-08-22T10:35:02Z

BUILDME.sh

@@ -51,7 +51,7 @@ function update_github_package_version {

 function get_kernel_version {
    CONFIG_FILE=.config
-    CONFIG_VAR=BR2_LINUX_KERNEL_VERSION


This variable is no longer used for custom repos.

AaronDewes · 2020-08-22T10:35:51Z

BUILDME.sh

-                    CONFIG_VAR=BR2_LINUX_KERNEL_VERSION
-                    sed -ri "s/(^$CONFIG_VAR=\")[0-9a-f]+(\")$/\1$GITREV\2/" "$CONFIG_FILE"


Same as above, This variable is no longer used for custom repos.

AaronDewes · 2020-08-22T10:53:49Z

BUILDME.sh

+        # Fix for modular kernel: Force recreation of rootfs image to include modules
+        rm "$IMAGES_DIR/rootfs.squashfs"
+        make


The build order changed in the latest buildroot. If this is not done, the image doesn't contain the kernel modules.

AaronDewes

Commit 434ed06

Fix PINN for QT5 & latest buildroot

Use QT5 features, fix include paths, ...

AaronDewes · 2020-08-22T10:58:43Z

recovery/bootselectiondialog.cpp

@@ -31,7 +31,7 @@
 #include <QSettings>
 #include <QDesktopWidget>
 #include <QScreen>
-#include <QWSServer>
+// #include <QWSServer>


I will remove everything related to QWS later and replace it with a custom implementation that connects to the compositor and tells it to change its settings.

AaronDewes · 2020-08-22T10:59:52Z

recovery/bootselectiondialog.cpp

@@ -56,7 +56,7 @@ BootSelectionDialog::BootSelectionDialog(
    _dsi(dsi),
    ui(new Ui::BootSelectionDialog),
    _inSelection(false),
-    _drive(drive.toAscii())
+    _drive(drive.toLatin1())


No functional change, ttoAscii was replaced with toLatin1 in 5.0.

AaronDewes · 2020-08-22T11:00:26Z

recovery/confeditdialog.cpp

@@ -15,7 +15,7 @@

 #include <QDir>
 #include <QFile>
-#include <QtGui/QPlainTextEdit>
+#include <QPlainTextEdit>


This is no longer part of the QtGui module, but still available.

AaronDewes · 2020-08-22T11:01:17Z

recovery/json.cpp

+#include <qjson-qt5/parser.h>
+#include <qjson-qt5/serializer.h>


QJson is now called qjson-qt5, but functionality stays the same.

AaronDewes · 2020-08-22T11:03:16Z

recovery/recovery.pro

@@ -4,11 +4,9 @@
 #
 #-------------------------------------------------

-QT       += core gui network dbus
+QT       += core gui network dbus widgets multimedia


As the qjson path change already broke QT4 support, I removed the QT5 check and added widgets directly here, some part also used the multimedia module, but I don't remember where this was needed.

AaronDewes · 2020-08-22T11:05:33Z

recovery/recovery.pro

-greaterThan(QT_MAJOR_VERSION, 4): QT += widgets
-
-RPI_USERLAND_DIR=../../staging/usr
+RPI_USERLAND_DIR=../../host/arm-buildroot-linux-uclibcgnueabi/sysroot/usr


Staging is not inbuildroot/output/staging anymore (Buildroot creates a symlink when creating the final image, but not earlier).

AaronDewes · 2020-08-22T11:06:23Z

recovery/translation_ca.ts

@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
 <!DOCTYPE TS>
-<TS version="2.0" language="ca_ES">
+<TS version="2.1" language="ca_ES">


This is done by default in qt5, all changes to translation files were done automatically.

AaronDewes · 2020-08-23T07:27:43Z

@procount While trying to get this working, I found out that a complete change of the boot partition is needed, which would break the update system, so an update to prepare PINN for the next update would be needed if these changes are implemented (at least if I understand the update system correctly).
Alternatively, it would be possible to have the main PINN in a file with another name (like pinn.rfs instead of recovery.squashfs), and recovery.rfs has a custom init that performs all changes needed and reboots, so it can be done in one update.

The change is needed because having the firmware named recovery.elf prevents it from reading fixup.dat, which results in a lot of memory-related errors.

Old name	New name
recovery.elf	start.elf
recover4.elf	start4.elf
recovery.cmdline	cmdline.txt
(NEW FILE)	config.txt
(NEW FILE)	fixup.dat
(NEW FILE)	fixup4.dat
recovery*.img (kernel)	kernel*.img

procount · 2020-08-23T18:01:07Z

Is it not possible to run Qt5 without OpenGL(ES) and a compositor, e.g. just using a simple framebuffer like is used in Qt4?

AaronDewes · 2020-08-23T19:07:27Z

Actually, only a single window can be used without compositor and OpenGL, which won't work for PINN (browser, language selection, config, ...)

procount · 2020-08-23T19:52:04Z

That is surprising. According to https://doc.qt.io/qt-5/qpaintengine.html#details

"Qt provides several premade implementations of QPaintEngine for the different painter backends we support. The primary paint engine provided is the raster paint engine, which contains a software rasterizer which supports the full feature set on all supported platforms. "

AaronDewes · 2020-08-24T04:41:23Z

Haven't seen that yet. The compositor uses other OpenGL features, but maybe I can get it to work properly.

procount · 2020-08-24T06:57:19Z

It seems you are correct. Sounds like this is a backward step by qt in my opinion.
Oh well, I'll have to think about the update issue.

procount · 2020-08-24T09:16:36Z

@AaronDewes - I've heard reports that Qt5 does not work well with buildroot on the RPi as you have found, mostly probably due to the Rpi specifics themselves. Whilst it may be possible to fix all the issues, this does result in a major change to the PINN software and a lot of risk. Previous attempts to advance PINN in this manner have not been totally successful. It also requires changes to the firmware, buildroot and the PINN recovery program, and extra bloat that may be difficult to manage on a Pi0 or 1.

I wonder if a better approach would be to do this one step at a time. It might be better to concentrate on just evolving buildroot to a later version and reverting to the use of Qt4. This may mean backtracking a bit and porting Qt4 into the new buildroot environment, but it may allow a new development environment to be used a lot sooner. Once that is working properly, converting to qt5 could be reconsidered at a later stage. And maybe some of the issues with it will have been resolved by then.

AaronDewes · 2020-08-24T15:13:08Z

I've heard reports that Qt5 does not work well with buildroot on the RPi as you have found, mostly probably due to the Rpi specifics themselves.

Most of these issues are fixed with a1ccdd1.

It also requires changes to the firmware,

No, with compositor 0.3 (will be pushed soon), the firmware doesn't require any changes.

buildroot

Only adding these patches (a1ccdd1), everything else is in 2020.08.

and the PINN recovery program

Only minor changes are needed there (remove some QWS code, and replace it with something else, I'm currently testing a solution for that)

and extra bloat that may be difficult to manage on a Pi0 or 1.

About 20 MB more RAM usage, I don't know if that''s an issue on the 256 MB models (I don't have one), but I don't think too many problems are caused there.

It might be better to concentrate on just evolving buildroot to a later version and reverting to the use of Qt4.

I've tried that, but haven't been successful. I'll try again if my QT5 attempt fails, but as I said before, compositor 0.3 is much better.

AaronDewes · 2020-08-24T15:19:27Z

But you're right, QT5 is really broken on RPi, but I'm trying to change that: https://codereview.qt-project.org/c/qt/qtwayland/+/310822.

Normally, this is disallowed because build failures occured in some test builds. I haven't experienced this, and parted is needed for PINN, so I changed this.

The path is different now, but the file was not changed.

Use QT5 features, fix include paths, ...

Build of qt5wayland fails on the Raspberry Pi for multiple reasons. These patches are already merged or proposed in qt and probably won't be needed when the next buildroot update is done.

AaronDewes · 2020-09-16T15:01:56Z

Sorry for not working on this, I had some issues, because I didn't get fixup.dat to work, because I didn't know it had to be called fixup_rc.dat, but NOOBS 3.5 resolved that issue, I'll rebase and update this PR after you merged NOOBS 3.5 (I assume you'll do that, If not, please tell me), I'll wait with rebasing because otherwise I'd have to do the rebase again.

procount · 2020-09-16T15:46:55Z

No worries at all. I appreciate all your efforts and hope it will work out in the end. Timescales are all yours. 😉
Yes I intend to merge the noobs 3.5 changes. Hopefully soon, but not sure when.

AaronDewes · 2020-11-22T13:52:41Z

I'll wait for Buildroot 2020.02 to release so I can directly update this to Qt 6, because QT 6 doesn't require any major changes from QT 5, but still needs some optimizations

AaronDewes force-pushed the buildroot-2018.11 branch from 3be6250 to e1876e2 Compare July 4, 2020 12:26

lurch reviewed Jul 6, 2020

View reviewed changes

buildroot/package/linux-headers/Config.in.host Outdated Show resolved Hide resolved

AaronDewes force-pushed the buildroot-2018.11 branch 2 times, most recently from 48124dc to f9e6afa Compare August 22, 2020 10:49

AaronDewes commented Aug 22, 2020

View reviewed changes

AaronDewes force-pushed the buildroot-2018.11 branch 4 times, most recently from d683900 to 662bb9a Compare August 22, 2020 17:52

AaronDewes force-pushed the buildroot-2018.11 branch from 7fe60ca to 053b5d3 Compare August 24, 2020 15:13

AaronDewes added 2 commits August 24, 2020 17:26

Import buildroot 2020.08-rc2

78056bf

Fix the linux kernel for RPi support

eefd70a

AaronDewes added 5 commits August 24, 2020 17:28

Add missing kernel patches

25399d3

Add PINN packages

8fd0699

Modify RPi Packages for PINN

cd54c0c

Update build script for latest buildroot

a83a1e8

Prevent buildroot from thinking we're using an experimental version

0011672

AaronDewes force-pushed the buildroot-2018.11 branch from 353a3fe to b4dab5c Compare August 24, 2020 15:39

AaronDewes added 9 commits August 25, 2020 18:20

Allow parted on uclibc

2be161d

Normally, this is disallowed because build failures occured in some test builds. I haven't experienced this, and parted is needed for PINN, so I changed this.

Add kernel config

1ae854a

Add buildroot config

38baa2e

Fix patch of the dhcpcd hooks

fcab648

The path is different now, but the file was not changed.

Fix PINN for QT5 & latest buildroot

3a808a7

Use QT5 features, fix include paths, ...

Update kernel

3fb04c7

Disable updates for test builds

24a1921

Fix ramdisk size

d3ff335

Use weston

085df40

AaronDewes force-pushed the buildroot-2018.11 branch from b4dab5c to 085df40 Compare August 25, 2020 16:26

Fix qt5wayland

b5882ea

Build of qt5wayland fails on the Raspberry Pi for multiple reasons. These patches are already merged or proposed in qt and probably won't be needed when the next buildroot update is done.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Update buildroot #401

WIP: Update buildroot #401

AaronDewes commented Jul 4, 2020

lurch commented Jul 4, 2020 •

edited

AaronDewes commented Jul 4, 2020 •

edited

AaronDewes commented Aug 15, 2020

AaronDewes commented Aug 21, 2020 •

edited

procount commented Aug 21, 2020

AaronDewes commented Aug 22, 2020

procount commented Aug 22, 2020

AaronDewes commented Aug 22, 2020 •

edited

AaronDewes left a comment •

edited

AaronDewes Aug 22, 2020

AaronDewes Aug 22, 2020

AaronDewes Aug 22, 2020

AaronDewes left a comment •

edited

AaronDewes Aug 22, 2020

AaronDewes Aug 22, 2020

AaronDewes Aug 22, 2020

AaronDewes Aug 22, 2020

AaronDewes Aug 22, 2020

AaronDewes Aug 22, 2020

AaronDewes Aug 22, 2020

AaronDewes commented Aug 23, 2020 •

edited

procount commented Aug 23, 2020

AaronDewes commented Aug 23, 2020

procount commented Aug 23, 2020

AaronDewes commented Aug 24, 2020

procount commented Aug 24, 2020

procount commented Aug 24, 2020

AaronDewes commented Aug 24, 2020

AaronDewes commented Aug 24, 2020

AaronDewes commented Sep 16, 2020

procount commented Sep 16, 2020

AaronDewes commented Nov 22, 2020

		CONFIG_VAR=BR2_LINUX_KERNEL_VERSION
		sed -ri "s/(^$CONFIG_VAR=\")[0-9a-f]+(\")$/\1$GITREV\2/" "$CONFIG_FILE"

		#include <qjson-qt5/parser.h>
		#include <qjson-qt5/serializer.h>

WIP: Update buildroot #401

Are you sure you want to change the base?

WIP: Update buildroot #401

Conversation

AaronDewes commented Jul 4, 2020

lurch commented Jul 4, 2020 • edited

AaronDewes commented Jul 4, 2020 • edited

AaronDewes commented Aug 15, 2020

AaronDewes commented Aug 21, 2020 • edited

procount commented Aug 21, 2020

AaronDewes commented Aug 22, 2020

procount commented Aug 22, 2020

AaronDewes commented Aug 22, 2020 • edited

AaronDewes left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AaronDewes left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AaronDewes commented Aug 23, 2020 • edited

procount commented Aug 23, 2020

AaronDewes commented Aug 23, 2020

procount commented Aug 23, 2020

AaronDewes commented Aug 24, 2020

procount commented Aug 24, 2020

procount commented Aug 24, 2020

AaronDewes commented Aug 24, 2020

AaronDewes commented Aug 24, 2020

AaronDewes commented Sep 16, 2020

procount commented Sep 16, 2020

AaronDewes commented Nov 22, 2020

lurch commented Jul 4, 2020 •

edited

AaronDewes commented Jul 4, 2020 •

edited

AaronDewes commented Aug 21, 2020 •

edited

AaronDewes commented Aug 22, 2020 •

edited

AaronDewes left a comment •

edited

AaronDewes left a comment •

edited

AaronDewes commented Aug 23, 2020 •

edited