Create a generic Arch Linux upgrade script

[LukeShortCloud]

Our existing winesapOS upgrade script is basically an Arch Linux upgrade script that addresses issues brought up on archlinux.org . It has some extra logic to handle winesapOS configurations and packages. I would like to refactor our upgrade script into a new project simply called arch-linux-upgrade (or archupgrade to mirror the archinstall project name that exists) to handle automated and unattended upgrades. Then winesapOS can re-use that script and add on our own logic for things very specific to our project.

[LukeShortCloud]

We should have a list of Arch Linux versions that we can use to reference container tags for testing upgrades via a CI/CD pipeline.

https://hub.docker.com/_/archlinux/tags

A list to start off with could be the winesapOS versions that used Arch Linux (probably skipping the minor releases):

• 2022-01-27 = SteamOS 3.0 through 3.3 used this (and so did winesapOS 3.0.0 and 3.0.1, by extension).
• 2022-06-01 = winesapOS 3.1.0. We started the switch back to an Arch Linux base here (instead of SteamOS).
• 2022-07-01 = winesapOS 3.1.1
• 2022-11-24 = winesapOS 3.2.0
• 2023-02-01 = winesapOS 3.2.1
• 2023-07-01 = winesapOS 3.3.0
• 2023-12-01 = winesapOS 3.4.0

https://github.com/LukeShortCloud/winesapOS/releases

Alternatively, we could simply target every 1 year, 6 months, or 3 months.

• 2022-01-01
• 2022-06-01
• 2023-01-01
• 2023-06-01
• 2024-01-01

[LukeShortCloud]

Goals:

• Addressing archlinux.org manual upgrade intervention
• Mutlilib support
• Manjaro support
• Upgrade support for GNOME and KDE Plasma desktop environments at first (with more possibly to come)

Non-goals:

• Support AUR upgrades
  • Later on, Chaotic AUR packages may be supported on a case-by-case basis

[LukeShortCloud]

Packages from https://archive.archlinux.org/ can be pre-installed into old Arch Linux containers for CI testing purposes.

[LukeShortCloud]

Pacman has an undocumented feature to automatically replace packages without confirmation:

$ sudo pacman -S <PACKAGE> --ask 4

https://unix.stackexchange.com/questions/274727/how-to-force-pacman-to-answer-yes-to-all-questions/584001#584001

[LukeShortCloud]

Arch Linux maintainers recommend using pacutils for automating Pacman operations (and not using pacman itself).

https://gitlab.archlinux.org/pacman/pacman/-/issues/102
https://github.com/andrewgregory/pacutils

[LukeShortCloud]

If using a pre- and post-hooks approach with archupgrade to integrate with winesapOS, we should move away from having upgrades organized for specific winesapOS versions. We have already had to reorganize upgrades many times in the past. Depending on what needs to be upgraded, it may need to happen or after any given X.Y.Z version. For example, saying "Running 3.0.1 to 3.1.0 upgrades..." is not entirely accurate since we do various repository modifications required for newer versions of winesapOS.

[LukeShortCloud]

We should come up with a list of Arch Linux based distributions we would want to support. Obviously winesapOS but I have heard that folks from ChimeraOS are interested (for their users who insist on turning off the read-only frzr file system). There is also ArcoLinux, Batocera, EndeavourOS, Garuda Linux, etc.

[LukeShortCloud]

@Thijzer @soredake @wallentx @ohaiibuzzle @Sebazzz @lucid-cloud-dreams @Frostswing @Equ1no0x @Haarolean @maxiride @DKRacingFan @Rapou7 @99powerbreaker @QushyQushy @rkr87 @AHandheldFan @A-mak88 @FabioLolix @mjkl-gh @liberodark @massatt212 @Penguin766 @KaiH-0

Are any of you interested in helping to make an archupgrade project? Similar to how there is a standardized archinsall project?

If I tagged you, that means you have helped contribute to winesapOS in a meaningful way and I am very grateful for that! In winesapOS, we have an upgrade script that fully automates Arch Linux upgrades. Instead of keeping this for winesapOS only, I want to take the experiences/learnings from that and create a new project that will work across a wider range of Arch Linux based distributions. With a fresh start and fresh new perspectives from the community, we can design this better and can create a better experience for end-users.

Why is this needed? Upgrading Arch Linux usually requires manual intervention and it is not always intuitive on how to solve it. Especially for new Linux users. I found a lot of people use Discover from KDE Plasma to do system upgrades. This upgrades one package a time. If the upgrade fails for some reason, it is left in a half broken state. I have also seen other users wait 1-2 years before upgrading which leads to various issues. Dealing with GPG keyrings is especially a big problem on older systems. KDE Plasma 5 to 6 upgrades were also horrible to deal with. This new upgrade script/program would handle everything automatically with no user interaction required.

Some questions we should answer:

• What are goals and non-goals for the project?
• What language should the project use? Bash, Go, Python, Rust, etc.?
• What Arch Linux based distributions and/or custom installations (ex., GNOME, KDE Plasma, etc.) will we officially support?
• What will the automation look like for this?
• Who will help maintain this?
• What GitHub namespace should this project live under?

I wrote down various ideas earlier in this thread related to a few of those questions that we can talk about more.

Where would we discuss this?
GitHub is too slow for communication. Let's use Discord instead.

April 20th Saturday at 4:00 P.M. (16:00) UTC
(Reach out to me on Discord @LukeShortCloud for the server invite) in the #misc-utilities channel

Thanks for your time! I hope to see ya'll online!

[massatt212]

You should use Manjaro KDE since it's simple to install, and it will replicate SteamDeck os closer, could always make a repository that have your custom kernel and mesa drivers, or U can make your own arch installer with a script to install steamos, cause one of the problems I hers is sddm and gdm switching back and fort from game mode to desktop.

[LukeShortCloud]

@massatt212 We have started work on that idea already. :-) Users who insist on having an installer can use Manjaro and then our conversion script. It is very basic right now and does not cover Gamescope session yet but we are looking into expanding it. We can include our repository, tweaks, system packages, etc.

https://github.com/LukeShortCloud/winesapOS/tree/4.0.0?tab=readme-ov-file#convert-to-winesapos

[LukeShortCloud]

In its simplest form, archupgrade will be able to automatically address upgrade concerns from the Arch Linux news feed.

For example:

• News feed: https://archlinux.org/news/incoming-changes-in-jdk-jre-21-packages-may-require-manual-intervention/
• Solution: https://github.com/LukeShortCloud/winesapOS/blob/4.0.0/scripts/winesapos-upgrade.sh#L847

[Thijzer]

Hi @LukeShortCloud, I'm definitely willing to help. Although I kinda shifted to ublue atomic images for most of my machines, I still like those arch based image builds as a base as you have more control of how you package..

[ohaiibuzzle]

I had a couple of ideas reading this. Noting the best one I thought of down so I won't forget it later. Might be absurd, or might very well be viable
winesapOS-upgrade.pdf

[LukeShortCloud]

@ohaiibuzzle Nice PDF! That really helps to clarify your vision and seems doable.

winesapOS 5.0.0 may include an optional read-only image and I will definitely need your help with that! We will still offer "legacy" images with the traditional writable file system for users who want and expect that. Short-term, my friends at ChimeraOS have made frzr to address this problem by shipping Btrfs volume updates. Long-term, bootc is the future by allowing us to use a Dockerfile to build the operating system file system. Directories such as /etc and /home remain writable and are kept during upgrades. Updates are delivered from a container registry. @Thijzer Universal Blue uses containers today (in a slightly different way that is RPM distro specific) and we would take a very similar approach. I spoke with a few members of their team and they will also be moving to bootc (which happens to be distro agnostic) eventually. Thanks for your continued interest and help!

archupgrade will be focused on traditional Arch Linux upgrades. Some folks have even suggested that perhaps archupgrade should help users on ChimeraOS or SteamOS switch to a writable file system. We should follow-up this winesapOS read-only discussion in the near future!

[Thijzer]

I completely agree with your comment @LukeShortCloud. I was thinking of picking up the work of chimera os built-tools and working out a foundation of good image building and easy image deployment. Also the layering of the cake like ublue images is what I would want in these tools. I'll read up on bootc and see if it aligns. Thanks!

[ohaiibuzzle]

Yeah, I would much prefer, for an Arch system anyway, for them to upgrade in an imaging-based manner rather than doing traditional package upgrade if we can. It would be much more predictable than trying to deal with potentially dicey package upgrades, since winesap is expected to just be used with Flatpaks and not having users managing packages themselves.

I could think of a few ideas to achieve package-based upgrades, I'll see which one could work and make a similar writeup if I do

[LukeShortCloud]

@Thijzer Do you have a Discord? You can add me @LukeShortCloud and I will send you an invite to the Discord server. We will be meeting April 20th Saturday at 4:00 P.M. (16:00) UTC. We will be focused on the archupgrade conversation there but, in the coming weeks or months, we can also have a follow-up discussion about a read-only variant of winesapOS.

[lucid-cloud-dreams]

@LukeShortCloud I feel really honored to be invited to this project :)
I have to say, that I am not a programmer by any means! But I would say about myself, that I'm a good enough sysadmin ;)
But I still want to join the call at 4pm UTC today.
I am really excited how I can help the project! :)

[LukeShortCloud]

@lucid-cloud-dreams Your system administrator experience would be great to help provide feedback on how we tackle upgrades! Reach out to me on Discord @LukeShortCloud and I will get you an invite to our Discord server. You can join us in the #misc-utils channel in an hour when the meeting starts.

[LukeShortCloud]

@ohaiibuzzle Thanks for sharing your idea about a potential YAML layout (pasted below)! You mentioned this was inspired by Debian. This would be great for helping define upgrade recipes for winesapOS, ChimeraOS, etc.

phases:
  first:
    preupgrade:
      - some_bash_commands && do_something
    packages:
      - package_a: https://archive.archlinux.org/p/package_a.tar.zst
      - package_b: https://archive.archlinux.org/p/package_b.tar.zst
    postupgrade:
      - some_more_bash_commands && do_some_more_things
  second:
    preupgrade:
      - some_bash_commands && do_something
    packages:
      - package_c: https://archive.archlinux.org/p/package_c.tar.zst
      - package_d: https://archive.archlinux.org/p/package_d.tar.zst
    postupgrade:
      - some_more_bash_commands && do_some_more_things

[ohaiibuzzle]

I'll make a writeup similar to the other, imaging-based upgrading strategy when I get back to my desktop later, expanding on the idea, would be more detailed since we have a direction to go now

[wallentx]

I intended to reply to this, but I just started a new job a few weeks ago, and my focus was pulled away. I'm sorry I missed the boat!
I actually have a lot of opinions about this, since it sounds like this is what the Steam Deck is doing. If you all need a hand on any ongoing/future work, I'd love to help!

[LukeShortCloud]

Hey @wallentx , I know you already read through everything on Discord but I'll give a quick update for everyone.

There are two goals we are targeting with archupgrade:

1. Upgrade to the latest Arch Linux. Deal with common issues.
2. Convert to different profiles. For example: gaming_desktop (winesapOS) or gaming_console (ChimeraOS).

With the conversion feature, distros can use archupgrade to take a minimal Arch Linux container and get a fully functional operating system. Users can either use this program on bare metal, distrobox/toolbox containers, or in a build system (ex., for bootc). Everything is declarative and state-driven.

For the programming language, most folks are leaning towards Rust but other proposed languages include Go and Bash.

@ohaiibuzzle Could you upload the latest PDF of your outline here on this GitHub Issue? That showcases the goals and non-goals plus the YAML configuration we are thinking about using.

[ohaiibuzzle]

Here is my notes on how I see it could be implemented, the "specification" refers to the file that contains what's explained by Luke as the "profile". "Implementation" here is the actual archupgrade binary, which may/can be implemented differently using different toolchains, so long as it fits the definition.

archupgrade.pdf