scripts: create kernel configuration upgrade script

[ehem]

Since GitHub has effectively displaced the mailing list, a different implementation already got in. This script though has some major advantages.

First, it is rather more robust. The one presently in the tree failed with the first thing I tried. There are several brittle spots which have a high probability of failing in the future.

Second, this does not touch the working tree at all. It generates the two appropriate commits, but then lets the person running decide whether they want a new branch versus bringing into their current branch.

Third, this gets everything in one pass. No need to run it 44 times for each and every device (worse, having one maintainer not know there is a script and manually copy files). This is extremely helpful for rebasing commits modifying the default values in generic.

Fourth, this has cleaner copyright. While not impossible for two people to come up with the same wording, having the exact same goof seems rather less likely.

Fifth, rather better messages. scripts/kernel_bump.sh yields an unhelpful mysterious error. This script emits its usage message. The formatting of the second commit message by the other script isn't very good.

Sixth, this is inherently much faster. Since it has no need to touch the working tree, it never has the two intermediate commit nor final commit unpacked on the filesystem (in fact this script only has one intermediate).

Okay, so Perl is no longer such common knowledge. Until there is a policy of getting rid of Perl, it was an appropriate tool for this script. Python may be better longer-term (and converting to Python seems a likely future enhancement), but Perl was simpler short-term due to better string handling.

The other trick is this is using git fast-import. This is a lesser known portion of Git. Fully two-thirds of the lines in this script are interfacing with git fast-import. Unless you know the tool this is going to seem puzzling. I suggest playing with git fast-export --no-data if you want to learn more.

I would suggest running:

scripts/update_kernel.pl -6.1 -6.6
git merge --ff-only <sha1 provided>
scripts/update_kernel.pl -5.15 -6.6
git merge --ff-only <sha1 provided>

If this goes in.

[ehem]

On second thought, might be best for a maintainer to quickly grab the script out of the commit and run:

scripts/update_kernel.pl -6.1 -6.6
git merge --ff-only <sha1 provided>
scripts/update_kernel.pl -5.15 -6.6
git merge --ff-only <sha1 provided>

(or at least the first two lines, looks like several device update pull requests exist and at least some are not using the history-preserving scripts).

[ehem]

Quick check, 3 were using the other script, 3 were doing manual copy. While the ratio may improve in the future, that is a Bad Sign.

Problem is it will take years before enough history is preserved to be really valuable, but then it takes a single miss to destroy that history. In light of this, doing everything in one pass really does seem the right way to do it.

[namiltd]

Take a look at this: #14713
Looks like someone beat you to it ;)

[hauke]

Please provide an example so that I can have a look at the result of this script.

[ehem]

Take a look at this: #14713 Looks like someone beat you to it ;)

Not really so (notice the date on this).

Please provide an example so that I can have a look at the result of this script.

Okay, since it is the example given, here. Beware, due to the structure of the merges, GitHub shows them backwards if you look at the commits online (easier to understand the blame data or log).

[ehem]

I should ask since it came to mind when examining #14713. How are devices currently without 6.1 configurations expected to be handled? Will they be required to skip 6.1 and immediately go to 6.6?

It would be possible to modify the script to take multiple source and/or destination versions. Something like scripts/kernel_upgrade.pl -s -5.15 -s -6.1 -d -6.1 -d -6.6 this would be handled by copying from -6.1 if it existed, but falling back to -5.15; and copying this to both -6.1 (if it doesn't exist) and -6.6.

This would be useful if the present situation is likely to occur again. This would synthesize 2 copy commits and 1 merge commit, instead of 2 copies and 2 merges (may look better in logs depending on PoV). Simply requires a degenerate octopus merge instead of 2 standard merges.

[hauke]

NACK, I do not like it

Using two times -C will make git blame to detect the moves:

git blame -C -C  target/linux/malta/config-6.1

After the first commit the tree is broken, git bisect will break with such commits which make the build break.
It takes some time to port each target we can not do it all in one shot. Doing such change for all targets will not work.

[ehem]

NACK, I do not like it

Using two times -C will make git blame to detect the moves:

git blame -C -C  target/linux/malta/config-6.1

Exactly the same as "--find-copies-harder" with the well documented problem. This is orders of magnitude slower than git blame without the option. This is the difference between a tool you can use on a regular basis to investigate minor oddities (thus avoiding breakage due to understanding a problem), versus one you take a coffee break while waiting for it to complete.

After the first commit the tree is broken, git bisect will break with such commits which make the build break. It takes some time to port each target we can not do it all in one shot. Doing such change for all targets will not work.

git bisect says on the label it is expected to find broken checkouts. The documentation suggests strategies for dealing with this because it is expected. If you can't deal with that then git bisect is not the tool for you!

There is a consensus having a few commits where git bisect fails, but git blame is fast is a good trade off.

There is an unresolved question of whether it is better to do this individually for each board, versus all boards at once. This is presenting the all boards at once strategy. Ends up being a question of whether it is worthwhile to have first-step copied kernel configs around for a long time while it is decided whether to drop a board versus updates being sluggish.

This approach has the advantage of there is a single bisect broken commit for each update, rather than >40 spread all over the place. This also makes it far easier to handle changing the fallback settings in generic, since you can update everything at once instead of having to continuously monitor for new board updates (this is truly a nightmare).

[ehem]

I'll be rebasing in a few hours, when I'll squash the third in. I've also got something others may like better for doing single device updates.

[ehem]

@hauke if you're truly that much against the idea, you'll need to revert 3561015. Otherwise most others think git blame being properly functional is kind of important.

[ehem]

An interesting issue was pointed out elsewhere, so update to take care of situation.

The second patch was adjusted for better efficiency.

[stintel]

OpenWrt is licensed GPL-2.0-only. Please use the same license if you expect the script to be accepted in this repo.

And if we're deliberately going to be breaking git bisect, please provide a git bisect script that is able to automatically skip these commits. Bisecting OpenWrt is painful enough already. If you don't know what I'm talking about, I suggest you give it a try yourself.

[ehem]

OpenWrt is licensed GPL-2.0-only. Please use the same license if you expect the script to be accepted in this repo.

That is certainly a topic for negotiation. Unless there is genuine hope this will be accepted you're not offering much in negotiation.

And if we're deliberately going to be breaking git bisect, please provide a git bisect script that is able to automatically skip these commits. Bisecting OpenWrt is painful enough already. If you don't know what I'm talking about, I suggest you give it a try yourself.

I haven't used git bisect with OpenWRT, but I can readily believe it is painful. Hard-coded URLs have an unpleasant tendency to expire if the upstream source moves. Several solutions come to mind:

1. git submodule so many projects are using git, perhaps requiring git might now be acceptable. I haven't played with this, but there is a legion of "don't use this" reports.
2. git subtree similar to git submodule, but with some differences. A similar tool with some advantages.
3. Use FreeBSD's strategy. Create a directory for mirrored git trees. Merge all the tools into a subdirectory of the OpenWRT repository (from reading this may be similar to what git subtree does).
4. Start mirroring all upstream tarballs.

There is the major issue of OpenWRT's build system being slow. Too much .NOTPARALLEL when now most developer machines are likely to have >12 processors. The FreeBSD project is worried about having too much single-threaded building, yet it still manages to average use of 10 of 24 processors during its build.

So yes, I can readily believe bisecting being painful. I don't know which reasons are your main concerns though. This doesn't break git bisect, it does generate a single unbuildable commit though. Yet even without this broken commits happen and you have to be able to cope with those during bisect sessions.

Perhaps you're proposing a change to the intermediate commit message to ensure it is more obvious not to try building it when git bisect shows you the single-line commit message? Perhaps standardize on unbuildable commits should start with "DONTBUILD", "DONOTBUILD", or "NONBUILD"? (this needs to be coordinated with #14913)

[ehem]

And if we're deliberately going to be breaking git bisect, please provide a git bisect script that is able to automatically skip these commits. Bisecting OpenWrt is painful enough already. If you don't know what I'm talking about, I suggest you give it a try yourself.

Uhm, I was kind of asking questions in order to figure out a best approach. Would be valuable to get a response...

@hauke and @stintel I regret to inform you the avalanche has already started, it is too late even for boulders to vote. Notice, 16edee9, e28492c and 25abc8f. I believe #14907 can do a better job with this (generate a single non-buildable commit per update, rather than one for every device), but without direct commit ability there is only so much I can do.

Guess I should concede git fast-import is riskier in some ways. It is readily able to synthesize arbitrary commits which command-line git won't allow. Yet in the hands of someone knowledgeable it is a powerful tool.

[ehem]

Gah, screwing up commands dealing with remote repositories in a different way this time.

Slowly waking up the Perl writing skills. Adjusting some argument orders to better match with how things need to be done in Perl.

[ehem]

Okay, seems to be a 2 step process to reopen.

[ehem]

Okay, bit of a delay in GitHub processing things. Now to deal with the bug which showed up right when I pushed this. Hrmm. (have a suboptimal workaround, but I would like to do better).

[ehem]

Unless another device update has happened in the meantime (quite likely), I would suggest doing:

scripts/kernel_upgrade.pl -5.15 -6.1 at91 bcm47xx lantiq octeontx oxnas pistachio qoriq realtek tegra zynq
git merge --ff-only <given sha1>
scripts/kernel_upgrade.pl -6.1 -6.6
git merge --ff-only <given sha1>

When bringing this in. Point being to update everything which hasn't been updated. I'm distinctly unsure of this since I'm wondering whether it might be better for bcm47xx and the other devices still on 5.15 to simply skip 6.1.

[ehem]

Then when looking some more, realize the quotes should instead be added in GitImporter->ls().

[ehem]

I've had that lurking for some time. I had noticed it was an issue, problem was I needed to reread the documentation several times to figure out why that hadn't been working.

I don't believe it actually matters right now, but I could see this being desired in the future.

[ehem]

Then go looking at things and @$ret=reverse(@$ret); should work here.