Portageq depreciated, causes warnings while building anything #883

Phoenix591 · 2023-06-07T09:20:20Z

See gentoo/portage@ab538e7

Used in

gentooLTO/sys-config/ltoize/files/bashrc.d/41-lto-patch.sh

Line 21 in a24c46a

    
           local lto_overlay_dir="$(portageq get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay)"

The text was updated successfully, but these errors were encountered:

parona-source · 2023-10-22T14:29:34Z

It will be fatal in the next portage release gentoo/portage@76a7c3c

kanyck · 2023-10-25T16:56:32Z

Yeah, I got my system broken today after a regular upgrade. Portage was upgraded to 3.0.54 and doesn't work anymore!

antonellocaroli · 2023-10-25T18:15:10Z

how to solve this problem?

kir68k · 2023-10-25T18:24:37Z

Yeah, I got my system broken today after a regular upgrade. Portage was upgraded to 3.0.54 and doesn't work anymore!

I had to emerge -c this package, modify the ebuild¹ to get a hardcoded /var/db/repos path instead of portageq, then emerge again, after removing /etc/portage/bashrc.d and bashrc.
This feels wrong, and is a temporary solution, at best.

I think I'm gonna try to work around this next week or in a few, as right now I am unable to.

¹ Currently mine doesn't have a lot of changes but I wouldn't recommend using it on production systems.

parona-source · 2023-10-25T18:26:06Z

GentooLTO is practically dead, either stop using it and manage over rides manually or fork GentooLTO to fix uses of portageq in the portage hooks.

To actually fix ltoize would either find some non banned way to find out repository paths (I doubt it) or change the way patches are available in the repo.

With the second options I think you could a package package which contains the patches and could be installed into a proper location. It just means you would have to start making releases so that it gets updated for people in addition to just syncing the repo.

Just spit balling with the later.

https://projects.gentoo.org/pms/8/pms.html#x1-157000B

I'm personally not invested in GentooLTO as I stopped using it a long time ago and instead having managing package.env and /e/p/patches for all the problem packages.

If someone were to fork it Id advocate removing the haphazard filter-flag removal. Some packages actually need it and its breaking stuff for no reason (binutils rebuilds when LTO version changes).

eternal-sorrow · 2023-10-25T18:31:03Z

I just edited the /etc/portage/bashrc.d/41-lto-patch.sh and replaced the $(portageq get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay) with an actual path to the repository. This file is a symlink that points inside the repository itself, so the change would be overwritten on the next sync, but because gentooLTO is dead, and no updates are happening, this is not a problem.

eternal-sorrow · 2023-10-25T18:35:03Z

I'm personally not invested in GentooLTO as I stopped using it a long time ago and instead having managing package.env and /e/p/patches for all the problem packages.

Could you somehow share your collection of package.env settings and patches?

eternal-sorrow · 2023-10-25T18:45:17Z

Welp, I have -march=znver3 -O3 -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=16 -fuse-linker-plugin -fvect-cost-model=cheap -pipe

kanyck · 2023-10-25T19:10:43Z

I just edited the /etc/portage/bashrc.d/41-lto-patch.sh

I actually did the same: just hard-coded the path. I feel it's a dirty and ugly hack but nevertheless my portage works again.
It's a pity that the project is dying, however I understand that it takes a lot of time and efforts to keep up with all the changes while the app devs are sometimes not very cooperative with LTO demands. Well, so be it...

antonellocaroli · 2023-10-25T19:21:45Z

I just edited the /etc/portage/bashrc.d/41-lto-patch.sh and replaced the $(portageq get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay) with an actual path to the repository. This file is a symlink that points inside the repository itself, so the change would be overwritten on the next sync, but because gentooLTO is dead, and no updates are happening, this is not a problem.

What path did it return? Can you post the change?

eternal-sorrow · 2023-10-25T19:24:49Z

What path did it return? Can you post the change?

Path to the lto-overlay repository. In my case it's /var/lib/portage/repos/lto-overlay/, but you probably have something different. Something in the /var/db/portage directory probably.

kanyck · 2023-10-25T19:27:39Z

What path did it return?

Check with
$> portageq get_repo_path / lto-overlay
/var/db/repos/lto-overlay in my case.

So the line in my case would be
local lto_overlay_dir="/var/db/repos/lto-overlay"

Have fun!

matoro · 2023-10-26T00:44:06Z

Another workaround is to change portageq to the absolute path /usr/bin/portageq.

jonasstein · 2023-10-26T00:48:50Z

Please close https://bugs.gentoo.org/916287, when fixed

eternal-sorrow · 2023-10-26T06:31:16Z

This overlay has been dropped from the gentoo overlays. Which is expected, I guess.

neeshy · 2023-10-26T12:30:26Z

You can also replace the offending line with this:

local lto_overlay_dir="$("$(command -pv portageq)" get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay)"

Which is a bit more generic than using the absolute path

kanyck · 2023-10-26T15:22:39Z

You can also replace the offending line with this:

local lto_overlay_dir="$("$(command -pv portageq)" get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay)"

portageq seems to be removed soon. Then it would stop working again...

kir68k · 2023-10-26T16:43:29Z

You can also replace the offending line with this:

local lto_overlay_dir="$("$(command -pv portageq)" get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay)"

This seems like a temporary workaround, still hacking on with portageq.

matoro · 2023-10-26T16:51:24Z

You can also replace the offending line with this:
local lto_overlay_dir="$("$(command -pv portageq)" get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay)"

portageq seems to be removed soon. Then it would stop working again...

Where are you getting that portageq itself is deprecated? It's not, only using it within ebuild scope.

parona-source · 2023-10-26T16:56:31Z

You can also replace the offending line with this:
local lto_overlay_dir="$("$(command -pv portageq)" get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay)"

portageq seems to be removed soon. Then it would stop working again...

Where are you getting that portageq itself is deprecated? It's not, only using it within ebuild scope.

These are used in ebuild hooks so they are in ebuild scope.

matoro · 2023-10-26T17:11:44Z

You can also replace the offending line with this:
local lto_overlay_dir="$("$(command -pv portageq)" get_repo_path ${PORTAGE_CONFIGROOT} lto-overlay)"

portageq seems to be removed soon. Then it would stop working again...

Where are you getting that portageq itself is deprecated? It's not, only using it within ebuild scope.

These are used in ebuild hooks so they are in ebuild scope.

It's only banned within ebuild scope because it's temporarily overridden via PATH to use a different executable which behaves differently. As long as you take care to call the real one outside the sandbox, it should continue to work just fine.

thesamesam · 2023-10-26T17:24:25Z

No, it's banned because you shouldn't call portageq in ebuilds, as PMS clearly says. The suggested workaround is a hacky workaround indeed and still not permitted. Even if it "works".

portageq will remain as a command outside of the ebuild environment.

thesamesam · 2023-10-26T17:28:06Z

I just edited the /etc/portage/bashrc.d/41-lto-patch.sh

I actually did the same: just hard-coded the path. I feel it's a dirty and ugly hack but nevertheless my portage works again. It's a pity that the project is dying, however I understand that it takes a lot of time and efforts to keep up with all the changes while the app devs are sometimes not very cooperative with LTO demands. Well, so be it...

It's not really relevant nowadays anyway, given for some time now, we've tried to handle this in ::gentoo. See e.g. https://wiki.gentoo.org/wiki/Project:Toolchain/LTO.

eternal-sorrow · 2023-10-26T17:32:21Z

It's not really relevant nowadays anyway, given for some time now, we've tried to handle this in ::gentoo. See e.g. https://wiki.gentoo.org/wiki/Project:Toolchain/LTO.

gentooLTO, despite its name, is not only about LTO, so it's still relevant.

thesamesam · 2023-10-26T17:34:02Z

The other flags it adds shouldn't really be passed unconditionally at all.

eternal-sorrow · 2023-10-26T17:35:13Z

Well, that's for users to decide.

thesamesam · 2023-10-26T17:38:29Z

Not sure where I've said anything that indicates you can't decide. But going against even GCC's advice on things like -fipa-pta is not wise, nor is it wise to pass flags which change runtime behaviour like -fno-semantic-interposition unconditionally.

vimproved · 2023-10-26T19:10:05Z

Well, that's for users to decide.

It's also for users to decide whether they want their system to work properly :). Just because you can use various dubious optimizations doesn't mean you should or that you'll get any support for it.

eternal-sorrow · 2023-10-26T19:13:58Z

It's also for users to decide whether they want their system to work properly

My system works properly.

Just because you can use various dubious optimizations doesn't mean you should

But I want to.

or that you'll get any support for it.

I don't ask anybody for support.

thesamesam · 2023-10-26T20:09:05Z

And yet you're interjecting here when I just responded to someone talking about LTO demands.

eternal-sorrow · 2023-10-26T20:10:38Z

And?

negril · 2023-10-27T15:44:26Z

And?

Thank you for you contribution. It's embraced and valued.

Can I send everyone that continues to have issues due to this overlay to you?

neeshy · 2023-10-28T04:39:30Z

I've done some work to move away from gentooLTO and I thought others might benefit from the details, so here they are:

First, modify your CFLAGS to use the relevant flags directly.

Next, run this one liner to find the intersection of packages installed on your system, and packages configured by gentooLTO:

cat /etc/portage/package.cflags/* | grep -E "(^#|$(head -c-1 <(qlist -I) | tr '\n' '|'))" > workarounds.conf

On zsh you can use *(@) to only select symlinks (to filter for files that ltoize provides).

Then, go through each package on a semi-individual basis to see if the workaround is still necessary. Spoiler: The vast majority of them are not. The workarounds in the "Packages which cannot be built with LTO at all", ICE, libtool hack, test failure and undefined symbol categories are easy enough to verify. If it's not clear from the comments, you'll have to go through the issue tracker or use git blame (or git log -p) to find the exact reason why the workaround was added.

Here are the workarounds I removed (I'm running on an unstable system, so YMMV. Some of these workarounds might still be relevant for stable.):

# lto
# Packages which cannot be built with LTO at all
app-emulation/qemu *FLAGS-=-flto* *FLAGS+=-fno-strict-aliasing # required to compile qemu
app-text/texlive-core *FLAGS-=-flto*
dev-lang/perl *FLAGS-=-flto*
dev-lang/rust *FLAGS-=-flto*
dev-libs/elfutils *FLAGS-=-flto*
dev-scheme/racket *FLAGS-=-flto* *FLAGS-="-fipa-pta" # Undefined references and multiple segfaults / violations during build.
dev-util/perf *FLAGS-=-flto*
dev-util/valgrind *FLAGS-=-flto*
media-libs/alsa-lib *FLAGS-=-flto*
media-libs/dav1d *FLAGS-=-flto* # Starting with GCC 11.1.0, various undefined reference errors during linking
media-video/ffmpeg *FLAGS-=-flto* # NOTE: Depending on your CPUFLAGS this may work with LTO.  The SSE intrinsics seem to cause problems in some cases. Only x86 requires workaround
net-analyzer/wireshark *FLAGS-=-flto* # Segfaults trying to load GUI on QT5, see issue 746
net-libs/nodejs *FLAGS-=-flto* # LTO is controlled by a USE flag; currently won't build with LTO and GCC 11
sys-apps/man-db *FLAGS-=-flto* # Issue #403, LTO causes runtime failures
sys-devel/clang *FLAGS-=-flto* # Issue #619 Same as above
sys-devel/gdb *FLAGS-=-flto*
sys-devel/llvm *FLAGS-=-flto* # Issue #619 temporarily disabled for now due to build errors
sys-fs/fuse-common *FLAGS-=-flto*
sys-fs/fuse *FLAGS-=-flto* # Issue #139 and #142, it appears the symbol __fuse_read_cmd goes missing if LTO is enabled, causing linking problems on some systems.  Using gold seems to help.
sys-libs/glibc *FLAGS-=-flto* # Alternative: *FLAGS+=-ffat-lto-objects *FLAGS-="-fipa-pta" Fails in configure stage with LTO enabled, checking for builtin redirect.  With -ffat-lto-objects, and no -fipa-pta, we get further, but get multiple definition errors  Forcing past that yields undefined symbol errors.
sys-libs/libomp *FLAGS-=-flto* # Issue #619 libomp-11.0.0_rc4 fails to build with LTO
sys-process/numactl *FLAGS-=-flto* # Undefined references
x11-base/xorg-server *FLAGS-=-flto* # linking error during compilation

# lto
# Packages which can be built with LTO but leads to problems/crashes/segfaults
dev-libs/libaio *FLAGS-=-flto* # Issue #314, missing symbols in LTO build compared to non-LTO build, leads to problems with sys-fs/lvm2
dev-libs/libbsd *FLAGS-=-flto* # Undefined symbol error when building mail-client/mailx
dev-libs/libpcre *FLAGS-=-flto* # Test failure
dev-libs/protobuf *FLAGS-=-flto* # Upstream bug https://github.com/protocolbuffers/protobuf/issues/4958
net-misc/openssh *FLAGS-=-flto* # hangs on exit with lto
sys-apps/attr *FLAGS-=-flto* # Issue #268, causes segfaults all over the system
sys-apps/sandbox *FLAGS-=-flto* # Issue #347, LTO breaks basic sandboxing functionality
sys-libs/libxcrypt *FLAGS-=-flto* # Undefined symbols in library files cause dependencies like net-misc/whois to fail to build

# fat-lto-objects
# Packages which require fat LTO objects in order to build (TESTING: STRIP_MASK) or which others depend on non-LTO symbols
media-libs/libvpx *FLAGS+=-ffat-lto-objects # Test failure

# Build Workarounds
# Packages which require build workarounds to be built with LTO that can be resolved directly in *FLAGS
sys-devel/gettext *FLAGS+=-lm # needed for linker to resolve libm's log10 -- only needed for LTO
sys-libs/ncurses-compat *FLAGS-=-Wl,--as-needed *FLAGS+=-ldl # checking whether able to link to dl*() functions... configure: error: Cannot link test program for libdl (#111)
sys-libs/ncurses *FLAGS-=-Wl,--as-needed *FLAGS+=-ldl # checking whether able to link to dl*() functions... configure: error: Cannot link test program for libdl (#111)
x11-libs/wxGTK NOLDADD=1 C*FLAGS-="-Wl,*" # Issue #40, does not build when -Wl linker flags are passed to the compiler through C*FLAGS

# ipa-pta
dev-libs/protobuf *FLAGS-="-fipa-pta" # Test failure
dev-lisp/sbcl *FLAGS-="-fipa-pta" # ICE on -fipa-pta
dev-vcs/git *FLAGS-="-fipa-pta" # Segfault in git fetch with GCC < 11.3.0
media-video/ffmpeg *FLAGS-="-fipa-pta" # Test failure
sys-devel/clang *FLAGS-="-fipa-pta" # Test failure
www-client/firefox *FLAGS-="-fipa-pta" # ICE on -fipa-pta

# no-semantic-interposition
media-video/ffmpeg *FLAGS-="-fno-semantic-interposition" # doesn't build, function inlining fails
net-print/cups *FLAGS-="-fno-semantic-interposition" # ICE
sys-libs/readline *FLAGS-="-fno-semantic-interposition" # breaks ncurses for some users, #681 #392

# no-common libtool hack workarounds (#484)
app-crypt/p11-kit NOCOMMON_OVERRIDE_LIBTOOL=yes
dev-libs/fribidi NOCOMMON_OVERRIDE_LIBTOOL=yes
dev-libs/libmspack NOCOMMON_OVERRIDE_LIBTOOL=yes
dev-libs/libpipeline NOCOMMON_OVERRIDE_LIBTOOL=yes
gnome-base/librsvg NOCOMMON_OVERRIDE_LIBTOOL=yes
media-gfx/imagemagick NOCOMMON_OVERRIDE_LIBTOOL=yes
media-libs/glu NOCOMMON_OVERRIDE_LIBTOOL=yes
media-sound/mpg123 NOCOMMON_OVERRIDE_LIBTOOL=yes
net-misc/curl NOCOMMON_OVERRIDE_LIBTOOL=yes
sys-devel/gettext NOCOMMON_OVERRIDE_LIBTOOL=yes
x11-libs/pango NOCOMMON_OVERRIDE_LIBTOOL=yes
x11-libs/xcb-util-cursor NOCOMMON_OVERRIDE_LIBTOOL=yes
x11-libs/xcb-util-xrm NOCOMMON_OVERRIDE_LIBTOOL=yes

# no-plt
dev-libs/nss *FLAGS-="-fno-plt"
media-video/ffmpeg *FLAGS-="-fno-plt"
sys-libs/glibc *FLAGS-="-fno-plt" # When using prelink, will cause a bunch of segfaults.
x11-base/xorg-server *FLAGS-="-fno-plt" # Undefined symbol in i956 driver

# tls-dialect=gnu2
dev-lang/python *FLAGS-="-mtls-dialect=gnu2"

I was fairly conservative and only removed a workaround if the relevant flag was already being filtered in the corresponding ebuild (or an inherited eclass; cargo.eclass filters out LTO and xorg-3.eclass filters out -no-plt, for example), or if I could contradict the reason why the workaround was added in the first place. The only one I'm not sure about is the python -mtls-dialect=gnu2 workaround.

These are the ones that I could directly verify are still relevant for me:

# O3
media-libs/lcms /-O3/-O2 # Test failure

# devirtualize-at-ltrans
app-text/lcdf-typetools *FLAGS-="-fdevirtualize-at-ltrans" # ICE on static-var pass

And these are the ones I kept just in case, or because I didn't want to bother investigating.

# ipa-pta
dev-qt/qtgui *FLAGS-="-fipa-pta" # Same problem as libwebp
media-libs/libwebp *FLAGS-="-fipa-pta" # no compilation issues, but -fipa-pta causes webp images to be displayed incorrectly
media-libs/mesa *FLAGS-="-fipa-pta" # Segfault with vulkan
sys-devel/gcc *FLAGS-="-fipa-pta"
x11-wm/bspwm *FLAGS-="-fipa-pta" # needed for version 0.9.7 on 17.0 profile running the testing branch everywhere with GCC 8.3.0

# no-semantic-interposition
sys-devel/llvm *FLAGS-="-fno-semantic-interposition"
sys-libs/glibc *FLAGS-="-fno-semantic-interposition"

# no-plt
app-emulation/wine-staging *FLAGS-="-fno-plt"

# tls-dialect=gnu2
www-client/firefox *FLAGS-="-mtls-dialect=gnu2"
sys-libs/glibc *FLAGS-="-mtls-dialect=gnu2"
net-libs/libtorrent-rasterbar *FLAGS-="-mtls-dialect=gnu2" # causes memory corruption at runtime

BTW: app-office/libreoffice *FLAGS-="-fno-semantic-interposition" should be used instead of the workaround in lto.conf

At this point, I have no workarounds from lto.conf and no-common-libtool.conf on my system. Seems like over the years, Gentoo has largely made LTO viable without the use of overlays.

Now, you can remove the ltoize package (make sure you don't remove portage-bashrc-mv with it). If you still have any workarounds from no-common-libtool.conf, you'll have to keep the /etc/portage/bashrc.d/43-no-common.sh script.

As for the patches applied by 41-lto-patch.sh, none of them are relevant (if you use an arm64 system, the gmp patch might still be useful).

thesamesam · 2023-10-28T05:54:39Z

@neeshy Please do file any bugs for issues on bugs.gentoo.org you do hit that remain as well, although it sounds like there won't be many of those. Thanks!

antonellocaroli · 2023-10-28T06:18:06Z

@neeshy can you explain better how you did it, to remove the overlai-lto?
You basically just created a file /etc/portage/package.env
file and put in:

app-emulation/qemu *FLAGS-=-flto* *FLAGS+=-fno-strict-aliasing # required to compile qemu
app-text/texlive-core *FLAGS-=-flto*
dev-lang/perl *FLAGS-=-flto*
dev-lang/rust *FLAGS-=-flto*
dev-libs/elfutils *FLAGS-=-flto*
dev-scheme/racket *FLAGS-=-flto* *FLAGS-="-fipa-pta" # Undefined references and multiple segfaults / violations during build.
dev-util/perf *FLAGS-=-flto*
dev-util/valgrind *FLAGS-=-flto*
media-libs/alsa-lib *FLAGS-=-flto*
media-libs/dav1d *FLAGS-=-flto* # Starting with GCC 11.1.0, various undefined reference errors during linking
media-video/ffmpeg *FLAGS-=-flto* # NOTE: Depending on your CPUFLAGS this may work with LTO.  The SSE intrinsics seem to cause problems in some cases. Only x86 requires workaround
net-analyzer/wireshark *FLAGS-=-flto* # Segfaults trying to load GUI on QT5, see issue 746
net-libs/nodejs *FLAGS-=-flto* # LTO is controlled by a USE flag; currently won't build with LTO and GCC 11
sys-apps/man-db *FLAGS-=-flto* # Issue #403, LTO causes runtime failures
sys-devel/clang *FLAGS-=-flto* # Issue #619 Same as above
sys-devel/gdb *FLAGS-=-flto*
.................................

what does your make.conf look like?

neeshy · 2023-10-28T09:03:44Z

@antonellocaroli

No, the workarounds that you listed are the ones that I removed.

Here's my current /etc/portage/package.cflags/workarounds.conf (yours will look different if you follow my steps):

# lto
app-emulation/dxvk *FLAGS-=-flto*
app-emulation/gallium-nine-standalone *FLAGS-=-flto*

# devirtualize-at-ltrans
app-text/lcdf-typetools *FLAGS-=-fdevirtualize-at-ltrans # ICE on static-var pass

# ipa-pta
sys-devel/gcc *FLAGS-=-fipa-pta

# no-semantic-interposition
app-office/libreoffice *FLAGS-=-fno-semantic-interposition
sys-devel/llvm *FLAGS-=-fno-semantic-interposition # makes DXVK crash (#174)
sys-libs/glibc *FLAGS-=-fno-semantic-interposition # gentooLTO commit a0d9f847

# no-plt
app-emulation/wine-staging *FLAGS-=-fno-plt # gentooLTO commit 899e2cfb

# tracer
media-libs/lcms *FLAGS-=-ftracer # Test failure
sys-devel/gdb *FLAGS-=-ftracer

# tls-dialect=gnu2
www-client/firefox *FLAGS-=-mtls-dialect=gnu2
sys-libs/glibc *FLAGS-=-mtls-dialect=gnu2 # gentooLTO commit a0d9f847
net-libs/libtorrent-rasterbar *FLAGS-=-mtls-dialect=gnu2 # causes memory corruption at runtime

# hash-style=gnu
app-emulation/dxvk *FLAGS-=-Wl,--hash-style=gnu

# gc-sections
sys-libs/glibc *FLAGS-=-Wl,--gc-sections

# z
app-emulation/dxvk *FLAGS-=-Wl,-z,relro
app-emulation/dxvk *FLAGS-=-Wl,-z,now

# debug
sys-libs/glibc *FLAGS+=-fno-builtin-strlen
sys-libs/glibc *FLAGS-=-Wl,-s

# noldadd
dev-util/mingw64-toolchain NOLDADD=1 C*FLAGS-="-Wl,*"

And my CFLAGS

CFLAGS="-O3 -pipe -march=native -flto=auto -fuse-linker-plugin -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -fno-plt -ftracer -fgcse-sm -fgcse-las -fira-loop-pressure -fsched-pressure -fmodulo-sched -fmodulo-sched-allow-regmoves -falign-functions=64 -flimit-function-alignment -ffunction-sections -fdata-sections -malign-data=cacheline -mtls-dialect=gnu2"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -Wl,-s -Wl,--gc-sections -Wl,-z,relro -Wl,-z,now"
MAKEOPTS="-j16"

EDIT: Removed some more workarounds. Couldn't reproduce the mesa vulkan segfault with vkcube (see #747). The lcms test fails due to -ftracer not -O3. Also, added some more comments.

negril · 2023-10-28T11:25:06Z

@neeshy in case you have an account on https://wiki.gentoo.org would you mind putting your findings there so we could refer people to that?

antonellocaroli · 2023-10-28T12:23:32Z

@neeshy
other question this CFLAGS you have in the make.conf
-fuse-linker-plugin -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -fno-plt -ftracer -fgcse-sm -fgcse-las -fira-loop-pressure -fsched-pressure -fmodulo-sched -fmodulo-sched-allow-regmoves -falign-functions=64 -flimit-function-alignment -ffunction-sections -fdata-sections -malign-data=cacheline -mtls-dialect=gnu2

and LDFLAGS

-hash-style=gnu -Wl,-s -Wl,--gc-sections -Wl,-z,relro -Wl,-z,now

are they for your specific CPU, or are they good for Intel or amd indifferently?

neeshy · 2023-10-28T18:39:48Z

@negril

I wouldn't really have anything relevant to say on the wiki since this is all very gentooLTO specific. I will create a new issue to increase visibility though.

@antonellocaroli

The only flag that's architecture specific is -falign-functions. On Intel CPUs, 32 seems to be better (see #164). The flags after -fno-plt are mostly overkill. Specifically -ftracer, -ffunction-sections, and -fdata-sections. -ftracer in particular will give you problems, so I would just leave it out if you're not keen on debugging build failures.

vimproved · 2023-10-28T18:56:13Z

@neeshy
A couple things about your workarounds:

app-text/lcdf-typetools *FLAGS-=-fdevirtualize-at-ltrans # ICE on static-var pass

PLEASE report this to bugs.gentoo.org, and upstream if applicable. ICEs are serious errors that the Gentoo devs need to know about.

sys-libs/glibc *FLAGS-=-fno-semantic-interposition # gentooLTO commit a0d9f847

Does the glibc ebuild not already filter this? Glibc generally should filter almost all CFLAGS since it's so sensitive.

app-emulation/dxvk *FLAGS-=-Wl,-z,relro
app-emulation/dxvk *FLAGS-=-Wl,-z,now

I've never had issues with dxvk (or anything) and relro/znow, albeit I use clang.

LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -Wl,-s -Wl,--gc-sections -Wl,-z,relro -Wl,-z,now"

You don't actually need -Wl,-z,now in LDFLAGS. You can enable it via the default-znow flag on GCC, which is enabled by default in the hardened profiles and will be enabled everywhere when the 23.0 profiles are released. Relro is enabled by default if you use clang and sys-devel/clang-common[hardened], not sure about GCC though.

CFLAGS="-O3 -pipe -march=native -flto=auto -fuse-linker-plugin -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -fno-plt -ftracer -fgcse-sm -fgcse-las -fira-loop-pressure -fsched-pressure -fmodulo-sched -fmodulo-sched-allow-regmoves -falign-functions=64 -flimit-function-alignment -ffunction-sections -fdata-sections -malign-data=cacheline -mtls-dialect=gnu2"

I still think these are overzealous, but to each their own. I'd recommend not using -fipa-pta though, since it's not very well maintained upstream in GCC and has had quite a few bugs stemming from it.

Generally, also please report any issues with the more sane CFLAGS (-flto or -O3), just please try to find the minimum flags required to reproduce.

parona-source · 2023-10-28T18:58:51Z

Does the glibc ebuild not already filter this? Glibc generally should filter almost all CFLAGS since it's so sensitive.

it does but this exists https://github.com/InBetweenNames/gentooLTO/blob/master/sys-config/ltoize/metadata.xml#L16

vimproved · 2023-10-28T19:10:14Z

Does the glibc ebuild not already filter this? Glibc generally should filter almost all CFLAGS since it's so sensitive.

it does but this exists https://github.com/InBetweenNames/gentooLTO/blob/master/sys-config/ltoize/metadata.xml#L16

Yes, but nobody should be using that, nor should it have been made in the first place. Flag-o-matic exists for a reason.

InBetweenNames · 2023-10-28T20:21:25Z

Should be fixed in 21d2ba4 , there doesn't seem to be a clean way to do it.
Or rather, this might be the cleanest way.

Also, a reminder to please keep conversations on topic. This bug is about portageq, not supported GentooLTO flags.

neeshy · 2023-10-28T21:24:05Z

@vimproved

PLEASE report this to bugs.gentoo.org, and upstream if applicable. ICEs are serious errors that the Gentoo devs need to know about.

OK, done: https://bugs.gentoo.org/916455

~~I couldn't get an account on GCC's bug tracker (says it's restricted). Maybe someone with an account already could submit the bug for me.~~

Seems like this is a known bug upstream: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100010

Does the glibc ebuild not already filter this? Glibc generally should filter almost all CFLAGS since it's so sensitive.

Oh, you're right. I wasn't searching for strip-flags when I made that post.

I've never had issues with dxvk (or anything) and relro/znow, albeit I use clang.

~~I'll have to revisit these ones.~~

It's due to mingw's linker not respecting -z

/usr/lib/mingw64-toolchain/lib/gcc/x86_64-w64-mingw32/13/../../../../x86_64-w64-mingw32/bin/ld: unrecognized option '-z'

I still think these are overzealous

Very true. Gotta go fast :^)

I'll have to revisit these as well. -ftracer in particular is basically a pessimization in most cases (not to mention the build failures it causes).

@InBetweenNames

Apologies, I figured this was the best place to post my text wall.

As a side note, thank you for maintaining this repo over the years.

eternal-sorrow · 2023-10-29T05:09:47Z

@InBetweenNames, the news item says that LTO_OVERLAY_PATH is only required if I set USE=override-flagomatic for sys-config/ltoize.

if you are using the override-flagomatic option, you must now specify the path to lto-overlay
in the variable LTO_OVERLAY_PATH. If you're not using this GentooLTO flag, this probably wouldn't affect you.

(emphasis mine)

I don't have this USE flag set, but any package I try to build fails with an error:

* You must set LTO_OVERLAY_PATH in your make.conf to point to where lto-overlay lives.
* You can use: 'portageq get_repo_path / lto-overlay' to obtain it if you're not sure.

InBetweenNames · 2023-10-29T13:47:03Z

Ah yeah, I probably overlooked something. No matter, the error tells you what you need to know. I'll accept a PR to fix it if anyone is interested.

paralin · 2023-12-07T02:14:32Z

It seems like there's still some need for a bunch of package.env overrides like @neeshy is posting above, so there's still a purpose to have a thing like gentooLTO around, no?

neeshy · 2023-12-13T16:55:21Z

@paralin

I've since gone through even more overrides and determined that most of them aren't needed. Here's my current workarounds.conf:

# lto
app-emulation/gallium-nine-standalone *FLAGS-=-flto* # build fails with: C shared or static library 'dl' not found

# ld
app-emulation/dxvk *FLAGS-=-Wl,--hash-style=gnu *FLAGS-=-Wl,-z,relro *FLAGS-=-Wl,-z,now # MinGW's linker doesn't recognize these flags
app-emulation/vkd3d-proton *FLAGS-=-Wl,--hash-style=gnu *FLAGS-=-Wl,-z,relro *FLAGS-=-Wl,-z,now # MinGW's linker doesn't recognize these flags
dev-util/mingw64-toolchain NOLDADD=1 # build fails with: C compiler cannot create executables
sys-libs/glibc *FLAGS-=-Wl,--gc-sections # build fails with: --gc-sections requires a defined symbol root specified by -e or -u

This removes the workarounds verified by neeshy to be unnecessary. See neeshy's comments reference in the below issue for more context. Refs: InBetweenNames#883 Signed-off-by: Jaremy Hatler <hatler.jaremy@gmail.com>

This comment was marked as off-topic.

Sign in to view

lubellier mentioned this issue Dec 15, 2023

qubes-core-qrexec-4.1.32 build fails in qubes 4.1 with "portageq is not allowed in ebuild scope" fepitre/qubes-gentoo#19

Closed

jhatler mentioned this issue Dec 26, 2023

Workaround cleanup jhatler/gentooLTO#3

Open

Portageq depreciated, causes warnings while building anything #883

Portageq depreciated, causes warnings while building anything #883

Comments

Phoenix591 commented Jun 7, 2023

parona-source commented Oct 22, 2023

kanyck commented Oct 25, 2023

antonellocaroli commented Oct 25, 2023

kir68k commented Oct 25, 2023 • edited

parona-source commented Oct 25, 2023 • edited

eternal-sorrow commented Oct 25, 2023 • edited

eternal-sorrow commented Oct 25, 2023

This comment was marked as off-topic.

eternal-sorrow commented Oct 25, 2023

kanyck commented Oct 25, 2023

antonellocaroli commented Oct 25, 2023 • edited

eternal-sorrow commented Oct 25, 2023 • edited

kanyck commented Oct 25, 2023 • edited

matoro commented Oct 26, 2023

jonasstein commented Oct 26, 2023

eternal-sorrow commented Oct 26, 2023

neeshy commented Oct 26, 2023

kanyck commented Oct 26, 2023

kir68k commented Oct 26, 2023

matoro commented Oct 26, 2023

parona-source commented Oct 26, 2023

matoro commented Oct 26, 2023

thesamesam commented Oct 26, 2023 • edited

thesamesam commented Oct 26, 2023

eternal-sorrow commented Oct 26, 2023 • edited

thesamesam commented Oct 26, 2023

eternal-sorrow commented Oct 26, 2023

thesamesam commented Oct 26, 2023 • edited

vimproved commented Oct 26, 2023

eternal-sorrow commented Oct 26, 2023 • edited

thesamesam commented Oct 26, 2023

eternal-sorrow commented Oct 26, 2023

negril commented Oct 27, 2023

neeshy commented Oct 28, 2023 • edited

thesamesam commented Oct 28, 2023

antonellocaroli commented Oct 28, 2023

neeshy commented Oct 28, 2023 • edited

negril commented Oct 28, 2023

antonellocaroli commented Oct 28, 2023

neeshy commented Oct 28, 2023

vimproved commented Oct 28, 2023

parona-source commented Oct 28, 2023

vimproved commented Oct 28, 2023

InBetweenNames commented Oct 28, 2023

neeshy commented Oct 28, 2023 • edited

eternal-sorrow commented Oct 29, 2023 • edited

InBetweenNames commented Oct 29, 2023

paralin commented Dec 7, 2023

neeshy commented Dec 13, 2023 • edited

kir68k commented Oct 25, 2023 •

edited

parona-source commented Oct 25, 2023 •

edited

eternal-sorrow commented Oct 25, 2023 •

edited

antonellocaroli commented Oct 25, 2023 •

edited

eternal-sorrow commented Oct 25, 2023 •

edited

kanyck commented Oct 25, 2023 •

edited

thesamesam commented Oct 26, 2023 •

edited

eternal-sorrow commented Oct 26, 2023 •

edited

thesamesam commented Oct 26, 2023 •

edited

eternal-sorrow commented Oct 26, 2023 •

edited

neeshy commented Oct 28, 2023 •

edited

neeshy commented Oct 28, 2023 •

edited

neeshy commented Oct 28, 2023 •

edited

eternal-sorrow commented Oct 29, 2023 •

edited

neeshy commented Dec 13, 2023 •

edited