lxc-attach does not work with unprivileged containers and shared net namespace

[ElJeffe]

If a container is started unprivileged with a shared namespace, then it is not possible to attach to the container with error:
lxc-attach netns 20240513090718.131 ERROR attach - ../src/lxc/attach.c:__attach_namespaces_nsfd:666 - Operation not permitted - Failed to attach to net namespace of 6468 lxc-attach netns 20240513090718.131 ERROR attach - ../src/lxc/attach.c:lxc_attach:1590 - Failed to enter namespaces

This is reproducible on lxc 5.0.2 and master (6.0.0) on linux < 5.8 since form 5.8 __attach_namespaces_pidfd is called which works.
Attached is a config that is used to reproduce.
config.txt
First changing to the shared namespaces before the other namespaces works, as is done here:
ElJeffe@2dfe2ff