Need help with proper usage of lxc.id_map

[Frimi]

Required information

• Distribution: Embedded Linux

lxc-start --version
1.1.4

~# lxc-checkconfig
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: missing
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: enabled
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: missing

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: missing
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

name -a
Linux 3.14.52 #1 SMP Tue Mar 19 20:45:00 UTC 2024 armv7l GNU/Linux

 cat /proc/self/cgroup
9:debug:/
8:blkio:/
7:freezer:/
6:devices:/system.slice/system-serial\x2dgetty.slice
5:memory:/
4:cpuset:/
3:cpu,cpuacct:/
2:name=systemd:/system.slice/system-serial\x2dgetty.slice/serial-getty@ttyFP0.service

Context:

• I'm working on a somewhat dated embedded device (based on yocto 2.0 - jethro) and we recently added LXC version 1.1.4 via meta-virtualization layer

• We currently use systemd-nspawn and we're now attempting a migration to LXC, so it'd be great to achieve a similar way of working then the one we have today with systemd-nspawn for containerized apps (in terms of networking, etc)

• The way we work today with containerized apps is that processes spawned with systemd-nspawn will:

  • 1. Have the network namespace shared between host and guest OS

       • I'm able to achieve this with LXC using lxc.network.type = none

  • 2. Be owned by a user with UID != 0 (non-root) and we have an iptables rule where traffic with UID != 0 will be redirected to one specific port where we have an ACL, etc...

       • This is where I'm having a hard time right now and would appreciate some clarification!

Question(s):

• I've made progress already, but I still need help fine-tuning my container config files:

  • Should I specify lxc.id_map in container creation time or execution time? (I've tried both and each one has it's issues)

  • Maybe I should substitute lxc.mount.entry by a different option that would make things persist after container creation?

My tests so far:

Test 1

• Description: Specifying lxc.id_map options in creation time
• Conclusion: rootfs is created with UID=1015 BUT I have permission issues in execution time

root@mydevice:~# cat conf_A.conf 
lxc.mount.entry = /etc/mydevice etc/mydevice none ro,bind,create=dir 0 0       
lxc.mount.entry = /dev/p4_time dev/p4_time none ro,bind,create=file 0 0
lxc.utsname = alpha
lxc.network.type = none
lxc.id_map = u 0 1015 1
lxc.id_map = g 0 1015 1
root@mydevice:~# cat /etc/subuid 
root:1015:1     
root@mydevice:~# cat /etc/subgid 
root:1015:1
root@mydevice:~# lxc-create -n foo -f conf_A.conf -t n2busybox
This is path: /var/lib/lxc/foo
This is rootfs: /var/lib/lxc/foo/rootfs
This is SSH:
This is in_userns: 1
root@mydevice:~# lxc-ls
foo
root@mydevice:~# ls -al /var/lib/lxc/foo/ 
drwxrwx---    3 1015     1015          4096 Jan  1 05:07 .
drwxr-xr-x    3 root     root          4096 Jan  1 05:07 ..    
-rw-r--r--    1 root     root          1529 Jan  1 05:07 config
drwxr-xr-x   15 1015     1015          4096 Jan  1 05:07 rootfs
root@mydevice:~# lxc-execute -n foo -- sh
lxc-execute: utils.c: safe_mount: 1642 Operation not permitted - Failed to mount sysfs onto /usr/lib/lxc/rootfs/sys
lxc-execute: conf.c: lxc_mount_auto_mounts: 828 Operation not permitted - error mounting sysfs on /usr/lib/lxc/rootfs/sys flags 14
lxc-execute: conf.c: lxc_setup: 3736 failed to setup the automatic mounts for 'foo'
lxc-execute: start.c: do_start: 702 failed to setup the container
lxc-execute: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-execute: start.c: __lxc_start: 1172 failed to spawn 'foo'

Test 2

• Description: Specifying lxc.id_map options in execution time
• Conclusion: rootfs is created with UID=0 and I can execute, BUT some lxc.mount.entry options are not mounted due to permission problems!

root@mydevice:~# cat conf_A.conf 
lxc.mount.entry = /etc/mydevice etc/mydevice none ro,bind,create=dir 0 0
lxc.mount.entry = /dev/p4_time dev/p4_time none ro,bind,create=file 0 0
lxc.utsname = alpha
lxc.network.type = none
root@mydevice:~# lxc-create -n bar -f conf_A.conf -t n2busybox
This is path: /var/lib/lxc/bar
This is rootfs: /var/lib/lxc/bar/rootfs
This is SSH:
This is in_userns: 0
root@mydevice:~# lxc-ls
bar  foo
root@mydevice:~# ls -al /var/lib/lxc/bar/
drwxrwx---    3 root     root          4096 Jan  1 05:13 .
drwxr-xr-x    4 root     root          4096 Jan  1 05:13 ..
-rw-r--r--    1 root     root           946 Jan  1 05:13 config
drwxr-xr-x   15 root     root          4096 Jan  1 05:13 rootfs
root@mydevice:~# cat conf_B.conf 
lxc.id_map = u 0 1015 1
lxc.id_map = g 0 1015 1
root@mydevice:~# lxc-execute -n bar -f conf_B.conf -- sh
init.lxc: log.c: log_open: 180 failed to open log file "/var/log/lxc/bar.log" : Permission denied
init.lxc: log.c: log_open: 180 failed to open log file "/var/lib/lxc/bar/bar.log" : Permission denied
init.lxc: log.c: log_open: 180 failed to open log file "/var/log/lxc/bar.log" : Permission denied
init.lxc: initutils.c: mount_fs: 36 failed to mount /dev/mqueue : No such device
sh-3.2# ls -al /dev/p4_time
ls: /dev/p4_time: No such file or directory
sh-3.2# ls -al /etc/mydevice/
drwxr-xr-x    2 nobody   nogroup       4096 Jan  1 00:03 .
drwxrwxr-x   35 nobody   nogroup       4096 Jan  1 00:11 ..
-rw-r--r--    1 nobody   nogroup       1559 Mar 15  2024 test.conf

• Follow up on Test2: trying to specify the lxc.mount.entry option of /dev/p4_time in execution time

root@mydevice:~# cat conf_B.conf 
lxc.mount.entry = /dev/p4_time dev/p4_time none ro,bind,create=file 0 0
lxc.id_map = u 0 1015 1
lxc.id_map = g 0 1015 1
root@mydevice:~# lxc-execute -n bar -f conf_B.conf -- sh
lxc-execute: utils.c: mkdir_p: 248 Permission denied - failed to create directory 'dev'
lxc-execute: conf.c: lxc_setup: 3746 failed to setup the mount entries for 'bar'
lxc-execute: start.c: do_start: 702 failed to setup the container
lxc-execute: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-execute: start.c: __lxc_start: 1172 failed to spawn 'bar'