[UNMAINTAINED] Android Container
This document describes how to setup and run multiple isolated Android user-space instances on a commercial mobile device like Nexus One. Operating system-level virtualization method Linux Containers (LXC) is applied to create and run Android Containers on a single control host. The standard Android Kernel is modified to boot GNU-Linux from SD-card and to provide isolation mechanisms and resource management features for Android user-space.
- Nexus One
- SD-card (> 2GB recommended)
- x86-Linux environment
- Sun Java 6 JDK, JRE
- Version Control tools Repo and Git (download)
In this section we describe how to modify the stock Nexus One firmware (mainly boot.img) to be able to boot a GNU-Linux directly from SD-card. This method enables us to run a plain GNU-Linux environment natively (no chroot!) on the device hardware. Additionally kernel configuration options and source code are patched to provide LXC container virtualization environment. Finally we describe how the modified kernel is properly packaged into a boot.img and flashed on the Nexus One device.
In order to compile the kernel from scratch a toolchain for cross-compiling is needed. Please follow instructions from Google here for setting up a Linux or MacOSX build environment and here for getting the Android SDK which contains a pre-built toolchain. The Android source directory is from now on indicated as ANDROID_DIR.
The kernel sources for Nexus One can be downloaded through the command:
$ git clone https://android.googlesource.com/kernel/msm.git
The repository is created in the subfolder msm
$ cd <MSM_DIR>
Note: Currently we can only provide a patch for 2.6.35.7 kernel.
Download the sdcard-lxc-kernel.tar.gz patch and extract the content:
- sdcard-lxc-kernel.patch - patches kernel for SD-card boot and LXC support
- readme.txt - description how to build
Apply the patch:
<MSM_DIR>$ patch -p1 < sdcard-lxc-kernel.patch
Now we are ready to build the kernel:
<MSM_DIR>$ make ARCH=arm CROSS_COMPILE=<ANDROID_DIR>/prebuilt/linux-x86/toolchain/arm-eabi-4.4.3/bin/arm-eabi- -j4
Note: In case the new kernel has not the correct kernel revision number or a -dirty flag is appended, you have to create a .scmversion file inside the kernel directory with a proper revision number and build the kernel again:
<MSM_DIR>$ echo '-g3cc95e3' > .scmversion
<MSM_DIR>$ make ARCH=arm CROSS_COMPILE=<ANDROID_DIR>/prebuilt/linux-x86/toolchain/arm-eabi-4.4.3/bin/arm-eabi- -j4
The newly created kernel MSM_DIR/arch/arm/boot/zImage needs to be packaged into a boot.img before it can be flashed onto device. In order to create and flash the boot.img we need to compile the necessary tools first:
$ cd <ANDROID_DIR>
<ANDROID_DIR>$ . build/envsetup.sh
<ANDROID_DIR>$ lunch full_passion-eng
<ANDROID_DIR>$ make -j4
After a successfull build tools like mkbootimg, fastboot and adb are available in the ANDROID_DIR/out/host/linux-x86/bin directory. We add it to the PATH variable:
$ export PATH=<ANDROID_DIR>/out/host/linux-x86/bin:$PATH
The goal of the next step is to package the newly created kernel zImage and the unmodified ramdisk.img into a new boot.img. Since we don't want to boot from the ramdisk but directly from the SD-card, the kernel cmdline needs to be modified permanently. The kernel replacement and cmdline modification is achieved by this command:
$ mkbootimg \
--kernel <MSM_DIR>/arch/arm/boot/zImage \
--ramdisk <ANDROID_DIR>/out/target/product/passion/ramdisk.img \
--cmdline "no_console_suspend=1 wire.search_count=5 root=/dev/mmcblk0p2 rw rootfs=ext2 init=/sbin/init rootwait noinitrd" \
--base 0x20000000 \
--output <ANDROID_DIR>/out/target/product/passion/boot.img
Note: For future target devices the correct mkbootimg parameters can be easily determined by recompiling the boot.img with the showcommands option:
<ANDROID_DIR>$ make bootimage showcommands
Note: If you don't want to build your own image a fully working boot.img can be downloaded here sdcard-lxc-boot.img
Now we can flash the new boot.img onto the boot partition. The device has to be rooted and the USB permissions have to be setup properly. In case of any problems check here and here (section: Configuring USB Access).
$ adb reboot bootloader
$ fastboot flash boot <ANDROID_DIR>/out/target/product/passion/boot.img
Note: Once you replaced the boot.img with modified cmdline to boot from SD-card, you can easily update the kernel zImage on the device without the need to create a boot.img again:
$ fastboot flash zimage <MSM_DIR>/arch/arm/boot/zImage
Since we don't touch the original system and userdata partitions, the original boot.img can be flashed to recovery partition to provide a convenient way to boot 'the normal way' from the bootloader menu.
In the next step we prepare the GNU-linux environment on the SD-card and configure LXC tools for the Android Container.
For our virtualization solution with Linux Containers (LXC) a standard GNU-linux, in this case Debian (squeeze) distribution is configured to run as host system directly from SD-card on the mobile device. This setup has several advantages, mainly a convenient way of installing new software packages currently available in the Debian repositories for the armel architecture and further avoiding possible space restrictions of the built-in device flash memory.
The following steps have to be performed on any external linux-system with SD-card slot.
- Create two partitions on the SD-card, a small vfat partition as first partition to simulate the standard SD-card and a second bigger ext2 partition (>2GB) where we place our host system and the Android containers (ext3 & ext4 is not recommended due to journaling capabilities which reduce the lifetime of SD-card).
- Mount the ext2 partition on the linux-system to any directory MNT_DIR and debootstrap a minimal Debian (sqeeze) for the armel architecture:
$ sudo mount /dev/mmcblk0p2 <MNT_DIR>
$ debootstrap --foreign --arch=armel --variant=minbase \
squeeze <MNT_DIR> http://ftp.debian.org/debian
For the second step of debootstrapping the system we need to access the debian ext2 partition from the mobile device. In this step all the necessary device nodes etc. are created, thus this task has to be performed in our target system environment, i.e. in the adb shell after a regular boot:
$ adb shell
# mount -t ext2 /dev/block/mmcblk0p2 <MNT_DIR>
# chroot <MNT_DIR> /debootstrap/debootstrap --second-stage
Note: mount/chroot can be executed with help of a busybox which can be downloaded here. How to use busybox on an Android device is explained for example here.
After a basic Debian system on the ext2 partition of the SD-card is created, several configuration changes have to be made in order to be able to directly boot from it. Thus minimum required settings are presented to perform the very first successful boot and then additional configurations are shown like networking, sound, X11, etc.
Since there is usually no serial cable available (example how to build a serial cable can be found here) we have to ensure that the GNU/Linux environment boots up without errors and the adb daemon is started at system startup. If that fails there is no possibility to access the system.
Perform a regular Android boot and execute these commands:
$ adb shell
# mount -t ext2 /dev/block/mmcblk0p2 <MNT_DIR>
# cp /sbin/adbd <MNT_DIR>/sbin/adbd
# cat > <MNT_DIR>/etc/rc.local << EOF
> /sbin/adbd &
> exit 0
> EOF
# mkdir -p <MNT_DIR>/system/bin/
# ln -s /bin/bash <MNT_DIR>/system/bin/sh
# mkdir -p /cgroup /media/system /media/cache /media/userdata
# cat > <MNT_DIR>/etc/fstab << EOF
> /dev/block/mmcblk0p2 / ext2 defaults 1 0
> proc /proc proc defaults 0 0
> devpts /dev/pts devpts defaults 0 0
> sysfs /sys sysfs defaults 0 0
>
> none /cgroup cgroup defaults 0 0
> /dev/mtdblock3 /media/system yaffs defaults 0 0
> /dev/mtdblock4 /media/cache yaffs defaults 0 0
> /dev/mtdblock5 /media/userdata yaffs defaults 0 0
> EOF
Note: Mounting of cgroup partition is needed later for LXC and the yaffs partitions gives us a conventient access to stock Android files, e.g. firmware.
Now the Debian host system is ready for the first native boot! Test after a few minutes via adb and if that works reboot the device normally to perform further configuration steps. Otherwise check previous steps for errors.
Next we need to install some packages from the repository, e.g. wpasupplicant for networking. Since we need obviously internet connection to do this, we have to boot Android normally and establish an internet connection by its means. Then we chroot into Debian to install the necessary packages:
$ adb shell
# mount -t ext2 /dev/block/mmcblk0p2 <MNT_DIR>
# chroot <MNT_DIR> /bin/bash
$ export PATH=/usr/bin:/usr/sbin:/bin:$PATH
$ export TERM=linux
$ echo 'deb http://ftp.debian.org/debian squeeze main' > /etc/apt/sources.list
$ apt-get update
$ apt-get install wpasupplicant wireless-tools
$ apt-get install locales vi
$ exit
In order to setup networking for the Debian host system, we have to copy the wifi kernel module and the firmware from the regular Android file system, thus they can be loaded at startup from SD-card. Then wpasupplicant is configured to be able to establish wifi connections from within the Debian environment. These commands should be executed inside the Android shell (not chroot!):
$ mkdir -p <MNT_DIR>/lib/modules/`uname -r`/drivers/net/wireless/
$ cp /system/lib/modules/bcm4329.ko \
<MNT_DIR>/lib/modules/`uname -r`/drivers/net/wireless/
$ mkdir -p <MNT_DIR>/system/vendor/firmware/
$ cp /system/vendor/firmware/fw_bcm4329.bin \
<MNT_DIR>/system/vendor/firmware/
$ echo 'bcm4329' > <MNT_DIR>/etc/modules
$ cat > <MNT_DIR>/etc/network/interfaces << EOF
< auto lo
< iface lo inet loopback
<
< auto eth0
< iface eth0 inet manual
< wpa-driver wext
<
< wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
<
< iface your_wlan_name inet dhcp
< EOF
$ cat > <MNT_DIR>/etc/wpa_supplicant/wpa_supplicant.conf << EOF
< ctrl_interface=/var/run/wpa_supplicant
< ap_scan=2
< fast_reauth=1
< network={
< ssid="your_wlan_ssid"
< id_str="your_wlan_name"
< scan_ssid=1
< mode=0
< proto=WPA
< key_mgmt=WPA-PSK
< pairwise=CCMP TKIP
< group=TKIP
< psk="your_wlan_password"
< }
< EOF
In case the wpasupplicant settings were correct, we should be able to connect to the local access point and establish an internet connection. The wpa-roam option makes it possible to preconfigure several APs which are dynamically chosen according to current location of the device.
Since the regular Android environment for networking activities is not required anymore, the Nexus One device can be booted directly from SD-card and all the following steps can be performed in the plain Debian environment accessible through the adb shell.
For convenience a desktop environment should be installed. Currently XFCE4 works best since it's lightweight and does not start with a login screen, which is a problem due the lack of a hardware keyboard. We use the Debian backports version of xserver-xorg 1:7.6+8~bpo60+1, since the stable version 1:7.5+8+squeeze1 has minor touchscreen handling issues.
$ apt-get install xfce4
$ echo 'deb http://backports.debian.org/debian-backports squeeze-backports main' \
>> /etc/apt/sources.list
$ apt-get update
$ apt-get install xserver-xorg
$ startx &
Note: To trigger the start of XFCE4 at startup, place 'startx &' inside the /etc/rc.local configuration file before the 'exit 0' statement.
A decent software keyboard solution is matchbox, which provides well proportioned buttons and also can be activated in foreground when a input field is active.
As next we fix the sound inside Debian to avoid crashing of our Android containers when a sound action is performed. We copy the according binaries to the standard firmware folder where the kernel can find it:
$ cp -pR /media/system/etc/firmware/default*.acdb /lib/firmware
In this section we have demonstrated how the basic Debian host system should be configured to enable networking, desktop environment and sound. Further possibilities should be explored to enable GSM radio and bluetooth.
This section is intended to give an overview on setting up the Linux Container (LXC) environment in our Debian host system running natively on a Nexus One device. Following prerequisites are described to run basically any Linux Container on this system setup without the restriction to a particular container like Android Container.
Since the control group file system (cgroup) was setup and mounted in previous steps, we continue with installing the patched LXC package.
Download this patch lxc-0.7.5-fd-patch.tar.gz and apply it to lxc-0.7.5.tar.gz sources, compile and install:
<LXC_SRC_DIR>$ patch -p1 < lxc-0.7.5-fd.patch
<LXC_SRC_DIR>$ ./configure
<LXC_SRC_DIR>$ make
<LXC_SRC_DIR>$ make install
Before we create a specific Android container, network bridging has to be configured to enable network access from within the isolated container environment:
$ cat >> /etc/network/interfaces << EOF
> auto br0
>
> iface br0 inet static
> address 192.168.99.1
> broadcast 0.0.0.0
> netmask 255.255.255.0
> bridge_ports none
> bridge_fd 0
> bridge_maxwait 0
> bridge_hello 0
> bridge_maxage 12
> bridge_stp off
> post-up ip route add 192.168.99.201 dev br0
> EOF
$ ifup br0
Note: An additional bridge device br0 is created with a static ip address. It's not connected to the Debian eth0 interface (bridge_ports none) since the lunch of the Android container alters the MAC address of the bridge which results in connectivity loss to the access point. Instead we use netfilter iptables to route the ip packets to the outer world. For DNS forwarding install dnsmasq and configure as local DNS server. Note: activating the ebtables kernel option in the kernel results in an immediate crash and reboot of the device when a network connection is about to be established. Keep that in mind if you decide to modify the kernel options.
Now we are ready to create our first Android Container inside our Debian host.
In order to create a complete Android system, for instance, the file system structure has to be recreated inside the container rootfs. Following folder tree is created, where ANDROID_LXC contains the rootfs subfolder and the neccesary configuration file for LXC:
+-- <ANDROID_LXC>
|-- config
+-- rootfs
Download the sdcard-android-lxc-rootfs.tar.gz tarball, copy to ANDROID_LXC/rootfs folder and extract:
<ANDROID_LXC>/rootfs$ tar -xzvf sdcard-android-lxc-rootfs.tar.gz
Note: All in this image included modifications were applied to Android 2.3.x Gingerbread and are explained in section 4.3.
As next step we introduce the LXC configuration file settings which enable us to run the Android Container (to be placed inside the ANDROID_LXC/config file):
lxc.utsname = android
lxc.tty = 4
lxc.rootfs = <ANDROID_LXC>//rootfs
# network
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.ipv4 = 192.168.99.202 0.0.0.0
lxc.network.name = eth0
lxc.network.veth.pair = vethvm2
lxc.cgroup.devices.deny = a # deny all first
# mount points
lxc.mount.entry=none <ANDROID_LXC>//rootfs/proc proc defaults 0 0
lxc.mount.entry=none <ANDROID_LXC>//rootfs/sys sysfs defaults 0 0
lxc.mount.entry=/lib <ANDROID_LXC>//rootfs/lib none ro,bind 0 0
lxc.mount.entry=/usr/share/locale <ANDROID_LXC>//rootfs/usr/share/locale none rw,bind 0 0
Through the extensive device nodes permission setting we are able to deny access to any particular device node throgh the LXC resource isolation mechanism. These settings have to be appended to the configuration file:
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:* rwm
lxc.cgroup.devices.allow = c 4:* rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
# /dev/pts/* - pts namespaces
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
# nexus specific
lxc.cgroup.devices.allow = c 1:7 rwm # dev/full
lxc.cgroup.devices.allow = c 1:11 rwm # dev/kmsg
lxc.cgroup.devices.allow = c 7:* rwm #
# cpu and memory
#lxc.cgroup.cpuset.cpus = 0
#lxc.cgroup.cpu.shares = 1024
#lxc.cgroup.memory.limit_in_bytes = 512M
#lxc.cgroup.memory.memsw.limit_in_bytes = 512M
# 10:* devices
lxc.cgroup.devices.allow = c 10:0 rwm # dev/pmem
lxc.cgroup.devices.allow = c 10:1 rwm # dev/pmem_adsp
lxc.cgroup.devices.allow = c 10:2 rwm # dev/pmem_camera
lxc.cgroup.devices.allow = c 10:223 rwm # dev/uinput
lxc.cgroup.devices.allow = c 10:30 rwm
lxc.cgroup.devices.allow = c 10:31 rwm
lxc.cgroup.devices.allow = c 10:32 rwm
lxc.cgroup.devices.allow = c 10:33 rwm
lxc.cgroup.devices.allow = c 10:34 rwm
lxc.cgroup.devices.allow = c 10:35 rwm
lxc.cgroup.devices.allow = c 10:36 rwm
lxc.cgroup.devices.allow = c 10:37 rwm
lxc.cgroup.devices.allow = c 10:39 rwm
lxc.cgroup.devices.allow = c 10:40 rwm # /dev/keychord
lxc.cgroup.devices.allow = c 10:41 rwm
lxc.cgroup.devices.allow = c 10:42 rwm
lxc.cgroup.devices.allow = c 10:43 rwm
lxc.cgroup.devices.allow = c 10:44 rwm
lxc.cgroup.devices.allow = c 10:45 rwm
lxc.cgroup.devices.allow = c 10:46 rwm
lxc.cgroup.devices.allow = c 10:47 rwm
lxc.cgroup.devices.allow = c 10:48 rwm
lxc.cgroup.devices.allow = c 10:49 rwm
lxc.cgroup.devices.allow = c 10:50 rwm
lxc.cgroup.devices.allow = c 10:51 rwm
lxc.cgroup.devices.allow = c 10:52 rwm
lxc.cgroup.devices.allow = c 10:53 rwm
lxc.cgroup.devices.allow = c 10:54 rwm
lxc.cgroup.devices.allow = c 10:55 rwm
lxc.cgroup.devices.allow = c 10:56 rwm
lxc.cgroup.devices.allow = c 10:57 rwm
lxc.cgroup.devices.allow = c 10:58 rwm
lxc.cgroup.devices.allow = c 10:59 rwm
lxc.cgroup.devices.allow = c 10:60 rwm
lxc.cgroup.devices.allow = c 10:61 rwm
lxc.cgroup.devices.allow = c 10:62 rwm
lxc.cgroup.devices.allow = c 10:63 rwm
# dev/input/*
lxc.cgroup.devices.allow = c 13:64 rwm # event0 -> lightsensor-level
lxc.cgroup.devices.allow = c 13:65 rwm # event1 -> h2w headset
lxc.cgroup.devices.allow = c 13:66 rwm # event2 -> compass
lxc.cgroup.devices.allow = c 13:67 rwm # event3 -> synaptics-rmi-touchscreen
lxc.cgroup.devices.allow = c 13:68 rwm # event4 -> proximity
lxc.cgroup.devices.allow = c 13:69 rwm # event5 -> mahimahi-keypad
lxc.cgroup.devices.allow = c 13:70 rwm # event6 -> mahimahi-nav
lxc.cgroup.devices.allow = c 29:0 rwm # dev/fb0
lxc.cgroup.devices.allow = c 31:* rwm # dev/block/mtdblock*
lxc.cgroup.devices.allow = c 90:* rwm # dev/mtd
lxc.cgroup.devices.allow = c 108:0 rwm # dev/ppp
lxc.cgroup.devices.allow = c 179:* rwm # dev/block/mmcblk0*
lxc.cgroup.devices.allow = c 248:* rwm # control0, config0, frame0
lxc.cgroup.devices.allow = c 251:0 rwm # dev/q6venc
lxc.cgroup.devices.allow = c 252:0 rwm # dev/vdec
lxc.cgroup.devices.allow = c 250:0 rwm # dev/ttyHS0
lxc.cgroup.devices.allow = c 253:* rwm
Here we describe which changes that had to be applied to Android 2.3.x Gingerbread to run the Android Container on the Nexus One device. We only list the components which were modified since an extensive description would extend the scope of this document:
init.rc, init, DalvikVM, system_server, busybox (route), dns, etc...
If neccessary a button trigger daemon can be installed to manage the start of Android Container by pressing predefined hardware buttons. For this purpose the Debian package triggerhappy can be installed to execute a shell script which starts the Android Container and which is triggered by events on /dev/input/event
* devices.Here is a list of ongoing feature development:
- sdcard emulation
- lxc-console
- parallel fb access
- secureSD
http://en.wikipedia.org/wiki/Operating_system-level_virtualization
http://lxc.sourceforge.net/
http://lxc.teegra.net/
http://www.irregular-expression.com/?p=30
http://www.saurik.com/id/10
http://source.android.com/source/downloading.html
- Introduction
- How To
- Access Control
- UICC Support
- ASSD Support
- MSC Support
- PC/SC Interface
- Service Layer
- Applications
- FAQS
- Abbreviations
- Contributions