Xen ARM with Virtualization Extensions/Allwinner

From Xen
Revision as of 09:07, 18 October 2013 by Bamvor (talk | contribs) (arm: sun7i: explain the dom0 kernel config)

this wiki talks about xen on A20(Cortex-A7 dual core, code name sun7i). A31(Cortex-A7 quad core, code name sun6i) should work as well.

Compile u-boot

as described in Firmware_Requirements, xen need boot from non-secure hyper mode.

get the code

a worked branch is here:


here is the details:

merge the hypmode branch
   git://git.linaro.org/people/aprzywara/u-boot.git hypmode_v4

branch into

   git://github.com/linux-sunxi/u-boot-sunxi.git u-boot-sunxi

i tested hypmode_v4 with commit 3d2fc4e8 in u-boot-sunxi. hypmode_v5 should work as well.

add Ian Campbell patches on it.
enable arm generic timer for sunxi, xen dom0 kernel need this
   diff --git a/include/configs/sun7i.h b/include/configs/sun7i.h
   index f0d9222..473d1eb 100644
   --- a/include/configs/sun7i.h
   +++ b/include/configs/sun7i.h
   @@ -36,6 +36,8 @@
    #define CONFIG_ARMV7_VIRT
   +#define CONFIG_SYS_CLK_FREQ     24000000
    #if defined(CONFIG_SYS_SECONDARY_ON)

refernce: bootwrapper can't be compiled for cubieboard2


for sd card

   make Cubieboard2 CROSS_COMPILE=arm-linux-gnueabihf- -j8

for usb boot

   make Cubieboard2_FEL CROSS_COMPILE=arm-linux-gnueabihf- -j8

Building Linux Dom0

First, we will build Linux for dom0 to have a device tree to provide to Xen.

You can get a tree from sun7i-xen-dom0 which is based on arokux sunxi usb branch. the emac patch in this tree is needed for enable network in dom0. i am confuse about what does "CONFIG XEN" mean. it will check the kernel features for dom0 and domU kernel. if you want to a dom0 kernel, you need to uncheck all the FRONTEND config, check CONFIG_XEN_BACKEND, and check the BACKEND config as you need:


or use sunxi dom0 kernel directly

 git clone -b sun7i-xen-dom0 https://github.com/bjzhang/linux-allwinner.git sun7i-xen-dom0
 cd sun7i-xen-dom0
 make sun7i_dom0_defconfig
 make zImage
 make dtbs

The device tree is located in arch/arm/boot/dts/sun7i-a20-cubieboard2-xen.dtb

Building Xen

the lastest xen should work. reference Build Xen on ARM and Building Xen and Linux Dom0 for building xen and generating the xen-uImage. i test pass on xen commit 720f45ad: Julien Grall, xen/evtchn: Fix build on ARM

use the following build command for enable early printk for a20:

   make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=sun7i CONFIG_DTB_FILE=/path/to/sun7i-a20-cubieboard2-xen.dtb

Building linux DomU

xen domU kernel is supported by upstream. and remember that domU guest do not aware of which hardware it running on, it only know about the mach-virt.

check the frontend driver in the config file.

currenly, xen directly use the dom0 gic dist base and cpu base as domU gic dist and cpu base. reference gicv_setup() in xen/arch/arm/gic.c for details. for sunxi, need chances the followings: (this patch is included in my branch)

   diff --git a/arch/arm/boot/dts/xenvm-4.2.dts b/arch/arm/boot/dts/xenvm-4.2.dts
   index 3369151..5d5689e 100644
   --- a/arch/arm/boot/dts/xenvm-4.2.dts
   +++ b/arch/arm/boot/dts/xenvm-4.2.dts
   @@ -50,13 +50,13 @@
           reg = <0 0x80000000 0 0x08000000>;
   -	gic: interrupt-controller@2c001000 {
   +	gic: interrupt-controller@1c81000{
           compatible = "arm,cortex-a15-gic", "arm,cortex-a9-gic";
           #interrupt-cells = <3>;
           #address-cells = <0>;
   -		reg = <0 0x2c001000 0 0x1000>,
   -		      <0 0x2c002000 0 0x100>;
   +		reg = <0 0x01c81000 0 0x1000>,
   +		      <0 0x01c82000 0 0x1000>;
       timer {

it should not need to change after ian auto-generate dts patch upstreamed.

 git clone -b sun7i_xen_domU https://github.com/bjzhang/linux-allwinner.git sun7i_xen_domU
 cd sun7i_xen_domU
 make sun7i_domU_defconfig
 make zImage
 make dtbs

Booting Xen and Dom0

To boot Xen and Dom0, you can use USB boot or sd boot.

Booting via PXE


Booting via USBBoot(FEL)

reference: http://linux-sunxi.org/FEL/USBBoot

Booting directly on the SD card

reference [1] for how to make a bootable sdcard.

  • Copy xen-uImage, zImage and the initrd the root directory of you SD card.
  • save the bootcmd
 setenv bootcmd "fatload mmc 0 45000000 xen-uImage; fatload mmc 0 50000000 zImage; fatload mmc 0 60000000 initrd; bootm 45000000"
  • boot

Assume the SD card has only 1 partition with fat filesystem.

Mount NFS after initrd boot

Using a distrbution is easily to compile xen tools. i use opensuse arm 12.3 rootfs for it. other distribution which support armhf is ok too.

Create domU

setup the harddisk
   losetup /dev/loop0 /root/domU/rootfs.ext3 
domU config example
   > cat domU_test
   kernel = "/root/domU/zImage_dtb" 
   memory = "128" 
   name = "domU" 
   vcpus = 1 
   disk = [ 'phy:/dev/loop0,xvda,w' ]
   vif=[ 'mac=00:16:3e:56:af:69,bridge=virbr0,type=netfront', ]
   xl create domU_test

or enable all the debug message

   xl -vvv create -d domU_testa
connect to console

xl console domU_test

DomU kernel debug

xen support varity debug method. you could use xl debug-keys for dumping information. use xen_raw_console_write for output DomU kernel output. but gdbsx is not work for arm right now.

   diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
   index b4e8500..060c163 100644
   --- a/kernel/printk/printk.c
   +++ b/kernel/printk/printk.c
   @@ -1488,6 +1488,7 @@ static size_t cont_print_text(char *text, size_t size)
       return textlen;
   +#include <xen/hvc-console.h>
    asmlinkage int vprintk_emit(int facility, int level,
                   const char *dict, size_t dictlen,
                   const char *fmt, va_list args)
   @@ -1546,6 +1547,7 @@ asmlinkage int vprintk_emit(int facility, int level,
        * prefix which might be passed-in as a parameter.
       text_len = vscnprintf(text, sizeof(textbuf), fmt, args);
   +    xen_raw_console_write(text);
       /* mark and strip a trailing newline */
       if (text_len && text[text_len-1] == '\n') {


allwinner sunxi resources [2]