Running Linux in QEMU’s aarch64 system emulation mode

Running Linux in QEMU’s aarch64 system emulation mode

Since I started working on aarch64 support for QEMU the most frequently asked question I got was “when can I run aarch64 system emulation on QEMU?”. Well wait no more as support for a VIRT-IO based aarch64 board was recently merged into the master branch of QEMU. In this post I’ll talk about building QEMU, a rootfs and a kernel that will allow you to start experimenting with the architecture.

Quick start

Let’s first start with building and running QEMU with some pre-built images.

BUILD DEPENDANCIES

As has been noted in the comments the configure script will automatically enable features as long as the pre-requisite developer libraries are installed on your sytem. With a Debian/Ubuntu system this is easily achieved by running:

sudo apt-get build-dep qemu

Of course if you want to enable a feature (either a bleeding edge or non-standard) that requires additional libraries then you will need to install the appropriate development packages manually. The config.log file is usually a useful first step in working out what headers are being looked for.

BUILDING QEMU

git clone git://git.qemu.org/qemu.git qemu.git cd qemu.git ./configure --target-list=aarch64-softmmu make

Assuming the build ran without any problems you should now have an executable ./aarch64-softmmu/qemu-system-aarch64 in your build directory. Grab a pre-built image from here and we’ll check it works. The image is a kernel that has been combined with an initial RAM disk (initrd) with a basic root file-system. I go into more details on how to create this later on.

Be aware the command line is quite long so make sure you copy it all 

wget http://people.linaro.org/~alex.bennee/images/aarch64-linux-3.15rc2-buildroot.img ./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0"

If all went well you should see the familiar Linux boot sequence and eventually get a login prompt. Login as root (no password) and play in the new sandbox.

... usual kernel boot output ...
Welcome to Buildroot
buildroot login: root
# uname -a
Linux buildroot 3.15.0-rc2ajb-00069-g1aae31c #39 SMP Thu Apr 24 11:48:57 BST 2014 aarch64 GNU/Linux

Once you are done type C-a c to enter QEMU’s monitor mode and then quit to exit.

QEMU 2.0.50 monitor - type 'help' for more information
(qemu) quit

Accessing your local file-system

This is all very well but the test image only has a fairly limited root file-system attached to it. It will be a lot more useful if you could access your host file-system to test other binaries. Thanks to VIRT FS we can achieve this without too much hassle. Use the following extended QEMU command line:

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" -fsdev local,id=r,path=/home/alex/lsrc/qemu/rootfs/trusty-core,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r

This sets up the selected path to be mountable by the guest. In this case I’m using an Ubuntu rootfs which can be downloaded from here. Once the system has booted the following commands on the guest will mount the local file-system:

Welcome to Buildroot buildroot login: root # mount -t 9p -o trans=virtio r /mnt # ls -l /mnt/ total 84 drwxr-xr-x 2 default default 4096 Apr 2 2014 bin drwxr-xr-x 2 default default 4096 Feb 27 2014 boot drwxr-xr-x 3 default default 4096 Apr 2 2014 dev drwxr-xr-x 64 default default 4096 Apr 3 2014 etc drwxr-xr-x 2 default default 4096 Feb 27 2014 home ..

Building your own rootfs

There are many solutions to this (including downloading Linaro engineering builds) but the simplest one I’ve found for rolling your own from scratch is the Buildroot project. It present the familiar kernel menuconfig interface and deals with all the hassle of setting up cross compilers for you.

git clone git://git.buildroot.net/buildroot buildroot.git cd buildroot.git make menuconfig

There are lots of configuration options to choose from but the following are what I use:

* Target Options -> Target Architecture(AArch64)
* Toolchain -> Toolchain type (External toolchain)
* Toolchain -> Toolchain (Linaro AArch64 14.02)
* System configuration -> Run a getty (login prompt) after boot (BR2_TARGET_GENERIC_GETTY)
* System configuration -> getty options -> TTY Port (ttyAMA0) (BR2_TARGET_GENERIC_GETTY_PORT)
* Target Packages -> Show packages that are also provided by busybox (BR2_PACKAGE_BUSYBOX_SHOW_OTHERS)
* Filesystem images -> cpio the root filesystem (for use as an initial RAM filesystem) (BR2_TARGET_ROOTFS_CPIO)

The last one will be important for when we build the kernel next. Once you have configured buildroot to your liking it’s time to type make and leave it for a while as you enjoy a nice lunch 

make .. lots of output ..

Building a kernel

$ sudo apt-get install gcc-aarch64-linux-gnu

And the usual kernel building process, with a few tweaks for cross compiling:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux.git cd linux.git ARCH=arm64 make defconfig ARCH=arm64 make menuconfig （one cmd）

I’ve put my full config up here but important options to note are:

CONFIG_CROSS_COMPILE="aarch64-linux-gnu-"                                               # needs to match your cross-compiler prefix
CONFIG_INITRAMFS_SOURCE="/home/alex/lsrc/qemu/buildroot.git/output/images/rootfs.cpio"  # points at your buildroot image
CONFIG_NET_9P=y                                                                         # needed for virtfs mount
CONFIG_NET_9P_VIRTIO=y

Finally you build it all with:

ARCH=arm64 make -j 8  
（one cmd）

The -j 8 just specifies how many parallel build threads to use. Generally set it to the number of cores you have on your machine.

Final test

All that remains is to test that the newly built kernel works:

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image --append "console=ttyAMA0" ... lots more output ... Welcome to Buildroot ajbtest login: root [root@ajbtest ~]# ls -l total 0 [root@ajbtest ~]# uname -a Linux ajbtest 3.15.0-rc4ajb-00320-gafcf0a2-dirty #41 SMP Fri May 9 13:05:31 BST 2014 aarch64 GNU/Linux

UPDATED: 27/05/2014
* Added notes about library dependencies
* Cleaned up formatting of shell sections, mention length of command line!
* Fix some spelling errors

Filed under geek   |  Tagged aarch64, arm64, linaro, qemu   |  31 Comments   |  Permalink

31 Comments

1. Tariq Masood
   May 20, 2014 at 6:58 pm

   Thanks for writing this Alex. Very easy to follow.

   Reply to this comment
2. Tariq Masood
   May 22, 2014 at 4:14 pm

   Note that VIRTFS and VIRTIO features need to be enabled in the configuration for building QEMU to get the option “-fsdev local,id=r,path=/home/alex/lsrc/qemu/rootfs/trusty-core,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r” to work.

   Reply to this comment
   • Alex
     May 22, 2014 at 8:15 pm

     I did a fresh re-build when writing this so I assume it’s enabled automatically when the pre-reqs are around. I shall check next week when I’m back in the office.

     Thanks for the feedback 

     Reply to this comment
3. Wei Xu
   May 23, 2014 at 8:15 am

   Thanks for your guide Alex.

   I also did not find the virtio-9p-device firstly.
   In my host, I found missing the “libcap-devel” and “libattr-devel”.
   After I installed the two libraries and re-build, I found that the VIRTFS was enabled automatically:)

   Reply to this comment
   • Alex
     May 27, 2014 at 4:08 pm

     I’ve updated the post to mention library dependencies. I assume your system is RPM based? Is there a similar build-dep type command that can pull in the extra development libraries for those systems?

     Reply to this comment
4. Raghu
   June 4, 2014 at 2:48 pm

   Alex, Thanks for the article.

   I am using Ubuntu 12.04 LTS. I was able to successfully run Linux on Qemu following the instructions above and using the pre-built image provided by you.
   However, on building the rootfs on my own as instructed above, Qemu is not showing any output and no buildroot login when invoked with the newly built “../linux.git/arch/arm64/boot/Image” on my system.

   Since the command “sudo apt-get install gcc-aarch64-linux-gnu” could not fetch toolchain on ubuntu 12.04, I have used the toolchain present in “buildroot.git/output/host/opt/ext-toolchain/bin” for compiling the kernel. I configured the kernel as instructed above.

   Any pointers on resolving the problem is very much appreciated.

   Thanks & Best Regards,
   Raghu.

   Reply to this comment
   • Alex
     June 5, 2014 at 8:49 am

     It sounds like the initrd image hasn’t linked correctly. Can you pastebin your boot-up output?

     Reply to this comment
   • Raghu
     June 5, 2014 at 8:50 am

     Problem Solved. Configuration file provided in the website used instead of changing options in menuconfig. rootfs.cpio path alone is modified in config file.

     However, AFAIK, the major differences between the config files(menuconfig changes ->.config & aarch64-kernel.config) are
     CONFIG_NET_9P=m, CONFIG_NET_9P_VIRTIO=m, CONFIG_9P_FS=m instead of
     CONFIG_NET_9P=y, CONFIG_NET_9P_VIRTIO=y, CONFIG_9P_FS=y.

     Thanks & Best Regards,
     Raghu.

     Reply to this comment
     • Alex
       June 11, 2014 at 10:47 am

       Ah yes modules add an extra level of confusion as you need to ensure the initramfs has them and they get loaded as required. They make a lot of sense for generic binary distributions that need to boot on ${RANDOMHW} but for personal kernels and well-known requirements for VMs it saves a lot of pain just to build them directly in.

       Reply to this comment
5. Ori Chalak
   June 24, 2014 at 1:05 pm

   Hi Alex,

   Thanks for the great post. Was easy to follow!

   I tried to run multiple guest cpus using: -smp 2
   Result:
   qemu-system-aarch64: mach-virt: must enable KVM to use multiple CPUs

   Then tried to enable KVM:
   ./configure –enable-kvm –target-list=aarch64-softmmu
   make
   ./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt,accel=kvm -nographic -smp 2 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image –append “console=ttyAMA0″ -fsdev local,id=r,path=/home/ori/lsrc/qemu/rootfs/trusty-core,security_model=none
   Result:
   KVM not supported for this target
   No accelerator found!

   What is the reason?

   Thanks!

   Ori

   Reply to this comment
   • Alex
     June 25, 2014 at 9:31 am

     KVM is only for running on aarch64 hardware where QEMU doesn’t do any translation but just controls the virtual machine and deals with hardware emulation. In that case you’ll want to change your command line include “-enable-kvm -cpu host”.

     As for running SMP on a TCG emulated system I can look into it but there’s not currently much point as internally QEMU will schedule one emulated core after another in a single thread.

     Reply to this comment
6. Ori Chalak
   June 25, 2014 at 3:13 pm

   It seems as the problem is on my host, which does not support KVM.
   I will investigate and if find a solution will post on this thread so that others can benefit.

   Reply to this comment
7. Brian Rzycki
   July 21, 2014 at 11:36 pm

   Hi Alex,
   Thank you for this write-up, it’s helped me get quite far using aarch64 with qemu. However, this is one area where I’m still stuck. I’m unable to get any kind of networking working in the Linux guest OS. the problem I have is all network devices I try cannot find a PCI bus:
   $ $HOME/git/qemu/aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" -netdev type=user,id=net0 -device virtio-net-pci,netdev=net0
qemu-system-aarch64: -device virtio-net-pci,netdev=net0: No 'PCI' bus found for device 'virtio-net-pci'
   Is there something special I need to do to get the networking subsystem to see the qemu internal PCI bus for aarch64?

   Thanks again!

   Reply to this comment
   • Alex
     July 22, 2014 at 9:41 am

     The thing to remember about ARM systems is they are typically System on a Chip (SOC) setups and don’t always have the standard bus architectures of the PC space. In our case the virt modeled system doesn’t need a PCI bus as it would just be an additional abstraction. However a PCI device is discoverable which is useful for the kernel to find out what hardware it has at runtime, in the case of ARM SOCs the location of the virt device are passed to the guest kernel via a device tree blob.

     You can describe your network hardware with the following:

       -netdev user,id=unet -device virtio-net-device,netdev=unet
     

     Reply to this comment
     • Brian Rzycki
       July 22, 2014 at 4:27 pm

       Alex, thank you! I missed that entry when I ran:
       
$ ./aarch64-softmmu/qemu-system-aarch64 -machine type=virt -device \?


       I also find it a bit odd that “virtio-net” is aliased to “virtio-net-pci” and not “virtio-net-device”. As of now my networking is working perfectly for apt-get updates in a virtualized environment. 

       Reply to this comment
       • Emma Zhang
         July 30, 2014 at 3:21 pm

         Hi Brian,

         Thanks for your write-up. I followed your steps, while my networking is not work. Within QEMU, there’s only a local network at first. I use DHCP to get an IP address: /sbin/udhcpc eth0. While I can neither reach my host nor connect out from QEMU.

         Is there something special I need to do? Do I need to create a tap?

         Thanks & Best Regards,

         Emma

         Reply to this comment
         • Nagesh BL
           February 2, 2015 at 7:50 pm

           Hi Emma,

           I’m facing the same issue. Were you able to get a working network interface in the end? Thanks for your help.

           Thanks,
           nagesh

       • pradeepkj
         August 28, 2014 at 2:16 pm

         Hi Brian,

         I am very new to this, I have couple of issues please help me out to fix the things.
         I have some issue with the network configuration, I am using the same steps mentioned above still the interfaces on virtualized environments are not visible. Just the “lo” loopback is visible.
         And also I built a root file system but it does not contain many commands and entries can you please help me out in this also what I need to set etc.

         Regards,
         Pradeep

         Reply to this comment
         • Nagesh BL
           February 2, 2015 at 7:51 pm

           Hi Pradeep,

           Were you able to get a working network interface in the end? How did you do it? Thank you for your help.

           Thanks,
           nagesh

     • Nagesh BL
       February 2, 2015 at 7:47 pm

       Hi Alex,

       Thank you for the writeup. I followed the above instructions and can I see only the loopback interface. What more should be done to get a working ethernet (or any other network) device?

       Thanks,
       nagesh

       Reply to this comment
       • Alex
         February 2, 2015 at 10:54 pm

         Are you running with the virtio-net lines? What does ifconfig -a show?

         Reply to this comment
         • Nagesh BL
           February 3, 2015 at 2:28 am

           Hi Alex,

           Yes, I’ve appended “-netdev user,id=unet -device virtio-net-device,netdev=unet” to my command line.

           ifconfig -a shows:

           lo Link encap:Local Loopback
           LOOPBACK MTU:65536 Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

           Thanks,
           nagesh

         • Alex
           February 3, 2015 at 8:16 pm

           Can you run “grep VIRT .config” on your kernel config for me please? Specifically do you have CONFIG_VIRTIO_NET set?

         • Nagesh BL
           February 18, 2015 at 7:53 pm

           Hi Alex,

           My apologies for not responding sooner. I am able to get a working loopback interface now. The problem was the networking service was not being started on the guest OS boot up.

           Thanks,
           nagesh

8. Emma Zhang
   July 30, 2014 at 3:34 pm

   Hi Alex,

   Thanks for writing this. It helps me a lot.

   I tried to copy a program from host to guest by mounting the local file-system. And the program is lost after I reboot the guest. I don’t know why. Is there a way to keep the things which is copied to the guest?

   I find out that there’s no compile tools in the linux kernel downloaded from https://git.kernel.org. I tried to install JDK and failed. It seems like missing bash instructions.

   Thanks & Best Regards,
   Emma

   Reply to this comment
   • Alex
     July 31, 2014 at 11:14 am

     Yes, you need to have a more permanent file-system. The contents of the initrd RAM disk are not persistent and will get reset to what was bundled in the kernel on each boot. Either run your programs from the 9p mounted file-system or mount a virtio block device for your persistant data.

     As to compilers I’m not sure why you mention the JDK. However Debian/Ubuntu does package a number of cross compilers. Failing that you can download the latest GCC from the Linaro website and add it to your path.

     Reply to this comment
9. cool_guy
   August 20, 2014 at 8:05 pm

   Hey, I want to transfer an executable file from my local host amd64 debian machine to this virtual machine . Can you tell me how to do that ?

   Reply to this comment
10. wannabe_aarch64、_dev
    January 13, 2015 at 10:53 am

    Hey Alex!
    Wonderful post, and very helpful too!
    One thing though.. how does one enable shared folders between Host and Target system?
    i’ve googled around, but i can’t seem to make sense of all the -fsdev and -device options.

    Thanks, in advance!

    Reply to this comment
    • Alex
      January 14, 2015 at 9:54 pm

      That’s what the 9p mounts allow you to do. Did it not work for you?

      Reply to this comment
11. Tino
    February 11, 2015 at 5:28 pm

    Alex,

    This is awesome. Thank you!

    Reply to this comment
    • Fe15 at 12:37 am

      Also: Any idea how to get this aarch64 Qemu online? I’ve been following the steps on other sites with the tap0 and eth0 method, but no luck. Getting the following warnings:

      “Warning: hub port hub0port0 has no peer
      Warning: vlan 0 with no nics
      Warning: netdev hub0port0 has no peer
      Warning: requested NIC (anonymous, model unspecified) was not created (not supported by this machine?)”

      Qemu then proceeds to launch and run just fine, I just can’t get it online.
      Thanks in advance.

      Reply to this comment

转载：

http://www.bennee.com/~alex/blog/2014/05/09/running-linux-in-qemus-aarch64-system-emulation-mode/