Running Linux in QEMU’s aarch64 system emulation mode

Running Linux in QEMU’s aarch64 system emulation mode

Since I started working on aarch64 support for QEMU the most frequently asked question I got was “when can I run aarch64 system emulation on QEMU?”. Well wait no more as support for a VIRT-IO based aarch64 board was recently merged into the master branch of QEMU. In this post I’ll talk about building QEMU, a rootfs and a kernel that will allow you to start experimenting with the architecture.

Quick start

Let’s first start with building and running QEMU with some pre-built images.

BUILD DEPENDANCIES

As has been noted in the comments the configure script will automatically enable features as long as the pre-requisite developer libraries are installed on your sytem. With a Debian/Ubuntu system this is easily achieved by running:

sudo apt-get build-dep qemu

Of course if you want to enable a feature (either a bleeding edge or non-standard) that requires additional libraries then you will need to install the appropriate development packages manually. The config.log file is usually a useful first step in working out what headers are being looked for.

BUILDING QEMU
git clone git://git.qemu.org/qemu.git qemu.git
cd qemu.git
./configure --target-list=aarch64-softmmu
make

Assuming the build ran without any problems you should now have an executable ./aarch64-softmmu/qemu-system-aarch64 in your build directory. Grab a pre-built image from here and we’ll check it works. The image is a kernel that has been combined with an initial RAM disk (initrd) with a basic root file-system. I go into more details on how to create this later on.

Be aware the command line is quite long so make sure you copy it all ;-)

wget http://people.linaro.org/~alex.bennee/images/aarch64-linux-3.15rc2-buildroot.img
./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img  --append "console=ttyAMA0"

If all went well you should see the familiar Linux boot sequence and eventually get a login prompt. Login as root (no password) and play in the new sandbox.

... usual kernel boot output ...
Welcome to Buildroot
buildroot login: root
# uname -a
Linux buildroot 3.15.0-rc2ajb-00069-g1aae31c #39 SMP Thu Apr 24 11:48:57 BST 2014 aarch64 GNU/Linux

Once you are done type C-a c to enter QEMU’s monitor mode and then quit to exit.

QEMU 2.0.50 monitor - type 'help' for more information
(qemu) quit

Accessing your local file-system

This is all very well but the test image only has a fairly limited root file-system attached to it. It will be a lot more useful if you could access your host file-system to test other binaries. Thanks to VIRT FS we can achieve this without too much hassle. Use the following extended QEMU command line:

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" -fsdev local,id=r,path=/home/alex/lsrc/qemu/rootfs/trusty-core,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r

This sets up the selected path to be mountable by the guest. In this case I’m using an Ubuntu rootfs which can be downloaded from here. Once the system has booted the following commands on the guest will mount the local file-system:

Welcome to Buildroot
buildroot login: root
# mount -t 9p -o trans=virtio r /mnt
# ls -l /mnt/
total 84
drwxr-xr-x    2 default  default       4096 Apr  2  2014 bin
drwxr-xr-x    2 default  default       4096 Feb 27  2014 boot
drwxr-xr-x    3 default  default       4096 Apr  2  2014 dev
drwxr-xr-x   64 default  default       4096 Apr  3  2014 etc
drwxr-xr-x    2 default  default       4096 Feb 27  2014 home
..

Building your own rootfs

There are many solutions to this (including downloading Linaro engineering builds) but the simplest one I’ve found for rolling your own from scratch is the Buildroot project. It present the familiar kernel menuconfig interface and deals with all the hassle of setting up cross compilers for you.

git clone git://git.buildroot.net/buildroot buildroot.git
cd buildroot.git
make menuconfig

There are lots of configuration options to choose from but the following are what I use:

* Target Options -> Target Architecture(AArch64)
* Toolchain -> Toolchain type (External toolchain)
* Toolchain -> Toolchain (Linaro AArch64 14.02)
* System configuration -> Run a getty (login prompt) after boot (BR2_TARGET_GENERIC_GETTY)
* System configuration -> getty options -> TTY Port (ttyAMA0) (BR2_TARGET_GENERIC_GETTY_PORT)
* Target Packages -> Show packages that are also provided by busybox (BR2_PACKAGE_BUSYBOX_SHOW_OTHERS)
* Filesystem images -> cpio the root filesystem (for use as an initial RAM filesystem) (BR2_TARGET_ROOTFS_CPIO)

The last one will be important for when we build the kernel next. Once you have configured buildroot to your liking it’s time to type make and leave it for a while as you enjoy a nice lunch 

make
.. lots of output ..

Building a kernel

$ sudo apt-get install gcc-aarch64-linux-gnu

And the usual kernel building process, with a few tweaks for cross compiling:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux.git
cd linux.git
ARCH=arm64 make defconfig
ARCH=arm64 make menuconfig  (one cmd)

I’ve put my full config up here but important options to note are:

CONFIG_CROSS_COMPILE="aarch64-linux-gnu-"                                               # needs to match your cross-compiler prefix
CONFIG_INITRAMFS_SOURCE="/home/alex/lsrc/qemu/buildroot.git/output/images/rootfs.cpio"  # points at your buildroot image
CONFIG_NET_9P=y                                                                         # needed for virtfs mount
CONFIG_NET_9P_VIRTIO=y

Finally you build it all with:

ARCH=arm64 make -j 8  
(one cmd)

The -j 8 just specifies how many parallel build threads to use. Generally set it to the number of cores you have on your machine.

Final test

All that remains is to test that the newly built kernel works:

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image  --append "console=ttyAMA0"
... lots more output ...
Welcome to Buildroot
ajbtest login: root
[root@ajbtest ~]# ls -l
total 0
[root@ajbtest ~]# uname -a
Linux ajbtest 3.15.0-rc4ajb-00320-gafcf0a2-dirty #41 SMP Fri May 9 13:05:31 BST 2014 aarch64 GNU/Linux

UPDATED: 27/05/2014
* Added notes about library dependencies
* Cleaned up formatting of shell sections, mention length of command line!
* Fix some spelling errors

Filed under geek   Tagged aarch64arm64linaroqemu   31 Comments   Permalink

31 Comments

  1. Tariq Masood

    Thanks for writing this Alex. Very easy to follow.

  2. Tariq Masood

    Note that VIRTFS and VIRTIO features need to be enabled in the configuration for building QEMU to get the option “-fsdev local,id=r,path=/home/alex/lsrc/qemu/rootfs/trusty-core,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r” to work.

    • Alex

      I did a fresh re-build when writing this so I assume it’s enabled automatically when the pre-reqs are around. I shall check next week when I’m back in the office.

      Thanks for the feedback :-)

  3. Wei Xu

    Thanks for your guide Alex.

    I also did not find the virtio-9p-device firstly.
    In my host, I found missing the “libcap-devel” and “libattr-devel”.
    After I installed the two libraries and re-build, I found that the VIRTFS was enabled automatically:)

    • Alex

      I’ve updated the post to mention library dependencies. I assume your system is RPM based? Is there a similar build-dep type command that can pull in the extra development libraries for those systems?

  4. Raghu

    Alex, Thanks for the article.

    I am using Ubuntu 12.04 LTS. I was able to successfully run Linux on Qemu following the instructions above and using the pre-built image provided by you.
    However, on building the rootfs on my own as instructed above, Qemu is not showing any output and no buildroot login when invoked with the newly built “../linux.git/arch/arm64/boot/Image” on my system.

    Since the command “sudo apt-get install gcc-aarch64-linux-gnu” could not fetch toolchain on ubuntu 12.04, I have used the toolchain present in “buildroot.git/output/host/opt/ext-toolchain/bin” for compiling the kernel. I configured the kernel as instructed above.

    Any pointers on resolving the problem is very much appreciated.

    Thanks & Best Regards,
    Raghu.

    • Alex

      It sounds like the initrd image hasn’t linked correctly. Can you pastebin your boot-up output?

    • Raghu

      Problem Solved. Configuration file provided in the website used instead of changing options in menuconfig. rootfs.cpio path alone is modified in config file.

      However, AFAIK, the major differences between the config files(menuconfig changes ->.config & aarch64-kernel.config) are
      CONFIG_NET_9P=m, CONFIG_NET_9P_VIRTIO=m, CONFIG_9P_FS=m instead of
      CONFIG_NET_9P=y, CONFIG_NET_9P_VIRTIO=y, CONFIG_9P_FS=y.

      Thanks & Best Regards,
      Raghu.

      • Alex

        Ah yes modules add an extra level of confusion as you need to ensure the initramfs has them and they get loaded as required. They make a lot of sense for generic binary distributions that need to boot on ${RANDOMHW} but for personal kernels and well-known requirements for VMs it saves a lot of pain just to build them directly in.

  5. Ori Chalak

    Hi Alex,

    Thanks for the great post. Was easy to follow!

    I tried to run multiple guest cpus using: -smp 2
    Result:
    qemu-system-aarch64: mach-virt: must enable KVM to use multiple CPUs

    Then tried to enable KVM:
    ./configure –enable-kvm –target-list=aarch64-softmmu
    make
    ./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt,accel=kvm -nographic -smp 2 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image –append “console=ttyAMA0″ -fsdev local,id=r,path=/home/ori/lsrc/qemu/rootfs/trusty-core,security_model=none
    Result:
    KVM not supported for this target
    No accelerator found!

    What is the reason?

    Thanks!

    Ori

    • Alex

      KVM is only for running on aarch64 hardware where QEMU doesn’t do any translation but just controls the virtual machine and deals with hardware emulation. In that case you’ll want to change your command line include “-enable-kvm -cpu host”.

      As for running SMP on a TCG emulated system I can look into it but there’s not currently much point as internally QEMU will schedule one emulated core after another in a single thread.

  6. Ori Chalak

    It seems as the problem is on my host, which does not support KVM.
    I will investigate and if find a solution will post on this thread so that others can benefit.

  7. Brian Rzycki

    Hi Alex,
    Thank you for this write-up, it’s helped me get quite far using aarch64 with qemu. However, this is one area where I’m still stuck. I’m unable to get any kind of networking working in the Linux guest OS. the problem I have is all network devices I try cannot find a PCI bus:
    $ $HOME/git/qemu/aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" -netdev type=user,id=net0 -device virtio-net-pci,netdev=net0
    qemu-system-aarch64: -device virtio-net-pci,netdev=net0: No 'PCI' bus found for device 'virtio-net-pci'

    Is there something special I need to do to get the networking subsystem to see the qemu internal PCI bus for aarch64?

    Thanks again!

    • Alex

      The thing to remember about ARM systems is they are typically System on a Chip (SOC) setups and don’t always have the standard bus architectures of the PC space. In our case the virt modeled system doesn’t need a PCI bus as it would just be an additional abstraction. However a PCI device is discoverable which is useful for the kernel to find out what hardware it has at runtime, in the case of ARM SOCs the location of the virt device are passed to the guest kernel via a device tree blob.

      You can describe your network hardware with the following:

        -netdev user,id=unet -device virtio-net-device,netdev=unet
      
      • Brian Rzycki

        Alex, thank you! I missed that entry when I ran:

        $ ./aarch64-softmmu/qemu-system-aarch64 -machine type=virt -device \?

        I also find it a bit odd that “virtio-net” is aliased to “virtio-net-pci” and not “virtio-net-device”. As of now my networking is working perfectly for apt-get updates in a virtualized environment. 

        • Emma Zhang

          Hi Brian,

          Thanks for your write-up. I followed your steps, while my networking is not work. Within QEMU, there’s only a local network at first. I use DHCP to get an IP address: /sbin/udhcpc eth0. While I can neither reach my host nor connect out from QEMU.

          Is there something special I need to do? Do I need to create a tap?

          Thanks & Best Regards,

          Emma

          • Nagesh BL

            Hi Emma,

            I’m facing the same issue. Were you able to get a working network interface in the end? Thanks for your help.

            Thanks,
            nagesh

        • pradeepkj

          Hi Brian,

          I am very new to this, I have couple of issues please help me out to fix the things.
          I have some issue with the network configuration, I am using the same steps mentioned above still the interfaces on virtualized environments are not visible. Just the “lo” loopback is visible.
          And also I built a root file system but it does not contain many commands and entries can you please help me out in this also what I need to set etc.

          Regards,
          Pradeep

          • Nagesh BL

            Hi Pradeep,

            Were you able to get a working network interface in the end? How did you do it? Thank you for your help.

            Thanks,
            nagesh

      • Nagesh BL

        Hi Alex,

        Thank you for the writeup. I followed the above instructions and can I see only the loopback interface. What more should be done to get a working ethernet (or any other network) device?

        Thanks,
        nagesh

        • Alex

          Are you running with the virtio-net lines? What does ifconfig -a show?

          • Nagesh BL

            Hi Alex,

            Yes, I’ve appended “-netdev user,id=unet -device virtio-net-device,netdev=unet” to my command line.

            ifconfig -a shows:

            lo Link encap:Local Loopback
            LOOPBACK MTU:65536 Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

            Thanks,
            nagesh

          • Alex

            Can you run “grep VIRT .config” on your kernel config for me please? Specifically do you have CONFIG_VIRTIO_NET set?

          • Nagesh BL

            Hi Alex,

            My apologies for not responding sooner. I am able to get a working loopback interface now. The problem was the networking service was not being started on the guest OS boot up.

            Thanks,
            nagesh

  8. Emma Zhang

    Hi Alex,

    Thanks for writing this. It helps me a lot.

    I tried to copy a program from host to guest by mounting the local file-system. And the program is lost after I reboot the guest. I don’t know why. Is there a way to keep the things which is copied to the guest?

    I find out that there’s no compile tools in the linux kernel downloaded from https://git.kernel.org. I tried to install JDK and failed. It seems like missing bash instructions.

    Thanks & Best Regards,
    Emma

    • Alex

      Yes, you need to have a more permanent file-system. The contents of the initrd RAM disk are not persistent and will get reset to what was bundled in the kernel on each boot. Either run your programs from the 9p mounted file-system or mount a virtio block device for your persistant data.

      As to compilers I’m not sure why you mention the JDK. However Debian/Ubuntu does package a number of cross compilers. Failing that you can download the latest GCC from the Linaro website and add it to your path.

  9. cool_guy

    Hey, I want to transfer an executable file from my local host amd64 debian machine to this virtual machine . Can you tell me how to do that ?

  10. wannabe_aarch64、_dev

    Hey Alex!
    Wonderful post, and very helpful too!
    One thing though.. how does one enable shared folders between Host and Target system?
    i’ve googled around, but i can’t seem to make sense of all the -fsdev and -device options.

    Thanks, in advance!

  11. Tino

    Alex,

    This is awesome. Thank you!

    • Also: Any idea how to get this aarch64 Qemu online? I’ve been following the steps on other sites with the tap0 and eth0 method, but no luck. Getting the following warnings:

      “Warning: hub port hub0port0 has no peer
      Warning: vlan 0 with no nics
      Warning: netdev hub0port0 has no peer
      Warning: requested NIC (anonymous, model unspecified) was not created (not supported by this machine?)”

      Qemu then proceeds to launch and run just fine, I just can’t get it online.
      Thanks in advance.


转载:
http://www.bennee.com/~alex/blog/2014/05/09/running-linux-in-qemus-aarch64-system-emulation-mode/


评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值