A quick look inside the Android emulator

http://benno.id.au/blog/2007/11/29/android-qemu

 

I've been working on getting OKL4 up and running on the Android simulator. In doing so I've ended up getting quite involved with the simulator code base and thought I would share some of my findings.

At the core the simulator is Qemu version 0.8.2. The Android team has provided a wrapper main() , which does some slightly nicer argument naming and parsing, and ends up calling the traditional Qemu entrypoint. Inside, the main change is that a new platform called goldfish has been added to supplement the existing Integrator and Versatile platforms.

When porting a different OS to a new platform, the first thing you need to do is get some basic device drivers, such as interrupt controller, serial console and timer, up and running. Usually, the way to do this is find the published spec sheet, and go off that. Unfortunately there is no published spec sheet for the goldfish, but we have something infinitely more useful; the actual source code to the simulated device. (The number of times I've found bit errors in device documentation is pretty amazing!).

This post will share some of the details of the simulated platform (as it stands at this point in time!), along with some commentary. I'm only covering the bits that I have needed in bootstrapping OKL4, so keypads, framebuffers, etc will have to wait for another day. (Hopefully, tomorrow).

Physical memory layout

The physical memory layout is about as simple as it gets. RAM starts at address 0, and continues up to size of ram, in one contiguous block.

Cache

The data cache is 16KiB, 4 way set associative, with 32 byte lines. This is pretty standard, although it would be nice to have higher associativity. (Of course, it makes not very much difference in terms of simulation, so one can only guess that this cache layout is going to be similar to some real system-on-chip being used in an actual phone.)

Interrupt controller

The interrupt controller has a 4KiB block of registers residing at 0xff000000 . It consist of 5 32-bit registers.

STATUS at offset 0x0 contains the number of pending interrupt. It is a read-only register.

NUMBER at offset 0x4 contains the lowest pending, enabled interrupt number. It is a read-only register.

DISABLE_ALL at offset 0x8 is a write-only register. Writing any value to it will disable all interrupts.

DISABLE at offset 0xC is a write-only register. Writing an interrupt number to it will disable to specified interrupt.

ENABLE at offset 0x10 is a write-only register. Writing an interrupt number to it will enable to specified interrupt.

This has to be the best interface to an interrupt controller ever. No messy shifting, or updating multiple registers to get the job done. Every function I need to implement in my driver ends up being just a register read or write. Bliss!

Serial

The serial controller has a 4KiB block of registers residing at 0xff002000 . It consist of 5 32-bit registers.

PUT_CHAR at offset 0x0 is a write-only register. Writing a value to it puts a character onto the console.

BYTES_READY at offset 0x4 returns the number of characters waiting to be read from the console. This register is read-only .

CMD at offset 0x8 is a write-only register. Writing a command performs one of four actions.

• CMD_INT_DISABLE (0) disables the console interrupt.
• CMD_INT_ENABLE (1) enables the console interrupt.
• CMD_WRITE_BUFFER (2) copies DATA_LEN bytes from virtual address DATA_PTR to the console.
• CMD_READ_BUFFER (3) copies DATA_LEN bytes from the console to virtual address DATA_PTR . The number of bytes should not exceed that specified by BYTES_READY .

DATA_PTR at offset 0x10 is a write-only register. The value in this register is the virtual address used in read and write buffer commands.

DATA_LEN at offset 0x14 is a write-only register. The value in this register is the number of bytes to copy on the read or write buffer commands.

This is a really nice interface. My one reservation is that it would be really nice if performing a read from PUT_CHAR returned a character if available. (Of course then it should be renamed from PUT_CHAR.) It was an interesting decision to use virtual addresses for the buffers, rather than a physical address. This will be different from most hardware out there.

Timer

The serial controller has a 4KiB block of registers residing at 0xff003000 . It consist of 5 32-bit registers. Time is represented by a flowing 64-bit counter.

TIME_LOW at offset 0x0 return the lowest 32-bit from the 64-bit counter. It also latches the high 32-bits into TIME_HIGH . You must read TIME_LOW , before reading TIME_HIGH to get consistent values. It is a read-only register.

TIME_HIGH at offset 0x4 is a read-only register storing the top 32-bits of the 64-bit counter. It should only be read after reading the TIME_LOW value.

ALARM_LOW at offset 0x8 is a write-only register storing the lowest 32-bits of the next alarm value. When written it takes the top 32-bits for the alarm value from ALARM_HIGH and stores the value in an internal register. To get consistent results the ALARM_HIGH should be stored first when setting an alarm. When the counter value reaches the alarm value and interrupt is triggered.

ALARM_HIGH at offset 0xc is a write-only register storing the top 32-bits of the next alarm value. Writing to this register does not update the internal 64-bit alarm register. This is done on writes to ALARM_LOW

CLEAR_INTERRUPT at offset 0x10 is a write-only register. When written to it will clear an interrupt previously posted by the alarm.

This is a nice simple way to access an OS timer. The only thing missing is a periodic mode so the next alarm value doesn't need to be calculated each time. (Of course, periodic ticks are on the way out, so this isn't very critical.)

Bug fixes

The first bug fix is to actually make the thing compile on my machine. This mostly involved removing what seems to be dead code. SDL is used, but the build system is set up to only use SDL on specific files, and on those files, the correct include would be #include <SDL.h> . It turns out this code is unused, so we can just get rid of it entirely. This isn't really a problem for anyone using Android, just if you want to try and recompile.

--- android-emulator-20071111.orig/qemu/vl.c 2007-11-12 17:58:42.000000000 +1100

+++ android-emulator-20071111/qemu/vl.c   2007-11-29 00:29:35.000000000 +1100

@@ -78,12 +78,6 @@

 extern void  android_emulation_setup( void );

 extern void  android_emulation_teardown( void );

 

-#ifdef CONFIG_SDL

-#ifdef __APPLE__

-#include 

-#endif

-#endif /* CONFIG_SDL */

-

 #ifdef CONFIG_COCOA

 #undef main

 #define main qemu_main

The next bug is a little odd. This was found during a run of L4 test. Basically, the code goes to a lot of trouble to register a real-time alarm, and hook up a signal handler to service this. (This is how timer interrupts end up being injected into the emulated machine.) The problem is that the default sigprogmask seems to have SIGALRM blocked, which means we don't end up getting timer interrupts, or at least not if the emulated code is running in a tight loop. This bug could actually affect people using Android. It is possible that in this case timer interrupts are missed and the system becomes unresponsive. I'm not sure if it is something strange in my setup that makes SIGALRM blocked by default, or if this is a general problem. It should probably be investigated further.

diff -ru android-emulator-20071111.orig/qemu/vl.c android-emulator-20071111/qemu/vl.c

--- android-emulator-20071111.orig/qemu/vl.c 2007-11-12 17:58:42.000000000 +1100

+++ android-emulator-20071111/qemu/vl.c   2007-11-29 00:43:34.000000000 +1100

@@ -1282,6 +1276,7 @@

     {

         struct sigaction act;

         struct itimerval itv;

+        sigset_t nset;

 

         sigfillset(&act.sa_mask);

         act.sa_flags = 0;

@@ -1304,6 +1299,10 @@

         /* we probe the tick duration of the kernel to inform the user if

            the emulated kernel requested a too high timer frequency */

         getitimer(ITIMER_REAL, &itv);

+

+        sigemptyset(&nset);

+        sigaddset(&nset, SIGALRM);

+        sigprocmask(SIG_UNBLOCK, &nset, NULL);

     }

 #endif

 }

The next bugs are nasty. Really nasty. And I didn't really debug them, I kind of guessed and looked at diffs to find them. They deal with the innards of the ARM MMU, and are only really exposed by kernels that make full use of the memory management unit (e.g: domains for fast context switching, super pages, PID relocation). OKL4 does this, which I why I'm hitting these bugs, where as Linux doesn't right now, so it avoids them. The first of these problems was fixed upstream in Qemu 0.9.0, the second was actually found by another engineer at OK Labs , Matt Warton, and has been pushed upstream already.

diff -ru android-emulator-20071111.orig/qemu/target-arm/helper.c android-emulator-20071111/qemu/target-arm/helper.c

--- android-emulator-20071111.orig/qemu/target-arm/helper.c 2007-11-12 17:58:42.000000000 +1100

+++ android-emulator-20071111/qemu/target-arm/helper.c   2007-11-29 00:26:44.000000000 +1100

@@ -247,7 +247,7 @@

 

   switch (ap) {

   case 0:

-      if (access_type != 1)

+      if (access_type == 1)

           return 0;

       switch ((env->cp15.c1_sys >> 8) & 3) {

       case 1:

@@ -428,6 +428,7 @@

         break;

     case 3: /* MMU Domain access control.  */

         env->cp15.c3 = val;

+        tlb_flush(env, 1);

         break;

     case 4: /* Reserved.  */

         goto bad_reg;

New features

Since I was having to modify the emulator anyway, I decided to add a feature to make my live just a little easier. Qemu expects to be loading and a Linux kernel, but not all kernels out there are Linux, and they have different expectations about where they should be loaded and what data if any should be passed to them.

I've extended Qemu, and the android wrapper to support a new -os-type flag, so that you can specify what type of OS is being emulated. By default this is set to Linux, and the normal Linux kernel loading algorithm applies. If it is set to anything else, then it simply loads the specified kernel directly in at the start of memory and doesn't do any string or command line passing.

diff -ru android-emulator-20071111.orig/qemu/android_sdl.c android-emulator-20071111/qemu/android_sdl.c

--- android-emulator-20071111.orig/qemu/android_sdl.c 2007-11-12 17:58:41.000000000 +1100

+++ android-emulator-20071111/qemu/android_sdl.c   2007-11-29 00:59:24.000000000 +1100

@@ -3537,6 +3537,7 @@

 static char *arg_nand0 = 0;

 static char *arg_nand1 = 0;

 static char *arg_sdcard = 0;

+static char *arg_os_type = 0;

 static char *arg_kernel = 0;

 static char *arg_ramdisk = 0;

 static char *arg_tracefile = 0;

@@ -3573,6 +3574,7 @@

     const char *help;   /* description text for this option */

 } argmap[] = {

     { "-system",   &arg_sysdir,    0,              0,                   "<dir>",    "search system, ramdisk and userdata images in <dir>" },

+    { "-os-type",  &arg_os_type,   0,             "linux",              "<os-type>","kernel image is of given OS type. E.g: linux, okl4" },

     { "-kernel",   &arg_kernel,    0,             "kernel-qemu",        "<file>",   "use <file> as the emulated kernel" },

     { "-ramdisk",  &arg_ramdisk,   0,             "ramdisk.img",        "<file>",   "use <file> as the ramdisk image (default is <system>/ramdisk.img)" },

     { "-image",    &arg_nand0,     0,             "system.img",         "<file>",   "use <file> as the system image (default is <system>/system.img)" },

@@ -4270,6 +4272,10 @@

 

     n = 1;

     /* generate arguments for the underlying qemu main() */

+    if(arg_os_type && arg_os_type[0]) {

+        args[n++] = "-os-type";

+        args[n++] = arg_os_type;

+    }

     if(arg_kernel && arg_kernel[0]) {

         args[n++] = "-kernel";

         args[n++] = arg_kernel;

--- android-emulator-20071111.orig/qemu/vl.c 2007-11-12 17:58:42.000000000 +1100

+++ android-emulator-20071111/qemu/vl.c   2007-11-29 00:56:47.000000000 +1100

@@ -199,6 +193,8 @@

 int dcache_store_miss_penalty = 5;

 #endif

 

+char *os_type = "linux";

+

 extern void  dprint( const char* format, ... );

 

 /***********************************************************/

@@ -6005,6 +6006,7 @@

     QEMU_OPTION_smb,

     QEMU_OPTION_redir,

 

+    QEMU_OPTION_os_type,

     QEMU_OPTION_kernel,

     QEMU_OPTION_append,

     QEMU_OPTION_initrd,

@@ -6095,6 +6097,7 @@

     { "redir", HAS_ARG, QEMU_OPTION_redir },

 #endif

 

+    { "os-type", HAS_ARG, QEMU_OPTION_os_type },

     { "kernel", HAS_ARG, QEMU_OPTION_kernel },

     { "append", HAS_ARG, QEMU_OPTION_append },

     { "initrd", HAS_ARG, QEMU_OPTION_initrd },

@@ -6564,6 +6567,9 @@

                 pstrcpy(serial_devices[0], sizeof(serial_devices[0]), "stdio");

                 nographic = 1;

                 break;

+            case QEMU_OPTION_os_type:

+                os_type = optarg;

+                break;

             case QEMU_OPTION_kernel:

                 kernel_filename = optarg;

                 break;

--- android-emulator-20071111.orig/qemu/hw/arm_boot.c 2007-11-12 17:58:41.000000000 +1100

+++ android-emulator-20071111/qemu/hw/arm_boot.c   2007-11-29 00:57:04.000000000 +1100

@@ -64,6 +64,8 @@

     stl_raw(p++, 0);

 }

 

+extern char *os_type;

+

 void arm_load_kernel(int ram_size, const char *kernel_filename,

                      const char *kernel_cmdline, const char *initrd_filename,

                      int board_id)

@@ -71,19 +73,27 @@

     int kernel_size;

     int initrd_size;

     int n;

+    int linux_image = (strcmp(os_type, "linux") == 0);

 

     /* Load the kernel.  */

     if (!kernel_filename) {

         fprintf(stderr, "Kernel image must be specified/n");

         exit(1);

     }

-    kernel_size = load_image(kernel_filename,

-                             phys_ram_base + KERNEL_LOAD_ADDR);

+

+    if (linux_image) {

+      kernel_size = load_image(kernel_filename,

+               phys_ram_base + KERNEL_LOAD_ADDR);

+    } else {

+      kernel_size = load_image(kernel_filename,

+               phys_ram_base);

+    }

     if (kernel_size < 0) {

         fprintf(stderr, "qemu: could not load kernel '%s'/n", kernel_filename);

         exit(1);

     }

-    if (initrd_filename) {

+

+    if (linux_image && initrd_filename) {

         initrd_size = load_image(initrd_filename,

                                  phys_ram_base + INITRD_LOAD_ADDR);

         if (initrd_size < 0) {

@@ -94,12 +104,14 @@

     } else {

         initrd_size = 0;

     }

-    bootloader[1] |= board_id & 0xff;

-    bootloader[2] |= (board_id >> 8) & 0xff;

-    bootloader[5] = KERNEL_ARGS_ADDR;

-    bootloader[6] = KERNEL_LOAD_ADDR;

-    for (n = 0; n < sizeof(bootloader) / 4; n++)

-        stl_raw(phys_ram_base + (n * 4), bootloader[n]);

-    set_kernel_args(ram_size, initrd_size, kernel_cmdline);

+    if (linux_image) {

+        bootloader[1] |= board_id & 0xff;

+        bootloader[2] |= (board_id >> 8) & 0xff;

+        bootloader[5] = KERNEL_ARGS_ADDR;

+        bootloader[6] = KERNEL_LOAD_ADDR;

+        for (n = 0; n < sizeof(bootloader) / 4; n++)

+            stl_raw(phys_ram_base + (n * 4), bootloader[n]);

+        set_kernel_args(ram_size, initrd_size, kernel_cmdline);

+    }

 }