Linux ARM Cross Compile On Intel x86

Linux ARM Cross Compile On Intel x86

For the complete list of my articles on Android devices and software, including analysis of devices and system firmware, lists of external resources and tools, and How-To instructions, check the front page of this wiki under the Android heading.

Most pocket Android devices currently utilise an ARM processor implementation. The ARM microprocessor instruction set and operation codes (op-codes) are not compatible with the typical Intel or AMD based PC workstation that derives from the 80x86 range of Intel Architecture (IA) microprocessors (known as x86 for IA32 32-bit CPUs and x64 or amd64 for IA64 64-bit CPUs). Therefore, in order to compile, assemble and link program code for an ARM target device on an x86 host PC a cross-compiler tool-chain is required that runs on x86 but creates executable code for ARM.

Tool-chain¶

The GNU/Linux kernel is so-called because the Linux kernel depends upon the GNU compiler collection (gcc) and binary utilities (binutils) to build the executable vmlinux and it's supporting kernel modules (.ko files). Without the comprehensive set of tools provided by gcc and binutils the Linux kernel cannot be built (there are various projects aimed at substituting them but all serious kernel developers I know will only use the GNU tools).

When building GNU/Linux on an x86-based host for the same x86-based target the regular x86 gcc and binutils are sufficient. However, when the target is a different architecture as in this case (ARM) a version of the tool-chain capable of emitting ARM-compatible executable code is required.

Many GNU/Linux distributions do not pre-package the full set of multi-arch tools required to successfully cross-build (e.g. the Debian/Ubuntu binutils-multiarch package is missing gas, the GNU assembler). Building the tool-chain can be quite a tortuous and time-consuming experience so the pragmatic solution is to install pre-built packages.

Code Sourcery is a company that is licensed and supported by ARM to maintain the GNU ARM tools. They provide several subscription-based pre-built packages for developers (Sourcery G++) that contain a multitude of additional time-saving tools, but they also provide a free-of-cost pre-packaged IA32 set of tools: Sourcery G++ Lite (Note that at this time Code Sourcery do not provide pre-built IA64 packages).

There are three package options provided by Code Sourcery: a GUI-based self-extracting installer, a compressed archive (known as a tar-ball), and the source-code. The GUI (Graphical User Interface) installer is bundled with a Java engine for running the installer and runs a series of 'wizard' dialogs before installing and configuring the package.

I prefer using the tar-ball installation since I can be sure where tools are and control what environment files are changed. Also, the tar-ball can be installed from a text terminal session whereas the GUI cannot.

Installing Multiple Versions¶

Most often it is required that existing versions remain installed and accessible for building existing projects and for specific devices to ensure API/ABI matches. There are three steps to achieve this:

1. Install new tool-chain version in modified location
2. If necessary, modify the locations of the currently installed version(s)
3. Create symbolic links to the preferred version and a simple shell script to easily switch the default tool-chain version

This requires installation to slightly modified locations so that newer installs do not over-write prior installations.

Choose an installation location¶

The convention from the FSH ( File System Hierarchy standard ) is for user-installed binaries and libraries to be placed under /usr/local/ so they do not over-write or otherwise interfere with the system-installed binaries and libraries installed under /usr/ . I chose to install under /usr/local/ but you could choose another location such as /opt/ . The reason for installing under /usr/local/ is that the environment's executable search path variable PATH usually already contains directories in this hierarchy (/usr/local/bin ). Installation to a system location does require super-user privileges.

Some of the paths in the tool-chain archive are already GCC version-specific. I.e. the 2010q1 archives contains GCC v4.4.1 whereas the 2010.09 archive contains GCC v4.5.1. Other paths such as the documentation, the common executables, and the internal executables are the same in each archive so these need modifying to appear in expected FSH locations.

Previously I provided manual instructions for installing and configuring the installation. After I met a requirement to have multiple versions of the ARM GCC installed in parallel I worked up a couple of shell scripts that do all the work. They create and switch symbolic links from standard locations to the currently active GCC version tools. They are attached to this article. Install them in /usr/local/bin/ so they are always available. Don't forget to make them executable:

sudo wget -O /usr/local/bin/arm-install-parallel http://tjworld.net/raw-attachment/wiki/Android/Linux/ARMCrossCompileOnIntel/arm-install-parallel
sudo wget -O /usr/local/bin/arm-gcc-switch http://tjworld.net/raw-attachment/wiki/Android/Linux/ARMCrossCompileOnIntel/arm-gcc-switch
sudo chmod a+x /usr/local/bin/arm-*

Download the latest version of Sourcery G++ Lite for GNU/Linux (or any other version you require). As of this writing it is the 2010.09 version GCC 4.5.1:

wget http://www.codesourcery.com/sgpp/lite/arm/portal/package7851/public/arm-none-linux-gnueabi/arm-2010.09-50-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2

Use arm-install-parallel:

$ arm-install-parallel arm-2010.09-50-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2 

arm-install-parallel version 1.0
Copyright 2010 TJ <linux@tjworld.net>
Licensed on the terms of the GNU GPL version 3

Extracting files from archive, this may take a while...done.
Configuring GCC version 4.5.1 located in /usr/local/arm-2010.09

Here's the same operation for the 2010q1 ARM GCC release:

$ arm-install-parallel arm-2010q1-202-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2

arm-install-parallel version 1.0
Copyright 2010 TJ <linux@tjworld.net>
Licensed on the terms of the GNU GPL version 3

Extracting files from archive, this may take a while...done.
Configuring GCC version 4.4.1 located in /usr/local/arm-2010q1

Decide which will be the default version: check what versions are installed:

$ arm-gcc-switch 

arm-gcc-switch version 1.0
Copyright 2010 TJ <linux@tjworld.net>
Licensed on the terms of the GNU GPL version 3

Usage: arm-gcc-switch [-f] <version>
 -f	force removal of existing executables in common bin/ directory
 Versions available (default*): 4.4.1 4.5.1

Now configure the preferred default GCC version that will be used when calling arm-none-linux-gnueabi-* tools:

$ arm-gcc-switch 4.5.1

arm-gcc-switch version 1.0
Copyright 2010 TJ <linux@tjworld.net>
Licensed on the terms of the GNU GPL version 3

Default version is not currently configured
Default version now 4.5.1
Done

Check the installation is where we expect and was successful:

which arm-none-linux-gnueabi-gcc

 /usr/local/bin/arm-none-linux-gnueabi-gcc

arm-none-linux-gnueabi-gcc --version

arm-none-linux-gnueabi-gcc (Sourcery G++ Lite 2010.09-50) 4.5.1
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Switching Between GCC Versions¶

As well as manually controlling a project's build settings by specifying the absolute location of the GCC version required, it is possible to leave the project configuration alone and quickly switch the symbolic links before running a configure or make operation.

Now arm-gcc-switch can be used to easily change all the symbolic links to tools in /usr/local/bin/, for example to check the current default version look for the asterisk (*):

$ arm-gcc-switch
arm-gcc-switch version 1.0
Copyright 2010 TJ <linux@tjworld.net>
Licensed on the terms of the GNU GPL version 3

Usage: arm-gcc-switch [-f] <version>
 -f	force removal of existing executables in common bin/ directory

 Versions available (default*): 4.4.1 4.5.1*

To change the version:

$ arm-gcc-switch 4.4.1
arm-gcc-switch version 1.0
Copyright 2010 TJ <linux@tjworld.net>
Licensed on the terms of the GNU GPL version 3

Replacing existing default version 4.5.1
[sudo] password for tj: 
Default version now 4.4.1
Done

Building Linux¶

Now the tool-chain is installed a build test is possible. There are so many sources for, and methods of delivery of, the GNU/Linux kernel - it could be from the main-line kernel repository, a distribution, a device manufacturer or a community special-interest project and it could be fetched as a tar-ball or using git, the distributed version control system used by the Linux kernel developers.

Because I originally wrote this guide to help amateur hackers working on the HTC Vision I'll use that in the example here.

Fetch the Source¶

Get the HTC Vision source code package and install it to a useful location:

mkdir sourcecode
cd sourcecode
wget http://member.america.htc.com/download/RomCode/Source_and_Binaries/vision-2.6.32-g814e0a1.tar.gz
tar -xzf vision-2.6.32-g814e0a1.tar.gz

If you wanted to get the Linux main-line, and have the git package installed (sudo apt-get install git-core on Debian and Ubuntu), you could do:

mkdir sourcecode
cd sourcecode
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

Understanding the Kernel Build Process¶

There are several stages to building the kernel and many options of use to a kernel hacker. The key tool for building the kernel is GNU Make. Make is a tool that takes one or more scripts that describe how to build a piece of software, combines it with configuration settings and the environment it discovers as it runs, and makes creating complex programs such as Linux extremely easy. The key to building Linux is its Makefile. This describes a range of targets. Each target declares rules for creating that target, which may include calling other targets. When using Make the default target, if none is specified on the command line, is "all". I.e. these are equivalent:

make
make all

Power-users Tips¶

Before showing how to cross-compile I want to recommend a couple of options for Make and tips that will significantly improve a hack-build-test cycle.

Concurrent Builds¶

The first one is concurrent builds (-j) - that is, more than one file being built at the same time. If the development PC has more than one CPU or core (dual core is extremely common and quad-cores are widespread too) then it makes sense to reduce the build time by allowing Make to run multiple tasks concurrently. On a dual core CPU a setting of "-j2" would fully utilise the system. On a quad-core a setting of "-j4" might be more appropriate. If you wish to do something else on the PC whilst make is running then leaving one core unused (-j3) by Make will allow the foreground applications to get sufficient processor resources to not feel sluggish.

Out-of-Tree builds¶

The second is out-of-tree builds. Usually, when a compiler converts source code into object code it writes the files it creates into the same directories as the source code file it is processing. For a complex project like Linux with hundreds of sub-directories and thousands of files, this can make quite a mess, especially when these files are being written into working directories of the git version control system since the object files - which are not wanted in the source package - will complicate and confuse several key git commands. The solution is to have all the intermediate and final object code files written to another directory that is out-of-tree - in other words, it is a directory out-side of the git working directory. First, create a directory for the builds. Convention dictates we create a new directory builds/ in the parent directory of the git working directory. In other words we want:

sourcecode/
sourcecode/vision-2.6.32-g814e0a1/
sourcecode/builds/

In the builds directory we want one sub-directory for each project we're building, so to create the sub-directory for the HTC Vision project:

cd sourcecode
mkdir -p builds/vision-2.6.32-g814e0a1

Now, when building the project using Make we pass it the path to this directory with the Make output option "O=". The path is usually specified as relative to the project directory, so something like this:

make O=../builds/vision-2.6.32-g814e0a1 [...other options...]

Short Command Line¶

The third is to create an environment variable containing all the options required to avoid typing or copying them each time, and to make the command-line easier to scan. Here's an example taken from the complete command line required to build the kernel as described later:

export MAKE_VISION='-j2 O=../builds/vision-2.6.32-g814e0a1 CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm EXTRA_AFLAGS=-mfpu=neon'

With this variable set in the environment you can use much shorter and clearer commands, E.g:

make $MAKE_VISION vision_defconfig
make $MAKE_VISION prepare
make $MAKE_VISION all
make $MAKE_VISION modules_prepare
make $MAKE_VISION M=../custom-module

Cross Compiling¶

Because the target architecture (ARM) is different to the host (IA32) Make needs to be instructed to create output suitable for ARM with "ARCH=arm".

In addition Make has to know how to find the cross-compile tool-chain. The convention is that the various tools have names prefixed with the architecture and platform they target. So the compiler that builds for the same target as the host is called "gcc" but the one that targets ARM linux is called "arm-none-linux-gnueabi-gcc" (This is the prefix used by all the tools installed by Sourcery G++ Lite). Because there are many tools, not just "gcc", Make accepts the CROSS_COMPILE option that gives the prefix all the tools use: in this case "CROSS_COMPILE=arm-none-linux-gnueabi-". Because the tools are already in the executable search path (/usr/local/bin/) we do not need to specify the absolute path.

Because the ARM target source-code contains instructions for the NEON (Advanced SIMD Single Instruction Multiple Data) FPU (floating point processor) we must also pass an extra option to the assembler to let it know this using "EXTRA_AFLAGS=-mfpu=neon" otherwise we'll see assembler errors early in the build:

  AS      arch/arm/mach-msm/idle-v7.o
/home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S: Assembler messages:
/home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:47: Error: selected processor does not support ARM mode `fmrx r1,fpexc'
/home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:48: Error: selected processor does not support ARM mode `fmrx r2,fpscr'
/home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:117: Error: selected processor does not support ARM mode `fmxr fpexc,r2'
/home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:125: Error: selected processor does not support ARM mode `fmxr fpscr,r3'
/home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:126: Error: selected processor does not support ARM mode `fmxr fpexc,r2'
make[2]: *** [arch/arm/mach-msm/idle-v7.o] Error 1
make[1]: *** [arch/arm/mach-msm] Error 2
make: *** [sub-make] Error 

So, the cross-compiler specific options are:

make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm EXTRA_AFLAGS=-mfpu=neon [...other options...]

Building¶

These examples rely upon creating the MAKE_VISION environment variable as described above in the section Short Command Line.

There are several steps involved in building the kernel. Start off by changing to the project directory:

cd vision-2.6.32-g814e0a1

Create .config for this target¶

The first and most important step is to provide Make with the kernel configuration file ".config" (Note that file-names that begin with a "." (dot a.k.a period) are usually hidden from directory listings). The kernel source tree contains many default config files for various architectures so we tell Make to use the one for the HTC Vision (which is in arch/arm/configs/vision_defconfig):

make $MAKE_VISION vision_defconfig

This will create the required configuration in ../builds/vision-2.6.32-g814e0a1

ls -A1 ../builds/vision-2.6.32-g814e0a1/
.config
include
Makefile
scripts
source

Prepare¶

Now we need to prepare some scripts and header files based on this configuration:

make $MAKE_VISION prepare

Build the Kernel¶

Now we're ready to build. Let's use the "all" target to build the entire kernel:

make $MAKE_VISION all

Build a Custom Module¶

If you're developing and/or building a custom module for this kernel there are several steps you can take to keep the source-code of the module separate from the kernel source, and to avoid rebuilding the kernel whilst testing and modifying.

Let's create a custom module in a new directory that is a sibling (brother or sister) of the kernel directory:

mkdir ../custom-module

And let's add a module skeleton there sufficient for a successful build. Using your preferred text-editor create the file ../custom-module/skeleton.c with the contents:

/*
 *  skeleton.c
 *
 * Copyright (C) 2010 TJ
 * Author: TJ <linux@tjworld.net>
 *
 * This software is licensed under the terms of the GNU General Public
 * License version 2, as published by the Free Software Foundation, and
 * may be copied, distributed, and modified under those terms.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
*/

#include <linux/module.h>
#include <linux/init.h>

static const char *module_name = "skeleton";

static int __init skeleton_init(void)
{
	printk(KERN_INFO "Module %s loaded\n", module_name);
	return 0;
}

static void __exit skeleton_exit(void)
{
	printk(KERN_INFO "Module %s unloaded\n", module_name);
	return;
}

module_init(skeleton_init);
module_exit(skeleton_exit);

MODULE_AUTHOR("TJ <linux@tjworld.net>");
MODULE_DESCRIPTION("Skeleton example module");
MODULE_LICENSE("GPL");

Create the accompanying ../custom-module/Makefile:

obj-m += skeleton.o

To avoid the compile-time warning:

  WARNING: Symbol version dump /home/all/SourceCode/android/HTC/builds/vision-2.6.32-g814e0a1/Module.symvers
           is missing; modules will have no dependencies and modversions.

Build the entire kernel at least once so the Module.symvers file is created in the out-of-tree build directory:

make $MAKE_VISION modules_prepare

To build the module give Make the "M=" option. This contains the path to the module's directory, in this case "M=../custom-module":

make $MAKE_VISION M=../custom-module

  CC [M]  ../custom-module/skeleton.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      ../custom-module/skeleton.mod.o
  LD [M]  ../custom-module/skeleton.ko

The files will be built in ../builds/custom-module:

ls -1 ../builds/custom-module/
built-in.o
modules.order
Module.symvers
skeleton.ko
skeleton.mod.c
skeleton.mod.o
skeleton.o

Installing on the Device¶

Now your custom loadable kernel module file (skeleton.ko) has been built it can be transferred from the host PC to the target device. Assuming the device is connected and the Android Debug Bridge (adb) is installed then:

adb push ../builds/custom-module/skeleton.ko /sdcard/

To install the module using ADB shell:

adb shell insmod /sdcard/skeleton.ko

If you are doing more sophisticated tasks, such as build and installing a replacement kernel image (boot.img in an RUU package), the file to use is the compressed kernel found at ../builds/vision-2.6.32-g814e0a1/arch/arm/boot/compressed/vmlinux

Clean and Ultra-Clean¶

After making some changes it can be necessary to clean out older build files. There are two Make targets for this depending on how clean you want to be.

clean¶

make $MAKE_VISION clean

This will remove intermediate and object files but leave the .config and related build environment files such as Modules.symvers in place.

mrproper¶

make $MAKE_VISION mrproper

mrproper will do a clean and then remove the files created by prepare such as .config and Modules.symvers.

After mrproper you'll need to install the .config file again (e.g. using the vision_defconfig target) and a prepare to install scripts and headers. Obviously the .config file you install will depend on the architecture and device you are targeting.