Linux ARM Cross Compile On Intel x86

Most pocket Android devices currently utilise an  ARM  processor implementation. The ARM microprocessor instruction set and operation codes (op-codes) are not compatible with the typical Intel or AMD based PC workstation that derives from the 80x86 range of Intel Architecture (IA) microprocessors (known as x86 for IA32 32-bit CPUs and x64 or amd64 for IA64 64-bit CPUs). Therefore, in order to compile, assemble and link program code for an ARM target device on an x86 host PC a cross-compiler tool-chain is required that runs on x86 but creates executable code for ARM.

Tool-chain

The GNU/Linux kernel is so-called because the Linux kernel depends upon the  GNU compiler collection (gcc) and  binary utilities (binutils) to build the executable vmlinuxand it's supporting kernel modules (.ko files). Without the comprehensive set of tools provided by gcc and binutils the  Linux kernel cannot be built (there are various projects aimed at substituting them but all serious kernel developers I know will only use the  GNUtools).

When building GNU/Linux on an x86-based host for the same x86-based target the regular x86 gcc and binutils are sufficient. However, when the target is a different architecture as in this case (ARM) a version of the tool-chain capable of emitting ARM-compatible executable code is required.

Many GNU/Linux distributions do not pre-package the full set of multi-arch tools required to successfully cross-build (e.g. the Debian/Ubuntu binutils-multiarch package is missing gas, the GNU assembler). Building the tool-chain can be  quite a tortuous and time-consuming experience so the pragmatic solution is to install pre-built packages.

 Code Sourcery is a company that is licensed and supported by ARM to maintain the GNU ARM tools. They provide several subscription-based pre-built packages for developers (Sourcery G++) that contain a multitude of additional time-saving tools, but they also provide a free-of-cost pre-packaged IA32 set of tools:  Sourcery G++ Lite (Note that at this time Code Sourcery do notprovide pre-built IA64 packages).

There are three package options provided by Code Sourcery: a GUI-based self-extracting installer, a compressed archive (known as atar-ball), and the source-code. The GUI (Graphical User Interface) installer is bundled with a Java engine for running the installer and runs a series of 'wizard' dialogs before installing and configuring the package.

I prefer using the tar-ball installation since I can be sure where tools are and control what environment files are changed. Also, the tar-ball can be installed from a text terminal session whereas the GUI cannot.

Installing Multiple Versions

Most often it is required that existing versions remain installed and accessible for building existing projects and for specific devices to ensure API/ABI matches. There are three steps to achieve this:

1. Install new tool-chain version in modified location
2. If necessary, modify the locations of the currently installed version(s)
3. Create symbolic links to the preferred version and a simple shell script to easily switch the default tool-chain version

This requires installation to slightly modified locations so that newer installs do not over-write prior installations.

Choose an installation location

The convention from the FSH ( File System Hierarchy standard ) is for user-installed binaries and libraries to be placed under /usr/local/ so they do not over-write or otherwise interfere with the system-installed binaries and libraries installed under /usr/ . I chose to install under /usr/local/ but you could choose another location such as /opt/ . The reason for installing under /usr/local/ is that the environment's executable search path variable PATH usually already contains directories in this hierarchy (/usr/local/bin ). Installation to a system location does require super-user privileges.

Some of the paths in the tool-chain archive are already GCC version-specific. I.e. the 2010q1 archives contains GCC v4.4.1 whereas the 2010.09 archive contains GCC v4.5.1. Other paths such as the documentation, the common executables, and the internal executables are the same in each archive so these need modifying to appear in expected FSH locations.

Previously I provided manual instructions for installing and configuring the installation. After I met a requirement to have multiple versions of the ARM GCC installed in parallel I worked up a couple of shell scripts that do all the work. They create and switch symbolic links from standard locations to the currently active GCC version tools. They are attached to this article. Install them in/usr/local/bin/ so they are always available. Don't forget to make them executable:

sudo wget -O /usr/local/bin/arm-install-parallel http://tjworld.net/raw-attachment/wiki/Android/Linux/ARMCrossCompileOnIntel/arm-install-parallel  sudo wget -O /usr/local/bin/arm-gcc-switch http://tjworld.net/raw-attachment/wiki/Android/Linux/ARMCrossCompileOnIntel/arm-gcc-switch  sudo chmod a+x /usr/local/bin/arm-*  

Download the latest version of  Sourcery G++ Lite for GNU/Linux (or any other version you require). As of this writing it is the 2010.09 version GCC 4.5.1:

wget http://www.codesourcery.com/sgpp/lite/arm/portal/package7851/public/arm-none-linux-gnueabi/arm-2010.09-50-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2  

Use arm-install-parallel:

$ arm-install-parallel arm-2010.09-50-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2     arm-install-parallel version 1.0  Copyright 2010 TJ <linux@tjworld.net>  Licensed on the terms of the GNU GPL version 3    Extracting files from archive, this may take a while...done.  Configuring GCC version 4.5.1 located in /usr/local/arm-2010.09  

Here's the same operation for the 2010q1 ARM GCC release:

$ arm-install-parallel arm-2010q1-202-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2    arm-install-parallel version 1.0  Copyright 2010 TJ <linux@tjworld.net>  Licensed on the terms of the GNU GPL version 3    Extracting files from archive, this may take a while...done.  Configuring GCC version 4.4.1 located in /usr/local/arm-2010q1  

Decide which will be the default version: check what versions are installed:

$ arm-gcc-switch     arm-gcc-switch version 1.0  Copyright 2010 TJ <linux@tjworld.net>  Licensed on the terms of the GNU GPL version 3    Usage: arm-gcc-switch [-f] <version>   -f	force removal of existing executables in common bin/ directory   Versions available (default*): 4.4.1 4.5.1  

Now configure the preferred default GCC version that will be used when calling arm-none-linux-gnueabi-* tools:

$ arm-gcc-switch 4.5.1    arm-gcc-switch version 1.0  Copyright 2010 TJ <linux@tjworld.net>  Licensed on the terms of the GNU GPL version 3    Default version is not currently configured  Default version now 4.5.1  Done  

Check the installation is where we expect and was successful:

which arm-none-linux-gnueabi-gcc     /usr/local/bin/arm-none-linux-gnueabi-gcc    arm-none-linux-gnueabi-gcc --version    arm-none-linux-gnueabi-gcc (Sourcery G++ Lite 2010.09-50) 4.5.1  Copyright (C) 2010 Free Software Foundation, Inc.  This is free software; see the source for copying conditions.  There is NO  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  

Switching Between GCC Versions

As well as manually controlling a project's build settings by specifying the absolute location of the GCC version required, it is possible to leave the project configuration alone and quickly switch the symbolic links before running a configure or make operation.

Now arm-gcc-switch can be used to easily change all the symbolic links to tools in /usr/local/bin/, for example to check the current default version look for the asterisk (*):

$ arm-gcc-switch  arm-gcc-switch version 1.0  Copyright 2010 TJ <linux@tjworld.net>  Licensed on the terms of the GNU GPL version 3    Usage: arm-gcc-switch [-f] <version>   -f	force removal of existing executables in common bin/ directory     Versions available (default*): 4.4.1 4.5.1*  

To change the version:

$ arm-gcc-switch 4.4.1  arm-gcc-switch version 1.0  Copyright 2010 TJ <linux@tjworld.net>  Licensed on the terms of the GNU GPL version 3    Replacing existing default version 4.5.1  [sudo] password for tj:   Default version now 4.4.1  Done  

Building Linux

Now the tool-chain is installed a build test is possible. There are so many sources for, and methods of delivery of, the GNU/Linux kernel - it could be from the main-line  kernel repository, a distribution, a device manufacturer or a community special-interest project and it could be fetched as a tar-ball or using  git, the distributed version control system used by the Linux kernel developers.

Because I originally wrote this guide to help amateur hackers working on the HTC Vision I'll use that in the example here.

Fetch the Source

Get the HTC Vision source code package and install it to a useful location:

mkdir sourcecode  cd sourcecode  wget http://member.america.htc.com/download/RomCode/Source_and_Binaries/vision-2.6.32-g814e0a1.tar.gz  tar -xzf vision-2.6.32-g814e0a1.tar.gz  

If you wanted to get the Linux main-line, and have the git package installed (sudo apt-get install git-core on Debian and Ubuntu), you could do:

mkdir sourcecode  cd sourcecode  git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git    

Understanding the Kernel Build Process

There are several stages to building the kernel and many options of use to a kernel hacker. The key tool for building the kernel is GNU Make. Make is a tool that takes one or more scripts that describe how to build a piece of software, combines it with configuration settings and the environment it discovers as it runs, and makes creating complex programs such as Linux extremely easy. The key to building Linux is its Makefile. This describes a range of targets. Each target declares rules for creating that target, which may include calling other targets. When using Make the default target, if none is specified on the command line, is "all". I.e. these are equivalent:

make  make all  

Power-users Tips

Before showing how to cross-compile I want to recommend a couple of options for Make and tips that will significantly improve a hack-build-test cycle.

Concurrent Builds

The first one is concurrent builds (-j) - that is, more than one file being built at the same time. If the development PC has more than one CPU or core (dual core is extremely common and quad-cores are widespread too) then it makes sense to reduce the build time by allowing Make to run multiple tasks concurrently. On a dual core CPU a setting of "-j2" would fully utilise the system. On a quad-core a setting of "-j4" might be more appropriate. If you wish to do something else on the PC whilst make is running then leaving one core unused (-j3) by Make will allow the foreground applications to get sufficient processor resources to not feel sluggish.

Out-of-Tree builds

The second is out-of-tree builds. Usually, when a compiler converts source code into object code it writes the files it creates into the same directories as the source code file it is processing. For a complex project like Linux with hundreds of sub-directories and thousands of files, this can make quite a mess, especially when these files are being written into working directories of the git version control system since the object files - which are not wanted in the source package - will complicate and confuse several key git commands. The solution is to have all the intermediate and final object code files written to another directory that is out-of-tree - in other words, it is a directory out-side of the git working directory. First, create a directory for the builds. Convention dictates we create a new directory builds/ in the parent directory of the git working directory. In other words we want:

sourcecode/  sourcecode/vision-2.6.32-g814e0a1/  sourcecode/builds/  

In the builds directory we want one sub-directory for each project we're building, so to create the sub-directory for the HTC Vision project:

cd sourcecode  mkdir -p builds/vision-2.6.32-g814e0a1  

Now, when building the project using Make we pass it the path to this directory with the Make output option "O=". The path is usually specified as relative to the project directory, so something like this:

make O=../builds/vision-2.6.32-g814e0a1 [...other options...]  

Short Command Line

The third is to create an environment variable containing all the options required to avoid typing or copying them each time, and to make the command-line easier to scan. Here's an example taken from the complete command line required to build the kernel as described later:

export MAKE_VISION='-j2 O=../builds/vision-2.6.32-g814e0a1 CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm EXTRA_AFLAGS=-mfpu=neon'  

With this variable set in the environment you can use much shorter and clearer commands, E.g:

make $MAKE_VISION vision_defconfig  make $MAKE_VISION prepare  make $MAKE_VISION all  make $MAKE_VISION modules_prepare  make $MAKE_VISION M=../custom-module  

Cross Compiling

Because the target architecture (ARM) is different to the host (IA32) Make needs to be instructed to create output suitable for ARM with "ARCH=arm".

In addition Make has to know how to find the cross-compile tool-chain. The convention is that the various tools have names prefixed with the architecture and platform they target. So the compiler that builds for the same target as the host is called "gcc" but the one that targets ARM linux is called "arm-none-linux-gnueabi-gcc" (This is the prefix used by all the tools installed by Sourcery G++ Lite). Because there are many tools, not just "gcc", Make accepts the CROSS_COMPILE option that gives the prefix all the tools use: in this case "CROSS_COMPILE=arm-none-linux-gnueabi-". Because the tools are already in the executable search path (/usr/local/bin/) we do not need to specify the absolute path.

Because the ARM target source-code contains instructions for the NEON (Advanced SIMD Single Instruction Multiple Data) FPU (floating point processor) we must also pass an extra option to the assembler to let it know this using "EXTRA_AFLAGS=-mfpu=neon" otherwise we'll see assembler errors early in the build:

  AS      arch/arm/mach-msm/idle-v7.o  /home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S: Assembler messages:  /home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:47: Error: selected processor does not support ARM mode `fmrx r1,fpexc'  /home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:48: Error: selected processor does not support ARM mode `fmrx r2,fpscr'  /home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:117: Error: selected processor does not support ARM mode `fmxr fpexc,r2'  /home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:125: Error: selected processor does not support ARM mode `fmxr fpscr,r3'  /home/all/SourceCode/android/HTC/vision-2.6.32-g814e0a1/arch/arm/mach-msm/idle-v7.S:126: Error: selected processor does not support ARM mode `fmxr fpexc,r2'  make[2]: *** [arch/arm/mach-msm/idle-v7.o] Error 1  make[1]: *** [arch/arm/mach-msm] Error 2  make: *** [sub-make] Error   

So, the cross-compiler specific options are:

make CROSS_COMPILE=arm-none-linux-gnueabi- ARCH=arm EXTRA_AFLAGS=-mfpu=neon [...other options...]  

Building

These examples rely upon creating the MAKE_VISION environment variable as described above in the section Short Command Line.

There are several steps involved in building the kernel. Start off by changing to the project directory:

cd vision-2.6.32-g814e0a1  

Create .config for this target

The first and most important step is to provide Make with the kernel configuration file ".config" (Note that file-names that begin with a "." (dot a.k.a period) are usually hidden from directory listings). The kernel source tree contains many default config files for various architectures so we tell Make to use the one for the HTC Vision (which is in arch/arm/configs/vision_defconfig):

make $MAKE_VISION vision_defconfig  

This will create the required configuration in ../builds/vision-2.6.32-g814e0a1

ls -A1 ../builds/vision-2.6.32-g814e0a1/  .config  include  Makefile  scripts  source  

Prepare

Now we need to prepare some scripts and header files based on this configuration:

make $MAKE_VISION prepare  

Build the Kernel

Now we're ready to build. Let's use the "all" target to build the entire kernel:

make $MAKE_VISION all  

Build a Custom Module

If you're developing and/or building a custom module for this kernel there are several steps you can take to keep the source-code of the module separate from the kernel source, and to avoid rebuilding the kernel whilst testing and modifying.

Let's create a custom module in a new directory that is a sibling (brother or sister) of the kernel directory:

mkdir ../custom-module  

And let's add a module skeleton there sufficient for a successful build. Using your preferred text-editor create the file ../custom-module/skeleton.c with the contents:

/*   *  skeleton.c   *   * Copyright (C) 2010 TJ   * Author: TJ <linux@tjworld.net>   *   * This software is licensed under the terms of the GNU General Public   * License version 2, as published by the Free Software Foundation, and   * may be copied, distributed, and modified under those terms.   *   * This program is distributed in the hope that it will be useful,   * but WITHOUT ANY WARRANTY; without even the implied warranty of   * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the   * GNU General Public License for more details.   *  */    #include <linux/module.h>  #include <linux/init.h>    static const char *module_name = "skeleton";    static int __init skeleton_init(void)  {  	printk(KERN_INFO "Module %s loaded\n", module_name);  	return 0;  }    static void __exit skeleton_exit(void)  {  	printk(KERN_INFO "Module %s unloaded\n", module_name);  	return;  }    module_init(skeleton_init);  module_exit(skeleton_exit);    MODULE_AUTHOR("TJ <linux@tjworld.net>");  MODULE_DESCRIPTION("Skeleton example module");  MODULE_LICENSE("GPL");  

Create the accompanying ../custom-module/Makefile:

obj-m += skeleton.o  

To avoid the compile-time warning:

  WARNING: Symbol version dump /home/all/SourceCode/android/HTC/builds/vision-2.6.32-g814e0a1/Module.symvers             is missing; modules will have no dependencies and modversions.  

Build the entire kernel at least once so the Module.symvers file is created in the out-of-tree build directory:

make $MAKE_VISION modules_prepare  

To build the module give Make the "M=" option. This contains the path to the module's directory, in this case "M=../custom-module":

make $MAKE_VISION M=../custom-module      CC [M]  ../custom-module/skeleton.o    Building modules, stage 2.    MODPOST 1 modules    CC      ../custom-module/skeleton.mod.o    LD [M]  ../custom-module/skeleton.ko  

The files will be built in ../builds/custom-module:

ls -1 ../builds/custom-module/  built-in.o  modules.order  Module.symvers  skeleton.ko  skeleton.mod.c  skeleton.mod.o  skeleton.o  

Installing on the Device

Now your custom loadable kernel module file (skeleton.ko) has been built it can be transferred from the host PC to the target device. Assuming the device is connected and the Android Debug Bridge (adb) is installed then:

adb push ../builds/custom-module/skeleton.ko /sdcard/  

To install the module using ADB shell:

adb shell insmod /sdcard/skeleton.ko  

If you are doing more sophisticated tasks, such as build and installing a replacement kernel image (boot.img in an RUU package), the file to use is the compressed kernel found at ../builds/vision-2.6.32-g814e0a1/arch/arm/boot/compressed/vmlinux

Clean and Ultra-Clean

After making some changes it can be necessary to clean out older build files. There are two Make targets for this depending on how clean you want to be.

clean

make $MAKE_VISION clean  

This will remove intermediate and object files but leave the .config and related build environment files such as Modules.symvers in place.

mrproper

make $MAKE_VISION mrproper  

mrproper will do a clean and then remove the files created by prepare such as .config and Modules.symvers.

After mrproper you'll need to install the .config file again (e.g. using the vision_defconfig target) and a prepare to install scripts and headers. Obviously the .config file you install will depend on the architecture and device you are targeting.