Ubuntu ContainerArchitecture

 

On the Ubuntu Touch images we run the Android HAL (Hardware Abstraction Layer) in a container to make use of the binary drivers and some daemons that are needed to drive the builtin hardware of a phone.

Booting by default into the Ubuntu rootfs

To run Android in a container Ubuntu needs to become the default system we are booting to. This is achieved by using a generic Ubuntu initramfs in the phones boot.img instead of the Android supplied one.

The initramfs script that manages the booting lives in the initramfs-tools-ubuntu-touch package. To get a binary initrd.img the ubuntu-touch-generic-initrd package exists, this will always contain the latest binary initrd.img based n the last initramfs-tools-ubuntu-touch package in the Ubuntu archive.

People that are porting their devices to the Ubuntu Touch container model should take into account that Ubuntu's Upstart is used as init system. Upstart likes to have an actually existing device as /dev/console. Many Android kernels do not provide this as default or even have something like console=none or console=ram hardcoded in their boot commandline arguments. Such drawbacks in the kernel or commandline need to be fixed to be able to use this model properly (here is for example a kernel patch used on the i9100 image to change the behavior of CONFIG_CMDLINE_EXTEND ... now using this option appends the kernel cmdline to the bootloader one instead of pre-pending, this way console=tty1 can be handed over as the last cmdline argument and withCONFIG_VT_CONSOLE and CONFIG_HW_CONSOLE set in the kernel config upstart is now happy)

System Preparation for using the LXC container

Since the Android container is supposed to initialize all hardware (beyond the disk and minimal bits required for booting) it needs to come up as early as possible in the boot process and since android ships its own device handling daemon in form of ueventd, we need to make sure that udev does not intefere with the devices on boot. For this purpose the lxc-android-config package (which is carrying all configuration, overrides, jobs and scripts for running the android container) ships an upstart override job for udev that delays its startup until the container is fully done tinkering with devices. Various udev rules like firmware rules, v4l or alsa rules are overridden as well.

To have similar permissions on the Ubuntu side as we do at the Android side, the lxc-android-config package ships a 70-$devicename.rules file for phones it knows and puts this file in place for udev usage on first boot.

On first boot there is also a script run that creates the proper Android mountpoints and fstab entries in the Ubuntu filesystem, you will find /data, /system, /vendor and /factory (or /efs, depending on the device) mountpoints in the Ubuntu root filesystem.

All first boot handling is done by the lxc-android-boot upstart job.

Booting the Android LXC container

Android does not, like other Linux, jump out of its initrd into a root filesystem on disk. Instead the initrd is actually the rootfs. The Android build that gets created from phablet.ubuntu.com takes care that the android initrd is shipped in /system/boot/android-ramdisk.img from where it gets copied to /boot at install time and on upgrades.

Once the system is ready to fire up a container, upstart sends an event to the lxc-android-config job which uses lxc-start to initialize the preconfigured "android" lxc container config.

The above mentioned "android" config ships in /var/lib/lxc/android with the lxc-android-config package. After the config file has been parsed /var/lib/lxc/android/pre-start.sh is executed. This script unpacks the android-ramdisk.img as the contrainer rootfs and parses its pre-start.d subdir which contains code snippets for temporary or permanent modification of the rootfs. Whole files of the rootfs can be replaced by putting a copy into /var/lib/lxc/android/overrides/ (very useful to work with a modified android init.rc file)

Now that the container is populated with a rootfs lxc-start executes /init inside it and returns to the lxc-android-config upstart job. The lxc-android-config job now emits the "android" event to upstart.

Based on this event delayed services like udev can now get started and apps can start talking to the container through libhybris. Logs inside the container can be read via /system/bin/logcat which (like all other binaries in this path) can just be executed from Ubuntu. The Android properties system is also fully accessible from Ubuntu and getprop/setprop are shipped in the image as management tools and persistent properties can be added to /data/properties if needed.

For further communication between the container and the host the /dev/socket directory from Android is shared across the two systems. Here ofono talks to rild (the vendor shipped binary daemon that manages mobile modems) for example.

Using a 64bit Android Container

With Android 5.0/Lollipop 64bit BSPs (Board Support Packages) for Android became available. Ubuntu will host these in a container similarly to 32 bit builds.

Common Requirements

A 64bit Android BSP will have a 64bit kernel. Therefore the Ubuntu kernel which hosts the android patches from such a BSP will also be 64bit.

In order to host the Android BSP user-side components, we will need a 64bit lxc container to host the Android parts.

Other system components interface to this container as before, eg via sockets or libHybris.

TODO: Add support in libHybris for 64bit libraries

Android documentation encourages dual 32/64bit builds for native libraries, but leaves any final decision to the implementor, so long as their system remains compatible with existing Android applications. Ubuntu does not have this constraint, so can potentially use a 64 bit BSP two different ways: 64bit only, or a 32/64 multiarch approach.

64bit

If all the services we depend on are made available in 64bit form, we have the option of building a 64bit touch image analogous to the 64bit desktop images. In this case all the user side packages are selected from the arm64 archive.

multi-arch 32/64bit

In practice, android BSPs must support the existing 32bit ecosystem of android applications, and so contain the equivalent of multiarch support. They ship a suite of 32 and 64 bit libraries. On a case-by-case basis, some of the libraries may be better tested or supported in either the 32bit or 64bit form.

If we assume that some key systems (RILD?) will be maintained for some time in 32bit form, then we may discover that building a pure 64bit Ubuntu image is not possible, or will be lower quality than a 32bit equivalent.

It seems likely that 64bit is being introduced aggressively to provide a marketing feature for handsets, and the ecosystem of software on a handset will transition more slowly.

As such, it may be advantageous to stay with our existing 32bit ubuntu images, and boot these on a multi-arch enabled 64bit kernel. Such a system would need a libybris that was aware of the fact it was being used in a multi-arch system, and would need to support both 32bit and 64bit libaries. Note that the linking would only ever be to the 'same' size, so 32bit Ubuntu to 32bit Android or 64bit Ubuntu to 64bit Android.

mixed-containers, 32bit Ubuntu, 64bit Android

Considering the discussion above about multi-arch systems, we've also got a third option, which is to boot just the container hosting android in 64 bit (which assumes a 64bit kernel), and then initialise the ubuntu user space as today, in 32 bit.

One key advantage of this arrangement is that it is contained within the device tarball, and therefore can re-use our existing rootfs images.

This diagram documents the arrangement:

 

https://wiki.ubuntu.com/Touch/ContainerArchitecture

Android Boot Time Optimization – Android boot process

Android is one of the mostly widely used operating systems in the world. It is a feature rich system that can be used in a wide range of application areas like CE devices, industrial HMI's, display interface for medical devices etc. Generally the boot time of the Android – from power up to showing the first screen is a time consuming process. In this series of posts, we will discuss about some of the tools and techniques for a fast Android boot.

This post is split in three parts:

1.     Android boot process

2.     Android boot time analysis

3.     Android boot optimization techniques

Now in this post, we will explain about the android boot process and its internals.

Android Boot Process

Understanding of the boot process of the target platform is the starting point for the optimizing the boot time. Generally on any platform, following components are loaded and executed step by step

·         Boot loader

·         U-boot (optional)

·         Kernel

·         Android

The Android process has the following sequence

·         Init

·         Zygote

·         System Server

·         Service Manager

·         Other Daemons and processes

·         Applications

The following diagram depicts the boot process.

Boot loader

Up on power up, the processor boots from a ROM area typically located internally. This code determines the boot media and loads the boot loader from the media. The boot loader can be used to initialize the DRAM and load another level of loader or directly the Linux kernel. IT is generally dependent on the processor architecture and implementation.

U-Boot

U-Boot is used as a first or second level boot loader. It reads the Linux and ramdisk images from boot media and validates them. While it may not be mandatory to use u-boot, it offers some flexibility like passing arguments to the kernel (easily used to enter the recovery mode), fast boot modes, as a backup option to reprogram the OS etc.

Kernel

Linux kernel is the heart of the Android responsible for the process creation, inter process communication, device drivers, file system management etc. Android applies a custom patch on the main stream kernel to support certain features like Wake locks etc needed for operation of the Android.

The kernel can be either loaded as uncompressed image or as a compressed image. Up on loading, it mounts the root file system (typically passed as kernel command line arguments) and starts the first application in user space.

Android

Android typically operates wholly on the user space. The android applications are executed over a Virtual Machine called the Dalvik. The following section explains the internals in detail.

init and init.rc

The first user space application executed on booting the kernel is the init executable located in the root folder. The process parses a start up script called the "init.rc" script. This is written in a language designed for android used to start all the necessary processes, daemons and services for a proper operation of android. It offers various types of execution timings such as early-init, on-boot, on-post-fs etc. A detailed explanation of the scripting model is available on Android documentation site.

Demons and Services

The init process creates various daemons and processes like rild, vold, mediaserver, adb, etc each responsible for its own functionality. Descriptions of these processes are not in the scope of this post. Rather we will discuss more about "Zygote" process.

Service Manager

The service manager process manages all the services running in the system. Every service created registers itself with this process and this information is used for future references by other processes/applications.

Zygote

Zygote is one of the first init process created on boot. The term "zygote" is based the biological "initial cell formed that divides to produce offsprings". Similarly "zygote in android" initializes the Dalivik VM and forks to create multiple instances to support each android process. It facilitates using a shared code across the VM instances resulting in a low memory foot print and short load time, ideal for an embedded system.

Zygote apart from installing a listener on the server socket, also preloads classes and resources to be used later in the Android applications. Once done, the system server is started.

Android System services (SystemServer)

SystemServer process starts all services available in the Android. Some of them are described below.

Service	Description
Activity Manager	Manages activities life cycle and new services
Package Manager	Manages application package handling (install, uninstall, upgrade, permissions)
Window Manager	Manages all the window manipulations (like input events, orientation).
AppWidget Service	Handles Android widgets
Backup Manager	Manages backup scheduling and transfer
Status Bar	Shows software/hardware status. It works with other managers like Notification, Network Status, Battery Status
Power Manager	Handles power management while Android's different modes (lock mode, sleep mode, Adjust brightness)
NetworkManagement Service	Deals with network related activities
Notification Manager	Manage all notifications (Toasts)
Location Manager	Manages location providers
Entropy Mixer	Handles (load & save periodically) kernel randomness
Display Manager	Manages display properties
Telephony Registry	Provides telephony information
Scheduling Policy	Manages the process scheduling
Account Manager	Handles the users account credential of different online services
Content Manager	Handles all the data's on a device
Battery Service	Manages battery level and charging states
Alarm Manager	Used to schedule the user applications to be run at future.
Input Manager	Handles input devices and key layouts
Device Policy	Enforces security policies for the device
Clipboard Service	Provides Clipboard based copy/past operations.
NetworkStats Service	Monitors Network connection Status
NetworkPolicy Service	Enforces network security policies.
Wi-Fi P2pService	Handles WiFi peer to peer connection
Ethernet Service	Manages Ethernet connectivity.
Wi-Fi Service	Manage WiFi connectivity
Connectivity Service	Monitors and handles network connection state changes
Network Service Discovery Service	Used to find local network devices to share app data

Once all the services are started and are executing, the Android broadcasts a "ACTION_BOOT_COMPLETED" message implying end of the boot process.

Android Home Screen

The Android package Manager on start up, parses each package (".apk" file) available in the "/system/app" and "/system/vendor/app" and validates its AndroidManifest.xml. The application that is configured as the "Home" in its manifest is launched there by showing the android UI. Typically the Launcher application is launched as it is the default home application.

Generally this whole process may take around 25 to 60 seconds depending up on various factors to be discussed in later posts. Now with the boot process understood, we will look in to the tools available for profiling the boot activity in the next post – Android Boot Time Analysis.

 

 

http://embien.com/blog/android-boot-process-and-optimization/

From PowerOn to Android – The Boot Sequence

 

One of the main issues when dealing when embedded systems -specially as an amateur developer- is understanding what happens since our board/tablet/phone/etc (we will refer to is as a device from now on) is powered on, until the OS takes control of the device and we are able to use it. Understanding this chain of events is crucial in order to develop low-level routines for any given embedded platform, and equally necessary for efficiently design high-level Apps.

Before getting into the boot sequence itself, we are going to say something about how the code is executed before reaching the root files system, this is, how the low partition table is organize, and how it is different from "normal partitions" recognized by the OS (and any other disk utility such as the one present in Ubuntu or Mac OS).

Low Partition Table (raw) and Normal Partition Table

One of the main issues -at least it was for me- is how to understand how code starts being executed once we press ON in our board/tablet/phone/etc. How does the CPU know which code to execute first? Where is that code? How is it possible that this code is found without even counting on a partition table as such, and even without an MBR? This, I will try to explain here.

Normally, in a boot device, the block 0 contains the Master Boot Record (MBR). This special sector (boot sector) contains information about how the device is partitioned in a structure commonly referred as thepartition table. In this way, anytime the SO needs anything knows where to start looking for it. However, when starting up the system for the first time we do it, with a blank device without MBR or partitions, even then, there are some things we can do.

Once the a device is powered on it starts code from a know location (ROM) and looks for the first stage bootloader in a specific block. After doing some initializations, this first bootloader points to a second stage bootloader, which is placed in another well-known location. This process of "pointing" is sometimes referred as Low Partition Table or Raw Partition. This is, there is no partition as such, but the code knows where to continue, since it is hard-coded where it has to look for things. If the ROM expects to find a bootloader in sector 1 and it is not, it stops its operation, otherwise it executes the code present there and continues. Normally in embedded systems (such as Android) the images of the RAMDisk and the Kernel are located in known positions outside the partition table, so that they can be loaded into the DRAM when the system boots up. These images are generated when compiling the kernel and Android itself.

The main advantage of this is that only with a bootloader, in our booting device -which we can copy ourselves there simply by using dd if dealing with a USB stick or an SD card- we can access a basic terminal with very useful commands such as fdisk or fastboot. (We will discuss how to deal with SD Cards (it is exactly the same for USB sticks) when using them as boot devices). By using these commands we can create the MBR and the partition table (fdisk) and upload the images of the kernel, the ramdisk or the system (fastboot) in a more friendly and faster way than "manually" copying them using dd.

Here we have an example of the hardkernel's ODROID-A4 partitions, both raw and normal (www.hardkernel.com):

Power On – ROM

When a system is first booted, the processor -as explained before- executes code place in a well-known location. In embedded systems, this location usually is an internal ROM that initializes some of the components in the board and finds a device to boot up from (in some platforms this small piece of code is referred as BL0)- it varies from board to board and from CPU to CPU also. In the case of a PC, this task is carried out by the BIOS, which is way more flexible in terms of choosing a device to boot up from, component configuration, etc. In either case, the result is the same and the system attempts to boot up.

If a device is eligible for booting up – this means, a first stage bootloader is found (referred as BL1, continuing with the convention started above)- the BL1 is copied intro an internal RAM, being responsible for setting up clocks, SDRAM and loading the remaining boot loader (BL2) into it. Once again this process varies from board to board, but we can assume that they all do something very similar, being the ultimate goal the same for all of them – placing the second stage bootloader (BL2) in DRAM and running it.

Here we have an example of the hardkernel's ODROID-A4 Bootloader Loading Process (www.hardkernel.com):

The Kernel

The bootloader (now referring to it as a whole, BL0 + BL1 + BL2) once it has initialized all the components in the board, loads into RAM -from a well-know location- a more complex software,  known as the kernel. The kernel, which has been specifically cross-compiled for our board (CPU, Memory and Components), is responsible for managing the system's resources, being the "bridge" between the hardware and the software. When the kernel is decompressed, it loads and start several processes, creating a low-level layer for supporting the root file system (Android in our case). This task is carried out by what is known as the Init Sequence (init.rc).

One remarkable thing here is that since the kernel know the hardware, and has to communicate with it, not only does it need to be cross-compile depending on the platform itself, but it is specific for each board. This implies and added complexity, given that not every kernel will be able to be loaded in every board – as it tends to happen with PCs, where there is a set of standard drivers that are compatible with most hardware, from CPUs to Graphic controllers. For example, if we want to change the kernel in a Samsung Galaxy III, we need to know that it comes with a Exynos 4412 , which is an ARM Quad Cortex-9 processor. With this knowledge we know the capabilities of our processor and the target platform to cross-compile, however, we lack details concerning the board as such, this means the schematics of the Samsung Galaxy III. Since our intention is merely development (not hacking), we can access what is called development boards -such as the ones provided by hardkernel (ODROID), which are very close to Samsung's and open source. Counting on information concerning the used components (FLASH and RAM chips, Etherner controller, etc), it is possible to create configuration files for a kernel to match a specific target platform. Unfortunately this changes in the kernel are very specific on the kernel version, being normally the case that we are tied to a very few versions of the kernel when dealing with a given board. It is common all these changes made in the kernel (version x) are group under a patch, which can be apply to an official kernel (version x) so that it is compatible with our board. (A future post will comment on how to patch a kernel).

Dalvik VM and Zygote

Normally, one of the last steps of the init sequence is to load the root file system and the OS, which is in itself the very last step before us -or our code- taking over control of the board/device/machine. Nonetheless, Android does not run directly on top of the kernel since it has been coded in Java and therefore needs to be run on top of a Virtual Machine – Java being marketed as "write once, run anywhere". 

The VM Android runs on top of is called Dalvik (Google decided to abandon both JME and JVM), in a move from traditionally favored stack-based VM architectures to register-based ones. One of the main reasons for this favoritism was mostly due to simplicity of VM implementation, ease of writing a compiler back-end (most VMs are originally designed to host a single language and code density (i.e., executables for stack architectures are invariably smaller than executables for register architectures) [1].

Since every application counts on its own instance of the VM to run in, VM instances are required to start quickly when a new application is launched. Also the memory foot print of the VM is required to be minimal. In order to achieve this, Android introduced a concept called Zygote, in order to enable both sharing of code across VM instances and to provide fast startup time of new VM instances. The Zygote process initializes one Dalvik VM, which preloads and preinitializes code library classes. Once the Zygote has initialized, it will sit and wait for socket requests coming from the runtime process indicating that it should fork newVM instances based on the ZygoteVM instance. [2]

As Zygotes has to be run before Android core processes can be executed, it is launched by the kernel, as part of the init sequence, already in the user space.

Root File System – ANDROID

Finally, Android code starts being executed: volume demons, libraries, the activity manager, core applications, etc., leading to the launching of the graphical interface.

Further explanation concerning processes running on the Dalvik VM and Android Apps. is beyond the scope of this tutorial.

Finally, and as a visual aid for understanding what we have discussed here, we have a simplified flow diagram comparison of the different boot sequences of a device running Linux (left), a device running Android (middle) and a PC running Linux.

REFERENCES

[1] Register vs. stack based VMs. Derek Jones. September, 2009. (here)

[2] The Dalvik Virtual Machine Architecture, David Ehringer. March, 2010. (here)

[3] Android Zygote Startup. (here)

[4] Hardkernel – ODROID (here)

[5] Samsung Exynos 4 (here)

[6] The Linux Kernel Archives (here)

[7] ARM – Cortex-A9 Processor (here)

I hope the post has been helpful and clarifying :)

 

http://javigon.com/2012/08/24/from-poweron-to-android-the-boot-sequence/

 

Android booting sequence Explained

Android booting sequence Explained

Today we will be looking into the Android boot process. When the user press power button then the booting sequence starts.

 

Diagram to show Android Booting Sequence

 

First thing that is loaded while booting Android is BOOTLOADER . CPU has a hard coded address from where the first instruction is fetched. This address generally points to the chip which has BOOTLOADER programmed on it. The main function of bootloader is to :

1.      Initialize RAM

2.      Put basic HW in quiescent state

3.      Load kernel and RAM disk

4.      Jump to kernel

Now kernel has been loaded into the RAM the control passes to the Kernel. The initial kernel startup is very hardware dependent, but its purpose is to set things up so that the CPU can start executing C code as early as possible. Once that's done, the kernel jumps to the architecture-independent start_kernel() function, initializes its
various subsystems, and proceeds to call the "init" functions of all built-in drivers. The majority of messages printed out by the kernel at startup come from these steps. The kernel then mounts its root filesystem and starts the init process.

That's when Android's init kicks in and executes the instructions stored in its /init.rc file to set up environment variables such as the system path, create mount points, mount filesystems, set OOM adjustments, and start native daemons.

These native daemons are started and they initialize their corresponding module. We will focus on special daemon called Zygote which plays a very crucial role in launching applications.  Its functionality is centralized here in order to unify the components shared by all apps and to shorten their start-up time. The init doesn't actually start the Zygote directly; instead it uses the app_process command to get Zygote started by the Android Runtime. The runtime then starts the first Dalvik VM of the system and tells it to invoke the Zygote's main().

Zygote is active only when a new app needs to be launched. To achieve a speedier app launch, the Zygote starts by preloading all Java classes and resources that an app may potentially need at runtime. This effectively loads those into the system's RAM. The Zygote then listens for connections on its socket (/dev/socket/zygote) for requests to start new apps. When it gets a request to start an app, it forks itself and launches the new app. The beauty of having all apps fork from the Zygote is that it's a "virgin" VM that has all the system classes and resources an app may need preloaded and ready to be used. In other words, new apps don't have to wait until those are loaded to start executing.

All of this works because the Linux kernel implements a copy-on-write (COW) policy for forks. As you may know, forking in Unix involves creating a new process that is an exact copy of the parent process. With COW, Linux doesn't actually copy anything. Instead, it maps the pages of the new process over to those of the parent process and
makes copies only when the new process writes to a page. But in fact the classes and resources loaded are never written to, because they're the default ones and are pretty much immutable within the lifetime of the system. So all processes directly forking from the Zygote are essentially using its own mapped copies. Therefore, regardless of the
number of apps running on the system, only one copy of the system classes and the resources is ever loaded in RAM.

Although the Zygote is designed to listen to connections for requests to fork new apps, there is one "app" that the Zygote actually starts explicitly: the System Server. This is the first app started by the Zygote, and it continues to live on as an entirely separate process from its parent. The System Server then starts initializing each system service it houses and registering it with the previously started Service Manager. One of the services it starts, the Activity Manager, will end its initialization by sending an intent of type Intent.CATEGORY_HOME. This starts the Launcher app, which then displays the home screen familiar to all Android users.

 

http://androidsrc.net/android-booting-sequence-explained/