|ELJ Issue 1: Building Tiny Linux Systems with Busybox, Part 2: Building the Kernel
Posted on Saturday, January 20, 2001 by Bruce Perens
Let Bruce help you put the BusyBox to work.
For this example I use Linux kernel version 2.2.17. The 2.4.0-test8 kernel that I tried did not size the RAM disk for the root file system properly, leading to a ``not enough memory'' message at boot time. That bug will probably be repaired in the 2.4 series of kernels by the time you read this.
We will build our example to run on an i386-architecture PC-compatible system with PC keyboard and VGA display, booting from a floppy disk and running the root file system entirely in RAM once the system is booted. This example should also boot from IDE disks and from FLASH EEPROM devices that masquerade as IDE disks. It can also be configured to boot from a CD-ROM.
Build a bzImage-style kernel with all of the facilities needed for the application, plus these three:
Don't use kernel modules, because this example system doesn't support them. Don't put any facilities in the kernel that you don't need, as they will use up space that you need on the floppy disk. A kernel with the facilities you need should be around half a megabyte in size and should fit easily on a floppy along with the ROM root file system. A kernel with many unnecessary bells and whistles will be a megabyte or more and won't leave sufficient room for your ROM root file system.
- RAM disk support (in the Block Devices menu)
- Initial RAM disk (initrd) support (also in the Block Devices menu)
- ROM file system support (in the File Systems menu)
If you're not familiar with building and installing kernels on a normal Linux PC, you'll need to study up on that. In short, I placed the kernel sources in /usr/src/linux and ran:
This created a compiled Linux kernel in /usr/src/linux/arch/i386/boot/bzImage.
Building a Static-Linked Busybox
In the busybox source directory, edit the Makefile, changing the variable DOSTATIC from false to true. Then run make. That will create a static-linked version of busybox. Confirm that it is static-linked by running this command:
This should print something like:
It's important to get this right; if you install a dynamic-linked version of Busybox, your system won't run because we aren't installing the runtime dynamic linker and its libraries on the floppy disk for this example.
Creating a ROM Root File System
We're going to go through all of the steps for creating a minimal root file system by hand so that you will understand just how little is necessary to boot your system rather than copying all of the files from the root of your Linux distribution and then being afraid to remove anything because you don't know whether it's necessary. You will need to become root (the superuser) to perform the following steps because the mknod command requires superuser privilege.
Create the tiny-linux directory and change directory into it:
Create the standard directories in it:
Enter the tiny-linux/dev directory:
Create the generic terminal devices:
Create the virtual terminal device for the VGA display:
Create the RAM disk device:
Create the null device, used to discard unwanted output:
Change directory to tiny-linux/etc/init.d, where startup scripts are stored:
Use an editor to create this shell script in tiny-linux/etc/init.d/rcS. It will be executed when the system boots:
Make the script executable:
Change directory to tiny-linux/etc:
Use an editor to create the file tiny-linux/etc/fstab, which says what file systems should be mounted at boot time:
Set the mode of tiny-linx/etc/fstab:
Use an editor to create the file tiny-linux/etc/inittab, which tells /bin/init, the system startup program, what processes to start:
The above example runs the script /etc/init.d/rcS at boot time and runs an interactive shell on the console device.
Set the modes of tiny-linux/etc/inittab:
That's everything necessary to create your root file system, except for the installation of the programs. Change directory to tiny-linux/bin:
Copy your static-linked version of Busybox from wherever you built it into tiny-linux/bin/busybox with a command similar to this one:
Add another command name ls to Busybox using the ln command:
Run ls, and the result should look like this:
Repeat the above ln command for all of these names:
Are you tired yet? Well, now is a good time to take a break--you've finished creating your ROM root file system.
Generate the ROM Root File System Image
You'll need the genromfs program to generate the ROM file system image. If you are using Debian or Red Hat, it's already packaged for you as part of the standard system; you'll just need to run the command to install it. If your Linux distribution doesn't include a prepackaged version, at this writing the program can be found at ftp://ibiblio.org/pub/Linux/system/recovery/genromfs-0.3.tar.gz. Install the program, change directory to the directory that contains tiny-linux and run these commands:
This creates the ROM file system image in the file fs. Now, compress the file system image using the gzip command:
That will create the file fs.gz, which is about half the size of the uncompressed version.
Build the Floppy
We'll need one more program to build our floppy: syslinux. This is an i386 bootstrap program that will load a kernel and a compressed root file system image from a floppy, hard disk or CD. Again, it's prepackaged with Debian or Red Hat and can be found, at this writing, at ftp://ibiblio.org/pub/Linux/system/boot/loaders/syslinux-1.48.tar.gz.
Create an MS-DOS file system on a floppy by using the Linux mformat command (or by another means). Put the floppy in your drive but don't mount it, and install the syslinux bootstrap with this command:
That will copy a first-stage bootstrap onto the first block of the floppy and a second-stage bootstrap into the file LDLINUX.SYS in the MS-DOS file system of the floppy. I'm assuming you have a directory called /mnt; substitute whatever directory you usually use for mounting floppy disks in the shell commands below. Now, it's time to copy our kernel and root file system onto the floppy:
Create the configuration file /mnt/syslinux.cfg with an editor:
This tells syslinux to wait for two seconds and then boot the default system. You can interrupt the default boot during those two seconds by pressing the shift key. Typing linux at the prompt will do the same thing as the default.
The kernel is booted with the arguments root=/dev/ram0 and initrd=fs.gz. These arguments tell the kernel that the root is a RAM disk, and that the RAM disk is loaded from the compressed ROM file system image fs.gz. Although the ``ROM'' root file system is actually in RAM, it will not be writable because the Linux ROM file system driver used in this example is meant to work with real ROMs and thus doesn't support writing.
The Smoke Test
Configure an i386 PC to boot from the floppy. Note that this is a setting in the BIOS preferences of most new PCs, and they are often configured to boot from the hard disk without first looking for a bootstrap on the floppy. Place the floppy in the first floppy drive, and restart the system. You should see something like the following:
Uncompressing linux, OK, booting the kernel
Tons of cybercrud about every device driver and facility in the kernel race by too rapidly to read...
RAMDISK: Compressed image found at block 0.
VFS: Mounted root (romfs file system)
You've done it! Press Enter and you should see something like this:
Pop out the floppy disk; it's not being used any longer. The root file system is entirely in RAM. You should be able to look around the system and try out commands, but you won't have any writable storage. This is the simplest, bootable-system, running busybox that I could think of, and thus I've left out the files in /dev that you'd need to mount writable RAM disks, floppies and hard disks. You now should be able to figure out what those devices are, add them and build a tiny Linux system specialized to your application.
Let's get smaller, writable and multiuser! A new library, new kernel and other bells and whistles gives us more practice building a highly functional Linux system with a tiny footprint.
This is the third article in the series ``Building Tiny Linux Systems with BusyBox''. You'll need to refer to my last article (part 2) on kernel building in the January/February 2001 issue of ELJ if you don't have a paper copy of the previous issue on hand, you can refer to the on-line magazine archive at http://embedded.linuxjournal.com/magazine/issue01/4395/ [Part 1 appeared in the ELJ supplement (November 2000) and is available at http://embedded.linuxjournal.com/magazine/issue00/4335/.]
In this article, we'll add writable storage and a multiuser capability to your tiny Linux system, and when we're done the system will be much smaller than when we started.
Time Marches On
Progress on BusyBox and its friends has continued since I wrote the first two articles, largely due to the efforts of Erik Andersen, whose name I so horribly misspelled in the first article. Erik's work on BusyBox and associated programs is supported by Lineo and benefits the entire Embedded Linux community. Erik and his colleagues have continued to contribute to BusyBox using the original license I chose, the GPL, and they have made great contributions to other embedded programs under the GPL and LGPL.
In my last article, we used the 2.2 kernel, due to a bug in 2.4. That has now been repaired, and since the 2.4 kernel has been released, let's try it. syslinux has also been updated to cope with the new kernel.
Software to Download
We'll be using some new software: TinyLogin and uClibc, and new versions of BusyBox and syslinux. Create a directory where you will work on this example, and change to that directory. Download the 2.4 kernel source from http://www.kernel.org or any of its mirrors. Download all of the files from ftp://ftp.perens.com/ELJ/3/. Read the README.txt file. Extract the compressed tar archives you've just downloaded. Don't run the shell script as_in_episode_2.sh yet. Build and install syslinux from the downloaded sources, replacing the old version if you've installed it on your system. Build BusyBox from the downloaded sources.
Getting Comfortable with the 2.4 Kernel
Take the time to install the 2.4 kernel on a BusyBox floppy, just to get comfortable with configuring and running it. We'll use some scripts in the elj-3 directory to spare you most of the typing you did in the kernel article.
If you still have the tiny Linux directory that we used in the second article, move it out of the way. We're about to create a new one.
Extract the kernel source and configure it as we configured the kernel in my last article under the subhead ``Build the Kernel'', but add two new features to the kernel if they are not already in it: the DOS FAT filesystem and the VFAT (Windows 95) filesystem. These are in the filesystems menu of the kernel configuration. You will need to use these filesystems on both your development system and your tiny Linux system, so you may end up building and installing two kernels. Remember, you need to keep the kernel on your tiny Linux system small, so it probably won't be the same kernel you run on your development system.
Go to the BusyBox source directory and edit the Makefile, setting DOSTATIC to true to build a statically linked version of BusyBox. Build BusyBox with the make command, and then run:
This will create the tiny Linux directory in the directory above the BusyBox directory and will create the bin, sbin and /usr/sbin directories within the tiny Linux directory. The Makefile will install BusyBox in that directory tree and will create all of the links for the various command names, automatically doing much of the work we performed manually in Part 2. It won't install scripts and files and won't create the /dev directory. One of the scripts you downloaded will do that. Change directory to ../elj-3 and run:
That will create the contents of the /dev and /etc directories and anything else you need to make a ROM filesystem.
Now, follow the instructions in my January/February 2001 article under the subheadings ``Generate the ROM Root Filesystem Image'' and ``Build the Floppy''. Finally, boot the floppy to test it.
Let's Get Small
Another good thing Erik and others have been working on is uClibc, the free embedded C library. I had previously identified this library as being under the GPL license, but I was mistaken. uClibc is under the LGPL and is thus fine for linking with proprietary applications. You can now make BusyBox statically linked to run with uClibc rather than libc5 or libc6. The result is less than 300K in size, and of course, it gets even smaller when the ROM filesystem is compressed. In the uClibc source directory, edit the configuration file and change HAS_MMU to true, HAS_FLOATS to true and KERNEL_SOURCE to the location of your kernel source. Change the definition of the CC to be $(CROSS)gcc -Os to optimize for smaller code size. If your version of gcc does not support the -Os flag, use -O2 instead. Run make to build the libc.a library. In this article, uClib supports only static linking, but dynamic linking has just started to work. We'll give it a little time to stabilize and will cover it in the next issue.
Now, go back to the BusyBox source, edit the Makefile and look for the lines that say ``to compile vs. an alternative libc, you may need to use/adjust the following lines of source to meet your needs''. Below that section are lines that define LIBCDIR, LDFLAGS, LIBRARIES, CROSS_CFLAGS and GCCINCDIR. Most or all of these definitions are commented out. Uncomment the definitions to enable linking with uClibc. Change the definition of LIBCDIR to be the location of your uClibc source. Edit Config.h, and comment out the BB_GETOPT and BB_FEATURE_NFSMOUNT definitions to disable the getopt command and the ability to mount remote NFS filesystems with the mount command. As I write this, there's a bug preventing the BusyBox getopt command from working with uClibc, but that might be fixed by the time you read this article. uClibc does not currently include network support, although I'm sure that someone will add it eventually.
Now, rebuild all of BusyBox with the make clean and make commands. On my system, the result is a statically linked executable of 297,620 bytes in size.
Remove the file tiny-linux/bin/getopt. That file won't be generated this time because we turned off the definition of BB_GETOPT. If we don't remove it, we'll end up with two versions of BusyBox and a lot of wasted space.
Use the instructions above under ``Getting Comfortable with the 2.4 Kernel'', starting with the command:
This generates another floppy. Boot the floppy to test it.
Add Writable Storage
Probably the biggest missing feature in your tiny Linux system so far is the lack of writable storage. Let's fix that now. We'll use a VFAT filesystem for the writable storage. This is an extension of the old DOS FAT filesystem to handle long and mixed-case filenames, introduced for Windows 95. Since your floppy already uses a DOS filesystem to hold the kernel and the compressed ROM image file, it's easiest to use VFAT to extend that filesystem for this example. Become root. Create a tiny var directory. This directory tree will hold files that will be writable on your tiny Linux system. You will copy it to the floppy later on, and it will be mounted on /floppy/var when your tiny Linux system boots.
Move your entire tiny-linux/etc directory to tiny-var/etc using the command:
Create two other directories: tiny-var/shm and tiny-var/home. The first directory is used by the mount command as a placeholder when providing shared memory, and the second is used to hold home directories for users.
Now we'll create symbolic links in tiny Linux to redirect some directories to /var so that you will have access to your writable storage and then redirect /var to /floppy/var. We do the symbolic links in two levels here so that the location of all of the writable storage can be changed by replacing only one link:
Create the shell script startup.sh as shown below:
This script will run when the system starts, instead of init, because we'll provide a special command-line argument to the kernel. It will mount the floppy so that the contents of /etc are correct before init starts. Note that exec/sbin/init is different from simply running /sbin/init. The exec command says to replace the shell with the given program, rather than run the program as a subprocess of the shell. Thus, init will run in the same process-ID as the shell, which in this case will be process-ID 1. This is important as the kernel gives process-ID 1 some special properties, and init will not run properly unless it's in process-ID 1.
Change the mode of the script so that it is executable:
Change directory up one level (cd ..):
Use an editor to modify the file tiny-var/etc/inittab, which tells /bin/init, the system startup program, which processes to start. The entire contents of the file should look like this:
We're adding lines here to cleanly unmount filesystems at shutdown time, so that the floppy filesystem containing the writable files will be unmounted. Tasks on lines that start with ::ctrlaltdel: will be executed just before the system halts. We also turn off any swap partitions that are activated, just in case they live on a mounted filesystem.
Create the special device file necessary to mount the floppy:
Now, regenerate the ROM filesystem from the tiny Linux directory, using genromfs as before, and compress it using gzip to create a new fs.gz.
Mount your floppy with this command:
and copy the new fs.gz to your tiny Linux floppy, replacing the old ROM filesystem.
Copy the tiny var directory tree to your floppy with the commands:
Tell the kernel to run /startup.sh instead of /sbin/init by editing the file /mnt/syslinux.cfg. Change the line that says
Change directory to /, and unmount the floppy with this command
You should now be able to boot the floppy, and the directories /var, /etc and /home should be writable.
A Multiuser System on One Floppy
Your tiny Linux system currently just logs you in as root. Let's make a more versatile system that supports user and group names, the login command, support for logins via the console, a modem or a serial line, and utilities to add and delete users. Let's also make the disk writable, unlike the system we made in the second article. Let's fit all of this, and our BusyBox toolkit of more than 100 general-purpose UNIX commands, on one floppy with room to spare!
Go into the TinyLogin source directory and edit the Makefile. We'll be using uClibc to build TinyLogin, so set DOSTATIC to true, comment out the -lcrypt in the definition of LIBRARIES, and set USE_SYSTEM_PWD_GRP to true. Look in the Makefile for the lines that say ``To compile vs. an alternative libc, you may need to use/adjust the following lines to meet your needs.'' Change the lines below that in the same way that you changed the BusyBox Makefile to enable building with uClibc.
Warning: if you ever link TinyLogin with a C library that uses the Name Service Switch (NSS), including GNU LIBC and probably the default libc on your development system, you must set USE_SYSTEM_PWD_GRP to false in the Makefile. That will replace the functions that read the /etc/passwd and /etc/group files, etc., with a version that does not use NSS. Our tiny Linux system doesn't provide the files and other facilities needed for NSS to work, and thus TinyLogin will fail when built with it. This is the most-often-reported ``bug'' for TinyLogin, and its author is rather weary of explaining how NSS works to everyone who reports this ``bug''.
Build TinyLogin with the make command. The result should be about 58K in size.
Install TinyLogin in your tiny Linux directory using
Change directory up one level (cd ..).
Use an editor to modify the file tiny-var/etc/inittab, which tells /bin/init, the system startup program, which processes to start. The entire contents of the file should look like this:
This configures the system to offer a login on the first four console virtual terminals and on the first two serial ports at 9,600 baud. Create the file tiny-var/etc/passwd containing this line:
Create tiny-var/etc/group containing this line:
Create the terminal devices:
Change directory up two levels (cd ../..).
Regenerate the compressed ROM filesystem, and install it on the floppy, as you have done before. Copy the new tiny var to the floppy using this command:
Unmount the floppy disk. Boot the floppy to test it. When the floppy boots, you will get a login prompt. Use root as your login--there's no password (yet). You'll see some complaints about missing files. Now, you can experiment with the adduser and passwd commands. You should be able to add users and set their passwords. Once you've added some users, you can log them in. To visit one of the other console virtual terminals, type Ctrl+Alt+F2 through Ctrl+Alt+F4. Ctrl+Alt+F1 will get you back to the first virtual console. You should be able to log in a different user to each virtual terminal. If you can, connect two serial terminals, set to 9,600 baud, to the first two serial ports on the system. You should be able to have users on each terminal too.
Feel free to add the missing files and test the other programs in the TinyLogin package. To have a fully functional system, you'll need /etc/shadow, /etc/gshadow, /etc/securetty and the utmp file. The location of the utmp file is defined in /usr/include/paths.h on your development system.
Security will be nonexistent because the VFAT filesystem doesn't support the ownership and mode information needed to provide security. You'll have to replace VFAT with another filesystem if you actually need this system to be secure. The USMSDOS filesystem is a good replacement for VFAT. Other filesystems are just a bit more complicated because they aren't compatible with the MS-DOS FAT filesystem used by syslinux to load the kernel and the ROM filesystem image. You could replace the FAT filesystem entirely with the MINIX or EXT2 filesystems, but you'd have to switch your boot loader from syslinux to LILO or grub. You could also mount the writable filesystem from another disk drive.
In the next issue, we'll work on security, and we'll provide the remaining facilities used by TinyLogin. Don't let that keep you from experimenting between now and then--you'll understand how the system works much better if you figure it out by yourself. We'll add dynamic linking and shared libraries and some other exciting new facilities. So, stay tuned!