BBC User's Guide

 

BBC User's Guide

http://www.inf.u-szeged.hu/jffs2/user/

 

Ferenc Havasi

Introduction

BBC (Block Based Compression framework) is an extension to the JFFS2 [ 1] file system used in embedded systems with Linux. JFFS2 has a compression feature which uses  zlib for compression. BBC replaces  zlib with a compressor framework, which is able to use an arbitrary set of compressors.

JFFS2 is a block based file system, so the stored information is compressed in blocks individually.

There are two kinds of compressors. A compressor plugin can be model-based or non-model-based. Model based compressors have the possiblity to collect information about the whole file system and store it into a (model) file which will be avaiable when the block of the file system is being compressed/decompressed. (zlib is a non-model-based one.)

 

BBC framework

 

Method Overview

JFFS2 is a block based file system, so the stored information is compressed in blocks individually. BBC is a complete compression framework, which has the capability to make this compression more powerful.

Figure 1: Overview of BBC

As we mentioned above there are two kinds of compressors:

Non-model-based ones

compress the block individually. (e.g. compressor B)

Model based ones

have the possiblity to collect information about the whole file system and store it into a (model) file which will be avaiable when the block of the file system is being compressed/decompressed. (e.g. compressor A)

There are three phases:

model generation:

Model based compressors have to generate  model file(s). The compression and decompression will be done using these  models. The input of this step is a file structure, the output is one or more  model file. The model-based compressors must provide their model creator programs. ( modelgen programs)

file system image creation:

The input of this phase is a file structure, the output is an  image file. It is done by the program  mkfs.jffs2. The files are first split into blocks of a given size ¹, and each block is compressed with one of the compressors (depending on the compression mode). For special reasons, the model(s) and some operating system file blocks must be compressed with  zlib.

active usage of file system:

After the image is ready, it is flashed onto a real device (for example an iPAQ or an Innovator Board). When these machines are operating and a file is read, the file system knows how to decompress each block of the file (due to a kernel modification), and assembles it. When a file is written (either modifying an existing file or creating a new one), the procedure is the same as in the image creation stage.

 

Compression modes

JFFS2 stores files in blocks, and every block is compressed individually. BBC uses the first two bytes of the compressed blocks to identify the corresponding compressor (and if necessary the model file). To compress a block the following compression modes can be used:

zlib mode:

in this mode the compressor framework always chooses  zlib. You must use it for example to upload kernel files (because of compatibility). This is the deafult.

size mode:

in this mode the compressor framework tries out all compressors with all matching models, and chooses the best (smallest) result. It serves the best compression ratio.

fastr mode:

in this mode the compressor framework calls the read speed estimator function of all compressors (using the model file which has the highest block signature) and uses the most promising one to compress the block - if it fails,  zlib will be called. This mode produces a slightly worse compression ratio than  size mode, but it should be much faster in reading.

fastw mode:

the same as above but calls the write speed estimator function.

fasts mode:

the same as above but calls the compressed size estimator function.

manual mode:

in this mode the user can specify a compressor. The framework will always call the specified compressor (with all matching model files), or if this fails  zlib will be called.

dummy mode:

does not compress

Important note: there is a further possibility to control/tune the compression: the user can deactivate/activate every compressor individually. If a compressor is deactivated it will be ignored in every compression mode. See Section  2.4.

 

Creation of a file system image

In this section the basic steps are given on how to create a working system with BBC tools. A proper cross-compiler (GCC) and Linux kernel source should be available on the host machine in order to be able to perform the following steps.

1. Compile model generators and mkfs.jffs2 as described in Section 2.6.1 and 2.6.3.
2. Compile the kernel for the embadded system (if you need), see Section 2.6.2.
3. After this, create a directory which will be the root of the file system (referred to as $(ROOT) in the following). Create the directory structure you wish in $(ROOT) and copy all the needed files.
4. Generate model file(s) using the modelgen programs for the directory $(ROOT). See the documentation of the compressor plugins!
5. Copy the kernel files into the appropriate (usually boot) directories under $(ROOT).
6. Copy the model file(s) into $(ROOT).
7. Create a configuration file (described in Section 2.7) that contains the model file name in the first line (or if you have multiple model files put them into further lines). It is followed by a line that contains only a "-" character. The remaining lines can contain the names of special files and the name of the required compressor separated by ":" (e.g. kernel files need to be compressed by zlib, so its lines are "boot/kernel_img:ZLIB").
8. Put this configuration file into $(ROOT), and rename it to bbc.conf.
9. Finally invoke mkfs.jffs2 for $(ROOT) (see Section 2.7). This creates the image that can be flashed.

 

Active usage of the file system

If the image is ready, it can be flashed onto the embedded system (eg. iPAQ or Innovator board). This image may contain model file(s), and may contain kernel file(s) in a bootable format. Of course, these kernel files must be created from the kernel source modified by BBC to be able to decompress blocks compressed by orher compressors. (See Section 2.6.2)

Using these modified kernel versions there will be a new file in the proc file system called /proc/jffs2_bbc. From this file you can get some information and statistics about the BBC subsystem (loaded compressors, models, compression statistics, etc.). You can also give a command to BBC by writing to this file. Valid commands are:

cm

S	switch to size compression mode
FR	switch to fastr compression mode
FW	switch to fastw compression mode
FS	switch to fasts compression mode
Z	switch to zlib compression mode
D	switch to dummy compression mode
M:compressor_name	switch to manual compression mode
0:compressor_name	deactivate the specified compressor
1:compressor_name	activate the specified compressor
R	reload bbc.conf and load/unload
	model files on every file system
C:compressor_name:command	give a command to the compressor

cm Valid compressor commands can be found in the description of the compressors.

The effect of the commands can be checked by reading the file /proc/jffs2_bbc.

 

Uploading a new model file

The model files of the model-based compressor are created specially for the given file system and it is not efficient to compress new (nonseen) files. If you upload new files onto the system almost every new block will be compressed with the non-model-based compressors.

A better solution can be the following procedure:

1. Create a new model file for the new executables
2. Upload the new model file to the root directory of the file system. Use the zlib compression method! Insert the name of the new model files into the file of bbc.conf. (see Section 2.3)
3. Activate the new model file using the proc interface of BBC by the following command: echo R >/proc/jffs2_bbc or reboot the system.
4. Upload the new files to the system.

Note: every model file must have a different block signature (see the BBC Technical Documentation and the documentations of the compressor plugins). Thesize compression mode tries out every model file, fastX modes use only one model per compressor (which has the highest block signature).

 

Compiling BBC tools and source details

This is the directory structure of BBC:

 

doc

documentation

src/jffs2

source of the framework, filesystem modification and the comprssor/decompressor part of the compressor plugins

src/util

source of the  mkfs modification and the model generation part of the model based compressors

src/patches

special patch utility and patch files for different kernel versions and mtd-util.

src/test

test utilities

src/models

pregenerated model files

 

Compiling model generators

Enter src/util and type make! You will find some modelgen.xxx executables where xxx is the name of the compressor.

 

Compiling the kernel

Apply the specific hpatch file using src/install.jffs2, and compile your kernel! The first parameter of the script is the name of the hpatch, the second is the kernel directory. You can ask a list of the avaiable hpatch files with paramter -l:

hafy@kpax:~/dev/bbc/src$ ./install.jffs2 -l
linux-2.4.19-rmk6-pxa1-hh30.hpatch  linux-2.4.25.hpatch linux-2.6.3.hpatch
hafy@kpax:~/dev/bbc/src$ ./install.jffs2 linux-2.4.25.hpatch /usr/src/linux/

The hpatch files are as general as possible - for example you can use the hpatch of 2.4.25 for almost all not too old 2.4.x kernels. If you downloaded the source of BBC from CVS, make sure, that  src/pathes/hpatch script is executable, and the first line points to the location of the perl interpreter! If the patching is successful you can individually select the compressor plugins (with  make menuconifg).

 

Compiling mkfs

Follow these steps:

1. Download the lastest MTD source from http://www.linux-mtd.infradead.org/, or use some later version if you want.
2. Type make mkfs.jffs2 to make some symlinks (it is necessary only for some older versions of MTD)!
3. Apply the specific hpatch file using src/install.mkfs. The usage of this script is similar to the one above.
4. Type make mkfs.jffs2 in the directory of util of the mtd source, and you will get the mkfs.jffs2 binary.

 

Using mkfs.jffs2

This program creates a file system image according to the desired compression mode (See Section 2.3). Before invoking the program, the directory structure of the image must be created with the model file copied into the root directory of the image.

The input is the root directory of the directory structure that will be flashed. The model file has to be in the root of this structure. There also must be an bbc.conf file. See details below.

Output: the image of the file system that can be flashed onto the embadded system.

 

mkfs.jffs2 [-m,--compression-mode=mode]
           [-x,--disable-compressor=name_of_the_compressor]
           [-b,--big-endian -l,--little-endian] [-e,--eraseblock=SIZE]
           [-s,--pagesize=SIZE] [-p,--pad[=SIZE]] [-h,--help]
           [-v,--version] [-d,-r,--root directory] -o,--output image.jffs2


  -x,--disable-compressor=name_of_the_compressor
                      Disabling the specified compressor
  -m,--compressor-mode=name_of_the_compressor
                      Set the compression mode.
		      Avaiable: size (default), fastr, fastw, fasts, zlib

  -b,--big-endian     These options specify the endianness of the image. Only
  -l,--little-endian  one can be used at a time. [def: same as host]

  -e,--eraseblock     Size of erasable blocks [def: 64kB]
  -s,--pagesize       Size of a page [def: 4kB]
  -p,--pad            Padding size [def: end of last erase block]
  -d,-r,--root        Root directory of the image [def: current dir]
  -o,--output         Output file

Example:
   mkfs.jffs2 -rroot_fs -oroot.jffs2

Options that take SIZE arguments can be specified either as decimal (e.g., 65536), octal (0200000), or hexadecimal (0x1000).

 

-b, -big-endian, -l, -little-endian

These options specify the endianness of the created image. -b and -big-endian mean big-endian, -l and -little-endian mean little-endian image. Only one of these options can be used at a time. The default is to make an image with the same endianness as the host. This value must be the same as that of modelgen.

-m, -compression-mode=mode

See section Compression modes. Only one of these options can be used at a time. The default is to make an image with size mode.

-x, -disble-compressor=name

Disable the specified compressor.

-e, -eraseblock

This option sets the size of the erasable block. The default is 64 kB. If an erase block size different than the erase block size of the target device is used, JFFS2 may not perform optimally.

-s, -pagesize

This option sets the page size. The default is 4 kB. This size is the maximum size of a data node. This value should be the same as the value of -blocksize parameter of program modelgen.armlib.

-p, -pad

Sets the size of padding. If not specified, the output is padded to the end of the last erase block. The padded bytes get the value 0xFF.

-d, -r, -root

The specified directory will be the root directory of the image. The default value is the current directory.

-o, -output

This specifies the file where the image is written.

-h, -help

Displays help.

-v, -version

Displays version information.

 

Configuration file (bbc.conf)

This configuration file is a text file, in which every line contains a file name relative to the specified root directory. It has two parts, separated by a line that contains a "-" character only. In the first part there are the names of the model files (one per line). They must be in the root directory of the image. In the second part the lines contain the name of the file and optionally a compressor name separated with ":". For examle: 
usr/bin/something:armlib 
All files listed in the configuration file are compressed using the specified compressor (or with zlib if no compressor is specified). A complete example:

armlib-1.model
armlib-2.model
-
boot/zImage

 

 

Compressors

They are described in seperated documents.

 

Bibliography

 

 

1

David Woodhouse.: 
JFFS: The Journalling Fash File System. 
January 2001. 
http://sources.redhat.com/jffs2/

 

 

About this document ...

BBC User's Guide

This document was generated using the LaTeX2HTML translator Version 2002-2-1 (1.70)

Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds. 
Copyright © 1997, 1998, 1999, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The command line arguments were: 
latex2html -split 0 -no_navigation user.tex

The translation was initiated by Havasi Ferenc on 2004-03-02

---

Footnotes

... size ¹

Typical size of a block is 4Kb.