Using smem on Android

smem is a handy tool for analyzing system memory.

Quick steps

For the impatient, here are some quick steps:

* download smem
  * hg clone http://selenic.com/repo/smem
  * cd smem
* (install python-matplotlib and python-matplotlib-tk, if needed)
* cross-compile smemcap for Android, using something like 'agcc'
  * agcc smemcap.c -o android-smemcap
* install android-smemcap onto your target
  * adb shell mkdir /data/smem
  * adb push android-smemcap /data/smem
* collect memory information on target
  * adb shell "/data/smem/android-smemcap >/data/smem/memdata1.tar"
* retreive the collected data
  * adb pull /data/smem/memdata1.tar .
* review the data on host
  * smem -S memdata1.tar

This document describes how to use 'smem' with Android.

Introduction

smem is a memory tool for analyzing the amount of memory usageon a Linux system. The 'ps' tool provided with Android only shows theVSIZE and RSS memory columns. Even if you install busybox on anandroid system, that tool's 'ps' will also only show you only the VSIZEvalue. VSIZE is the total size of the virtual image, and RSS is theresident set size - the amount of physical pages accounted by the kernelto a particular process. These numbers are not very useful to indicatethe actual cost in memory of a particular application or daemon.

On the smem web site it says:

smem is a tool that can give numerous reports on memory usage on Linux systems.Unlike existing tools, smem can report proportional set size (PSS), which is amore meaningful representation of the amount of memory used by libraries andapplications in a virtual memory system.

Because large portions of physical memory are typically shared among multipleapplications, the standard measure of memory usage known as resident set size (RSS)will significantly overestimate memory usage. PSS instead measures eachapplication's "fair share" of each shared area to give a realistic measure.of the activity of a system during boot.

smem consists of a data capture tool, for use on an android target,and an analysis tool for use on a development host machine.

Overview of Steps to Use smem

To use smem on an Android System, you need to performthe following steps:

* install the smem tool on your host machine
* build smemcap on your host machine (or use a pre-built binary)
* install smemcap on your Android target system
* use smemcap to capture the memory information from /proc on the target
* retrieve the collected data from the system
* use the smem tool with the collected data, on the host
* view memory information and charts
* [optionally] provide additional system information

These steps will be described in detail in the followingsections. This paper was written based on Android 1.5software and SDK.

Detailed Instructions

Installing the smem tool

smem can be downloaded from http://selenic.com/repo/smem

Download using the mercurial tool (hg)

    $ hg clone http://selenic.com/repo/smem
destination directory: smem
requesting all changes
adding changesets
adding manifests
adding file changes
added 37 changesets with 40 changes to 5 files
5 files updated, 0 files merged, 0 files removed, 0 files unresolved

This will create a directory called 'smem' in the current directory.

Usually, you will want to place the 'smem' program on your path. This can be done by copying smem to /usr/local/bin, like so (as root):

   $ cp smem/smem /usr/local/bin

smem has a few requirements:

• a reasonably modern kernel (> 2.6.27 or so)
• a reasonably recent version of Python (2.4 or so)

The smem tool can be used in text mode with no further installation.

However, smem also provides the ability to produce nice charts and graphs.For this, it needs the python matplotlib plotting library.

On Fedora, you can install this with the command:

  $ yum install python-matplotlib python-matplotlib-tk

Other Linux distributions may also have packages for matplotlib.

Here are links to information if you need to directly download and/or install matplotlib

• http://matplotlib.sourceforge.net/index.html - home page
• http://sourceforge.net/projects/matplotlib - download area
• http://matplotlib.sourceforge.net/users/installing.html - install instructions

Building 'smemcap' (or use pre-built binary)

To collect the memory information from the /proc filesystem on the targetyou need to run the 'smemcap' program. smem also includes a shell scriptcalled 'capture' which can collect the data on a standard Linux desktop.However, the Android environment does not have the necessary shell environmentor command line tools for this program to work. So you will need to usethe compiled C program.

Compile the smemcap program as a native Android program against thebionic C library. Details about this are outside the scope of thisdocument, but you can use the instructions here:Compiling Native Apps for Android

If you have installed 'agcc', then the following steps should work:

  $ cd smem
  $ agcc smemcap.c -o android-smemcap

Pre-built binary (for G1)

Alternatively, you can use a pre-built capture program. This one worked onmy Android G1 phone and on an OMAP EVM board.

Note: Due to constraints on file uploads on this wiki, I had to upload it as a tar file. Download it and untar with: 'tar -xvf Android-smemcap.tar'

Installing 'android-smemcap' onto your Android target

To install the android-smemcap program on your target, you just need to copy theprogram, and make sure it is executable. It is recommended to make a separate directoryfor smem capture data, on the /data portion of the target file system. (However,any writable directory can be used, such as one on an sdcard.)

Make the smem directory, copy the file:

  $ adb shell mkdir /data/smem
  $ adb push android-smemcap /data/smem

You should check that the program is executable:

  $ adb shell chmod 777 /data/smem/android-smemcap
  $ adb shell ls -l /data/smem
  -rwxrwxrwx root     root         7391 2009-06-24 14:47 android-smemcap

Collect memory information on the target

Now, to measure the memory usage on the target, use the android-smemcap program to collect the data from the /proc filesystem.

  $ adb shell
  $ cd /data/smem
  $ ./android-smemcap >memdata1.tar
  $ exit

You can also do this directly from the host, using the adb shell:

  $ adb shell "/data/smem/android-smemcap >/data/smem/memdata1.tar"

Note the use of quotes to have the shell redirection on the targetstay on the target. Due to the binary nature of the data outputby android-smemcap, you can not redirect it directly throughthe 'adb shell' command to the host filesystem.

Retrieving the collected data from the system

Now, collect the data from the target, and retrieve it to the host:

   $ adb pull /data/smem/memdata1.tar .

This will pull the just-created file 'memdata1.tar' from the targetand place it in the current directory.

Review the data on the host

Now, you can use the smem tool on the host, to view the target process data:

   $ smem -S memdata1.tar
  PID User     Command                         Swap      USS      PSS      RSS
   32 0        /system/bin/debuggerd              0       64       70      252
   30 1000     /system/bin/servicemanager         0       80       86      264
   36 1002     /system/bin/dbus-daemon --s        0       92      106      368
   39 0        /system/bin/sh /runme.sh           0       72      106      316
   37 0        /system/bin/installd               0      100      108      332
 2462 0        /system/bin/sh -                   0       84      118      332
   31 0        /system/bin/vold                   0      136      144      392
   41 0        /sbin/adbd                         0      144      144      156
    1 0        /init                              0      152      154      172
 3780 0        ./android-smemcap                  0      188      195      384
   40 1008     /system/bin/akmd                   0      268      303      688
   33 0        /system/bin/rild                   0      444      504      984
   35 1013     /system/bin/mediaserver            0      876     1458     2948
  181 0        com.android.voicedialer            0      948     1550    10668
  263 0        com.instamapper.gpstracker         0     1072     1633    10992
  252 0        com.rerware.android.MyBacku        0     1080     1665    11348
  289 0        com.android.music                  0     1204     1892    11436
  271 0        com.biggu.shopsavvy                0     1236     2050    12344
  241 0        com.android.alarmclock             0     1488     2087    12264
  215 0        com.android.email                  0     1524     2164    12116
  302 0        com.google.android.gm              0     1508     2170    11928
  280 0        com.acme.android.powermanag        0     1884     2533    12688
  135 0        com.android.mms                    0     2036     2711    12708
  164 0        android.process.media              0     2208     2840    13076
  412 0        com.biggu.shopsavvy:remote         0     2036     2904    13668
  228 0        com.android.calendar               0     2296     3005    13596
   34 0        zygote /bin/app_process -Xz        0     1088     3218    16688
  202 0        com.google.android.apps.map        0     2472     3334    14324
  106 0        com.android.phone                  0     4928     5891    16664
  155 0        com.google.process.gapps           0     4628     5933    17408
  108 0        android.process.acore              0     7288     9023    29560
   66 0        system_server                      0    10188    12636    35056

Viewing information and charts

There are a number of interesting options that smem provides, for filtering,analyzing and visualizing the process data.

Here are some interesting options:

show pie chart 

'smem -S memdata1.tar --pie=command'

This produces a pie chart labeled by the command names (with pss as the data key)

Also: 'smem -S memdata1.tar -P com.* --pie=command'

Only showing Android applications (apps starting with "com.")

show a bar chart 

'smem -S memdata1.tar --bar=command'

See below for a sample

show all memory mappings on the system 

'smem -S memdata1.tar -m'

show the mappings for a specific program 

'smem -S memdata1.tar -m -P email'

show the system libraries 

'smem -S memdata1.tar -m -M /system/lib/*'

show memory maps for application packages 

'smem -S memdata1.tar -m -M .*.apk'

find the libraries with the most users 

'smem -S memdata1.tar -m -M lib -s pids'

That is, show mappings for libraries, with 'lib' in the name, sorted by number of processes (pids) using them.

sample output:

Map                                       PIDs   AVGPSS      PSS
/system/lib/hw/copybit.msm7k.so              1        8        8
/system/lib/hw/lights.msm7k.so               1        8        8
/system/lib/hw/sensors.trout.so              1       12       12
/system/lib/libagl.so                        1       48       48
/system/lib/libandroid_servers.so            1       16       16
/system/lib/libhtc_acoustic.so               1        4        4
/system/lib/libhtc_ril.so                    1      168      168
/system/lib/libril.so                        1       28       28
/system/lib/libsoundpool.so                  1        4        4
/system/lib/libsurfaceflinger.so             1      108      108
/system/lib/libsystem_server.so              1        8        8
/system/lib/liba2dp.so                       2        6       12
/system/lib/libaudio.so                      2       14       28
/system/lib/libaudioflinger.so               2       62      124
/system/lib/libcamera.so                     2       14       28
/system/lib/libcameraservice.so              2       20       40
/system/lib/libmediaplayerservice.so         2       52      104
/system/lib/libmedia_jni.so                 19        1       35
/system/lib/libwebcore.so                   19       10      191
/system/lib/libEGL.so                       20        0       15
/system/lib/libGLESv1_CM.so                 20        0       12
/system/lib/libOmxCore.so                   20        0        4
/system/lib/libaes.so                       20        0        4
/system/lib/libandroid_runtime.so           20       13      267
/system/lib/libbluedroid.so                 20        0        8
/system/lib/libcorecg.so                    20        2       52
/system/lib/libcrypto.so                    20       21      424
/system/lib/libdrm1.so                      20        0        7
/system/lib/libdvm.so                       20       18      375
/system/lib/libemoji.so                     20        0        4
/system/lib/libhardware.so                  20        0        7
/system/lib/libicudata.so                   20       13      260
/system/lib/libicui18n.so                   20       15      304
/system/lib/libicuuc.so                     20       18      363
/system/lib/libmedia.so                     20        5      102
/system/lib/libnativehelper.so              20        6      135
/system/lib/libopencoreauthor.so            20       13      265
/system/lib/libopencorecommon.so            20        9      185
/system/lib/libopencoreplayer.so            20       29      593
/system/lib/libpixelflinger.so              20        3       71
/system/lib/libsgl.so                       20       29      583
/system/lib/libskiagl.so                    20        0        4
/system/lib/libsonivox.so                   20        0        7
/system/lib/libsqlite.so                    20       16      325
/system/lib/libssl.so                       20        4       97
/system/lib/libui.so                        20       12      245
/system/lib/libvorbisidec.so                20        4       94
/system/lib/libdbus.so                      21        0       16
/system/lib/libexpat.so                     21        3       67
/system/lib/libgps.so                       21        2       52
/system/lib/libhardware_legacy.so           21        1       40
/system/lib/libnetutils.so                  21        0       16
/system/lib/librpc.so                       21        0       16
/system/lib/libwpa_client.so                21        0       12
/system/lib/libutils.so                     22       18      404
/system/lib/libz.so                         22        3       77
/system/lib/libcutils.so                    24        5      131
/system/lib/liblog.so                       25        0       24
/system/lib/libstdc++.so                    29        1       40
/system/lib/libc.so                         30       14      444
/system/lib/libm.so                         30        2       88

show processes using libc, sorted by PSS 

'smem -S memdata1.tar -M libc'

sample output:

  PID User     Command                         Swap      USS      PSS      RSS
  135 0        com.android.mms                    0        8       16      264
  164 0        android.process.media              0        8       16      264
  181 0        com.android.voicedialer            0        8       16      260
  202 0        com.google.android.apps.map        0        8       16      264
  215 0        com.android.email                  0        8       16      260
  228 0        com.android.calendar               0        8       16      264
  241 0        com.android.alarmclock             0        8       16      264
  252 0        com.rerware.android.MyBacku        0        8       16      260
  263 0        com.instamapper.gpstracker         0        8       16      260
  271 0        com.biggu.shopsavvy                0        8       16      260
  280 0        com.acme.android.powermanag        0        8       16      260
  289 0        com.android.music                  0        8       16      260
  302 0        com.google.android.gm              0        8       16      264
  412 0        com.biggu.shopsavvy:remote         0        8       16      264
   30 1000     /system/bin/servicemanager         0       16       19      124
   32 0        /system/bin/debuggerd              0       16       19      132
   39 0        /system/bin/sh /runme.sh           0       16       19      132
 2462 0        /system/bin/sh -                   0       16       19      136
   36 1002     /system/bin/dbus-daemon --s        0       16       20      148
 3780 0        ./android-smemcap                  0       16       20      144
   31 0        /system/bin/vold                   0       20       25      192
   37 0        /system/bin/installd               0       20       25      168
   40 1008     /system/bin/akmd                   0       20       25      176
  106 0        com.android.phone                  0       12       25      276
   33 0        /system/bin/rild                   0       20       30      204
  108 0        android.process.acore              0       16       36      312
   34 0        zygote /bin/app_process -Xz        0        8       83      472
   35 1013     /system/bin/mediaserver            0       44       95      344
  155 0        com.google.process.gapps           0       28      187      612
   66 0        system_server                      0       32      248      772

Sample Charts

Here are some examples of graphical charts produced by smem

product a pie chart 

'smem -S memdata1.tar --pie=command'

Pie chart is labeled with the command names (and pss is used as the data key)

---

Produce a bar chart 

'smem -S memdata1.tar --bar=command'

Providing additional system information

You can provide additional information to smem (such as the kernel image size), using the -R and -K options:

Here is an example showing the 'whole system' memory information, while specifing a kernel vmlinux image:

   $ smem -S ftrace-kernel.tar -w -t -K /home/tbird/work/android/kernel-msm/build/g1/vmlinux
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                   4515          0       4515
kernel dynamic memory         17500       4784      12716
userspace memory              78544      28736      49808
free memory                    1988       1988          0
----------------------------------------------------------
                             102547      35508      67039

In this example, my kernel was a test kernel that was significantly larger than a standard productionkernel.

Using the graphical viewer

The graphical viewer provides options for zooming and panning portions of the graph, for adjusting graph display parameters, switching between views, and saving a view to an image file.

Here is a snapshot of the smem (matplotlib) chart viewer:

Here are some tips:

Sometimes it is difficult to read the chart labels. You can view the labels more easilyif you pan and zoom the chart.

To zoom in on an area, click on the 'zoom to rectangle' button. Drag a rectangle in the viewing area.For a bar chart, the rectangle must reside within the graph area of the chart (above the x axis and to theright of the y axis). Once you unclick your mouse button, a zoomed image is then shown.You can pan the zoomed image by clicking the 'pan axes' button, and dragging the image withthe left mouse button. (This "pan axes" button puts the chart in 'pan/zoom' mode. Clickthe button again to revert to regular view mode.)

You can also zoom using the right mouse button in the 'pan/zoom' mode.

You can adjust the location of the chart on the page (which is also sometimes necessary to see the full labels), by clicking on the "configure subplots" button. Experiment with changing different parameters, such as the bottomor left parameters.

Save a view to a file using the "save the figure" button. Select the image format to save in, from the drop-downlist at the bottom of the dialog. Available formats are: bmp, eps, jpg, png, ps, svg.

Interpreting the results

See the smem web site for an explanation of USS and PSS, and how these measurements should be interpretedto analyze a running system.