I needed to debug a kernel crash on SUSE Linux Enterprise Server 11 today. If you’re not familiar with debugging Linux kernel crashes, you need the kernel debug symbols in order to analyze the crash dump. These are typically not part of the kernel image itself, but instead are bundled into a kernel debuginfo package corresponding to the kernel that produced the crash dump.
Although I’ve done this on RedHat Enterprise Linux many times, I had never debugged a kernel crash on SUSE before, so I was not familiar with the process for acquiring the debuginfo packages with that distro. I couldn’t find any single set of instructions explaining how to get the packages, and although it wasn’t hard, I figured I’d try to save somebody else a little time by writing down the steps I followed.
SUSE uses a package manager called ZYpp. I used zypper, the command-line interface to ZYpp, to install the packages.
Step 1: enable the debuginfo repositories
Before zypper can install the debuginfo packages, it must be able to find them. The packages reside in specialized debuginfo repositories, which are normally not enabled, although the system is aware of them. Use zypper repos to get a list of the repositories:
lin4-ea6:~ # zypper repos
# | Alias | Name | Enabled | Refresh
--+----------------------------------------------------------+--------------------------------------------------------+---------+--------
1 | SUSE-Linux-Enterprise-Server-11 11-0 | SUSE-Linux-Enterprise-Server-11 11-0 | Yes | No
2 | SUSE-Linux-Enterprise-Software-Development-Kit-11_11-0 | SUSE-Linux-Enterprise-Software-Development-Kit-11 11-0 | Yes | No
3 | SUSE-Linux-Enterprise-Software-Development-Kit-11_11-0_1 | SUSE-Linux-Enterprise-Software-Development-Kit-11 11-0 | Yes | No
4 | nu_novell_com:SLE11-Debuginfo-Pool | SLE11-Debuginfo-Pool | No | Yes
5 | nu_novell_com:SLE11-Debuginfo-Updates | SLE11-Debuginfo-Updates | No | Yes
6 | nu_novell_com:SLES11-Extras | SLES11-Extras | No | Yes
7 | nu_novell_com:SLES11-Pool | SLES11-Pool | No | Yes
8 | nu_novell_com:SLES11-Updates | SLES11-Updates | Yes | Yes
You want the two Debuginfo repos. To enable them, use zypper modifyrepo with the alias of the repo:
lin4-ea6:~ # zypper modifyrepo --enable nu_novell_com:SLE11-Debuginfo-Pool
Repository 'nu_novell_com:SLE11-Debuginfo-Pool' has been sucessfully enabled.
lin4-ea6:~ # zypper modifyrepo --enable nu_novell_com:SLE11-Debuginfo-Updates
Repository 'nu_novell_com:SLE11-Debuginfo-Updates' has been sucessfully enabled.
Step 2: find the debuginfo package for your crash
It’s critical to get the debuginfo package that matches the kernel that created your crash dump. It’s easy to determine the version you need: check the README.txt alongside the vmcore in the crash directory:
lin4-ea6:~ # cat /var/crash/2010-11-30-00:43/README.txt
Kernel crashdump
----------------
Crash time : 2010-11-30 00:43 (+0000)
Kernel version : 2.6.27.45-0.1-pae
Host : SLES-11-32
Dump level : 0
Dump format : compressed
In this case, I need the debuginfo for the pae variant of kernel version 2.6.27.45-0.1. Now, search the package repository for kernel debuginfo packages with zypper search:
lin4-ea6:~ # zypper search -s kernel-*-debuginfo*
Loading repository data...
Reading installed packages...
S | Name | Type | Version | Arch | Repository
--+--------------------------+---------+------------------+------+------------------------
| kernel-default-debuginfo | package | 2.6.27.54-0.2.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.48-0.6.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.48-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.45-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.42-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.39-0.3.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.37-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.29-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.25-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.23-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.21-0.1.2 | i586 | SLE11-Debuginfo-Updates
| kernel-default-debuginfo | package | 2.6.27.19-5.1 | i586 | SLE11-Debuginfo-Pool
| kernel-pae-debuginfo | package | 2.6.27.54-0.2.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.48-0.6.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.48-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.45-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.42-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.39-0.3.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.37-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.29-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.25-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.23-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.21-0.1.2 | i586 | SLE11-Debuginfo-Updates
| kernel-pae-debuginfo | package | 2.6.27.19-5.1 | i586 | SLE11-Debuginfo-Pool
| kernel-source-debuginfo | package | 2.6.27.54-0.2.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.48-0.6.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.48-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.45-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.42-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.39-0.3.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.37-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.29-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.25-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.23-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.21-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-source-debuginfo | package | 2.6.27.19-5.1 | i586 | SLE11-Debuginfo-Pool
| kernel-vmi-debuginfo | package | 2.6.27.54-0.2.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.48-0.6.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.48-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.45-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.42-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.39-0.3.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.37-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.29-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.25-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.23-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.21-0.1.2 | i586 | SLE11-Debuginfo-Updates
| kernel-vmi-debuginfo | package | 2.6.27.19-5.1 | i586 | SLE11-Debuginfo-Pool
| kernel-xen-debuginfo | package | 2.6.27.54-0.2.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.48-0.12.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.48-0.6.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.48-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.45-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.42-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.39-0.3.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.37-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.29-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.25-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.23-0.1.1 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.21-0.1.2 | i586 | SLE11-Debuginfo-Updates
| kernel-xen-debuginfo | package | 2.6.27.19-5.1 | i586 | SLE11-Debuginfo-Pool
You can see there are several versions of each variant available. One tricky thing is that there isn’t an exact match for the kernel version I need. I’m looking for 2.6.27.45-0.1; the closest thing to it is 2.6.27.45-0.1.1. This seems to be nothing more than a minor inconsistency in labeling: the 2.6.27.45-0.1.1 is the correct package.
Step 3: install the kernel debuginfo package
Having identified the package, you are ready to install it with zypper install:
lin4-ea6:~ # zypper install kernel-pae-debuginfo=2.6.27.45-0.1.1
Loading repository data...
Reading installed packages...
Resolving package dependencies...
The following NEW package is going to be installed:
kernel-pae-debuginfo
The following package is not supported by its vendor:
kernel-pae-debuginfo
Overall download size: 153.1 M. After the operation, additional 673.8 M will be used.
Continue? [YES/no]:
At this prompt, you should type YES and hit Enter. zypper will download the package (which may take a while depending on the speed of your internet connection), showing a progress bar as it does:
Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [90% (1.1 M/s)]
Then zypper will install the package, again with a progress bar:
Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [done (244.7 K/s)]
Installing: kernel-pae-debuginfo-2.6.27.45-0.1.1 [84%]
When installation is complete, there is no more notification than that the progress bar reads “done”:
Retrieving package kernel-pae-debuginfo-2.6.27.45-0.1.1.i586 (1/1), 153.1 M (673.8 M unpacked)
Retrieving: kernel-pae-debuginfo-2.6.27.45-0.1.1.i586.rpm [done (244.7 K/s)]
Installing: kernel-pae-debuginfo-2.6.27.45-0.1.1 [done]
Step 4: symlink the debuginfo into the crash directory
The debuginfo package, for some reason, does not install the debuginfo files to a location known to the crash utility, so the final step is to make the debuginfo file available to crash before we invoke it. We do so by creating a symlink to the debuginfo file alongside the kernel image in the crash directory:
lin4-ea6:~ # ln -s /usr/lib/debug/boot/vmlinux-2.6.27.45-0.1-pae.debug /var/crash/2010-11-30-00:43
Decompress the kernel image and you’re in business:
lin4-ea6:~ # gzip -d /var/crash/2010-11-30-00:43/vmlinux-2.6.27.45-0.1-pae.gz
lin4-ea6:~ # cd /var/crash/2010-11-30-00:43
lin4-ea6:/var/crash/2010-11-30-00:43 # crash vmlinux-2.6.27.45-0.1-pae vmcore
886
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
