Linux: Debugging With "git bisect"

Following up to a bug report against the 2.6.22 kernel, Andrew Morton and Linus Torvalds offered some tips on how to debug kernel problems. Andrew first pointed to netconsole.txt for instructions on setting up a netconsole, "when the machine has stalled, see if you can get a task trace with ALT-SYSRQ-t. This will require CONFIG_MAGIC_SYSRQ=y and possibly setting ignore_loglevel on the kernel boot command line."

Linus Torvalds suggested "git bisect" as an alternative, "[it] will take some time, but is really a lot easier" He explains, "there's almost 7000 commits in between 2.6.21 and 22, but that still means that in about fourteen recompiles/reboots, "git bisect" should tell us where your problem starts, which will hopefully make it obvious what the problem is (or at least pinpoint it a *lot*)." He goes on to detail how to install git, obtain the latest kernel, and run "git bisect", "doing a git bisect isn't really that hard, but fourteen compiles/reboots will take some time (well, the compiles will, the reboots aren't that bad). But even if you're not a git user, it really is very simple". Specifically, he notes, "start the 'git bisect' with 'git bisect good v2.6.21', 'git bisect bad v2.6.22', and it will pick a kernel version about half-way between the two points, and you can now start testing. For each kernel you try, if it boots fine, do 'git bisect good', otherwise boot into a working kernel, and then do 'git bisect bad'. Git will then pick the next 'halfway' kernel for that case."

---

From: Andrew Morton [email blocked]
Subject: Re: Linux 2.6.22 released
Date:	Tue, 10 Jul 2007 01:52:07 -0700

On Tue, 10 Jul 2007 09:17:18 +0200 Stefano Rivoir wrote:

> Linus Torvalds wrote:
> > It's out there now (or at least in the process of mirroring out - if you 
> > don't see everything, give it a bit of time).
> 
> Hi all.
> 
> 2.6.22 hangs at boot on my box. Here attached a original dmesg from 
> 2.6.21, and a copy of it where it stops on 2.6.22 (I can't attach the 
> original 2.6.22 dmesg because it's not logged to disk yet); it actually 
> stops right after 'init' launches.
> 
> Thinking it was something related to USB mass storage, I've disabled 
> (not in the attached .config) it but with no results.

If you have another Linux box on the LAN, please set up netconsole
(Documentation/networking/netconsole.txt) to gather the boot logs.

When the machine has stalled, see if you can get a task trace with
ALT-SYSRQ-t.  This will require CONFIG_MAGIC_SYSRQ=y and possibly setting
ignore_loglevel on the kernel boot command line.

Thanks.

(mad guesses: try the following on the boot command line: clock=pit,
noacpi, noapic)



From:	Linus Torvalds [email blocked]
cc:	Linux Kernel Mailing List [email blocked]
Subject: Re: Linux 2.6.22 released
Date:	Tue, 10 Jul 2007 08:39:11 -0700 (PDT)



On Tue, 10 Jul 2007, Stefano Rivoir wrote:
> 
> 2.6.22 hangs at boot on my box. Here attached a original dmesg from 2.6.21,
> and a copy of it where it stops on 2.6.22 (I can't attach the original 2.6.22
> dmesg because it's not logged to disk yet); it actually stops right after
> 'init' launches.

Ok, without any oops or hang in any really obvious place, that doesn't 
really tell anybody anything specific enough to even start trying to debug 
this, so you'd need to do one of two things:

 - poke a lot at the machine to try to get more specific information. In 
   particular, get things like SysRQ-T output. That, in turn, probably 
   would mean trying to get a serial console hooked up or something.

   The next thing that you got on 2.6.21 after the point it hangs was 
   apparently 

	..
	input: PC Speaker as /class/input/input1
	ieee1394: Initialized config rom entry `ip1394'
	ACPI: PCI Interrupt Link [APC4] enabled at IRQ 19
	ACPI: PCI Interrupt 0000:04:05.0[A] -> Link [APC4] -> GSI 19 (level, low) -> IRQ 19
	usbcore: registered new interface driver hiddev
	..

   so you could _try_ to disable the PC speaker or firewire, and see 
   what's up. Did you switch from the old firewire drivers to the new one, 
   for example? Or if you didn't, try it.

   IOW, we'd need a lot more debug information.

The second alternative will take some time, but is really a lot easier:

 - Get a kernel git tree, and do a "git bisect".

   There's almost 7000 commits in between 2.6.21 and 22, but that still 
   means that in about fourteen recompiles/reboots, "git bisect" should 
   tell us where your problem starts, which will hopefully make it obvious 
   what the problem is (or at least pinpoint it a *lot*).

Doing a git bisect isn't really that hard, but fourteen compiles/reboots 
will take some time (well, the compiles will, the reboots aren't that 
bad). But even if you're not a git user, it really is very simple:

 - get started with 'git': on most distros it's now as simple as doing 
   something like

	yum install git

   and while you might not get the latest version (Debian stable is at 
   some ancient 1.4.4.4 version that isn't as nice as the 1.5.x series), 
   for something like this you won't care that deeply.

 - get the kernel git tree (this will take a while to download about 
   180MB)

	git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6

 - start the "git bisect" with

	git bisect good v2.6.21
	git bisect bad v2.6.22

   and it will pick a kernel version about half-way between the two 
   points, and you can now start testing. For each kernel you try, if it 
   boots fine, do "git bisect good", otherwise boot into a working kernel, 
   and then do "git bisect bad". Git will then pick the next "halfway" 
   kernel for that case.

Thanks,

		Linus

---