程序调试之非法指令（龙芯平台）

更换内核后运行 qrcreator 报非法指令之调试过程

背景：
在中标5.0系统上qtcreator 运行正常，由于中标5.0 自带内核为3.10比较旧，故采用我们自己移植的4.19。结果换了内核之后qtcreator 运行报非法指令。

主要思路为通过gdb 找出产生非法指令的语句。
具体过程如下：

[sancog@localhost ~]$ qtcreator 
非法指令
[sancog@localhost ~]$ whereis qtcreator
qtcreator: /usr/bin/qtcreator /usr/lib64/qtcreator /usr/libexec/qtcreator /usr/share/qtcreator
[sancog@localhost ~]$ cd /usr/bin/

[sancog@localhost bin]$ gdb ./qtcreator 
GNU gdb (GDB) NeoKylin 7.8.2-39.nd7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mips64el-neokylin-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./qtcreator...Reading symbols from /usr/bin/./qtcreator...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install qt-creator-4.3.1-2.nd7.mips.4.mips64el
(gdb) r
Starting program: /usr/bin/qtcreator 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib64/libthread_db.so.1".
[New Thread 0xfff247f0a0 (LWP 5059)]
[New Thread 0xfff0de30a0 (LWP 5060)]
[New Thread 0xffe9f970a0 (LWP 5062)]
[New Thread 0xffe97970a0 (LWP 5063)]
[New Thread 0xffe8f970a0 (LWP 5064)]
[New Thread 0xffdffff0a0 (LWP 5065)]
[Thread 0xffdffff0a0 (LWP 5065) exited]
[Thread 0xffe8f970a0 (LWP 5064) exited]
[New Thread 0xffe8f970a0 (LWP 5067)]
[New Thread 0xffdffff0a0 (LWP 5068)]

Program received signal SIGILL, Illegal instruction.
0x000000fff623624c in QV4::Moth::VME::run(QV4::ExecutionEngine*, unsigned char const*, void***) () from /usr/bin/../lib64/qtcreator/../libQt5Qml.so.5

反汇编找到地址 0x000000fff623624c 的代码

(gdb) disassemble 0x000000fff623624c 
Dump of assembler code for function _ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv:
   0x000000fff623615c <+7140>:	lwu	a0,32(s4)
   0x000000fff6236160 <+7144>:	b	0xfff6234b28 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+1456>
   0x000000fff6236164 <+7148>:	lwu	a1,36(s4)
   0x000000fff6236168 <+7152>:	lwu	v0,8(s4)
   0x000000fff623616c <+7156>:	li	a3,1
   0x000000fff6236170 <+7160>:	lwu	a2,12(s4)
   0x000000fff6236174 <+7164>:	lwu	a0,16(s4)
   0x000000fff6236178 <+7168>:	dsll	v0,v0,0x3
   0x000000fff623617c <+7172>:	lwu	a1,20(s4)
   0x000000fff6236180 <+7176>:	gsldx	v0,0(s5,v0)
   0x000000fff6236184 <+7180>:	dsll	a2,a2,0x3
   0x000000fff6236188 <+7184>:	dsll	a0,a0,0x3
   0x000000fff623618c <+7188>:	gsldx	a0,0(s5,a0)
   0x000000fff6236190 <+7192>:	dsll	a1,a1,0x3
   0x000000fff6236194 <+7196>:	daddu	v0,v0,a2
   0x000000fff6236198 <+7200>:	ld	a2,0(v0)
   0x000000fff623619c <+7204>:	daddu	a1,a0,a1
   0x000000fff62361a0 <+7208>:	dsrl32	a0,a2,0x11
   0x000000fff62361a4 <+7212>:	beq	a0,a3,0xfff6236498 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+7968>
   0x000000fff62361a8 <+7216>:	daddiu	s7,s4,32
   0x000000fff62361ac <+7220>:	beqz	a0,0xfff6236988 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+9232>
   0x000000fff62361b0 <+7224>:	ld	t9,8088(gp)
   0x000000fff62361b4 <+7228>:	ld	v0,0(a1)
   0x000000fff62361b8 <+7232>:	dsrl32	a3,v0,0x11
   0x000000fff62361bc <+7236>:	dsrl32	a4,a2,0x0
   0x000000fff62361c0 <+7240>:	lui	a0,0x3
   0x000000fff62361c4 <+7244>:	beq	a4,a0,0xfff62369e0 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+9320>
   0x000000fff62361c8 <+7248>:	li	a0,-1
   0x000000fff62361cc <+7252>:	dsll32	a0,a0,0xf
   0x000000fff62361d0 <+7256>:	xor	a0,a2,a0
   0x000000fff62361d4 <+7260>:	sd	a0,24(s8)
   0x000000fff62361d8 <+7264>:	beqz	a3,0xfff6236954 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+9180>
   0x000000fff62361dc <+7268>:	lui	a0,0x3
   0x000000fff62361e0 <+7272>:	dsrl32	a1,v0,0x0
   0x000000fff62361e4 <+7276>:	beq	a1,a0,0xfff62369b8 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+9280>
   0x000000fff62361e8 <+7280>:	li	a0,-1
   0x000000fff62361ec <+7284>:	dsll32	a0,a0,0xf
   0x000000fff62361f0 <+7288>:	xor	v0,v0,a0
   0x000000fff62361f4 <+7292>:	dmtc1	v0,$f0
   0x000000fff62361f8 <+7296>:	ldc1	$f2,24(s8)
   0x000000fff62361fc <+7300>:	li	a0,-1
   0x000000fff6236200 <+7304>:	dsll32	a0,a0,0xf
   0x000000fff6236204 <+7308>:	mul.d	$f2,$f2,$f0
   0x000000fff6236208 <+7312>:	dmfc1	v0,$f2
   0x000000fff623620c <+7316>:	xor	a0,v0,a0
   0x000000fff6236210 <+7320>:	lw	v0,16(s6)
   0x000000fff6236214 <+7324>:	bnez	v0,0xfff62349f0 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+1144>
---Type <return> to continue, or q <return> to quit---
   0x000000fff6236218 <+7328>:	nop
   0x000000fff623621c <+7332>:	lwu	v0,24(s4)
   0x000000fff6236220 <+7336>:	lwu	a1,28(s4)
   0x000000fff6236224 <+7340>:	dsll	v0,v0,0x3
   0x000000fff6236228 <+7344>:	gsldx	v0,0(s5,v0)
   0x000000fff623622c <+7348>:	dsll	a1,a1,0x3
   0x000000fff6236230 <+7352>:	b	0xfff6234bb0 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+1592>
   0x000000fff6236234 <+7356>:	gssdx	a0,0(v0,a1)
   0x000000fff6236238 <+7360>:	lwu	a0,8(s4)
   0x000000fff623623c <+7364>:	lwu	v0,12(s4)
   0x000000fff6236240 <+7368>:	dsll	v1,a0,0x3
   0x000000fff6236244 <+7372>:	gsldx	a5,0(s5,v1)
   0x000000fff6236248 <+7376>:	dsll	v0,v0,0x3
=> 0x000000fff623624c <+7380>:	b	0xfff6234a00 <_ZN3QV44Moth3VME3runEPNS_15ExecutionEngineEPKhPPPv+1160> 
   0x000000fff6236250 <+7384>:	gsldx	v0,0(a5,v0)			0x000000fff623624c 处是跳转指令，根据mips延迟槽的特点故此处是触发非法指令的指令
   0x000000fff6236254 <+7388>:	nop
   0x000000fff6236258 <+7392>:	lwu	v0,8(s4)
   0x000000fff623625c <+7396>:	move	a0,s6
   0x000000fff6236260 <+7400>:	daddiu	a1,s4,16
   0x000000fff6236264 <+7404>:	lwu	s7,12(s4)
   0x000000fff6236268 <+7408>:	ld	t9,-1712(gp)
   0x000000fff623626c <+7412>:	dsll	v0,v0,0x3
   0x000000fff6236270 <+7416>:	dsll	s7,s7,0x3
   0x000000fff6236274 <+7420>:	sd	a1,32(s8)

通过查看寄存器cause的值可以知道cause 寄存器的BD位被置1，说明异常发生在转移指令的延迟槽中。另外通过解码cause寄存器中可知ExcCode值为4. 查表《see mips run》可知异常码4对应的是： （取数、取指或者存数时）地址错误：这要么是在用户态试图存取kuseg以外的空间，或者是试图从未对齐的地址读取一个双字、字或者半字。
寄存器pc 则记录了发生异常时的地址。
badvaddr这个寄存器保存引发异常的地址；*在发生MMU相关的异常时、在用户程序试图访问kuseg以外的地址时、或者地址没有正确对齐时，该寄存器被设置。*此处的值与a5的值相同。 正好与gsldx v0,0(a5,v0) 相对应。
ls2k是mips64架构，并且运行的nk5操作系统是64位系统，故对齐要求为8字节对齐。而此时a5的值为ffd412347c,从而触发非对齐访问。

通过上面的分析找到了产生非法指令的语句是gsldx v0,0(a5,v0),一看gsldx指令并非mips体系的标准指令，而是龙芯自己扩展的非对齐访问指令。由于mips对非齐访问指令拥有专利权，所以龙芯自己实现一套以绕开专利问题。

自定义指令需要在binuntils添加实现或是在内核中做指令模拟支持才能被识别。由于中标自带3.10能正常，而自移植的4.19的有问题，显然问题出在内核。经查发现内核中没有实现gsldx 指令。
之后在linux 源码树[1]上找到了相应的代码。
在这里面龙梦的源码树[2]更是有针对linux 4.19的相关代码，将补丁取下来之后给内核打上，重新编译之后一切正常。

[1] https://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git/commit/?id=f83e4f9896eff614d0f2547a561fa5f39f9cddde
[2] http://dev.lemote.com:8000/cgit/?p=linux-official.git;a=commit;h=09156ac5d92e7cd9729bf7accca1cfd0fc68905b