怎样使用srcML对Java或C++源代码进行tokenization

这个需求我觉得很多做软件分析的朋友都会有，我之前的方法比较傻：https://blog.csdn.net/qysh123/article/details/89810018，Understand有时候会有点问题（特别是在处理C++源码的时候），而且Understand跑的时间也比较长。如果在tokenization的时候顺便进行一些处理，如数字换成“<num>”，函数调用换成抽象的“funccall”等等，这时候用Understand也不是不可以，就是比较笨重，所以想尝试一下srcML。下面以Windows系统为例：

将srcml.exe加入path环境变量中后，例如我有下面这个Example1.java（这里给出一些稍微复杂一点的例子），

  public static long toLong(byte[] a) {
    long x = 0;
    for (int i = 0; i < 8; i++) {
      int j = (7 - i) << 3;
      x |= ((0xFFL << j) & ((long) a[i] << j));
    }
    return x;
  }

那么只需要运行：

srcml.exe Example1.java > test.xml

就可以生成下面这个test.xml文件了：

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<unit xmlns="http://www.srcML.org/srcML/src" revision="0.9.5" language="Java" filename="Example1.java">  <function><specifier>public</specifier> <specifier>static</specifier> <type><name>long</name></type> <name>toLong</name><parameter_list>(<parameter><decl><type><name><name>byte</name><index>[]</index></name></type> <name>a</name></decl></parameter>)</parameter_list> <block>{
    <decl_stmt><decl><type><name>long</name></type> <name>x</name> <init>= <expr><literal type="number">0</literal></expr></init></decl>;</decl_stmt>
    <for>for <control>(<init><decl><type><name>int</name></type> <name>i</name> <init>= <expr><literal type="number">0</literal></expr></init></decl>;</init> <condition><expr><name>i</name> <operator>&lt;</operator> <literal type="number">8</literal></expr>;</condition> <incr><expr><name>i</name><operator>++</operator></expr></incr>)</control> <block>{
      <decl_stmt><decl><type><name>int</name></type> <name>j</name> <init>= <expr><operator>(</operator><literal type="number">7</literal> <operator>-</operator> <name>i</name><operator>)</operator> <operator>&lt;&lt;</operator> <literal type="number">3</literal></expr></init></decl>;</decl_stmt>
      <expr_stmt><expr><name>x</name> <operator>|=</operator> <operator>(</operator><operator>(</operator><literal type="number">0xFFL</literal> <operator>&lt;&lt;</operator> <name>j</name><operator>)</operator> <operator>&amp;</operator> <operator>(</operator><operator>(</operator><name>long</name><operator>)</operator> <name><name>a</name><index>[<expr><name>i</name></expr>]</index></name> <operator>&lt;&lt;</operator> <name>j</name><operator>)</operator><operator>)</operator></expr>;</expr_stmt>
    }</block></for>
    <return>return <expr><name>x</name></expr>;</return>
  }</block></function></unit>

看起来有点复杂，其实也不麻烦。在Python中我们可以将上面输出的xml直接取出来：

import subprocess
from lxml import etree

output=subprocess.run(['srcml.exe', 'Example1.java'], capture_output=True, check=False)
root=etree.fromstring(output.stdout)

for func in root.xpath('//*[local-name() = "function"]'):
    func_name = func.xpath('./*[local-name() = "name"]/text()')[0]
    print(func_name)

匹配的时候用的就是XPath的规则，可以参考我之前博客中的介绍：https://blog.csdn.net/qysh123/article/details/79802250

唯一需要注意的一点是，如果我们用for func in root.xpath('//function'): 这种形式是取不到function的，原因是name space，具体可以参考这里：https://www.cnblogs.com/cxd4321/archive/2007/09/24/903896.html，这里我就不细究了，总之用上面的方法就可以取到这个function了。

然后我们再用一些简单粗暴的方法就可以取到每一个token了：

import subprocess
from lxml import etree

output=subprocess.run(['srcml.exe', 'Example1.java'], capture_output=True, check=False)
root=etree.fromstring(output.stdout)
for func in root.xpath('//*[local-name() = "function"]'):
    func_name = func.xpath('./*[local-name() = "name"]/text()')[0]
    print(func_name)
    content = func.xpath('.//text()')
    content = [str(v).strip() for v in content]
    content = list(filter(None, content))
    print(content)

这里用的匹配规则也很好理解，这样就把原来代码中的token都取出来了。得到的结果是：

['public', 'static', 'long', 'toLong', '(', 'byte', '[]', 'a', ')', '{', 'long', 'x', '=', '0', ';', 'for', '(', 'int', 'i', '=', '0', ';', 'i', '<', '8', ';', 'i', '++', ')', '{', 'int', 'j', '=', '(', '7', '-', 'i', ')', '<<', '3', ';', 'x', '|=', '(', '(', '0xFFL', '<<', 'j', ')', '&', '(', '(', 'long', ')', 'a', '[', 'i', ']', '<<', 'j', ')', ')', ';', '}', 'return', 'x', ';', '}']

如果我们想把0之类的数字换掉呢，我尝试了一下，倒是能匹配到，但是会导致token stream的顺序变化。另外，上面这种方法是没有过滤掉注释的，这个也比较讨厌啊。总体上来讲，srcML的结果比较详细，连Method Call之类的也能识别出来，但是xml还是有点复杂的，我不太会用。

另一方面，Understand的功能比较强大，但是运行起来较慢。下面提供另外一种只针对Java的方法：可以用Javalang这个轻量级的工具：

https://github.com/c2nes/javalang

这个工具的好处是比srcML的结果更简洁：

import javalang

this_file=open("Example1.java",'r')
file_content=this_file.read()
tokens = list(javalang.tokenizer.tokenize(file_content))
for each_token in tokens:
    print(each_token)

就短短几行代码就搞定了。结果如下：

Modifier "public" line 1, position 3
Modifier "static" line 1, position 10
BasicType "long" line 1, position 17
Identifier "toLong" line 1, position 22
Separator "(" line 1, position 28
BasicType "byte" line 1, position 29
Separator "[" line 1, position 33
Separator "]" line 1, position 34
Identifier "a" line 1, position 36
Separator ")" line 1, position 37
Separator "{" line 1, position 39
BasicType "long" line 2, position 5
Identifier "x" line 2, position 10
Operator "=" line 2, position 12
DecimalInteger "0" line 2, position 14
Separator ";" line 2, position 15
Keyword "for" line 3, position 5
Separator "(" line 3, position 9
BasicType "int" line 3, position 10
Identifier "i" line 3, position 14
Operator "=" line 3, position 16
DecimalInteger "0" line 3, position 18
Separator ";" line 3, position 19
Identifier "i" line 3, position 21
Operator "<" line 3, position 23
DecimalInteger "8" line 3, position 25
Separator ";" line 3, position 26
Identifier "i" line 3, position 28
Operator "++" line 3, position 29
Separator ")" line 3, position 31
Separator "{" line 3, position 33
BasicType "int" line 4, position 7
Identifier "j" line 4, position 11
Operator "=" line 4, position 13
Separator "(" line 4, position 15
DecimalInteger "7" line 4, position 16
Operator "-" line 4, position 18
Identifier "i" line 4, position 20
Separator ")" line 4, position 21
Operator "<<" line 4, position 23
DecimalInteger "3" line 4, position 26
Separator ";" line 4, position 27
Identifier "x" line 5, position 7
Operator "|=" line 5, position 9
Separator "(" line 5, position 12
Separator "(" line 5, position 13
HexInteger "0xFFL" line 5, position 14
Operator "<<" line 5, position 20
Identifier "j" line 5, position 23
Separator ")" line 5, position 24
Operator "&" line 5, position 26
Separator "(" line 5, position 28
Separator "(" line 5, position 29
BasicType "long" line 5, position 30
Separator ")" line 5, position 34
Identifier "a" line 5, position 36
Separator "[" line 5, position 37
Identifier "i" line 5, position 38
Separator "]" line 5, position 39
Operator "<<" line 5, position 41
Identifier "j" line 5, position 44
Separator ")" line 5, position 45
Separator ")" line 5, position 46
Separator ";" line 5, position 47
Separator "}" line 6, position 5
Keyword "return" line 7, position 5
Identifier "x" line 7, position 12
Separator ";" line 7, position 13
Separator "}" line 8, position 3

缺点是没有srcML功能强大，sigh，真的还是需要一个即功能强大，又简单易用的tokenizer啊。

就简单总结这么多。