0x00 前言
随着业务的日益壮大,在集成构建实践中发现,dalvik上的MultiDex拆包频繁出现main dex capacity exceeded问题导致编译失败,对app的年末上线构成了严峻挑战。本文通过控制maindexlist中class的数量,达到减少MainDex体积,避免exceeded的目的。
0x01 为什么会main dex capacity exceeded
在入口类com.android.dx.command.dexer.Main
中,
processOne
会去调用processClass
,一旦发现main dex的方法数超65535,会通过createDexFile
创建一个新的Byte[]对象放入dexOutputArrays。processAllFiles
遇到dexOutputArrays.size
> 0就会抛DexException
,告诉我们”Too many classes in maindexlixt, main dex capacity exceeded”。
dalvik/dx/src/com/android/dx/command/dexer/Main.java
processAllFiles
if (args.mainDexListFile != null) { // with --main-dex-list ... // forced in main dex for (int i = 0; i < fileNames.length; i++) { processOne(fileNames[i], mainPassFilter); } if (dexOutputArrays.size() > 0) { throw new DexException("Too many classes in " + Arguments.MAIN_DEX_LIST_OPTION + ", main dex capacity exceeded"); } ... } |
processClass
private static boolean processClass(String name, byte[] bytes) { ... int numMethodIds = outputDex.getMethodIds().items().size(); int numFieldIds = outputDex.getFieldIds().items().size(); int constantPoolSize = cf.getConstantPool().size(); int maxMethodIdsInDex = numMethodIds + constantPoolSize + cf.getMethods().size() + MAX_METHOD_ADDED_DURING_DEX_CREATION; int maxFieldIdsInDex = numFieldIds + constantPoolSize + cf.getFields().size() + MAX_FIELD_ADDED_DURING_DEX_CREATION; if (args.multiDex && (outputDex.getClassDefs().items().size() > 0) && ((maxMethodIdsInDex > args.maxNumberOfIdxPerDex) || (maxFieldIdsInDex > args.maxNumberOfIdxPerDex))) { DexFile completeDex = outputDex; createDexFile(); } |
0x02 怎么避免
怎么才能避免maindex方法数超65535呢? 关键在控制好maindexlist.txt中类的数量。
0x03 maindexlist.txt的生成
android gradle plugin中,有一个类专门负责创建maindexlist.txt,叫做CreateMainDexList
,源码位于:
tools/base/build-system/gradle-core/src/main/groovy/com/android/build/gradle/internal/tasks/multidex/CreateMainDexList.groovy
@TaskAction void output() { if (getAllClassesJarFile() == null) { throw new NullPointerException("No input file") } // manifest components plus immediate dependencies must be in the main dex. File _allClassesJarFile = getAllClassesJarFile() Set<String> mainDexClasses = callDx(_allClassesJarFile, getComponentsJarFile()) |
callDx
最终调用AndroidBuilder.createMainDexList
,实际是通过开启后台进程执行ClassReferenceListBuilder.main
(5.0之前叫MainDexListBuilder
)去分析类的依赖关系,生成一个maindexlist.txt。
public static void main(String[] args) { ... ZipFile jarOfRoots; jarOfRoots = new ZipFile(args[0]); ... Path path = null; try { path = new Path(args[1]); ClassReferenceListBuilder builder = new ClassReferenceListBuilder(path); builder.addRoots(jarOfRoots); printList(builder.toKeep); ... |
为了确定ClassReferenceListBuilder.main
输入参数,用javassist.jar来运行时改写ClassReferenceListBuilder.class文件。在main方法第一行加上System.err.println(java.util.Arrays.toString(args))
,目的是把标准错误输出流打印到终端。另外保险起见,通过PrintWriter同时把参数输出到本地文件log.txt。
修改后发现,gradle build进程的标准错误输出流没打印出来,很可能被重定向了,不过本地log.txt记录下了参数信息:
[[省略...]/release/componentClasses.jar, [省略...]/release/classes.jar]
|
可见arg[0]是componentClasses.jar, arg[1]是app完整的classes.jar。
ClassReferenceListBuilder.addRoots
通过读文件遍历componentClasses.jar的每个entry,再调用addDependencies
分析这个类的依赖关系:
public void addRoots(ZipFile jarOfRoots) throws IOException { ... for (Enumeration<? extends ZipEntry> entries = jarOfRoots.entries(); entries.hasMoreElements();) { ZipEntry entry = entries.nextElement(); String name = entry.getName(); if (name.endsWith(CLASS_EXTENSION)) { DirectClassFile classFile; ... classFile = path.getClass(name); ... addDependencies(classFile.getConstantPool()); } } } |
addDependencies
从ConstantPool得到import类,调用addClassWithHierachy
继续分析继承关系(其实也可以通过javap -verbose先反汇编,再分析匹配”= class”的字符串来获取,具体可以参考本文的依赖分析工具)
private void addDependencies(ConstantPool pool) { for (Constant constant : pool.getEntries()) { if (constant instanceof CstType) { Type type = ((CstType) constant).getClassType(); String descriptor = type.getDescriptor(); if (descriptor.endsWith(";")) { int lastBrace = descriptor.lastIndexOf('['); if (lastBrace < 0) { addClassWithHierachy(descriptor.substring(1, descriptor.length()-1)); } else { assert descriptor.length() > lastBrace + 3 && descriptor.charAt(lastBrace + 1) == 'L'; addClassWithHierachy(descriptor.substring(lastBrace + 2, descriptor.length() - 1)); } } } } } |
依赖关系分析结束后,输出maindexlist.txt,这里标准输出已经重定向到了maindexlist.txt。
private static void printList(Set<String> toKeep) { for (String classDescriptor : toKeep) { System.out.print(classDescriptor); System.out.println(CLASS_EXTENSION); } } |
0x04 componentClasses.jar的来源
经过前面的分析,其实可以看出componentClasses.jar最终决定了maindexlist.txt的大小。
而componentClasses.jar是proguardComponentsTask根据manifest_keep.txt从allclasses.jar中抽取生成的,manifest_keep.txt内容如下:
-keep class com.xxxx.sdk.app.XXXXApplication { <init>(); void attachBaseContext(android.content.Context); } -keep class com.xxxx.sdk.splash.XXXXActivity { <init>(); } -keep class com.xxxx.sdk.app.MainActivity { <init>(); } -keep class com.xxxx.sdk.login.xxxx.LoginActivity { <init>(); } -keep class com.xxxx.sdk.sidebar.account.XXAccountActivity { <init>(); } ... |
那么manifest_keep.txt由谁生成的呢?
其实是CreateManifestKeepList
解析AndroidManifest.xml文件得到的,可以找到:
./tools/base/build-system/gradle-core/src/main/groovy/com/android/build/gradle/internal/tasks/multidex/CreateManifestKeepList.groovy
-keep public class * extends android.app.backup.BackupAgent { <init>(); } -keep public class * extends java.lang.annotation.Annotation { *; } |
@TaskAction void generateKeepListFromManifest() { SAXParser parser = SAXParserFactory.newInstance().newSAXParser() Writer out = new BufferedWriter(new FileWriter(getOutputFile())) try { parser.parse(getManifest(), new ManifestHandler(out)) out.write( """-keep public class * extends android.app.backup.BackupAgent { <init>(); } -keep public class * extends java.lang.annotation.Annotation { *; } """) |
上面getOutputFile
返回的就是manifest_keep.txt,CreateManifestKeepList
私有内部类ManifestHandler
用CreateManifestKeepList.KEEP_SPECS[qName]
决定哪些类需要放入manifest_keep.txt。
private class ManifestHandler extends DefaultHandler { ... @Override void startElement(String uri, String localName, String qName, Attributes attr) { String keepSpec = CreateManifestKeepList.KEEP_SPECS[qName] if (keepSpec) { boolean keepIt = true if (CreateManifestKeepList.this.filter) { Map<String, String> attrMap = [:] for (int i = 0; i < attr.getLength(); i++) { attrMap[attr.getQName(i)] = attr.getValue(i) } keepIt = CreateManifestKeepList.this.filter(qName, attrMap) } if (keepIt) { String nameValue = attr.getValue('android:name') if (nameValue != null) { out.write((String) "-keep class ${nameValue} $keepSpec\n") } |
用来过滤的KEEP_SPECS
:
private static String DEFAULT_KEEP_SPEC = "{ <init>(); }" private static Map<String, String> KEEP_SPECS = [ 'application' : """{ <init>(); void attachBaseContext(android.content.Context); }""", 'activity' : DEFAULT_KEEP_SPEC, 'service' : DEFAULT_KEEP_SPEC, 'receiver' : DEFAULT_KEEP_SPEC, 'provider' : DEFAULT_KEEP_SPEC, 'instrumentation' : DEFAULT_KEEP_SPEC, ] |
可见,至少AndroidManifest.xml中application
,activity
,service
,receiver
,provider
,instrumentation
这6种标签的类,以及继承至java.lang.annotation.Annotation
和android.app.backup.BackupAgent
的类会用来产生maindexlist.txt。
0x05 解决之道
本文绕过gradle build工具make componentClasses.jar的方式,不纠结于如何控制componentClasses.jar,而是直接指定哪些类需要放进maindexlist.txt,从而减少maindexlist类数量,避免main dex capacity exceeded的出现。具体方法如下:
首先从app的com.xxxx.sdk.app.XXXXApplication类出发,分析出MultiDex.install之前的所有必须放入maindex的类(MultiDex包也需并入分析),输出到maindexlist.txt。正确分析依赖关系非常重要,否则运行时一定出现ClassNotFoundException
。XXXXApplication对外依赖则越少越好,甚至可以通过java反射和动态加载特性让其仅依赖android.jar和部分接口类。
另外加载过程中,被加载类的static initializer块里(clinit)用到的类和inner类也会被classloader主动加载,需要确保在maindexlist.txt中,可以使用本人scala写的依赖分析工具来进行分析,得到足够小的maindexlist.txt。
最后,还需在build.gradle中加上:
afterEvaluate { tasks.matching { it.name.startsWith("dex") }.each { dx -> if (dx.additionalParameters == null) { dx.additionalParameters = [] } // optional dx.additionalParameters += "--main-dex-list=$projectDir/maindexlist.txt".toString() dx.additionalParameters += "--minimal-main-dex" } } |
本文中,最后的maindexlist.txt类数量成功由4000减少到1116,maindex体积由7M减少到1.3M,du -sh *.dex输出如下:
从此再也不用担心main dex capacity exceeded了^_-