0x00 为什么需要MultiDex
如果你是一名android开发者,随着app功能复杂度的增加,代码量的增多和库的不断引入,你迟早会在5.0以下的某款设备上遇到:
Conversion to Dalvik format failed: Unable to execute dex: method ID not in [0, 0xffff]: 65536 |
或者
trouble writing output: Too many field references: 131000; max is 65536. You may try using --multi-dex option. |
这说明你的app的main dex方法数已经超过65535
,如果打算继续兼容5.0以下手机,你可以采用google提供的MultiDex方案,但main dex方法数为什么不能超过65535呢?
其实在Dalvik的invoke-kind指令集中,method reference index只留了16bits,最多能调用65535个方法。
invoke-kind {vC, vD, vE, vF, vG}, meth@BBBB B: method reference index (16 bits) |
所以在生成dex文件的过程中,当方法数超过65535就会报错。我们可以在dx工具源码中找到一些线索:
dalvik/dx/src/com/android/dx/merge/IndexMap.java
/** * Maps the index offsets from one dex file to those in another. For example, if * you have string #5 in the old dex file, its position in the new dex file is * {@code strings[5]}. */ public final class IndexMap { private final Dex target; public final int[] stringIds; public final short[] typeIds; public final short[] protoIds; public final short[] fieldIds; public final short[] methodIds; |
可以看到,methodIds,typeIds,protoIds,fieldIds都是short[]类型,对于每个具体的method来说都是限制在16bits。google dalvik开发者在这上面挖了个坑,MultiDex就是来填坑的。
0x01 使用MultiDex
MultiDex出现在google官方提供的support包里面,使用的时候需要在build.gradle中加上依赖:
compile 'com.android.support:multidex:1.0.0'
|
同时让app的Application继承MultiDexApplication
public class MultiDexApplication extends Application { public MultiDexApplication() { } protected void attachBaseContext(Context base) { super.attachBaseContext(base); MultiDex.install(this); } } |
也别忘了在build.gradle中修改:
multiDexEnabled true
|
0x02 MultiDex.install过程
先从MultiDex.install开始分析,传入的context是MultiDexApplication,loader是dalvik.system.PathClassLoader,运行时会先去app的dexDir /data/data/pkgname/code_cache/secondary-dexes下找secondary dex(除了main dex,其他都叫secondary dex,一个apk中可能存在多个secondary dex),找到后先校验zip格式,没问题就直接installSecondaryDexes,否则会去强制从apk中重新extract secondary dex。
MultiDex.java
public static void install(Context context) { ... ClassLoader loader; try { loader = context.getClassLoader(); ... File dexDir = new File(applicationInfo.dataDir, SECONDARY_FOLDER_NAME); List<File> files = MultiDexExtractor.load(context, applicationInfo, dexDir, false); if (checkValidZipFiles(files)) { installSecondaryDexes(loader, dexDir, files); } else { Log.w(TAG, "Files were not valid zip files. Forcing a reload."); // Try again, but this time force a reload of the zip file. files = MultiDexExtractor.load(context, applicationInfo, dexDir, true); if (checkValidZipFiles(files)) { installSecondaryDexes(loader, dexDir, files); } else { // Second time didn't work, give up throw new RuntimeException("Zip files were not valid."); } } |
MultiDexExtractor.load方法中,sourceApk指向的是/data/app/pkgname.apk,然后通过getZipCrc获取apk的CRC校验码,去和最后一次CRC校验码对比,若一致或者不是forceReload,那么直接loadExistingExtractions,loadExistingExtractions直接为/data/data/pkgname/code_cache/secondary-dexes/下已经存在的.dex创建File对象;如果不一致说明apk已经被修改了,dex需要重新从apk中抽取,此时执行performExtractions。
MultiDexExtractor.java
static List<File> load(Context context, ApplicationInfo applicationInfo, File dexDir, boolean forceReload) throws IOException { Log.i(TAG, "MultiDexExtractor.load(" + applicationInfo.sourceDir + ", " + forceReload + ")"); final File sourceApk = new File(applicationInfo.sourceDir); long currentCrc = getZipCrc(sourceApk); List<File> files; if (!forceReload && !isModified(context, sourceApk, currentCrc)) { try { files = loadExistingExtractions(context, sourceApk, dexDir); } catch (IOException ioe) { Log.w(TAG, "Failed to reload existing extracted secondary dex files," + " falling back to fresh extraction", ioe); files = performExtractions(sourceApk, dexDir); putStoredApkInfo(context, getTimeStamp(sourceApk), currentCrc, files.size() + 1); } } else { Log.i(TAG, "Detected that extraction must be performed."); files = performExtractions(sourceApk, dexDir); putStoredApkInfo(context, getTimeStamp(sourceApk), currentCrc, files.size() + 1); } Log.i(TAG, "load found " + files.size() + " secondary dex files"); return files; } |
performExtractions中真正抽取在extract方法中,输入参数分别是:
- apk(ZipFile) 指向/data/app/pkgname.apk
- dexFile(ZipEntry) 指向classes2.dex
- extractTo(File) 指向/data/data/pkgname/code_cache/secondary-dexes/pkgname.apk.classes2.zip
- extractedFilePrefix(String) pkgname.apk.classes
extract把dexFile写入到extractTo指向的一个entry,其中InputStream读的是apk中的classes2.apk,ZipOutputStream指向的是一个tmpFile,具体步骤为:
- 在/data/data/pkgname/code_cache/secondary-dexes/创建pkgname.apk.classes12345.zip的tmpFile;
- 对步骤1的tmpFile建立ZipOutputStream;
- 创建一个指向classes.dex的ZipEntry对象;
- 向tmpFile写入这个entry;
- 将tmpFile重命名为pkgname.apk.classes2.zip,tmpFile此时还存在;
- 删除tmpFile;
extract的实际效果就是在/data/data/pkgname/code_cache/secondary-dexes下创建了:
/data/data/pkgname/code_cache/secondary-dexes/pkgname.apk.classes2.zip
/data/data/pkgname/code_cache/secondary-dexes/pkgname.apk.classes3.zip
…
/data/data/pkgname/code_cache/secondary-dexes/pkgname.apk.classesN.zip
N是MultiDex拆分后dex的个数
private static void extract(ZipFile apk, ZipEntry dexFile, File extractTo, String extractedFilePrefix) throws IOException, FileNotFoundException { InputStream in = apk.getInputStream(dexFile); ZipOutputStream out = null; File tmp = File.createTempFile(extractedFilePrefix, EXTRACTED_SUFFIX, extractTo.getParentFile()); Log.i(TAG, "Extracting " + tmp.getPath()); try { out = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(tmp))); try { ZipEntry classesDex = new ZipEntry("classes.dex"); // keep zip entry time since it is the criteria used by Dalvik classesDex.setTime(dexFile.getTime()); out.putNextEntry(classesDex); byte[] buffer = new byte[BUFFER_SIZE]; int length = in.read(buffer); while (length != -1) { out.write(buffer, 0, length); length = in.read(buffer); } out.closeEntry(); } finally { out.close(); } Log.i(TAG, "Renaming to " + extractTo.getPath()); if (!tmp.renameTo(extractTo)) { throw new IOException("Failed to rename \"" + tmp.getAbsolutePath() + "\" to \"" + extractTo.getAbsolutePath() + "\""); } } finally { closeQuietly(in); tmp.delete(); // return status ignored } } |
最后到真正的install阶段,在MultiDex中有三个私有嵌套类V19,V14和V4来负责具体的系统版本,分别对应android 4.4以上,4.0以上和4.0以下系统。
private static void installSecondaryDexes(ClassLoader loader, File dexDir, List<File> files) throws IllegalArgumentException, IllegalAccessException, NoSuchFieldException, InvocationTargetException, NoSuchMethodException, IOException { if (!files.isEmpty()) { if (Build.VERSION.SDK_INT >= 19) { V19.install(loader, files, dexDir); } else if (Build.VERSION.SDK_INT >= 14) { V14.install(loader, files, dexDir); } else { V4.install(loader, files); } } } |
以4.4为例,V19.install其实就是对输入additionalClassPathEntries反射调用makeDexElements创建Element[]对象,再去修改dalvik.system.BaseDexClassLoader的pathList字段表示的DexPathList类中的dexElements字段内容,把Element[]对象添加进去,这样以后dalvik.system.PathClassLoader就可以找到在secondary dex中的class了。
private static final class V19 { private static void install(ClassLoader loader, List<File> additionalClassPathEntries, File optimizedDirectory) throws IllegalArgumentException, IllegalAccessException, NoSuchFieldException, InvocationTargetException, NoSuchMethodException { /* The patched class loader is expected to be a descendant of * dalvik.system.BaseDexClassLoader. We modify its * dalvik.system.DexPathList pathList field to append additional DEX * file entries. */ Field pathListField = findField(loader, "pathList"); Object dexPathList = pathListField.get(loader); ArrayList<IOException> suppressedExceptions = new ArrayList<IOException>(); expandFieldArray(dexPathList, "dexElements", makeDexElements(dexPathList, new ArrayList<File>(additionalClassPathEntries), optimizedDirectory, suppressedExceptions)); if (suppressedExceptions.size() > 0) { for (IOException e : suppressedExceptions) { Log.w(TAG, "Exception in makeDexElement", e); } Field suppressedExceptionsField = findField(loader, "dexElementsSuppressedExceptions"); IOException[] dexElementsSuppressedExceptions = (IOException[]) suppressedExceptionsField.get(loader); if (dexElementsSuppressedExceptions == null) { dexElementsSuppressedExceptions = suppressedExceptions.toArray( new IOException[suppressedExceptions.size()]); } else { IOException[] combined = new IOException[suppressedExceptions.size() + dexElementsSuppressedExceptions.length]; suppressedExceptions.toArray(combined); System.arraycopy(dexElementsSuppressedExceptions, 0, combined, suppressedExceptions.size(), dexElementsSuppressedExceptions.length); dexElementsSuppressedExceptions = combined; } suppressedExceptionsField.set(loader, dexElementsSuppressedExceptions); } } |
还有一点需要注意的是,DexPathList:makeDexElements最终会去做dex2opt,其中optimizedDirectory就是之前的dexDir,优化后的dex文件是pkgname.apk.classes2.dex,然而dex2opt会消耗较多的cpu时间,如果全部放在main线程去处理的话,比较影响用户体验,甚至可能引起ANR。
DexPathList.java
private static Element[] makeDexElements(ArrayList<File> files, File optimizedDirectory, ArrayList<IOException> suppressedExceptions) { ArrayList<Element> elements = new ArrayList<Element>(); for (File file : files) { File zip = null; DexFile dex = null; String name = file.getName(); if (file.isDirectory()) { } else if (file.isFile()){ if (name.endsWith(DEX_SUFFIX)) { // Raw dex file (not inside a zip/jar). try { dex = loadDexFile(file, optimizedDirectory); } catch (IOException ex) { System.logE("Unable to load dex file: " + file, ex); } } else { ... } } else { System.logW("ClassLoader referenced unknown path: " + file); } if ((zip != null) || (dex != null)) { elements.add(new Element(file, false, zip, dex)); } } return elements.toArray(new Element[elements.size()]); } |
0x03 探明的坑
-
坑1 如果secondary dex文件太大,可能导致应用在安装过程中出现ANR,这个在0x02 MultiDex.install的最后也提到过,规避方法以后将继续介绍。
-
坑2 Dalvik linearAlloc bug (Issue 22586):采用MutilDex方案的app在Android4.0以前的设备上可能会启动失败。
-
坑N-1 Dalvik linearAlloc limit (Issue 78035):使用MultiDex的app需要申请一个大内存,运行时可能导致程序crash,这个Issue在Android4.0已经修复了, 不过还是有可能在低于Android5.0的设备上出现。
-
坑N main dex capacity exceeded,一边愉快地编译apk,一边写着代码,突然出现“main dex capacity exceeded”,build failed了… 这个时候怎么办,一种看似有效的办法是指定dex中的method数,例如:
android.applicationVariants.all { variant -> dex.doFirst{ dex-> if (dex.additionalParameters == null) { dex.additionalParameters = [] } dex.additionalParameters += '--set-max-idx-number=55000' } } |
然并卵,虽然编译没问题了,但是运行时会大概率出现ClassNotFoundException
和NoClassDefFoundError
导致crash,原因很简单,MultiDex.install之前依赖的所有类必须在main dex中,暴力指定main dex数量,可能导致这些类被划分到了secondary dex,系统的PathClassLoader并不能在main dex中找到全部需要加载的类!好在5.0之后,安装app时ART会对apk中所有的classes(..N).dex预编译输出为一个.oat文件,因此找不到类的情况会彻底解决,但是编译时dex过程中main dex capacity exceeded的问题却仍然存在。
一个解决办法是build时指定maindexlist.txt,具体可以参考本博客的另一篇文章MultiDex中出现的main dex capacity exceeded解决之道。