Android APK 方法数过多的问题

随着新功能越来越多，开始遇到一个奇怪的问题：Conversion to Dalvik format failed: Unable to execute dex:method ID not in [0, 0xffff]: 65536

特别是在低版本的手机（2.2 & 2.3）不能安装。查了一些资料后，大部分推荐用Load Dex的方式，开始尝试用DexClassLoader，在运行时加载需要的Class的方式，这样虽然APK中classes.dex的数量是将下来了，APK也能安装运行了，但是后来看到一个文章：https://www.facebook.com/notes/facebook-engineering/under-the-hood-dalvik-patch-for-facebook-for-android/10151345597798920。  Facebook也遇到这个问题，而且DexClassLoader不能解决根本问题。

所以做了一个的测试。利用之前生成二维码的Demo，Demo中点击按钮才通过DexClassLoader加载Class。测试是在项目中添加T1,T2,T3…等Class，每个Class中有1024个方法。最终使APK中的方法数达到65526，比64K的上限少10。而待加载的DEX中方法数是300+。也就是说加载DEX后，当前进程的实际方法总数超过了64K。测试结果是加载后崩溃，确定了文章中说的问题。

下面大概翻译下，问题原因：在Android中有个程序Dexopt，Dexopt使用一个叫做LinearAlloc(https://github.com/android/platform_dalvik/blob/android-2.3.7_r1/vm/LinearAlloc.h#L33)的Buffer来保持你App的所有方法的信息，虽然最新的版本Android使用了8MB或16MB buffer，但是早版本的Froyo和Gingerbread (2.2和 2.3)只用5MB，所有当有大量方式时，超出了buffer size，导致Dexopt崩溃。

这个Dexopt会在App安装的时候检测，所以会导致APK安装失败，而且Dexopt会在App运行时也会保持App实际使用过的方法数，所以DexClassLoader加载的也计入LinearAlloc时，一样会崩溃。

最终解决办法，Facebook是用JNI，用更大的buffer去替换Dalvik VM中的buffer。

以下直接引用原文:

That's when we had the idea of using a JNI extension to replace the existing buffer with a larger one. At first, this idea seemed completely insane. Modifying the internals of the Java class loader is one thing, but modifying the internals of the Dalvik VM while it was running our code is incredibly dangerous. But as we pored over the code, analyzing all the uses of LinearAlloc, we began to realize that it should be safe as long as we did it at the start of our program. All we had to do was find the LinearAllocHdr object, lock it, and replace the buffer.

 

Finding it turned out to be the hard part. Here’s where it’s stored(https://github.com/android/platform_dalvik/blob/android-2.3.7_r1/vm/Globals.h#L519), buried within the DvmGlobals object, over 700 bytes from the start. Searching the entire object would be risky at best, but fortunately, we had an anchor point: the vmList object just a few bytes before. This contained a value that we could compare to the JavaVM pointer available through JNI.

 

The plan was finally coming together: find the proper value for vmList, scan the DvmGlobals object to find a match, jump a few more bytes to the LinearAlloc header, and replace the buffer. So we built the JNI extension, embedded it in our app, started it up, and...we saw the app running on a Gingerbread phone for the first time in weeks.The plan had worked. 

 

But for some reason it failed on the Samsung Galaxy S II...

The most popular Gingerbread phone...

Of all time...

 

It seems that Samsung made a small change to Android that was confusing our code. Other manufacturers might have done the same, so we realized we needed to make our code more robust. 

 

Manual inspection of the GSII revealed that the LinearAlloc buffer was only 4 bytes from where we expected it, so we adjusted our code to look a few bytes to each side if it failed to find the LinearAlloc buffer in the expected location. This required us to parse our process's memory map to ensure we didn't make any invalid memory references (which would crash the app immediately) and also build some strong heuristics to make sure we would recognize the LinearAlloc buffer when we found it. As a last resort, we found a (mostly) safe way to scan the entire process heap to search for the buffer. 

 

Now we had a version of the code that worked on a few popular phones--but we needed more than just a few. So we bundled our code up into a test app that would run the same procedure we were using for the Facebook app, then just display a large green or red box, indicating success or failure. 

 

We used manual testing, DeviceAnywhere, and a test lab that Google let us borrow to run our test app on 70 different phone models, and fortunately, it worked on every single one!

 

We released this code with Facebook for Android 2.0 in December. It's now running on hundreds of different phone models, and we have yet to find one where it doesn't work. The great speed improvements in that release would not have been possible without this crazy hack. And needless to say, without Android’s open platform, we wouldn’t have had the opportunity to ship our best version of the app. There’s a lot of opportunity for building on Android, and we’re excited to keep bringing the Facebook experience to more people and devices.