Android P 图形显示系统（四） Android VirtualDisplay解析

Android VirtualDisplay解析

Android支持多个屏幕：主显，外显，和虚显，虚显就是我们要说的VirtualDisplay。VirtualDisplay的使用场景很多，比如录屏，WFD显示等。其作用就是抓取屏幕上显示的内容。VirtualDisplay抓取屏幕内容，其实现方式有很多。在API中就提供了ImageReader进行读取VirtualDisplay里的内容。

下面我们就结合ImageReader，来看看VirtualDisplay及其相关流程。

ImageReader和VirtualDisplay使用示例

我们以VirtualDisplayTest为示例：

1.在测试setUp时，初始化 DisplayManager, ImageReader 和 ImageListener ，代码如下：

* frameworks/base/core/tests/coretestssrc/android/hardware/display/VirtualDisplayTest.java

    protected void setUp() throws Exception {
        super.setUp();

        mDisplayManager = (DisplayManager)mContext.getSystemService(Context.DISPLAY_SERVICE);
        mHandler = new Handler(Looper.getMainLooper());
        mImageListener = new ImageListener();

        mImageReaderLock.lock();
        try {
            mImageReader = ImageReader.newInstance(WIDTH, HEIGHT, PixelFormat.RGBA_8888, 2);
            mImageReader.setOnImageAvailableListener(mImageListener, mHandler);
            mSurface = mImageReader.getSurface();
        } finally {
            mImageReaderLock.unlock();
        }
    }

• DisplayManager 管理Display的，系统中有对应的DisplayManagerService。
• ImageListener实现OnImageAvailableListener接口。
• ImageReader是一个图片读取器，它是OnImageAvailableListener接口的触发者
• 另外，注意这里的mSurface。

2.以测试项目testPrivateVirtualDisplay为例

    public void testPrivateVirtualDisplay() throws Exception {
        VirtualDisplay virtualDisplay = mDisplayManager.createVirtualDisplay(NAME,
                WIDTH, HEIGHT, DENSITY, mSurface, 0);
        assertNotNull("virtual display must not be null", virtualDisplay);

        Display display = virtualDisplay.getDisplay();
        try {
            assertDisplayRegistered(display, Display.FLAG_PRIVATE);

            // Show a private presentation on the display.
            assertDisplayCanShowPresentation("private presentation window",
                    display, BLUEISH,
                    WindowManager.LayoutParams.TYPE_PRIVATE_PRESENTATION, 0);
        } finally {
            virtualDisplay.release();
        }
        assertDisplayUnregistered(display);
    }

• 测试时，先通过mDisplayManager，创建一个虚拟显示。
• 通过assertDisplayRegistered判断虚显是否已经注册
• 通过assertDisplayCanShowPresentation判断是否能显示私有的Presentation
• 将虚显释放后，通过assertDisplayUnregistered判断是否已经撤销注册。

这里Presentation是Andorid的一个显示控件，能够实现将要显示的内容显示到制定的显示屏上。

实例代码就这么多，接下来，我们来看具体的流程。

ImageReader介绍

ImageReader，简单来说，就是使应用能够以图片数据的形式读取绘制到Surface中的内容。图片数据用Image描述。

1.ImageReader的定义
ImageReader的定义如下：

* frameworks/base/media/java/android/media/ImageReader.java

    public static ImageReader newInstance(int width, int height, int format, int maxImages) {
        return new ImageReader(width, height, format, maxImages, BUFFER_USAGE_UNKNOWN);
    }

这里的参数maxImages表示，能同时访问的Image数量，这里概念上和BufferQueue中的maxnumber也是类似的。

ImageReader关键代码如下：

    protected ImageReader(int width, int height, int format, int maxImages, long usage) {
        mWidth = width;
        mHeight = height;
        mFormat = format;
        mMaxImages = maxImages;

        .. ...

        mNumPlanes = ImageUtils.getNumPlanesForFormat(mFormat);

        nativeInit(new WeakReference<>(this), width, height, format, maxImages, usage);

        mSurface = nativeGetSurface();

        mIsReaderValid = true;
        // Estimate the native buffer allocation size and register it so it gets accounted for
        // during GC. Note that this doesn't include the buffers required by the buffer queue
        // itself and the buffers requested by the producer.
        // Only include memory for 1 buffer, since actually accounting for the memory used is
        // complex, and 1 buffer is enough for the VM to treat the ImageReader as being of some
        // size.
        mEstimatedNativeAllocBytes = ImageUtils.getEstimatedNativeAllocBytes(
                width, height, format, /*buffer count*/ 1);
        VMRuntime.getRuntime().registerNativeAllocation(mEstimatedNativeAllocBytes);
    }

• 我们的格式是PixelFormat.RGBA_8888，所以这里的mNumPlanes值为1
• nativeInit，native方法，创建一个native的ImageReader实例。
• nativeGetSurface，native方法，获取对应的Native实例的Surface，注意，我们的Surface是从哪儿来的。

2.ImageReader的JNI实现
ImageReader的JNI实现如下，这里包含了ImageReader的方法和SurfaceImage的方法。

* frameworks/base/media/jni/android_media_ImageReader.cpp

static const JNINativeMethod gImageReaderMethods[] = {
    {"nativeClassInit",        "()V",                        (void*)ImageReader_classInit },
    {"nativeInit",             "(Ljava/lang/Object;IIIIJ)V",  (void*)ImageReader_init },
    {"nativeClose",            "()V",                        (void*)ImageReader_close },
    {"nativeReleaseImage",     "(Landroid/media/Image;)V",   (void*)ImageReader_imageRelease },
    {"nativeImageSetup",       "(Landroid/media/Image;)I",   (void*)ImageReader_imageSetup },
    {"nativeGetSurface",       "()Landroid/view/Surface;",   (void*)ImageReader_getSurface },
    {"nativeDetachImage",      "(Landroid/media/Image;)I",   (void*)ImageReader_detachImage },
    {"nativeDiscardFreeBuffers", "()V",                      (void*)ImageReader_discardFreeBuffers }
};

static const JNINativeMethod gImageMethods[] = {
    {"nativeCreatePlanes",      "(II)[Landroid/media/ImageReader$SurfaceImage$SurfacePlane;",
                                                              (void*)Image_createSurfacePlanes },
    {"nativeGetWidth",         "()I",                        (void*)Image_getWidth },
    {"nativeGetHeight",        "()I",                        (void*)Image_getHeight },
    {"nativeGetFormat",        "(I)I",                        (void*)Image_getFormat },
};

nativeInit对应的方法为ImageReader_init：

static void ImageReader_init(JNIEnv* env, jobject thiz, jobject weakThiz, jint width, jint height,
                             jint format, jint maxImages, jlong ndkUsage)
{
    ... ...
    sp<JNIImageReaderContext> ctx(new JNIImageReaderContext(env, weakThiz, clazz, maxImages));

    sp<IGraphicBufferProducer> gbProducer;
    sp<IGraphicBufferConsumer> gbConsumer;
    BufferQueue::createBufferQueue(&gbProducer, &gbConsumer);
    sp<BufferItemConsumer> bufferConsumer;
    String8 consumerName = String8::format("ImageReader-%dx%df%xm%d-%d-%d",
            width, height, format, maxImages, getpid(),
            createProcessUniqueId());
    ... ...
    bufferConsumer = new BufferItemConsumer(gbConsumer, consumerUsage, maxImages,
            /*controlledByApp*/true);
    if (bufferConsumer == nullptr) {
        jniThrowExceptionFmt(env, "java/lang/RuntimeException",
                "Failed to allocate native buffer consumer for format 0x%x and usage 0x%x",
                nativeFormat, consumerUsage);
        return;
    }
    ctx->setBufferConsumer(bufferConsumer);
    bufferConsumer->setName(consumerName);

    ctx->setProducer(gbProducer);
    bufferConsumer->setFrameAvailableListener(ctx);
    ImageReader_setNativeContext(env, thiz, ctx);
    ctx->setBufferFormat(nativeFormat);
    ctx->setBufferDataspace(nativeDataspace);
    ctx->setBufferWidth(width);
    ctx->setBufferHeight(height);

    // Set the width/height/format/dataspace to the bufferConsumer.
    res = bufferConsumer->setDefaultBufferSize(width, height);
    if (res != OK) {
        jniThrowExceptionFmt(env, "java/lang/IllegalStateException",
                          "Failed to set buffer consumer default size (%dx%d) for format 0x%x",
                          width, height, nativeFormat);
        return;
    }
    res = bufferConsumer->setDefaultBufferFormat(nativeFormat);
    if (res != OK) {
        jniThrowExceptionFmt(env, "java/lang/IllegalStateException",
                          "Failed to set buffer consumer default format 0x%x", nativeFormat);
    }
    res = bufferConsumer->setDefaultBufferDataSpace(nativeDataspace);
    if (res != OK) {
        jniThrowExceptionFmt(env, "java/lang/IllegalStateException",
                          "Failed to set buffer consumer default dataSpace 0x%x", nativeDataspace);
    }
}

• 创建了一个JNIImageReaderContext实例，这个就是ImageReader的Native对应的对象。

JNIImageReaderContext::JNIImageReaderContext(JNIEnv* env,
        jobject weakThiz, jclass clazz, int maxImages) :
    mWeakThiz(env->NewGlobalRef(weakThiz)),
    mClazz((jclass)env->NewGlobalRef(clazz)),
    mFormat(0),
    mDataSpace(HAL_DATASPACE_UNKNOWN),
    mWidth(-1),
    mHeight(-1) {
    for (int i = 0; i < maxImages; i++) {
        BufferItem* buffer = new BufferItem;
        mBuffers.push_back(buffer);
    }
}

这里的mDataSpace是数据空间，用以描述格式的。native的Buffer用BufferItem描述，在mBuffers中。

• 创建对应的BufferQueue，生产者gbProducer，消费者gbConsumer。
  这里用的还是BufferQueue，Consumer端用BufferItemConsumer进行了封装。还记得我们Androdi正常显示的时候，Consumer是什么吗？没错BufferLayerConsumer，需要注意这其间的差别。BufferItemConsumer中持有gbConsumer对象。

• 创建完BufferQueue后，再设置到 JNIImageReaderContext 中。注意BufferItemConsumer的FrameAvailableListener为JNIImageReaderContext中实现的FrameAvailableListener。

• 最后通过ImageReader_setNativeContext，将native对象和Java的对象关联。

JNIImageReaderContext的类图

VirtualDisplay的创建

通过DisplayManager创建VirtualDisplay。

* frameworks/base/core/java/android/hardware/display/DisplayManager.java

    public VirtualDisplay createVirtualDisplay(@Nullable MediaProjection projection,
            @NonNull String name, int width, int height, int densityDpi, @Nullable Surface surface,
            int flags, @Nullable VirtualDisplay.Callback callback, @Nullable Handler handler,
            @Nullable String uniqueId) {
        return mGlobal.createVirtualDisplay(mContext, projection,
                name, width, height, densityDpi, surface, flags, callback, handler, uniqueId);
    }

DisplayManagerGlobal是一个单例，Android系统中就这么一个。

    public DisplayManager(Context context) {
        mContext = context;
        mGlobal = DisplayManagerGlobal.getInstance();
    }

DisplayManagerGlobal的createVirtualDisplay方法实现如下：

* frameworks/base/core/java/android/hardware/display/DisplayManagerGlobal.java

    public VirtualDisplay createVirtualDisplay(Context context, MediaProjection projection,
            String name, int width, int height, int densityDpi, Surface surface, int flags,
            VirtualDisplay.Callback callback, Handler handler, String uniqueId) {
        ... ...
        int displayId;
        try {
            displayId = mDm.createVirtualDisplay(callbackWrapper, projectionToken,
                    context.getPackageName(), name, width, height, densityDpi, surface, flags,
                    uniqueId);
        } catch (RemoteException ex) {
            throw ex.rethrowFromSystemServer();
        }
        if (displayId < 0) {
            Log.e(TAG, "Could not create virtual display: " + name);
            return null;
        }
        Display display = getRealDisplay(displayId);
        if (display == null) {
            Log.wtf(TAG, "Could not obtain display info for newly created "
                    + "virtual display: " + name);
            try {
                mDm.releaseVirtualDisplay(callbackWrapper);
            } catch (RemoteException ex) {
                throw ex.rethrowFromSystemServer();
            }
            return null;
        }
        return new VirtualDisplay(this, display, callbackWrapper, surface);
    }

mDm是DisplayManagerservice（DMS）的Stub。mDm.