前言
前面两篇文章介绍了图像生产者和图像消费者,终于到了最后一篇——图像缓冲区,这三者一起构成了一个完整的Android图像显示系统。Android中的图像生产者OpenGL,Skia,Vulkan将绘制的数据存放在图像缓冲区中,Android中的图像消费SurfaceFlinger从图像缓冲区将数据取出,进行加工及合成。那么图像缓冲区是什么呢?它是如何创建出来的呢?又要如何使用它呢?它的存储原理是什么呢?读完这篇文章,你就能回答这些问题了。
图像缓冲区
在讲解图像的生产者时,多次提到了Surface,我们知道Surface可以存储用来绘制的图形数据。在硬件加速中,需要调用 ThreadedRenderer.initialize(mSurface) 函数将Surface缓冲区传递到OpenGL或者Vulkan的渲染管线;在软件绘制中调用 drawSoftware(surface, mAttachInfo, xOffset, yOffset, scalingRequired, dirty) 函数将Surface缓冲区传递到native层的SkiaCanvas。当OpenGL,Vulkan或Skia拥有Surface后,就可以获取到图形缓冲区,将需要绘制的内容存储在图形缓冲区内了。
同时,在讲解图像的消费者时,也多次提到了Layer,我们知道通过Layer可以取出缓冲区的图形数据。在SurfaceFlinger需要消费图像数据,进行图层混合时,需要遍历Layer,取出Layer中缓冲区的图形数据后,再加过或合成处理。
我们也知道一个Layer唯一对应了一个Surface,他们的关系如下。
所以,Surface和Layer都属于图形缓冲区的组成部分。那么Android图形缓冲区由哪些部分组成呢?Surface和Layer又是什么呢?他们是怎么关联起来的呢?他们又分别是如何创建、获取以及使用缓冲区的呢?这篇文章主要的内容都是围绕这四个问题来讲解,在深入讲解这四个问题之前,这里我们先高屋建瓴的了解这个四个问题的答案,这样在接下面的篇幅中,才不会迷失在冗长的代码中。
1.Android图形缓冲区由哪些部分组成呢?
- Android的图形缓冲区由Surface,BufferQueue,Layer,GraphicBuffer四部分组成。BufferQueue中的slots数组最多能存储64个GraphicBuffer,而GraphicBuffer就是真正被分配内存,并能存储图形数据的缓冲区
2.Surface和Layer又是什么呢?
- Surface是提供给图形生产者控制缓冲区的类,Surface持有GraphicBufferProducer,简称gbp,gbp专门用来创建或获取可用的GraphicBuffer以及提交绘制后的GraphicBuffer
- Layer是提供给图像消费者获取缓冲区的类,Layer持有GraphicBufferConsumer,简称gbc,通过gbc用来获取和释放GraphicBuffer
3.他们是怎么关联起来的呢?
- Surface和Layer通过BufferQueue关联起来,Surface持有BufferQueue中的gbp,Layer持有BufferQueue中的gbc,gbp和gbc的GraphicBuffer都存储在GraphicBufferCore的slots数组中。
4.如何创建、获取以及使用缓冲区呢?
- Surface通过调用gbp的dequeue函数获取GraphicBuffer,调用queue函数提交使用完毕的GraphicBuffer。
- Layer通过调用gbc的acquire函数获取有数据的GraphicBuffer,调用release释放GraphicBuffer
总结一下上面提到的知识:当我们想要绘制图像时,需要创建Surface和Layer,图像生产者如Skia,OpenGL通过Surface调用dequeue函数获取一块缓冲区GraphicBuffer,有了这块缓冲区,就可以在缓冲区上绘制图像了,当绘制完毕后,通过Surface调用queue函数,将GraphicBuffer归还到BufferQueue。之后,Vsync通知图像消费者SurfaceFlinger调用Layer的acquire函数,获取绘制好内容的GraphicBuffer进行合成与处理,处理完毕后通过release函数释放这块缓冲区。
了解了大概的背景和流程,我们接着开始对图像缓冲区的深入学习。
缓冲区的创建
我们已经知道缓冲区由Surface,Layer,BufferQueue和GraphicBuffer组成。这一节主要讲这四个部分是如何创建的,先从Surface和Layer的创建开始。
Surface和Layer的创建
为了更容易的理解如何创建Surface和Layer,这里先从开机动画这个案例讲起。
开机动画创建Surface和Layer
在前面图像生产者中讲如何通过OpenGL ES播放开启动画时,已经提到了创建Surface的流程,它的流程很简单,这里再回顾一下通过Android开机动画的启动流程。开机动画的对象为BootAnimation,它的构造函数如下。
/frameworks/base/cmds/bootanimation/BootAnimation.cpp
BootAnimation::BootAnimation() : Thread(false), mClockEnabled(true), mTimeIsAccurate(false),
mTimeFormat12Hour(false), mTimeCheckThread(NULL) {
//创建SurfaceComposerClient
mSession = new SurfaceComposerClient();
……
}
可以看到构造函数里面创建了SurfaceComposerClient。
我们接着看BootAnimation的初始化函数——readyToRun函数
/frameworks/base/cmds/bootanimation/BootAnimation.cpp
status_t BootAnimation::readyToRun() {
mAssets.addDefaultAssets();
sp<IBinder> dtoken(SurfaceComposerClient::getBuiltInDisplay(
ISurfaceComposer::eDisplayIdMain));
DisplayInfo dinfo;
//获取屏幕信息
status_t status = SurfaceComposerClient::getDisplayInfo(dtoken, &dinfo);
if (status)
return -1;
// 通知SurfaceFlinger创建Surface,创建成功会返回一个SurfaceControl代理
sp<SurfaceControl> control = session()->createSurface(String8("BootAnimation"),
dinfo.w, dinfo.h, PIXEL_FORMAT_RGB_565);
SurfaceComposerClient::openGlobalTransaction();
//设置这个layer在SurfaceFlinger中的层级顺序
control->setLayer(0x40000000);
//获取surface
sp<Surface> s = control->getSurface();
// 以下是EGL的初始化流程
const EGLint attribs[] = {
EGL_RED_SIZE, 8,
EGL_GREEN_SIZE, 8,
EGL_BLUE_SIZE, 8,
EGL_DEPTH_SIZE, 0,
EGL_NONE
};
EGLint w, h;
EGLint numConfigs;
EGLConfig config;
EGLSurface surface;
EGLContext context;
//步骤1:获取Display
EGLDisplay display = eglGetDisplay(EGL_DEFAULT_DISPLAY);
//步骤2:初始化EGL
eglInitialize(display, 0, 0);
//步骤3:选择参数
eglChooseConfig(display, attribs, &config, 1, &numConfigs);
//步骤4:传入SurfaceFlinger生成的surface,并以此构造EGLSurface
surface = eglCreateWindowSurface(display, config, s.get(), NULL);
//步骤5:构造egl上下文
context = eglCreateContext(display, config, NULL, NULL);
//步骤6:绑定EGL上下文
if (eglMakeCurrent(display, surface, surface, context) == EGL_FALSE)
return NO_INIT;
……
}
通过BootAnimation的构造函数和readyToRun函数可以看到,创建Surface的步骤如下:
-
创建SurfaceComponentClient
-
通过SurfaceComponentClient的createSurface函数创建SurfaceControl,SurfaceControl是Surface的控制类
-
有了SurfaceControl之后,我们就可以并能通过getSurface获取到Surface
只需要三步,Surface的创建就完成了,非常的简单。但是不我们不能止步于如何创建Surface,我们还需要了解SurfaceComponentClient什么?createSurface经历了哪些流程?getSurface又经历了哪些流程?
SurfaceComponentClient
我们先看看SurfaceComponentClient对象,它的构造函数如下。
/frameworks/native/libs/gui/SurfaceComposerClient.cpp
ComposerService::ComposerService()
: Singleton<ComposerService>() {
connectLocked();
}
void ComposerService::connectLocked() {
const String16 name("SurfaceFlinger");
//获取SurfaceFlinger
while (getService(name, &mComposerService) != NO_ERROR) {
usleep(250000);
}
//注册binder death的通知
……
}
从SurfaceComposerClient可以看到,它在connectLocked函数中获取了SurfaceFlinger的客户端Binder代理mComposerService。
接着在看看它初始化函数onFirstRef
/frameworks/native/libs/gui/SurfaceComposerClient.cpp
void SurfaceComposerClient::onFirstRef() {
sp<ISurfaceComposer> sm(ComposerService::getComposerService());
if (sm != 0) {
auto rootProducer = mParent.promote();
sp<ISurfaceComposerClient> conn;
conn = (rootProducer != nullptr) ? sm->createScopedConnection(rootProducer) :
sm->createConnection();
if (conn != 0) {
mClient = conn;
mStatus = NO_ERROR;
}
}
}
onFirstRef函数中通过mComposerService调用createConnection方法。mComposerService就是SurfaceFlinger的binder代理,所以这里会最终调用SurfaceFlinger的createConnection函数。
接着看SurfaceFlinger的createConnection函数
/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp
sp<ISurfaceComposerClient> SurfaceFlinger::createConnection() {
return initClient(new Client(this));
}
static sp<ISurfaceComposerClient> initClient(const sp<Client>& client) {
status_t err = client->initCheck();
if (err == NO_ERROR) {
return client;
}
return nullptr;
}
可以看到createConnection方法创建了Client,这个Client封装了对Layer和Surface的操作,我们看一下Client的头文件
/frameworks/native/services/surfaceflinger/Client.h
class Client : public BnSurfaceComposerClient
{
public:
explicit Client(const sp<SurfaceFlinger>& flinger);
Client(const sp<SurfaceFlinger>& flinger, const sp<Layer>& parentLayer);
~Client();
status_t initCheck() const;
// protected by SurfaceFlinger::mStateLock
void attachLayer(const sp<IBinder>& handle, const sp<Layer>& layer);
void detachLayer(const Layer* layer);
sp<Layer> getLayerUser(const sp<IBinder>& handle) const;
void setParentLayer(const sp<Layer>& parentLayer);
private:
// ISurfaceComposerClient interface
virtual status_t createSurface(
const String8& name,
uint32_t w, uint32_t h,PixelFormat format, uint32_t flags,
const sp<IBinder>& parent, uint32_t windowType, uint32_t ownerUid,
sp<IBinder>* handle,
sp<IGraphicBufferProducer>* gbp);
virtual status_t destroySurface(const sp<IBinder>& handle);
virtual status_t clearLayerFrameStats(const sp<IBinder>& handle) const;
virtual status_t getLayerFrameStats(const sp<IBinder>& handle, FrameStats* outStats) const;
virtual status_t onTransact(
uint32_t code, const Parcel& data, Parcel* reply, uint32_t flags);
sp<Layer> getParentLayer(bool* outParentDied = nullptr) const;
// constant
sp<SurfaceFlinger> mFlinger;
// protected by mLock
DefaultKeyedVector< wp<IBinder>, wp<Layer> > mLayers;
wp<Layer> mParentLayer;
// thread-safe
mutable Mutex mLock;
};
}
可以看到,Client继承自BnSurfaceComposerClient,并提供了创建和销毁Layer和Surface的操作函数。
createSurface
创建好了SurfaceComponentClinet,并有了一个在SurfaceFlinger对应的Clinet。我们可以接着看第二步:createSurface函数。
/frameworks/native/libs/gui/SurfaceComposerClient.cpp
sp<SurfaceControl> SurfaceComposerClient::createSurface(
const String8& name,
uint32_t w,
uint32_t h,
PixelFormat format,
uint32_t flags,
SurfaceControl* parent,
uint32_t windowType,
uint32_t ownerUid)
{
sp<SurfaceControl> sur;
if (mStatus == NO_ERROR) {
sp<IBinder> handle;
sp<IBinder> parentHandle;
sp<IGraphicBufferProducer> gbp;
if (parent != nullptr) {
parentHandle = parent->getHandle();
}
status_t err = mClient->createSurface(name, w, h, format, flags, parentHandle,
windowType, ownerUid, &handle, &gbp);
if (err == NO_ERROR) {
sur = new SurfaceControl(this, handle, gbp);
}
}
return sur;
}
createSurface函数主要做了两件事情
- 调用Client的createSurface方法
- 创建SurfaceControl
先看第一件事情:调用Client的createSurface函数
/frameworks/native/services/surfaceflinger/Client.cpp
status_t Client::createSurface(
const String8& name,
uint32_t w, uint32_t h, PixelFormat format, uint32_t flags,
const sp<IBinder>& parentHandle, uint32_t windowType, uint32_t ownerUid,
sp<IBinder>* handle,
sp<IGraphicBufferProducer>* gbp)
{
……
class MessageCreateLayer : public MessageBase {
SurfaceFlinger* flinger;
Client* client;
sp<IBinder>* handle;
sp<IGraphicBufferProducer>* gbp;
status_t result;
const String8& name;
uint32_t w, h;
PixelFormat format;
uint32_t flags;
sp<Layer>* parent;
uint32_t windowType;
uint32_t ownerUid;
public:
MessageCreateLayer(SurfaceFlinger* flinger,
const String8& name, Client* client,
uint32_t w, uint32_t h, PixelFormat format, uint32_t flags,
sp<IBinder>* handle, uint32_t windowType, uint32_t ownerUid,
sp<IGraphicBufferProducer>* gbp,
sp<Layer>* parent)
: flinger(flinger), client(client),
handle(handle), gbp(gbp), result(NO_ERROR),
name(name), w(w), h(h), format(format), flags(flags),
parent(parent), windowType(windowType), ownerUid(ownerUid) {
}
status_t getResult() const {
return result; }
virtual bool handler() {
result = flinger->createLayer(name, client, w, h, format, flags,
windowType, ownerUid, handle, gbp, parent);
return true;
}
};
sp<MessageBase> msg = new MessageCreateLayer(mFlinger.get(),
name, this, w, h, format, flags, handle,
windowType, ownerUid, gbp, &parent);
mFlinger->postMessageSync(msg);
return static_cast<MessageCreateLayer*>( msg.get() )->getResult();
}
可以看到,createSurface实际是通过SurfaceFlinger的消息队列调用createLayer函数,接着看SurfaceFlinger中createLayer函数的实现。
/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp
status_t SurfaceFlinger::createLayer(
const String8& name,
const sp<Client>& client,
uint32_t w, uint32_t h, PixelFormat format, uint32_t flags,
uint32_t windowType, uint32_t ownerUid, sp<IBinder>* handle,
sp<IGraphicBufferProducer>* gbp, sp<Layer>* parent)
{
status_t result = NO_ERROR;
sp<Layer> layer;
String8 uniqueName = getUniqueLayerName(name);
switch (flags & ISurfaceComposerClient::eFXSurfaceMask) {
case ISurfaceComposerClient::eFXSurfaceNormal:
//创建普通的Layer
result = createNormalLayer(client,
uniqueName, w, h, flags, format,
handle, gbp, &layer);
break;
case ISurfaceComposerClient::eFXSurfaceDim:
//创建有遮罩效果的Layer
result = createDimLayer(client,
uniqueName, w, h, flags,
handle, gbp, &layer);
break;
default:
result = BAD_VALUE;
break;
}
if (result != NO_ERROR) {
return result;
}
layer->setInfo(windowType, ownerUid);
result = addClientLayer(client, *handle, *gbp, layer, *parent);
if (result != NO_ERROR) {
return result;
}
mInterceptor.saveSurfaceCreation(layer);
setTransactionFlags(eTransactionNeeded);
return result;
}
status_t SurfaceFlinger::createNormalLayer(const sp<Client>& client,
const String8& name, uint32_t w, uint32_t h, uint32_t flags, PixelFormat& format,
sp<IBinder>* handle, sp<IGraphicBufferProducer>* gbp, sp<Layer>* outLayer)
{
// initialize the surfaces
switch (format) {
case PIXEL_FORMAT_TRANSPARENT:
case PIXEL_FORMAT_TRANSLUCENT:
format = PIXEL_FORMAT_RGBA_8888;
break;
case PIXEL_FORMAT_OPAQUE:
format = PIXEL_FORMAT_RGBX_8888;
break;
}
*outLayer = new Layer(this, client, name, w, h, flags);
status_t err = (*outLayer)->setBuffers(w, h, format, flags);
if (err == NO_ERROR) {
*handle = (*outLayer)->getHandle();
*gbp = (*outLayer)->getProducer();
}
ALOGE_IF(err, "createNormalLayer() failed (%s)", strerror(-err));
return err;
}
到这里我们可以发现,原来createSurface函数实际上并不是创建Surface,而是创建了Layer。接着看Layer的初始化函数onFirstRef
/frameworks/native/services/surfaceflinger/Layer.cpp
void Layer::onFirstRef() {
sp<IGraphicBufferProducer> producer;
sp<IGraphicBufferConsumer> consumer;
//创建BufferQueue
BufferQueue::createBufferQueue(&producer, &consumer, true);
mProducer = new MonitoredProducer(producer, mFlinger, this);
mSurfaceFlingerConsumer = new SurfaceFlingerConsumer(consumer, mTextureName, this);
mSurfaceFlingerConsumer->setConsumerUsageBits(getEffectiveUsage(0));
mSurfaceFlingerConsumer->setContentsChangedListener(this);
mSurfaceFlingerConsumer->setName(mName);
if (mFlinger->isLayerTripleBufferingDisabled()) {
mProducer->setMaxDequeuedBufferCount(2);
}
const sp<const DisplayDevice> hw(mFlinger->getDefaultDisplayDevice());
updateTransformHint(hw);
}
可以看到,Layer的初始化函数里通过BufferQueue::createBufferQueue创建了BufferQueue,而BufferQueue又会创建GraphicBufferProducer和GraphicBufferConsumer,关于BufferQueue后面在详讲。
在看第二件事情:创建SurfaceControl
/frameworks/native/libs/gui/SurfaceControl.cpp
SurfaceControl::SurfaceControl(
const sp<SurfaceComposerClient>& client,
const sp<IBinder>& handle,
const sp<IGraphicBufferProducer>& gbp)
: mClient(client), mHandle(handle), mGraphicBufferProducer(gbp)
{
}
SurfaceControl的构造函数没有任何操作,仅仅只是传入Client,以及在BufferQueue中创建的GraphicBufferProducer。
getSurface
Layer和SurfaceControl都创建好了,就差最后一步了:调用getSurface函数获取Surface。
/frameworks/native/libs/gui/SurfaceControl.cpp
sp<Surface> SurfaceControl::getSurface() const
{
Mutex::Autolock _l(mLock);
if (mSurfaceData == 0)<