android 264 hardware encode,Android* Hardware Codec – MediaCodec

Android has a great media library that provides countless possibilities for application development. However, until just recently, the Android OS had no low-level API to directly encode and decode audio/video, which would give developers the ability to create almost anything.

Fortunately, the most recent Jelly Bean release introduced the Android.media.MediaCodec API. This API was designed using the same principles and architecture as OpenMAX* (a well-known standard in the media industry), effectively transitioning from a pure, high-level media player to a low-level encoder/decoder.

From MediaPlayer to MediaCode

Android has included MediaPlayer since API level 1. MediaPlayer provides a simple way to play back audio and video. However, it’s limited by the fact that it offers only three media formats: mp4, 3gpp, and mkv (beginning with Android 4.0). To play unsupported formats, many developers have used the ffmpeg software decoder.

There is an x86 binary version of ffmepg that provides excellent performance. However, including both the x86 binary and an ffmepg ARM* binary results in a large binary image.

As of the release of Android 4.1, MediaCodec provides a reasonable solution for this issue. Since MediaCodec is written using the Java* API, it allows the interface to access low-level system codecs, either hardware codec, or, as with the audio codec, a highly optimized software codec. Use of MediaCodec provides reasonable performance as well as accompanying power savings.

Function Call Flow for MediaCodec

The function call flow is very simple and follows the diagram below. After configuration, the developer uses dequeueInputBuffer to get the hardware buffer ID. After copying the decoding raw buffer in to the input queue, queueInputBuffer is used to make the hardware codec perform the decoding function. The developer uses dequeueOutputBuffer to get the decoding buffer and then uses releaseOutputBuffer to release the hardware buffer, which offers a choice to output to screen or not.

This sample code uses MediaExtractor as the media format parser. Due to the limited media formats supported by Android, most developers use AVFormat (a component of FFMPEG) as the media format parser. The configuration for MediaCodec is different for this function. Developers need to use MediaFormat.createVideoFormat to create a video format (if using MediaExtractor, getTrackFormat is used to get the format). Some decoders (for example, H.264) require additional configuration information. This information can be added with MediaFormat or the refresh frame. For example, H.264 refresh frame is an IDR frame that has included the additional configuration data in its header.

Sample code to create a video format for H.264:

videoformat = MediaFormat.createVideoFormat(“video/avc”, width,height);

videoformat.setByteBuffer("csd-0", extra configure data);

Use OpenGL* with MediaCodec

OpenGL can be used to render the MediaCodec output to screen. It is also useful to add some video effect when doing playback. The key is SurfaceTexture class.

SurfaceTexture is included in API Level 15 (Android 4.0.3). Developers can use it to stream images to a given OpenGL texture. So, if configuring MediaCodec with SurfaceTexture, the output buffer will be rendered to an OpenGL texture but not to the device screen. Thus, video post-processing is required.

Developers need to add frame listener (the function is setOnFrameAvailableListener) to get the image update message and then use updateTexImage to get the newest image. OpenGL supports only the RGB color format, but the output for the video decoder is the YUV color format, so GL_TEXTURE_EXTERNAL_OES (the value is 0x8D65) must be used to support the YUV color format. For example, to change the texture type, use the function GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, textureId).

Please reference https://github.com/MasDennis/Rajawali/wiki/Tutorial-25-Video-Material to get details of how to use SurfaceTexture for MediaPlayer. The same process is used for MediaCodec, so if MediaCodec is configured with the SurfaceTexture, the same result will be achieved.

Support Pipeline on Android 4.3

The new InputSurface and OutputSurface classes are included in Android 4.3. With the help of InputSurface and OutputSurface, a transcoding pipeline can be done with zero-copy, that is to say all decoding video memory will not be copied from the GPU to the CPU. The encoder can get the memory handler directly from the pipeline. InputSurface and OutputSurface will bind with the current OpenGL environment, and OutputSurface can serve as a kind of “packaging” for SurfaceTexture.

The work flow is as follows:

Get Low Latency When Using MediaCodec on Intel® Architecture (IA) platforms

In some conditions, for example screen projection or cloud games, we need low latency and high performance decoding. Using the hardware decoder, we can get good performance, but the side effect is high latency due to the software decoder.

If the encoder type is H.264, MediaCodec may have a delay of several frames (for IA platforms, it is 7 frames, while others are 3-5 frames on main or high profile). If the frame rate is 30 FPS, 7 frames will have a 200-millisecond delay. What causes this delay in the hardware decoder? The reason could be that the main or high profile has a B frame, and if the decoder has no buffer when the B frame references the subsequent buffer, the play will be blocked for a brief time. Intel has considered this condition for the hardware decoder and provides 7 buffers for main or high profile but reduces that to zero buffer for baseline because baseline has no B frame. Other vendors may use 4 buffers for all profiles.

So, on IA-based Android platforms, the solution is to change the encoder to a baseline profile. If the encoder profile cannot be changed, a viable workaround solution is to change the configuration data on the decoder side. Generally, the profile is in the fifth byte, so changing the baseline profile to 66 will produce the lowest latency on IA platforms.

Conclusion

Developers writing Android media apps for IA platforms should consider using the hardware solution described in this paper, as it performs the best on IA platforms. Their customers will get more benefit from the powerful hardware and this, in turn, will build a h3 multimedia software ecosystem for IA-based Android devices.

Notices

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations, and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license.