How to initialize MediaFormat to configure a MediaCodec to decode raw AAC data?

Question

I have a strange issue with my StreamPlayer and I need any help I can get.

The main goal I need to achieve is StreamPlayer which is able to play back MPEG-2 Transport Streams with smallest possible latency. For this I am following this approach:

The stream is parsed by a Java based TS Parser. I have implemented a TSExtractor which is similar to the MediaExtractor and which works fine. I can receive all the media samples for a selected track the same way it is possible using the MediaExtractor with

extractor.readSampleData(...);
extractor.advance();

To decode the AAC data I want to create and configure an instance of MediaCodec. Using the MediaExtractor class this is usually done by

MediaFormat mediaFormat = extractor.getTrackFormat(i);
decoder = MediaCodec.createDecoderByType(mediaFormat.getString(MediaFormat.KEY_MIME));
decoder.configure(mediaFormat, null, null, 0);

As I have to initialize the MediaFormat in the TSExtractor.getTrackFormat(int track) method I use

MediaFormat mf = MediaFormat.createAudioFormat ("audio/mp4a-latm", getSampleRate(), getChannelCount());

and because all my AAC samples include an ADTS I do

mediaFormat.setInteger(MediaFormat.KEY_IS_ADTS, 1);

After reading this post I finally add an ESDS frame using the "csd-0" key

mediaFormat.setByteBuffer("csd-0", ByteBuffer.allocate(2).put(new byte[]{(byte) 0x11, (byte)0x90}));

where the values 0x11 and 0x90 are extracted from the ADTS.

When I now want to decode the AAC samples the decoder posts

AAC decoder returned error 4097, substituting silence

to the Log.

To verify that my TSExtractor extracts the samples correctly I recorded the same stream using VLC remuxing it to an mp4 file without transcoding so the raw stream is unchanged. Now I can initialize the MediaExtractor with the recorded mp4 file and compare the samples created by my TSExtractor and the MediaExtractor. Using trail and error I found a very strange behavior:

When I configure the MediaCodec using the mediaFormat created by the MediaExtractor the MediaCodec decodes the AAC samples returned by my TSExtractor without any problems. Comparing the MediaFormat, which basically wraps a HashMap, created by my TSExtractor and the one created by the MediaExtractor gives this differences:

Created by MediaExtractor:

mediaFormat: {max-input-size=1212, durationUs=77428875, is-adts=1, channel-count=2, mime=audio/mp4a-latm, csd-0=java.nio.ByteArrayBuffer[position=0,limit=2,capacity=2], sample-rate=48000}

Created by TSExtractor:

mediaFormat: {is-adts=1, channel-count=2, mime=audio/mp4a-latm, csd-0=java.nio.ByteArrayBuffer[position=2,limit=2,capacity=2], sample-rate=48000}

Even when I adopt the MediaFormat created by the TSExtractor to be similar to the one created by the MediaExtractor the decoder gives the same error using the self created and decodes without any problems using the other one.

Any help would be really helpful.

Awesome work...could you provide me with some pointers on how you created the TS parser in java — anz, Nov 13 '13 at 9:33

twetjen · Answer 1 · 2013-10-16 13:20:46Z

I really don't know why, but it turns out that initializing the "csd-0" ByteBuffer this way

mediaFormat.setByteBuffer("csd-0", ByteBuffer.allocate(2).put(new byte[]{(byte) 0x11, (byte)0x90}));

doesn't work, but initializing it this way

byte[] bytes = new byte[]{(byte) 0x11, (byte)0x90};
ByteBuffer bb = ByteBuffer.wrap(bytes);
mediaFormat.setByteBuffer("csd-0", bb);

does.

BTW, comparing these two byteBuffers using

bb1.equals(bb2);

returns true.

Very strange!

user2241968 · Answer 2 · 2013-11-24 00:54:02Z

In the failing case u may need to call ByteBuffer's rewind method first. If u look carefully u'll see the position is different between the MediaExtractor and the TSExtractor:

csd-0=java.nio.ByteArrayBuffer[position=0,limit=2,capacity=2]

vs

csd-0=java.nio.ByteArrayBuffer[position=2,limit=2,capacity=2]

ByteBuffer's equals only compares the bytes after position until a mismatch; in ur case one buffer is already positioned at the end hence no mismatch.

Nik · Answer 3 · 2015-08-22 08:42:34Z

Values in csd-0 depends on ADTS header.

ADTS header length is up to 9 bytes. To generate csd-0 you need the second and the third byte of header.

int profile = (header[2] & 0xC0) >> 6;
int srate = (header[2] & 0x3C) >> 2;
int channel = ((header[2] & 0x01) << 2) | ((header.[3] & 0xC0) >> 6)

ByteBuffer csd = ByteBuffer.allocate(2);
csd.put(0, (byte)(profile << 4 | srate >> 1));
csd.put(1, (byte)((srate & 0x01) << 7 | channel << 3));

Now you got valid csd-0 for this aac audio stream.

Sojan P R · Answer 4 · 2016-03-29 08:02:09Z

Thanks for the above code for calculating CSD. Unfortunately this was not working for me. My decoder was failing with the above csd setting. Finally I found the issue. According to the documentation first "5 bits " of CSD is the object type ( Profile). Above code profile is added to only 4 bits. So changing the code as below works fine for me

  int profile = (header[2] & 0xC0) >> 6;
  int srate = (header[2] & 0x3C) >> 2;
  int channel = ((header[2] & 0x01) << 2) | ((header.[3] & 0xC0) >> 6)

  ByteBuffer csd = ByteBuffer.allocate(2);
  csd.put(0, (byte)(profile << 3 | srate >> 1));
  csd.put(1, (byte)((srate & 0x01) << 7 | channel << 3));

How to initialize MediaFormat to configure a MediaCodec to decode raw AAC data?

4 Answers