比特流
NALU
在讲比特流之前先了解下VCL和NAL，HEVC编码分成两个层次，高层处理编码具体细节的被称为VCL（视频编码层）、底层处理比特流的被称为NAL（网络适配层）。预测编码、变换、量化、环路滤波以及熵编码都属于VCL。而处理比特流封装细节的部分则属于NAL。
编码之后的数据要在网络上传输，必须按照一定的格式进行封装成报文，这样的数据报文被称为nal unit简称NALU。一个NALU包含了一个参数集或者一个SS（slice segment）的数据；包含一个参数集或者其他信息的NALU被称为non-VCLU；包含一个SS的压缩数据的NALU被称为VCLU。
HEVC规定一副图像中的VCLU具有相同的时域重要性及与其他图像的时域依赖关系。
NALU可以包含一个SS的压缩数据、vps、sps、pps、补充增强信息（SEI）、也可以为定界、序列结束、比特流结束、填充数据等
NALU的结构如下，它包含NALU头部（2字节）和NALU载荷（称为RBSP，整数字节）。
注意，RBSP和压缩的数据（SODB）是有区别的。SODB通过下面方式转换成RBSP：
1、把下面的字节流进行转换：
0x000000 ——> 0x00000300
0x000001 ——> 0x00000301
0x000002 ——> 0x00000302
0x000003 ——> 0x00000303
因为，0x000001是NALU的起始码，0x000000是NALU的结束码，0x000002是预留码，对0x000003进行替换是为了避免与0x0000030X（0x00000300，0x00000301，0x00000302，0x00000303）冲突。
2、如果转换之后的SODB不足整数个字节，那么在后面填充比特0，直到包含整数个字节。
3、最后可能会加入16个比特的0作为填充比特。
NALU的头部：
1比特的forbidden_zero_bit（固定的0比特）
6比特的nal_unit_type（NALU类型）
6比特的nuh_layer_id（当前应该取0，非0值用于3D视频等）
3比特的nuh_temporal_id_plus1，该值减去1表示NALU所在时域层的标识号temporalId，不能为0，temporalId表示NALU的时域层级，根据temporalId可以确定图像的重要性，配合nal_unit_type就可以实现视频流的时域分级。
Access Unit
HEVC中引入了接入单元（Access Unit，AU）的概念，一个AU表示多个按照顺序（解码顺序）排列的NALU，这些NALU刚好可以解码生成一个图像。
AU可以看作压缩视频比特流的基本单位，即压缩的视频流由多个按顺序排列的AU组成。
一个AU应该包含一幅图像的所有VCLU，还可以包含non-VCLU。一个AU可以从定界NALU、SEI类型的NALU或者第一个SS的NALU开始，可以用最后一个SS的NALU、序列结束NALU或者比特流结束NALU来结束。
一般情况下，一个AU不包含下面类型的NALU：参数集，保留VCL、填充、保留non-VCLU、未明确等。
NALU在网络上的传输
NALU在网上传输的时候分为两种类型：
1、字节流。
NALU生成字节流的过程如下：
（1）在每个NALU前面插入3字节的起始码start_code_prefix_one_3bytes，其值为0x000001
（2）如果NALU的类型为：VPS_NUT,SPS_NUT,PPS_NUT或者解码顺序为一个AU的第一个NALU，则在其起始码前再插入一个zero_byte，值为0x00
（3）在视频首个NALU的起始码（可能包含zero_byte）前插入leading_zero_8bits，值是0x00
（4）根据需要可在每个NALU后增加trailing_zero_8bits，值是0x00，作为填充数据。
2、分组流。使用RTP、RTMP等协议，把NALU直接作为网络分组的有效载荷。
比特流在HEVC中的实现
HEVC中的实现：
1、NALUnit表示一个NALU头部
// NAL单元头部
struct NALUnit
{
// NAL单元的类型
NalUnitType m_nalUnitType; ///< nal_unit_type
UInt m_temporalId; ///< temporal_id
// 保留的六个为0的比特
UInt m_reservedZero6Bits; ///< reserved_zero_6bits
/** construct an NALunit structure with given header values. */
// 构造函数
NALUnit(
NalUnitType nalUnitType,
Int temporalId = 0,
Int reservedZero6Bits = 0)
:m_nalUnitType (nalUnitType)
,m_temporalId (temporalId)
,m_reservedZero6Bits(reservedZero6Bits)
{}
/** default constructor - no initialization; must be perfomed by user */
NALUnit() {}
/** returns true if the NALunit is a slice NALunit */
// 判断是否为条带，即判断这个NAL单元存放的是否为条带数据
Bool isSlice()
{
return m_nalUnitType == NAL_UNIT_CODED_SLICE_TRAIL_R
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_TRAIL_N
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_TSA_R
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_TSA_N
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_STSA_R
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_STSA_N
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_BLA_W_LP
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_BLA_W_RADL
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_BLA_N_LP
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_W_RADL
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_N_LP
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_CRA
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_RADL_N
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_RADL_R
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_RASL_N
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_RASL_R;
}
// 判断这个NAL单元中存放是否为sei（增强信息）
Bool isSei()
{
return m_nalUnitType == NAL_UNIT_PREFIX_SEI
|| m_nalUnitType == NAL_UNIT_SUFFIX_SEI;
}
// 判断是否为vcl
Bool isVcl()
{
return ( (UInt)m_nalUnitType < 32 );
}
};// 用于输出的NAL单元
struct OutputNALUnit : public NALUnit
{
/**
* construct an OutputNALunit structure with given header values and
* storage for a bitstream. Upon construction the NALunit header is
* written to the bitstream.
*/
OutputNALUnit(
NalUnitType nalUnitType,
UInt temporalID = 0,
UInt reserved_zero_6bits = 0)
: NALUnit(nalUnitType, temporalID, reserved_zero_6bits)
, m_Bitstream()
{}
OutputNALUnit& operator=(const NALUnit& src)
{
m_Bitstream.clear();
static_cast<NALUnit*>(this)->operator=(src);
return *this;
}
// 输出比特流，它里面有个vector可以装编码后的数据
TComOutputBitstream m_Bitstream;
};// 一个辅助类，作用是把NALU写到ostringstream（相当于buffer）中
struct NALUnitEBSP : public NALUnit
{
std::ostringstream m_nalUnitData; // 会把NALU的头部连同数据写到这里
/**
* convert the OutputNALUnit #nalu# into EBSP format by writing out
* the NALUnit header, then the rbsp_bytes including any
* emulation_prevention_three_byte symbols.
*/
NALUnitEBSP(OutputNALUnit& nalu);
}; 4、AccessUnit表示了Access Unit，实际就是一个list，list的元素是NALUnitEBSPclass AccessUnit : public std::list<NALUnitEBSP*>
{
public:
~AccessUnit()
{
for (AccessUnit::iterator it = this->begin(); it != this->end(); it++)
{
delete *it;
}
}
};class TComBitIf
{
public:
virtual Void writeAlignOne () {};
virtual Void writeAlignZero () {};
virtual Void write ( UInt uiBits, UInt uiNumberOfBits ) = 0;
virtual Void resetBits () = 0;
virtual UInt getNumberOfWrittenBits() const = 0;
virtual ~TComBitIf() {}
};class TComOutputBitstream : public TComBitIf
{
/**
* FIFO for storage of bytes. Use:
* - fifo.push_back(x) to append words
* - fifo.clear() to empty the FIFO
* - &fifo.front() to get a pointer to the data array.
* NB, this pointer is only valid until the next push_back()/clear()
*/
// 用vector存放数据
std::vector<uint8_t> *m_fifo;
// 已经持有，但是还没有冲刷的比特数
UInt m_num_held_bits; /// number of bits not flushed to bytestream.
// 总的比特数，包括还没有冲刷的
UChar m_held_bits; /// the bits held and not flushed to bytestream.
/// this value is always msb-aligned, bigendian.
public:
// create / destroy
TComOutputBitstream();
~TComOutputBitstream();
// interface for encoding
/**
* append uiNumberOfBits least significant bits of uiBits to
* the current bitstream
*/
// 写入比特，数据存放在uiBits，比特数存放在uiNumberOfBits
Void write ( UInt uiBits, UInt uiNumberOfBits );
/** insert one bits until the bitstream is byte-aligned */
// 写入比特1直到按字节对其
Void writeAlignOne ();
/** insert zero bits until the bitstream is byte-aligned */
// 写入比特0知道按字节对其
Void writeAlignZero ();
/** this function should never be called */
void resetBits() { assert(0); }
// utility functions
/**
* Return a pointer to the start of the byte-stream buffer.
* Pointer is valid until the next write/flush/reset call.
* NB, data is arranged such that subsequent bytes in the
* bytestream are stored in ascending addresses.
*/
Char* getByteStream() const;
/**
* Return the number of valid bytes available from getByteStream()
*/
UInt getByteStreamLength();
/**
* Reset all internal state.
*/
Void clear();
/**
* returns the number of bits that need to be written to
* achieve byte alignment.
*/
Int getNumBitsUntilByteAligned() { return (8 - m_num_held_bits) & 0x7; }
/**
* Return the number of bits that have been written since the last clear()
*/
// 返回已经写入的比特数
UInt getNumberOfWrittenBits() const { return UInt(m_fifo->size()) * 8 + m_num_held_bits; }
void insertAt(const TComOutputBitstream& src, UInt pos);
/**
* Return a reference to the internal fifo
*/
std::vector<uint8_t>& getFIFO() { return *m_fifo; }
// 返回比特数
UChar getHeldBits () { return m_held_bits; }
TComOutputBitstream& operator= (const TComOutputBitstream& src);
/** Return a reference to the internal fifo */
std::vector<uint8_t>& getFIFO() const { return *m_fifo; }
// 添加子数据流
Void addSubstream ( TComOutputBitstream* pcSubstream );
Void writeByteAlignment();
//! returns the number of start code emulations contained in the current buffer
Int countStartCodeEmulations();
};8、 NALUnit（OutputNALUnit）与TComBitIfTComOutputBitstream）之间关系：NALUnit是NALU的头部，而TComBitIf表示有效的编码数据
9、那么NALUnit与TComBitIf（TComOutputBitstream）是怎么样结合起来的，这就要从熵编码器入手了：
（1）先定义一个OutputNALUnit，因为OutputNALUnit包含TComOutputBitstream成员，因此比特流也同时存在了；
OutputNALUnit nalu(NAL_UNIT_VPS);m_pcEntropyCoder->setBitstream(&nalu.m_Bitstream);write(std::ostream& out, OutputNALUnit& nalu)函数，这个函数用来把NALU写到NALUnitEBSP的ostringstream（相当于一个buffer）成员中，最后，NALUnitEBSP被添加到AccessUnit中。
accessUnit.push_back(new NALUnitEBSP(nalu));inline NALUnitEBSP::NALUnitEBSP(OutputNALUnit& nalu)
: NALUnit(nalu)
{
write(m_nalUnitData, nalu);
}// 写一个NAL单元到ostream中
void write(ostream& out, OutputNALUnit& nalu)
{
writeNalUnitHeader(out, nalu);
/* write out rsbp_byte's, inserting any required
* emulation_prevention_three_byte's */
/* 7.4.1 ...
* emulation_prevention_three_byte is a byte equal to 0x03. When an
* emulation_prevention_three_byte is present in the NAL unit, it shall be
* discarded by the decoding process.
* The last byte of the NAL unit shall not be equal to 0x00.
* Within the NAL unit, the following three-byte sequences shall not occur at
* any byte-aligned position:
* - 0x000000
* - 0x000001
* - 0x000002
* Within the NAL unit, any four-byte sequence that starts with 0x000003
* other than the following sequences shall not occur at any byte-aligned
* position:
* - 0x00000300
* - 0x00000301
* - 0x00000302
* - 0x00000303
*/
vector<uint8_t>& rbsp = nalu.m_Bitstream.getFIFO();
if (rbsp.size() == 0)
{
return;
}
for (vector<uint8_t>::iterator it = rbsp.begin(); it != rbsp.end();)
{
/* 1) find the next emulated 00 00 {00,01,02,03}
* 2a) if not found, write all remaining bytes out, stop.
* 2b) otherwise, write all non-emulated bytes out
* 3) insert emulation_prevention_three_byte
*/
vector<uint8_t>::iterator found = it;
do
{
/* NB, end()-1, prevents finding a trailing two byte sequence */
found = search_n(found, rbsp.end()-1, 2, 0);
found++;
/* if not found, found == end, otherwise found = second zero byte */
if (found == rbsp.end())
break;
if (*(++found) <= 3)
break;
} while (true);
it = found;
if (found != rbsp.end())
{
it = rbsp.insert(found, emulation_prevention_three_byte[0]);
}
}
out.write((Char*)&(*rbsp.begin()), rbsp.end() - rbsp.begin());
/* 7.4.1.1
* ... when the last byte of the RBSP data is equal to 0x00 (which can
* only occur when the RBSP ends in a cabac_zero_word), a final byte equal
* to 0x03 is appended to the end of the data.
*/
if (rbsp.back() == 0x00)
{
out.write(emulation_prevention_three_byte, 1);
}
}（4）最后我们得到了一个完整的AccessUnit（包含一个图像的所有的NALU），压缩的数据都在AccessUnit的NALUnitEBSP的ostringstream成员中
（5）最后我们需要把AccessUnit写到网络或者本地磁盘上，需要使用writeAnnexB函数（这个函数被xWriteOutput函数调用），该函数使用字节流的方式，把数据写到网络或者磁盘上。static std::vector<UInt> writeAnnexB(std::ostream& out, const AccessUnit& au)
{
std::vector<UInt> annexBsizes;
for (AccessUnit::const_iterator it = au.begin(); it != au.end(); it++)
{
const NALUnitEBSP& nalu = **it;
UInt size = 0; /* size of annexB unit in bytes */
static const Char start_code_prefix[] = {0,0,0,1};
if (it == au.begin() || nalu.m_nalUnitType == NAL_UNIT_VPS || nalu.m_nalUnitType == NAL_UNIT_SPS || nalu.m_nalUnitType == NAL_UNIT_PPS)
{
/* From AVC, When any of the following conditions are fulfilled, the
* zero_byte syntax element shall be present:
* - the nal_unit_type within the nal_unit() is equal to 7 (sequence
* parameter set) or 8 (picture parameter set),
* - the byte stream NAL unit syntax structure contains the first NAL
* unit of an access unit in decoding order, as specified by subclause
* 7.4.1.2.3.
*/
out.write(start_code_prefix, 4);
size += 4;
}
else
{
out.write(start_code_prefix+1, 3);
size += 3;
}
out << nalu.m_nalUnitData.str();
size += UInt(nalu.m_nalUnitData.str().size());
annexBsizes.push_back(size);
}
return annexBsizes;
}
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
