H.264 JM的学习笔记

原文转自：http://www.360doc.cn/article/1412027_27456278.html

Useful Links:

Rate Control and H.264:     http://www.pixeltools.com/rate_control_paper.html

JM online documents:

encoder:     http://iphome.hhi.de/suehring/tml/doc/lenc/html/index.html

decoder:     http://iphome.hhi.de/suehring/tml/doc/ldec/html/index.html

T264:    H.264网络传输中马赛克问题的解决

video sequence:     http://yufeng1684.bokee.com/6760108.html  or  转载

back to top

JM14.0:

|| Reading Notes || Actions || Q&A || My understanding || Useful Links ||

Reading Notes:

-------------------------encoder-----------------------------------------

encoder architecture

encoding:

main()-->encode_one_frame()-->frame_picture()--code_a_picture(frame)-->encode_one_slice()-->start_macroblock()/encode_one_macroblock()

void (*encode_one_macroblock) (Macroblock *currMB)==>void encode_one_macroblock_high (Macroblock *currMB);

compute_mode_RD_cost(mode, currMB, enc_mb, &min_rdcost, &min_dcost, &min_rate, i16mode, bslice, &inter_skip);

packetization:

main()-->encode_one_frame()-->writeout_picture()-->writeUnit()

save decoded picture buffer (DPB):

main()-->encode_one_frame()-->store_picture_in_dpb(enc_frame_picture[0])-->insert_picture_in_dpb(dpb.fs[dpb.used_size],p)

in dump_dpb(), we may set DUMP_DPB = 1 to get debug information!

write test_rec.yuv:

main()-->flush_dpb()-->output_one_frame_from_dpb()-->write_stored_frame(dpb.fs[pos], p_dec)-->write_picture(fs->frame, p_out, FRAME)-->write_out_picture(p, p_out)

-->img2buf (p->imgY, buf, p->size_x, p->size_y, symbol_size_in_bytes, crop_left, crop_right, crop_top, crop_bottom)

-->write(p_out, buf, (p->size_y-crop_bottom-crop_top)*(p->size_x-crop_right-crop_left)*symbol_size_in_bytes)

在JM14.0中有两个结构体比较重要：ImageParameters和StorablePicture

在global.h中定义： ImageParameters用于保存程序运算过程的图像参数

1）imgpel mpr [MAX_PLANE][16][16];    用于保存预测的像素值

2）int m7 [MAX_PLANE][16][16];    用于保存residue data的处理过程临时数据（注释写的有问题“the diff pixel values between the original macroblock/block and its prediction”）

3）int ****cofAC; / int ***cofDC;    用于保存经过transform和quantization以后的宏块系数

4）Slice *currentSlice;

DataPartition *partArr;

Bitstream *bitstream;

byte *streamBuffer;    用于保存最终的编码结果，输出到test.264

在mbuffer.h中定义：StorablePicture用于保存图像处理的结果

imgpel ** imgY; / imgpel *** imgUV;    用于保存重构图像的像素值，输出到test_rec.yuv

imgpel ** p_curr_img; ??

short **** mv;    用于保存motion vector的值

这两个结构体各定义了一个全局变量用来保存图像的处理结果：

在lencod.c中定义：ImageParameters images, *img = &images;

在image.c中定义：StorablePicture *enc_picture;（其实还定义了StorablePicture **enc_frame_picture;但是enc_frame_picture[i]取决于rd_pass变量对i有不同的取值，在实验中只用到enc_frame_picture[0]）

在zhifeng.c和zhifeng.h两个文件中用到img->mpr，enc_picture->imgY/enc_picture->imgUV和enc_picture->mv来得到所有像素的motion vector和residue data。

---------------------------------------------------

me_fullfast.c中的FastFullPelBlockMotionSearch()函数中：

mcost = block_sad[pos_00[list][ref]] + MV_COST_SMP (lambda_factor, 0, 0, pred_mv[0], pred_mv[1]);

是在根据RDOptimization做rate distortion optimazatin。虽然这里RateControlEnable=0，但是对于每p帧中任一宏块都有7中预测模式，理论上用4×4的block得到的distortion最小，但是rate最大，这条语句就是做两者的balance。

RateControlEnable的设置是对整个video sequence做的。

用-d "*.cfg"不用-f "*.cfg"!!!(-f好像是组合参数，lencod -f "encoder_baseline.cfg"对于没有修改的JM中的lencod运行出错)

可以用Bit_Buffer来计算每一帧的大小，但是好像不能计算slice的大小。

image.c：Bit_Buffer[total_frame_buffer] = (int) (stats->bit_ctr - stats->bit_ctr_n);

original图像：

/ buf = malloc (xs*ys * symbol_size_in_bytes)

image.c-->ReadOneFrame()-->

\ read(p_in, buf, bytes_y)

但是buf的值在ReadOneFrame()最后被释放：free (buf);

通过imgY_org_frm保存的是16位，高8位为0x00：

buf2img(imgY_org_frm_JV[0], buf, xs, ys, symbol_size_in_bytes);

buf2img(imgUV_org_frm[0], buf, xs_cr, ys_cr, symbol_size_in_bytes);

buf2img(imgUV_org_frm[1], buf, xs_cr, ys_cr, symbol_size_in_bytes);

重要参数：

stats->bit_ctr_parametersets_n：SPS和PPS的位数

start_sequence()-->stats->bit_ctr_parametersets_n = len;

encode_one_frame（）-->stats->bit_ctr_parametersets_n=0;

stats->bit_ctr_n: 当前处理的帧之前的所有位数

encode_one_frame（）-->stats->bit_ctr_n = stats->bit_ctr;

stats->bit_ctr: 在ReportFirstframe(）中清零！

ReportFirstframe(）-->stats->bit_ctr = 0;

ok:

output:

.cfg

ReportFrameStats = 0 # (0:Disable Frame Statistics 1: Enable)

DisplayEncParams = 0 # (0:Disable Display of Encoder Params 1: Enable)

Verbose = 1 # level of display verboseness (0:short, 1:normal, 2:detailed)

in dump_dpb(), we may set DUMP_DPB = 1 to get debug information!

全局变量：

frame_pic

defined in global.h: Picture **frame_pic;

*img:

defined in lencod.c: ImageParameters images, *img = &images;

referenced in global.h: extern ImageParameters *img;

difference between JM13 and JM14:

there is no LoopFilterDisable in JM14, but deblocking filter!

-------------------------decoder-----------------------------------------

decoder architecture

运动补偿过程：

预测mv：在预测一个宏块的mv时，相邻的a，b，c若不可用，则置零。

mv_a = block_a.available ? tmp_mv[list][block_a.pos_y][block_a.pos_x][hv] : 0;

mv_b = block_b.available ? tmp_mv[list][block_b.pos_y][block_b.pos_x][hv] : 0;

mv_c = block_c.available ? tmp_mv[list][block_c.pos_y][block_c.pos_x][hv] : 0;

decode_one_slice()-->read_one_macroblock()-->SetMotionVectorPredictor()

读入运动矢量:

curr_mv [k] = (short)(curr_mvd[k] + pred_mv[k]);

decode_one_slice()-->read_one_macroblock()-->readMotionInfoFromNAL()-->readMBMotionVectors()

读入coefficients：

read_one_macroblock()-->readCBPandCoeffsFromNAL()

执行mpr：（并没有加上residue）

memcpy(&(curr_mpr[0][ioff]), &(block[0][0]), hor_block_size * ver_block_size * sizeof(imgpel));

decode_one_slice()-->decode_one_macroblock()-->perform_mc()-->mc_prediction()

执行transform:

inverse4x4()

decode_one_slice()-->decode_one_macroblock()-->iTransform()-->iMBtrans4x4()-->itrans_4x4()

mpr加上residue：

m7[j][i] = iClip1(max_imgpel_value, rshift_rnd_sf((m7[j][i] + ((long)mpr[j][i] << DQ_BITS)), DQ_BITS));

decode_one_slice()-->decode_one_macroblock()-->iTransform()-->iMBtrans4x4()-->itrans_4x4()

mb_type defined in defines.h

enum {

PSKIP = 0,

BSKIP_DIRECT = 0,

P16x16 = 1,

P16x8 = 2,

P8x16 = 3,

SMB8x8 = 4,

SMB8x4 = 5,

SMB4x8 = 6,

SMB4x4 = 7,

P8x8 = 8,

I4MB = 9,

I16MB = 10,

IBLOCK = 11,

SI4MB = 12,

I8MB = 13,

IPCM = 14,

MAXMODE = 15

} MBModeTypes;

计算SNR：

snr->snr[k]=(float)(10*log10(max_pix_value_sqd[k]*(double)((double)(comp_size_x[k])*(comp_size_y[k]) / diff_comp[k])));

计算平均SNR：

snr->snra[k]=(float)(snr->snra[k]*(snr->frame_ctr)+snr->snr[k])/(snr->frame_ctr + 1); // average snr chroma for all frames

read_new_slice()-->init_picture()-->exit_picture()-->store_picture_in_dpb()-->direct_output()-->find_snr()

丢包时的执行过程：

设置标志位：

1）currSlice->dpC_NotPresent =1; @ if ((slice_id_c != slice_id_a)|| (nalu->lost_packets))

read_new_slice()

2) currMB->dpl_flag = 1; @ if (IS_INTER (currMB) && currSlice->dpC_NotPresent )

decode_one_slice()-->read_one_macroblock()-->readCBPandCoeffsFromNAL()

3) cbp = 0; / currMB->cbp = cbp; @ if (currMB->dpl_flag)

decode_one_slice()-->read_one_macroblock()-->readCBPandCoeffsFromNAL()

修复：

read_new_slice()-->init_picture()-->exit_picture()-->ercConcealIntraFrame()-->concealBlocks()-->ercPixConcealIMB()-->pixMeanInterpolateBlock()

error concealment有两种：

intra:

read_new_slice()-->init_picture()-->exit_picture()-->ercConcealIntraFrame()

inter:

read_new_slice()-->init_picture()-->exit_picture()-->ercConcealInterFrame()

根据erc_mvperMB和MVPERMB_THR的比较来选用怎么inter conceal：

if(erc_mvperMB >= MVPERMB_THR)

concealByTrial();

else

concealByCopy();

erc_mvperMB的计算方法：

8x4, 4x8, 4x4用平均值得到8x8的erc_mvperMB

16x16, 16x8, 8x16, 8x8直接赋值

然后累加成一个MB的erc_mvperMB。（若MVPERMB_THR设成8，则MB的平均MVx和MVy的和不能超过2，而mv的一个pixel值为4，所以JM14.0中把MVPERMB_THR设成8，MVx和MVy和不能超过0.5个pixel，基本上不执行concealByCopy（））

decode_one_slice()-->ercWriteMBMODEandMV()

如何根据设置来选择concealment：

if (img->conceal_mode==1)

exit_picture()-->store_picture_in_dpb()-->output_one_frame_from_dpb()-->write_lost_ref_after_idr()-->copy_to_conceal()

但是在write_lost_ref_after_idr()之前poc的获取get_smallest_poc()@（while (dpb.used_size==dpb.size)）好像有问题，在16帧以后才执行。然后dpb.last_output_poc = poc

另外：

dpb.size = getDpbSize()

init_dpb()

size = imin( size, 16);

init_dpb()-->getDpbSize()

Actions:

Q&A:

Why put WriteOneFrameMV() in main() in lencod.c file， while put WriteOneFrameResidue() right after encode_one_macroblock() in encode_one_slice() in the slice.c file?

1) Since img->mpr only contains one Macroblock data, so it can not be put in lencod.c like WriteOneFrameMV() where enc_picture->mv contains one frame data.

img->mpr is changed in encode_one_macroblock_low()-->LumaResidualCoding()/(Line617)-->LumaResidualCoding8x8()-->LumaPrediction(): memcpy(&(curr_mpr[j][block_x]), l0pred, block_size_x * sizeof(imgpel));

2) The best place to put WriteOneFrameResidue() is right after encode_one_macroblock() because enc_picture->imgY/enc_picture->imgUV is changed in three locations:

a) encode_one_macroblock_low()-->LumaResidualCoding()/(Line617)-->LumaResidualCoding8x8()-->pDCT_4x4()/(Line901)<-->dct_4x4()-->SampleReconstruct(): *imgOrg++ = iClip1( max_imgpel_value, rshift_rnd_sf(*m7++, dq_bits) + *imgPred++);

b) encode_one_macroblock_low()-->LumaResidualCoding()/(Line617)-->LumaResidualCoding8x8(): memcpy(&enc_picture->imgY[img->pix_y + j][img->pix_x + mb_x], &img->mpr[0][j][mb_x], 2 * BLOCK_SIZE * sizeof(imgpel));

c) encode_one_macroblock_low()-->LumaResidualCoding()/(Line617): memcpy(&enc_picture->imgY[img->pix_y+j][img->pix_x], img->mpr[0][j], MB_BLOCK_SIZE * sizeof (imgpel));

So, we may put WriteOneFrameResidue() right after LumaResidualCoding(), however to generally support other encode_one_macroblock mrthods, we put WriteOneFrameResidue() right after encode_one_macroblock().

Why not use img->m7 as residue data ouput?

Although img->m7 is used as a temporary variable for processing residue data, but it is the never accurate residue data value. img->m7 is changed in two locations:

a) encode_one_macroblock_low()-->LumaResidualCoding()/(Line617)-->LumaResidualCoding8x8()-->ComputeResidue(): *m7++ = *imgOrg++ - *imgPred++; (where m7 has not been quantized)

b) after forward4x4() and quant_4x4(),

If there is nonzero coefficient, img->m7 is changed in inverse4x4() and the value should be rshift_rnd_sf() to get residue data as in SampleReconstruct().

If there is no nonzero coefficient, img->m7 is invalid because only predicted value is used for enc_picture->p_curr_img.

If no residue data output in the global structure, how does JM encode residue data?

The residue data is needed only after transform and quantization in H.264 decoder, so JM14.0 does not keep original residue data in global variable.

The one used for H.264 decoder is defined in img->cofAC/img->cofDC:

which are changed in quant_4x4(),<-->quant_4x4_around() (int* ACLevel = img->cofAC[b8][b4][0]; defined in dct_4x4())

which are used to be VLC coded in main()-->encode_one_frame()-->frame_picture()-->code_a_picture(frame)-->encode_one_slice()-->write_one_macroblock()-->writeMBLayer()-->writeCoeff16x16()-->writeCoeff8x8()-->writeCoeff4x4_CAVLC()

test_rec.yuv通过什么变量写入？

dpb.fs[pos]: write_stored_frame(dpb.fs[pos], p_dec)

enc_frame_picture[0]：

初始化地址：prepare_enc_frame_picture( &enc_frame_picture[0] )<--frame_picture()

1）通过get_mem2Dpel (&(s->imgY), size_y, size_x)，给s->imgY赋地址，s是alloc_storable_picture的返回值。

2）通过(*stored_pic) = alloc_storable_picture ((PictureStructure) img->structure, img->width, img->height, img->width_cr, img->height_cr)给函数prepare_enc_frame_picture的参数StorablePicture **stored_pic赋地址

注意：不是分配内存，所以不用对enc_frame_picture[0]赋值，而是用enc_frame_picture[0]作为函数prepare_enc_frame_picture的参数来赋地址！

写地址内容：encode_one_macroblock_high()-->compute_mode_RD_cost()-->RDCost_for_macroblocks()

-->Intra16x16_Mode_Decision()-->currMB->cbp = dct_16x16 (currMB, PLANE_Y, *i16mode)-->img_Y[i] = iClip1( max_imgpel_value, rshift_rnd_sf(M1[j][i], DQ_BITS) + predY[i])这里一次向&enc_frame_picture[0]写两个字节的内容

compute_mode_RD_cost（）一次写16次（的两个字节）

-->currMB->cbp = Mode_Decision_for_Intra4x4Macroblock (currMB, lambda, &dummy_d)-->Mode_Decision_for_8x8IntraBlocks()-->Mode_Decision_for_4x4IntraBlocks()-->RDCost_for_4x4IntraBlocks()

-->*nonzero = pDCT_4x4 (currMB, PLANE_Y, block_x, block_y, &dummy, 1)-->orig_img[i] = iClip1( max_imgpel_value, rshift_rnd_sf(m7[i], DQ_BITS) + pred_img[i])

在insert_picture_in_dpb(dpb.fs[dpb.used_size],p)中，将dpb.fs指向enc_frame_picture[0]

main()-->encode_one_frame()-->store_picture_in_dpb(enc_frame_picture[0])-->insert_picture_in_dpb(dpb.fs[dpb.used_size],p)

打开文件：p_dec=open(params->ReconFile, OPENFLAGS_WRITE, OPEN_PERMISSIONS)

main()-->Configure()-->PatchInp()

关闭文件：close(p_dec)

main()

test.264通过什么变量写入？

frame_pic[img->rd_pass]:

初始化：

定义在global.h的全局变量 Picture **frame_pic;

将frame_pic[0]赋给frame：frame_picture (frame_pic[0], 0)

main()-->encode_one_frame()-->frame_picture()

将frame/frame_pic[0]赋给pic：code_a_picture(frame)；

main()-->encode_one_frame()-->frame_picture()--code_a_picture(frame)

在函数encode_one_slice()中pic参数只被赋给img->currentPicture: img->currentPicture = pic

将pic/frame/frame_pic[0]赋给img->currentPicture：img->currentPicture = pic

img是全局变量：ImageParameters images, *img (lencod.c) = &images; / extern ImageParameters *img; (global.h)

将img->currentPicture赋给currPic: Picture *currPic = img->currentPicture;

在init_slice()中分配内存：currPic->slices[currPic->no_slices-1] = malloc_slice();

即: frame_pic[0]->slices[currPic->no_slices-1] = malloc_slice() / img->currentPicture->slices[currPic->no_slices-1] = malloc_slice()

main()-->encode_one_frame()-->frame_picture()-->code_a_picture(frame)-->encode_one_slice()-->init_slice()

写内容：

currStream->streamBuffer[currStream->byte_pos++] = currStream->byte_buf

encode_one_macroblock_high()-->submacroblock_mode_decision()-->RDCost_for_8x8blocks()-->writeCoeff8x8()-->writeCoeff4x4_CAVLC()

-->writeSyntaxElement_NumCoeffTrailingOnes()-->writeUVLC2buffer()

-->writeSyntaxElement_Level_VLC1()/writeSyntaxElement_Level_VLCN()-->writeUVLC2buffer()

-->writeSyntaxElement_TotalZeros()-->writeUVLC2buffer()

encode_one_macroblock_high()-->compute_mode_RD_cost()/共(max_index = 9)次-->RDCost_for_macroblocks()-->writeMBLayer()-->writeCoeff16x16()-->writeCoeff8x8()-->writeCoeff4x4_CAVLC()

写到test.264：

currSlice = pic->slices[slice] 这里currSlice通过行参pic指向writeout_picture (frame_pic[img->rd_pass])实参frame_pic[img->rd_pass]: currSlice = pic->slices[slice] (=frame_pic[img->rd_pass]->slices[slice])

currStream = (currSlice->partArr[partition]).bitstream （=(frame_pic[img->rd_pass]->slices[slice]->partArr[partition]).bitstream）

main()-->encode_one_frame()-->writeout_picture()

memcpy (&nalu->buf[1], currStream->streamBuffer, nalu->len-1); 然后：WriteNALU (nalu)

main()-->encode_one_frame()-->writeout_picture()-->writeUnit()

打开文件：f = fopen (Filename, "wb")

main()-->start_sequence()-->OpenAnnexbFile()

关闭文件：fclose (f)

main()-->terminate_sequence()-->CloseAnnexbFile()

在哪里估计mv?

1)

mv[0] = offset_x + spiral_search_x[best_pos];

mv[1] = offset_y + spiral_search_y[best_pos];

encode_one_macroblock_low()-->PartitionMotionSearch()

encode_one_macroblock_low()-->submacroblock_mode_decision()-->PartitionMotionSearch()

-->BlockMotionSearch()-->IntPelME()<-->FastFullPelBlockMotionSearch()

2)

然后，skip模式可能重写mv：

mv[0] = img->all_mv [0][0][0][0][0][0];

mv[1] = img->all_mv [0][0][0][0][0][1];

encode_one_macroblock_low()-->PartitionMotionSearch()-->BlockMotionSearch()

encode_one_macroblock_low()-->submacroblock_mode_decision()-->PartitionMotionSearch()-->BlockMotionSearch()

3)

关于mv的内容： short*** all_mv = &img->all_mv[list][ref][blocktype][block_y]; //!< block type (1-16x16 ... 7-4x4)

all_mv[0][i][0] = mv[0];

all_mv[0][i][1] = mv[1];

encode_one_macroblock_low()-->PartitionMotionSearch()-->BlockMotionSearch()

encode_one_macroblock_low()-->submacroblock_mode_decision()-->PartitionMotionSearch()-->BlockMotionSearch()

PSliceSkip设置：enc_mb->valid[0] = (!intra && params->InterSearch[bslice][0])

encode_one_macroblock_low()-->init_enc_mb_params()

img->all_mv

分配内存：

int get_mem_mv (short ******* mv)

{

// LIST, reference, block_type, block_y, block_x, component

get_mem6Dshort(mv, 2, img->max_num_references, 9, 4, 4, 2);

return 2 * img->max_num_references * 9 * 4 * 4 * 2 * sizeof(short);

}

get_mem_mv (&(img->pred_mv));

get_mem_mv (&(img->all_mv));

main()->init_img()

enc_picture->mv

初始化

写内容：通过img->all_mv写入enc_picture->mv

memcpy(enc_picture->mv [LIST_0][block_y][block_x + i], img->all_mv[LIST_0][best_l0_ref][mode][j][i], 2 * sizeof(short))

encode_one_macroblock_low()-->assign_enc_picture_params（）

enc_picture->mv[LIST_0][by][bx][0] = all_mv [LIST_0][ ref][mode8][j][i][0];

enc_picture->mv[LIST_0][by][bx][1] = all_mv [LIST_0][ ref][mode8][j][i][1];

encode_one_macroblock_low()-->SetMotionVectorsMB (currMB, bslice)

enc_frame_picture[0]

什么时候写出mv？

在encode_one_frame()之后

mv在什么时候变化?

init_frame ()

main()-->encode_one_frame()

什么时候写出residue？

//===== S E T F I N A L M A C R O B L O C K P A R A M E T E R S ======

通过(*curr_mpr)[16] = img->mpr[0]

memcpy(&(curr_mpr[j][block_x]), l0pred, block_size_x * sizeof(imgpel));

通过m7 = &img_m7[j][mb_x]：*m7++ = *imgOrg++ - *imgPred++;

encode_one_macroblock_low()-->LumaResidualCoding()-->LumaResidualCoding8x8()-->ComputeResidue()/Line863

//===== DCT, Quantization, inverse Quantization, IDCT, Reconstruction =====

通过img->m7：

(*curr_res)[MB_BLOCK_SIZE] = img->m7[pl];

m7 = &img_m7[j][mb_x];

SampleReconstruct (imgpel **curImg, imgpel mpr[16][16], int img_m7[16][16], int mb_y, int mb_x, int opix_y, int opix_x, int width, int height, int max_imgpel_value, int dq_bits)

*imgOrg++ = iClip1( max_imgpel_value, rshift_rnd_sf(*m7++, dq_bits) + *imgPred++);

encode_one_macroblock_low()-->LumaResidualCoding()/(Line617)-->LumaResidualCoding8x8()-->pDCT_4x4()/(Line901)<-->dct_4x4()-->SampleReconstruct()

encode_one_macroblock_low()-->ChromaResidualCoding()/(Line645)

/*!

计算宏块的残差m7：

*m7++ = *imgOrg++ - *imgPred++;

encode_one_macroblock_low()-->submacroblock_mode_decision()-->LumaResidualCoding8x8()-->ComputeResidue()

m7_line = &m7[j][block_x];

*m7_line++ = (int) (*cur_line++ - *prd_line++)

encode_one_macroblock_low()-->Mode_Decision_for_Intra4x4Macroblock()-->Mode_Decision_for_8x8IntraBlocks()-->Mode_Decision_for_4x4IntraBlocks_JM_Low()-->generate_pred_error()

img->m7[uv + 1][j][i] = imgUV_org[uv][img->opix_c_y+j][img->opix_c_x+i] - curr_mpr[j][i]; （在global.h中定义imgpel ***imgUV_org;）

encode_one_macroblock_low()-->ChromaResidualCoding()/line645

*/

几种intrapred：

encode_one_macroblock_low()-->Mode_Decision_for_Intra4x4Macroblock()-->Mode_Decision_for_8x8IntraBlocks()-->Mode_Decision_for_4x4IntraBlocks_JM_Low()-->intrapred_4x4()

encode_one_macroblock_low()-->Mode_Decision_for_new_Intra8x8Macroblock()-->(point functions below)-->Mode_Decision_for_new_8x8IntraBlocks_JM_High()/Mode_Decision_for_new_8x8IntraBlocks_JM_Low()-->intrapred_8x8()

encode_one_macroblock_high()/encode_one_macroblock_highfast()/encode_one_macroblock_highloss()-->compute_mode_RD_cost()-->RDCost_for_macroblocks()-->Mode_Decision_for_new_Intra8x8Macroblock()-->(above)-->Mode_Decision_for_new_8x8IntraBlocks_JM_High()/Mode_Decision_for_new_8x8IntraBlocks_JM_Low()-->intrapred_8x8()

Mode_Decision_for_new_8x8IntraBlocks = Mode_Decision_for_new_8x8IntraBlocks_JM_Low;

Mode_Decision_for_new_8x8IntraBlocks = Mode_Decision_for_new_8x8IntraBlocks_JM_High;

encode_one_macroblock_low()-->intrapred_16x16()

encode_one_macroblock_high()/encode_one_macroblock_highfast()/encode_one_macroblock_highloss()-->compute_mode_RD_cost()-->RDCost_for_macroblocks()-->Intra16x16_Mode_Decision()-->intrapred_16x16()

？？为什么不能用img->m7来保存residue（就算没有post-scaling, the last step(8-356) of 8.5.10）？

因为在LumaResidualCoding8x8()中pDCT_4x4执行后，还有一行修改了enc_picture->imgY值，所以m7并不是residue：

memcpy(&enc_picture->imgY[img->pix_y + j][img->pix_x + mb_x], &img->mpr[0][j][mb_x], 2 * BLOCK_SIZE * sizeof(imgpel));

encode_one_macroblock_low()-->submacroblock_mode_decision()-->LumaResidualCoding8x8()

-------------------------decoder-----------------------------------------

？总是会执行exit_picture()中的DeblockPicture( img, dec_picture );

不是，在函数内部判断

如何在处理完一帧后，进行error concealment？

在init_picture()中：

if (dec_picture)

{

// this may only happen on slice loss

exit_picture();

}

read_new_slice()-->init_picture()

应该放到decode_one_frame()中的最后！！！但是因为while ((currSlice->next_header != EOS && currSlice->next_header != SOP))一直为真，所以并没有被执行到。而是从if (current_header == EOS)内退出了

My understanding:

对JM的建议：

1）应该增加一个标志位在MB中标明哪几个8x8的subMB的coeff被置成零！因为几个地方置coeff，而img->m7没有相应被修改，导致很难得到residue data。

有两个地方用到LumaResidualCoding8x8()，这里第二次需要重新reset_residuex_zero()

encode_one_macroblock_low()-->submacroblock_mode_decision()-->LumaResidualCoding8x8()

encode_one_macroblock_low()-->LumaResidualCoding()-->LumaResidualCoding8x8()

有两个地方用mpr的值重设imgY：

memcpy(&enc_picture->imgY[img->pix_y+j][img->pix_x], tr8x8.mpr8x8[j], MB_BLOCK_SIZE * sizeof(imgpel));

或者：memcpy (&enc_picture->imgY[img->pix_y+j][img->pix_x],tr4x4.mpr8x8[j], MB_BLOCK_SIZE * sizeof(imgpel));

encode_one_macroblock_low()-->SetCoeffAndReconstruction8x8()

memcpy(&enc_picture->imgY[img->pix_y+j][img->pix_x], img->mpr[0][j], MB_BLOCK_SIZE * sizeof (imgpel));

encode_one_macroblock_low()-->LumaResidualCoding()

-------------------------decoder-----------------------------------------

？当partitioning C的RTP包不是在partitioning A的RTP包之后到达时，解码器会有问题？（在NALU_TYPE_DPA内部执行read_next_nalu(nalu)）

对JM的建议：

1）MVPERMB_THR的取值过小（JM14.0中用8，在erc_api.h中）

2）erc_mvperMB的计算有问题：

不能用erc_mvperMB /= dec_picture->PicSizeInMbs;

因为在decode_one_slice()-->ercWriteMBMODEandMV()中累加erc_mvperMB，而丢失partitionA包时不执行decode_one_slice()

erc_mvperMB += iabs(pRegion->mv[0]) + iabs(pRegion->mv[1]);

read_new_slice()-->init_picture()-->exit_picture()

Useful Links:

0) for beginner:

http://www.h263l.com/h264/h264_overview.pdf

1) JM14.0 source code:

http://iphome.hhi.de/suehring/tml/download/

2) H.264 standard:

http://www.itu.int/rec/T-REC-H.264

3) H.264 Wiki:

http://en.wikipedia.org/wiki/H.264

4) JM online document:

http://iphome.hhi.de/suehring/tml/doc/lenc/html/index.html

http://iphome.hhi.de/suehring/tml/doc/ldec/html/index.html

5) JM manual:

http://iphome.hhi.de/suehring/tml/JM%20Reference%20Software%20Manual%20(JVT-X072).pdf

6) Overview of the H.264/AVC Video Coding Standard

http://ieeexplore.ieee.org/iel5/76/27384/01218189.pdf

7) H264Visa.exe

8) YUVviewer.exe

9) RTP Payload Format for H.264 Video

http://tools.ietf.org/html/rfc3984

10) x264

http://blog.csdn.net/sunshine1314/archive/2005/05/20/377158.aspx

http://lspbeyond.go1.icpcn.com/x264/index.htm

11) white paper

http://www.vcodex.com/h264.html(in sections)

http://bbs.chinavideo.org/viewthread.php?tid=3336&extra=page%3D1(whole)

back to top

FFMPEG :

|| Reading Notes || Actions || Q&A || My understanding || Useful Links ||

Reading Notes:

How to support camera in windows for FFMPEG？

ffmpeg -r 15 -f vfwcap -i 0 output.mpg

http://ffmpeg.arrozcru.org/forum/viewtopic.php?f=8&t=763

Actions:

Q&A:

My understanding: