更新：最近又看了VTM9.0中estIntraPredLumaQT函数，相对VTM7.0还是更新了许多内容，加入了ACT变换等等
最近研究了VTM7.0中estIntraPredLumaQT函数的大体功能，之前大概看过一点，但是被这繁乱的代码搞自闭了，没有仔细研究过，这次借助H师兄的博客才勉强看懂大部分代码。
estIntraPredLumaQT函数主要是完成亮度分量的预测，并选出亮度预测的最佳模式。
主要流程如下：
一、初始化各种参数；
二、为了减少最终RDcost的次数，降低编码端的复杂度，estIntraPredLumaQT使用帧内快速搜索算法，主要包含三个阶段：
1、RMD：为了减少运算复杂度，直接对残差进行哈达玛变换求得SATD失真计算代价，将最小代价前几的模式加入全率失真模式列表中（不进行变换量化等操作）
2、MPM：根据相邻块推导出当前块的最可能模式，加入到全率失真模式列表中。
3、RDO：遍历全率失真模式列表中的模式，进行完整的变换、量化、反变换、反量化等操作，计算失真和比特并求出RD Cost，选择代价最小的模式作为当前块的最优角度模式。
具体过程如下：
1、第一轮SATD粗选：对35种非扩展的传统的亮度帧内角度预测模式进行第一轮SATD循环粗选，这里粗选的时候使用predIntraAng函数计算预测值，然后计算SAD和SATD（HAD哈达玛变换后的SAD），并选其最小值计算每种模式的Cost，然后调用updateCandList函数更新全率失真候选模式列表uiRdModeList，候选列表长度为numModesForFullRD。
2、第二轮SATD粗选：
（1）对第一轮选出的numModesForFullRD种非扩展角度模式，分别对其左右相邻的扩展角度模式使用predIntraAng函数计算预测值，然后计算SAD和SATD，并选其最小值计算每种模式的Cost，然后调用updateCandList函数更新全率失真候选模式列表uiRdModeList，候选列表长度为numModesForFullRD。（numModesForFullRD通常为3，使用MIP时翻倍为6）
（2）遍历多参考行MRL模式，使用getIntraMPMs函数获取MPM列表，然后对每个扩展的参考行（1、2）进行遍历，选出最有可能的几种模式加入到全率失真优化列表中。粗选过程与之前类似，循环遍历MPM列表中的每种模式，再调用predIntraAng函数计算预测值，然后计算SAD和SATD，并选其最小值计算每种模式的Cost，调用updateCandList函数更新全率失真候选模式列表uiRdModeList。
（3）使用哈达玛变换对MIP模式单独进行候选模式的推导。根据当前CU块的尺寸获取到MIP的模式数，然后对每一种模式调用predIntraMip函数计算预测值，然后计算SAD和SATD，并选其最小值计算每种模式的Cost，再调用updateCandList函数更新全率失真候选模式列表uiRdModeList。在这之后，调用reduceHadCandList函数对全率失真候选模式列表uiRdModeList进行缩减处理。
（4）获取FastUDI最可能的MPM模式列表，仅遍历左相邻CU和上相邻CU的模式，判断它们是否在全率失真候选列表中，若不在则将其加入进去。同样地，对ISP模式列表m_ispCandListHor进行相同的处理。
（5）如果需要测试ISP模式，则需为ISP在全率失真候选模式列表中保留位置，即将所有的ISP模式添加进uiRdModeList。
3、第三轮RD Cost细选：遍历RD候选模式列表，如果当前模式为ISP模式，则将ISP列表重新排列（仅排列一次），然后调用xIntraCodingLumaISP函数进行ISP模式的残差变换、量化并计算RD Cost；否则调用xRecurIntraCodingLumaQT函数进行普通的残差变换、量化并计算RD Cost。
4、收尾工作：判断最优模式是否是MIP以及是否使用多参考行，并将最优模式保存入pu.intraDir里。
注意：这里涉及到许多快速算法
对于DCT-2变换和MTS变换，为降低复杂度，使用mtsUsageFlag变量标记MTS以实现快速算法。
- mtsUsageFlag=0时，表示MTS不适用于当前CU，即一次变换只需要检查DCT-2，此时需要进行第一轮SATD粗选和第二轮SATD粗选，并将粗选结果保存下来。
- mtsUsageFlag=1时，表示当前CU可以使用DCT-2和MTS，且当前正在检查DCT-2，此时需要进行第一轮SATD粗选和第二轮SATD粗选，并将粗选过程保存下来。
- mtsUsageFlag=2时，表示当前CU正在检查MTS，这里由于之前检查DCT-2的时候已经进行了一轮和二轮的SATD粗选，所以可以直接将粗选结果加载出来。
对于LFNST变换，当lfnstIdx=0时候需要进行完整的一轮、二轮粗选和RD Cost细选过程，并将第一轮SATD和第二轮SATD粗选的（1）（2）（3）过程的全率失真候选列表保存下来。当lfnstIdx=1、2时直接加载lfnst=0时保存的全率失真候选列表，并且不再将ISP模式添加进候选模式列表，这样可以大大降低编码端的复杂度（相关变量LFNSTSaveFlag、LFNSTLoadFlag）。
至于DCT-2变换、MTS变换和LFNST变换的遍历检查，在estIntraPredLumaQT的上层函数xCheckRDCostIntra中进行。
代码及注释如下：
bool IntraSearch::estIntraPredLumaQT(CodingUnit &cu, Partitioner &partitioner, const double bestCostSoFar, bool mtsCheckRangeFlag, int mtsFirstCheckId, int mtsLastCheckId, bool moreProbMTSIdxFirst, CodingStructure* bestCS)
{
CodingStructure &cs = *cu.cs;//获取当前CU的编码结构
const SPS &sps = *cs.sps;//获取该编码结构的语法元素
const uint32_t uiWidthBit = floorLog2(partitioner.currArea().lwidth() );
const uint32_t uiHeightBit = floorLog2(partitioner.currArea().lheight());
// Lambda calculation at equivalent Qp of 4 is recommended because at that Qp, the quantization divisor is 1.
// 建议在等效qp为4时进行lambda的计算，因为在该qp时，用于量化的除数为1。
const double sqrtLambdaForFirstPass = m_pcRdCost->getMotionLambda( ) * FRAC_BITS_SCALE;
//===== loop over partitions =====
//上下文模型的开始
const TempCtx ctxStart ( m_CtxCache, m_CABACEstimator->getCtx() );
//Mip模式的flag上下文模型
const TempCtx ctxStartMipFlag ( m_CtxCache, SubCtx( Ctx::MipFlag, m_CABACEstimator->getCtx() ) );
//ISP模式的上下文
const TempCtx ctxStartIspMode ( m_CtxCache, SubCtx( Ctx::ISPMode, m_CABACEstimator->getCtx() ) );
//Planar模式的flag上下文模型
const TempCtx ctxStartPlanarFlag ( m_CtxCache, SubCtx( Ctx::IntraLumaPlanarFlag, m_CABACEstimator->getCtx() ) );
//其他帧内模式的上下文
const TempCtx ctxStartIntraMode(m_CtxCache, SubCtx(Ctx::IntraLumaMpmFlag, m_CABACEstimator->getCtx()));
//多参考行的索引
const TempCtx ctxStartMrlIdx ( m_CtxCache, SubCtx( Ctx::MultiRefLineIdx, m_CABACEstimator->getCtx() ) );
CHECK( !cu.firstPU, "CU has no PUs" );
// variables for saving fast intra modes scan results across multiple LFNST passes
//用于保存多个LFNST遍的快速内部模式扫描结果的变量
//这两个变量主要用于使用LFNST时，只需要进行一遍一轮和二轮的SATD扫描
//即当lfnstIdx=0时，使用LFNSTSaveFlag将一轮和二轮扫描结果保存下来
//当lfnstIdx=1、2时，使用LFNSTLoadFLag直接加载一轮和二轮的扫描结果
bool LFNSTLoadFlag = sps.getUseLFNST() && cu.lfnstIdx != 0;//SPS层开启LFNST，且正在检查lfnst
bool LFNSTSaveFlag = sps.getUseLFNST() && cu.lfnstIdx == 0;//SPS层开启LFNST且没有检查LFNST
//若SPS层开启帧内MTS，判断当前是否检查的是DCT-2；若没有开启帧内MTS，则说明变换必是DCT2，故为true
LFNSTSaveFlag &= sps.getUseIntraMTS() ? cu.mtsFlag == 0 : true;
const uint32_t lfnstIdx = cu.lfnstIdx;
double costInterCU = findInterCUCost( cu );//帧间CU的cost
//当前亮度CU的宽和高
const int width = partitioner.currArea().lwidth();
const int height = partitioner.currArea().lheight();
// Marking MTS usage for faster MTS
// 0: MTS is either not applicable for current CU (cuWidth > MTS_INTRA_MAX_CU_SIZE or cuHeight > MTS_INTRA_MAX_CU_SIZE), not active in the config file or the fast decision algorithm is not used in this case
// 1: MTS fast algorithm can be applied for the current CU, and the DCT2 is being checked
// 2: MTS is being checked for current CU. Stored results of DCT2 can be utilized for speedup
// 标记MTS的使用以获得更快的MTS
// 0:MTS不适用于当前CU（cuWidth>MTS_INTRA_MAX_CU_SIZE或cuHeight>MTS_INTRA_MAX_CU_SIZE），在配置文件中不活动，或者在这种情况下不使用快速决策算法
// 1:当前CU可以采用MTS快速算法，正在检查DCT2
// 2:正在检查当前CU的MTS。DCT2的存储结果可用于加速
uint8_t mtsUsageFlag = 0;
const int maxSizeEMT = MTS_INTRA_MAX_CU_SIZE;
if( width <= maxSizeEMT && height <= maxSizeEMT && sps.getUseIntraMTS() )
{// SPS层开启LFNST，且当前检查的为MTS，则mtsUsageFlag为2；否则mtsUsageFlag为1
mtsUsageFlag = ( sps.getUseLFNST() && cu.mtsFlag == 1 ) ? 2 : 1;
}
//宽度*高度小于64 且没有开启LFNST快速算法
if( width * height < 64 && !m_pcEncCfg->getUseFastLFNST() )
{
mtsUsageFlag = 0;
}
//是否启用ACT变换
const bool colorTransformIsEnabled = sps.getUseColorTrans() && !CS::isDualITree(cs);
//是否是第一种颜色空间
const bool isFirstColorSpace = colorTransformIsEnabled && ((m_pcEncCfg->getRGBFormatFlag() && cu.colorTransform) || (!m_pcEncCfg->getRGBFormatFlag() && !cu.colorTransform));
//是否是第二种颜色空间
const bool isSecondColorSpace = colorTransformIsEnabled && ((m_pcEncCfg->getRGBFormatFlag() && !cu.colorTransform) || (!m_pcEncCfg->getRGBFormatFlag() && cu.colorTransform));
//定义一个当前最优的代价
double bestCurrentCost = bestCostSoFar;
//是否可以使用ISP：SPS层开启ISP且当前正在检查仅仅DCT-2变换且当前块的尺寸符号ISP尺寸限制
bool ispCanBeUsed = sps.getUseISP() && cu.mtsFlag == 0 && cu.lfnstIdx == 0 && CU::canUseISP(width, height, cu.cs->sps->getMaxTbSize());
//为ISP模式保存数据
bool saveDataForISP = ispCanBeUsed && (!colorTransformIsEnabled || isFirstColorSpace);
//测试ISP
bool testISP = ispCanBeUsed && (!colorTransformIsEnabled || !cu.colorTransform);
if ( saveDataForISP )
{
//reset the intra modes lists variables
//重置ISP帧内模式列表变量
m_ispCandListHor.clear();
m_ispCandListVer.clear();
}
if( testISP )
{
//reset the variables used for the tests
//重置用于测试的变量
m_regIntraRDListWithCosts.clear();
//水平划分所得子分区数
int numTotalPartsHor = (int)width >> floorLog2(CU::getISPSplitDim(width, height, TU_1D_VERT_SPLIT));
//垂直划分所得子分区数
int numTotalPartsVer = (int)height >> floorLog2(CU::getISPSplitDim(width, height, TU_1D_HORZ_SPLIT));
m_ispTestedModes[0].init( numTotalPartsHor, numTotalPartsVer );
//the total number of subpartitions is modified to take into account the cases where LFNST cannot be combined with ISP due to size restrictions
//修改子分区的总数，以考虑由于大小限制而无法将LFNST与ISP结合的情况
numTotalPartsHor = sps.getUseLFNST() && CU::canUseLfnstWithISP(cu.Y(), HOR_INTRA_SUBPARTITIONS) ? numTotalPartsHor : 0;
numTotalPartsVer = sps.getUseLFNST() && CU::canUseLfnstWithISP(cu.Y(), VER_INTRA_SUBPARTITIONS) ? numTotalPartsVer : 0;
for (int j = 1; j < NUM_LFNST_NUM_PER_SET; j++)
{
m_ispTestedModes[j].init(numTotalPartsHor, numTotalPartsVer);
}
}
//BDPCM模式测试标志
const bool testBDPCM = sps.getBDPCMEnabledFlag() && CU::bdpcmAllowed(cu, ComponentID(partitioner.chType)) && cu.mtsFlag == 0 && cu.lfnstIdx == 0;
static_vector<ModeInfo, FAST_UDI_MAX_RDMODE_NUM> uiHadModeList;//哈达玛变换（即SATD）粗选后的候选列表
static_vector<double, FAST_UDI_MAX_RDMODE_NUM> CandCostList;//代价候选列表
static_vector<double, FAST_UDI_MAX_RDMODE_NUM> CandHadList;//哈达玛变换（SATD/HAD）后的候选代价列表
auto &pu = *cu.firstPU;
bool validReturn = false;
//代码块
{
CandHadList.clear();
CandCostList.clear();
uiHadModeList.clear();
CHECK(pu.cu != &cu, "PU is not contained in the CU");
//===== determine set of modes to be tested (using prediction signal only) =====
//===== 确定要测试的模式集（仅使用预测信号）======
// 所有67种传统帧内模式的数量，不包括MIP模式，MIP模式另行处理
int numModesAvailable = NUM_LUMA_MODE; // total number of Intra modes
const bool fastMip = sps.getUseMIP() && m_pcEncCfg->getUseFastMIP();//MIP快速算法
//MIP允许使用标志
const bool mipAllowed = sps.getUseMIP() && isLuma(partitioner.chType) && ((cu.lfnstIdx == 0) || allowLfnstWithMip(cu.firstPU->lumaSize()));
//MIP模式测试标志：宽高比不能超过1:8/8:1
const bool testMip = mipAllowed && !(cu.lwidth() > (8 * cu.lheight()) || cu.lheight() > (8 * cu.lwidth()));
//MIP支持的最大块尺寸为64
const bool supportedMipBlkSize = pu.lwidth() <= MIP_MAX_WIDTH && pu.lheight() <= MIP_MAX_HEIGHT;
static_vector<ModeInfo, FAST_UDI_MAX_RDMODE_NUM> uiRdModeList;
/****这里定义的就是能够进入到最终的RDcost环节的最优的几种预测模式的数量，numModesForFullRD初始化为3****/
int numModesForFullRD = 3;//最终的RDcost环节的最优的几种模式的数量
//根据CU的尺寸选取模式数量
numModesForFullRD = g_aucIntraModeNumFast_UseMPM_2D[uiWidthBit - MIN_CU_LOG2][uiHeightBit - MIN_CU_LOG2];
#if INTRA_FULL_SEARCH
numModesForFullRD = numModesAvailable;
#endif
if (isSecondColorSpace)
{ //第二种颜色空间
uiRdModeList.clear();
//直接从第一个颜色空间的模式列表中加载测试的模式列表
if (m_numSavedRdModeFirstColorSpace[m_savedRdModeIdx] > 0)
{
for (int i = 0; i < m_numSavedRdModeFirstColorSpace[m_savedRdModeIdx]; i++)
{
uiRdModeList.push_back(m_savedRdModeFirstColorSpace[m_savedRdModeIdx][i]);
}
}
else
{
return false;
}
}
else
{ //mtsUsageFlag为0或者为1，此时一次变换为DCT-2
if (mtsUsageFlag != 2)
{
// this should always be true
CHECK(!pu.Y().valid(), "PU is not valid");
//是否是CTU的第一行
bool isFirstLineOfCtu = (((pu.block(COMPONENT_Y).y) & ((pu.cs->sps)->getMaxCUWidth() - 1)) == 0);
//使用的扩展参考行的数量，如果为CTU的第一行或者SPS层关闭MRL，则最多只能用1行；如果不是，则最多可以用三行
int numOfPassesExtendRef = ((!sps.getUseMRL() || isFirstLineOfCtu) ? 1 : MRL_NUM_REF_LINES);
pu.multiRefIdx = 0;
//numModesForFullRD的数量应该属少于所有可用的帧内模式的数量的，因此这里是主要的RD的入口
if (numModesForFullRD != numModesAvailable)
{
CHECK(numModesForFullRD >= numModesAvailable, "Too many modes for full RD search");
const CompArea &area = pu.Y();//获取当前亮度PU的区域
PelBuf piOrg = cs.getOrgBuf(area);//获取当前亮度PU区域的亮度原始值
PelBuf piPred = cs.getPredBuf(area);//获取当前亮度PU区域的亮度预测值
DistParam distParamSad;// 定义一个使用SAD的失真参数
DistParam distParamHad;//定义一个使用SATD（加了哈达玛变换）的失真参数
if (cu.slice->getLmcsEnabledFlag() && m_pcReshape->getCTUFlag())
{
CompArea tmpArea(COMPONENT_Y, area.chromaFormat, Position(0, 0), area.size());
PelBuf tmpOrg = m_tmpStorageLCU.getBuf(tmpArea);
tmpOrg.copyFrom(piOrg);
tmpOrg.rspSignal(m_pcReshape->getFwdLUT());
m_pcRdCost->setDistParam(distParamSad, tmpOrg, piPred, sps.getBitDepth(CHANNEL_TYPE_LUMA), COMPONENT_Y,
false); // Use SAD cost
m_pcRdCost->setDistParam(distParamHad, tmpOrg, piPred, sps.getBitDepth(CHANNEL_TYPE_LUMA), COMPONENT_Y,
true); // Use HAD (SATD) cost
}
else
{
m_pcRdCost->setDistParam(distParamSad, piOrg, piPred, sps.getBitDepth(CHANNEL_TYPE_LUMA), COMPONENT_Y,
false); // Use SAD cost
m_pcRdCost->setDistParam(distParamHad, piOrg, piPred, sps.getBitDepth(CHANNEL_TYPE_LUMA), COMPONENT_Y,
true); // Use HAD (SATD) cost
}
distParamSad.applyWeight = false;
distParamHad.applyWeight = false;
if (testMip && supportedMipBlkSize)
{//添加MIP模式到候选模式数目中
numModesForFullRD += fastMip
? std::max(numModesForFullRD, floorLog2(std::min(pu.lwidth(), pu.lheight())) - 1)
: numModesForFullRD;
}
//定义SATD变换候选的数量
const int numHadCand = (testMip ? 2 : 1) * 3;//若是MIP模式为6，否则为3
//*** Derive (regular) candidates using Hadamard
//*** 使用哈达吗变换的推导角度候选（即使用SATD）
cu.mipFlag = false;
//===== init pattern for luma prediction =====
//===== 初始化亮度预测模式 =====
//获取参考像素，并对参考像素进行滤波
initIntraPatternChType(cu, pu.Y(), true);
bool bSatdChecked[NUM_INTRA_MODE];//定义一个存放已经被STAD粗选过的帧内模式的bool数组
memset(bSatdChecked, 0, sizeof(bSatdChecked));
if (!LFNSTLoadFlag)
{
//首先对67种非扩展的传统的亮度帧内模式进行第一轮SATD循环粗选
for (int modeIdx = 0; modeIdx < numModesAvailable; modeIdx++)
{
uint32_t uiMode = modeIdx;
Distortion minSadHad = 0;
// Skip checking extended Angular modes in the first round of SATD
// 在第一轮SATD中跳过检查扩展角模式
// 主要检查0、1、2、4、6...66这些角度模式
if (uiMode > DC_IDX && (uiMode & 1))
{
continue;
}
bSatdChecked[uiMode] = true;
//pu.intraDir中存放的就是当前CU最终选中的预测模式
pu.intraDir[0] = modeIdx;
//初始化帧内预测参数
initPredIntraParams(pu, pu.Y(), sps);
//传统角度预测模式的函数入口，进行传统的角度预测
predIntraAng(COMPONENT_Y, piPred, pu);
// Use the min between SAD and HAD as the cost criterion
// SAD is scaled by 2 to align with the scaling of HAD
// 使用SAD和HAD之间的最小值作为代价标准(HAD其实就是SATD，是加了哈达玛变换后的SAD)
// SAD的缩放比例为2，与HAD的缩放比例一致。
minSadHad += std::min(distParamSad.distFunc(distParamSad) * 2, distParamHad.distFunc(distParamHad));
// NB xFracModeBitsIntra will not affect the mode for chroma that may have already been pre-estimated.
// NB xFracModeBitsIntra不会影响可能已经预先估计的色度模式。
m_CABACEstimator->getCtx() = SubCtx( Ctx::MipFlag, ctxStartMipFlag );
m_CABACEstimator->getCtx() = SubCtx( Ctx::ISPMode, ctxStartIspMode );
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaPlanarFlag, ctxStartPlanarFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaMpmFlag, ctxStartIntraMode);
m_CABACEstimator->getCtx() = SubCtx( Ctx::MultiRefLineIdx, ctxStartMrlIdx );
//估计编码帧内预测模式所需的比特数
uint64_t fracModeBits = xFracModeBitsIntra(pu, uiMode, CHANNEL_TYPE_LUMA);
//计算每种模式的Cost，比较相互之间的Cost进行模式更新
double cost = (double) minSadHad + (double) fracModeBits * sqrtLambdaForFirstPass;
DTRACE(g_trace_ctx, D_INTRA_COST, "IntraHAD: %u, %llu, %f (%d)\n", minSadHad, fracModeBits, cost, uiMode);
//更新SAD之后的RD候选模式列表以及代价列表
updateCandList(ModeInfo(false, false, 0, NOT_INTRA_SUBPARTITIONS, uiMode), cost, uiRdModeList,
CandCostList, numModesForFullRD);
//更新SATD之后的uiHadModeList、CandHadList这些列表
updateCandList(ModeInfo(false, false, 0, NOT_INTRA_SUBPARTITIONS, uiMode), double(minSadHad),
uiHadModeList, CandHadList, numHadCand);
}//遍历67种角度模式
//如果SPS层关闭MIP且需要为LFNST保存粗选结果
if (!sps.getUseMIP() && LFNSTSaveFlag)
{
// save found best modes
m_uiSavedNumRdModesLFNST = numModesForFullRD;
m_uiSavedRdModeListLFNST = uiRdModeList;
m_dSavedModeCostLFNST = CandCostList;
// PBINTRA fast
m_uiSavedHadModeListLFNST = uiHadModeList;
m_dSavedHadListLFNST = CandHadList;
LFNSTSaveFlag = false;
}
} // NSSTFlag (!LFNSTLoadFlag)
//如果SPS层关闭MIP且可以为LFNST直接加载之前的粗选结果
if (!sps.getUseMIP() && LFNSTLoadFlag)
{
// restore saved modes
numModesForFullRD = m_uiSavedNumRdModesLFNST;
uiRdModeList = m_uiSavedRdModeListLFNST;
CandCostList = m_dSavedModeCostLFNST;
// PBINTRA fast
uiHadModeList = m_uiSavedHadModeListLFNST;
CandHadList = m_dSavedHadListLFNST;
} // !LFNSTFlag
//SPS层关闭MIP或者不能为LFNST加载粗选结果时
if (!(sps.getUseMIP() && LFNSTLoadFlag))
{
// 将第一轮选出来的最优模式放到父模式中
static_vector<ModeInfo, FAST_UDI_MAX_RDMODE_NUM> parentCandList = uiRdModeList;
// Second round of SATD for extended Angular modes
// 这里开始第二轮对剩下的扩展角度模式的SATD粗选
for (int modeIdx = 0; modeIdx < numModesForFullRD; modeIdx++)
{
// 定义父模式，父模式放入的是第一轮选出来的最优的模式列表
unsigned parentMode = parentCandList[modeIdx].modeId;
// 从扩展模式开始循环处理
// 模式号是从3-65
if (parentMode > (DC_IDX + 1) && parentMode < (NUM_LUMA_MODE - 1))
{
for (int subModeIdx = -1; subModeIdx <= 1; subModeIdx += 2)
{
//给父模式减一或者加一，正好就是上一轮那些最优候选列表中非扩展角度模式的左右相邻的扩展角度模式
unsigned mode = parentMode + subModeIdx;
//如果mode没有进行过SATD粗选
if (!bSatdChecked[mode])
{
pu.intraDir[0] = mode;
initPredIntraParams(pu, pu.Y(), sps);
predIntraAng(COMPONENT_Y, piPred, pu);
// Use the min between SAD and SATD as the cost criterion
// SAD is scaled by 2 to align with the scaling of HAD
// 使用SAD和SATD之间的最小值作为代价标准
// SAD按2缩放以与HAD的缩放对齐
Distortion minSadHad =
std::min(distParamSad.distFunc(distParamSad) * 2, distParamHad.distFunc(distParamHad));
// NB xFracModeBitsIntra will not affect the mode for chroma that may have already been
// pre-estimated.
m_CABACEstimator->getCtx() = SubCtx(Ctx::MipFlag, ctxStartMipFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::ISPMode, ctxStartIspMode);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaPlanarFlag, ctxStartPlanarFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaMpmFlag, ctxStartIntraMode);
m_CABACEstimator->getCtx() = SubCtx(Ctx::MultiRefLineIdx, ctxStartMrlIdx);
uint64_t fracModeBits = xFracModeBitsIntra(pu, mode, CHANNEL_TYPE_LUMA);
// 计算第二轮SATD每一种模式的cost
double cost = (double) minSadHad + (double) fracModeBits * sqrtLambdaForFirstPass;
updateCandList(ModeInfo(false, false, 0, NOT_INTRA_SUBPARTITIONS, mode), cost, uiRdModeList,
CandCostList, numModesForFullRD);
updateCandList(ModeInfo(false, false, 0, NOT_INTRA_SUBPARTITIONS, mode), double(minSadHad),
uiHadModeList, CandHadList, numHadCand);
bSatdChecked[mode] = true;
}
}
}
}//for (int modeIdx = 0; modeIdx < numModesForFullRD; modeIdx++)
if (saveDataForISP)
{ //为后面测试ISP保存粗选出的传统帧内预测模式
// we save the regular intra modes list
// 我们保存常规帧内模式列表
m_ispCandListHor = uiRdModeList;
}
pu.multiRefIdx = 1;
const int numMPMs = NUM_MOST_PROBABLE_MODES;//定义最可能模式的数量，即MPM列表的大小
unsigned multiRefMPM[numMPMs];//定义多参考行的MPM列表
PU::getIntraMPMs(pu, multiRefMPM);//获取该MPM列表的函数入口
// 对每个扩展的参考行进行循环
for (int mRefNum = 1; mRefNum < numOfPassesExtendRef; mRefNum++)
{
int multiRefIdx = MULTI_REF_LINE_IDX[mRefNum];
pu.multiRefIdx = multiRefIdx;//修改当前多参考行索引
{
initIntraPatternChType(cu, pu.Y(), true);//获取参考像素并滤波
}
// 对MPM列表中的6个模式进行SATD比较选择
for (int x = 1; x < numMPMs; x++)
{
uint32_t mode = multiRefMPM[x];
{
pu.intraDir[0] = mode;
initPredIntraParams(pu, pu.Y(), sps);
predIntraAng(COMPONENT_Y, piPred, pu);
// Use the min between SAD and SATD as the cost criterion
// SAD is scaled by 2 to align with the scaling of HAD
Distortion minSadHad =
std::min(distParamSad.distFunc(distParamSad) * 2, distParamHad.distFunc(distParamHad));
// NB xFracModeBitsIntra will not affect the mode for chroma that may have already been pre-estimated.
m_CABACEstimator->getCtx() = SubCtx(Ctx::MipFlag, ctxStartMipFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::ISPMode, ctxStartIspMode);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaPlanarFlag, ctxStartPlanarFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaMpmFlag, ctxStartIntraMode);
m_CABACEstimator->getCtx() = SubCtx(Ctx::MultiRefLineIdx, ctxStartMrlIdx);
uint64_t fracModeBits = xFracModeBitsIntra(pu, mode, CHANNEL_TYPE_LUMA);
double cost = (double) minSadHad + (double) fracModeBits * sqrtLambdaForFirstPass;
//更新候选模式列表
updateCandList(ModeInfo(false, false, multiRefIdx, NOT_INTRA_SUBPARTITIONS, mode), cost, uiRdModeList,
CandCostList, numModesForFullRD);
updateCandList(ModeInfo(false, false, multiRefIdx, NOT_INTRA_SUBPARTITIONS, mode), double(minSadHad),
uiHadModeList, CandHadList, numHadCand);
}
}//for (int x = 1; x < numMPMs; x++)
}//for (int mRefNum = 1; mRefNum < numOfPassesExtendRef; mRefNum++)
CHECKD(uiRdModeList.size() != numModesForFullRD, "Error: RD mode list size");
//为下次运行保存不同的集
if (LFNSTSaveFlag && testMip
&& !allowLfnstWithMip(cu.firstPU->lumaSize())) // save a different set for the next run
{
// save found best modes
m_uiSavedRdModeListLFNST = uiRdModeList;
m_dSavedModeCostLFNST = CandCostList;
// PBINTRA fast
m_uiSavedHadModeListLFNST = uiHadModeList;
m_dSavedHadListLFNST = CandHadList;
m_uiSavedNumRdModesLFNST =
g_aucIntraModeNumFast_UseMPM_2D[uiWidthBit - MIN_CU_LOG2][uiHeightBit - MIN_CU_LOG2];
m_uiSavedRdModeListLFNST.resize(m_uiSavedNumRdModesLFNST);
m_dSavedModeCostLFNST.resize(m_uiSavedNumRdModesLFNST);
// PBINTRA fast
m_uiSavedHadModeListLFNST.resize(3);
m_dSavedHadListLFNST.resize(3);
LFNSTSaveFlag = false;
}
//*** Derive MIP candidates using Hadamard
//在两轮传统角度模式的SATD以及MPM的SATD结束以后，对使用哈达玛的MIP模式单独进行候选模式的推导
if (testMip && !supportedMipBlkSize)
{
// avoid estimation for unsupported blk sizes
// 避免估计不支持的块大小
const int transpOff = getNumModesMip(pu.Y());
const int numModesFull = (transpOff << 1);
for (uint32_t uiModeFull = 0; uiModeFull < numModesFull; uiModeFull++)
{
const bool isTransposed = (uiModeFull >= transpOff ? true : false);
const uint32_t uiMode = (isTransposed ? uiModeFull - transpOff : uiModeFull);
numModesForFullRD++;
uiRdModeList.push_back(ModeInfo(true, isTransposed, 0, NOT_INTRA_SUBPARTITIONS, uiMode));
CandCostList.push_back(0);
}
}
else if (testMip)
{
cu.mipFlag = true;
pu.multiRefIdx = 0;
double mipHadCost[MAX_NUM_MIP_MODE] = { MAX_DOUBLE };
// 初始化Mip模式
initIntraPatternChType(cu, pu.Y());
initIntraMip(pu, pu.Y());
// getNumModesMip函数根据当前CU的块尺寸获取不同的Mip模式数量16 8 6
const int transpOff = getNumModesMip(pu.Y());
const int numModesFull = (transpOff << 1);//MIP模式总数分别是32 16 12
for (uint32_t uiModeFull = 0; uiModeFull < numModesFull; uiModeFull++)
{
//对于另一半MIP模式，其需要进行转置
const bool isTransposed = (uiModeFull >= transpOff ? true : false);
const uint32_t uiMode = (isTransposed ? uiModeFull - transpOff : uiModeFull);
pu.mipTransposedFlag = isTransposed;
pu.intraDir[CHANNEL_TYPE_LUMA] = uiMode;
// Mip预测模式的函数入口，进行MIP预测
predIntraMip(COMPONENT_Y, piPred, pu);
// Use the min between SAD and HAD as the cost criterion
// SAD is scaled by 2 to align with the scaling of HAD
Distortion minSadHad =
std::min(distParamSad.distFunc(distParamSad) * 2, distParamHad.distFunc(distParamHad));
m_CABACEstimator->getCtx() = SubCtx(Ctx::MipFlag, ctxStartMipFlag);
uint64_t fracModeBits = xFracModeBitsIntra(pu, uiMode, CHANNEL_TYPE_LUMA);
double cost = double(minSadHad) + double(fracModeBits) * sqrtLambdaForFirstPass;
mipHadCost[uiModeFull] = cost;
DTRACE(g_trace_ctx, D_INTRA_COST, "IntraMIP: %u, %llu, %f (%d)\n", minSadHad, fracModeBits, cost,
uiModeFull);
// 更新候选列表，将最优的Mip模式加入到RDCost的候选列表中
updateCandList(ModeInfo(true, isTransposed, 0, NOT_INTRA_SUBPARTITIONS, uiMode), cost, uiRdModeList,
CandCostList, numModesForFullRD + 1);
updateCandList(ModeInfo(true, isTransposed, 0, NOT_INTRA_SUBPARTITIONS, uiMode),
0.8 * double(minSadHad), uiHadModeList, CandHadList, numHadCand);
}
// 在总共四轮的STAD之后，最终更新好RDcost列表后，还要对该列表进行一个缩减的处理
const double thresholdHadCost = 1.0 + 1.4 / sqrt((double) (pu.lwidth() * pu.lheight()));
reduceHadCandList(uiRdModeList, CandCostList, numModesForFullRD, thresholdHadCost, mipHadCost, pu,
fastMip);
}
//如果SPS层开启MIP且需要为LFNST保存粗选结果
if (sps.getUseMIP() && LFNSTSaveFlag)
{
// save found best modes
m_uiSavedNumRdModesLFNST = numModesForFullRD;
m_uiSavedRdModeListLFNST = uiRdModeList;
m_dSavedModeCostLFNST = CandCostList;
// PBINTRA fast
m_uiSavedHadModeListLFNST = uiHadModeList;
m_dSavedHadListLFNST = CandHadList;
LFNSTSaveFlag = false;
}
}
else // if( sps.getUseMIP() && LFNSTLoadFlag)
{//SPS层开启MIP且需要为LFNST直接加载粗选结果
// restore saved modes
numModesForFullRD = m_uiSavedNumRdModesLFNST;
uiRdModeList = m_uiSavedRdModeListLFNST;
CandCostList = m_dSavedModeCostLFNST;
// PBINTRA fast
uiHadModeList = m_uiSavedHadModeListLFNST;
CandHadList = m_dSavedHadListLFNST;
}
//获取FastUDI可能的MPM列表，如果满足条件也将其加入RD候选列表
if (m_pcEncCfg->getFastUDIUseMPMEnabled())
{
const int numMPMs = NUM_MOST_PROBABLE_MODES;
unsigned uiPreds[numMPMs];
pu.multiRefIdx = 0;
//获取MPM列表中最可能的模式,然后只对MPM中最可能的那几个模式进行SATD粗选，其余模式不进行
const int numCand = PU::getIntraMPMs(pu, uiPreds);
for (int j = 0; j < numCand; j++)
{
bool mostProbableModeIncluded = false;
ModeInfo mostProbableMode( false, false, 0, NOT_INTRA_SUBPARTITIONS, uiPreds[j] );
for (int i = 0; i < numModesForFullRD; i++)
{
mostProbableModeIncluded |= (mostProbableMode == uiRdModeList[i]);
}
if (!mostProbableModeIncluded)
{
numModesForFullRD++;
uiRdModeList.push_back(mostProbableMode);
CandCostList.push_back(0);
}
}
if (saveDataForISP)
{
// we add the MPMs to the list that contains only regular intra modes
// 我们将MPMs添加到仅包含常规帧内模式的列表中
for (int j = 0; j < numCand; j++)
{
bool mostProbableModeIncluded = false;
ModeInfo mostProbableMode(false, false, 0, NOT_INTRA_SUBPARTITIONS, uiPreds[j]);
for (int i = 0; i < m_ispCandListHor.size(); i++)
{
mostProbableModeIncluded |= (mostProbableMode == m_ispCandListHor[i]);
}
if (!mostProbableModeIncluded)
{
m_ispCandListHor.push_back(mostProbableMode);
}
}
}
}//End if(m_pcEncCfg->getFastUDIUseMPMEnabled())
}//End if (numModesForFullRD != numModesAvailable)
else
{
THROW("Full search not supported for MIP");
}
if (sps.getUseLFNST() && mtsUsageFlag == 1)//为进行MTS变换检查保存RD模式列表
{
// Store the modes to be checked with RD
// 存储要进行RD Cost检查的模式
m_savedNumRdModes[lfnstIdx] = numModesForFullRD;
std::copy_n(uiRdModeList.begin(), numModesForFullRD, m_savedRdModeList[lfnstIdx]);
}
}//End if(mtsUsageFlag != 2)
else // mtsUsage = 2 (here we potentially reduce the number of modes that will be full-RD checked)
{
if ((m_pcEncCfg->getUseFastLFNST() || !cu.slice->isIntra()) && m_bestModeCostValid[lfnstIdx])
{ //LFNST快速算法或者是B帧或P帧
numModesForFullRD = 0;
double thresholdSkipMode = 1.0 + ((cu.lfnstIdx > 0) ? 0.1 : 1.0) * (1.4 / sqrt((double) (width * height)));
// Skip checking the modes with much larger R-D cost than the best mode
// 跳过检查RD Cost比最佳模式大得多的模式
for (int i = 0; i < m_savedNumRdModes[lfnstIdx]; i++)
{
if (m_modeCostStore[lfnstIdx][i] <= thresholdSkipMode * m_bestModeCostStore[lfnstIdx])
{
uiRdModeList.push_back(m_savedRdModeList[lfnstIdx][i]);
numModesForFullRD++;
}
}
}
else // this is necessary because we skip the candidates list calculation, since it was already obtained for
// the DCT-II. Now we load it
{ //这里的操作是必要的，因为已经从DCT - 2中获得了计算数据，因此这里跳过了候选列表的计算，现在我们加载它
// Restore the modes to be checked with RD
numModesForFullRD = m_savedNumRdModes[lfnstIdx];
uiRdModeList.resize(numModesForFullRD);
std::copy_n(m_savedRdModeList[lfnstIdx], m_savedNumRdModes[lfnstIdx], uiRdModeList.begin());
CandCostList.resize(numModesForFullRD);
}
}
CHECK(numModesForFullRD != uiRdModeList.size(), "Inconsistent state!");
// after this point, don't use numModesForFullRD
//在这一点之后，不要使用numModesForFullRD
// PBINTRA fast
// PBINTRA fast快速帧内模式
if (m_pcEncCfg->getUsePbIntraFast() && !cs.slice->isIntra() && uiRdModeList.size() < numModesAvailable
&& !cs.slice->getDisableSATDForRD() && (mtsUsageFlag != 2 || lfnstIdx > 0))
{
double pbintraRatio = (lfnstIdx > 0) ? 1.25 : PBINTRA_RATIO;
int maxSize = -1;
ModeInfo bestMipMode;
int bestMipIdx = -1;
for (int idx = 0; idx < uiRdModeList.size(); idx++)
{
if (uiRdModeList[idx].mipFlg)
{
bestMipMode = uiRdModeList[idx];
bestMipIdx = idx;
break;
}
}
const int numHadCand = 3;
for (int k = numHadCand - 1; k >= 0; k--)
{
if (CandHadList.size() < (k + 1) || CandHadList[k] > cs.interHad * pbintraRatio)
{
maxSize = k;
}
}
if (maxSize > 0)
{
uiRdModeList.resize(std::min<size_t>(uiRdModeList.size(), maxSize));
if (bestMipIdx >= 0)
{
if (uiRdModeList.size() <= bestMipIdx)
{
uiRdModeList.push_back(bestMipMode);
}
}
if (saveDataForISP)
{
m_ispCandListHor.resize(std::min<size_t>(m_ispCandListHor.size(), maxSize));
}
}
if (maxSize == 0)
{
cs.dist = std::numeric_limits<Distortion>::max();
cs.interHad = 0;
//===== reset context models =====
m_CABACEstimator->getCtx() = SubCtx(Ctx::MipFlag, ctxStartMipFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::ISPMode, ctxStartIspMode);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaPlanarFlag, ctxStartPlanarFlag);
m_CABACEstimator->getCtx() = SubCtx(Ctx::IntraLumaMpmFlag, ctxStartIntraMode);
m_CABACEstimator->getCtx() = SubCtx(Ctx::MultiRefLineIdx, ctxStartMrlIdx);
return false;
}
}
}
int numNonISPModes = (int)uiRdModeList.size();//非ISP模式的数量
if ( testISP )
{
// we reserve positions for ISP in the common full RD list
// 我们为ISP在通用的完整RD列表中保留位置
const int maxNumRDModesISP = sps.getUseLFNST() ? 16 * NUM_LFNST_NUM_PER_SET : 16;
m_curIspLfnstIdx = 0;
for (int i = 0; i < maxNumRDModesISP; i++)
{
uiRdModeList.push_back( ModeInfo( false, false, 0, INTRA_SUBPARTITIONS_RESERVED, 0 ) );
}
}
//===== check modes (using r-d costs) =====
//===== 使用RDcost检查预测模式 =====
ModeInfo uiBestPUMode;//定义当前PU最优的亮度预测帧内模式
int bestBDPCMMode = 0;
double bestCostNonBDPCM = MAX_DOUBLE;//定义再不使用BDPCM模式的时候的最优代价
//临时缓存CS
CodingStructure *csTemp = m_pTempCS[gp_sizeIdxInfo->idxFrom( cu.lwidth() )][gp_sizeIdxInfo->idxFrom( cu.lheight() )];
//最好的模式缓存CS
CodingStructure *csBest = m_pBestCS[gp_sizeIdxInfo->idxFrom( cu.lwidth() )][gp_sizeIdxInfo->idxFrom( cu.lheight() )];
csTemp->slice = cs.slice;
csBest->slice = cs.slice;
csTemp->initStructData();
csBest->initStructData();
csTemp->picture = cs.picture;
csBest->picture = cs.picture;
// just to be sure
numModesForFullRD = ( int ) uiRdModeList.size();
TUIntraSubPartitioner subTuPartitioner( partitioner );
if ( testISP )
{
m_modeCtrl->setIspCost( MAX_DOUBLE );
m_modeCtrl->setMtsFirstPassNoIspCost( MAX_DOUBLE );
}
int bestLfnstIdx = cu.lfnstIdx;
/********************这里才进入到最终对于RD候选列表中的模式的进一步精选RDcost操作***********/
for (int mode = isSecondColorSpace ? 0 : -2 * int(testBDPCM); mode < (int)uiRdModeList.size(); mode++)
{
// set CU/PU to luma prediction mode
// 首先设置亮度CU/PU的预测模式
ModeInfo uiOrgMode;
if (sps.getUseColorTrans() && !m_pcEncCfg->getRGBFormatFlag() && isSecondColorSpace && mode)
{
continue;
}
if (mode < 0 || (isSecondColorSpace && m_savedBDPCMModeFirstColorSpace[m_savedRdModeIdx][mode]))
{//如果是BDPCM模式
cu.bdpcmMode = mode < 0 ? -mode : m_savedBDPCMModeFirstColorSpace[m_savedRdModeIdx][mode];
uiOrgMode = ModeInfo( false, false, 0, NOT_INTRA_SUBPARTITIONS, cu.bdpcmMode == 2 ? VER_IDX : HOR_IDX );
}
else
{
cu.bdpcmMode = 0;
uiOrgMode = uiRdModeList[mode];
}
//如果是ISP模式，
if (!cu.bdpcmMode && uiRdModeList[mode].ispMod == INTRA_SUBPARTITIONS_RESERVED)
{
if (mode == numNonISPModes) // the list needs to be sorted only once列表只需排序一次
{
if (m_pcEncCfg->getUseFastISP())
{
m_modeCtrl->setBestPredModeDCT2(uiBestPUMode.modeId);
}
if (!xSortISPCandList(bestCurrentCost, csBest->cost, uiBestPUMode))//排列ISP模式列表
{
break;
}
}
//获取下一个ISP模式
xGetNextISPMode(uiRdModeList[mode], (mode > 0 ? &uiRdModeList[mode - 1] : nullptr), Size(width, height));
if (uiRdModeList[mode].ispMod == INTRA_SUBPARTITIONS_RESERVED)
{
continue;
}
cu.lfnstIdx = m_curIspLfnstIdx;
uiOrgMode = uiRdModeList[mode];
}
cu.mipFlag = uiOrgMode.mipFlg;//当前候选模式是否是Mip模式
pu.mipTransposedFlag = uiOrgMode.mipTrFlg;//当前候选模式是否是ISP模式
cu.ispMode = uiOrgMode.ispMod;
pu.multiRefIdx = uiOrgMode.mRefId;//当前候选模式是否是使用MRL
pu.intraDir[CHANNEL_TYPE_LUMA] = uiOrgMode.modeId;//当前候选模式放入intraDir
CHECK(cu.mipFlag && pu.multiRefIdx, "Error: combination of MIP and MRL not supported");
CHECK(pu.multiRefIdx && (pu.intraDir[0] == PLANAR_IDX), "Error: combination of MRL and Planar mode not supported");
CHECK(cu.ispMode && cu.mipFlag, "Error: combination of ISP and MIP not supported");
CHECK(cu.ispMode && pu.multiRefIdx, "Error: combination of ISP and MRL not supported");
CHECK(cu.ispMode&& cu.colorTransform, "Error: combination of ISP and ACT not supported");
pu.intraDir[CHANNEL_TYPE_CHROMA] = cu.colorTransform ? DM_CHROMA_IDX : pu.intraDir[CHANNEL_TYPE_CHROMA];
// set context models
// 设置上下文模型
m_CABACEstimator->getCtx() = ctxStart;
// determine residual for partition
// 确定分区的残差
cs.initSubStructure( *csTemp, partitioner.chType, cs.area, true );
bool tmpValidReturn = false;
if( cu.ispMode )//如果是ISP模式
{
if ( m_pcEncCfg->getUseFastISP() )
{
m_modeCtrl->setISPWasTested(true);
}
tmpValidReturn = xIntraCodingLumaISP(*csTemp, subTuPartitioner, bestCurrentCost);
if (csTemp->tus.size() == 0)
{
// no TUs were coded
csTemp->cost = MAX_DOUBLE;
continue;
}
// we save the data for future tests
// 我们保存数据以备将来的测试
m_ispTestedModes[m_curIspLfnstIdx].setModeResults((ISPType)cu.ispMode, (int)uiOrgMode.modeId, (int)csTemp->tus.size(), csTemp->cus[0]->firstTU->cbf[COMPONENT_Y] ? csTemp->cost : MAX_DOUBLE, csBest->cost);
csTemp->cost = !tmpValidReturn ? MAX_DOUBLE : csTemp->cost;
}
else
{
if (cu.colorTransform)//ACT变换
{
tmpValidReturn = xRecurIntraCodingACTQT(*csTemp, partitioner, mtsCheckRangeFlag, mtsFirstCheckId, mtsLastCheckId, moreProbMTSIdxFirst);
}
else
{
tmpValidReturn = xRecurIntraCodingLumaQT(
*csTemp, partitioner, uiBestPUMode.ispMod ? bestCurrentCost : MAX_DOUBLE, -1, TU_NO_ISP,
uiBestPUMode.ispMod, mtsCheckRangeFlag, mtsFirstCheckId, mtsLastCheckId, moreProbMTSIdxFirst);
}
}
if (!cu.ispMode && !cu.mtsFlag && !cu.lfnstIdx && !cu.bdpcmMode && !pu.multiRefIdx && !cu.mipFlag && testISP)
{
m_regIntraRDListWithCosts.push_back( ModeInfoWithCost( cu.mipFlag, pu.mipTransposedFlag, pu.multiRefIdx, cu.ispMode, uiOrgMode.modeId, csTemp->cost ) );
}
//如果当前CU是ISP模式，且第一块TU的CBF为0（无残差）
if( cu.ispMode && !csTemp->cus[0]->firstTU->cbf[COMPONENT_Y] )
{
csTemp->cost = MAX_DOUBLE;
csTemp->costDbOffset = 0;
tmpValidReturn = false;
}
validReturn |= tmpValidReturn;
//如果SPS层开启LFNST且可以使用MTS并正在检查DCT-2，将当前RD Cost保存下来以在检查MTS时使用快速算法
if( sps.getUseLFNST() && mtsUsageFlag == 1 && !cu.ispMode && mode >= 0 )
{
m_modeCostStore[lfnstIdx][mode] = tmpValidReturn ? csTemp->cost : (MAX_DOUBLE / 2.0); //(MAX_DOUBLE / 2.0) ??
}
DTRACE(g_trace_ctx, D_INTRA_COST, "IntraCost T [x=%d,y=%d,w=%d,h=%d] %f (%d,%d,%d,%d,%d,%d) \n", cu.blocks[0].x,
cu.blocks[0].y, (int) width, (int) height, csTemp->cost, uiOrgMode.modeId, uiOrgMode.ispMod,
pu.multiRefIdx, cu.mipFlag, cu.lfnstIdx, cu.mtsFlag);
if( tmpValidReturn )//如果是有效返回值，比较最优模式代价和当前模式代价
{
if (isFirstColorSpace)
{
if (m_pcEncCfg->getRGBFormatFlag() || !cu.ispMode)
{
sortRdModeListFirstColorSpace(uiOrgMode, csTemp->cost, cu.bdpcmMode, m_savedRdModeFirstColorSpace[m_savedRdModeIdx], m_savedRdCostFirstColorSpace[m_savedRdModeIdx], m_savedBDPCMModeFirstColorSpace[m_savedRdModeIdx], m_numSavedRdModeFirstColorSpace[m_savedRdModeIdx]);
}
}
// check r-d cost
if( csTemp->cost < csBest->cost )//如果当前模式的cost小于目前最优的模式的cost
{
std::swap( csTemp, csBest );//当前模式成为最新的最优模式
uiBestPUMode = uiOrgMode;
bestBDPCMMode = cu.bdpcmMode;
if( sps.getUseLFNST() && mtsUsageFlag == 1 && !cu.ispMode )
{
m_bestModeCostStore[ lfnstIdx ] = csBest->cost; //cs.cost;
m_bestModeCostValid[ lfnstIdx ] = true;
}
if( csBest->cost < bestCurrentCost )
{
bestCurrentCost = csBest->cost;
}
if ( cu.ispMode )//如果当前模式时ISP模式，保存最优的ISP模式和相应的lfnstIdx
{
m_modeCtrl->setIspCost(csBest->cost);
bestLfnstIdx = cu.lfnstIdx;
}
else if ( testISP )//如果需要测试ISP，则保存MTS的第一个流程中非ISP模式的Cost
{
m_modeCtrl->setMtsFirstPassNoIspCost(csBest->cost);
}
}
if( !cu.ispMode && !cu.bdpcmMode && csBest->cost < bestCostNonBDPCM )
{
bestCostNonBDPCM = csBest->cost;
}
}//End if( tmpValidReturn )
csTemp->releaseIntermediateData(); // 清空CU, PU, TU
if( m_pcEncCfg->getFastLocalDualTreeMode() )
{
if( cu.isConsIntra() && !cu.slice->isIntra() && csBest->cost != MAX_DOUBLE && costInterCU != COST_UNKNOWN && mode >= 0 )
{
if( m_pcEncCfg->getFastLocalDualTreeMode() == 2 )
{
//Note: only try one intra mode, which is especially useful to reduce EncT for LDB case (around 4%)
//注意：仅尝试一种帧内模式，这对于减少LDB情况下的EncT尤其有用（大约4％）
break;
}
else
{
if( csBest->cost > costInterCU * 1.5 )
{
break;
}
}
}
}
if (sps.getUseColorTrans() && !CS::isDualITree(cs))
{
if ((m_pcEncCfg->getRGBFormatFlag() && !cu.colorTransform) && csBest->cost != MAX_DOUBLE && bestCS->cost != MAX_DOUBLE && mode >= 0)
{
if (csBest->cost > bestCS->cost)
{
break;
}
}
}
} // Mode loop RD Cost模式循环
/*********RD候选模式的循环RDCost遍历结束,选出最优的模式**************/
cu.ispMode = uiBestPUMode.ispMod;//设置最优的ISP模式
cu.lfnstIdx = bestLfnstIdx;
if( validReturn )
{
if (cu.colorTransform)
{
cs.useSubStructure(*csBest, partitioner.chType, pu, true, true, KEEP_PRED_AND_RESI_SIGNALS, KEEP_PRED_AND_RESI_SIGNALS, true);
}
else
{
cs.useSubStructure(*csBest, partitioner.chType, pu.singleChan(CHANNEL_TYPE_LUMA), true, true, KEEP_PRED_AND_RESI_SIGNALS,
KEEP_PRED_AND_RESI_SIGNALS, true);
}
}
csBest->releaseIntermediateData();
if( validReturn )
{
//=== update PU data ====
//更新PU数据
cu.mipFlag = uiBestPUMode.mipFlg;
pu.mipTransposedFlag = uiBestPUMode.mipTrFlg;
pu.multiRefIdx = uiBestPUMode.mRefId;
pu.intraDir[ CHANNEL_TYPE_LUMA ] = uiBestPUMode.modeId;
cu.bdpcmMode = bestBDPCMMode;
if (cu.colorTransform)
{
CHECK(pu.intraDir[CHANNEL_TYPE_CHROMA] != DM_CHROMA_IDX, "chroma should use DM mode for adaptive color transform");
}
}//End if( validReturn )
}//代码块
//===== reset context models =====
//===== 重置上下文模型 =====
m_CABACEstimator->getCtx() = ctxStart;
return validReturn;
}感谢H师兄的博客（狗头保命），多亏了师兄的博客我才能勉强看懂这个函数（顺带吐槽一下写VTM代码的人，真是连注释都不给，变量作用全靠猜）。
扫一扫
2858
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
