连续多帧图像光流对齐和光流运动检测

光流运动检测

本方案适用于手持设备拍摄连续多帧拍摄,  使用光流检测场景运动信息.
1. 由于设备或者手抖动, 产生画面的整体偏移--需要先对齐不同的帧.
2. 为了计算性能, 将原始输入图, resize到较小尺寸(此处采用320x200).
3. 从大图直接resize到小图容易产生明显的摩尔纹, 此处采用三级下采样, 有效避免摩尔纹的干扰.

主要流程

1. 预测多帧图像的相对清晰度
   通过sobel算子,计算不同帧的图像平均梯度值sobelValue, 代表相对清晰度.sobelValue最大的作为Reference, 并根据sobelValue利用统计分析, 标出不可用的BlurImg.

2. 光流对齐
   Reference下采样到目标Size(OPT_W, OPT_H).
   遍历其他帧(除了Ref img和Blur img), 同样下采样到Size(OPT_W, OPT_H). 进行光流对齐

	//Read Reference frame from pBuffers[refIdx], and resize to target Size
	cv::Mat smallRef, resize1, resize2;
	cv::Mat inputRef(cropheight, cropwidth, CV_8UC1, (uint8_t*)pBuffers[refIdx]);

	cv::resize(inputRef, resize1, cv::Size(OPT_W*4, OPT_H*4), cv::INTER_CUBIC);
	cv::resize(resize1, resize2, cv::Size(OPT_W*2, OPT_H*2), cv::INTER_CUBIC);
	cv::resize(resize2, smallRef, cv::Size(OPT_W, OPT_H), cv::INTER_CUBIC);
	
	//Buffers for storing aligned small image
	uint8_t* pSmallBuffers[7];
	for(int k=0;k<burstImgNum;k++) {
		pSmallBuffers[k] = (uint8_t*)malloc(OPT_W*OPT_H);
		if(!pSmallBuffers[k] )
			return;
	}

	//Align the smallNoise to the smallRef, and store it in the alignSmall.
	for(int imgId = 0; imgId < burstImgNum; imgId++) {
		if(imgId == refIdx || imgIsBlur[imgId] == true) continue;
		
		cv::Mat smallNoise;
		cv::Mat inputNoise(cropheight, cropwidth, CV_8UC1, (uint8_t*)pBuffers[imgId]);
		cv::Mat alignSmall(OPT_H, OPT_W, CV_8UC1, (uint8_t*)pSmallBuffers[imgId]);
		
		cv::resize(inputNoise, resize1, cv::Size(OPT_W*4, OPT_H*4), cv::INTER_CUBIC);
		cv::resize(resize1, resize2, cv::Size(OPT_W*2, OPT_H*2), cv::INTER_CUBIC);
		cv::resize(resize2, smallNoise, cv::Size(OPT_W, OPT_H), cv::INTER_CUBIC);

		alignWithDenseOpticalFlow(smallNoise, smallRef, alignSmall);
		}
		

	光流对齐部分代码alignWithDenseOpticalFlow

void alignWithDenseOpticalFlow(cv::Mat& imMov, cv::Mat& imRef, cv::Mat& imAli)
{
	Mat matFlow; 
	//cv::Mat visFlow(cv::Size(imRef.cols, imRef.rows),CV_8UC1);
	
	//Get matFlow from openCV api.
	calcOpticalFlowFarneback(imRef, imMov, matFlow, 0.5, 3, 15, 3, 5, 1.2, 0);
	
	//Sample 100 points uniformly from the middle part of the optical matFlow.
	std::vector<float> sort_x;
	std::vector<float> sort_y;
	for(int y=30; y<150; y+=12){
		for(int x=40; x<280; x+=24){
			sort_x.push_back(matFlow.at<cv::Vec2f>(y, x)[0]);
			sort_y.push_back(matFlow.at<cv::Vec2f>(y, x)[1]);
			//visFlow.at<uchar>(y, x) = 255;
		}
	}
	
	//Sort the sampling points, select the middle 20 points, calculate the average value, use it as the offset vector of the whole image.
	std::sort(sort_x.begin(), sort_x.end());
	std::sort(sort_y.begin(), sort_y.end());

	float sum_x = 0.0, sum_y = 0.0;
	for(int i=40; i<60; i++){
		sum_x += sort_x.at(i);
		sum_y += sort_y.at(i);
	}
	sum_x /= 20.0;
	sum_y /= 20.0;
	
	//Map imMov to imAli according to the offset vector.
	Mat map(matFlow.size(), CV_32FC2);
	for (int y = 0; y < map.rows; ++y)
	{
	    for (int x = 0; x < map.cols; ++x)
	    {
	        map.at<Point2f>(y, x) = Point2f(x + sum_x, y + sum_y);
	    }
	}

	Mat mapping[2];
	split(map, mapping);
	remap(imMov, imAli, mapping[0], mapping[1], INTER_CUBIC);//INTER_LINEAR

	// static int save_id = 0;
	// std::string save_name = "/sdcard/sr/sr_results/vismat_" + std::to_string(save_id) + ".jpg";
	// cv::imwrite(save_name, visFlow);
	// save_id++;
}

3. 光流运动场检测.
   使用opencv api --calcOpticalFlowFarneback计算稠密光流(高频场景有噪声, 感觉稀疏光流可以一试, 后话了, 先交作业…),
   由于上面的alignment, 引起图像的平移, 边缘应舍弃, 对光流图裁掉边缘.
   根据光流场, 计算每个像素位置的光流强度并缩放到可视化的范围.
   统计光流强度, 通过阈值判断运动场景.

int getMotionPixels(cv::Mat& iRefFrame, cv::Mat& iCurrFrame, float optThresh)
{
	int iCount = 0;
	cv::Mat matFlow;
	
	//Get matFlow from openCV api.
	calcOpticalFlowFarneback(iRefFrame, iCurrFrame, matFlow, 0.5, 3, 15, 3, 5, 1.2, 0);
	
	//Crop matFlow 0.1 edges
	matFlow = matFlow(Rect(iRefFrame.cols*0.1, iRefFrame.rows*0.1,  iRefFrame.cols*0.8, iRefFrame.rows*0.8));
	
	//cv::Mat visFlow(cv::Size(matFlow.cols, matFlow.rows),CV_8UC1);

	
	for (int i = 0; i < matFlow.rows; i++)
	{
		for (int j = 0; j < matFlow.cols; j++)
		{
			float tmp0 = matFlow.at<cv::Vec2f>(i, j)[0];
            float tmp1 = matFlow.at<cv::Vec2f>(i, j)[1];
            float fImp = sqrt(tmp0 * tmp0 + tmp1 * tmp1);

			//unsigned char elem = (uchar)(fImp * 51);
			//visFlow.at<uchar>(i, j) = elem < 255 ? elem : 255;

			if( fImp*51 > optThresh) iCount++;
		}
	}
	
	// static int save_id = 0;
	// std::string save_name = "/sdcard/sr/sr_results/smallFlow_" + std::to_string(save_id) + ".jpg";
	// cv::imwrite(save_name, visFlow);
	// save_id++;
	std::cout<< "***********************Motion pixels     >       "<< optThresh << " iCount: "<< iCount << endl;
	return iCount;
}

效果对比

1, 输入图4k原图, 连续4帧

2. 对齐之前

3, 去掉摩尔纹, 对齐后

4. 高频场景存在得问题. 感觉可以结合稀疏光流试试

5. 静态场景光流图

6. 动态场景, 光流图