1、大作业2题目如下:开发媒体数据处理的应用,应用的输入为视频码流、输出为JPEG图片,且JPEG 图片与视频的分辨率不同。
根据作业提示,转换的思路如下:
媒体数据处理各功能串联时,可实现内存复用:
− 视频码流解码的输出可以作为图片缩放的输入;
− 图片缩放的输出可以作为 JPEG 图片编码的输入。
视频解码接口这一块,通过课程的学习还有资料参考,是通过VDEC视频解码获取视频流
一般使用多线程方式调用,一路线程视频解码,一路做其他操作:
各个主要接口作用:
1、调用aclvdecCreateChannel接口创建视频解码处理的通道
创建视频解码处理通道前,需先执行以下操作:
调用aclvdecCreateChannelDesc接口创建通道描述信息。
调用aclvdecSetChannelDesc系列接口设置通道描述信息的属性,包括解码通道号、线程、回调函数、视频编码协议等,其中:
回调函数需由用户提前创建,用于在视频解码后,获取解码数据,并及时释放相关资源,回调函数的原型前参见aclvdecCallback。
在回调函数内,用户需调用acldvppGetPicDescRetCode接口获取retCode返回码判断是否解码成功,retCode为0表示解码成功,为1表示解码失败。如果解码失败,需要根据日志中的返回码判断具体的问题,返回码请参见返回码说明。
解码结束后,建议用户在回调函数内及时释放VDEC的输入码流内存、输出图片内存以及相应的视频码流描述信息、图片描述信息。
aclvdecCreateChannel接口内部封装了如下接口,无需用户单独调用:
aclrtCreateStream接口:显式创建Stream,VDEC内部使用。
aclrtSubscribeReport接口:指定处理Stream上回调函数的线程,回调函数和线程是由用户调用aclvdecSetChannelDesc系列接口时指定的。
2、调用aclvdecSendFrame接口将视频码流解码成YUV420SP格式的图片。
视频解码前,需先执行以下操作:
调用acldvppCreateStreamDesc接口创建输入视频码流描述信息,并调用acldvppSetStreamDesc系列接口设置输入视频的内存地址、内存大小、码流格式等属性。
调用acldvppCreatePicDesc接口创建输出图片描述信息,并调用acldvppSetPicDesc系列接口设置输出图片的内存地址、内存大小、图片格式等属性。
aclvdecSendFrame接口内部封装了aclrtLaunchCallback接口,用于在Stream的任务队列中增加一个需要执行的回调函数。用户无需单独调用aclrtLaunchCallback接口。
如果用户需要获取解码的帧序号,则可以在aclvdecSendFrame接口的userData参数处定义,然后解码的帧序号可以通过userData参数传递给VDEC的回调函数,用于确定回调函数中处理的是第几帧数据。
3、调用aclvdecDestroyChannel接口销毁视频处理的通道
系统会等待已发送帧解码完成且用户的回调函数处理完成后再销毁通道。
aclvdecDestroyChannel接口内部封装了如下接口,无需用户单独调用:
aclrtUnSubscribeReport接口:取消线程注册(Stream上的回调函数不再由指定线程处理)。
aclrtDestroyStream接口:销毁Stream。
销毁通道后,需调用aclvdecDestroyChannelDesc接口销毁通道描述信息
2、sample参考
根据学习的内容,原来的sample仓里面,包含三个VDEC、VPC Resize、JPEGE单独的样例:
2.1 ffmpeg切帧
输入为mp4文件,切帧输出到device侧:
//intialize ffmpeg decoder
FFmpegDecoder();
ACLLITE_LOG_INFO("Get Vdec type: %d", GetVdecType());
//verify video type
if (kInvalidTpye == GetVdecType()) {
ACLLITE_LOG_ERROR("Video %s type is invalid", streamName_.c_str());
}
//Get video fps, if no fps, use 1 as default
if (fps_ == 0) {
fps_ = kDefaultFps;
ACLLITE_LOG_INFO("Video %s fps is 0, change to %d",
streamName_.c_str(), fps_);
}
Decode();
2.2 VDEC 视频解码
输入h264或者h265单帧,解码为YUV420格式,保存为yuv格式的文件
VDEC样例中关键代码实现:
线程创建:
/* 3. Vdec init */
// create threadId
pthread_create(&threadId_, nullptr, ThreadFunc, nullptr);
(void)aclrtSubscribeReport(static_cast<uint64_t>(threadId_), stream_);
创建输出图片的disc:
//Create output image description information, set the image description information properties
//picOutputDesc_ is acldvppPicDesc
picOutputDesc_ = acldvppCreatePicDesc();
ret = acldvppSetPicDescData(picOutputDesc_, picOutBufferDev_);
ret = acldvppSetPicDescSize(picOutputDesc_, DataSize);
ret = acldvppSetPicDescFormat(picOutputDesc_, static_cast<acldvppPixelFormat>(format_));
VDEC解码:
/* Perform video stream decoding. After decoding each frame of data, the system automatically
calls callback callback function to write the decoded data to the file, and then timely release
relevant resources*/
ret = aclvdecSendFrame(vdecChannelDesc_, streamInputDesc_, picOutputDesc_, nullptr, nullptr);
回调函数:
void callback(acldvppStreamDesc *input, acldvppPicDesc *output, void *userdata)
{
/*Get the output memory decoded by VDEC, call the custom function WriteToFile to write
the data in the output memory to the file, and then call the acldvppFree interface to release
the output memory*/
void *vdecOutBufferDev = acldvppGetPicDescData(output);
uint32_t size = acldvppGetPicDescSize(output);
static int count = 1;
std::string fileNameSave = "./output/image" + std::to_string(count) + ".yuv";
if (!WriteToFile(fileNameSave.c_str(), vdecOutBufferDev, size)) {
ERROR_LOG("write file failed.");
}
aclError ret = acldvppFree(reinterpret_cast<void *>(vdecOutBufferDev));
// Release acldvppPicDesc type data, representing output picture description data after decoding
ret = acldvppDestroyPicDesc(output);
count++;
}
sample运行截图,保存了10帧yuv文件:
2.3 VPC Resize
输入YUV图片,根据Resize的尺寸,保存resize之后的YUV图片,sample运行截图:
2.4 JPEGE 编码
输入yuv图片,调用编码接口,保存JPEG图片,sample运行截图:
3、ffmPeg+ VDEC + VPC Resize + JPEGE串联
根据学习的内容,按照以下思路将完整的功能串联在一起输出代码,实现内存的复用,ffmpeg切帧,将mp4视频的单帧输出,拷贝到device内存,VDEC解码后的YUV输出内存给到VPC Resize,Resize之后的内存给到JPEGE,编码之后根据内存输出地址和Buffersize,写文件图片即可。
关键实现代码如下,完整代码请看作业贴回复附件。
3.1.视频解码
主要接口:
aclvdecSendFrame视频解码
代码实现:
ret = aclvdecSendFrame(vdecChannelDesc_, streamInputDesc_, picOutputDesc_, nullptr, nullptr);
3.2. VPC Resize
acldvppCreateResizeConfig缩放配置接口
acldvppVpcResizeAsync异步接口,将输入图片缩放到输出图片大小,输出大小这里设置为640*480,可以自行修改。
代码实现:
acldvppResizeConfig *resizeConfig_ = acldvppCreateResizeConfig();
uint32_t modelInputWidth = 640;
uint32_t modelInputHeight = 480;
uint32_t sizeAlignment = 3;
uint32_t sizeNum = 2;
acldvppPicDesc *vpcOutputDesc_ = acldvppCreatePicDesc();
void *vpcOutBufferDev_ = nullptr;
acldvppSetPicDescData(vpcOutputDesc_, vpcOutBufferDev_);
acldvppSetPicDescFormat(vpcOutputDesc_, PIXEL_FORMAT_YUV_SEMIPLANAR_420);
acldvppSetPicDescWidth(vpcOutputDesc_, modelInputWidth);
acldvppSetPicDescHeight(vpcOutputDesc_, modelInputHeight);
acldvppSetPicDescWidthStride(vpcOutputDesc_, resizeOutWidthStride);
acldvppSetPicDescHeightStride(vpcOutputDesc_, resizeOutHeightStride);
acldvppSetPicDescSize(vpcOutputDesc_, vpcOutBufferSize_);
ret = acldvppVpcResizeAsync(dvppChannelDesc_, output,
vpcOutputDesc_, resizeConfig_, stream2_);
3.3. JPEGE编码
主要接口:
acldvppCreateJpegeConfig创建图片编码配置数据
acldvppJpegEncodeAsync异步编码
代码实现:
uint32_t encodeLevel = 100; // default optimal level (0-100)
acldvppJpegeConfig *jpegeConfig_;
void* encode_out_buffer_dev_;
jpegeConfig_ = acldvppCreateJpegeConfig();
//call Asynchronous api
aclRet = acldvppJpegEncodeAsync(dvppChannelDesc_, vpcOutputDesc_, encode_out_buffer_dev_,
&encode_outbuffer_size_, jpegeConfig_, stream2_);
3.4. 保存JPEG图片
根据编码输出内存地址和Buffersize,写入到JPEG图片:
int dir_tail_index = image_path.find("/data");
std::string outfile_dir = image_path.substr(0, dir_tail_index) + "/" + "out/output/";
std::string outfile_path = outfile_dir + image_path.substr(dir_tail_index+5+1, image_path.rfind(".jpg")-dir_tail_index-5-1)
+ "_jpege_" + std::to_string(modelInputWidth) + "_" + std::to_string(modelInputHeight) + ".jpg";
INFO_LOG("outfile_path=%s", outfile_path.c_str());
ret = save_dvpp_outputdata(outfile_path.c_str(), encode_out_buffer_dev_, encode_outbuffer_size_);
if (ret != SUCCESS) {
ERROR_LOG("save dvpp output data failed");
// allow not return
}
3.5. 运行效果
编译:
cd vdec_resize_jpede/
cd scripts/
bash sample_build.sh
编译Log:
[INFO] Sample preparation
please input TargetKernel? [arm/x86]:x86
[INFO] input is normal, start preparation.
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/gitee/samples/cplusplus/level2_simple_inference/0_data_process/vdec_resize_jpege/build/intermediates/host
Scanning dependencies of target main
[ 50%] Building CXX object CMakeFiles/main.dir/main.cpp.o
[100%] Linking CXX executable /home/gitee/samples/cplusplus/level2_simple_inference/0_data_process/vdec_resize_jpege/out/main
[100%] Built target main
[INFO] Sample preparation is complete
运行:
bash sample_run.sh
运行Log:
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
[INFO] create dvppChannelDesc_ success
[INFO] VPC Resized_Width=640, Resized_Hight=480, Output_Buffer_Size=460800
[INFO] Call acldvppCreateJpegeConfig success
[INFO] Call acldvppJpegEncodeAsync success
从原来的cat.mp4视频,解码输出100帧Resize之后的图片:
图片分辨率为640*480:
图片是根据ffmpeg解码的帧变化而变化: