背景
直播SDK接入iOS ARKit。
WebRTC数据传入
创建PeerConnection的时候需要创建一个VideoTrackSourceInterface对象,该对象可以作为外部视频数据传入WebRTC的入口。
ARKit数据来源
ARKit有两种数据,一种是直接从摄像头采集到的原始数据,一种是经过AR渲染的数据,也就是我们直观看到的增强现实的图像。无论是哪种数据,都要经过CVPixelBuffer ==> RTCCVPixelBuffer ==> RTCVideoFrame的转换,才能交给VideoTrackSourceInterface,进而通过WebRTC推流。
摄像头原始数据
实现ARSessionDelegate协议的以下方法:
……
RTCVideoCapturer *_dummyCapturer = [[RTCVideoCapturer alloc] init];
……
- (void)session:(ARSession *)session didUpdateFrame:(ARFrame *)frame {
CVPixelBufferRef pixelBuffer = frame.capturedImage; // 获得CVPixelBufferRef
RTCCVPixelBuffer *rtcPixelBuffer = [[RTCCVPixelBuffer alloc] initWithPixelBuffer:pixelBuffer]; // 转成RTCCVPixelBuffer.
int64_t timeStampNs = frame.timestamp * 1000000000; // 时间单位转换,单位为ns.
RTCVideoRotation rotation = RTCVideoRotation_0; //TBD, check rotation.
RTCVideoFrame *videoFrame = [[RTCVideoFrame alloc] initWithBuffer:rtcPixelBuffer rotation:rotation timeStampNs:timeStampNs]; //转成RTCVideoFrame.
[_videoSource capturer:_dummyCapturer didCaptureVideoFrame:videoFrame]; //发送数据给WebRTC, _dummyCapturer弄个假的即可。
}
渲染后的AR数据
需要创建一个SCNRenderer,以一定的帧率调用snapshotAtTime抓取场景。由于该方法返回的是UIImage,需要先把UIImage转成CVPixelBufferRef,才能使用上述方法发送数据到WebRTC。代码示例如下:
#define screenWidth [UIScreen mainScreen].bounds.size.width
#define screenHeight [UIScreen mainScreen].bounds.size.height
……
ARSCNView *_arView;
ARSession *_arSession;
SCNRenderer *_scnRenderer;
……
- (ARSession *)arSession {
if(_arSession == nil) {
_arSession = [[ARSession alloc] init];
_arSession.delegate = self;
}
return _arSession;
}
- (ARSCNView *)arView {
if (_arView == nil) {
_arView = [[ARSCNView alloc] initWithFrame:CGRectMake(
0,
0,
screenWidth,
screenHeight)];
_arView.session = self.arSession;
_arView.automaticallyUpdatesLighting = YES;
_arView.delegate = self;
}
return _arView;
}
- (SCNRenderer*)scnRenderer {
if (_scnRenderer == nil) {
_scnRenderer = [SCNRenderer rendererWithDevice:nil options:nil];
_scnRenderer.scene = self.arView.scene;
}
return _scnRenderer;
}
-(CVPixelBufferRef)capturePixelBuffer : (NSTimeInterval)timestamp {
UIImage *image = [self.scnRenderer snapshotAtTime:timestamp withSize:CGSizeMake(_outputSize.width, _outputSize.height) antialiasingMode:SCNAntialiasingModeMultisampling4X];
CVPixelBufferRef pixelBuffer = [self imageToPixelBuffer:image.CGImage]; // UIImage ==> CVPixelBuffer.
return pixelBuffer;
}
//CGImageRef ==> CVPixelBufferRef,也就是UIImage ==> CVPixelBuffer.
- (CVPixelBufferRef)imageToPixelBuffer:(CGImageRef)image {
CGSize frameSize = CGSizeMake(_outputSize.width, _outputSize.height); // _outputSize = CGSizeMake(720, 1280), 分辨率越高,转换消耗越高.
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithBool:YES],kCVPixelBufferCGImageCompatibilityKey,[NSNumber numberWithBool:YES],kCVPixelBufferCGBitmapContextCompatibilityKey,nil];
CVPixelBufferRef pxbuffer = NULL;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, frameSize.width, frameSize.height,kCVPixelFormatType_32BGRA, (__bridge CFDictionaryRef)options, &pxbuffer);
NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, frameSize.width, frameSize.height,8, CVPixelBufferGetBytesPerRow(pxbuffer),rgbColorSpace,(CGBitmapInfo)kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CGContextDrawImage(context, CGRectMake(0, 0, _outputSize.width, _outputSize.height), image);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}
- (void)captureAndSend {
CVPixelBufferRef pixelBuffer = [self capturePixelBuffer:[self getCurrentTimestamp]]; //获取当前时间戳的图像
RTCCVPixelBuffer *rtcPixelBuffer = [[RTCCVPixelBuffer alloc] initWithPixelBuffer:pixelBuffer]; // 转成RTCCVPixelBuffer.
int64_t timeStampNs = frame.timestamp * 1000000000; // 时间单位转换,单位为ns.
RTCVideoRotation rotation = RTCVideoRotation_0; //TBD, check rotation.
RTCVideoFrame *videoFrame = [[RTCVideoFrame alloc] initWithBuffer:rtcPixelBuffer rotation:rotation timeStampNs:timeStampNs]; //转成RTCVideoFrame.
[_videoSource capturer:_dummyCapturer didCaptureVideoFrame:videoFrame]; //发送数据给WebRTC, _dummyCapturer弄个假的即可。
}
在一个定时器里周期地调用captureAndSend即可。
原始AR Demo
https://github.com/miliPolo/ARSolarPlay
把这个流通过WebRTC推出去,还是蛮有意思的,AR+WebRTC,也许是一个值得尝试的方向。