iOS平台如何实现毫秒级延迟的RTMP|RTSP播放器

技术背景

在我的blog里面，最近很少有提到iOS平台RTMP推送|轻量级RTSP服务和RTMP|RTSP直播播放模块，实际上，我们在2016年就发布了iOS平台直播推拉流、转发模块，只是因为传统行业，对iOS的需求比较少，所以一直没单独说明，本文主要介绍下，如何在iOS平台播放RTMP或RTSP流。

技术实现

先说播放实现，iOS端，RTMP|RTSP直播播放，我们实现的功能如下：

•  [支持播放协议]高稳定、超低延迟（毫秒级）
•  [多实例播放]支持多实例播放；
•  [事件回调]支持网络状态、buffer状态等回调；
•  [视频格式]支持RTMP扩展H.265，H.264；
•  [音频格式]支持AAC/PCMA/PCMU/Speex；
•  [H.264/H.265软解码]支持H.264/H.265软解；
•  [H.264硬解码]Windows/Android/iOS支持特定机型H.264硬解；
•  [H.265硬解]Windows/Android/iOS支持特定机型H.265硬解；
•  [H.264/H.265硬解码]Android支持设置Surface模式硬解和普通模式硬解码；
•  [缓冲时间设置]支持buffer time设置；
•  [首屏秒开]支持首屏秒开模式；
•  [低延迟模式]支持低延迟模式设置(公网200~400ms)；
•  [复杂网络处理]支持断网重连等各种网络环境自动适配；
•  [快速切换URL]支持播放过程中，快速切换其他URL，内容切换更快；
•  [实时静音]支持播放过程中，实时静音/取消静音；
•  [实时音量调节]支持播放过程中实时调节音量；
•  [实时快照]支持播放过程中截取当前播放画面；
•  [渲染角度]支持0°，90°，180°和270°四个视频画面渲染角度设置；
•  [渲染镜像]支持水平反转、垂直反转模式设置；
•  [等比例缩放]支持图像等比例缩放绘制(Android设置surface模式硬解模式不支持)；
•  [实时下载速度更新]支持当前下载速度实时回调(支持设置回调时间间隔)；
•  [解码前视频数据回调]支持H.264/H.265数据回调；
•  [解码后视频数据回调]支持解码后YUV数据回调；
•  [解码前音频数据回调]支持AAC/PCMA/PCMU/SPEEX数据回调；
•  [音视频自适应]支持播放过程中，音视频信息改变后自适应；
•  [扩展录像功能]完美支持和录像SDK组合使用。

下面，我们看看技术实现细节，先说开始播放逻辑：

//
//  ViewController.m
//  SmartiOSPlayerV2
//
//  Author: daniusdk.com
//  Created by daniulive on 2016/01/03.
//
- (void)playBtn:(UIButton *)button {
    
    NSLog(@"playBtn only++");
    
    button.selected = !button.selected;
    
    if (button.selected)
    {
        if(is_playing_)
            return;
        
        [self InitPlayer];
        
        //如需处理回调的用户数据+++++++++
        __weak __typeof(self) weakSelf = self;
        
        _smart_player_sdk.spUserDataCallBack = ^(int data_type, unsigned char *data, unsigned int size, unsigned long long timestamp, unsigned long long reserve1, long long reserve2, unsigned char *reserve3)
        {
            [weakSelf OnUserDataCallBack:data_type data:data size:size timestamp:timestamp reserve1:reserve1 reserve2:reserve2 reserve3:reserve3];
        };
        
        Boolean enableUserDataCallback = YES;
        [_smart_player_sdk SmartPlayerSetUserDataCallback:enableUserDataCallback];
         //如需处理回调的用户数据---------
        
        if(![self StartPlayer])
        {
            NSLog(@"Call StartPlayer failed..");
        }
        
        [playbackButton setTitle:@"停止播放" forState:UIControlStateNormal];
        
        is_playing_ = YES;
    }
    else
    {
        if ( !is_playing_ )
            return;
        
        [self StopPlayer];
        
        if(!is_recording_)
        {
            [self UnInitPlayer];
        }
        
        [playbackButton setTitle:@"开始播放" forState:UIControlStateNormal];
        
        is_mute_ = NO;
        [muteButton setTitle:@"实时静音" forState:UIControlStateNormal];
        
        is_playing_ = NO;
    }
}

其中，InitPlayer实现如下：

-(bool)InitPlayer
{
    NSLog(@"InitPlayer++");
    
    if(is_inited_player_)
    {
        NSLog(@"InitPlayer: has inited before..");
        return true;
    }
    
    //NSString* in_cid = @"";
    //NSString* in_key = @"";
    
    //[SmartPlayerSDK SmartPlayerSetSDKClientKey:in_cid in_key:in_key reserve1:0 reserve2:nil];
    
    _smart_player_sdk = [[SmartPlayerSDK alloc] init];
    
    if (_smart_player_sdk ==nil ) {
        NSLog(@"SmartPlayerSDK init failed..");
        return false;
    }
    
    if (playback_url_.length == 0) {
        NSLog(@"playback url is nil..");
        return false;
    }
    
    if (_smart_player_sdk.delegate == nil)
    {
        _smart_player_sdk.delegate = self;
        NSLog(@"SmartPlayerSDK _player.delegate:%@", _smart_player_sdk);
    }
    
    NSInteger initRet = [_smart_player_sdk SmartPlayerInitPlayer];
    if ( initRet != DANIULIVE_RETURN_OK )
    {
        NSLog(@"SmartPlayerSDK call SmartPlayerInitPlayer failed, ret=%ld", (long)initRet);
        return false;
    }
    
    [_smart_player_sdk SmartPlayerSetPlayURL:playback_url_];
    //[self try_set_rtsp_url:playback_url_];
    
    //超低延迟模式设置
    [_smart_player_sdk SmartPlayerSetLowLatencyMode:(NSInteger)is_low_latency_mode_];
    
    //buffer time设置
    if(buffer_time_ >= 0)
    {
        [_smart_player_sdk SmartPlayerSetBuffer:buffer_time_];
    }
    
    //快速启动模式设置
    [_smart_player_sdk SmartPlayerSetFastStartup:(NSInteger)is_fast_startup_];
    
    NSLog(@"[SmartPlayerV2]is_fast_startup_:%d, buffer_time_:%ld", is_fast_startup_, (long)buffer_time_);
    
    //RTSP TCP还是UDP模式
    [_smart_player_sdk SmartPlayerSetRTSPTcpMode:is_rtsp_tcp_mode_];
 
    //设置RTSP超时时间
    NSInteger rtsp_timeout = 10;
    [_smart_player_sdk SmartPlayerSetRTSPTimeout:rtsp_timeout];
    
    //设置RTSP TCP/UDP自动切换
    NSInteger is_tcp_udp_auto_switch = 1;
    [_smart_player_sdk SmartPlayerSetRTSPAutoSwitchTcpUdp:is_tcp_udp_auto_switch];
    
    //快照设置 如需快照 参数传1
    [_smart_player_sdk SmartPlayerSaveImageFlag:save_image_flag_];
    
    //如需查看实时流量信息，可打开以下接口
    NSInteger is_report = 1;
    NSInteger report_interval = 3;
    [_smart_player_sdk SmartPlayerSetReportDownloadSpeed:is_report report_interval:report_interval];
    
    //录像端音频，是否转AAC后保存
    NSInteger is_transcode = 1;
    [_smart_player_sdk SmartPlayerSetRecorderAudioTranscodeAAC:is_transcode];
    
    //录制MP4文件 是否录制视频
    NSInteger is_record_video = 1;
    [_smart_player_sdk SmartPlayerSetRecorderVideo:is_record_video];
    
    //录制MP4文件 是否录制音频
    NSInteger is_record_audio = 1;
    [_smart_player_sdk SmartPlayerSetRecorderAudio:is_record_audio];
    
    
    is_inited_player_ = YES;
    
    NSLog(@"InitPlayer--");
    return true;
}

停止播放StopPlayer实现如下：

-(bool)StopPlayer
{
    NSLog(@"StopPlayer++");
    
    if (_smart_player_sdk != nil)
    {
        [_smart_player_sdk SmartPlayerStop];
    }
    
    if (!is_audio_only_) {
        if (_glView != nil) {
            [_glView removeFromSuperview];
            [SmartPlayerSDK SmartPlayeReleasePlayView:(__bridge void *)(_glView)];
            _glView = nil;
        }
    }
    
    NSLog(@"StopPlayer--");
    return true;
}

UnInitPlayer实现如下：

-(bool)UnInitPlayer
{
    NSLog(@"UnInitPlayer++");
    
    if (_smart_player_sdk != nil)
    {
        [_smart_player_sdk SmartPlayerUnInitPlayer];
        
        if (_smart_player_sdk.delegate != nil)
        {
            _smart_player_sdk.delegate = nil;
        }
        
        _smart_player_sdk = nil;
    }
    
    is_inited_player_ = NO;
    
    NSLog(@"UnInitPlayer--");
    return true;
}

实时录像：

- (void)RecorderBtn:(UIButton *)button {
    
    NSLog(@"record Stream only++");
    
    button.selected = !button.selected;
    
    if (button.selected)
    {
        if(is_recording_)
            return;
        
        [self InitPlayer];
        
        //设置录像目录
        NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
        NSString *recorderDir = [paths objectAtIndex:0];
        
        if([_smart_player_sdk SmartPlayerSetRecorderDirectory:recorderDir] != DANIULIVE_RETURN_OK)
        {
            NSLog(@"Call SmartPlayerSetRecorderDirectory failed..");
        }
        
        //每个录像文件大小
        NSInteger size = 200;
        if([_smart_player_sdk SmartPlayerSetRecorderFileMaxSize:size] != DANIULIVE_RETURN_OK)
        {
            NSLog(@"Call SmartPlayerSetRecorderFileMaxSize failed..");
        }
        
        [_smart_player_sdk SmartPlayerStartRecorder];
        [recButton setTitle:@"停止录像" forState:UIControlStateNormal];
        
        is_recording_ = YES;
    }
    else
    {
        [_smart_player_sdk SmartPlayerStopRecorder];
        [recButton setTitle:@"开始录像" forState:UIControlStateNormal];
        
        if(!is_playing_)
        {
            [self UnInitPlayer];
        }
        
        is_recording_ = NO;
    }
}

实时快照：

- (void)SaveImageBtn:(UIButton *)button {
    if ( _smart_player_sdk != nil )
    {
        //设置快照目录
        NSLog(@"[SaveImageBtn] path++");
        
        NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
        NSString *saveImageDir = [paths objectAtIndex:0];
        
        NSLog(@"[SaveImageBtn] path: %@", saveImageDir);
        
        NSString* symbol = @"/";
        
        NSString* png = @".png";
        
        // 1.创建时间
        NSDate *datenow = [NSDate date];
        // 2.创建时间格式化
        NSDateFormatter *formatter = [[NSDateFormatter alloc] init];
        // 3.指定格式
        formatter.dateFormat = @"yyyyMMdd_HHmmss";
        // 4.格式化时间
        NSString *timeSp = [formatter stringFromDate:datenow];
        
        NSString* image_name =  [saveImageDir stringByAppendingString:symbol];
        
        image_name = [image_name stringByAppendingString:timeSp];
        
        image_name = [image_name stringByAppendingString:png];
        
        NSLog(@"[SaveImageBtn] image_name: %@", image_name);
        
        [_smart_player_sdk SmartPlayerSaveCurImage:image_name];
    }
}

Event回调处理如下：

- (NSInteger) handleSmartPlayerEvent:(NSInteger)nID param1:(unsigned long long)param1 param2:(unsigned long long)param2 param3:(NSString*)param3 param4:(NSString*)param4 pObj:(void *)pObj;
{
    NSString* player_event = @"";
    NSString* lable = @"";
    
    if (nID == EVENT_DANIULIVE_ERC_PLAYER_STARTED) {
        player_event = @"[event]开始播放..";
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_CONNECTING)
    {
        player_event = @"[event]连接中..";
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_CONNECTION_FAILED)
    {
        player_event = @"[event]连接失败..";
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_CONNECTED)
    {
        player_event = @"[event]已连接..";
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_DISCONNECTED)
    {
        player_event = @"[event]断开连接..";
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_STOP)
    {
        player_event = @"[event]停止播放..";
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_RESOLUTION_INFO)
    {
        NSString *str_w = [NSString stringWithFormat:@"%ld", (long)param1];
        NSString *str_h = [NSString stringWithFormat:@"%ld", (long)param2];
        
        lable = @"[event]视频解码分辨率信息: ";
        player_event = [lable stringByAppendingFormat:@"%@*%@", str_w, str_h];
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_NO_MEDIADATA_RECEIVED)
    {
        player_event = @"[event]收不到RTMP数据..";
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_SWITCH_URL)
    {
        player_event = @"[event]快速切换url..";
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_CAPTURE_IMAGE)
    {
        if ((int)param1 == 0)
        {
            NSLog(@"[event]快照成功: %@", param3);
            lable = @"[event]快照成功:";
            player_event = [lable stringByAppendingFormat:@"%@", param3];
            
            tmp_path_ = param3;
            
            image_path_ = [ UIImage imageNamed:param3];
            
            UIImageWriteToSavedPhotosAlbum(image_path_, self, @selector(image:didFinishSavingWithError:contextInfo:), NULL);
        }
        else
        {
            lable = @"[event]快照失败";
            player_event = [lable stringByAppendingFormat:@"%@", param3];
        }
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_RECORDER_START_NEW_FILE)
    {
        lable = @"[event]录像写入新文件..文件名:";
        player_event = [lable stringByAppendingFormat:@"%@", param3];
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_ONE_RECORDER_FILE_FINISHED)
    {
        lable = @"一个录像文件完成..文件名:";
        player_event = [lable stringByAppendingFormat:@"%@", param3];
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_START_BUFFERING)
    {
        //NSLog(@"[event]开始buffer..");
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_BUFFERING)
    {
        NSLog(@"[event]buffer百分比: %lld", param1);
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_STOP_BUFFERING)
    {
        //NSLog(@"[event]停止buffer..");
    }
    else if (nID == EVENT_DANIULIVE_ERC_PLAYER_DOWNLOAD_SPEED)
    {
        NSInteger speed_kbps = (NSInteger)param1*8/1000;
        NSInteger speed_KBs = (NSInteger)param1/1024;
        
        lable = @"[event]download speed :";
        player_event = [lable stringByAppendingFormat:@"%ld kbps - %ld KB/s", (long)speed_kbps, (long)speed_KBs];
    }
    else if(nID == EVENT_DANIULIVE_ERC_PLAYER_RTSP_STATUS_CODE)
    {
        lable = @"[event]RTSP status code received:";
        player_event = [lable stringByAppendingFormat:@"%ld", (long)param1];
        
        dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
            dispatch_async(dispatch_get_main_queue(), ^{
                UIAlertController *aleView=[UIAlertController alertControllerWithTitle:@"RTSP错误状态" message:player_event preferredStyle:UIAlertControllerStyleAlert];
                UIAlertAction *action_ok=[UIAlertAction actionWithTitle:@"确定" style:UIAlertActionStyleCancel handler:nil];
                [aleView addAction:action_ok];
                
                [self presentViewController:aleView animated:YES completion:nil];
            });
        });
    }
    else if(nID == EVENT_DANIULIVE_ERC_PLAYER_NEED_KEY)
    {
        player_event = @"[event]RTMP加密流，请设置播放需要的Key..";
    }
    else if(nID == EVENT_DANIULIVE_ERC_PLAYER_KEY_ERROR)
    {
        player_event = @"[event]RTMP加密流，Key错误，请重新设置..";
    }
    else
        NSLog(@"[event]nID:%lx", (long)nID);
    
    NSString* player_event_tag = @"当前状态:";
    NSString* event = [player_event_tag stringByAppendingFormat:@"%@", player_event];
    
    if ( player_event.length != 0)
    {
        NSLog(@"%@", event);
    }
    
    dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
            dispatch_async(dispatch_get_main_queue(), ^{
                self.textPlayerEventLabel.text = event;
            });
    });
    
    return 0;
}

总结

iOS平台播放，由于设备和系统比较单一，所以优先考虑硬解码，除了基础播放外，我们还实现了实时快照、实时录像、实时回调YUV数据、实时音量调节等，实际体验下来，iOS平台RTMP和RTSP，可以轻松毫秒级，感兴趣的开发者，可以和我单独交流。