WebRTC-Android实践02

02 本地回环

本文是基于 Android WebRTC完整入门教程 这篇文章的实践过程记录，自己新增的内容主要体现在代码的注释中

这部分还可以参考大神PIASY的WebRTC native源码解析系列文章：https://blog.piasy.com/2017/08/30/WebRTC-P2P-part1/index.html

介绍WebRTC中最核心的概念PeerConnection , 给同一手机中的前后摄像头建立虚拟的连接, 相互传输画面

PeerConnection

PeerConnection也就是Peer-to-Peer connection(P2P), 就是两个"人"的连接. 双方分别创建PeerConnection对象, 然后向对方发送自己的网络状况ICE和多媒体编码格式SDP(因为这时候连接还没建立, 所以发送内容是通过服务器完成的). 当双方网络和编码格式协商好后, 连接就建立好了, 这时从PeerConnection中能获取到对方的MediaStream数据流, 也就能播放对方的音视频了

ICE

Interactive Connectivity Establishment, 交互式连接建立. 其实是一个整合STUN和TURN的框架, 给它提供STUN和TURN服务器地址, 它会自动选择优先级高的进行NAT穿透

SDP

Session Description Protocol: 会话描述协议. 发送方的叫Offer, 接受方的叫Answer, 除了名字外没有区别. 就是一些文本描述本地的音视频编码和网络地址等

主要流程

A(local)和B(remote)代表两个人, 初始化PeerConnectionFactory并分别创建PeerConnection , 并向PeerConnection 添加本地媒体流

1. A创建Offer
2. A保存Offer(set local description)
3. A发送Offer给B
4. B保存Offer(set remote description)
5. B创建Answer
6. B保存Answer(set local description)
7. B发送Answer给A
8. A保存Answer(set remote description)
9. A发送ICE Candidates给B
10. B发送ICE Candidates给A
11. A, B收到对方的媒体流并播放

流程图如下

准备步骤

主要是初始化PeerConnectionFactory和使用相机

public class MainActivity extends AppCompatActivity {
    PeerConnectionFactory peerConnectionFactory;
    PeerConnection peerConnectionLocal;
    PeerConnection peerConnectionRemote;
    SurfaceViewRenderer localView;
    SurfaceViewRenderer remoteView;
    // 在webrtc中，MediaStream代表一个媒体流，AudioTrack，VideoTrack代表音频”轨道”和视频“轨道”，如同一个MP4文件可以包含许多音轨和视频轨一样，一个MediaStream中可以包含多个AudioTrack和VideoTrack
    MediaStream mediaStreamLocal;
    MediaStream mediaStreamRemote;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        verifyStoragePermissions(this);

        // 创建EglBase对象, WebRTC 把 EGL 的操作封装在了 EglBase 中，并针对 EGL10 和 EGL14 提供了不同的实现, 而 OpenGL 的绘制操作则封装在了 EglRenderer 中
        // 获取EglBase对象的上下文
        EglBase.Context eglBaseContext = EglBase.create().getEglBaseContext();

        // create PeerConnectionFactory
        // PeerConnectionFactory负责创建PeerConnection、VideoTrack、AudioTrack等重要对象
        // 初始化PeerConnectionFactory
        PeerConnectionFactory.initialize(PeerConnectionFactory.
                InitializationOptions.
                builder(this).
                createInitializationOptions();
        // PeerConnection的选项类
        PeerConnectionFactory.Options options = new PeerConnectionFactory.Options();
        // 视频编码工厂类，负责创建视频编码类
        DefaultVideoEncoderFactory defaultVideoEncoderFactory = new DefaultVideoEncoderFactory(eglBaseContext,
                true,
                true);
        // 视频解码工厂类
        DefaultVideoDecoderFactory defaultVideoDecoderFactory = new DefaultVideoDecoderFactory(eglBaseContext);
        // 设置PeerConnection工厂类，并创建PeerConnection工厂对象
        peerConnectionFactory = PeerConnectionFactory.
                builder().
                setOptions(options).
                setVideoEncoderFactory(defaultVideoEncoderFactory).
                setVideoDecoderFactory(defaultVideoDecoderFactory).
                createPeerConnectionFactory();
        
        // SurfaceTextureHelper 负责创建 SurfaceTexture，接收 SurfaceTexture 数据，相机线程的管理
        SurfaceTextureHelper localSurfaceTextureHelper = SurfaceTextureHelper.create("localCaptureThread", eglBaseContext);
        // create VideoCapturer
        // 获取前置摄像头
        // WebRTC 视频采集的接口定义为 VideoCapturer，其中定义了初始化、启停、销毁等操作，以及接收启停事件、数据的回调
        VideoCapturer localVideoCapturer = createCameraCapturer(true);
        // peerConnectionFactory通过localVideoCapturer创建视频源
        VideoSource localVideoSource = peerConnectionFactory.createVideoSource(localVideoCapturer.isScreencast());
        localVideoCapturer.initialize(localSurfaceTextureHelper, getApplicationContext(), localVideoSource.getCapturerObserver());
        localVideoCapturer.startCapture(480, 640, 30);//480, 640, 30分别是width, height, fps

        //视频数据在 native 层处理完毕后会抛出到 VideoRenderer.Callbacks#renderFrame 回调中，在这里也就是 SurfaceViewRenderer#renderFrame，而 SurfaceViewRenderer 又会把数据交给 EglRenderer 进行渲染
        localView = findViewById(R.id.localView);
        localView.setMirror(true);
        localView.init(eglBaseContext, null);

        // create VideoTrack
        // 创建视频轨，用于网络传输
        VideoTrack localVideoTrack = peerConnectionFactory.createVideoTrack("100", localVideoSource);
//        // display in localView
//        localVideoTrack.addSink(localView);

        SurfaceTextureHelper remoteSurfaceTextureHelper = SurfaceTextureHelper.create("remoteCaptureThread", eglBaseContext);
        // create VideoCapturer
        // 获取后置摄像头
        VideoCapturer remoteVideoCapturer = createCameraCapturer(false);
        VideoSource remoteVideoSource = peerConnectionFactory.createVideoSource(remoteVideoCapturer.isScreencast());
        remoteVideoCapturer.initialize(remoteSurfaceTextureHelper, getApplicationContext(), remoteVideoSource.getCapturerObserver());
        remoteVideoCapturer.startCapture(480, 640, 30);

        remoteView = findViewById(R.id.remoteView);
        remoteView.setMirror(true);
        remoteView.init(eglBaseContext, null);

        // create VideoTrack
        // VideoTrack是webrtc中视频流最上层的接口，它内部其实是经过层层封装
        VideoTrack remoteVideoTrack = peerConnectionFactory.createVideoTrack("100", remoteVideoSource);
//        // display in remoteView
//        remoteVideoTrack.addSink(remoteView);

        // 向媒体流中添加视频轨
        mediaStreamLocal = peerConnectionFactory.createLocalMediaStream("mediaStreamLocal");
        mediaStreamLocal.addTrack(localVideoTrack);

        mediaStreamRemote = peerConnectionFactory.createLocalMediaStream("mediaStreamRemote");
        mediaStreamRemote.addTrack(remoteVideoTrack);

        call(mediaStreamLocal, mediaStreamRemote);
    }
    ...
}

使用相机

对createCameraCapturer()方法略作修改, 传入boolean参数就能分别获取前后摄像头, MainActivity.java中

// MainActivity.java中
// isFront==true 获取前置摄像头, 反之获取后置摄像头
private VideoCapturer createCameraCapturer(boolean isFront){
    Camera1Enumerator enumerator = new Camera1Enumerator(false);
    final String[] deviceNames = enumerator.getDeviceNames();

    // First, try to find front facing cammera
    for (String deviceName : deviceNames){
        if (isFront ? enumerator.isFrontFacing(deviceName) : enumerator.isBackFacing(deviceName)){
            VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);

            if (videoCapturer != null){
                return videoCapturer;
            }
        }
    }

    return null;
}

拨打

建立连接的两人肯定有一个是拨打方, 另一个是接受方. 拨打方创建Offer发给接受方, 接收方收到后回复Answer。

// MainActivity.java中
private void call(MediaStream mediaStreamLocal, MediaStream mediaStreamRemote){
    List<PeerConnection.IceServer> iceServers = new ArrayList<>();
    // 创建本地的peerConnection
    peerConnectionLocal = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("localConnection"){
        @Override
        // peerConnectionLocal发送收集到的iceCandidate
        public void onIceCandidate(IceCandidate iceCandidate) {
            super.onIceCandidate(iceCandidate);
            // 远端添加iceCandidate
            peerConnectionRemote.addIceCandidate(iceCandidate);
        }

        @Override
        public void onAddStream(MediaStream mediaStream) {
            super.onAddStream(mediaStream);
            // 远端通过传输过来的媒体流将视频轨添加到localView进行渲染
            VideoTrack remoteVideoTrack = mediaStream.videoTracks.get(0);
            // “返回”到主线程，更新应用 UI
            runOnUiThread(()->{
                remoteVideoTrack.addSink(localView);
            });
        }
    });

    peerConnectionRemote = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("remoteConnection"){
        @Override
        public void onIceCandidate(IceCandidate iceCandidate) {
            super.onIceCandidate(iceCandidate);
            peerConnectionLocal.addIceCandidate(iceCandidate);
        }

        @Override
        public void onAddStream(MediaStream mediaStream) {
            super.onAddStream(mediaStream);
            VideoTrack localVideoTrack = mediaStream.videoTracks.get(0);
            runOnUiThread(()->{
                localVideoTrack.addSink(remoteView);
            });
        }
    });

    // 添加本地的媒体流
    peerConnectionLocal.addStream(mediaStreamLocal);
    // 创建offer
    peerConnectionLocal.createOffer(new SdpAdapter("local offer sdp"){
        @Override
        public void onCreateSuccess(SessionDescription sessionDescription) {
            super.onCreateSuccess(sessionDescription);
            peerConnectionLocal.setLocalDescription(new SdpAdapter("local set local"), sessionDescription);
            peerConnectionRemote.addStream(mediaStreamRemote);
            peerConnectionRemote.setRemoteDescription(new SdpAdapter("remote set remote"), sessionDescription);
            peerConnectionRemote.createAnswer(new SdpAdapter("remote answer sdp"){
                @Override
                public void onCreateSuccess(SessionDescription sessionDescription) {
                    super.onCreateSuccess(sessionDescription);
                    peerConnectionRemote.setLocalDescription(new SdpAdapter("remote set local"), sessionDescription);
                    peerConnectionLocal.setRemoteDescription(new SdpAdapter("local set remote"), sessionDescription);
                }
            }, new MediaConstraints());
        }
    }, new MediaConstraints());
}

添加PeerConnectionAdapter类作为PeerConnection的观察者

public class PeerConnectionAdapter implements PeerConnection.Observer {

    private String tag;

    public PeerConnectionAdapter(String tag){
        this.tag = "bo" + tag;
    }

    public void log(String str){
        Log.d(this.tag, str);
    }

    @Override
    public void onSignalingChange(PeerConnection.SignalingState signalingState) {
        log("onSignalingChange" + signalingState);
    }

    @Override
    public void onIceConnectionChange(PeerConnection.IceConnectionState iceConnectionState) {
        log("onIceConnectionChange" + iceConnectionState);
    }

    @Override
    public void onIceConnectionReceivingChange(boolean b) {
        log("onIceConnectionReceivingChange" + b);
    }

    @Override
    public void onIceGatheringChange(PeerConnection.IceGatheringState iceGatheringState) {
        log("onIceGatheringChange" + iceGatheringState);
    }

    @Override
    public void onIceCandidate(IceCandidate iceCandidate) {
        log("onIceCandidate" + iceCandidate);
    }

    @Override
    public void onIceCandidatesRemoved(IceCandidate[] iceCandidates) {
        log("onIceCandidatesRemoved" + iceCandidates);
    }

    @Override
    public void onAddStream(MediaStream mediaStream) {
        log("onAddStream" + mediaStream);
    }

    @Override
    public void onRemoveStream(MediaStream mediaStream) {
        log("onRemoveStream" + mediaStream);
    }

    @Override
    public void onDataChannel(DataChannel dataChannel) {
        log("onDataChannel" + dataChannel);
    }

    @Override
    public void onRenegotiationNeeded() {
        log("onRenegotiationNeeded");
    }

    @Override
    public void onAddTrack(RtpReceiver rtpReceiver, MediaStream[] mediaStreams) {
        log("onAddTrack" + mediaStreams);
    }
}

添加SdpAdapter(继承SdpObserver)作为Sdp的观察者

public class SdpAdapter implements SdpObserver {

    private String tag;

    public SdpAdapter(String tag){
        this.tag =  "bo" + tag;
    }

    public void log(String str){
        Log.d(tag, str);
    }

    @Override
    public void onCreateSuccess(SessionDescription sessionDescription) {
        log("onCreateSuccess " + sessionDescription);
    }

    @Override
    public void onSetSuccess() {
        log("onSetSuccess ");
    }

    @Override
    public void onCreateFailure(String s) {
        log("onCreateFailure " + s);
    }

    @Override
    public void onSetFailure(String s) {
        log("onSetFailure " + s);
    }
}

注意: 虽然这里没有真正使用到网络, 但是要添加网络权限

<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>

Android6.0及以上申请权限

private static final int REQUEST_ALL = 1;

private static String[] PERMISSIONS_ALL = {
        "android.permission.CAMERA",
        "android.permission.RECORD_AUDIO",
        "android.permission.INTERNET",
        "android.permission.ACCESS_NETWORK_STATE"
};

//然后通过一个函数来申请
public static void verifyStoragePermissions(Activity activity) {
    try {
        int permission = 0;
        //检测所有需要的权限
        for(String temp : PERMISSIONS_ALL){
            permission = ActivityCompat.checkSelfPermission(activity, temp);
            if (permission != PackageManager.PERMISSION_GRANTED){
                break;
            }
        }

        if (permission != PackageManager.PERMISSION_GRANTED) {
            // 没有写的权限，去申请写的权限，会弹出对话框
            ActivityCompat.requestPermissions(activity, PERMISSIONS_ALL,REQUEST_ALL);
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}

网上大部分本地回环(Loopback)的Demo都只用到一个摄像头, 这里使用到同一个手机的前后摄像头, 把它们当做两个客户端, 建立模拟连接, 发送媒体数据. 这跟实际WebRTC工作流程非常接近了, 只有一点差别–这里的数据传输是内存共享, 而实际是通过网络发送