android WebRTC 教程,Android WebRTC完整入门教程04: 多人视频

多人视频有三种理论方案, 如下图所示, 从左到右分别是Mesh,SFU,MCU.

multi-peers.png

Mesh 网格, 每个人都跟其他人单独建立连接. 4个人的情况下, 每个人建立3个连接, 也就是3个上传流和3个下载流. 此方案对客户端网络和计算能力要求最高, 对服务端没有特别要求.

SFU(Selective Forwarding Unit) 可选择转发单元, 有一个中心单元, 负责转发流. 每个人只跟中心单元建立一个连接, 上传自己的流, 并下载别人的流. 4个人的情况下, 每个人建立一个连接, 包括1个上传流和3个下载流. 此方案对客户端要求较高, 对服务端要求较高.

MCU(Multipoint Control Unit) 多端控制单元, 有一个中心单元, 负责混流处理和转发流. 每个人只跟中心单元建立一个连接, 上传自己的流, 并下载混流. 4个人的情况下, 每个人建立一个连接, 包括1个上传流和1个下载流. 此方案对客户端没有特别要求, 对服务端要求最高.

Mesh实现

先从理论上分析一下, 客户端A与B之间建立连接完全是通过PeerConnection对象, 那么只要客户端A有多个PeerConnection对象, 它就可以同时跟B,C,D...连接.

虽然PeerConnection有多个, 但是客户端A跟信令服务器仍然是一个socket连接, 这样A向服务器发送信令时就要指定发送给谁, 收到信令时要判断来自谁, 服务端收到信令时要判断发给谁. 这就需要在所有信令中添加两个字段 from 和 to, 代表信令发送方和接收方. 每个socket连接都有唯一socketId, 可以用socketId来标识一个客户端. 每个客户端用一个HashMap(key是socketId)来保存自己的连接.

拨号方案: 客户端A加入房间, 如果房间内还有其他客户端B和C. 服务端向B和C发送A的socketId, B和C收到后各自给A发送Offer建立连接, A分别回复Answer被动建立多个连接. 这样保证每个客户端的逻辑是一样的, 如果它新加入房间, 那么只需要等待其他人的Offer; 如果它已在房间中, 那么等待别人加入时向别人发送Offer.

信令服务端

在上一篇基础上做如下修改,

转发message时根据其中的to, 来选择发送目标

某人加入房间时, 向其他人发送此人的socketId

去掉房间内最多两个人的限制

socket.on('message', function(message) {

// for a real app, would be room-only (not broadcast)

// socket.broadcast.emit('message', message);

var to = message['to'];

log('from:' + socket.id + " to:" + to, message);

io.sockets.sockets[to].emit('message', message);

});

socket.on('create or join', function(room) {

log('Received request to create or join room ' + room);

var clientsInRoom = io.sockets.adapter.rooms[room];

var numClients = clientsInRoom ? Object.keys(clientsInRoom.sockets).length : 0;

log('Room ' + room + ' now has ' + numClients + ' client(s)');

if (numClients === 0) {

socket.join(room);

log('Client ID ' + socket.id + ' created room ' + room);

socket.emit('created', room, socket.id);

} else {

log('Client ID ' + socket.id + ' joined room ' + room);

io.sockets.in(room).emit('join', room, socket.id);

socket.join(room);

socket.emit('joined', room, socket.id);

io.sockets.in(room).emit('ready');

}

});

MainActivity.java

在上一篇的基础上, 添加HashMap peerConnectionMap(key是socketId)管理所有的PeerConnection连接, 收到信令时判断来源的socketId, 发送时加上自己和对方的socketId.

public class MainActivity extends AppCompatActivity implements SignalingClient.Callback {

EglBase.Context eglBaseContext;

PeerConnectionFactory peerConnectionFactory;

SurfaceViewRenderer localView;

MediaStream mediaStream;

List iceServers;

HashMap peerConnectionMap;

SurfaceViewRenderer[] remoteViews;

int remoteViewsIndex = 0;

@Override

protected void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

setContentView(R.layout.activity_main);

peerConnectionMap = new HashMap<>();

iceServers = new ArrayList<>();

iceServers.add(PeerConnection.IceServer.builder("stun:stun.l.google.com:19302").createIceServer());

eglBaseContext = EglBase.create().getEglBaseContext();

// create PeerConnectionFactory

PeerConnectionFactory.initialize(PeerConnectionFactory.InitializationOptions

.builder(this)

.createInitializationOptions());

PeerConnectionFactory.Options options = new PeerConnectionFactory.Options();

DefaultVideoEncoderFactory defaultVideoEncoderFactory =

new DefaultVideoEncoderFactory(eglBaseContext, true, true);

DefaultVideoDecoderFactory defaultVideoDecoderFactory =

new DefaultVideoDecoderFactory(eglBaseContext);

peerConnectionFactory = PeerConnectionFactory.builder()

.setOptions(options)

.setVideoEncoderFactory(defaultVideoEncoderFactory)

.setVideoDecoderFactory(defaultVideoDecoderFactory)

.createPeerConnectionFactory();

SurfaceTextureHelper surfaceTextureHelper = SurfaceTextureHelper.create("CaptureThread", eglBaseContext);

// create VideoCapturer

VideoCapturer videoCapturer = createCameraCapturer(true);

VideoSource videoSource = peerConnectionFactory.createVideoSource(videoCapturer.isScreencast());

videoCapturer.initialize(surfaceTextureHelper, getApplicationContext(), videoSource.getCapturerObserver());

videoCapturer.startCapture(480, 640, 30);

localView = findViewById(R.id.localView);

localView.setMirror(true);

localView.init(eglBaseContext, null);

// create VideoTrack

VideoTrack videoTrack = peerConnectionFactory.createVideoTrack("100", videoSource);

// // display in localView

videoTrack.addSink(localView);

remoteViews = new SurfaceViewRenderer[]{

findViewById(R.id.remoteView),

findViewById(R.id.remoteView2),

findViewById(R.id.remoteView3),

};

for(SurfaceViewRenderer remoteView : remoteViews) {

remoteView.setMirror(false);

remoteView.init(eglBaseContext, null);

}

mediaStream = peerConnectionFactory.createLocalMediaStream("mediaStream");

mediaStream.addTrack(videoTrack);

SignalingClient.get().init(this);

}

private synchronized PeerConnection getOrCreatePeerConnection(String socketId) {

PeerConnection peerConnection = peerConnectionMap.get(socketId);

if(peerConnection != null) {

return peerConnection;

}

peerConnection = peerConnectionFactory.createPeerConnection(iceServers, new PeerConnectionAdapter("PC:" + socketId) {

@Override

public void onIceCandidate(IceCandidate iceCandidate) {

super.onIceCandidate(iceCandidate);

SignalingClient.get().sendIceCandidate(iceCandidate, socketId);

}

@Override

public void onAddStream(MediaStream mediaStream) {

super.onAddStream(mediaStream);

VideoTrack remoteVideoTrack = mediaStream.videoTracks.get(0);

runOnUiThread(() -> {

remoteVideoTrack.addSink(remoteViews[remoteViewsIndex++]);

});

}

});

peerConnection.addStream(mediaStream);

peerConnectionMap.put(socketId, peerConnection);

return peerConnection;

}

@Override

public void onCreateRoom() {

}

@Override

public void onPeerJoined(String socketId) {

PeerConnection peerConnection = getOrCreatePeerConnection(socketId);

peerConnection.createOffer(new SdpAdapter("createOfferSdp:" + socketId) {

@Override

public void onCreateSuccess(SessionDescription sessionDescription) {

super.onCreateSuccess(sessionDescription);

peerConnection.setLocalDescription(new SdpAdapter("setLocalSdp:" + socketId), sessionDescription);

SignalingClient.get().sendSessionDescription(sessionDescription, socketId);

}

}, new MediaConstraints());

}

@Override

public void onSelfJoined() {

}

@Override

public void onPeerLeave(String msg) {

}

@Override

public void onOfferReceived(JSONObject data) {

runOnUiThread(() -> {

final String socketId = data.optString("from");

PeerConnection peerConnection = getOrCreatePeerConnection(socketId);

peerConnection.setRemoteDescription(new SdpAdapter("setRemoteSdp:" + socketId),

new SessionDescription(SessionDescription.Type.OFFER, data.optString("sdp")));

peerConnection.createAnswer(new SdpAdapter("localAnswerSdp") {

@Override

public void onCreateSuccess(SessionDescription sdp) {

super.onCreateSuccess(sdp);

peerConnectionMap.get(socketId).setLocalDescription(new SdpAdapter("setLocalSdp:" + socketId), sdp);

SignalingClient.get().sendSessionDescription(sdp, socketId);

}

}, new MediaConstraints());

});

}

@Override

public void onAnswerReceived(JSONObject data) {

String socketId = data.optString("from");

PeerConnection peerConnection = getOrCreatePeerConnection(socketId);

peerConnection.setRemoteDescription(new SdpAdapter("setRemoteSdp:" + socketId),

new SessionDescription(SessionDescription.Type.ANSWER, data.optString("sdp")));

}

@Override

public void onIceCandidateReceived(JSONObject data) {

String socketId = data.optString("from");

PeerConnection peerConnection = getOrCreatePeerConnection(socketId);

peerConnection.addIceCandidate(new IceCandidate(

data.optString("id"),

data.optInt("label"),

data.optString("candidate")

));

}

@Override

protected void onDestroy() {

super.onDestroy();

SignalingClient.get().destroy();

}

private VideoCapturer createCameraCapturer(boolean isFront) {

Camera1Enumerator enumerator = new Camera1Enumerator(false);

final String[] deviceNames = enumerator.getDeviceNames();

// First, try to find front facing camera

for (String deviceName : deviceNames) {

if (isFront ? enumerator.isFrontFacing(deviceName) : enumerator.isBackFacing(deviceName)) {

VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);

if (videoCapturer != null) {

return videoCapturer;

}

}

}

return null;

}

}

多人视频

启动node.js服务器, 在多个安卓手机上安装客户端, 先后启动, 随后就能在一个客户端上看到其他所有人的画面. (这里布局文件只放了4个SurfaceViewRenderer, 因此支持2,3,4个手机同时连接).

连接4个手机