The WebRTC native APIs are implemented based on the following WebRTC spec.
The code that implements WebRTC native APIs (including the Stream and the PeerConnection APIs) are available in libjingle. A sample client application is also provided there.
The target audience of this document are those who want to use WebRTC Native APIs to implement WebRTC javascript APIs or to develop native RTC applications.
What is new
Compared to the previous version of WebRTC native APIs (code is available here as part of libjingle r115), the major difference is the new version contains the implementation of the Stream APIs. With the Stream APIs, the audio/video I/O will be set to the MediaTrack instead of directly to the PeerConnection. Also PeerConnection now takes and returns the MediaStream object instead of just the media stream label as in the previous version.
In addition to this, the signaling protocol has also changed to ROAP in this version, however, this should be transparent to the users of the APIs.
For those who have already developed the application based on the previous version, please check this patch as an example on how to migrate to the new APIs. The sample client application mentioned above can also be used as a reference.
Block diagram
Calling sequences
Set up a call
Receive a call
Close down a call
Threading model
WebRTC native APIs use two globally available threads: the signaling thread and the worker thread. Depending on how the PeerConnection factory is created, the application can either provide those 2 threads or just let them be created internally.
The calls to the Stream APIs and the PeerConnection APIs will be proxied to the signaling thread which means that the application can call those APIs from whatever thread.
All callbacks will be made on the signaling thread. The application should return the callback as quickly as possible to avoid blocking the signaling thread. Resource intensive processes should be posted to a different thread.
The worker thread is used to handle more resource intensive processes such as data streaming.
Class MediaStreamTrackInterface
This class declares an abstract interface to the media stream track which represents a media source in the user agent.
class MediaStreamTrackInterface : public talk_base::RefCountInterface,
public NotifierInterface {
public:
enum TrackState {
kInitializing,
kLive = 1,
kEnded = 2,
kFailed = 3,
};
virtual std::string kind() const = 0;
virtual std::string label() const = 0;
virtual bool enabled() const = 0;
virtual TrackState state() const = 0;
virtual bool set_enabled(bool enable) = 0;
virtual bool set_state(TrackState new_state) = 0;
};
MediaStreamTrackInterface::TrackState
This enumerator is used to specify the type of the track states.
Syntax
enum TrackState {
kInitializing,
kLive = 1,
kEnded = 2,
kFailed = 3,
};
Remarks
kInitializing - Track is beeing negotiated.
kLive - Track is alive.
kEnded - Track has ended.
kFailed - Track negotiation failed.
MediaStreamTrackInterface::kind
Returns the string "audio" if the track is an audio track, "video" if the track is a video track, or a user-agent defined string otherwise.
Syntax
virtual std::string kind() const = 0;
MediaStreamTrackInterface::label
Returns the label of the track, if any. If the track has no label, it returns the empty string.
Syntax
virtual std::string label() const = 0;
MediaStreamTrackInterface::enabled
Returns “true” if the track is enabled, “false” otherwise.
Syntax
virtual bool enabled() const = 0;
MediaStreamTrackInterface::state
Returns the current state of the track.
Syntax
virtual TrackState state() const = 0;
MediaStreamTrackInterface::set_enabled
Enable (true) or disable (false) a track.
Syntax
virtual bool set_enabled(bool enable) = 0;
Remarks
Not implemented.
MediaStreamTrackInterface::set_state
Set the track to a new state.
Syntax
virtual bool set_state(TrackState new_state) = 0;
Remarks
This method is used by the PeerConnection internally, the application should not call it directly.
Class VideoTrackInterface
The VideoTrackInterface derives from MediaStreamTrackInterface with two extra interfaces to get and set the video renderer.
class VideoTrackInterface : public MediaStreamTrackInterface {
public:
virtual void SetRenderer(VideoRendererWrapperInterface* renderer) = 0;
virtual VideoRendererWrapperInterface* GetRenderer() = 0;
protected:
virtual ~VideoTrackInterface() {}
};
Class LocalVideoTrackInterface
The LocalVideoTrackInterface derives from VideoTrackInterface with one extra interface to get the video capture associated with the track.
class LocalVideoTrackInterface : public VideoTrackInterface {
public:
virtual cricket::VideoCapturer* GetVideoCapture() = 0;
protected:
virtual ~LocalVideoTrackInterface() {}
};
Class AudioTrackInterface
The AudioTrackInterface derives from MediaStreamTrackInterface and currently without extra interface.
class AudioTrackInterface : public MediaStreamTrackInterface {
public:
protected:
virtual ~AudioTrackInterface() {}
};
Class LocalAudioTrackInterface
The LocalAudioTrackInterface derives from AudioTrackInterface with one extra interface to get audio device associated with this track.
class LocalAudioTrackInterface : public AudioTrackInterface {
public:
virtual AudioDeviceModule* GetAudioDevice() = 0;
protected:
virtual ~LocalAudioTrackInterface() {}
};
Class cricket::VideoRenderer, cricket::VideoCapturer
These classes are defined by the open source project libjingle and will not be described in detail here..
Class webrtc::AudioDeviceModule
The AudioDeviceModule is defined by the open source project webrtc. Please refer to this link for a detailed definition.
Class MediaStreamInterface
This class declares an abstract interface to MediaStream which, typically but not necessarily, represents streams of media data of audio and/or video content.
Each MediaStream object can contain zero or more tracks, in particular audio and video tracks. All tracks in a MediaStream are intended to be synchronized when rendered. Different MediaStreams do not need to be synchronized.
class MediaStreamInterface : public talk_base::RefCountInterface,
public NotifierInterface {
public:
virtual std::string label() const = 0;
virtual AudioTracks* audio_tracks() = 0;
virtual VideoTracks* video_tracks() = 0;
enum ReadyState {
kInitializing,
kLive = 1, // Stream alive
kEnded = 2, // Stream have ended
};
virtual ReadyState ready_state() = 0;
protected:
virtual ~MediaStreamInterface() {}
};
MediaStreamInterface::label
Returns a label that is unique to this stream, so that streams can be recognized after they are sent through the PeerConnection APIs.
Syntax
virtual std::string label() const = 0;
MediaStreamInterface::audio_tracks
Returns a pointer to a list of objects of type MediaStreamTrack representing the audio tracks associated with this MediaStream.
Syntax
virtual AudioTracks* audio_tracks() = 0;
MediaStreamInterface::video_tracks
Returns a pointer to a list of objects of type MediaStreamTrack representing the video tracks associated with this MediaStream.
Syntax
virtual VideoTracks* video_tracks() = 0;
MediaStreamInterface::ready_state
Returns the current ready state of the MediaStream.
Syntax
virtual ReadyState ready_state() = 0;
Class LocalMediaStreamInterface
The LocalMediaStreamInterface derives from the MediaStreamInterface with extra interfaces to add media tracks to the MediaStream.
class LocalMediaStreamInterface : public MediaStreamInterface {
public:
virtual bool AddTrack(AudioTrackInterface* track) = 0;
virtual bool AddTrack(VideoTrackInterface* track) = 0;
};
Class StreamCollectionInterface
This class declares a MediaStream container interface.
class StreamCollectionInterface : public talk_base::RefCountInterface {
public:
virtual size_t count() = 0;
virtual MediaStreamInterface* at(size_t index) = 0;
virtual MediaStreamInterface* find(const std::string& label) = 0;
protected:
~StreamCollectionInterface() {}
};
StreamCollectionInterface::count
Returns the number of MediaStreams in the collection.
Syntax
size_t count() = 0;
StreamCollectionInterface::at
Returns a reference to the MediaStream at the position index in the collection.
Syntax
MediaStreamInterface* at(size_t index) = 0;
Parameters
index [in] Position of a MediaStream in the collection.
StreamCollectionInterface::find
Searches the collection for a MediaStream with given label and returns a reference to it if found, otherwise it returns NULL.
Syntax
MediaStreamInterface* find(const std::string& label) = 0;
Parameters
label [in] The label value to be searched for.
Class PeerConnectionObserver
This class declares an abstract interface for a user defined observer. It is up to the PeerConnection user to implement a derived class which implements the observer class. The observer is registered when the PeerConnection is created using the PeerConnectionFactoryInterface.
class PeerConnectionObserver {
public:
enum StateType {
kReadyState,
kIceState,
kSdpState,
};
virtual void OnError() = 0;
virtual void OnMessage(const std::string& msg) = 0;
virtual void OnSignalingMessage(const std::string& msg) = 0;
virtual void OnStateChange(StateType state_changed) = 0;
virtual void OnAddStream(MediaStreamInterface* stream) = 0;
virtual void OnRemoveStream(MediaStreamInterface* stream) = 0;
protected:
~PeerConnectionObserver() {}
};
PeerConnectionObserver::StateType
This enumerator is used to specify the type of the state machines.
Syntax
enum StateType {
kReadyState,
kIceState,
kSdpState,
};
PeerConnectionObserver::OnError
This method will be called when an error occurs in PeerConnection.
Syntax
void OnError() = 0;
Remarks
Not implemented.
PeerConnectionObserver::OnMessage
This method will be called when a text message is received from the remote peer.
Syntax
void OnMessage(const std::string& msg) = 0;
PeerConnectionObserver::OnSignalingMessage
This method is called once a signaling message is ready.
Syntax
void OnSignalingMessage(const std::string& msg) = 0;
Parameters
msg [in] A ROAP format signaling message.
Remarks
The user should send the signaling message from the callback to the remote peer.
PeerConnectionObserver::OnStateChange
This method is called when any of the 3 state machines (ReadyState, SdpState or IceState) has a state change.
Syntax
virtual void OnStateChange(StateType state_changed) = 0;
Parameters
state_changed [in] Specify which state machine’s state has changed.
Remarks
The IceState is not implemented.
PeerConnectionObserver::OnAddStream
This method is called when media is received on a new stream from remote peer.
Syntax
virtual void OnAddStream(MediaStreamInterface* stream) = 0;
Parameters
stream [in] The handler to the remote media stream.
Remarks
The user may use this event to set the renderer for the received media stream.
PeerConnectionObserver::OnRemoveStream
This method is called when the remote peer closes a stream.
Syntax
virtual void OnRemoveStream(MediaStreamInterface* stream) = 0;
Parameters
stream [in] The handler to the closed remote media stream.
Class PortAllocatorFactoryInterface
This class declares a factory interface for creating the cricket::PortAllocator, which is used for the ICE negotiation. The PeerConnection factory will use this interface (if supplied) to create the PortAllocator for PeerConnection. The application may provide its own implementation of the PortAllocator by implementing the PortAllocatorFactoryInterface - basically the CreatePortAllocator method.
class PortAllocatorFactoryInterface : public talk_base::RefCountInterface {
public:
struct StunConfiguration {
StunConfiguration(const std::string& address, int port)
: server(address, port) {}
talk_base::SocketAddress server;
};
struct TurnConfiguration {
TurnConfiguration(const std::string& address,
int port,
const std::string& user_name,
const std::string& password)
: server(address, port),
username(username),
password(password) {}
talk_base::SocketAddress server;
std::string username;
std::string password;
};
virtual cricket::PortAllocator* CreatePortAllocator(
const std::vector<StunConfiguration>& stun_servers,
const std::vector<TurnConfiguration>& turn_configurations) = 0;
protected:
PortAllocatorFactoryInterface() {}
~PortAllocatorFactoryInterface() {}
};
PortAllocatorFactoryInterface::CreatePortAllocator
This method returns an instance of the PortAllocator class.
Syntax
virtual cricket::PortAllocator* CreatePortAllocator(
const std::vector<StunConfiguration>& stun_servers,
const std::vector<TurnConfiguration>& turn_configurations) = 0;
Parameters
stun_servers [in] A configuration list of the STUN servers.
turn_servers [in] A configuration list of the TURN servers.
Remarks
TURN is currently not implemented.
Class PeerConnectionFactoryInterface
The PeerConnectionFactoryInterface is the factory interface used for creating PeerConnection, media stream and media tracks.
class PeerConnectionFactoryInterface : public talk_base::RefCountInterface {
public:
virtual talk_base::scoped_refptr<PeerConnectionInterface>
CreatePeerConnection(const std::string& config,
PeerConnectionObserver* observer) = 0;
virtual talk_base::scoped_refptr<LocalMediaStreamInterface>
CreateLocalMediaStream(const std::string& label) = 0;
virtual talk_base::scoped_refptr<LocalVideoTrackInterface>
CreateLocalVideoTrack(const std::string& label,
cricket::VideoCapturer* video_device) = 0;
virtual talk_base::scoped_refptr<LocalAudioTrackInterface>
CreateLocalAudioTrack(const std::string& label,
AudioDeviceModule* audio_device) = 0;
protected:
PeerConnectionFactoryInterface() {}
~PeerConnectionFactoryInterface() {}
};
PeerConnectionFactoryInterface::CreatePeerConnection
Create an instance of the PeerConnection.
Syntax
virtual talk_base::scoped_refptr<PeerConnectionInterface>
CreatePeerConnection(const std::string& config,
PeerConnectionObserver* observer) = 0;
Parameters
config [in] The configuration string gives the address of a STUN or TURN server to use to establish the connection. The format is defined in the webrtc-api.
observer [in] A pointer to an instance of the PeerConnectionObserver derived class.
Remarks
TURN is currently not supported.
The allowed formats for the config string are:
Indicates a specific IP address and port for the server.
"TYPE relay.example.net:3478"
Indicates a specific host and port for the server; the user agent will look up the IP address in DNS.
Indicates a specific domain for the server; the user agent will look up the IP address and port in DNS.
The "TYPE" is one of:
Indicates a STUN server
Indicates a STUN server that is to be contacted using a TLS session.
Indicates a TURN server
Indicates a TURN server that is to be contacted using a TLS session.
PeerConnectionFactoryInterface::CreateLocalMediaStream
Create an instance of a local media stream.
Syntax
virtual talk_base::scoped_refptr<LocalMediaStreamInterface>
CreateLocalMediaStream(const std::string& label) = 0;
Parameters
label [in] Desired local media stream label.
PeerConnectionFactoryInterface::CreateLocalVideoTrack
Create an instance of a local video track.
Syntax
virtual talk_base::scoped_refptr<LocalVideoTrackInterface>
CreateLocalVideoTrack(const std::string& label,
cricket::VideoCapturer* video_device) = 0;
Parameters
label [in] Desired local video track label.
video_device [in] Pointer to the video capture device that is going to associate with this track.
PeerConnectionFactoryInterface::CreateLocalAudioTrack
Create an instance of a local audio track.
Syntax
virtual talk_base::scoped_refptr<LocalVideoTrackInterface>
CreateLocalAudioTrack(const std::string& label,
AudioDeviceModule* audio_device) = 0;
Parameters
label [in] Desired local audio track label.
audio_device [in] Pointer to the audio device that is going to associate with this track.
Function CreatePeerConnectionFactory
Create a new instance of PeerConnectionFactoryInterface.
Syntax
talk_base::scoped_refptr<PeerConnectionFactoryInterface>
CreatePeerConnectionFactory();
Remarks
The PeerConnectionFactoryInterface instance generated by this function will create required resources internally - including libjingle threads, socket and network manager factory classes for networking.
Function CreatePeerConnectionFactory
Create a new instance of PeerConnectionFactoryInterface with the given libjingle threads and portallocator factory.
Syntax
talk_base::scoped_refptr<PeerConnectionFactoryInterface>
CreatePeerConnectionFactory(talk_base::Thread* worker_thread,
talk_base::Thread* signaling_thread,
PortAllocatorFactoryInterface* factory,
AudioDeviceModule* default_adm);
Remarks
This function can be used when the application wants to provide its own implementation of the threads and the portallocator.
Ownership of the arguments are not transferred to this object and must remain in scope for the lifetime of the PeerConnectionFactoryInterface.
Class PeerConnectionInterface
class PeerConnectionInterface : public talk_base::RefCountInterface {
public:
enum ReadyState {
kNew,
kNegotiating,
kActive,
kClosing,
kClosed,
};
enum SdpState {
kSdpNew,
kSdpIdle,
kSdpWaiting,
};
virtual void ProcessSignalingMessage(const std::string& msg) = 0;
virtual bool Send(const std::string& msg) = 0;
virtual talk_base::scoped_refptr<StreamCollectionInterface>
local_streams() = 0;
virtual talk_base::scoped_refptr<StreamCollectionInterface>
remote_streams() = 0;
virtual void AddStream(LocalMediaStreamInterface* stream) = 0;
virtual void RemoveStream(LocalMediaStreamInterface* stream) = 0;
virtual void CommitStreamChanges() = 0;
virtual void Close() = 0;
virtual ReadyState ready_state() = 0;
virtual SdpState sdp_state() = 0;
protected:
~PeerConnectionInterface() {}
};
PeerConnectionInterface::ReadyState
This enumerator is used to specify the type of the ready state.
- kNew - The object was just created and its ICE and SDP Agent have not yet been started.
- kNegotiating - The peerConenction object is attempting to get to the point where media can flow.
- kActive - A connection has been formed and if any media streams were successfully negotiated, any relevant media can be streaming.
- kClosing - The object is starting to shut down after the close() method has been invoked.
- kClosed - The shutting down is completed.
Syntax
enum ReadyState {
kNew,
kNegotiating,
kActive,
kClosing,
kClosed,
};
PeerConnectionInterface::SdpState
This enumerator is used to specify the type of the SDP state.
- kSdpNew - The object was just created and SDP Agent have not yet been started.
- kSdpIdle - A valid offer answer pair has been exchanged and the SDP Agent is waiting for the next SDP transaction.
- kSdpWaiting - The SDP Agent has sent an SDP offer and is waiting for a response.
kSdpNew, // TODO(ronghuawu): kSdpNew is not defined in the spec.
PeerConnectionInterface::ProcessSignalingMessage
Handle the signaling message from the remote peer.
virtual void ProcessSignalingMessage(const std::string& msg) = 0;
msg
[in] A
ROAP
format signaling message.
The order of messages is important. Passing messages to the PeerConnection in a different order than they were generated by the remote peer can prevent a successful connection from being established or degrade the connection's quality if one is established.
PeerConnectionInterface::Send
Send a text message to the remote peer over the data stream.
virtual bool Send(const std::string& msg) = 0; // TODO(ronghuawu): This is not defined in the spec.
msg
[in] The text message to be sent to the remote peer.
Not currently implemented.
PeerConnectionInterface::local_streams
Returns a live array containing the streams that the user agent is currently attempting to transmit to the remote peer (those that were added with A
ddStream()
).
virtual talk_base::scoped_refptr<StreamCollectionInterface>
PeerConnectionInterface::remote_streams
Returns a live array containing the streams that the user agent is currently receiving from the remote peer.
This array is updated when OnAddStream and OnRemoveStream callbacks are fired.
virtual talk_base::scoped_refptr<StreamCollectionInterface>
PeerConnectionInterface::AddStream
Add a local stream to the array of the streams that the user agent is currently attempting to transmit to the remote peer. This function only adds the stream to the array and does not trigger any changes until CommitStreamChanges is called.
virtual void AddStream(LocalMediaStreamInterface* stream) = 0;
stream
[in] Pointer to the local media stream to be added.
PeerConnectionInterface::RemoveStream
Remove a local stream from the array of the streams that the user agent is currently attempting to transmit to the remote peer. This function only removes the stream from the array and does not trigger any changes until CommitStreamChanges is called.
virtual void RemoveStream(LocalMediaStreamInterface* stream) = 0;
stream
[in] Pointer to the local media stream to be removed.
PeerConnectionInterface::CommitStreamChanges
Commit Stream changes made by the AddStream and RemoveStream. This will start sending media on new added streams and stop sending media on removed streams.
virtual void CommitStreamChanges() = 0;
PeerConnectionInterface::Close
Close the current session. This will trigger a Shutdown message to be sent and the ready state will change to kClosing.
virtual void Close() = 0;
PeerConnectionInterface::ready_state
virtual ReadyState ready_state() = 0;
PeerConnectionInterface::sdp_state
Return the state of the PeerConnection SDP Agent, represented by enum SdpState.
virtual SdpState sdp_state() = 0;
Reference
The current HTML5 specification for WebRTC is here:
The source code of the WebRTC Native API is here: