WebRTC, GStreamer and HTML5 - Part 2

An easy 360º solution for realtime multimedia communication.

First part is available in Part 1 - The story so far...

Part 2 - The GstWebRTC API #

The new gstwebrtc-api Javascript library developed at Igalia offers a full integration of the GStreamer webrtcsrc/webrtcsink protocols within a web browser or a mobile WebView.

You can easily and transparently interconnect a realtime streaming web application with GStreamer native components. Some interesting use-cases may be:

• ingestion of audio/video from a mobile phone to a backend infrastructure with low latency,
• broadcasting of low-latency media to a web page,
• video conferencing with native desktop clients and web clients,
• mobile video heavy post-processing (like AI) on server-side with direct feedback on client-side,
• mixing videos from different remote sources on the server-side and broadcasting back with low end-to-end latency,
• etc...

A full-featured demo showing how to use the gstwebrtc-api is available on the source code repository in the index.html file.

Following the instructions in the repository’s README, you can launch the demo and interact with GStreamer pipelines.

This demo opens a simple web page that, on one hand, offers to stream out the device webcam as a producer and, on the other hand, automatically detects any new remote producer available on the signalling network and offers to connect to them to consume their WebRTC streams.

The gstwebrtc-api is very simple and articulates around 3 groups of functions:

1 - Management of the connection to the signalling server. #

The client is automatically connected to the signalling server when the DOMContentLoaded event is triggered and, in case of unwanted disconnection, it is automatically reconnected. The signalling server address and reconnection timeout can be configured in the gstWebRTCConfig configuration.

const connectionListener = {
    connected: function(clientId)
    {
        // New connection established with the unique identifier: clientId
    },

disconnected: function()
    {
        // Disconnected from the signalling server
    }
};

gstWebRTCAPI.registerConnectionListener(connectionListener);
gstWebRTCAPI.unregisterConnectionListener(connectionListener);
gstWebRTCAPI.unregisterAllConnectionListeners();

2 - Management of the producer role. #

Only one producer session can be created at once. While a producer session is active, it is offering the provided MediaStream to the signalling network. The producer session can be stopped at any moment by calling the close() method on it.

const gstWebRTCAPI.SessionState = Object.freeze({
    idle: 0,       // Session has just been created, you need to call start() or connect()
    connecting: 1, // Session is connecting to the other peer
    streaming: 2,  // Session is actively streaming (out for a producer, in for a consumer)
    closed: 3      // Session is definitively closed and object can be recycled
});

class gstWebRTCAPI.ProducerSession
{
    get stream() -> MediaStream; // The stream broadcasted out by this producer
    get state() -> SessionState; // Current producer session state
    start() -> boolean;          // Must be called to start the session after registering eventual events listeners
    close();                     // Closes definitively the session
}

// The provided MediaStream is the stream that will be broadcasted out by the created producer session
gstWebRTCAPI.createProducerSession(stream) -> ProducerSession;

The ProducerSession class inherits from EventTarget and emits the following events:

• error when a network or streaming error occurs,
• stateChanged each time the session state changes,
• closed when the session has been closed,
• clientConsumerAdded each time a new remote consumer connects to the session,
• clientConsumerRemoved each time a remote consumer disconnects from the session.

3 - Management of the consumer role #

You can connect to any remote producer using its unique identifier. You can get the list of remote producers by calling gstWebRTCAPI.getAvailableProducers(). You can also register a listener to stay informed when a producer appears or disappears.

gstWebRTCAPI.Producer = Object.freeze({
    id: {non-empty string},
    meta: {non-null object}
});

gstWebRTCAPI.getAvailableProducers() -> Producer[];

const producersListener = {
    producerAdded: function(producer)
    {
        // A new producer is available (producer is a gstWebRTCAPI.Producer object)
    },

producerRemoved: function(producer)
    {
        // A producer has been closed (producer is a gstWebRTCAPI.Producer object)
    }
};

gstWebRTCAPI.registerProducersListener(producersListener);
gstWebRTCAPI.unregisterProducersListener(producersListener);
gstWebRTCAPI.unregisterAllProducersListeners();

class gstWebRTCAPI.ConsumerSession
{
    get peerId() -> string;         // Unique identifier of the producer peer to which this session is (or must be)
                                    // connected (always non-empty)
    get sessionId() -> string;      // Unique identifier of this session provided by the signalling server during
                                    // connection (empty until connection succeeds)
    get state() -> SessionState;    // Current consumer session state
    get streams() -> MediaStream[]; // List of remote streams received by this consumer session
    connect() -> boolean;           // Must be called to connect the session after registering eventual events
                                    // listeners
    close();                        // Closes definitively the session
}

gstWebRTCAPI.createConsumerSession(producerId) -> ConsumerSession;

The ConsumerSession class inherits from EventTarget and emits the following events:

• error when a network or streaming error occurs,
• stateChanged each time the session state changes,
• closed when the session has been closed,
• streamsChanged each time the underlying media streams change when media tracks are added or removed.

You can find a reference usage of the gstwebrtc-api in the index.html file.

Conclusion #

By using the new gstwebrtc-api in your web application in conjunction with the new webrtcsrc and webrtcsink GStreamer elements, you can now easily build a 360º realtime audio/video communication product.

You can transparently integrate web components and native components for desktop and/or backend applications.

The web integration is as easy as including a single script with a few straightforward functions (see index.html), whereas you can write your desktop or backend application using any wrapping language (C/C++, Rust, Python, .Net, Java, Javascript, Shell Script, etc.) and/or UI framework (GTK, QT, WxWidgets, Tcl/Tk, Avalonia, etc.) supported by GStreamer.

If you want to integrate multimedia and realtime communication into your project, don't hesitate to contact us at [email protected] or on our website: https://www.igalia.com/technology/multimedia. We will be pleased to offer our expertise to unblock your developments, reduce your innovation risks, or to help you add astonishing features to your products.

• Previous: WebRTC, GStreamer and HTML5 - Part 1