API with NestJS #142. A video chat with WebRTC and React

January 22, 2024

This entry is part 142 of 142 in the API with NestJS

Let’s say two people want to video chat using our app. One solution would be for the first person to stream their camera feed to our server. Then, our application would pass this data to the other caller. Unfortunately, our application acting as a middleman could introduce a significant lag, especially if our server is far from the call participants. Fortunately, we can use WebRTC to connect our users directly without sending all of the data through our server. This improves performance and helps us achieve a smooth experience.

Even if the video and audio are not streamed through our application, we still need to build a server to allow the users to establish a connection with each other. In this article, we create a NestJS application to connect our users. Besides that, we also create a frontend app using React that manages and displays the video stream.

For the code of our backend application, check out this repository.
If you want to take a look at the frontend code, open this repository.

Capturing the camera

In this article, we gradually build both the frontend and backend applications. For the frontend part, we use Vite.

If you want to know more about Vite, check out React with Vite and TypeScript and its common challenges

Let’s start by capturing a stream from the local camera. To do that, we need to use the mediaDevices object.

useLocalCameraStream.tsx

import { useEffect, useState } from 'react';

export function useLocalCameraStream() {

const [localStream, setLocalStream] = useState<MediaStream | null>(null);

useEffect(() => {

navigator.mediaDevices

.getUserMedia({ video: true, audio: true })

.then((stream) => {

setLocalStream(stream);

}, []);

return {

localStream,

To display it, we need to use the <video> element. Since we need to attach the media stream to its srcObject property, we must use a React reference.

VideoFeed.tsx

import { FunctionComponent } from 'react';

interface Props {

mediaStream: MediaStream;

isMuted?: boolean;

export const VideoFeed: FunctionComponent<Props> = ({

mediaStream,

isMuted = false,

}) => {

return (

<video

ref={(ref) => {

if (ref) {

ref.srcObject = mediaStream;

autoPlay={true}

muted={isMuted}

The last step is to combine our hook with the VideoFeed component.

App.tsx

import { useLocalCameraStream } from './useLocalCameraStream';

import { VideoFeed } from './VideoFeed';

export function App() {

const { localStream } = useLocalCameraStream();

if (!localStream) {

return null;

return <VideoFeed mediaStream={localStream} isMuted={true} />;

Creating video chat rooms

We want our users to be able to join a particular chat room and communicate in real time with the other person in the room. To do that, we need to use WebSockets.

If you want to know more about websockets, check out API with NestJS #26. Real-time chat with WebSockets

To use WebSockets in our NestJS application, we need to install a few additional libraries.

npm install @nestjs/websockets @nestjs/platform-socket.io

In this article, we use Socket.IO, because it is the most popular implementation of WebSockets for NestJS.

Let’s start with allowing the users to join a chat room with a particular name. In this straightforward implementation, we only handle a maximum of two people at once in the room.

chat.gateway.ts

import {

ConnectedSocket,

MessageBody,

SubscribeMessage,

WebSocketGateway,

WebSocketServer,

} from '@nestjs/websockets';

import { Server, Socket } from 'socket.io';

@WebSocketGateway()

export class ChatGateway {

@WebSocketServer()

server: Server;

@SubscribeMessage('join_room')

async joinRoom(

@MessageBody() roomName: string,

@ConnectedSocket() socket: Socket,

const room = this.server.in(roomName);

const roomSockets = await room.fetchSockets();

const numberOfPeopleInRoom = roomSockets.length;

// a maximum of 2 people in a room

if (numberOfPeopleInRoom > 1) {

room.emit('too_many_people');

return;

socket.join(roomName);

Thanks to the above code, the user joins a room with the provided name as soon as the frontend application emits the join_room message.

Dealing with Cross-Origin Resource Sharing

There is a solid chance that our frontend and backend applications are deployed to different origins. This can cause issues with Cross-Origin Resource Sharing when our frontend application tries to talk with the API.

If you want to know more about CORS, check out API with NestJS #117. CORS – Cross-Origin Resource Sharing

To deal with this, we need to define the URL of our frontend application on the backend side. A good way to do that is through environment variables.

FRONTEND_URL=http://localhost:5173

app.module.ts

import { Module } from '@nestjs/common';

import { AppController } from './app.controller';

import { AppService } from './app.service';

import { ChatModule } from './chat/chat.module';

import * as Joi from 'joi';

import { ConfigModule } from '@nestjs/config';

@Module({

imports: [

ChatModule,

ConfigModule.forRoot({

validationSchema: Joi.object({

FRONTEND_URL: Joi.string().required(),

controllers: [AppController],

providers: [AppService],

export class AppModule {}

To be able to provide the frontend URL dynamically, we should extend the IoAdapter built into NestJS.

websocket-adapter.ts

import { IoAdapter } from '@nestjs/platform-socket.io';

import { INestApplicationContext } from '@nestjs/common';

import { Server, ServerOptions } from 'socket.io';

import { CorsOptions } from 'cors';

export class WebsocketAdapter extends IoAdapter {

constructor(

appOrHttpServer: INestApplicationContext,

private readonly corsOptions: CorsOptions,

super(appOrHttpServer);

create(port: number, options?: ServerOptions): Server {

return super.create(port, {

...options,

cors: this.corsOptions,

We can now use it in our bootstrap function.

main.ts

import { NestFactory } from '@nestjs/core';

import { AppModule } from './app.module';

import { ConfigService } from '@nestjs/config';

import { WebsocketAdapter } from './chat/websocket-adapter';

async function bootstrap() {

const app = await NestFactory.create(AppModule);

const configService = app.get(ConfigService);

app.enableCors({

origin: configService.get('FRONTEND_URL'),

app.useWebSocketAdapter(

new WebsocketAdapter(app, {

origin: configService.get('FRONTEND_URL'),

await app.listen(3000);

bootstrap();

Joining a room

One way of letting the user join a particular room is through URL path parameters. For example, if the users enter the /video-chat-room/important-meeting URL, we assume that the room name is the important-meeting.

The React Router is the most straightforward way to achieve that with React.

npm install react-router-dom

App.tsx

import { BrowserRouter, Route, Routes } from 'react-router-dom';

import { VideoChatRoom } from './VideoChatRoom';

import React from 'react';

import { useLocalCameraStream } from './useLocalCameraStream';

export const App = () => {

const { localStream } = useLocalCameraStream();

return (

<Route

path="video-chat-room/:roomName"

element={localStream && <VideoChatRoom localStream={localStream} />}

</Routes>

</BrowserRouter>

To connect to our API, we need its URL. The best way to store it is through environment variables.

VITE_API_URL=http://localhost:3000

vite-env.d.ts

/// <reference types="vite/client" />

interface ImportMetaEnv {

readonly VITE_API_URL: string;

We need to use the socket.io-client library to establish the WebSocket connection.

npm install socket.io-client

socket.tsx

import { io } from 'socket.io-client';

export const socket = io(import.meta.env.VITE_API_URL, {

autoConnect: false,

We can now connect to our NestJS application and join a room.

useChatConnection.tsx

import { useParams } from 'react-router-dom';

import { useCallback, useEffect } from 'react';

import { socket } from '../socket';

export function useChatConnection() {

const { roomName } = useParams();

const handleConnection = useCallback(() => {

socket.emit('join_room', roomName);

}, [roomName]);

useEffect(() => {

socket.connect();

socket.on('connect', handleConnection);

return () => {

socket.off('connect', handleConnection);

}, [roomName, handleConnection, roomName]);

Above, we emit the join_room event as soon as we connect to the WebSocket and provide the name of the chat based on the URL.

VideoChatRoom.tsx

import { VideoFeed } from './VideoFeed';

import { FunctionComponent } from 'react';

import { useChatConnection } from './useChatConnection';

interface Props {

localStream: MediaStream;

export const VideoChatRoom: FunctionComponent<Props> = ({ localStream }) => {

useChatConnection();

return <VideoFeed mediaStream={localStream} isMuted={true} />;

Connecting the people in the room

Once we have precisely two people in our chat room, we want one of them to initiate a connection. To be able to do that, we need to let one of the people know that the other person is ready. To do that, we can emit the another_person_ready event.

chat.gateway.ts

import {

ConnectedSocket,

MessageBody,

SubscribeMessage,

WebSocketGateway,

WebSocketServer,

} from '@nestjs/websockets';

import { Server, Socket } from 'socket.io';

@WebSocketGateway()

export class ChatGateway {

@WebSocketServer()

server: Server;

@SubscribeMessage('join_room')

async joinRoom(

@MessageBody() roomName: string,

@ConnectedSocket() socket: Socket,

const room = this.server.in(roomName);

const roomSockets = await room.fetchSockets();

const numberOfPeopleInRoom = roomSockets.length;

// a maximum of 2 people in a room

if (numberOfPeopleInRoom > 1) {

room.emit('too_many_people');

return;

if (numberOfPeopleInRoom === 1) {

room.emit('another_person_ready');

socket.join(roomName);

We need to start by creating an RTC Peer Connection. To do that, we need to provide the ICE server. In WebRTC, ICE is a framework used to facilitate the connection between the users. It helps us find the best path to connect two peers online. A part of the ICE framework is the STUN protocol used to discover the public address we need to establish the connection.

Fortunately, Google provides a public STUN server we can use.

usePeerConnection.tsx

import { useMemo } from 'react';

export function usePeerConnection(localStream: MediaStream) {

const peerConnection = useMemo(() => {

const connection = new RTCPeerConnection({

iceServers: [{ urls: 'stun:stun2.1.google.com:19302' }],

localStream.getTracks().forEach((track) => {

connection.addTrack(track, localStream);

return connection;

}, []);

return {

peerConnection,

Above, we create an instance of the RTCPeerConnection and add video and audio tracks from the local stream created thanks to our webcam.

Let’s pass the peerConnection object to the useChatConnection hook we created before.

VideoChatRoom.tsx

import { VideoFeed } from './VideoFeed';

import { FunctionComponent } from 'react';

import { useChatConnection } from './useChatConnection';

import { usePeerConnection } from './usePeerConnection.tsx';

interface Props {

localStream: MediaStream;

export const VideoChatRoom: FunctionComponent<Props> = ({ localStream }) => {

const { peerConnection } = usePeerConnection(localStream);

useChatConnection(peerConnection);

return <VideoFeed mediaStream={localStream} isMuted={true} />;

Sending the connection offer

The first step in establishing the WebRTC connection is sending the connection offer. To do that, we need to use the createOffer and setLocalDescription methods.

Once the offer is created, we send it to our NestJS server through the send_connection_offer event.

useOfferSending.tsx

import { useCallback } from 'react';

import { socket } from '../socket.tsx';

import { useParams } from 'react-router-dom';

export function useOfferSending(peerConnection: RTCPeerConnection) {

const { roomName } = useParams();

const sendOffer = useCallback(async () => {

const offer = await peerConnection.createOffer();

await peerConnection.setLocalDescription(offer);

socket.emit('send_connection_offer', {

roomName,

offer,

}, [roomName]);

return { sendOffer };

We should send the offer as soon as we receive the another_person_ready event indicating that the other person in the chat room is ready.

useChatConnection.tsx

import { useParams } from 'react-router-dom';

import { useCallback, useEffect } from 'react';

import { socket } from '../socket';

import {useOfferSending} from "./useOfferSending.tsx";

export function useChatConnection(peerConnection: RTCPeerConnection) {

const { roomName } = useParams();

const { sendOffer } = useOfferSending(peerConnection);

const handleConnection = useCallback(() => {

socket.emit('join_room', roomName);

}, [roomName]);

useEffect(() => {

socket.connect();

socket.on('connect', handleConnection);

socket.on('another_person_ready', sendOffer);

return () => {

socket.off('connect', handleConnection);

socket.off('another_person_ready', sendOffer);

}, [roomName, handleConnection, roomName]);

We now need to adjust our NestJS server to handle this event. Fortunately, it is very straightforward. As soon as our WebSocket server gets the send_connection_offer, it sends it to the other person in the room.

chat.gateway.ts

import {

ConnectedSocket,

MessageBody,

SubscribeMessage,

WebSocketGateway,

WebSocketServer,

} from '@nestjs/websockets';

import { Server, Socket } from 'socket.io';

@WebSocketGateway()

export class ChatGateway {

@WebSocketServer()

server: Server;

// ...

@SubscribeMessage('send_connection_offer')

async sendConnectionOffer(

@MessageBody()

offer,

roomName,

offer: RTCSessionDescriptionInit;

roomName: string;

@ConnectedSocket() socket: Socket,

this.server.in(roomName).except(socket.id).emit('send_connection_offer', {

offer,

roomName,

Sending the answer

The other person in the room must prepare the answer as soon as they receive the offer. First, we need to call the setRemoteDescription function to acknowledge the offer. Then, we must create the answer and send it to our NestJS server through the WebSocket.

useSendingAnswer.tsx

import { useCallback } from 'react';

import { socket } from '../socket.tsx';

import { useParams } from 'react-router-dom';

export function useOffersListening(peerConnection: RTCPeerConnection) {

const { roomName } = useParams();

const handleConnectionOffer = useCallback(

async ({ offer }: { offer: RTCSessionDescriptionInit }) => {

await peerConnection.setRemoteDescription(offer);

const answer = await peerConnection.createAnswer();

await peerConnection.setLocalDescription(answer);

socket.emit('answer', { answer, roomName });

[roomName],

return {

handleConnectionOffer,

We should configure our frontend application to send the answer once it receives the offer.

useChatConnection.tsx

import { useParams } from 'react-router-dom';

import { useCallback, useEffect } from 'react';

import { socket } from '../socket';

import { useOfferSending } from './useOfferSending.tsx';

import { useSendingAnswer } from './useSendingAnswer.tsx';

export function useChatConnection(peerConnection: RTCPeerConnection) {

const { roomName } = useParams();

const { sendOffer } = useOfferSending(peerConnection);

const { handleConnectionOffer } = useSendingAnswer(peerConnection);

const handleConnection = useCallback(() => {

socket.emit('join_room', roomName);

}, [roomName]);

useEffect(() => {

socket.connect();

socket.on('connect', handleConnection);

socket.on('another_person_ready', sendOffer);

socket.on('send_connection_offer', handleConnectionOffer);

return () => {

socket.off('connect', handleConnection);

socket.off('another_person_ready', sendOffer);

socket.off('send_connection_offer', handleConnectionOffer);

}, [roomName, handleConnection, roomName, handleConnectionOffer]);

We should configure our server to get the answer and pass it to the other person in the chat room.

chat.gateway.ts

import {

ConnectedSocket,

MessageBody,

SubscribeMessage,

WebSocketGateway,

WebSocketServer,

} from '@nestjs/websockets';

import { Server, Socket } from 'socket.io';

@WebSocketGateway()

export class ChatGateway {

@WebSocketServer()

server: Server;

// ...

@SubscribeMessage('answer')

async answer(

@MessageBody()

answer,

roomName,

answer: RTCSessionDescriptionInit;

roomName: string;

@ConnectedSocket() socket: Socket,

this.server.in(roomName).except(socket.id).emit('answer', {

answer,

roomName,

Processing the answer

Finally, we need to process the answer to the offer. We need to start by acknowledging the answer through the setRemoteDescription method.

useAnswerProcessing.tsx

import { useCallback } from 'react';

export function useAnswerProcessing(peerConnection: RTCPeerConnection) {

const handleOfferAnswer = useCallback(

({ answer }: { answer: RTCSessionDescriptionInit }) => {

peerConnection.setRemoteDescription(answer);

[peerConnection],

return {

handleOfferAnswer,

We need to do that as soon as we receive the answer event.

useChatConnection.tsx

import { useParams } from 'react-router-dom';

import { useCallback, useEffect } from 'react';

import { socket } from '../socket';

import { useOfferSending } from './useOfferSending.tsx';

import { useSendingAnswer } from './useSendingAnswer.tsx';

import { useAnswerProcessing } from './useAnswerProcessing.tsx';

export function useChatConnection(peerConnection: RTCPeerConnection) {

const { roomName } = useParams();

const { sendOffer } = useOfferSending(peerConnection);

const { handleConnectionOffer } = useSendingAnswer(peerConnection);

const { handleOfferAnswer } = useAnswerProcessing(peerConnection);

const handleConnection = useCallback(() => {

socket.emit('join_room', roomName);

}, [roomName]);

useEffect(() => {

socket.connect();

socket.on('answer', handleOfferAnswer);

socket.on('connect', handleConnection);

socket.on('another_person_ready', sendOffer);

socket.on('send_connection_offer', handleConnectionOffer);

return () => {

socket.off('connect', handleConnection);

socket.off('another_person_ready', sendOffer);

socket.off('send_connection_offer', handleConnectionOffer);

socket.off('answer', handleOfferAnswer);

roomName,

handleConnection,

roomName,

handleConnectionOffer,

handleOfferAnswer,

sendOffer,

When we receive the stream from the other person in the room, it causes the track event to be dispatched. Let’s listen to it and store it in the state so that we can display it later.

usePeerConnection.tsx

import { useMemo, useState } from 'react';

import { socket } from '../socket.tsx';

import { useParams } from 'react-router-dom';

export function usePeerConnection(localStream: MediaStream) {

const { roomName } = useParams();

const [guestStream, setGuestStream] = useState<MediaStream | null>(null);

const peerConnection = useMemo(() => {

const connection = new RTCPeerConnection({

iceServers: [{ urls: 'stun:stun2.1.google.com:19302' }],

connection.addEventListener('track', ({ streams }) => {

setGuestStream(streams[0]);

localStream.getTracks().forEach((track) => {

connection.addTrack(track, localStream);

return connection;

}, [localStream, roomName]);

return {

peerConnection,

guestStream,

The icecandidate event

The icecandidate event in WebRTC is triggered when the local ICE agent discovers a new network candidate for establishing a peer-to-peer connection. This event provides the candidate’s information, which includes details like the IP address and port number. We must send this candidate to the peer we want to connect to. To do that, we need to send an event to our NestJS server.

usePeerConnection.tsx

import { useMemo, useState } from 'react';

import { socket } from '../socket.tsx';

import { useParams } from 'react-router-dom';

export function usePeerConnection(localStream: MediaStream) {

const { roomName } = useParams();

const [guestStream, setGuestStream] = useState<MediaStream | null>(null);

const peerConnection = useMemo(() => {

const connection = new RTCPeerConnection({

iceServers: [{ urls: 'stun:stun2.1.google.com:19302' }],

connection.addEventListener('icecandidate', ({ candidate }) => {

socket.emit('send_candidate', { candidate, roomName });

// ...

return connection;

}, [localStream, roomName]);

return {

peerConnection,

guestStream,

Now, let’s handle the send_candidate event by passing the data to the other person in the chat room.

chat.gateway.ts

import {

ConnectedSocket,

MessageBody,

SubscribeMessage,

WebSocketGateway,

WebSocketServer,

} from '@nestjs/websockets';

import { Server, Socket } from 'socket.io';

@WebSocketGateway()

export class ChatGateway {

@WebSocketServer()

server: Server;

// ...

@SubscribeMessage('send_candidate')

async sendCandidate(

@MessageBody()

candidate,

roomName,

candidate: unknown;

roomName: string;

@ConnectedSocket() socket: Socket,

this.server.in(roomName).except(socket.id).emit('send_candidate', {

candidate,

roomName,

The last step is to handle receiving the candidate.

useChatConnection.tsx

import { useParams } from 'react-router-dom';

import { useCallback, useEffect } from 'react';

import { socket } from '../socket';

import { useOfferSending } from './useOfferSending.tsx';

import { useSendingAnswer } from './useSendingAnswer.tsx';

import { useAnswerProcessing } from './useAnswerProcessing.tsx';

export function useChatConnection(peerConnection: RTCPeerConnection) {

const { roomName } = useParams();

const { sendOffer } = useOfferSending(peerConnection);

const { handleConnectionOffer } = useSendingAnswer(peerConnection);

const { handleOfferAnswer } = useAnswerProcessing(peerConnection);

const handleConnection = useCallback(() => {

socket.emit('join_room', roomName);

}, [roomName]);

const handleReceiveCandidate = useCallback(

({ candidate }: { candidate: RTCIceCandidate }) => {

peerConnection.addIceCandidate(candidate);

[peerConnection],

useEffect(() => {

socket.connect();

socket.on('answer', handleOfferAnswer);

socket.on('send_connection_offer', handleConnectionOffer);

socket.on('another_person_ready', sendOffer);

socket.on('connect', handleConnection);

socket.on('send_candidate', handleReceiveCandidate);

return () => {

socket.off('answer', handleOfferAnswer);

socket.off('send_connection_offer', handleConnectionOffer);

socket.off('another_person_ready', sendOffer);

socket.off('connect', handleConnection);

socket.off('send_candidate', handleReceiveCandidate);

roomName,

handleConnection,

roomName,

handleConnectionOffer,

handleOfferAnswer,

sendOffer,

handleReceiveCandidate

Displaying both video streams

We now have access to the video stream of the other person in the room through the guestStream variable. Let’s use it to render both videos.

VideoChatRoom.tsx

import { VideoFeed } from './VideoFeed';

import { FunctionComponent } from 'react';

import { useChatConnection } from './useChatConnection';

import { usePeerConnection } from './usePeerConnection.tsx';

interface Props {

localStream: MediaStream;

export const VideoChatRoom: FunctionComponent<Props> = ({ localStream }) => {

const { peerConnection, guestStream } = usePeerConnection(localStream);

useChatConnection(peerConnection);

return (

<div>

{guestStream && (

<div>

guest

</div>

Summary

In this article, we’ve gone into detail about what Web Real-Time Communication (WebRTC) is and what problem it solves. We explained how it allows people to communicate in real-time directly through web browsers and keeps the role of the server to a minimum. We implemented a WebSocket server using NestJS and a frontend application with React to show how it works.

By implementing a real-life example, we got from the basics and progressed into more advanced aspects. Thanks to that, we established a solid understanding of WebRTC and used this knowledge in our code.

Series Navigation<< API with NestJS #141. Getting distinct records with Prisma and PostgreSQL

Capturing the camera

useLocalCameraStream.tsx

VideoFeed.tsx

App.tsx

Creating video chat rooms

chat.gateway.ts

Dealing with Cross-Origin Resource Sharing

app.module.ts

websocket-adapter.ts

main.ts

Joining a room

App.tsx

vite-env.d.ts

socket.tsx

useChatConnection.tsx

VideoChatRoom.tsx

Connecting the people in the room

chat.gateway.ts

usePeerConnection.tsx

VideoChatRoom.tsx

Sending the connection offer

useOfferSending.tsx

useChatConnection.tsx

chat.gateway.ts

Sending the answer

useSendingAnswer.tsx

useChatConnection.tsx

chat.gateway.ts

Processing the answer

useAnswerProcessing.tsx

useChatConnection.tsx

usePeerConnection.tsx

The icecandidate event

usePeerConnection.tsx

chat.gateway.ts

useChatConnection.tsx

Displaying both video streams

VideoChatRoom.tsx

Summary

Recommend

About Joyk