How Dropbox Replay keeps everyone in sync

How do you recreate the experience of an in-person screening room with a remote, distributed team? This is one of the reasons we built Dropbox Replay, our new video collaboration tool. Dropbox customers told us the shift to virtual work had turned their once-straightforward review sessions into lengthy, inefficient video calls. It was clear their previous in-person workflows hadn’t made an elegant transition online.

When we looked at the market for virtual screening tools, we mostly found expensive Hollywood-scale solutions or clunky DIY experiences run over video conferencing tools. Nothing quite offered the kind of collaborative, live, and synchronous playback experience we envisioned—something that would approximate the feeling of being together in a screening room, with shared feedback, cursors, and playback controls. We knew there was an opportunity for an accessible yet high-quality online screening experience where people could collaborate in realtime as they would in person, but virtually—and where everyone could see the same thing and seamlessly annotate what they were seeing in real time. We created Dropbox Replay’s Live Review feature in response.

But keeping a virtual screening room with multiple collaborators in sync is a harder problem to solve than you might think. Anyone in a Live Review session can pause, adjust the playback speed, or scrub to a different frame at anytime. This is great for keeping screenings open and collaborative,but it also means that everyone might be sending conflicting commands at once. What if two people try to change the position of the video at the same time? 

For Live Review to work, we need to make sure that everyone converges on the same playback state in a timely manner. When a person is ready to discuss a frame, they must be able to trust that everyone is seeing the same frame too.

Changing states

In a Live Review session, we care about two types of state—single client state and shared client state. The position of someone’s mouse cursor or the drawings they make on a video frame are examples of single client state. These can’t conflict with other clients, so they’re relatively easy to handle: changes in single client state are sent to the server, which echoes them to the other clients, and that’s that. 

The shared client state, on the other hand, is what keeps local playback synchronized between all of the clients in a Live Review session. When someone joins a Live Review session, their client opens a WebSocket connection with a Dropbox Replay server (we use an open source Go library called Gorilla, which is already used elsewhere within Dropbox). Anytime someone presses play, pause, or changes the position of the video, the client sends a message conveying this change to the server, which updates the shared client state and then sends it to everyone else in the session.

The playback state is encoded using Protocol Buffers, which offer an extensible, schematized, and space-efficient way to package the data. We like the combination of Protocol Buffers and WebSockets; Protocol Buffers don’t concern themselves with message framing and leave the format on the wire up to the developer. WebSockets, meanwhile, are largely content-agnostic and offer text or binary messages that are framed, delivered reliably and in order, and even take care of ping/pong heartbeats, which can keep sessions alive through proxies or firewalls that might otherwise terminate an idle connection.

In a perfect world, only one person would interact with the video at a time. But that’s not how most screenings work. Everyone has something to say, or something they want the group to see—often at the same time! It’s precisely because anyone can influence playback at any time that our protocol must be resilient enough to handle multiple concurrent interactions and still produce a consistent outcome for all participants.

Establishing an order of events

One approach might be to simply send all state changes—“video paused at frame” or “cursor is now at position”—to all other clients. However, when the state is shared between clients you can quickly see how this approach will lead to inconsistent states if more than one client changes the state at the same time—for example, if Patty makes a change, sends it to Steven, but Steven makes and sends a change before Patty’s change has arrived.