Launch HN: Lifecast (YC W22) – 3D video for VR

Launch HN: Lifecast (YC W22) – 3D video for VR

Launch HN: Lifecast (YC W22) – 3D video for VR 28 points by fbriggs 1 hour ago | hide | past | favorite | 9 comments Hi HN, I’m Forrest of Lifecast (https://www.lifecastvr.com), with my co-founder Mateusz. We make software to create 3D video for VR, robotics simulation, and virtual production. We convert any VR180 video or photo into our 6DOF VR video format, or into meshes compatible with Unreal Engine. Our 3D reconstruction is based on computer vision for dual fisheye lenses and deep learning.

VR video can be categorized as 3DOF (three degrees of freedom) or 6DOF (six degrees of freedom). 3DOF responds only to rotation, while 6DOF responds to both rotation and translation—meaning you get to move your head. VR games are 6DOF, but most VR videos are 3DOF. 3DOF can cause motion sickness and eye strain due to incorrect 3D rendering. 6DOF VR video fixes these problems for a more comfortable and immersive experience, but it is harder to make because it requires a 3D model of each frame of video.

There are some prototypes for 6DOF VR video systems in big tech, but they typically involve arrays of many cameras, so they are expensive, not very portable, and generate an impractical amount of data. Because of these challenges, 6DOF hasn't been widely adopted by VR video creators.

In 2015 I was working on ads at Facebook, but I was more excited about VR. I built 3D cameras out of legos and GoPros, showed some of this at a hackathon, and eventually they let me do that as my day job. I was the first engineer on Facebook's 3D VR camera team, which made Surround 360 (an open-source hardware/software 3D VR camera), and Manifold (a ball of 20+ cameras for 6DOF). After Facebook, I was a tech lead on Lyft's self-driving car project, and Google X's everyday robot project.

I started Lifecast because I wasn't satisfied with the progress on 6DOF VR video since I left Facebook. I learned new ideas from robotics which can improve VR video. The Oculus Quest 2 has just enough power to do something interesting with 6DOF. There have also been advances in computer vision and deep learning in the last few years that make it possible to do 6DOF better.

Our software makes it simple to create 6DOF VR video using any VR180 camera. It's a GUI for Mac or Windows, which takes VR180 video or photos as input, and produces Lifecast's 6DOF VR video format (more info: https://fbriggs.medium.com/6dof-vr-video-from-vr180-cameras-...). VR180 video can be created with any VR180 camera; the Canon R5 is one of the best on the market right now. We make a video player for WebVR which runs on desktop, mobile or VR. Playing the videos on the Quest 2 doesn't require installing any software, just visiting a web page in the Oculus Browser.

In addition to our 6DOF format, the software can also output point clouds (.pcd) or triangle meshes (.obj) compatible with Unreal Engine. We are seeing interest in using this for virtual production (2D film-making in a game engine), and creating environments for robotics simulation.

This recent video review/tutorial does a nice job of explaining our tech: https://www.youtube.com/watch?v=_4a-RnTLu-I (video by Hugh Hou, not us). For something more interactive, the thumbnails on https://lifecastvr.com are links to demos that run in browser/VR.

6DOF VR video is one piece of a larger puzzle. We envision a future where people wear AR glasses with 3D cameras, and use them to record and live-stream their experience. 3DOF is not sufficient for this because it causes motion sickness if the camera moves. We have prototypes which fix motion sickness in 3D POV VR video from wearable cameras. Watching the videos in VR feels like reliving a memory. Here's a demo: https://lifecastvr.com/trickshot.html

You can download a free trial from https://lifecastvr.com after entering your email address, but do not need to create a full account. The free trial is not limited in any way other than putting a watermark on the output.

We’d love to hear your thoughts and experiences about VR video, virtual production and robotics!