Table of Contents
Professional visual works sorted by most to least recent.
Light Field Lab TIME - Best Inventions 2022
- High Fidelity
- Reaping Rewards
- Gary The Gull Cinequest 2018 - Most Innovative VR
Light Field Lab
What was Light Field Lab?
What Light Can Do
Light Field Lab created the world's best holographic digital displays. We're talking about Holodeck level stuff here. I was employee number 6 and got to watch it grow to 40+ people over 7 years. We made devices that could recreate bundles of light as they would have bounced off of real world objects. In other words, if our display was rendering a cube in front of the display surface and you had two people standing next to each other looking towards the display, the left person would see the front and left sides of the cube while the person on the right would see the front and right sides of the cube. This is not limited to 2 "views" or even the horizontal axis. If you add a child to the room, they would see the underside of the cube as well. And this is NOT your typical lenticular display with a few tens of "views" in which stepped transitions are visible. If you could fit a hundred people into the room with the display, each person would be seeing the hologram slightly differently just like a real object. We rendered and displayed the equivalent of hundreds or even thousands of views for each holographic pixel depending on the customer specifications.
Going back to the simple two tall and one short person example, we could also render the scene such that each person sees a different object. The child could see a monkey while the other two see something else. It should also be noted that because we are rendering to bundles of light, our holographic objects had view-dependent lighting and reflections, everything you'd expect from a real-world object. If you were to build a room in which all the walls, ceiling, and floor were our displays, you'd have a Holodeck. Not one quite as good as in Star Trek, but it would be as good as real-world physics would allow!
If you're really interested about this, the CEO gave quite a good talk at South by Southwest that you can watch on YouTube.
Electrons Into Photons
Traditional rendering cannot handle this so I prototyped and then directed the development of new renderers built from the ground up specifically for highly optimized rendering of light field information directly and threw out the concept of "views" entirely. Our display design was also modular, so we could build any size a customer was willing to pay us to make. Because of this, our custom renderer was also designed to work distributed over any number of GPUs and servers. Driving this much data also required going beyond the bounds of typical desktop environments. I got very familiar with configuring and automating lightweight X11 window managers, compositors, and Nvidia drivers to get the exact pixel output we needed for our displays.
The First Contact Experience
In Collaboration With: The SETI Institute
The alien was rendered using our custom real time renderer synchronized to the scene data within a running instance of Unreal Engine 5 containing all show logic and processing of motion and facial capture systems. This demo combined everything from the previous two press demos (Aztec and Chameleon) into a much larger production with the addition of: full body motion capture, motors, and professional OSC driven lighting and audio. Abstracted various networked physical control system APIs into Blueprint nodes to allow designers to easily activate things, like motors, safely and without breaking them.
Aztec | DEFY™ Experience
Interactive SOLIDLIGHT™ Real-Time Motion Capture
Integrated various cameras, sensors, and microphones with Unity and Unreal. Most demos used Orbbec, LeapMotion, Intel RealSense, and Unreal LiveLink. There were many versions of this demo. In this video we used LiveLink, RealSense, and Unreal to create a holographic avatar for an actor in another room that you can chat with. In other demos, the Orbbec was used to have an autonomous holographic character, controlled by Unity, know where everyone was in the room such that the character could turn its head and react based on viewer's locations and hand gestures in the room. Because investors kept asking for it, we added ChatGPT integration to create an interactive AI character.
Chameleon
Real World Integration with Digital SOLIDLIGHT™ Chameleon
Wrote control software for custom optical systems to be able to dynamically and spatially blend our digital holograms with traditional analog holograms. This enabled our digital holograms to appear in front of real objects such that you could look behind the digital hologram and see the physical objects of the analog hologram behind it without seeing through the digital hologram. Slightly transparent holograms (wings on a fairy for example) could use an alpha channel to control the transparency of the digital "shadow" on the analog hologram.
The chameleon was rendered using my custom light field optimized fork of Blender.
Lytro
Acquired by Google
I came to Lytro via its acquisition of Limitless and started working just 2 months before it was announced that Google would acquire Lytro.
Unity Light Field Integration
Integrated light fields into game engines. Worked closely with post production teams in order to integrate AR/VR interactivity with light field data using cross-platform techniques and targeting devices with varying performance envelopes.
Limitless Ltd
Acquired by Lytro
Shipped interactive narrative driven VR experiences for both Desktop and PS4. Developed tools to increase productivity, ease, and enjoyability of creating VR narrative driven experiences. Created and managed load balanced and auto scaled speech recognition servers.
Limitless Creative Environment
Platform for creating interactive VR experiences used by major Hollywood studios. Using the Vive with this product, users could author their interactive VR experiences fully within the headset without going back to traditional Desktop applications. Worked on the core animation authoring system, audio capture framework, server infrastructure, and interactivity authoring system. Featured at Siggraph Real Time Live 2017 where an interactive VR experience was created live on stage in just 6 minutes.
Reaping Rewards
Emotional VR interactive experience created in Unreal Engine 4 and prototyped and pre-vised in the Limitless Creative Environment. Featured at Siggraph VR Village. Directed by Matthew Ward (ex-Lucasfilm). Worked on the demo shown at AMD's Summer of Radeon during GDC 2017. Also worked on the PS4 VR trailer.
I have never seen such a wide variety of reactions from people that played this at Siggraph. Some people came out of it teary and thanking us for the experience, one person came out of it so angry that they didn't finish it and threatened to throw the VR headset at us! Thankfully, most people liked it.
Gary The Gull
Cinequest 2018 - Most Innovative VR
VR interactive experience directed by Mark Walsh (ex-Pixar). Oculus Rift Touch and PSVR launch title with Vive support. In charge of optimizing for and releasing on PSVR which included meeting or receiving waivers for specific TRC and PSVR Consult items. Also set up streamed voice recognition and refactored logic for branching narrative. Featured at Siggraph Real Time Live 2016. Created in Unreal Engine 4.