Wave

Live virtual entertainment & interactive concerts

director, vfx art lead, tech art lead may 2018 - feb 2024

Unity
C#
Unreal Engine
HLSL
Houdini
Mobile & desktop VR

Wave is one of the most prominent pioneers in real-time, live & interactive virtual concerts and entertainment. From multi-player VR to interactive live streams, Wave developed a new way for experiencing live music in a fully immersive, interactable 3D virtual world, blurring the lines between video game and virtual metaverse.

Throughout my 6 years at Wave (4 full time, 2 as a contractor) I worked on a tremendous amount of things from directing show production, developing visual effects & shaders and building out systems in Unity & Ureal to enable content production for large & small teams.

The challenge

Bringing live music to the virtual world

Instead of replacing in-person concerts, Wave introduced a new way to experience them, in both VR and interactive live streams that fans across the world can experience together live, with interactive elements and fantastical audio-visual experiences.

This comes with numerous challenges:

Capturing the performers live. Audiences relate a great deal to the visual presence of a performer, and capturing that with body/face tracking is already difficult let alone doing it in real-time with no cleanup / post processing
Enabling the performer and audience to truly feel that they are co-habitating the same virtual space together (eye contact & acknolwedging someone in the audience are both very impactful for example)
Creating engaging interactions for live stream (thousands of participants and only one shared screen, tension between linear performance and gamified interactions)
Heavy technical constraints when working in networked mobile VR, which is in direct conflict with needing to work fast and create visually stunning content to keep audience attention, because there is no core game loop, rather the performance itself is the main draw
Creating bespoke shows for artists under difficult time crunches, with the restraints of real-time tech and with very little access to the musicians you are working with

My role

Directed teams to produce flagship concerts including for The Weeknd, REZZ, Tinashe, Galantis & Pentakill (Riot Games’s virtual metal band)
Greatly simplified execution of show visuals with our custom performance tools, going from 3-4 people down to 1-2 through better UX, automated camera systems, MIDI based timeline content and more
Helped the company grow from 10-20 people to over 100, working on many crucial demos for investors as well as spearheading proper production pipelines & design guidelines as the company grew
Made countless shaders, visual effects, lighting rigs and camera systems for audio-reactive, highly expressive real-time visual content with live mocapped performers
Designed & built out systems within Unity and Unreal used by our art and tech-art teams to create and execute live shows
Helped build out the Wave Maker, spatial UI in their multi-player VR experience that enabled anyone to make their own audio-visual shows
Directed motion capture shoots and VJ’ed for our live show broadcasts

Process

Real-time rendering

All of our shows were built with game engines (Unity and Unreal Engine) enabling immersive multi-player experiences and allowing environments, vfx, lighting, and camera movements to react dynamically during a live set.

There were two core systems at play:

In engine creator tools: Runtime & editor code inside Unity & Unreal for rapidly making content for the shows, hooking up interactions, audio-reactive logic…etc
Live performance tools: The tools that allowed us to perform shows live. Responsible for triggering events in game, routing live mocap and audio streams to avatars and vfx, enabled us to manually switch camera feeds, control broadcast elements… etc… basically anything needed for a live performance. These tools were shaped through years of building and performing shows and seeing what works.

MoCap-driven avatars

Performers and their instruments get tracked in real-time via advanced mocap systems. Their movement and facial performances are mapped instantaneously to custom 3D avatars.

Our mocap systems at Wave went through several iterations:

XSens: The very first system we used was XSens, an accelerometer based mocap system that required no cameras, but could suffer problems with drift quite easily that we were constantly battling around.
Vicon: For some of our bigger shows like with Justin Bieber and Pentakill, we used high-end film grade optical systems like Vicon. Tracking was incredibly precise but of course incredibly expensive.
Captuary: The final installment of mocap was the markeless camera based Captuary system. We owned and operated one of these systems in our Culver City based broadcast studio. This was the dream, artists did not have to put on any suit or markers

For the performer to see into the world without wearing a VR headset, we had screens and an immersive LED wall for them to watch.

Audience interaction loop

The audience is a core part of every Wave show. Every Wave show has had different ways of experiencing and interacting with the show, including:

Live VR multiplayer experience: talk with friends, explore the environment, trigger visual effects, get up close to the performer
live broadcast: interact through the broadcast chat. Vote for events to happen in the show, see your name in the show feed, and more
in game zoom feed: during the pandemic we came up with a way to pipe in a live private zoom audience into the show so fans could see their faces in the live show feed