When Riot Games wanted to add an eye-catching augmented reality extravaganza to the Pro League Finals event based on their League Of Legends game characters, the company decided to partner with Pixotope Technologies and their innovative mixed reality platform.
League of Legends can be described without exaggeration as an internet phenomenon. The team-based strategy game enables players to choose a ‘champion’ to play as. With a unique backstory and specific skills, each champion helps drive player affiliation in the same way that real-world sports players attract a fan base.
The venue for the live esports event was Shanghai’s 18,000-seat Mercedes-Benz Arena and the pinnacle of the augmented reality content featured Akali, one of League of Legends’ most popular champions, singing and dancing with real dancers on stage, before being interviewed live in front of the packed arena.
Pixotope technology was to be found at the heart of the AR action.
● Over 100 million monthly players
● Riot Games generates annual revenue in excess of $1.5 billion
● 4 Pixotope augmented reality platforms
● 4 live and backup Pixotope engines
“We felt there was an opportunity to utilize AR to bring a live, real-time performance of one of our champions, which became the live interview of Akali with Candice, one of the event hosts. We found that the most compelling cases for AR character use had taken place in live, unscripted expressions. There is something magical and authentic when a virtual character interacts ‘in the moment’ that goes beyond a fully scripted and pre-recorded expression. AR allowed us to marry the virtual character to the real world.”
Kevin Scharff, Riot Games
“Virtual idols are very popular in China, and the technology and people’s understanding of virtual idols are maturing super-fast day by day. Therefore, bringing Akali, the most popular champion from the virtual band K/DA back to the LPL Summer Final stage, seemed to be a great opportunity to further deepen our IP and technology, celebrate our hard-core fans, and hopefully reach a more pan-entertainment audience in China.”
Sophie Xiao, Riot
Akali is a CG character and even though augmenting a CG character to shots from the live event doesn’t sound that difficult, there is a great deal involved. The level of interaction with show lighting, free-moving cameras, recorded and live motion capture, as well as facial expression capture with voice lip-sync, led to a uniquely complicated scenario to achieve the end results.
“To complement the Riot team’s efforts in developing the character assets for the event, we called in Pixotope, who were such fantastic partners at Worlds 2018 in Korea, to help bring the AR experience together.”
Kevin Scharff, Riot Games
Pixotope is not only a developer of a mixed reality platform but also retains expertise across all the various disciplines that are called into play when creating an augmented reality event. This enables the team to work closely with their customers’ creative departments to bring technically complicated ideas to life, while also having the capacity to suggest alternative routes and how to achieve the maximum impact from ever-advancing technology.
For the Riot Games event, Pixotope deployed a team of five augmented reality experts, including the Chief Creative Officer, Øystein Larsen, and Senior Product Specialist, David Stroud, to assess and solve the various challenges.
A significant challenge to solve when adding an augmented character is to match the lighting of the background scene. A secondary function of lighting is the creation of shadows. The human eye is curiously adept at calling out ‘fake’ images where shadows have not been correctly simulated. To avoid this, Pixotope enabled the use of ‘ray-traced’ shadows for Akali’s interview. Computationally time-consuming ray tracing is normally the domain of long-render-time post-production.
However, thanks to Pixotope having full use of the Unreal Game Engine, together with a bespoke custom rendering pipeline, these highly accurate shadows could be produced in real-time. Since the show was broadcast at 59.94 frames per second, challenging mathematics had to be completed and composited over every single frame of the live-action in less than 17 thousandths of a second.
“Failure to achieve this accurately leads to characters that look ‘stuck on’ and not embedded into the real-world scene. The specific challenge for this event was that Akali was to perform a dance song under pulsing show lighting that synchronized with a music track. Added to this, projected images were mapped to the arena floor.”
Øystein Larsen, Chief Creative Officer, Pixotope
“We used high dynamic range (HDR) site photography to accurately map the real-world lighting rigs to the virtual world created within Pixotope, where the digital Akali would be lit before being composited over the live camera shots. The arena lighting was pre-programmed and driven by timecode, as were the virtual lights within Pixotope, to ensure synchronicity.”
David Stroud, Product Specialist, Pixotope
One of the key aspects of this production was that Akali had to move realistically. Specifically, she first had to dance in sync with real-world dancers and then later, had to respond to a live interviewer and audience. The dance moves were directed and choreographed by the Riot team and then motion captured in Los Angeles by Animatrik Film Design. The movement data was combined with the Akali character model within Pixotope and produced in real-time to a timecode reference to ensure synchronization with the music track.
For the interview sequence, Pixotope replaced the pre-recorded dance motion capture data with a live motion capture stream derived from an actress performing off-stage. This enabled Akali to generate moments of interactivity and spontaneity which were so sought after.
Apart from the full-body motion capture, Cubic Motion’s Persona system streamed facial capture data live from the off-stage actress, who was also providing the voice for Akali. Of course, the arena was a very noisy environment so in fact, the body and face motion capture, together with audio recording, took place some 100 meters away in a soundproofed anechoic room. Pixotope combined both the body and facial motion capture feeds to drive the geometry of the CG Akali model. This in fact required some specific intervention from Pixotope technicians. When making games, motion capture data is recorded, cleaned up, and stored within the game to be recalled during gameplay.
With live events such as this, the motion capture data needs to directly drive the CG geometry live.
“We built the systems in Manchester and were joined by the actress who would provide both the voice and actions for Akali. Together with the creatives from Riot, we helped her perform at the appropriate level of expressiveness and get used to working in isolation away from her interviewer. Also, we were able to work through and solve the various technical challenges, such as synchronizing body and facial motion capture data together with vocal audio.”
David Stroud, Product Specialist, Pixotope
“We worked together across three continents for this experience – in Los Angeles for the original motion capture of the dance beat, in Shanghai for casting, modeling, location scouting, and pre-visualization, and in Manchester for a technical test and capture of the live interview, and finally in Shanghai for the live event. The key to the success was the collaboration and seamless integration of Riot teams in LA and China with world-class partners in Pixotope, Animatrik, and Cubic Motion.”
Kevin Scharff, Riot Games
Having completed the lighting and motion setups for Akali, the next challenge was to ensure that she would be rendered from a viewpoint that precisely matched the carefully choreographed live-action dancers.
In fact, there were three different live-action shots to match; one from a moving camera mounted on a jib arm, one on a free-roaming Steadicam, and one high above the arena. Each camera feed was sent to a dedicated Pixotope system, along with camera positional, directional, and lens information, encapsulated in a data stream provided by camera tracking specialist, Stype.”
Øystein Larsen, Chief Creative Officer, Pixotope
Each of the three Pixotope systems then created a matching camera inside their respective virtual scenes to “photograph” the digital, moving, lit Akali model and composite her perfectly onto each of the respective camera feeds. Those feeds were then sent to the broadcaster’s production switcher, just as normal camera feeds would otherwise have been. The resulting program was broadcast both externally as well as shown on giant screens within the arena so that the attending audience could enjoy the augmented reality experience.
Live broadcasts and events inevitably throw up last-minute and unexpected challenges to solve. As the venue was sponsored by Mercedes Benz, two of their cars adorned the stage and periodically, went for a spin - dislodging and scattering precisely laid out AT tracking markers in the process.
“I can still hear our AR camera lead Luka from Stype screaming in anxiety each time those cars started moving around and him scrambling to recalibrate the AR setup each time. But you know what was magical? All along the way, the team, Rioters, and incredible partners were total professionals throughout it all.”
Kevin Scharff, Riot Games
“It is so important in live events and broadcast situations to have
technology and people that can react quickly to evolving issues. The Pixotope system we make and the expertise we have is the product of many adventures in AR production. There’s always some unexpected element that needs to be accommodated.”David Stroud, Product Specialist, Pixotope
The event was incredibly successful, generating millions of views via broadcast and lasting social media engagement. The live interview with Akali, complete with real-time ray tracing at 60 FPS, was groundbreaking.
“Pixotope was critical to the success of the event in the use of their technology. They ensured the core of the experience – the fantastic and unexpected arrival of Akali who then fully integrated into a killer dance with live performers and the magical illusion of Akali and Candice having a post-match interview shined front and center. The magic trick was not the point though. What I love about Pixotope and its implementation of AR is that it met our goals of creating this fantastic immersive player experience of having a virtual idol perform at the biggest regional League of Legends event in China. From the visual smoke effects to the matched lighting of her glow stick, they ensured Akali looked and felt present at the event. They also solved every crazy hurdle that came upon this project, all with total professionalism and a great sense of humor. They are a stacked team of creative and technical wizards, that is for sure.”
Kevin Scharff, Riot Games
“It was a privilege for us at Pixotope to be so central to such an amazing event. Riot Games is a great client. They are bold and inventive and allow us to really push the envelope as to what is possible in live AR, while also helping to lay the foundations as to what incredible things come next.”
Øystein Larsen, Chief Creative Officer, Pixotope
Pixotope supports advanced augmented reality solutions for ambitious productions across various industries.
Read more about Pixotope Augmented Reality Solutions.