Rapid Evolution At The Intersection Of Ai And Vfx

Crafty Apes’s AI division came into existence during the pandemic. Company Co-Founder Chris LeDoux recalls, “It all started during COVID with me watching YouTube videos from authors like Bycloud and Two Minute Papers. Then our VFX Supervisor and resident mad scientist Aldo Ruggiero began to show me a number of incredible things he was using AI for on the film he was supervising.” It became clear to LeDoux “that AI was going to shake up our industry in a massive way.” He explains, “Developments in AI/ML seemed like they would create a fundamental shift in how we approached and solved problems as it relates to shot creation and augmentation. I knew we had to make it a top priority.” Since then, Crafty Apes has applied AI to a wide range of VFX projects, reflecting an accelerating implementation of AI technology by the visual effects industry.

LeDoux comments, “I can tell you that we have leveraged machine learning [ML] for tasks like deepfake creations, de-aging effects, facial manipulation, rotoscoping, image and video processing and style transfer, and the list continues to grow.” He notes that once AI tools are integrated into the pipeline, they “speed up the workflow drastically, lower the costs of VFX significantly, and allow the artists to put more time into creativity.”

Regarding the teaming up of AI with VFX, “the first challenge is really managing expectations, in both directions,” says Hanno Basse, Chief Technology Officer of Digital Domain. He adds, “We shouldn’t overestimate what AI will be able to do, and there is a lot of hype out there now. At the same time, it will have a significant and immediate impact on all aspects of content creation, and we need to recognize the consequences of that.”

Digital Domain

The industry is looking at “many concepts and implementations for AI and ML that are very promising, and [is] using some of them already today,” according to Basse. Digital Domain has utilized machine learning on high-profile movies such as Avengers: Infinity War and Avengers: Endgame, and She-Hulk: Attorney at Law limited series for Disney+. It also created a 3D “visual simulation” of famed NFL coach Vince Lombardi – with the help of Charlatan, Digital Domain’s machine learning neural rendering software – for the February 2021 Super Bowl.

“AI, and especially its close cousin, machine learning, have been in our toolbox for five years or so. We use it on things like facial animation, face-swapping, cloth simulation and other applications,” Basse says. “Our work on She-Hulk last year made extensive use of this technology. In fact, we don’t believe we could have delivered that many shots without it given the time and resources we had to work on this project. [We also did] some fantastic work with cloth simulation on Blue Beetle. We’re basically using this technology now on virtually any show we get to work on.”

Prior to that, the digital creation of the character Thanos’ face in Avengers: Infinity War was Digital Domain’s first major application of machine learning and utilized the Masquerade facial-capture system. Avengers: Endgame followed close on its heels. “Since then, DD has done a lot more work with this technology,” Basse remarks. “For example, we created an older version of David Beckham for his ‘Malaria Must Die – So Millions Can Live’ campaign and used our ML-based face-swapping technology Charlatan to bring deceased Taiwanese singer Teresa Teng back to life, virtually.”

Basse adds, “In general, machine learning has proven very useful to help create more photorealistic and accurate results. But it’s really the interplay of AI and the craft of our artists – which they acquired over decades, in many cases – that enables us to create believable results.”

Wētā FX

“We have been working with various ML tools and basic AI models for a long time,” says Wētā FX Senior Visual Effects Supervisor Joe Letteri. In fact, Massive software, employed all the way back on The Lord of the Rings, uses primitive fuzzy logic AI to drive its agents. Letteri notes, “Machine learning has also been prevalent in rendering for de-noising for years across the industry. For Gemini Man we used a deep learning solver to help us achieve greater consistency with the muscle activations in our facial system. It helped us streamline the combinations that were involved in complex movements across the face to build a more predictable result.”

Wētā changed its facial animation system for Avatar 2 and adapted new deep learning methodologies. Letteri says, “Our FACS-based facial animation system yielded great results, but we felt we could do better. As our animators and facial modelers got better, we needed increasingly more flexible and complex systems to accommodate their work. So, we took a neural network approach that allowed us to leverage more of what the actor was doing and hide away some of the complexity from the artists while giving them more control. We were also able to get more complex secondary muscle activations right from the start, so the face was working as a complete system, within a given manifold space, much like the human face,”

Letteri and his crew created another neural network to do real time depth compositing during live-action filming. He explains, “During that setup process, we utilized rendered images to train the deep learning model in addition to photographed elements. This allowed us to gather more reference of different variations and positions than we could feasibly get onset. We could train the system to understand a given set environment and the placement of characters in nearly every position on the set in a wide range of poses – something that would be impractical to do with actors on a working film set.”

Comments Letteri, “VFX pipelines are always evolving, sometimes driven by hardware or software advancements, sometimes through new and innovative techniques. There is no reason to think that we won’t find new ways to deploy AI-enhanced workflows within VFX. Giving artists ways to rapidly iterate and explore many simultaneous outcomes at the same time can be enormously powerful. It also has great potential as a QC or consistency tool, the way many artists are using it now.”

Autodesk

“AI has the potential to be revolutionary for VFX as artists design and make the future,” says Ben Fischler, Director of Product Management, Content Creation at Autodesk. “The Internet took shape over many years, and it took time for it to become a part of our daily lives, and it will be similar with AI. For the visual effects industry, it’s all about integrating it into workflows to make them better. It won’t be an immediate flip of the switch, and while certain areas will be rapid, others will take longer.”

It has been more than two years since Autodesk embraced AI tools in Flame. “Flame puts a sprinkle of AI into an artist’s workflow and supercharges it dramatically. Things like rotoscoping, wire removal and complex face matte creation are processes that go back to the origins of visual effects when we did things optically, not digitally, and they’re still labor intensive. These are the processes where a little AI in the right places goes a long way,” Fischler explains. “In the case of Flame, we can take a process that had an artist grinding away for hours and turn it into a 20-minute process.”

Autodesk recently launched a private beta of Maya Assist in collaboration with Microsoft. “It was developed for new users to Maya and 3D animation and uses voice prompts via ChatGPT to interface with Maya,” Fischler says.

Rising Sun Pictures

Some five years ago, RSP began collaborating with the Australian Institute for Machine Learning (AIML), which is associated with the University of Adelaide, on ways to incorporate emerging technologies into its visual effects pipeline. AIML post-doctoral researchers John Bastian and Ben Ward saw the potential for AI in filmmaking and joined RSP; they now lead its AI development team with Troy Tobin.

One of the multiple projects that benefited from their work was Marvel’s Thor: Love and Thunder, in which RSP applied data collected from a human baby (the grandson of former Disney CEO Bob Chapek) to a CG infant. Working in tandem with the film’s production team, they were able to “direct” their digital baby to perform specific gestures and exhibit emotions required by the script. According to the Senior VFX Producer on the film, Ian Cope, quoted on the RSP website, “The advantage of this technique over standard ‘deep fake’ methods is that the performance derives from animation enhanced by a learned library of reference material.” The look was honed over many iterations to achieve a digital baby that would seem real to audiences.

“The work we’re doing is not just machine learning,” adds Ward, now Senior Machine Learning Developer at RSP. “Our developers are also responsible for integrating our tools into the pipeline used by the artists. That means production tracking and asset management and providing artists with the control they need from a creative point of view.”

Working together across several projects, the AI and compositing teams have grown in their mutual understanding. Having explored this space early, “we’ve learned a lot about how the two worlds collide and how we can utilize their [AI] tools in our production environment,” observes RSP Lead Compositor Robert Beveridge. “The collaboration has improved with each project, and that’s helped us to one-up what we’ve done before. The quality of the work keeps getting better.”

Jellyfish Pictures

“AI and ML offer exciting opportunities to our workflows. and we are exploring how to best implement them,” says Paul J. Baaske, Jellyfish Pictures Head of Technical Direction. “For example, how we can leverage AI to create cloth and muscle simulation with higher fidelity. This is a really intriguing avenue for us. Other areas are in imaging – from better denoising, roto-masks, to creating textures quicker. But some of the greatest gains we see [are] in areas like data or library management.”

Baaske adds, “The key moving forward will be for studios to look at their output and data through the lens of ‘how can we learn and develop our internal models further?’ Having historic data available for training and cleverly deploying it to gain competitive advantage can help make a difference and empower artists to focus more on the creative than waiting for long calculations.”

Vicon

“One of the most significant ways I think AI is going to impact motion capture is the extent to which it’s going to broaden both its application and the core user base,” comments David “Ed” Edwards, VFX Product Manager for Vicon. “Every approach to motion capture has its respective strengths and shortcomings. What we’ve seen from VFX to date – and certainly during the proliferation of virtual production – is that technical accessibility and suitability to collaboration are driving forces in adoption. AI solutions are showing a great deal of promise in this respect.”

Edwards adds, “The demands and expectations of modern audiences means content needs to be produced faster than ever and to a consistently high standard. As AI is fast becoming ubiquitous across numerous applications, workflows and pipelines, it’s already making a strong case for itself as a unifier, as much as an effective tool in its own right.”

Studio Lab/Dimension 5

“I think with the use of AI we will see many processes be streamlined, which will allow us to see multiple variations of a unique look,” says Ian Messina, Director of Virtual Production at Studio Lab and owner of real-time production company Dimension 5. Wesley Messina, Dimension 5 Director of Generative AI, says, “Some trailblazers, like Wonder.ai, are pushing the boundaries of technology by developing tools that can turn any actor into a digital character using just video footage. This gets rid of the need for heavy motion-tracking suits and paints a promising picture of what’s to come in animation.”

Rising Sun Pictures’ machine learning incorporated data from a “learned library of reference material” to help create Baby Thor for Thor: Love and Thunder. An early adopter of AI, RSP used machine learning to give the baby and uncannily lifelike quality while exhibiting behaviors required by the script. (Images courtesy of Marvel Studios)

Wesley Messina adds, “As the technology becomes more widely available, we can expect to see AI tools being used by more and more creators. This will change the way we make movies and other visual content, bringing stories to life in ways we’ve never seen before.”

“The Internet took shape over many years, and it took time for it to become a part of our daily lives, and it will be similar with AI. For the visual effects industry, it’s all about integrating it into workflows to make them better. It won’t be an immediate flip of the switch, and while certain areas will be rapid, others will take longer.”

—Ben Fischler, Director of Product Management, Content Creation, Autodesk