Collaborative Workflows: Meeting The Global Time Zone Challenge

01.04.24Driven by the continued demand for visual effects, the industry is booming. Once upon a time, VFX studios were solely working on their own projects. Today, with the VFX surge, studios often rely on collaborating with other studios and vendors and utilize their skill sets. The majority of the larger VFX companies such as ILM, Digital Domain, Wētā FX and RISE VFX have multiple facilities across the globe. With this increase, as well as technological advancements, studios are able to take advantage of these skill sets and available resources in different locations around the world. With work being distributed between numerous vendors, there are many factors to take into consideration, including pipeline integration, asset and data exchange and project planning.

For RISE VFX, Blue Beetle and The Last Voyage of the Demeter have been shared shows between RISE Berlin and RISE Stuttgart teams, while RISE London and RISE Munich have been sharing Miss Marvel. Florian Gellinger, Owner/Executive VFX Producer at RISE VFX, details how it’s determined whether parts of the project will be handled in-house and which parts will be outsourced to external studios. “Usually, we break each show we work on down into creative and non-creative tasks. For us, it is of the utmost importance that we have complete creative ownership and control over every single aspect of the VFX work that we’re being trusted with,” Gellinger says. “Of course, we do share creative work between our own five facilities in Berlin, Cologne, London, Munich and Stuttgart from time to time as necessary because of skillset and provincial rebate requirements. But even any of the non-creative work that we consider outsourcing needs to be in accordance with our clients security requirements, audits and approval process. The work we usually outsource consists of rotoscoping, paint fixes and camera match moves, if we don’t have the capacity to do it ourselves.”

Lola VFX Executive Producer William Anderson explains the initial process for Lola VFX. “Our producers here will first review the shots with one of our in-house show supervisors and break down all of the tasks needed on a per-shot basis. Once all tasks have been defined, we will then consider our current staff and workload. We will typically outsource the more complex/arduous rotoscoping and 3D tracking as they tend to be our biggest bottlenecks. That being said, we always keep a good chunk of it in-house for our juniors to work on and gain experience from.”

For Digital Domain, the workload usually stays in-house while utilizing other Digital Domain facilities across the globe. “The current footprint of our global Digital Domain studio is such that we often consider multiple denominators as we assign the project work distribution between our Digital Domain locations. Should there be overflow work that we can consider some of our partner outsource providers, we will tap into them on an as-needed basis; however, our primary work distribution means within Digital Domain’s global studio footprint,” says Lala Gavgavian, Digital Domain President and COO.

“Digital Domain’s strategic locations across the globe… allow us the advantage to scale when needed and provide resources and access to various global productions in the various regions where we reside. The disadvantage we face is that our teams are not always able to work together in a single physical location. As a result, we need to incorporate additional layers of communication and team-building opportunities to ensure that we maintain strong human connections even when our teams are distributed across different locations.”

—Lala Gavgavian, President and COO, Digital Domain

ILM’s Jeanie King, Vice President of Production/Executive Producer, explains that they look at determining which parts of the project will be handled in-house and which parts will be outsourced to external studios when they are looking at the bidding stage. “There are many things we try to hit right with the client. First, we have to see the number that they want and the net number, and how do we get to that within the facilities we have,” King says. “We are looking at that and the show, and what capacity we have across our studios and what’s going to be more economical. We have collaborators in countries that have better rebates, so we will go to them and partner with them on certain shots and do it that way. We’ve been doing this for decades now, where we’ve joined and worked with these people and companies and have great relationships now. Over the years we have developed a shorthand with them which benefits everyone. Internally, we have a couple of people whose role is to make sure we are sourcing different companies that we might be wanting to work with, trying new relationships and keeping the relationships that we have alive and making sure that it’s working properly.”

When it comes to the data and assets exchange process, Gellinger notes that for the creative work that is shared with any of the five RISE facilities, they have a unified network architecture that makes sharing work very easy. “Files are automatically synchronized between our servers and tasks are assigned by our production teams to specific artists, no matter where they are based, initializing the synchronization. Sometimes, we will have artists log into a machine remotely at the office that is spearheading our efforts on a specific show, a common practice that has established itself as a standard for home office work during the pandemic anyway. The production database RISEBASE is our central hub to distribute work and track process across all sites. For non-creative outsource work, we use our Signiant server, sharing file sequences with approved third-party vendors, just like our clients would share the materials with us.”

For Lola VFX, the bulk of this exchange process is done in-house. “We try to always be cognizant of what we ourselves would ask for in order to complete specific tasks,” Anderson says. “Obviously, this can vary from project to project based on scope and amount of data captured on set. The actual exchange of data itself is a fairly simple process. We utilize secure and approved file transfer systems such as Aspera to deliver EXRs, QTs, Maya files, etc. Once the vendor has received the material, they will review and let us know if they have any additional requests. Based on their feedback, we will then try to accommodate as best we can.”

For digital artist Xi “Jake” Wang, currently employed by Refik Anadol Studio, there are the occasional issues with the data and assets exchange process. “This process is not always linear, as clients may occasionally lack sufficient data and resort to our proxy files for updates. For example, when working on the previz scene for HBO’s TV series The Last of Us, set in the Bostonian museum, the initial data received from the client included only the basic 3D environment and stage set blueprints. There were no textures and concept art provided. Consequently, I had to conduct research related to the original video game and the physical location it was based on.”

Michael DiComo, Head of Computer Graphics at ILM, notes that externally, there are fairly common sharing standards. “Every studio’s pipeline is different; everyone’s renderers are different, lookdev, texture, etc.,” he explains. “We provide the basic building blocks that then any company can say, ‘We’ll figure out some ingest scripts to say ‘we know how you are delivering the stuff, this is how we can ingest and convert it into our own pipeline needs.’ From there, we can say it comes into our package for us to do look development on it. We can match the material, the skin, the cloth, the hair and all the other things. All that stuff is some form of a very low common denominator of shareable data, which is typically Maya and a subdirectory of textures or classic texture maps, defuse maps and specular maps and bump maps – all those things that have been around in computer graphics forever. That’s the way it has been, and it’s still pretty common to keep doing that.”

Lola VFX and RISE Visual Effects Studios shared VFX on Stranger Things with DNEG, Alchemy 24, AB VFX Studios, BOT VFX, Crazy Horse Effects, Fix FX, MARZ, Rodeo FX, Spin FX, PowerHouse VFX and others. (Image courtesy of Netflix)

“We have collaborators in countries that have better rebates, so we will go to them and partner with them on certain shots and do it that way. … Internally, we have a couple of people whose role is to make sure we are sourcing different companies that we might be wanting to work with, trying new relationships and keeping the relationships that we have alive and making sure that it’s working properly.”

—Jeanie King, Vice President of Production/Executive Producer, ILM

“USD is a format that Pixar, our sister company, developed and subsequently open-sourced,” DiComo continues. “It’s now a global standard that most visual effects companies use. It’s an open way to describe all of the components that go into assets and into shots. With more and more software vendors and visual effects companies adopting USD as the backbone of their shot and asset pipeline, we realize that there are ways to do a higher common denominator of sharing, where you can put together an asset in a way with all its component files. For example, right now, there’s a shader called Autodesk Standard Surface. It comes for free with Maya and virtually every company uses Maya, or at least has a license floating around. If you have Maya, you can render an Autodesk Standard Surface asset. We’re working with other vendors who have similar concepts on how we can share things that will make it easier for all of us, because the game of visual effects now is not one VFX studio getting everything, for the most part. Jobs are split between studios, and it lowers the risk by having multiple vendors on it.”

Freefolk VFX is a U.K.-based company who have recently worked on projects such as Napoleon and Fast X, and stresses the importance of collaboration. “We are a resident team of approximately 100 with space for further artists both in our new Spitalfields building, and of course any number of remote artists able to join us too,” says Freefolk VFX Executive Producer Meg Guidon. “Collaboration is great – it works best when all parties are happy to be on board together and sharing the load as it were – then all communications flow, opinion, assets, skills are all shared and result is a reality-positive effort that bonds people, forges new working relationships via humor, trust and hard work.”

DNEG was one of the leading vendors on Fast X. Others included Lola FX, Freefolk VFX, Belo FX, Outpost VFX, Ghost VFX, BlueBolt and ILM. (Image courtesy of Universal Pictures)

The advantage of a worldwide collaborative workflow for Digital Domain is first getting access to global talent and unique skill sets that are required for specific projects. “Digital Domain’s strategic locations across the globe allow us to collaborate and have access to talent globally without the requirement for relocation to one specific hub. This allows us the advantage to scale when needed and provide resources and access to various global productions in the various regions where we reside. The disadvantage we face is that our teams are not always able to work together in a single physical location. As a result, we need to incorporate additional layers of communication and team-building opportunities to ensure that we maintain strong human connections even when our teams are distributed across different locations,” Gavgavian adds.

“The successful implementation of such global workflows necessitates meticulous strategic planning, deployment of effective communication tools, establishment of robust security protocols and the vigilant oversight of proficient project management. These measures collectively serve to ameliorate the inherent challenges while capitalizing on the manifold advantages conferred by this collaborative approach.”

—Xi “Jake” Wang, Digital Artist

“I can’t think of many projects that we are not collaborating on with external vendors. Similar could be said for any of our Star Wars projects. Tapping into global resources is really great,” DiComo says. “As tough as it can be dealing with different time zones, there is something magical about coming in to work to a pile of shots to review that you feel like magically arrived overnight. You don’t have people working graveyard [shifts] anymore in San Francisco, like in the old days.” King also supports this view and adds: “If you are in multiple sites around the globe, as a company you are almost working 24/7. Logistically, it’s challenging, but our production teams are top notch.”

Time zones are simultaneously the biggest advantage and disadvantage of worldwide collaboration for Lola VFX. “Our main facility is located in Los Angeles, but we have a location in London as well. Since the two are almost a full workday apart, we can task a shot to a U.K. artist at the end of our day, let him work on it while we sleep and wake up to an excellent first pass for review. While this is beneficial, it can also be disadvantageous when our U.K. artists are working against U.S.-based deadlines. These same factors, good and bad, apply to our collaboration with vendors located throughout the world. Regardless, the advantages far outweigh the disadvantages. Having a worldwide workflow is a gift that we utilize every day. We have the ability to collaborate with experts across the globe and at the end of the day, I believe it truly shows in the quality of our work,” Anderson explains.

“These same [time zone] factors, good and bad, apply to our collaboration with vendors located throughout the world. Regardless, the advantages far outweigh the disadvantages. Having a worldwide workflow is a gift that we utilize every day. We have the ability to collaborate with experts across the globe and at the end of the day, I believe it truly shows in the quality of our work.”

—William Anderson, Executive Producer, Lola VFX

Global collaborative workflows offer numerous advantages, as Wang details: “Most notably, the unfettered access to an international pool of talent. In the contemporary landscape, cutting-edge global workflows and technological advancements empower both individuals and studios to harness this extensive talent reservoir. Given that VFX production typically entails diverse teams comprising artists, technicians and creative professionals collaborating harmoniously, this dynamic facilitates the acquisition of specialized skills and innovative creative insights, often beyond the scope of local resources. Furthermore, it substantially bolsters production efficiency by facilitating 24/7 operations across disparate time zones. To illustrate, industry-leading VFX studio Digital Domain, boasting three decades at the forefront of its field, has expanded its global presence, with offices situated in Los Angeles, Vancouver, Montreal, Beijing, Shanghai, Hong Kong, Taipei and Hyderabad.”

RISE London and RISE Munich share the VFX tasks on the Disney+ series Ms. Marvel. Also in the mix are PixStone Images, MARZ, Od Studio, Perception and Trixter. (Image courtesy of Marvel Studios and Disney+)

Continues Wang, “Nonetheless, global collaborative workflows present a suite of challenges, spanning communication intricacies, security considerations and more. For instance, during my tenure as a freelance 3D designer on a music video project, I encountered a situation where the VFX team was situated in Vietnam. Due to the utilization of an early preview version of UE5 and associated communication lapses, we encountered unexpected delays in rectifying issues stemming from improper handling of the 3D scene within UE5. In summation, the successful implementation of such global workflows necessitates meticulous strategic planning, deployment of effective communication tools, establishment of robust security protocols and the vigilant oversight of proficient project management. These measures collectively serve to ameliorate the inherent challenges while capitalizing on the manifold advantages conferred by this collaborative approach.”