Moviemakers continue to up the ante in their quest to make film animation as realistic as live action, thanks to improvements in 3-D computer-generated (CG) graphics. These efforts can pay off in big ways—James Cameron's Avatar earned a mountain of money and three Academy Awards. But, as New Zealand digital effects–maker Weta Digital can attest, painstakingly creating three-meter-tall blue bioluminescent aliens required an unprecedented amount of computing power and data storage—and those resources are likely be dwarfed by subsequent projects.

"The biggest changes I've seen are in the complexity of the movies, starting with The Lord of the Rings," says Paul Gunn, data center systems administrator for Weta, the company responsible for the stunning visuals in The Lord of the Rings film trilogy as well those in as Avatar. A significant part of this complexity comes from moviemakers' demands that digitally rendered characters and scenes become more lifelike even when shot in close-up.

The main job of Weta's data center is rendering, a process that adds texture, shading, reflection and other visual aspects to digital images, "turning them into something we can produce on the screen," Gunn says. Avatar's graphics rendering required the services of more than 4,300 computer servers (containing nearly 35,000 central processing unit cores) to process digital images into movie-quality visuals—a system the company refers to as its "renderwall". For Avatar hundreds of visual effects artists fed terabytes' worth of work into the renderwall, which refined those computer-designed images into something closer to the finished product.

"We can't predict what an artist will need so we have to provide them with a smorgasbord of resources to work with," Gunn says. "That can be troublesome, because the dynamics of our data center environment changes quickly. We end up with fairly large surges in demand from several artists for a particular movie shot, which consists of a pile of individual frames."

This boils down to a heavy demand for similar or the same pieces of data, such as the computer code that creates the texture of the leaves in the rainforest on Pandora, the Saturnian moon where Avatar is set. "Texture is a set of data that's commonly used to give the movie a uniform look," Gunn says. "In the past we had this code on lots of different file servers." This was inefficient because the same four-terabyte master copy of the movie's images resided in 10 other locations throughout the data center. If the data were updated on one server, Gunn and his team would then have to make sure that same data were updated on all the servers on which it was stored.

For Avatar, instead of generating 40 terabytes of data that included 10 copies of the same information, Weta and data storage provider NetApp in Sunnyvale, Calif., devised a system that gave artists access cached data. A cache is a temporary memory buffer used to store data that is used most often, a setup designed to make data access faster and more efficient. Weta and NetApp created several caching servers on Weta's network to handle the large number of users requesting access to the movie's visual effects files.

All of the renderwall machines accessed these NetApp-caching file servers, which in turn accessed the master copy of the movie's images. When changes were made to the master images, these changes would automatically be reflected in the caching file servers as well, with minimal lag time. The cache servers held only the data most in demand by the movie artists, which turned out to be about 800 gigabytes of the original four-terabyte data set. "However, that 800 gigabytes of data was enough to answer more than 97 percent of all data requests," Gunn says.

NetApp's FlexCache software was used to automatically balance the renderwall's throughput requirements to keep data request bottlenecks from forming. "The key to caching is to understand what data is most in demand and who's demanding it," says Brendon Howe, a NetApp vice president and general manager.

Gunn says he cannot talk about any of the technology Weta is using for current and future movie projects, including The Hobbit or Planet of Apes sequel Rise of the Apes, but he can say that since the company implemented the caches for Avatar they have been able to improve the technology to make their system even more efficient.

3 Comments

1. 1. FINNesse 01:03 PM 6/23/10

   As a Visual Effects Professional this story interested me but I did not expect to be so pleased with the way in which the technology was explained. The processes were made understandable to both myself and the majority of readers with an interest in the technology used in the creation of these complex effects.
   Following up and reading several more of Larry's stories proved that his style and the word wise descriptions he uses work. They work on the most complex and the most difficult to understand technical issues faced by the world today. If more people could or would read these works I'm sure that they would understand much more clearly and be much less frustrated by what is happening in our world with the technology we now have access to.
   Most people are scared away by thechnical and scientific stories in print because is is so easy to get lost in the use of terminolgy and rethoric tha very few of us understand.
   
   Thank you Larry your work it is greatly appreciated.

   Reply | Report Abuse | Link to this

2. 2. Moodie-1 03:51 PM 6/23/10

   As good as Avatar's CG was it still can't hold a candle to the CG in the Pirates Of The Caribbean movies, which IMO are the benchmark for realistic CG creatures. Just compare Davy Jones (or any of his shipmates) to any of the Pandorans and it's clear that PotC takes the prize. In fact, anything CG in these older movies is equally as good (or better!) than anything in Avatar. Re Avatar, either the lighting or shading or motion capture (probably all three) are just off! When I saw this film (in 3D) it really surprised me how the scenes with Jake Sully in his human body looked as realistic as with any other film but as soon as he was loaded into his Pandoran body (and in all subsequent scenes) he acquired a simplistic, unnaturally-lit 'fake' appearance. And every time a Pandoran moved it seemed that the CG wasn't taking gravity into account properly. None of the Pandoran creatures seemed to have the appropriate weight. (And if Pandora had less gravity than Earth it sure wasn't stressed well enough in the movie). None of the PotC movies had any of these problems. In fact the scenes of the kraken attacks (in PotC #2) are positively amazing! It's obvious to me that someone somewhere is doing something wrong with Avatar. James Cameron please take note! You can learn a lot from the CGers who worked on the PotC franchise!

   Reply | Report Abuse | Link to this

3. 3. Quinn the Eskimo 08:54 PM 6/24/10

   At $20 per ticket for 3D movies, I think, the future may not be as colorful as the money hungry gluttons of Hollywood expect.
   
   This Rose will not bloom (at $20) for that long.

   Reply | Report Abuse | Link to this