Compositing for me is the art of manipulating the picture in motion.Filip Suska
Everyone has at least once wondered how the movies we can daily see at our closest cinema screening facility, the computer games we spend so much time playing or even the Netflix TV shows we got hooked on in the past years are being made? Let me give you a closer look at this topic. I currently study as a Visual Effects Artist (VFX), which means that I will soon join a fast-growing industry, which provides all the stunning visuals, some may call it computer-generated imaginary (CGI) or simply computer graphics, for our entertainment industry. To begin with, you should definitely know that there are always two ways to create these effects to be seen in the final render (picture). You can either physically create them on the set (real gunpowder explosions, car collisions, muzzle flashes, background buildings and pretty much anything you have the budget on) or just use the power of VFX and recreate the sets needed inside a computer.
Many people would argue about the realism of the effect you can produce using these techniques, but I guarantee you that most of the work that is done by the VFX department is not even noticed. In the end, the best recipe to create these stunning visuals is simply to add these two together. By placing on set special effects (SFX) side by side with VFX, you can easily trick the human eye to believe that what is happening in front of him is 100% real. Or at least could be possible.
Like every industry, the VFX industry has a long organized pipeline to get to the final product. At the end of this chain is the compositing department, whose role is to put together all the filmed and rendered assets produced by previous departments to create the final lifelike image.
Breakdown of compositing shot
How it all began
Compositing may be considered a very peculiar job, which roots go back to the birth of cinema. The first use of compositing techniques is dated all the way back to 1857 when Oscar G. Rejlander combined 2 different versions of 32 negatives to a single image. In the beginning, before the digital era, these effects were done all by cutting the film recorded by the camera and overlaying it on the top of each other to later be projected on the screen. All of the fundamental techniques used in this era were adopted and brought to the digital world after the computing power of our rendering machines increased. Hand in hand with the evolution of computers and the decrease in their price, the VFX industry got bigger and bigger to the stage we recognize it today. The creation of VFX spread out from Hollywood to cities like Vancouver, Montreal, London, Singapore, Sydney and many more. A lot of companies have been created for a specific purpose to play a role in the VFX world and slowly grew into big studios, which employ 1000s of people around the world. Weta, ILM, Framestore, Double Negative, Cinesite, Sony Pictures Imageworks, MPC, Animal Logic, Digital Domain, Method Studios are currently the leaders on the market and are looking for talented artists to become a part of this constantly progressing industry.
It is becoming more and more affordable for all of us, even you reading this blog to create some of the incredible images we see in the Hollywood movies, nicely and comfortably from our home office. Talking about the industry standards for compositing today there is one piece of software, which stands out the most. It is a handy and complex program called Nuke, produced by The Foundry. It offers a customizable interface and is based purely on mathematical expressions. For smaller studios and freelancers it is usually much more affordable to do their compositing in either Adobe After Effects or Fusion. There are many differences in the working procedure of this software, but we will likely talk about it later.
To learn more about all the work and practises used in compositing I would definitely advise to follow up with this blog, but if you are one of those “I want to know it all in detail” people there is an amazing book called: The Art and science of digital compositing by Ron Brinkmann, where you will find out exactly what the title reads. You should also check out a book from Steve Wright: Digital Compositing for Film and Video: Production Workflows and Techniques 4th Edition.
Is it really just A over B?
Let’s get back to the heart of compositing. The most common technique, so-called A over B is simply inserting image A into image B using a variety of tools and processes. Can’t be that hard right? Well, imagine repeating this process 10s even 100s times, using huge multilayered files called EXR containing tones of data that you have to go through pixel by pixel, just to create a few seconds of video to entertain the audience. When I was younger and had no idea about the world of VFX, this simple technique of overlaying images to create something new, impossible or unusual always inspired me and made me think about a way to accomplish it. Lucky for you, now I am going to share all my knowledge I have learnt so far, so you can step out with the right foot and be compozitive. If there is nothing you knew about the world of VFX, you definitely must have seen the use of a green screen to create a mask around a foreground actor or and object. Masking objects is the most important skill to know for every compositor.
By creating a mask you basically create a hole in the digital plate to insert another layer. Remember? A over B. There are multiple methods to accomplish this. Creating roto shapes to manually cut out an object or Rotoscoping is the easiest method to mask out an object. The second method that many compositors use is putting an already mentioned green screen or blue screen behind a character. By doing this you can later key out (cut out) the subject and create a very effective mask. The keying method is not easy and requires a lot practice to pull a good key with enough details without dark or light edges. 2D elements such as smoke, dust particles, blood, flares, mist, muzzle flashes and so on are usually recorded against a black backdrop due to the high contrast between those elements and can be keyed out with different keying method as well.
3rd method, that I am going to mention is using 3D generated ID masks (last few years also Crypto mattes). These ID masks are easily generated during the rendering process and can be used to mask out basically anything. From reflections and lights shining on the model to different parts of the 3D model. Communication between the 3D and compositing department is very crucial because they increase the render time on one side but help the compositors tweak little bits and pieces much faster. ID masks are only a fraction of the layers produced by 3D department. EXR files that store the render information usually contain different AOV layers that can be composited together to create to the final image, also called beauty pass. Constructing and deconstructing layers of a 3D render to get as much control over the final image as possible is another one of the key skills for a compositor.
Compositing for me is the art of manipulating the picture in motion. One of the key skills for every compositor is to correctly integrate elements in the scene and match their motion. To achieve this we use either 2D or 3D tracking or sometimes planar tracking. All of the compositing software has an in-built program to do this job or you can reach out for 3rd party software such as PFtrack or Mocha. By 2D tracking an image you match the exact position of the chosen feature across the timeline. Using the 3D or camera track instead recreates the exact movement of the camera used on the set, which can be later used to composite CG (Computer Generated) objects inside the scene. Another important part of compositing is removing unwanted objects from the scene such as wires, cables, camera rigs or even blemishes and logos. You can basically paint out everything and it is very important to know how to do well. These subtle visual effects are never seen by the audience, but require a lot of time and effort to be executed. Imagine spending a month working on an effect, which cannot be even seen. Ridiculous? Maybe, but a very important step in the pipeline to create the final image which everyone is so eager to see. Compositing is a very challenging job, which requires a lot of attention to detail and precise execution of the methods learned, but this is what makes the work we do so delicate and rewarding. One of the detailed parts we compositors do is colour matching. Different assets produced across the VFX departments are given to a compositor and are not always matching at first glance.
To make you understand, imagine a scene where you have to put together a 3D animated model of a car with a camera recorded scene of the road it is driving on. My job then would consist of matching the darkest point of the car to the darkest point of the recorded plate (match the blacks) and then look at the overall colour cast of the car and tweak the colour values to match the recorded plate. For example, Chappie, produced by Image Engine, came out so perfectly after the render that all the compositors needed to do is match the blacks and the image was finalized. A perfect example of A over B compositing.
Chappie – VFX breakdown
Over the years of development, compositing progressed as well and there are some advanced techniques that have been relatively recently developed. The need to be more versatile made at first strictly 2D compositing department become more 3D oriented. Particles are handy to create multiple easily renderable effects. You can orientate them in the 3D world, make them interact with geometry and by using real-world physics parameters to achieve the look desired. Deep compositing is another term you may have heard. Being firstly used by PIXAR under the name deep shadow maps, deep compositing has become a very flexible way of merging layers in space. You can hide deep data information to every pixel and tell the program how far away from the camera it lays. This offers us a very indulging way to composite assets over each other, because you have the capability of working with the 2D scene while thinking in 3D.
As I have already previously mentioned, compositing is based purely on mathematical expressions and to master it you will definitely take a closer look at the math behind the tools you use. Expressions or programmatic commands let you modify the tools you are using to Nuke customize parameters inside the software. From the most basic to the most advanced techniques, compositing is definitely a long journey, but short enough to master it if you spend your time learning these arts. Hope this short blog was a helpful start to the world of VFX and a motivation to learn more.