Amazing technology, bluescreen. Without it, most of the cool visual effects in movies and television just wouldn’t be possible. But as anyone who has done much work with bluescreen techniques can testify, it isn’t always as easy as it looks. Narrative applications that involve large, difficult-to-light screen areas can suddenly create challenges that your NLE’s built-in keyer doesn’t handle well. Bluescreen work is a tightrope walk, a matter of balancing many conflicting factors. It’s rarely perfect. By the way, I’ll use the term “bluescreen” to refer to all color-based compositing, whether the screen color is blue, green, or even red.
The tough problems in keying more commonly originate from haste or carelessness in production — uneven lighting on the background, conflicting costume choices and the basic poorly exposed shot are far more common causes of expletives in post. Once it’s “in the can,” it’s usually extremely difficult or expensive to go back and reshoot. So how do you deal with problem footage?
Professional compositors rarely create a composite shot using a single filter layer. As a base, they will usually use a garbage matte to remove extraneous bluescreen (so the keying process doesn’t have to deal with it), an edge matte to create the cleanest possible edge for the subject, and a core matte to take care of the rest of bluescreen further away from the edges. Usually, the mattes will be “pulled” as discrete grayscale files that can be manipulated separately by other filters, then combined and applied to the foreground footage manually in a compositing program like After Effects, Apple Shake or Autodesk Combustion. It’s not uncommon to have a final matte that is composed of 6 or 8 submattes. While this may seem like a lot of trouble to the average editor, it’s how top quality keyed composites are done. Often the compositor will find that one keyer works best on edges, while another cleans up the rest of the background better. For instance, with an unevenly lit background where the foreground has motion blur, the compositor may use Keylight to create the edge matte with transparent motion blur, but find that the varying tones of the rest of the unevenly lit background can best be keyed using Primatte or a color range keyer.
But you can actually do a “poor man’s” version of this multilayer process in most current NLEs without sinking a fortune into specialized plug-ins and separate compositing programs. The first part of the secret is to use multiple applications of keying plug-ins to render narrow sections of the troublesome background transparent, rather than depending on a single application with very aggressive settings. Color-based keyers are really designed to deal effectively with a very narrow color range; when pushed too far, their performance suffers. The second is to also use your NLE’s Garbage Matte function to get rid of portions of the background that aren’t really part of the action — not just to get rid of extraneous set pieces and off-the-screen areas. In many cases, a bit of garbage matting can get rid of the most troublesome, poorly lit edges of the bluescreen through brute force and substantially reduce the range of color that the keyers must deal with.
Our original image. While the model is properly lit, the greenscreen is not. The combination of darker areas on the screen and a poor wardrobe choice make this shot impossible to key cleanly with a single application of the stock color keyer in your NLE.
Let’s try this with a less-than-perfect foreground plate — though this sample is nowhere near as bad as it gets! The background is unevenly lit, the subject is too close to the screen so there’s a shadow, and she’s wearing a dark green sweater in front of a green background. The combination of factors mean that this shot won’t key properly with the basic keyers built into most NLE’s.
From top: Here we see the garbage matte has cleaned up the left side of the shot and taken away the darkest area of green backing. The first application of the color keyer has been adjusted to key out the fully lit green. In the center, another instance of the keyer is adjusted to take out the model’s shadow. On the bottom, the final composite, with a third instance of the keyer adjusted to remove the darkest green areas..
The first step is to use a garbage matte — necessary anyway here due to the extraneous “offscreen” material. However, we can also use the garbage matte to get rid of the darkest areas of the edge of the screen, as well. Then test the different keyers that are available in your NLE package to see which one performs best on a specific scene; with the garbage matte cleanup, one of them may be able to do the job. If the results are less than optimal, move on to step two.
Step two is to use multiple applications of the filter, each one keying out a narrow band of hue and value. Let’s try the color keyer in Premiere Pro, which is a really basic filter. After applying the filter in Premiere (or Final Cut or Vegas or Avid, the filters aren’t that different), use the eyedropper to select the brightest area of background right next to the foreground subject. Adjust the tolerance until the background of that hue and value around her is transparent; but don’t worry about the shadow or darker areas yet. Adjust the choke (shrink) and blur controls to get a clean edge — generally you want to apply as little choke and blur as possible.
Now, apply a second instance of the chroma keyer and use the eyedropper to select a darker area of green in the shadow. Adjust the tolerance until that area is transparent; again, don’t worry about the darker areas at the edge yet if they are still messy. Adjust the choke (shrink) and blur controls to get a clean edge.
If the darker edges are still opaque, apply a third instance of the keyer, use the eyedropper to select a representative area of darker green, and adjust the tolerance to make it transparent. As long as the darker green is not actually matching the dark green of her sweater, you’ll be able to pull an acceptable composite. Scroll through the entire scene, watching for areas where the background becomes visible or appears as excessive noise; or areas where the model’s sweater turns partially transparent. You may have to adjust tolerance on one of the instances of the filter a bit more to clean these up.
The difference between this approach and simply using a single filter with aggressive settings will not always be visible on a single frame, but will be more noticeable with the subject in motion. That’s where the aggressive settings may create sizzling or odd-looking edges, requiring excessive blur. When each of the filters can deal with a limited range of hue and value, using lower choke/shrink and blur settings, you’ll be able to create a more pleasing and consistent edge to the foreground subject and better control the way the foreground plate blends with the background. The final composite isn’t ideal; for better results you’ll need a higher-end third-party keyer such as Keylight, Ultimatte AdvantEdge, Zmatte or Primatte.
So what happens when it’s worse than this shot, for example when a piece of wardrobe is almost exactly the same color as a dark area of the backing color? That’s when you’ll need to step up to a full compositing program and use the same concepts to create multiple regions of transparency, and combine them with keyframed motion masks to create the final composite. And that’s when you will be able to hang out a shingle as a compositor!