Using Masks to Aid Keying Effects

Keying is rarely a one-step process. Although adding a keying filter is the first step, there is usually additional work that needs to be done to create a good key that retains detail around the edge of your subject. The mask tools and alpha channel filters presented in this section can be used to turn a decent key into a great one. For more information about using Motion’s keying filters see Keying Filters.

Garbage Masks

The second step in many keying shots is to create a garbage mask to crop out unwanted objects in the shot that can’t be keyed, such as the edge of a blue screen stage, lighting rigs, or tape that appears in the background.

Figure. Canvas window showing an unmodified clip.

You can also use a garbage mask to conceal parts of the background that are too difficult to key without the loss of foreground detail.

To create a garbage mask
  • Select a keyed layer and use one of the mask tools to draw a mask around the foreground subject.

    Note: The garbage mask must be animated if the subject is moving.

    By default, the mask is set to Add, and crops out everything outside the mask, while leaving the transparent areas within the mask alone.

    Figure. Canvas window showing a garbage mask applied to a clip.

    For more information about using Motion’s keying filters, see Keying Filters.

Holdout Masks

Sometimes, while pulling a key, you lose part of the image you’re trying to keep. This can happen when the color of the subject’s clothing is too close to the color of the background being keyed out, or when you need to use keying values that are too aggressive. In this example, very small parts of the lioness’s head and face are getting removed with the key, rather than just the blue sky.

The following images show an example of a key that needs to be manually masked. The top image is the original shot to be keyed, and the bottom image is the keyed shot. The lion image key leaves black spots or “dirt” on the key. If left uncorrected, the background image would show through the lion in these areas.

Figure. Canvas window showing an example of a holdout mask.
Figure. Canvas window showing keyed shot.

In these cases, you can duplicate the original layer, mask the part of the subject that’s being incorrectly keyed, and composite it over the keyed version to fill it back in.

Note: The Spill Suppressor filter can modify the color of the foreground subject as well. If you’re using the Spill Suppressor filter on the keyed layer, you may have to apply the same filter to the holdout mask layer to make sure the color matches. For more information on the Spill Suppressor filter, see Keying Filters.

To create a holdout mask
  1. Key and mask the foreground subject.

    For more information about using Motion’s keying filters, see Keying Filters.

  2. Duplicate the keyed layer, then move it so that it appears above the original layer in the Layers tab or Timeline layers list.

  3. Mask the area of the foreground that is incorrectly keyed.

    Make sure that the mask is entirely within the subject being keyed.

    Figure. Layers tab showing a mask applied to a duplicated layer.

    Note: The holdout mask must be animated if the subject is moving.

  4. On the newly duplicated layer, delete the original keying filters since they’re unnecessary.

    If you’ve used a Spill Suppressor filter, don’t delete it, since it’s probably changing the color of the subject.

  5. Feather the edge of the holdout mask you’ve just created, to make sure that it blends in with the object you initially keyed.

    In the following image, the mask on the lioness’s head creates a clean key by removing the “dirt” left by the original key.

    Figure. Canvas window showing holdout mask applied to the object.
  6. As an optional step, you can nest the originally keyed layer along with the holdout matte you just created inside a dedicated group so that you can manipulate the entire subject as a single object.

Performing Multiple Keys on a Single Subject

In some instances, it may be difficult to key an entire subject with a single keying filter while retaining fine details. For example, uneven lighting or background color might mean that the best settings to key a subject’s hair may not work elsewhere.

If you attempt to key the entire subject with a single keying filter, chances are that you need to use such extreme settings that a lot of edge detail is lost. In such cases, masks can be used to isolate different parts of a subject, so that you can apply different keying settings to each area.

To segment a subject into separate keying zones
  1. Create a new group, and place the layer you’re keying within the new group.

  2. Duplicate the layer you’re keying until you have a separate duplicate for each part of the image you want to key separately.

  3. Use one of the mask tools to mask each part of the subject in each of the duplicate layers.

    Make sure that all of the duplicate layers overlap, so that there are no gaps in the foreground.

    Note: You may need to animate the overlapping masks if the foreground subject is moving.

  4. Apply the appropriate keying filter to each of the isolated parts of the subject, and adjust each filter’s settings as necessary.

Eventually, you may find you can achieve an optimal key for each part of the subject. Furthermore, with all duplicates nested within a single group, you can continue to manipulate the subject as a single object.