Using Masks to Create Transparency

When you create a mask to define transparency within a layer, you’re drawing a shape to use as that layer’s alpha channel. Because a layer’s alpha channel is a grayscale channel, masks are grayscale objects, where white defines solid areas, progressively darker levels of gray define decreasingly opaque areas, and black defines areas of complete transparency. If you choose View > Channels > Alpha to set the Canvas to display the alpha channel only, you can see the grayscale alpha channel directly.

Figure. Canvas window showing a masked object and the corresponding alpha channel.

Masks can be used with layers that already have an alpha channel, and can add to, subtract from, intersect, or replace the layer’s original alpha channel.

Masking a Layer or Group

Use the mask tools to create different shapes with which to mask a layer or group. There are five mask tools available that are analogous to the five shape drawing tools covered earlier in this chapter:

  • Rectangle Mask

  • Circle Mask

  • Freehand Mask (used like the Paint Stroke tool)

  • Bezier Mask

  • B-Spline Mask

Note: You can also use an Image Mask to mask a layer. For more information, see Applying Image Masks to a Layer.

Important: A mask cannot be applied to a nonflattened 3D group. (To flatten a 3D group, select the Flatten checkbox in the Group tab of the Inspector.) A mask can be applied to a layer within a 3D group. To add a mask to a layer that has been manipulated in 3D space, use the Isolate button. Masks cannot be manipulated in 3D space. For example, the Position parameter for a mask only includes the X and Y position parameters, but not the Z position parameter; the Rotation parameter only includes Z, but not X or Y. For more information, see Masking Objects in a 3D Project.

Drawing and editing masks using these tools works identically to drawing and editing shapes. The main difference is that while drawing a shape creates a new layer in your project, masks must be drawn on top of another layer in your project. Once drawn, masks are nested within the specific layers or groups for which they were created, in the Layers tab and Timeline. Once a mask is attached to a layer, a Masks icon appears to the right of the name of the layer. Turn the mask display on or off by clicking this icon.

Figure. Layers tab showing Masks icon.

Note: Masks created for one layer can be copied or moved to other layers.

Important: When a mask is applied to a 2D or flattened 3D group, the group is rasterized. For more information, see Groups and Rasterization.

To mask a layer
  1. Select a layer in the Canvas or Layers tab.

    Figure. Canvas window showing lion image.
  2. Choose a mask drawing tool (the Rectangle Mask tool, the Circle Mask tool, the Freehand Mask tool, the Bezier Mask tool, or the B-Spline Mask tool).

    Figure. Mask tool set in the Toolbar.

    Note: If you haven’t selected a layer, the mask tools are not available.

  3. Begin drawing the shape you need in order to mask the layer.

    For more information about how to draw and edit shapes, see Shape and Mask Drawing Tools.

    Figure. Canvas window showing mask being drawn around lion's head.
  4. When you’re done, you can finish the mask by doing one of the following:

    • Click the first point to close the mask.

    • Press C to close the mask, joining the first point you created to the last.

    • Double-click to close the mask, joining the first point you created to the last.

    • Press Return to close the mask, joining the first point you created to the last.

    • When using the Freehand Mask tool, join the end of the mask to the target circle at the beginning of the mask. If you don’t reach the beginning of the mask, the mask shape is automatically closed when you lift the stylus or release the mouse button.

      Note: Unlike standard shapes, masks are always closed.

After you’ve finished drawing the mask, its effect is immediate, and the mask you created appears nested underneath the layer to which it’s applied in the Layers tab and Timeline.

Figure. Canvas window showing background masked, leaving just the lion's head.

You can create multiple masks for a single layer. To add additional masks, select the object again, then follow the above procedure. By default, all new masks are set to the Add blend mode. You can change the Mask Blend Mode parameter in the Inspector to combine masks for different effects. For more information about using multiple masks together, see Combining Multiple Masks.

Masking Objects in a 3D Project

When working in a 3D project, you may want to mask a layer once it has been transformed or the camera has been rotated. You can immediately switch a transformed object into a “work view” using the Isolate button in the Layers tab or Timeline layers list, or the Isolate command in the Object menu. Once a camera is added to a project, the Isolate option is available for any layer, group, or camera in the project.

Note: Masks can be applied to the layers within a 3D group, but not the 3D group itself (unless you’ve activated the Flatten checkbox in the Group tab of the Inspector). For more information about the Flatten checkbox, see Mask Parameters.

To isolate a layer
  1. In the 3D project, select the layer that you want to mask. In this example, the center leopard image is the layer to be masked.

    Figure. Canvas window showing a group of objects.

    The Isolate button appears to the right of the layer name.

    Note: A project must contain a camera in order to access the Isolate option.

  2. Click the Isolate button or choose Object > Isolate.

    Figure. Layers tab showing Isolate button (inactive).

    Note: You can also Control-click the layer or group and choose Isolate from the shortcut menu.

    The layer is switched into “work view,” allowing you to mask the layer at its original orientation.

    Figure. Canvas window showing isloated object.
  3. To return to your previous view (the view before you isolated the layer), click the Isolate button again.

    Figure. Layers tab showing Isolate button (active).

    Note: Clicking a camera’s Isolate button activates that camera’s view.

Combining Multiple Masks

Often, you may find that it is impossible to create the mask shape you need using only a single mask. For example, when masking something that has a hole in it, you need to use at least two masks—one set to Add to mask the overall object, and a second one set to Subtract to cut the necessary hole in the middle of it. In the following example, the only way to mask the space between the diver’s arms and legs is to create three overlapping masks set to Subtract.

Figure. Canvas window showing three overlapping masks.

Other times, you may find that it’s simply more convenient to mask a complex object using multiple masks. This is frequently the case when rotoscoping an object and animating the masks. In this example, the dog’s head is masked with four different masks set to Add—one for the head, one for each ear, and a separate mask for the mouth. This makes it easier to make adjustments later on, especially when animating a mask to do rotoscoping.

Figure. Canvas window showing four overlapping masks.

Whatever the reason, you can easily combine multiple masks for any object by adjusting each mask’s Mask Blend Mode parameter in the Inspector to create any combination.

Figure. Mask tab showing Mask Blend Mode pop-up menu.

Each mask’s blend mode determines whether the mask adds to, subtracts from, or replaces a layer’s previously existing alpha channel. Additionally, blend modes affect how masks interact with one another. Ultimately, every mask applied to a layer combines according to the specified blend modes to create that layer’s final alpha channel. Mask Blend Modes can be changed at any time.

Blend mode operations are also determined by the order in which each mask appears in the Layers tab, particularly with the Replace and Intersect modes. For example, a mask set to the Intersect mode masks out all other regions of a layer that do not overlap it. A mask set to Intersect affects all masks nested underneath it. Masks added above it are unaffected by this behavior.

Masks That Add and Subtract

New masks always default to the Add blend mode, allowing you to add to an existing alpha channel without cropping what’s already there. Once you’ve drawn the first mask, you may need to use additional masks set to Add or Subtract to refine it.

Masks set to Add create white areas in the alpha channel, making those parts of the object opaque. This can be useful when masking several unconnected subjects in a single image.

Figure. Canvas window showing two masks added together.

Note: To display the rest of a masked layer so you can trace it to create a second mask, turn off the first mask in either the Layers tab or Timeline. The outline of the first mask remains visible, even though it’s no longer affecting the layer’s transparency.

Predictably, masks set to Subtract have the opposite effect, creating black areas in the layer’s alpha channel and making those parts of the object transparent. This is particularly useful for cutting holes in an object.

Figure. Canvas window showing one mask subtracted from another.

Intersecting Masks

In many instances, you may find that you want to cut out parts of an object, while leaving the rest of the object alone. Using a mask set to Intersect is a good way to do this.

For example, if you import a TIFF file with an alpha channel and you want to cut out part of it without losing the alpha information that’s already there, the Intersect blend mode is the only way to do this. This is also true for layers with keying filters applied.

Figure. Canvas window showing two intersecting masks.

Replace Masks

A mask set to Replace completely replaces the previously existing alpha channel. It also completely replaces all other masks appearing underneath it in the Layers tab. For this reason, it is unlikely that any but the bottommost mask would be set to this blend mode.

Feathering Masks

One of the easiest ways to adjust a mask is to feather its edges. Feathering a mask by a few pixels can make the masked object blend more seamlessly with a background. Feathering a mask by larger amounts allows you to create progressively softer edges for a variety of design purposes.

You can feather mask edges either outward or inward. The direction in which you feather a mask is dictated by the subject you’re masking. When rotoscoping a subject, a combination of masks using both inward and outward feathered masks may be necessary for the best results.

Figure. Canvas window showing a mask feathered in both color and alpha channel views.
Figure. Canvas window showing a mask feathered out in both color and alpha channel views.
To feather the edge of a mask
  1. Select a mask.

  2. Open the Mask tab in the Inspector.

  3. Do one of the following:

    • Drag the Feather value slider to the right for a positive value to feather the mask outward.

    • Drag the Feather value slider to the left for a negative value to feather the mask inward.

Applying Filters to Masks

As with shapes, you can apply filters to masks, which remain editable even after the application. Although uses for this technique might not be immediately obvious, it’s actually an extremely powerful way to further manipulate the alpha channel created by one or more masks.

For example, you could apply the Crystallize filter to a mask to stylize its edges. This results in jagged edges that resemble facets of crystal around the edges.

Figure. Canvas window showing a mask with no filters applied.
Figure. Canvas window showing a mask with a filter applied.

When applying filters to masks, it’s useful to bear in mind that masks are just grayscale images. In the example above, the Crystallize filter creates patterns in the grayscale mask image, which results in a rippled translucent pattern.

To apply a filter to a mask
Do one of the following:
  • Drag a filter from the Library directly onto a mask in the Layers tab or Timeline.

  • Select a mask in the Layers tab or Timeline layers list, then choose a filter from the Add Filter pop-up menu in the Toolbar.

    The filter appears nested underneath the mask layer to which it’s applied.

Applying Behaviors to Masks

You can apply behaviors directly to masks. When you apply a behavior to a mask, the mask is animated like any other layer. However, since the mask only affects the transparency within a layer, and not the position, rotation, or scale of the layer itself, the result is similar to that of a “traveling matte,” where the mask moves within the layer to which it’s applied, hiding and showing different areas of the image as it moves.

To apply a behavior to a mask
Do one of the following:
  • Drag a behavior from the Library directly onto a mask in the Layers tab or Timeline.

  • Select a mask in the Layers tab or Timeline, then choose a behavior from the Add Behavior pop-up menu in the Toolbar.

    The behavior appears nested underneath the mask to which it’s applied.

Keyframing a Mask’s Shape for Animation and Rotoscoping

You can keyframe a mask’s animation parameter to change its shape over time. This can serve many purposes. For example, you can animate a subtractive mask to change the shape of a hole in a foreground object, allowing other layers in the background to show through.

A more conventional use of animated masks is to rotoscope a foreground subject. Rotoscoping is the process of manually tracing a foreground subject in order to isolate it from the background. The end result is similar to a blue screen or green screen effect. Why would you bother? In a wide variety of situations, keying is either impractical or impossible if the shot wasn’t well planned. Even for shots where keying is possible, manual rotoscoping is often necessary to create garbage or holdout mattes to improve the effect. (For more information on creating garbage and holdout mattes or masks using masks, see Using Masks to Aid Keying Effects.)

All mask animation is stored in the Shape Animation parameter. It’s worth noting that animated masks trigger the same motion blur as any other keyframed parameter in Motion. For example, if you animate a layer’s position so that it moves really fast, the layer is blurred when you enable motion blur. If you animate a layer’s mask so that it also moves fast, the edges of the mask are blurred as well. This is important as an animated mask’s blur should ideally match any blur that’s present in the foreground subject itself.

To rotoscope a subject by animating a mask
  1. Choose View > Resolution > Full to ensure that you are viewing the Canvas at full resolution.

    Important: If the Canvas resolution is not set to full, the outlines of objects and images may shift slightly. As a result, masks created to trace a subject at less than full resolution may not be accurate.

  2. Move to the first frame where the mask animation should begin, then draw a mask that accurately isolates the subject.

  3. When you’re finished drawing the first mask, enable Record (press A).

    Figure. Record button in the Canvas window transport controls.
  4. Move to the first frame of the Timeline where you want to change the shape of the mask, then make any necessary changes to its control points.

    In this example, the best place to start when rotoscoping the dolphin is the frame in which all the fins are showing.

    Note: For some tips on rotoscoping, see Rotoscoping Tips.

    Figure. Canvas window showing a masked object ready to be rotoscoped.

    When rotoscoping a moving subject, it’s a good idea to play through the clip and examine its direction and speed to get a sense of how the mask needs to be animated. In particular, you may find it helpful to place markers indicating frames where the subject or camera stops, changes direction, or changes speed, since these are likely candidates for your first pass of keyframed mask changes.

    By noting these changes, you can attempt to reduce the amount of keyframing by making your first keyframed mask shape adjustments at these major changes in speed and direction. Because animated masks are automatically interpolated to change from one keyframed shape to another, you can let Motion do some of your work for you.

    Figure. Canvas window showing a mask being changed over time.

    Note: If at any time the mask is deselected, you need to select the mask you’re working on in the Layers tab so that its control points become visible in the Canvas. Make sure that the Show Masks button is selected in the Layers tab so that masks are visible.

  5. Continue moving the playhead and making changes as needed, since the interpolated shape of the mask isn’t going to be perfect.

    One imprecise rule of thumb is to move to the frame that’s halfway between any two keyframed mask shapes and make new adjustments. Continue keyframing shape changes at the halfway point between every two keyframes until the mask accurately follows the motion of the subject. For irregularly shaped objects or objects with complex motion, don’t be surprised if you need to add a large number of keyframes. Nobody ever said that rotoscoping was fast!

    Every time you move the playhead to a new frame and make a change to the mask’s shape, a keyframe is automatically created in that shape’s Shape Animation parameter. If you move the playhead directly on top of a previously existing keyframe, you change the mask’s shape at that point without creating a new keyframe.

    Figure. Keyframe Editor showing Shape Animation parameter with several keyframes.

    Note: You can also transform and shear masks as you would any other layer. If you move, rotate, scale, shear, or change the anchor point for a mask while animation recording is enabled, you add keyframes to additional channels in the Keyframe Editor.

  6. When you’ve finished animating the mask, disable Record (press A).

    You can also create, delete, and edit the timing of a mask’s Shape Animation keyframes directly in the Keyframe Editor. For simplicity, each change you make to a mask is recorded as a single keyframe, no matter how many control points you edited. The Shape Animation parameter only allows keyframes set to Constant—you cannot use any other form of keyframe interpolation.

    The process used to animate both masks and shapes is identical. To see an example of shape animation, see Animating Shapes. For more information on keyframing in general, see Keyframes and Curves.

Rotoscoping Tips

A good way to start rotoscoping an image is to find the frame with the most detail showing in the subject you’re masking, and use that as your starting frame. For example, if you’re masking someone walking, choose a frame where both that person’s arms and legs are out in mid-stride. This allows you to decide how many control points you need to start with. Using the minimum number of control points you can to achieve the necessary level of detail in your mask makes it much easier to animate.

You should also bear in mind that it’s not always necessary to rotoscope an entire subject with a single mask. Motion allows you to apply multiple masks to a single object, so you can rotoscope different parts of a subject with separate masks. This can make your job much easier. For example, if you’re rotoscoping someone carrying something while walking across a room, you could use one mask for the upper part of the person’s body which doesn’t change shape very much, and another two masks for the legs, which change shape considerably as the person walks.

You can also use the Track Points behavior to apply tracking data to a shape or mask. For more information on the Track Points behavior, see Shape Track Points Behavior.