The cleanest roto in the world will look wrong if the edges are handled the wrong way. A hard-cut matte that should have been feathered reads as a paper cutout pasted onto the background. A soft-feathered matte that should have been hard-cut produces ghostly edges that betray the comp. The line between these two failures is narrow, and the right edge treatment depends on the subject, the motion, and the destination composite.
This post is the working roto artist’s guide to deciding on edges. The principles are simple to state and harder to apply consistently across hundreds of frames. The artists who consistently produce roto that compositors don’t have to fight are the ones who’ve internalized the decision process to the point where it’s automatic.
What’s Actually Happening at an Edge
Every edge in a real photographed image is some combination of three things:
1. The actual silhouette of the subject. Where the subject’s surface ends and the background begins. This is the geometry being recovered.
2. Motion blur. When the subject (or the camera) is moving fast enough relative to the shutter speed, the edge becomes a smear. The silhouette isn’t a hard line; it’s a gradient where partially exposed pixels show some of the subject and some of the background.
3. Optical effects. Defocus, atmospheric haze, lens diffraction, chromatic aberration. These produce edges that aren’t crisp even when the subject is stationary and well-defined.
Real edges in real footage are some combination of all three. A pure silhouette edge (sharp subject, no motion, no optical effects) is rare. Most edges have at least some softness from one of the other two factors.
What this means for roto: the matte has to encode not just the silhouette, but also the partial-transparency information that comes from motion blur and optical effects. A binary matte (1 or 0 per pixel) loses that information. A continuous matte (values between 0 and 1) preserves it.
The Hard-Cut Edge
A hard-cut edge is a roto matte where the transition from “subject” to “background” happens within one or two pixels. The matte values go from near 1 to near 0 abruptly. There’s no gradient.
When this is correct: the underlying edge in the plate is genuinely hard. The subject is in sharp focus, isn’t moving fast, and isn’t subject to significant optical effects. Crisp architectural edges, props in good light, talent in static portraits — all of these can have hard edges in the plate, and a hard-cut matte preserves them faithfully.
When this is wrong: the underlying edge has motion blur or optical softness. A hard-cut matte through a motion-blurred edge produces a ghost — the original blurred pixels were partially transparent, and the matte treats them as fully opaque or fully transparent. The shot looks correct only in still frames; in motion, the edges read as fake.
The signal that a hard-cut is wrong is usually visible in playback. Frames that look fine on their own come together into a sequence where the subject’s edges crawl, jitter, or pop against the background. The fix is feather.
The Feathered Edge
A feathered edge is a roto matte with a gradient at the boundary — values transitioning smoothly from 1 to 0 over several pixels. The width of the feather depends on the underlying edge.
When this is correct: any edge with motion blur, defocus, or optical softness. A subject moving across the frame at speed has motion-blurred edges; the matte should have feathered edges of matching width. A subject in shallow depth-of-field has defocused edges; the matte should follow.
When this is wrong: the underlying edge is genuinely sharp. A feathered matte through a sharp edge produces visible halos — pixels that should have been entirely the subject are now partially transparent, showing a hint of what was behind them. In a clean composite onto a different background, the halo reveals the edge of the original plate’s content.
The signal that a feather is wrong is usually visible at the edge color. The matte should pick up only the subject’s pixels, not the background’s. If the feathered region is showing color from what was behind the subject in the plate, the feather is too soft for the underlying edge.
The Skill of Matching Feather to Edge
Real shots have edges with varying characteristics across the same matte. The subject’s hand, in sharp focus and barely moving, has hard edges. The subject’s hair, blowing in a slight breeze, has soft edges. The subject’s silhouette around the body has mid-soft edges from motion blur. A single matte has to handle all three correctly.
This is where the craft lives. A sloppy matte applies the same edge treatment everywhere — uniform hard, or uniform feathered. A careful matte varies the treatment locally, matching feather width to the underlying edge characteristics.
Tools help, but only up to a point. Modern roto software can analyze edges and apply feathering automatically. The automatic feather is usually approximately right, but it misses the specific cases — a feathered edge where the feather is locally too narrow because the software didn’t see the motion blur, a hard edge where it added unnecessary softness. Manual refinement of the auto-feather is what produces clean results.
A working principle: after the automatic pass, scrub through the playback at full speed. The frames where the edge looks wrong tell you where to manually adjust. Wrong-feeling edges almost always pop in motion even when they look fine in stills.
Spill Suppression as Edge Treatment
For roto on greenscreen plates (or any plate with strong color contamination at the edges), spill suppression is part of edge handling. Color from the green or blue background bleeds onto the subject’s edges through the optics of the lens; the matte alone doesn’t remove this color.
The pattern that works: pull the matte first (extract the silhouette), then spill suppress the edges (clean up the residual color), then composite. Trying to do both in a single operation usually produces edges that are either over-suppressed (the subject’s actual color is being removed) or under-suppressed (residual green visible at the edges of the comp).
For roto on real backgrounds, spill suppression is rarely needed at the edges, but ambient color reflection (warm light bouncing off a red wall onto the subject’s skin, for example) sometimes produces a similar effect. The fix is the same: clean up the edge color separately from the matte itself.
When the Plate Doesn’t Cooperate
Sometimes the underlying edge in the plate is just bad. Heavy compression artifacts, uneven motion blur from a rolling shutter, severe chromatic aberration. The matte can be perfect and the comp will still look wrong, because the source pixels at the edge contain garbage.
In these cases, the roto artist has options:
Plate cleanup before roto. Some artifacts can be cleaned up at the plate level — chromatic aberration removed, compression artifacts smoothed, rolling shutter corrected. The cleanup is itself a comp operation, and it adds time, but the resulting matte is much cleaner.
Edge replacement in the comp. Instead of preserving the plate’s edge through to the final composite, the comp can synthesize a new edge that matches the destination scene. The plate’s edge is masked out entirely; new pixels are generated based on the destination’s lighting and the subject’s silhouette. This is more work but produces cleaner results when the plate’s edges are unrecoverable.
Plate replacement. In severe cases, the original plate is too damaged to use, and the only fix is to reshoot or substitute. This is rare on professional productions but happens occasionally on ones with compromised capture.
The Compositor’s Perspective
The compositor downstream of the roto artist has a specific test: can they integrate the matte into the destination scene without fighting it? A good matte is one the compositor doesn’t think about. They drop it into the comp, and it just works.
A bad matte is one the compositor has to fix before they can do their actual job. Edges that need re-feathering. Spill that wasn’t suppressed. Holes in the matte that have to be patched. Each of these is roto work that’s happening in the comp instead of in roto, which means it’s slower and more expensive than it should be.
Roto artists who care about the relationship with downstream compositors deliver mattes that are comp-ready: edges treated correctly per the underlying plate, spill suppressed where needed, no holes, no fluttering, consistent across frames. The compers don’t notice them, which is the highest compliment.
How FXiation Digitals Handles Edge Decisions
Our roto team trains on edge handling specifically because it’s where most matte failures show up. The default workflow includes a frame-by-frame review of edge treatment, with manual adjustment where the automatic pass missed. Spill suppression is a standard pass on every greenscreen-derived matte, separate from the matte pull itself.
We deliver mattes built so the compositor doesn’t fight them. If we send a matte that needs adjustment in the comp, we’ll flag it explicitly so the comp team knows what they’re walking into. Surprises in the matte are surprises that propagate through the rest of the work, and we’d rather catch them in roto than discover them in comp.
If you’re scoping a project with significant roto requirements, reach out and we can talk through edge expectations before any work starts. Some productions need every edge feathered to the maximum to feel cinematic; some need hard cuts to maintain a graphic style; most need a mix calibrated per shot. Defining the edge expectations upfront saves cycles in review and gets the look right faster.
Common Questions