The eyeball is in a group set to "Hide All" masking
The lowest layer in the group is the white of the eye, set to "Add to Mask"
The highest layer is the eyelids, which are only being masked
Everything in between is just there, being masked.
When I try to use the lids to cover everything, the white of the eye is rendered covered properly, except for its AA, which creates a lovely "rim" around the covered white.
How do I get rid of this accursed rim?
In Moho, perfectly aligned edges in separate layers with anti-aliasing enabled will do that. If the mask layers are hidden, disable AA for those layers and that should eliminate the matte line.
If the masks are 'visible' and you need AA enabled on those layers, then you may need to overlap the layers. I do this by adding a point on one edge where I want the overlap to occur and then simply drag that point slightly over the other edge. This is a common trick used in joint areas like elbows and knees.
If you're compositing in After Effects, there may be another option: Render all the layers without AA, and use the free OLM Smoother to apply the AA as a post effect. With AA disabled, this may open other interesting possibilities for effects and selections within Ae. (The OLM tools are available here:
Edit: Oops, I almost forgot this one: in some situations, enabling Gap Fill can fix the problem. Try that first since it's the easiest solution. (I forgot about it because it usually doesn't work the way I need it to.)
Last edited by Greenlaw on Wed Apr 27, 2022 2:41 pm, edited 1 time in total.
As a seasoned graphic designer I fought many a night with trapping (decades ago, nowadays software is smarter). One habit I've learned from that is to never stack identical shapes and hope it has no side effect. Always make the top one a bit larger than the other if you expect it to cover something.
Depending on your style it might be helpful to have just an fat outline, not masked, on top of everything, either in black or in skin colour. This covers a lot of problematic areas in the complex field of eyebball masking.
AS 9.5 MacPro Quadcore 3GHz 16GB OS 10.6.8 Quicktime 7.6.6
AS 11 MacPro 12core 3GHz 32GB OS 10.11 Quicktime 10.7.3
Moho 13.5 iMac Quadcore 2,9GHz 16GB OS 10.15
Moho 14.1 Mac Mini Plus OS 13.5
Don't know if this will affect your problem but checking *Expand mask by a pixel* may help. I don't need it often but have used it for a couple of characters (also never needed it for the eyes)
"Animation is not the art of drawings that move but the art of movements that are drawn."
Norman McLaren
Sorry for this being so late, but after trying it out for a while, I just left it alone till just now.
I tried doing it without AA, and it worked, but now I got to import the whole file into other software, and no one else seems to have that problem, so I didn't want to have to go through all that.
Just recently, though, I tried the "Expand mask by a pixel", and instead of it working, which I felt it should, and was the whole reason I gave this a shot again, it just made a perfect circle, which is the eye, wonky around the edges.
What ended up working is I swapped out how the whole thing was masking. So before, I had it masking as "Hide All" with the white of the eye being added to the mask and the eye lid simply covering it higher in the layer order. That created the rim around the mask. But now I have it set to "Reveal All" with the eye white simply being masked, and the lid, now under the white in the layer order, subtracting from the mask. Now there's no rim. Strange.
I guess, in some way, I can make some loose sense of it, being that adding to a mask is punching a hole into it, and thus creating a residual rim, but since the mask isn't the same color as the surrounding material, it's clearly visible, as opposed to my solution, in which the eyelid is the same color as the surrounding, and thus, rim or no, would show no artifacts upon rendering.
Either way, thank you all for your suggestions; I just realized I never bothered to thank you all for even giving me the time of day: sorry about that.
Are you checking that the aliasing also occurs in the render? There are certain masking configurations that will make the mask edges look aliased in the display but the results will render correctly. In these situations, you can safely ignore what you see on the screen. Sometimes changing a display option (i.e., enable or disable GPU acceleration) can correct this in the display preview but either way, it shouldn't affect the render result.
In general, if something looks odd on screen, you should do a render to check that this isn't just a display artifact.
Greenlaw wrote: ↑Wed Apr 27, 2022 2:46 pm
Are you checking that the aliasing also occurs in the render? There are certain masking configurations that will make the mask edges look aliased in the display but the results will render correctly.
Yes, I did, since I knew that you can't trust the edit window. That's actually how I found the wonkiness of using the "Expand-mask-by-one-pixel" thing. So, yes, I changed the masking, and it rendered perfectly.
