Three-tier masking problem

[OlgaI]

I rigged a character, it's a bone layer with a group layer inside (eyes mask). The bone layer also contains two masks, they mask arms and legs. The whole lot works beautifully but at some point I need to mask the whole character making it invisible behind a part of the background. But when I create a new group layer with Reveal all masking option and put my rig into it, along with the mask layer, it just doesn't work. Nothing happens except that the eyes mask shows incorrectly in the workplace. I tried to do the same tith a bone layer instead of group layer, nothing happens either.
Is there a limit of layers "tiers" one can mask?
I use v.10.

[synthsin75]

You need to render complex masking to see what the final result will actually be. These do not show correctly in the workspace.

[dueyftw]

Their is a trick you could use. I call it a two pass render. Render your character with out the background into a QuickTime with an alpha or a sequence image. And then re-import and add your mask.

Remember you can cut and render just parts. IE:

Character goes behind tree from frame 525 to 531. So render 1 to 524 normal then use a sequence image for 525 to 531 with the mask and then go back to normal setup for 532. This could be setup in one file or render in three and composite in a video editor.

Dale

[OlgaI]

Thank you Dale! For now I have used a roundabout to solve the problem but next time I will do as you advised.
Hope masking will be enhanced with future upgrades, though.

[synthsin75]

I'm sorry, but multiple render passes are completely unnecessary for this task. Render to see actual masking result. If you have an example file, I can show you how it needs to work very easily, but render passes, and especially image sequences, are not in the least bit needed.

[dueyftw]

I have run into the same problem with multiple nested masks to generate alpha images. After three nests they stop working and give strange results. I no longer need to mask that deep for alpha's so I have no recent files to show you.

Dale

[synthsin75]

I have never had any problems, no matter how many or how complex the masking.

[slowtiger]

I had a problem years ago with nested masking combined with transfer modes, at some level the images turned black and white. But that was the only incident.

[OlgaI]

I'm sorry, but multiple render passes are completely unnecessary for this task. Render to see actual masking result. If you have an example file, I can show you how it needs to work very easily, but render passes, and especially image sequences, are not in the least bit needed.

As I said I'd fixed it by other means. When I was trying to do that complex masking it didn't show correctly in frame previews, I take it doesn't differ from actual render?

[dueyftw]

I had complex masking that was giving me such a hard time that I ended up using that trick. It seems I'm the only one that can't get a mask to work after nesting a few times.

AS dose not show all features or elements until you render. Masking is one of those things where the working view will be different from the render.

Dale

[heyvern]

How complex is the masking? Is it the grouping depth or is it just a lot of clear and add etc, type things?

Basically, if I have masking that goes below two groups or is nested in a way that I know won't display in preivew I just ignore the preview and rely on what I know "works". I have run into trouble when I forget that the masking works when rendered but is not accurate in preview. I only trust simple top level masking in preview. Any time I have multiple layered complex masking I only rely on renders or my own knowledge of how masking works.

Another trick is to "test" subsets of complex masking groups at a "higher level" so I know it works before moving it to a lower grouping nested level and render tests. For example I will sometimes create group layers that have their own masking to create a final "result" used to mask other layers. Because of the nested grouping the preview won't show it properly. I will test the sub groups first at higher levels so I know the results are correct before moving them to where they should be.

Also with some really complex masking on a complex character, I will create a "test" file that is very very very basic shapes. I know the masking I want to end up with and use this test file to test out the masking I want to end up with. This is much easier than fiddling around with a complex character file. Once the test works out the way I want I can apply the same set up to my final character or project.

I use a lot of complex masking and haven't had too much trouble. The trick is knowing how it works and relying on that knowledge (and of course rendering).

I do have situations where masking makes the preview impossible to see anything below it. Often the masking completely blocks the view. I will then simply set layers to not show in preview and only render. This "hides" the masking in the preview so I can see layers under the mask that normally would hide everything. I once had a really complex mask that involved multiple layer blending modes that didn't display in preview. These mask layers covered the entire character so I set those to not display in preview. Sometimes I will even turn off masking completely in the display quality settings and hide all the masking in preview.

The key is you only need to "see" what needs to be animated for dramatic motion and action. You don't really need to see all the minute fine details of all the effects to animate.

EDIT:

The is similar to 3D animation. Most people don't have super fantastic high end expensive GPU cards that display high quality final render previews in real time when working in 3D. You must rely on a low resolution preview while animating in 3D.