Layer shading without strokes

[onurdemirsoy]

Hello, I’d like to use layer shading in Moho, but I don’t want it to affect the strokes. Is there a way to do that?

[Greenlaw]

Hi, onurdemirsoy!

Feeling extra verbose today, so get ready for a long one.

...I’d like to use layer shading in Moho, but I don’t want it to affect the strokes. Is there a way to do that?

Technically, no. Strokes are part of the layer, so they have to be included in the Layer Shading. But there are a few workarounds you can try.

#1 - I don't recommend the following, but it's one method I've used in the past: Create a Reference of the original group, and knock out the fills in the Reference’s Styles. This approach can get complicated because it involves recoloring the fills and using a mask and/or blend mode to isolate the strokes. You could try Moho's Color key, but that's getting messy. (Actually, I think Moho 14/13.5 doesn’t have a color key tool, so you might have to do that part in another program...I have to check this.)

I wanted to mention this method in case you were considering it. The following suggestions are better....

#2 - In very old versions of Moho (ASP 10 or maybe 11?), what worked for me was to render with no anti-aliasing, so I could easily key out the strokes in a compositing program. This allowed me to stamp an unshaded version of the strokes over the shaded render, and then add AA to everything as a post-process effect in the compositing program. In After Effects, for example, this is done using the free OLM Color Keep and OLM Smoother tools from OLM Digital.

Unfortunately, disabling AA in the current Moho 14 (and I believe Moho 12.5 as well) causes strokes narrower than 2 points to render as broken lines, and the result is unusable for compositing.

So...not very useful nowadays, at least until the AA problem is fixed.

Well, okay, this method can be used with anti-aliased renders from Moho if the strokes are rendered in a keyable color. After keying the strokes in a duplicate layer, the Strokes can be comped on top of the version with the shading, just like the non-anti-aliased version. Then I can roll the Stroke Color back to what it's supposed to be.

I don’t like doing this because the anti-aliased edges don't key as cleanly, and it takes more effort to get a clean edge. And even with a good key, the quality can be slightly degraded compared to keying with a hard clean edge.

That said, this method is still easier than isolating strokes as described in method #1!

Despite the issues, this has been my preferred method for several years.

However…

#3 - Here's something I've been thinking about but haven't actually tried yet, and it's possibly the easiest method with the current version of Moho:

1. Save a copy of the project, and in the copy...
2. Open Project Settings and...
3. Set the Render Style > Fill Style to Background.

This fills all the shapes that have a Fill with the solid Background color. I would set the background color to pure white, which would allow me to use a luminance key to knock out the filled areas, and then I would comp this visible 'Strokes-Only' layer over the shaded version of the render.

Here's an example…



On the left is the original render, in the middle is the render from the Strokes-Only project, and on the right is the luma-keyed version of the Strokes-Only render. If this were for a real production, I'd slap the keyed Fill With Background Color rendering over the shaded version.

Alternatively, I might set the background color to full-green, -blue, or -red (255, 255, 255) and pull a color key. It really depends on the stroke colors I wish to keep. (Full-green would be a poor choice for Mame-chan.)

I’m not sure how clean the keyed edges will be with this version because it still has anti-aliasing pre-applied, but the workflow should be easier than method #2.

Interesting note: If you check the other Fill options, you'll find None and Back Transparent. Neither of these options will do what you might think. None simply means don't render with fills, and this will reveal strokes that should be hidden. Back Transparent sounds like it should replace the background with transparency, but instead, I get a weird semi-transparent background that partially reveals strokes (maybe a bug?)

I have a project coming up where I can try this method, and I’ll share the results when ready.

In a future version of Moho, I really hope we get a proper way to break out a Strokes-Only pass. The above hacks work okay for me in a pinch, but for serious production work, we need a more efficient way to do this.

Anyway, these are my options for isolating Strokes. If I had to do this right now, I’d probably try method #3 first, and fall back on #2 if necessary.

If I were feeling really lazy, it would be tempting to simply allow the Layer Shading to affect the Strokes. But it looks pretty bad, and I'm never THAT lazy.

(If anyone here has a better method, please share. Thanks!)

I also hope the AA problem gets fixed in a future version of Moho. Rendering without AA from any animation package, and then applying AA in compositing after applying effects, is a fairly common practice in the industry.

Or even better, I wish Moho supported the EXR file format and offered the Cryptomatte option. This is a system that lets you pre-assign IDs to objects and shaders so they can be easily selected in compositing. Additionally, the AA data can be stored in a channel and applied to the selection for cleaner edges. This is the method I use when working in 3D animation, and it really is the easiest and cleanest method. In Moho, I’d love to be able to assign Cryptomatte IDs to Styles and other properties and items so I can easily select them in Fusion, Nuke, or After Effects.

Hope this info helps. Good luck!

[onurdemirsoy]

Thank you very much, you’ve offered great solutions. I found a workaround myself. For example, let’s say I want to add side lighting to a character — if I use layer shading, the outline also gets affected. So instead, I render the character as a PNG without layer shading, and render the background separately. Then, in After Effects, I can add light to the character’s layer using Layer Styles and apply it from the inside with blend mode.

I hope this feature comes to Moho as well. I’d really like to use Layer Shading for lighting, but without affecting the outlines. Maybe if the developers hear this, they might add such an option.

[Greenlaw]

Hi onurdemirsoy! Cool! I'm glad you found a solution that worked for you!

...Then, in After Effects, I can add light to the character’s layer using Layer Styles and apply it from the inside with blend mode.

I agree! Lighting Moho animations in After Effects (usually) provides the most flexibility, and it's what I typically do here. The video I made about rendering Layer Comps for Compositing shows a basic example.

BTW, since you're using AE's Layer Styles, you might be interested in this AE lighting preset from Reuben Lara: 2D Rim Light (After Effects preset). Reuben’s script uses AE’s Layer Styles but adds a regular Effects GUI, like you get with Plugins. I find using this preset is easier and more efficient than accessing the Layer Styles controls directly, and it almost always looks great with my Moho renders!

I occasionally run into one issue when applying the effect to large, irregular shapes, where the shading can flicker, but this is actually a limitation of AE’s Layer Styles, not the script. Fortunately, there are many other ways to light Moho animations in AE. I’ll make a video about them later. (Regarding the issue, my 'pony' example from our original tutorial promo video comes to mind. Maybe I can use this as a tutorial example for using the preset.)

I’ll also make videos about applying lighting effects in Moho, as there are many situations where it makes sense to keep it all inside Moho.

[onurdemirsoy]

Thnk you so much. I also use Shadow Studio plugin to get this effect. The video you shared is very useful. Thanks.

[Greenlaw]

Oo! That looks pretty cool. I'll have to keep it in mind for the future. Thanks for the tip!