Which mouth shapes for accurate lip syncing

[Mattyj]

Hi there,

I was wondering what mouth shapes you would recommend to include for accurate lip syncing, formate for naming the layers inside your mouth switch layer, and any other tips people might have.

Thanks heaps

Matt

[synthsin75]

The Preston Blair phonemes, except I make AI, E, and etc the same and O, U, and WQ the same. That way there's only 6 instead of 10, which can make lip sync too busy looking.

[Greenlaw]

Yeah, the Preston Blair set is a classic go to. I use a slightly modified version but the one from his book is a good standard to follow.

Mouth visemes alone may not be enough though, the character should also be able to emote. This means you may need animate the shapes for expressive states: neutral, happy, excitement, etc.

There are different systems you can set up for this. Here are some suggestions...

1. Make sub-switch layers. In the parent Switch, make three Switch layers, i.e., one for neutral shapes, sad and happy. A trick I learned recently from Wes is that if these layers are references, the layers will stay in sync, so in this case, the visemes will stay in sync when you switch between the three child Switch layers.

2. Make one set of mouth shapes but use an SBD to control the emotion of the each mouth. For example, make an SBD called 'emo', and use that to drive a mouth from neutral to happy in one direction, and neutral to sad in the other. ('sad' can usually work for 'anger' too; the eyes can determine which.) This is the method I typically setup for the rigs I create at work. Not all mouth shapes I put in the Switch layer have the 'emo' animations but the main visemes do.

3. Or, you can just use the Magnet tool to make expressive adjustments to your mouth shapes as you need them. This is what I do when I'm using Interpolate Sublayers for animated shape transitions. If you go this route, I recommend reshaping the drawings in a second pass, after the main mouth shapes in the Switch have been keyframed. If you don't want to keep reshaping the mouths, you can use regular Actions to setup presets for this.

IMO, only switching between mouth shape drawing isn't enough. I also like to add secondary transforms on the parent Switch layer. Set the origin in the middle of where the mouth appears in the switch, and use a simple layer transform to add a little squash and stretch. As described above, this should be done in a secondary pass. The result should be a much smoother mouth animation than using the Switch layer alone. (Take a look at my DWA demo reels for examples.)

Depending on how expressive you want your character to be, the head and body can also move to the dialog. This doesn't need to be complicated, it can simply be squash and stretch layer transform added to the Head group and/or stretching added to the the 'chest' bone. Like the mouth Switch transforms, add this animation as a separate pass after the lip sync animation is done.

Put it all together, and it's called acting!

There are other ways to animate mouths and lip sync in Moho (there are actually many ways to do almost anything in Moho,) but these are the three most common methods I use for animating mouth shapes.

Hope this helps.

[Greenlaw]

Another tip:

Keep a small mirror on your desk. It may seem obvious but a lot of people don't consider that the best reference for face and mouth animations is your own face. (Your own hands can be pretty decent reference for hand animations too.)

[Mattyj]

Thanks heaps to both of you for your advice, really appreciate it, i’m a real noob. Just quickly Greenlaw, what does SBD stand for? And I notice an interpolate check box on the Switch layer sync tab, are there any good tutorials for using this function?

Again, advice super appreciated.

Cheers

Matt

[synthsin75]

SBD = smart bone dial
If you use smart bones, interpolate sublayers will not work. That option makes switches animate smoothly from one to another, as long as they have the same number of points created in the same order...basically duplicates you've pushed the points around on.

Interpolated switches can sometimes look a bit mushy for lip sync. While it can work fine, think more singing than speaking.

[Greenlaw]

What Wes said.

Here's another commonly used abbreviation: SBA = Smart Bone Action.

A Smart Bone Dial (SBD) is a bone you can rotate to drive a predefined animation, a.k.a., a Smart Bone Action (SBA). The SBD usually sits outside of the character's skeleton and it gets its name because you turn it like a dial. There can be several SBDs in a character rig.

SBDs can drive almost anything you can animate inside Moho, including other other SBDs. For example, you could make two separate SBDs, one for each eye, called EyeWinkL and EyeWinkR, and then make a third SBD called EyesBlink that operates the other two SBDs together.

For Switch layers, you can use an SBD to flip through different mouth shapes. I used to set up mouth controls this way, and it's just one of many methods available for animating a Switch layer. These days, I prefer using the Switch Selection window to switch between mouth shapes, but Wes has a new system called Switch Icons that looks very interesting. (Sorry Wes, I still haven't tried this...just too busy with work these days. But now that I'm thinking about it, maybe try it tonight.)

A little more about Interpolate Sublayers; this is a cool feature but it's an advanced one and I suggest getting familiar with using Switch layers with the standard settings first. FWIW, I don't typically use this feature in the rigs I animate at work, and my mouth animations can look smooth enough without it. But when I really need it, I'm glad it's available.

About using SBDs and SBAs: It's tempting for beginners to create SBDs and SBAs to animate almost every aspect of character. IMO, 'SBD-featuritus' can become counter productive and make the character difficult to animate, or even worse, you wind up with animation that looks 'canned' and uninspired.

While SBDs and SBAs are among the strongest reasons to animate a character with Moho, before you add one, you should consider if it's really necessary. In many case, it might be easier to animate the feature directly. For example, do you really need an SBD to bend an elbow? In this case, it's easier to just bend the elbow directly or with IK. But an SBA used in the elbow joint to correct unwanted distortions totally makes sense. Or to animate eyes, do you really need separate controls to move the eyes in different directions? IMO, it's much easier to move the eyes directly by hand...no SBD or SBA necessary. But when you want to dilate or constrict the pupils, using an SBD makes complete sense.

A well-considered set of SBDs and SBAs will enhance your animation and speed up production. A poorly considered set will just complicate things and make it more difficult to edit the rig for special situations.

Hope this helps.

[synthsin75]

... but Wes has a new system called Switch Icons that looks very interesting. (Sorry Wes, I still haven't tried this...just too busy with work these days. But now that I'm thinking about it, maybe try it tonight.

No worries. I am interested in getting your feedback (since you're well-versed in just about every method of animating switches in production), but it's nothing crucial.

[Mattyj]

Hey guys I think I used all the Preston Blair Phenomes as mouth layers within my Switch layer, but the result was still pretty terrible. I was pretty rough with my mouth designs though. Also my audio, though it was me speaking, was purposely distorted. Do you guys know if there are any mouth libraries apart from those within the library already, available for download? None of the mouth sets available really suit my character tbh. Part of me thinks it may be best to draw them in procreate then export them as separate PNG layers, then import into Moho and stack them in a switch layer. What are your thoughts?

thanks again.

[Mattyj]

also i only have Debut atm so can’t use smart bones though am amping to upgrade.

[slowtiger]

It's always best to create your own mouths, fitting your design.

Have a look at your voice track and design style. Is it very cartoony, or is it relaxed and underplayed? Mouth shapes need to fit to that.

[Hoptoad]

I start with the Preston Blair phonemes and make 10 mouths for my character. Then I animate some lip sync and see what I think. I ask myself, Can I reduce the number of mouths? Or do I need to add more? I also tweak the phonemes: make some mouth wider, change the shape of the tongue, etc.

My most recent Switch Layer for mouths has 13 phonemes - a few more than Preston Blair.

I added one I called "misc". This is a slight variation of the CDGKNRSThYZ that I use when I would have two CDGKNRSThYZ phonemes in a row.

I made an R that is different than the one in CDGKNRSThYZ. The mouth isn't as wide.

I made a ChJSh. It looks like the R, but the mouth is even less wide.

I made a Th that is different than the one in CDGKNRSThYZ. The mouth is more open and you see the tongue.

My avatar is a phoneme, in case you couldn't tell.

It's always best to create your own mouths, fitting your design.

Have a look at your voice track and design style. Is it very cartoony, or is it relaxed and underplayed? Mouth shapes need to fit to that.

I agree. Refer to reference, but ideally make your own phonemes.

[Greenlaw]

Here are some notes for rigging and animating lip sync in Moho. I hope it gives you ideas for animating lip sync in your own characters.

This is the setup for a character I've been using in my tutorial videos...



To select the Switch layer, I click on the Mouth button in the workspace using a script called Layer Shortcuts, and the mouth shapes are automatically loaded into the Switch Selection window. Then, using a slider, I choose the mouth drawings from this window as I step through the timeline, which has the audio waveform and the dialog notated, as shown here...



I scrub back and forth in the timeline to check if the mouth shapes I've selected are suitable for the audio recording.

I mostly use the Preston Blair set here. If you can't read what's in the Layer Window, the mouth shapes in the Switch layer are:

blank
smile
Th
FV
MPB
WQ
L (D, Th)
CDGKNRSYZ
U
E
O
A, I
neutral

Not counting blank, smile, and neutral, this is technically 10 phonemes in the Switch. The character has the 'emo' SBD described earlier, which turns the mouth 'happy' when I rotate up, and 'sad' when I rotate down. The control is shown here...



You may have noticed the 'mouth angle' SBD. This control turns the mouths slightly from left to right angles. I think all I'm doing in this rig is rotating the mouth Switch layer on the y-axis in 3D, but for some rigs I'll make a sliding point animation SBA for each mouth shape. Obviously, it's a lot easier to just rotate a single (Switch) group layer in 3D space. Normally, I'll add this animation the the Head Turn SBD, but for this character, I wanted to control the mouth angle manually.

After I get the dialog satisfactorily keyframed, I'll go back and add squash and stretch layer transforms to the Mouth Switch layer. You can see the Layer Transform keyframes in the timeline image above.

With this workflow, I can get the basic lip sync animation done in a couple minutes. But making it look good, takes a little longer...as described earlier, this is where animating the performance comes in. I don't want to say this process easy, but with practice, it's really not that hard either. After a while, you'll figure out little tricks to animate quicker and in making it look consistent from shot to shot. But to get there, you have to do it...and do it...and do it. The result is as good as the effort and time you decide to put into it.

Tip: When you're animating the basic lip sync (before you add the mouth squash and stretch, and head animation,) don't feel you need to keyframe every syllable you hear...that can make the dialog animation look 'chattery' and unnatural. When dialog is spoken out loud, some mouth shapes can run together and some mouth shapes may get dropped entirely. Also, don't feel you need to open and close the mouth completely after each sound. Look at yourself in the small table mirror I suggested earlier and try speaking the dialog slowly but naturally, and you may be surprised to see that you're not making all the mouth shapes you thought your were making.

As mentioned earlier, there are many ways to animate lip sync in Moho, but I think this is a good workflow for most situations.

To see this animation, scrub to around 0:15 in this video: MQC for Moho Pro. It's a simple animation, but I think it illustrates the above points.

...I ask myself, Can I reduce the number of mouths? Or do I need to add more?

Yes, it can vary, and it depends on the level of realism you're going for, and what best serves your production. Even though I'm using 10 mouth shapes for dialog in the above example, I think it's acceptable to use far fewer mouth shapes. In a recent Fast And Furious Spy Racers episode, I created a Moho animation segment that used no more than 5 mouth shapes for each character, and it looked fine. (I'll have to check if that episode has come out yet. If it has, I'll point it out.) In a lot of anime I watch, I see only three or four mouth shapes in general dialog and a full set is used only for important scenes or dialogs. This is done to meet time and budgetary constraints of course, and I think these shortcuts are fine when applied consistently.

Hope this helps.

(Hmm...I guess I'm going to have add a 'lip sync' tutorial video to my list since I've just written a chunk of it for this post.)

[Greenlaw]

Something just occurred to me: I described rotating the mouth group in 3D because it's easier than creating individual 2D turn animations for every mouth shape...however, I can alternatively use a Smart Mesh Warp to reshape all of the mouths using the same mesh animation. This can be a good compromise...it's a little more work and simply rotating the Switch layer, but the quality might possibly be a little better.

[cgrotke]

" When parts of the dialog are spoken quickly, mouth shapes can run together and mouth shapes may get dropped. "

One of the best tricks I was taught was to put your elbow on a table and rest your jaw in your hand, then speak the lines. You will literally feel where the accents/wide mouths need to be.

"Chocolate Milk" , for example, opens your jaw twice (not 4 or 5 times if you go every syllable).

I use this trick when I find myself getting lost in the details and need to stop and reconsider the goal.. : )

....
Tips for lazy lip sync based on volume.... I just did a project with very cartoony mouths and didn't use Preston Blair. I just made one closed mouth, duplicated it about 4 times, and made each other subsequent mouths a bit bigger. So, closed, slightly open, open some more, open wide, and open really wide. These were basically half moon shapes. Then I made a circular "oo" to add in ( i kept it at the top of the switch layers... the built-in audio sync rarely decides to use the highest switch layer, but I can manually select it when needed.) I let the frames interpolate, to give some variety to the in-between shapes.

I used the built-in Moho audio sync to set the switch layers roughly to the audio file. It goes by volume, and tends to put in too many switches, so the next step is to go through the whole thing and remove some of the extras, and re-align things to hit accents a bit better. And if I come across any "o" or "oo" sounds, I swapped in that one where necessary.

It's not true lip sync, but it works. : )