Facial MoCap experiments

[Bendytoons]

I think people should realize that Zign Track is a facial motion capture solution, not an overall workaround for a good face rig and lip synch poses. The more you rig your face and setup mouth poses the better Zign Track's results will be for you... am I right Luuk?

thanks for putting that suucinctly. That's pretty much what I meant.

 

Ben

[Luuk Steitner]

I think people should realize that Zign Track is a facial motion capture solution, not an overall workaround for a good face rig and lip synch poses. The more you rig your face and setup mouth poses the better Zign Track's results will be for you... am I right Luuk?

 

Yep

[Luuk Steitner]

Maybe it's a good idea to show what I'm working on for version 1.2. The new A:M action feature works quite well, as far as I have tested. Here is a picture so you can see what the possibilities for the poses are.

The eyebrow poses involve up and down movement, the others speak for themselves I guess. I did some tests with these poses, and it looked pretty good. One thing I'm not sure about yet is if the mouth poses work 100% like they should. I need to do a few more tests to see if the result is good on all movements, and maybe I have to adjust the scaling of those movements. I hope Paul and Ben have some time to test the mouth poses, if not I'll do an extensive test later this week.

I''l explain what the mouth poses do: The mouth wide/purse pose should contain exactly what it's name says. This is also true for the mouth smile/frown pose. As you know, the mouth also gets wider on a smile, so when Zign Track detects a wide smile it reduces the mouth wide pose, so they don't add. The mouth open/close and shift left/right would be obvious.

Using these poses is a little bit less accurate like the original tracked footage, but the advantage is that the shapes it produces always suit the face (if the modeler made nice poses of course).

 

Tell me what you guys think about this. I want to have it as good as possible before I release it. Maybe I'll change some things if that can give better results.

 

Other than this, I'm working on some improvements for the output for all formats. On the left you can see that I have added an extra "neutral frame" spin box. This can be used to specify on which frame the face is most neutral. I did this because Zign Track was giving some strange offsets if the face in the first frame was not neutral. I'm also changing some calculations to improve the accuracy while the face is moving. This should solve the lip issues that sometimes occur. When this is done there should be very little manual tweaking required for the animation.

[Bendytoons]

As you know, the mouth also gets wider on a smile, so when Zign Track detects a wide smile it reduces the mouth wide pose, so they don't add.

Hey Luuk,

 

You might make adjustments like this optional.

 

The way I build poses, the widening of the smile is driven by the wide/purse pose. the smile pose drives only the corners of the mouth. I never let any pose overlap action with another pose, that way they can all combine together but never create unnatural action.

[Luuk Steitner]

Hey Luuk,

 

You might make adjustments like this optional.

 

The way I build poses, the widening of the smile is driven by the wide/purse pose. the smile pose drives only the corners of the mouth. I never let any pose overlap action with another pose, that way they can all combine together but never create unnatural action.

 

That is actually a good idea. The reason I made the smile reduce the wide pose is because it is harder to 'model' the smile in the the pose when the mouth isn't wide. But that could be solved by applying the wide pose while creating the smile, and when the smile is done remove the link the 'wide' from the pose. Is that how you do it?

[Bendytoons]

Hey Luuk,

 

You might make adjustments like this optional.

 

The way I build poses, the widening of the smile is driven by the wide/purse pose. the smile pose drives only the corners of the mouth. I never let any pose overlap action with another pose, that way they can all combine together but never create unnatural action.

 

That is actually a good idea. The reason I made the smile reduce the wide pose is because it is harder to 'model' the smile in the the pose when the mouth isn't wide. But that could be solved by applying the wide pose while creating the smile, and when the smile is done remove the link the 'wide' from the pose. Is that how you do it?

In a nutshell, yes. When I create a pose, I apply all the poses that will interact, one by one, and then delete them from the pws. Gives the most performance from the fewest poses, and builds in limits that prevent any kind of overshoot.

[Luuk Steitner]

It appears to be mostly just up and down movements of the jaw.

No sound forming lip shapes as far as I can tell.

Now why is this?

 

Mike, you were right! This video was made with the A:M action bone control. I was just doing a lot of tests to improve the output accuracy (for all formats) and I used A:M actions with bone control because that's the fastest way to test it. For some reason the mouth shape wouldn't do what it's supposed to do. I took me a while before it occurred to me what mistake I had made; I used the same method as I did for the BVH for controlling the bones. I forgot that the BVH axis orientations are different and I didn't adjust it for the mouth bones. So, in stead of rotating the Y axis I was rotating the Z axis... doh...

 

Sorry Paul, I'll finish some more things and I'll send you the new test version. You'll find the output results are much better this time

[Paul Forwood]

Cool!

 

---------

 

Edit: I haven't had a chance to play with the new pose feature yet but I think I understand the concept better now and I think it's great. This will enable a single action to drive a wide variety of different faces, human or otherwise.

[Paul Forwood]

Here is an example of the previous test but with the Zign Track update to fix the rotation of those bones:

 

Magic Mouth Motion 2

 

It doesn't look all that different to me and the mouth does something strange in the last few seconds. It is possibly a combination of me not getting the bones positioned correctly and bad miming. I really should finish this model and get the weighting done too.

 

No real animation here. Just a few rough poses at the beginning.

[Luuk Steitner]

Here is an example of the previous test but with the Zign Track update too fix the rotation of those bones:

 

Magic Mouth Motion 2

 

It doesn't look all that different to me and the mouth does something strange in the last few seconds. It is possibly a combination of me not getting the bones positioned correctly and bad miming. I really should finish this model and get the weighting done too.

 

No real animation here. Just a few rough poses at the beginning.

 

Can you show a picture of your face rig? And can you check what happens at the end of the time line for the mouth bones in the action? I noticed in one of my tests that one bone had a strange ending. I'm not sure what caused it, but I'll try to find out what happens.

[Paul Forwood]

Much the same thing with a static camera: Magic Mouth Motion 2b Once again, just partially blocked. I'll delete the older versions tomorrow.

 

I was just about to do a screen capture of my face rig when I noticed that most of the bones are sitting too low in the face. I guess I forgot to move them after adjusting the mesh.

Oh, well. It's a learning game.

I will test again and see what happens. Logic tells me that moving the bones up will only have made the problem worse. I'm going to try the pose method soon but first I want to check that the mouth corners are tracking correctly by videoing some extreme poses and tracking them without any filtering.

--------------------

Edit: I just moved the bones up and then scrubbed through to the end of the choreography and it all looks much the same.

[Luuk Steitner]

Much the same thing with a static camera: Magic Mouth Motion 2b Once again, just partially blocked. I'll delete the older versions tomorrow.

 

I was just about to do a screen capture of my face rig when I noticed that most of the bones are sitting too low in the face. I guess I forgot to move them after adjusting the mesh.

Oh, well. It's a learning game.

I will test again and see what happens. Logic tells me that moving the bones up will only have made the problem worse. I'm going to try the pose method soon but first I want to check that the mouth corners are tracking correctly by videoing some extreme poses and tracking them without any filtering.

--------------------

Edit: I just moved the bones up and then scrubbed through to the end of the choreography and it all looks much the same.

 

Are you sure all mouth bones are actually moving? I think you might have given them wrong names so they don't move. I only see the jaw and lower lip moving. Maybe I'm wrong, but that's what it looks like.

[Paul Forwood]

Well, it turned out to be a little glitch in the software which was affecting the motion but it seems to be fixed now. Thanks, Luuk!

 

This is my first semi-successful test using Zign Track to drive my own poses:

 

Magic Poses 1

 

I haven't tweaked the poses at all so I am sure that I will be able to bring more life to this face when I do. I like the fact that I am able to bring more forward/backward motion to the mouth using this method but I think that I still prefer the bone driven action export to this pose driver action. That view may change as I get more comfortable with this method though.

 

I need to add a pose for "F/V" and animate the tongue. That should help. Then I want to play with exaggerating everything.

---------------------

Edit: One thing that bothers me about my pose driver tests is that my model has picked up jitter in his hands and eyes. I assumed that I must have touched the pose sliders for those parts while setting up my facial poses but I have looked at all the poses and there is no sign of any other pose being active. I'm sure it is just something stupid that I have done.

[Luuk Steitner]

Not bad I wonder what causes the jitter on the hands and eyes. I can't imagine that it's caused by the action, that would be strange. I wouldn't worry too much about using the smooth filter, if you use 1 or 2 passes for the head only, it removes the jitter but it leaves the total motion almost unaffected. The smoothing routine is designed to reduce small peaks.

[Paul Forwood]

I'm just trying tracking the eyelids and exporting as a second action. It works but this example didn't have the bones set up quite right. Also the sync is out by a few frames. Still worth a look though, maybe.

 

I wonder.

 

I have tried to view this on Google video but it isn't playing well for me so can you let me know if you are unable to view it. I have a better version in the works so I will replace it when it is ready.

[Luuk Steitner]

That looks pretty good, you might want to use the smooth function to get rid of the jitter.

Why don't you try using a pose for the eyelids? If you use the eyebrow B features for the eyelids you could exaggerate them to ~150% and drive blink poses that close on -100% I think that's easier to do then trying to find a correct size and place for the eyelid bones, plus you will have better control over the motion curve.

 

Keep going , nice character btw.

[Paul Forwood]

I agree, Luuk. I will try using poses next and add some smoothing too.

 

In the meantime here is a render which shows the eyelid motion closer to the original.

I Wonder 2

I was reading from a sheet of paper and my eyes were down most of the time. I need to set up some kind of auto que on a monitor so that I can read it without my glasses.

[Paul Forwood]

Well, I know that I said I would use pose actions for the eyes this time but I'm enjoying using bone actions too much at the moment. This is getting closer now. Plenty of weighting issues to iron out and modelling to finish and textures to do though.

 

(No sound - Just pulling faces)

 

It seems that the grooming vanished from the hair somehow.

 

 

[John Bigboote]

Two words: Wow!

[Luuk Steitner]

Nice Paul. What are the smoothing settings you have used? I think the head bone can use a bit more smoothing.

[Paul Forwood]

What are the smoothing settings you have used? I think the head bone can use a bit more smoothing.

 

I just had pre-smoothing for the head and neck set to 1 or 2 and everything else set to 0.

I have tried exporting with higher values for the neck and head now and have kept the mouth smoothing to a minimum. With smoothing set to 10 for the neck and head the results are much better. Here is an example. (No eye motion in this one):

 

 

 

Smoothies A00

 

(Google Video has compressed everything out of this video. It must be due to the non-standard proportions. Play it in full-screen to see the difference that high smoothing values make to head jitter.)

 

Next I am going to concentrate on using Zign Track's Pose Driving Actions

[Luuk Steitner]

Well, the one on the right is very smooth

With 10 passes you'll loose some motion, personally I would never use values above 5 for the head (if a normal quality video is used) With 10 passes bigger movements will still be visible, but in this video there are no big movements.

I hope you'll enjoy the ease of direct pose control

[Paul Forwood]

I found this gathering dust today. Just another Zign Track test.

 

Ain't... animated.

 

[John Bigboote]

Gatherin dust, INDEED! That's the BEST sample I've seen yet!