Misaki Thank you! 😊
Here's how I do the lipsync, in a nutshell...
I use multiple animation tracks. One for the base animation (e.g. idle, talk, walk, etc), one for any additional stuff (e.g. holding a fishing rod) and one for lipsync. So any base animation that I know will involve talking does not specifiy any mouth movement.
Then, I create phoneme animations for each character. These are all just 1 frame long, and only affect the mouth/teeth/tongue of the character.
I use Papagayo-NG, a lipsyncing program, to break down the lines of text into phonemes and then align those phonemes to the audio. It generates short text files with timing, like a subtitle for a movie, but this has phonemes, instead of words.
Finally, when a character starts talking I set their base animation to "talk", and I use the tiny lipsync subtitle files to blend between the different phoneme animations on the lipsync animation track.
I am planning to write about this in detail in our blog, once we start recording voice overs for the chapter (next week! 😱) I'll come back and post it here when it's done.
