Simultaneous speech and animation synthesis

by Dietmar Schabus, Michael Pucher and Gregor Hofer
Abstract:
Talking computer animated characters are a common sight in video games and movies. Although doing the mouth animation by hand gives the best results, because of cost and time constraints it is not always feasible. Furthermore the amount of speech in current games is ever increasing with some games having more than 200,000 lines of dialogue. This work proposes a system that can produce speech and the corresponding lip animation simultaneously using a statistical machine learning framework based on Hidden Markov Models (HMMs). The key point is that with the developed system never before seen or heard animated dialogues can be produced at a push of a button.
Reference:
Simultaneous speech and animation synthesis (Dietmar Schabus, Michael Pucher and Gregor Hofer), In ACM SIGGRAPH Posters, 38th International Conference and Exhibition on Computer Graphics and Interactive Techniques, 2011.
Bibtex Entry:
@InProceedings{Schabus2011,
  author    = {Dietmar Schabus and Michael Pucher and Gregor Hofer},
  title     = {Simultaneous speech and animation synthesis},
  booktitle = {ACM SIGGRAPH Posters, 38th International Conference and Exhibition on Computer Graphics and Interactive Techniques},
  year      = {2011},
  pages     = {8:1-8:1},
  address   = {Vancouver, BC, Canada},
  month     = aug,
  abstract  = {Talking computer animated characters are a common sight in video games and movies. Although doing the mouth animation by hand gives the best results, because of cost and time constraints it is not always feasible. Furthermore the amount of speech in current games is ever increasing with some games having more than 200,000 lines of dialogue. This work proposes a system that can produce speech and the corresponding lip animation simultaneously using a statistical machine learning framework based on Hidden Markov Models (HMMs). The key point is that with the developed system never before seen or heard animated dialogues can be produced at a push of a button.},
  acmid     = {2037724},
  articleno = {8},
  comment   = {<br><a href="/phd/schabus_siggraph_2011.mov">Demonstration video</a> (32 MB)},
  doi       = {10.1145/2037715.2037724},
  file      = {/download/schabus_siggraph_2011},
  groups    = {FTW, Visual},
  isbn      = {978-1-4503-0971-4},
  numpages  = {1},
  owner     = {schabus},
  timestamp = {2013.11.21},
}