Computer scientists achieve virtual ventriloquism by making celebrities move and talk like other famous people.
What makes people look like themselves? How do you look at a picture and recognize yourself or a friend? Is it the facial expressions we make? Our facial features? Computer scientists at the University of Washington have been working to determine this.
In a published paper called: “What Makes Tom Hanks Look Like Tom Hanks,” scientists used thousands of photographs of Tom Hanks, taken over the course of his career, to map his face and replicate it digitally. A 3D simulation is just one of the replicas they have been working with that can mimic particular people’s expressions by using 49 predefined points on a person’s face. The algorithm developed by the researchers uses shadows to help determine the shape of someone’s face.
“Tom Hanks has appeared in many acting roles over the years. He’s played young and old, smart and simple, characters with a wide variety of temperaments and personalities. Yet, we always recognize him as Tom Hanks. Why? Is it his shape? His appearance? The way he moves?” the authors wrote in their paper.
Although Tom Hanks isn’t of particular scientific interest, studying his face could lead to important breakthroughs in virtual reality and filmmaking.
According to The Atlantic, “What’s special about the latest work by Kemelmacher-Shlizerman and her colleagues is that the modeling comes from pre-existing photo collections—hence the focus on Hanks, who has been photographed a lot over the years. Other researchers in this space have relied on specific environments and lighting levels for their subjects in order to make realistic models.”
The researchers didn’t stop at simply modeling a person’s face — they used the captured facial expressions and projected them onto models of other people’s faces. According to Kemelmacher-Shlizerman, the purpose of this is to evaluate how successfully they captured a person’s likeness. Does Barack Obama still look like himself even when he is making George W. Bush’s facial expressions?
“If we can produce a realistic model after transferring expressions, it means that we captured something,” Kemelmacher-Shlizerman told The Atlantic. “Something in the [data] flow that seems like it captures that person’s identity. We cannot say yet that because George Bush raises his eyebrows some particular way, this is what makes George Bush George Bush.”
Aside from the scientific applications, the research has led to some pretty funny video clips!