Stories

A summer research program taught Isabela Figueira ’18 how to use machine learning in computer animation—and changed her career path along the way. 

Isabela Figueira ’18 spent an entire summer focused just on hands. Specifically fingers. What do they do during certain gestures? What makes their movement human?  

The uncanny valley is big, so to speak. Making convincing computer animation isn’t just about faces. It’s the details—like fingers. And to perfect the details, Figueira called on machine learning.   

Figueira traveled to Clemson University last summer to work on an NSF funded research project led by assistant professor Sophie Jörg. The goal of the project was to use machine learning to automate parts of the animation process. 

“Finger motions and conversations are really tedious to manually animate,” Figueira explains. “When you’re animating a character, you have key frames. You have one frame and it moves and the computer automatically fills in. But the computer doesn’t know how the fingers move.”

Figueira used motion capture technology to track finger movements when people make gestures—like a shrug or a wave. The data points could then be used to develop algorithms that could successfully predict the label of a gesture and eventually generate realistic finger motions to group with body movements.

Essentially, the data and algorithms would teach the computer how fingers are supposed to move so it could automate the process for computer animators of movies, TV shows, and video games. Rather than taking time animating finger movements, animators could build a database of motions that could be reused. The algorithm would then pick pieces of a host of gestures to make a new gesture that would portray the intended emotion. All made possible through machine learning.

“I spent so much time in the lab coding and just thinking about it,” Figueira says of the research. “I learned so much about animation I never knew. It was amazing.” 

Figueira first encountered machine learning during a research project she worked on with assistant professor Yi Fang. She’d met Fang when she took his Introduction to Programming class and in 2016, he asked her if she would be interested in working on a machine learning project involving Twitter.  

The point of the project was to develop machine learning algorithms that would learn about Twitter users based on their account bios and their tweets. Essentially developing a model that identified what types of bios corresponded with what kinds of tweets. Figueira used sample tweets and bios to help develop the algorithms, then tested the algorithms against new tweets. 

While computer animation and Twitter might seem different, they both come down to data. For computer animation, the data is the configuration of fingers. For Twitter, it’s words: how often certain words appear, their correlation with each other, and their correlation with bios and tweets.  

Getting involved in machine learning and research projects outside of class not only taught Figueira new skills, but about the real-world impact of her computer engineering major. It also introduced her to new applications for her interests and even a new career path.   

“The greatest part of the research experience last summer was the exposure I got to a different application of computer science,” Figueira says. “I never would have encountered that had I not applied to the research program.” 

Figueira is still riding the wave from her summer experience. She continued to explore computer animation this year, taking 3-D Modeling and Animation (COEN 165) at Santa Clara, and continued her research on hand gestures.   

Her latest computer animation algorithms successfully predicted gesture labels 87.5 percent of the time. She presented a poster at the Grace Hopper Celebration for women in computing in Orlando, Fla. in October and hopes to publish a paper. She also applied for the NCWIT Collegiate Award using her research, and was one of 60 finalists for the award.  

Figueira will attend UC Irvine in the fall to pursue her Ph.D. in computer science with a focus on computer graphics. She hopes to include machine learning in her studies.