How OnPoint Studios is getting a hand on motion capture fingers

By PETE CARVILL, OnPoint Studios

Motion capture has given us great moments in acting over the last ten-to-fifteen years, from Sam Worthington and Zoe Saldana in Avatar to Andy Serkis’s and Benedict Cumberbatch’s portrayals of Gollum and Smaug in The Lord of the Rings and The Hobbit film series, to Mark Ruffalo as The Hulk in the Marvel Cinematic Universe. Audiences have become increasingly used to ever-more-sophisticated parts of an actor’s craft in motion capture, seeing the flickers and tremors of emotion that often pass across flesh, transported into an augmented performance.

Hand is holding an axe for hand capture with gloves from StretchSense

But one aspect of this technology still proves difficult today, and that is the rendering of hands.

There are twenty-seven joints and bones in the human hand, held together by over one hundred tendons and ligaments, and flexed by thirty-four muscles; and ten fingers, all of them capable of individual movement, and not always moving in concert. Some things are very easy. Hands, let us be clear, are difficult.

Niklas Bothe is head of motion capture at OnPoint Studios in Berlin. He says that there are reasons why full-performance capture with fingers has not been common over the years. “It’s hard with a passive system like ours,” he says. “It means that you have to put lots of little dots on the gloves and fingers. Having a lot of markers so close to each other produces a lot of work in post production.”

There has not been much in the ways of solutions to these problems, just workarounds. The problem is that the result often looks a little tarnished. Says Niklas, “Cleaning the motion capture data takes forever. It’s much faster to put hand poses together in post-production. This is especially true when there are several actors at the same time. Capturing all their finger movements is quite the challenge.”

Recently, the motion capture studio in Berlin has begun a partnership with the Auckland, New Zealand-based firm StretchSense. The aim of the partnership is to improve the rendering of hands as part of the Berlin studio’s motion capture offerings. Given that there is roughly 17,735 kilometers between Berlin and Auckland, this is a truly-international cooperation.

Shin Jeong Park is commercial team manager at StretchSense. She says that there is a really-good synergy between that company and OnPoint. “They are a nimble team that works really fast in validating and integrating new tech into their pipeline. They had so many cool showcases that they had done and were planning to do that we really wanted to be a part of.”

Gif shows the precision of hand capture with gloves

StretchSense was co-founded in 2012 by CEO Benjamin O’Brien and, since 2018, has been dedicated to developing and producing a glove to improve the capture of data in motion capture. A key part of the glove’s development has been a type of stretch sensor that tracks how each finger and thumb moves. Within each glove, every digit has a splay sensor with three sensing zones to capture each knuckle’s bend as well as the sideways splay, or finger spread, of each digit. A wrist sensor captures the lateral movement of the wrist for improved calibration. A low-profile sensing board connects to all 16 sensing channels, with a glove circuit that supports Bluetooth and USB communication protocols, timecode input, MicroSD logging, and an optional IMU for hand orientation and tracking in space.

Park says, “Our technology doesn’t rely on the use of optical markers you don’t need cameras to record data and you don’t need to worry about occlusion issues. I think the most ground-breaking improvement that studios get, though, is just the very high level of clean and accurate data.”

The gloves are also adjustable. “We know,” adds Park, “that hands come in different shapes, sizes, and range of motion. We’ve taken this into account and designed our software with the ability to tailor in mind. Studios are able to calibrate the gloves to their performers and build custom-pose libraries to get the perfect effect they are looking for.”

Both sides are currently benefitting from the arrangement. While StretchSense is gaining a more in-depth understanding of how their technology aligns with the full-body motion capture, OnPoint are able to offer a better, more-tailored product for their customers. Says Bothe, “These gloves make it possible to pick up the smallest details of hands, especially when they’re doing things like playing musical instruments or holding a pen. We are now able to capture all of this more accurately. It’s not only shortening dramatically the time we spend in post production, it’s giving us a better product.”

Recent posts

How OnPoint Studios is getting a hand on motion capture fingers

By PETE CARVILL, OnPoint Studios

Motion capture has given us great moments in acting over the last ten-to-fifteen years, from Sam Worthington and Zoe Saldana in Avatar to Andy Serkis’s and Benedict Cumberbatch’s portrayals of Gollum and Smaug in The Lord of the Rings and The Hobbit film series, to Mark Ruffalo as The Hulk in the Marvel Cinematic Universe. Audiences have become increasingly used to ever-more-sophisticated parts of an actor’s craft in motion capture, seeing the flickers and tremors of emotion that often pass across flesh, transported into an augmented performance.

Hand is holding an axe for hand capture with gloves from StretchSense

But one aspect of this technology still proves difficult today, and that is the rendering of hands.

There are twenty-seven joints and bones in the human hand, held together by over one hundred tendons and ligaments, and flexed by thirty-four muscles; and ten fingers, all of them capable of individual movement, and not always moving in concert. Some things are very easy. Hands, let us be clear, are difficult.

Niklas Bothe is head of motion capture at OnPoint Studios in Berlin. He says that there are reasons why full-performance capture with fingers has not been common over the years. “It’s hard with a passive system like ours,” he says. “It means that you have to put lots of little dots on the gloves and fingers. Having a lot of markers so close to each other produces a lot of work in post production.”

There has not been much in the ways of solutions to these problems, just workarounds. The problem is that the result often looks a little tarnished. Says Niklas, “Cleaning the motion capture data takes forever. It’s much faster to put hand poses together in post-production. This is especially true when there are several actors at the same time. Capturing all their finger movements is quite the challenge.”

Recently, the motion capture studio in Berlin has begun a partnership with the Auckland, New Zealand-based firm StretchSense. The aim of the partnership is to improve the rendering of hands as part of the Berlin studio’s motion capture offerings. Given that there is roughly 17,735 kilometers between Berlin and Auckland, this is a truly-international cooperation.

Shin Jeong Park is commercial team manager at StretchSense. She says that there is a really-good synergy between that company and OnPoint. “They are a nimble team that works really fast in validating and integrating new tech into their pipeline. They had so many cool showcases that they had done and were planning to do that we really wanted to be a part of.”

Gif shows the precision of hand capture with gloves

StretchSense was co-founded in 2012 by CEO Benjamin O’Brien and, since 2018, has been dedicated to developing and producing a glove to improve the capture of data in motion capture. A key part of the glove’s development has been a type of stretch sensor that tracks how each finger and thumb moves. Within each glove, every digit has a splay sensor with three sensing zones to capture each knuckle’s bend as well as the sideways splay, or finger spread, of each digit. A wrist sensor captures the lateral movement of the wrist for improved calibration. A low-profile sensing board connects to all 16 sensing channels, with a glove circuit that supports Bluetooth and USB communication protocols, timecode input, MicroSD logging, and an optional IMU for hand orientation and tracking in space.

Park says, “Our technology doesn’t rely on the use of optical markers you don’t need cameras to record data and you don’t need to worry about occlusion issues. I think the most ground-breaking improvement that studios get, though, is just the very high level of clean and accurate data.”

The gloves are also adjustable. “We know,” adds Park, “that hands come in different shapes, sizes, and range of motion. We’ve taken this into account and designed our software with the ability to tailor in mind. Studios are able to calibrate the gloves to their performers and build custom-pose libraries to get the perfect effect they are looking for.”

Both sides are currently benefitting from the arrangement. While StretchSense is gaining a more in-depth understanding of how their technology aligns with the full-body motion capture, OnPoint are able to offer a better, more-tailored product for their customers. Says Bothe, “These gloves make it possible to pick up the smallest details of hands, especially when they’re doing things like playing musical instruments or holding a pen. We are now able to capture all of this more accurately. It’s not only shortening dramatically the time we spend in post production, it’s giving us a better product.”