.svg)
13,401
edits
Motion capture: Difference between revisions
→Methods and systems: BP:N'T
(robot: Update article (please report if you notice any mistake or error in this edit)) |
|||
| Line 1: | Line 1: | ||
{{short description|Process of recording the movement of objects or people}} | {{short description|Process of recording the movement of objects or people}} | ||
{{original research | date= June 2013}} | {{original research | date= June 2013}} | ||
[[File:Temporal-Control-and-Hand-Movement-Efficiency-in-Skilled-Music-Performance-pone.0050901.s001.ogv|thumb|300px|Motion capture of two [[pianist]]s' right hands playing the same piece (slow-motion, no-sound)<ref>{{Cite journal | last1 = Goebl | first1 = W. | last2 = Palmer | first2 = C. | editor1-last = Balasubramaniam | editor1-first = Ramesh | doi = 10.1371/journal.pone.0050901 | title = Temporal Control and Hand Movement Efficiency in Skilled Music Performance | journal = PLOS ONE | volume = 8 | issue = 1 | pages = e50901 | year = 2013 | pmid = 23300946| pmc =3536780 | bibcode = 2013PLoSO...850901G | doi-access = free }}</ref>]] | [[File:Temporal-Control-and-Hand-Movement-Efficiency-in-Skilled-Music-Performance-pone.0050901.s001.ogv|thumb|300px|Motion capture of two [[pianist]]s' right hands playing the same piece (slow-motion, no-sound)<ref>{{Cite journal | last1 = Goebl | first1 = W. | last2 = Palmer | first2 = C. | editor1-last = Balasubramaniam | editor1-first = Ramesh | doi = 10.1371/journal.pone.0050901 | title = Temporal Control and Hand Movement Efficiency in Skilled Music Performance | journal = PLOS ONE | volume = 8 | issue = 1 | pages = e50901 | year = 2013 | pmid = 23300946| pmc =3536780 | bibcode = 2013PLoSO...850901G | doi-access = free }}</ref>]] | ||
[[File:Two repetitions of a walking sequence of an individual recorded using a motion-capture system.gif|thumb|300px|Two repetitions of a walking sequence recorded using | [[File:Two repetitions of a walking sequence of an individual recorded using a motion-capture system.gif|thumb|300px|Two repetitions of a walking sequence recorded using motion capture<ref>{{Citation |last1=Olsen | first1=NL |last2=Markussen |first2=B | last3=Raket | first3=LL| year=2018 |title=Simultaneous inference for misaligned multivariate functional data |journal= Journal of the Royal Statistical Society, Series C |volume=67 |issue=5 |pages=1147–76 |doi=10.1111/rssc.12276|arxiv=1606.03295 | s2cid=88515233 }}</ref>]] | ||
'''Motion capture''' (sometimes referred as '''mo-cap''' or '''mocap''', for short) is the process of recording the [[motion (physics)|movement]] of objects or people. It is used in [[Military science|military]], [[entertainment]], [[sports]], medical applications, and for validation of computer vision<ref>David Noonan, Peter Mountney, Daniel Elson, Ara Darzi and Guang-Zhong Yang. A Stereoscopic Fibroscope for Camera Motion and 3-D Depth Recovery During Minimally Invasive Surgery. In proc ICRA 2009, pp. 4463–68. http://www.sciweavers.org/external.php?u=http%3A%2F%2Fwww.doc.ic.ac.uk%2F%7Epmountne%2Fpublications%2FICRA%25202009.pdf&p=ieee</ref> and robots.<ref>Yamane, Katsu, and Jessica Hodgins. "[https://pdfs.semanticscholar.org/8de6/2ececd067c3d9e7d6f3462164a9a821d9e0a.pdf Simultaneous tracking and balancing of humanoid robots for imitating human motion capture data]." Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on. IEEE, 2009.</ref> In [[filmmaking]] and [[video game development]], it refers to recording actions of [[Motion capture acting|human actors]] | '''Motion capture''' (sometimes referred as '''mo-cap''' or '''mocap''', for short) is the process of recording the [[motion (physics)|movement]] of objects or people. It is used in [[Military science|military]], [[entertainment]], [[sports]], medical applications, and for validation of computer vision<ref>David Noonan, Peter Mountney, Daniel Elson, Ara Darzi and Guang-Zhong Yang. A Stereoscopic Fibroscope for Camera Motion and 3-D Depth Recovery During Minimally Invasive Surgery. In proc ICRA 2009, pp. 4463–68. http://www.sciweavers.org/external.php?u=http%3A%2F%2Fwww.doc.ic.ac.uk%2F%7Epmountne%2Fpublications%2FICRA%25202009.pdf&p=ieee</ref> and robots.<ref>Yamane, Katsu, and Jessica Hodgins. "[https://pdfs.semanticscholar.org/8de6/2ececd067c3d9e7d6f3462164a9a821d9e0a.pdf Simultaneous tracking and balancing of humanoid robots for imitating human motion capture data]." Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on. IEEE, 2009.</ref> In [[filmmaking]] and [[video game development]], it refers to recording actions of [[Motion capture acting|human actors]] and using that information to animate [[digital character]] models in 2D or 3D [[computer animation]].<ref>NY Castings, Joe Gatt, [http://www.nycastings.com/dmxreadyv2/blogmanager/v3_blogmanager.asp?post=motioncaptureactors Motion Capture Actors: Body Movement Tells the Story] {{webarchive|url=https://web.archive.org/web/20140703113656/http://www.nycastings.com/dmxreadyv2/blogmanager/v3_blogmanager.asp?post=motioncaptureactors |date=2014-07-03 }}, Accessed June 21, 2014</ref><ref name=twsBackstage>Andrew Harris Salomon, Feb. 22, 2013, Backstage Magazine, [http://www.backstage.com/news/spotlight/growth-performance-capture-helping-gaming-actors-weather-slump/ Growth In Performance Capture Helping Gaming Actors Weather Slump], Accessed June 21, 2014, "..But developments in motion-capture technology, as well as new gaming consoles expected from Sony and Microsoft within the year, indicate that this niche continues to be a growth area for actors. And for those who have thought about breaking in, the message is clear: Get busy...."</ref><ref name=twsGuardian>Ben Child, 12 August 2011, ''The Guardian'', [https://www.theguardian.com/film/2011/aug/12/andy-serkis-motion-capture-acting Andy Serkis: why won't Oscars go ape over motion-capture acting? Star of Rise of the Planet of the Apes says performance capture is misunderstood and its actors deserve more respect], Accessed June 21, 2014</ref> When it includes face and fingers or captures subtle expressions, it is often referred to as '''performance capture'''.<ref name=twsWired>Hugh Hart, January 24, 2012, Wired magazine, [https://www.wired.com/2012/01/andy-serkis-oscars/ When will a motion capture actor win an Oscar?], Accessed June 21, 2014, "...the Academy of Motion Picture Arts and Sciences' historic reluctance to honor motion-capture performances ... Serkis, garbed in a sensor-embedded Lycra body suit, quickly mastered the then-novel art and science of performance-capture acting. ..."</ref> In many fields, motion capture is sometimes called '''motion tracking''', but in filmmaking and games, motion tracking usually refers more to '''[[match moving]]'''. | ||
[[File:MoCap.gif|thumb|Raw data from Husky Sense motion capture clothing (with IMU sensors) on top of a real-time recorded moves|301x301px]] | |||
In motion capture sessions, movements of one or more actors are sampled many times per second. Whereas early techniques used [[3D reconstruction from multiple images|images from multiple cameras to calculate 3D positions]],<ref>Cheung, German KM, et al. "[https://www.researchgate.net/profile/Takeo_Kanade/publication/3854315_Real_time_system_for_robust_3D_voxel_reconstruction_of_human_motions/links/02e7e51c9c14d5ba39000000/Real-time-system-for-robust-3D-voxel-reconstruction-of-human-motions.pdf A real time system for robust 3D voxel reconstruction of human motions]." Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on. Vol. 2. IEEE, 2000.</ref> often the purpose of motion capture is to record only the movements of the actor, not their visual appearance. This ''animation data'' is mapped to a 3D model so that the model performs the same actions as the actor. This process may be contrasted with the older technique of [[rotoscoping]]. | In motion capture sessions, movements of one or more actors are sampled many times per second. Whereas early techniques used [[3D reconstruction from multiple images|images from multiple cameras to calculate 3D positions]],<ref>Cheung, German KM, et al. "[https://www.researchgate.net/profile/Takeo_Kanade/publication/3854315_Real_time_system_for_robust_3D_voxel_reconstruction_of_human_motions/links/02e7e51c9c14d5ba39000000/Real-time-system-for-robust-3D-voxel-reconstruction-of-human-motions.pdf A real time system for robust 3D voxel reconstruction of human motions]." Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on. Vol. 2. IEEE, 2000.</ref> often the purpose of motion capture is to record only the movements of the actor, not their visual appearance. This ''animation data'' is mapped to a 3D model so that the model performs the same actions as the actor. This process may be contrasted with the older technique of [[rotoscoping]]. | ||
Camera movements can also be motion captured so that a virtual camera in the scene will pan, tilt or dolly around the stage driven by a camera operator while the actor is performing. At the same time, the motion capture system can capture the camera and props as well as the actor's performance. This allows the computer-generated characters, images and sets to have the same perspective as the video images from the camera. A computer processes the data and displays the movements of the actor, providing the desired camera positions in terms of objects in the set. Retroactively obtaining camera movement data from the captured footage is known as ''match moving'' or ''[[camera tracking]]''. | Camera movements can also be motion captured so that a virtual camera in the scene will pan, tilt or dolly around the stage driven by a camera operator while the actor is performing. At the same time, the motion capture system can capture the camera and props as well as the actor's performance. This allows the computer-generated characters, images and sets to have the same perspective as the video images from the camera. A computer processes the data and displays the movements of the actor, providing the desired camera positions in terms of objects in the set. Retroactively obtaining camera movement data from the captured footage is known as ''match moving'' or ''[[camera tracking]]''. | ||
The first virtual actor animated by motion-capture was produced in 1993 by Didier Pourcel and his team at Gribouille. It involved "cloning" the body and face of French comedian Richard Bohringer, and then animating it with still-nascent motion-capture tools. | |||
==Advantages== | ==Advantages== | ||
| Line 15: | Line 18: | ||
* Low latency, close to real time, results can be obtained. In entertainment applications this can reduce the costs of keyframe-based [[animation]].<ref name="Xsens MVN Animate - Products">{{Cite web|url=https://www.xsens.com/products/xsens-mvn-animate/|title=Xsens MVN Animate – Products|website=Xsens 3D motion tracking|language=en-US|access-date=2019-01-22}}</ref> The [[Hand Over]] technique is an example of this. | * Low latency, close to real time, results can be obtained. In entertainment applications this can reduce the costs of keyframe-based [[animation]].<ref name="Xsens MVN Animate - Products">{{Cite web|url=https://www.xsens.com/products/xsens-mvn-animate/|title=Xsens MVN Animate – Products|website=Xsens 3D motion tracking|language=en-US|access-date=2019-01-22}}</ref> The [[Hand Over]] technique is an example of this. | ||
* The amount of work does not vary with the complexity or length of the performance to the same degree as when using traditional techniques. This allows many tests to be done with different styles or deliveries, giving a different personality only limited by the talent of the actor. | * The amount of work does not vary with the complexity or length of the performance to the same degree as when using traditional techniques. This allows many tests to be done with different styles or deliveries, giving a different personality only limited by the talent of the actor. | ||
* Complex movement and realistic physical interactions such as secondary | * Complex movement and realistic physical interactions such as [[secondary motion]]s, weight and exchange of forces can be easily recreated in a physically accurate manner.<ref>{{cite magazine|title=The Next Generation 1996 Lexicon A to Z: Motion Capture|magazine=[[Next Generation (magazine)|Next Generation]]|issue=15 |publisher=[[Imagine Media]]|date=March 1996|page=37}}</ref> | ||
* The amount of animation data that can be produced within a given time is extremely large when compared to traditional animation techniques. This contributes to both cost effectiveness and meeting production deadlines.<ref>{{cite journal|title=Motion Capture|journal=[[Next Generation (magazine)|Next Generation]]|issue=10|publisher=[[Imagine Media]]|date=October 1995|page=50}}</ref> | * The amount of animation data that can be produced within a given time is extremely large when compared to traditional animation techniques. This contributes to both cost-effectiveness and meeting production deadlines.<ref>{{cite journal|title=Motion Capture|journal=[[Next Generation (magazine)|Next Generation]]|issue=10|publisher=[[Imagine Media]]|date=October 1995|page=50}}</ref> | ||
*Potential for free software and third party solutions reducing its costs. | *Potential for free software and third-party solutions reducing its costs. | ||
==Disadvantages== | ==Disadvantages== | ||
| Line 28: | Line 31: | ||
* Movement that does not follow the laws of physics cannot be captured. | * Movement that does not follow the laws of physics cannot be captured. | ||
* Traditional animation techniques, such as added emphasis on anticipation and follow through, secondary motion or manipulating the shape of the character, as with [[squash and stretch]] animation techniques, must be added later. | * Traditional animation techniques, such as added emphasis on anticipation and follow through, secondary motion or manipulating the shape of the character, as with [[squash and stretch]] animation techniques, must be added later. | ||
* If the computer model has different proportions from the capture subject, artifacts may occur. For example, if a cartoon character has large, oversized hands, these may intersect the character's body if the human performer is not careful with | * If the computer model has different proportions from the capture subject, artifacts may occur. For example, if a cartoon character has large, oversized hands, these may intersect the character's body if the human performer is not careful with their physical motion. | ||
==Applications== | ==Applications== | ||
| Line 34: | Line 37: | ||
[[File:Motion Capture Performers.png|thumb|right|250px|Motion capture performers from Buckinghamshire New University]] | [[File:Motion Capture Performers.png|thumb|right|250px|Motion capture performers from Buckinghamshire New University]] | ||
[[Video games]] often use motion capture to animate athletes, [[martial artists]], and other in-game characters.<ref>Jon Radoff, Anatomy of an MMORPG, {{cite web |url=http://radoff.com/blog/2008/08/22/anatomy-of-an-mmorpg/ |title= | [[Video games]] often use motion capture to animate athletes, [[martial artists]], and other in-game characters.<ref>Jon Radoff, Anatomy of an MMORPG, {{cite web |url=http://radoff.com/blog/2008/08/22/anatomy-of-an-mmorpg/ |title=Anatomy of an MMORPG |access-date=2009-11-30 |url-status=dead |archive-url=https://web.archive.org/web/20091213053756/http://radoff.com/blog/2008/08/22/anatomy-of-an-mmorpg/ |archive-date=2009-12-13 }}</ref><ref name="GPro82">{{cite magazine|title=Hooray for Hollywood! Acclaim Studios|magazine=[[GamePro]]|issue=82|publisher=[[International Data Group|IDG]]|date=July 1995|pages=28–29}}</ref> As early as 1988, an early form of motion capture was used to animate the [[2D computer graphics|2D]] [[player characters]] of [[Martech]]'s video game ''[[Vixen (video game)|Vixen]]'' (performed by model [[Corinne Russell]])<ref>{{cite magazine|magazine=[[Retro Gamer]]|title=Martech Games - The Personality People|page=51|issue=133|first=Graeme|last=Mason|url=https://issuu.com/michelfranca/docs/retro_gamer____133}}</ref> and [[Magical Company]]'s 2D arcade [[fighting game]] ''Last Apostle Puppet Show'' (to animate digitized [[Sprite (computer graphics)|sprites]]).<ref>{{cite web |title=Pre-Street Fighter II Fighting Games |url=http://www.hardcoregaming101.net/fighters/fighters8.htm |website=Hardcore Gaming 101 |page=8 |access-date=26 November 2021}}</ref> Motion capture was later notably used to animate the [[3D computer graphics|3D]] character models in the [[Sega Model 1|Sega Model]] [[arcade games]] ''[[Virtua Fighter (video game)|Virtua Fighter]]'' (1993)<ref name="CVG158">{{cite magazine |url=https://retrocdn.net/images/8/84/CVG_UK_158.pdf#page=12 |title=Sega Saturn exclusive! Virtua Fighter: fighting in the third dimension |magazine=[[Computer and Video Games]] |publisher=[[Future plc]] |issue=158 (January 1995) |date=15 December 1994 |pages=12–3, 15–6, 19}}</ref><ref name="Maximum">{{cite journal|title=Virtua Fighter|journal=Maximum: The Video Game Magazine|issue=1|publisher=[[Emap International Limited]]|date=October 1995|pages=142–3}}</ref> and ''[[Virtua Fighter 2]]'' (1994).<ref>{{cite web|last=Wawro|first=Alex|title=Yu Suzuki Recalls Using Military Tech to Make Virtua Fighter 2 |url=http://www.gamasutra.com/view/news/228512/Yu_Suzuki_recalls_using_military_tech_to_make_Virtua_Fighter_2.php|website=[[Gamasutra]]|access-date=18 August 2016|date=October 23, 2014}}</ref> In mid-1995, developer/publisher [[Acclaim Entertainment]] had its own in-house motion capture studio built into its headquarters.<ref name="GPro82"/> [[Namco]]'s 1995 arcade game ''[[Soul Edge]]'' used passive optical system markers for motion capture.<ref>{{cite web |url=http://www.motioncapturesociety.com/resources/industry-history |title=History of Motion Capture |publisher=Motioncapturesociety.com |access-date=2013-08-10 |archive-url=https://web.archive.org/web/20181023162411/http://www.motioncapturesociety.com/resources/industry-history |archive-date=2018-10-23 |url-status=dead }}</ref> Motion capture also uses athletes in based-off animated games, such as [[Naughty Dog]]'s [[Crash Bandicoot (video game)|Crash Bandicoot]], [[Insomniac Games]]' [[Spyro the Dragon]], and [[Rare (company)|Rare]]'s [[Star Fox Adventures#Development|Dinosaur Planet]]. | ||
Movies use motion capture for | Movies use motion capture for CGI effects, in some cases replacing traditional cel animation, and for completely [[computer-generated imagery|CGI]] creatures, such as [[Gollum]], [[The Mummy (1999 film)|The Mummy]], [[Peter Jackson's King Kong|King Kong]], [[Davy Jones (Pirates of the Caribbean)|Davy Jones]] from ''[[Pirates of the Caribbean (film series)|Pirates of the Caribbean]]'', the [[Pandoran biosphere#Na'vi|Na'vi]] from the film [[Avatar (2009 film)|''Avatar'']], and Clu from ''[[Tron: Legacy]]''. The Great Goblin, the three [[Troll (Middle-earth)#Troll types|Stone-trolls]], many of the orcs and goblins in the 2012 film ''[[The Hobbit: An Unexpected Journey]]'', and [[Smaug]] were created using motion capture. | ||
The film ''[[Batman Forever]]'' (1995) used some motion capture for certain special effects. [[Warner Bros]] had acquired motion capture technology from [[arcade video game]] company Acclaim Entertainment for use in the film's production.<ref>{{cite magazine |title=Coin-Op News: Acclaim technology tapped for "Batman" movie |magazine=[[Play Meter]] |date=October 1994 |volume=20 |issue=11 |page=22 |url=https://archive.org/details/play-meter-volume-20-number-11-october-1994/Play%20Meter%20-%20Volume%2020%2C%20Number%2011%20-%20October%201994/page/22}}</ref> Acclaim's 1995 [[Batman Forever (video game)|video game of the same name]] also used the same motion capture technology to animate the digitized [[Sprite (computer graphics)|sprite]] graphics.<ref>{{cite magazine |title=Acclaim Stakes its Claim |magazine=RePlay |date=January 1995 |volume=20 |issue=4 |page=71 |url=https://archive.org/details/re-play-volume-20-issue-no.-4-january-1995/RePlay%20-%20Volume%2020%2C%20Issue%20No.%204%20-%20January%201995/page/n68}}</ref> | The film ''[[Batman Forever]]'' (1995) used some motion capture for certain special effects. [[Warner Bros.]] had acquired motion capture technology from [[arcade video game]] company Acclaim Entertainment for use in the film's production.<ref>{{cite magazine |title=Coin-Op News: Acclaim technology tapped for "Batman" movie |magazine=[[Play Meter]] |date=October 1994 |volume=20 |issue=11 |page=22 |url=https://archive.org/details/play-meter-volume-20-number-11-october-1994/Play%20Meter%20-%20Volume%2020%2C%20Number%2011%20-%20October%201994/page/22}}</ref> Acclaim's 1995 [[Batman Forever (video game)|video game of the same name]] also used the same motion capture technology to animate the digitized [[Sprite (computer graphics)|sprite]] graphics.<ref>{{cite magazine |title=Acclaim Stakes its Claim |magazine=RePlay |date=January 1995 |volume=20 |issue=4 |page=71 |url=https://archive.org/details/re-play-volume-20-issue-no.-4-january-1995/RePlay%20-%20Volume%2020%2C%20Issue%20No.%204%20-%20January%201995/page/n68}}</ref> | ||
''[[Star Wars: Episode I – The Phantom Menace]]'' (1999) was the first feature-length film to include a main character created using motion capture (that character being [[Jar Jar Binks]], played by [[Ahmed Best]]), and [[India]]n-[[United States|American]] film ''[[Sinbad: Beyond the Veil of Mists]]'' (2000) was the first feature-length film made primarily with motion capture, although many character animators also worked on the film, which had a very limited release. 2001's ''[[Final Fantasy: The Spirits Within]]'' was the first widely released movie to be made primarily with motion capture technology. Despite its poor box-office intake, supporters of motion capture technology took notice. ''[[Total Recall (1990 film)|Total Recall]]'' had already used the technique, in the scene of the x-ray scanner and the skeletons. | ''[[Star Wars: Episode I – The Phantom Menace]]'' (1999) was the first feature-length film to include a main character created using motion capture (that character being [[Jar Jar Binks]], played by [[Ahmed Best]]), and [[India]]n-[[United States|American]] film ''[[Sinbad: Beyond the Veil of Mists]]'' (2000) was the first feature-length film made primarily with motion capture, although many character animators also worked on the film, which had a very limited release. 2001's ''[[Final Fantasy: The Spirits Within]]'' was the first widely released movie to be made primarily with motion capture technology. Despite its poor box-office intake, supporters of motion capture technology took notice. ''[[Total Recall (1990 film)|Total Recall]]'' had already used the technique, in the scene of the x-ray scanner and the skeletons. | ||
''[[The Lord of the Rings: The Two Towers]]'' was the first feature film to utilize a real-time motion capture system. This method streamed the actions of actor [[Andy Serkis]] into the computer generated skin of Gollum / Smeagol as it was being performed.<ref>{{cite magazine|last1=Savage|first1=Annaliza|title=Gollum Actor: How New Motion-Capture Tech Improved The Hobbit|url=https://www.wired.com/2012/12/andy-serkis-interview/|magazine=[[Wired (website)|Wired]]|access-date=29 January 2017|date=12 July 2012}}</ref> | ''[[The Lord of the Rings: The Two Towers]]'' was the first feature film to utilize a real-time motion capture system. This method streamed the actions of actor [[Andy Serkis]] into the computer-generated skin of Gollum / Smeagol as it was being performed.<ref>{{cite magazine|last1=Savage|first1=Annaliza|title=Gollum Actor: How New Motion-Capture Tech Improved The Hobbit|url=https://www.wired.com/2012/12/andy-serkis-interview/|magazine=[[Wired (website)|Wired]]|access-date=29 January 2017|date=12 July 2012}}</ref> | ||
Storymind Entertainment, which is an independent [[Ukrainians|Ukrainian]] studio, created a [[neo-noir]] [[Third-person shooter|third-person]] / shooter video game called ''[[My Eyes On You (video game)|My Eyes On You]],'' using motion capture in order to animate its main character, Jordan Adalien, and along with non-playable characters.<ref>{{Cite web |title=INTERVIEW: Storymind Entertainment Talks About Upcoming 'My Eyes On You' |url=https://www.thatmomentin.com/my-eyes-on-you/ |access-date=2022-09-24 |website=That Moment In |language=en-US}}</ref> | |||
Out of the three nominees for the 2006 [[Academy Award for Best Animated Feature]], two of the nominees (''[[Monster House (film)|Monster House]]'' and the winner ''[[Happy Feet]]'') used motion capture, and only [[Walt Disney Pictures|Disney]]'''·'''[[Pixar]]'s ''[[Cars (film)|Cars]]'' was animated without motion capture. In the ending credits of [[Pixar]]'s film ''[[Ratatouille (film)|Ratatouille]]'', a stamp appears labelling the film as "100% Pure Animation – No Motion Capture!" | Out of the three nominees for the 2006 [[Academy Award for Best Animated Feature]], two of the nominees (''[[Monster House (film)|Monster House]]'' and the winner ''[[Happy Feet]]'') used motion capture, and only [[Walt Disney Pictures|Disney]]'''·'''[[Pixar]]'s ''[[Cars (film)|Cars]]'' was animated without motion capture. In the ending credits of [[Pixar]]'s film ''[[Ratatouille (film)|Ratatouille]]'', a stamp appears labelling the film as "100% Pure Animation – No Motion Capture!" | ||
Since 2001, motion capture | Since 2001, motion capture has been used extensively to simulate or approximate the look of live-action cinema, with nearly [[Photorealism|photorealistic]] digital character models. ''[[The Polar Express (film)|The Polar Express]]'' used motion capture to allow [[Tom Hanks]] to perform as several distinct digital characters (in which he also provided the voices). The 2007 adaptation of the saga ''[[Beowulf (2007 film)|Beowulf]]'' animated digital characters whose appearances were based in part on the actors who provided their motions and voices. James Cameron's highly popular ''[[Avatar (2009 film)|Avatar]]'' used this technique to create the Na'vi that inhabit Pandora. [[The Walt Disney Company]] has produced [[Robert Zemeckis]]'s ''[[A Christmas Carol (2009 film)|A Christmas Carol]]'' using this technique. In 2007, Disney acquired Zemeckis' [[ImageMovers Digital]] (that produces motion capture films), but then closed it in 2011, after a [[box office failure]] of ''[[Mars Needs Moms]]''. | ||
Television series produced entirely with motion capture animation include ''[[Et Dieu créa... Laflaque|Laflaque]]'' in Canada, ''[[Sprookjesboom]]'' and ''{{ill|Cafe de Wereld|nl|Cafe de Wereld|vertical-align=sup}}'' in The Netherlands, and ''[[Headcases]]'' in the UK. | Television series produced entirely with motion capture animation include ''[[Et Dieu créa... Laflaque|Laflaque]]'' in Canada, ''[[Sprookjesboom]]'' and ''{{ill|Cafe de Wereld|nl|Cafe de Wereld|vertical-align=sup}}'' in The Netherlands, and ''[[Headcases]]'' in the UK. | ||
[[Virtual reality]] and [[Augmented reality]] providers, such as [[uSens]] and [[Gestigon]], allow users to interact with digital content in real time by capturing hand motions. This can be useful for training simulations, visual perception tests, or performing | [[Virtual reality]] and [[Augmented reality]] providers, such as [[uSens]] and [[Gestigon]], allow users to interact with digital content in real time by capturing hand motions. This can be useful for training simulations, visual perception tests, or performing virtual walk-throughs in a 3D environment. Motion capture technology is frequently used in [[digital puppetry]] systems to drive computer-generated characters in real time. | ||
[[Gait analysis]] is one application of motion capture in [[clinical medicine]]. Techniques allow clinicians to evaluate human motion across several biomechanical factors, often while streaming this information live into analytical software. | [[Gait analysis]] is one application of motion capture in [[clinical medicine]]. Techniques allow clinicians to evaluate human motion across several biomechanical factors, often while streaming this information live into analytical software. | ||
| Line 56: | Line 61: | ||
Some physical therapy clinics utilize motion capture as an objective way to quantify patient progress.<ref>{{Cite web|url=https://www.eumotus.com|title=Markerless Motion Capture {{!}} EuMotus|website=Markerless Motion Capture {{!}} EuMotus|language=en|access-date=2018-10-12}}</ref> | Some physical therapy clinics utilize motion capture as an objective way to quantify patient progress.<ref>{{Cite web|url=https://www.eumotus.com|title=Markerless Motion Capture {{!}} EuMotus|website=Markerless Motion Capture {{!}} EuMotus|language=en|access-date=2018-10-12}}</ref> | ||
During the filming of James Cameron's [[Avatar (2009 film)|''Avatar'']] all of the scenes involving this process were directed in | During the filming of James Cameron's [[Avatar (2009 film)|''Avatar'']] all of the scenes involving this process were directed in real-time using [[Autodesk MotionBuilder]] software to render a screen image which allowed the director and the actor to see what they would look like in the movie, making it easier to direct the movie as it would be seen by the viewer. This method allowed views and angles not possible from a pre-rendered animation. Cameron was so proud of his results that he invited [[Steven Spielberg]] and [[George Lucas]] on set to view the system in action. | ||
In Marvel's ''[[The Avengers (2012 film)|The Avengers]]'', Mark Ruffalo used motion capture so he could play his character [[Bruce Banner (Marvel Cinematic Universe)|the Hulk]], rather than have him be only CGI as in previous films, making Ruffalo the first actor to play both the human and the Hulk versions of Bruce Banner. | In Marvel's ''[[The Avengers (2012 film)|The Avengers]]'', Mark Ruffalo used motion capture so he could play his character [[Bruce Banner (Marvel Cinematic Universe)|the Hulk]], rather than have him be only CGI as in previous films, making Ruffalo the first actor to play both the human and the Hulk versions of Bruce Banner. | ||
[[FaceRig]] software uses facial recognition technology from ULSee.Inc to map a player's facial expressions and the body tracking technology from Perception Neuron to map the body movement onto a 3D | [[FaceRig]] software uses facial recognition technology from ULSee.Inc to map a player's facial expressions and the body tracking technology from Perception Neuron to map the body movement onto a 2D or 3D character's motion on-screen.<ref>{{cite web|url=http://www.polygon.com/2014/6/30/5858610/this-facial-recognition-software-lets-you-be-octodad|title=This facial recognition software lets you be Octodad|first=Alexa Ray|last=Corriea|date=30 June 2014|access-date=4 January 2017|via=www.polygon.com}}</ref><ref>{{cite web|url=http://kotaku.com/turn-your-human-face-into-a-video-game-character-1490049650|title=Turn Your Human Face Into A Video Game Character|first=Luke|last=Plunkett|work=kotaku.com|date=27 December 2013 |access-date=4 January 2017}}</ref> | ||
During ''[[Game Developers Conference]]'' 2016 in San Francisco ''[[Epic Games]]'' demonstrated full-body motion capture live in Unreal Engine. The whole scene, from the upcoming game ''[[Hellblade: Senua's Sacrifice|Hellblade]]'' about a woman warrior named Senua, was rendered in real-time. The keynote<ref>{{cite web|url=https://www.fxguide.com/featured/put-your-digital-game-face-on/|title=Put your (digital) game face on|date=24 April 2016|work=fxguide.com|access-date=4 January 2017}}</ref> was a collaboration between ''[[Unreal Engine]]'', ''[[Ninja Theory]]'', ''[[3Lateral]]'', ''Cubic Motion'', ''IKinema'' and ''[[Xsens]]''. | During ''[[Game Developers Conference]]'' 2016 in San Francisco ''[[Epic Games]]'' demonstrated full-body motion capture live in Unreal Engine. The whole scene, from the upcoming game ''[[Hellblade: Senua's Sacrifice|Hellblade]]'' about a woman warrior named Senua, was rendered in real-time. The keynote<ref>{{cite web|url=https://www.fxguide.com/featured/put-your-digital-game-face-on/|title=Put your (digital) game face on|date=24 April 2016|work=fxguide.com|access-date=4 January 2017}}</ref> was a collaboration between ''[[Unreal Engine]]'', ''[[Ninja Theory]]'', ''[[3Lateral]]'', ''Cubic Motion'', ''IKinema'' and ''[[Xsens]]''. | ||
==Methods and systems== | ==Methods and systems== | ||
| Line 70: | Line 73: | ||
[[File:Silhouette tracking.PNG|thumb|Silhouette tracking]] | [[File:Silhouette tracking.PNG|thumb|Silhouette tracking]] | ||
Motion tracking or motion capture started as a photogrammetric analysis tool in biomechanics research in the 1970s and 1980s, and expanded into education, training, sports and recently [[computer animation]] for [[television]], [[film|cinema]], and [[video game]]s as the technology matured. Since the 20th century the performer has to wear markers near each joint to identify the motion by the positions or angles between the markers. Acoustic, inertial, [[LED]], magnetic or reflective markers, or combinations of any of these, are tracked, optimally at least two times the frequency rate of the desired motion. The resolution of the system is important in both the spatial resolution and temporal resolution as motion blur causes almost the same problems as low resolution. Since the beginning of the 21st century and because of the rapid growth of technology new methods | Motion tracking or motion capture started as a photogrammetric analysis tool in biomechanics research in the 1970s and 1980s, and expanded into education, training, sports and recently [[computer animation]] for [[television]], [[film|cinema]], and [[video game]]s as the technology matured. Since the 20th century, the performer has to wear markers near each joint to identify the motion by the positions or angles between the markers. Acoustic, inertial, [[LED]], magnetic or reflective markers, or combinations of any of these, are tracked, optimally at least two times the frequency rate of the desired motion. The resolution of the system is important in both the spatial resolution and temporal resolution as motion blur causes almost the same problems as low resolution. Since the beginning of the 21st century - and because of the rapid growth of technology - new methods have been developed. Most modern systems can extract the silhouette of the performer from the background. Afterwards all joint angles are calculated by fitting in a mathematical model into the silhouette. For movements you can not see a change of the silhouette, there are hybrid systems available that can do both (marker and silhouette), but with less marker.{{citation needed|date=January 2019}} In robotics, some motion capture systems are based on [[simultaneous localization and mapping]].<ref>Sturm, Jürgen, et al. "[http://jsturm.de/publications/data/sturm12iros.pdf A benchmark for the evaluation of RGB-D SLAM systems]." Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, 2012.</ref> | ||
{{Further|Left-hand–right-hand activity chart|Kinematics}} | |||
==Optical systems== | ==Optical systems== | ||
''Optical systems'' utilize data captured from image sensors to [[triangulation (computer vision)|triangulate]] the 3D position of a subject between two or more cameras calibrated to provide overlapping projections. Data acquisition is traditionally implemented using special markers attached to an actor; however, more recent systems are able to generate accurate data by tracking surface features identified dynamically for each particular subject. Tracking a large number of performers or expanding the capture area is accomplished by the addition of more cameras. These systems produce data with three degrees of freedom for each marker, and rotational information must be inferred from the relative orientation of three or more markers; for instance shoulder, elbow and wrist markers providing the angle of the elbow. Newer hybrid systems are combining inertial sensors with optical sensors to reduce occlusion, increase the number of users and improve the ability to track without having to manually clean up data{{Citation needed|date=January 2017}} | ''Optical systems'' utilize data captured from image sensors to [[triangulation (computer vision)|triangulate]] the 3D position of a subject between two or more cameras calibrated to provide overlapping projections. Data acquisition is traditionally implemented using special markers attached to an actor; however, more recent systems are able to generate accurate data by tracking surface features identified dynamically for each particular subject. Tracking a large number of performers or expanding the capture area is accomplished by the addition of more cameras. These systems produce data with three degrees of freedom for each marker, and rotational information must be inferred from the relative orientation of three or more markers; for instance shoulder, elbow and wrist markers providing the angle of the elbow. Newer hybrid systems are combining inertial sensors with optical sensors to reduce occlusion, increase the number of users and improve the ability to track without having to manually clean up data.{{Citation needed|date=January 2017}} | ||
===Passive markers=== | ===Passive markers=== | ||
[[Image:MotionCapture.jpg|thumb|A dancer wearing a suit used in an optical motion capture system]] | [[Image:MotionCapture.jpg|thumb|A dancer wearing a suit used in an optical motion capture system]] | ||
[[File:Motion capture facial.jpg|thumb|Markers are placed at specific points on an actor's face during facial optical motion capture.]] | [[File:Motion capture facial.jpg|thumb|Markers are placed at specific points on an actor's face during facial optical motion capture.]] | ||
''Passive optical'' systems use markers coated with a [[retroreflective]] material to reflect light that is generated near the | ''Passive optical'' systems use markers coated with a [[retroreflective]] material to reflect light that is generated near the camera's lens. The camera's threshold can be adjusted so only the bright reflective markers will be sampled, ignoring skin and fabric. | ||
The centroid of the marker is estimated as a position within the two-dimensional image that is captured. The grayscale value of each pixel can be used to provide sub-pixel accuracy by finding the centroid of the [[Gaussian]]. | The centroid of the marker is estimated as a position within the two-dimensional image that is captured. The grayscale value of each pixel can be used to provide sub-pixel accuracy by finding the centroid of the [[Gaussian]]. | ||
An object with markers attached at known positions is used to calibrate the cameras and obtain their positions and the lens distortion of each camera is measured. If two calibrated cameras see a marker, a three-dimensional fix can be obtained. Typically a system will consist of around 2 to 48 cameras. Systems of over three hundred cameras exist to try to reduce marker swap. Extra cameras are required for full coverage around the capture subject and multiple subjects. | An object with markers attached at known positions is used to calibrate the cameras and obtain their positions, and the lens distortion of each camera is measured. If two calibrated cameras see a marker, a three-dimensional fix can be obtained. Typically a system will consist of around 2 to 48 cameras. Systems of over three hundred cameras exist to try to reduce marker swap. Extra cameras are required for full coverage around the capture subject and multiple subjects. | ||
Vendors have constraint software to reduce the problem of marker swapping since all passive markers appear identical. Unlike active marker systems and magnetic systems, passive systems do not require the user to wear wires or electronic equipment.<ref>{{cite journal|title=Motion Capture: Optical Systems|journal=[[Next Generation (magazine)|Next Generation]]|issue=10|publisher=[[Imagine Media]]|date=October 1995|page=53}}</ref> Instead, hundreds of rubber balls are attached with reflective tape, which needs to be replaced periodically. The markers are usually attached directly to the skin (as in biomechanics), or they are [[velcro]]ed to a performer wearing a full-body spandex/lycra [[Mo-cap suit|suit designed specifically for motion capture]]. This type of system can capture large numbers of markers at frame rates usually around 120 to 160 fps although by lowering the resolution and tracking a smaller region of interest they can track as high as 10,000 fps. | Vendors have constraint software to reduce the problem of marker swapping since all passive markers appear identical. Unlike active marker systems and magnetic systems, passive systems do not require the user to wear wires or electronic equipment.<ref>{{cite journal|title=Motion Capture: Optical Systems|journal=[[Next Generation (magazine)|Next Generation]]|issue=10|publisher=[[Imagine Media]]|date=October 1995|page=53}}</ref> Instead, hundreds of rubber balls are attached with reflective tape, which needs to be replaced periodically. The markers are usually attached directly to the skin (as in biomechanics), or they are [[velcro]]ed to a performer wearing a full-body spandex/lycra [[Mo-cap suit|suit designed specifically for motion capture]]. This type of system can capture large numbers of markers at frame rates usually around 120 to 160 fps although by lowering the resolution and tracking a smaller region of interest they can track as high as 10,000 fps. | ||
===Active marker=== | ===Active marker=== | ||
[[File:Body_Motion_Capture.jpg|thumb| | [[File:Body_Motion_Capture.jpg|thumb|Body motion capture]] | ||
Active optical systems triangulate positions by illuminating one LED at a time very quickly or multiple LEDs with software to identify them by their relative positions, somewhat akin to celestial navigation. Rather than reflecting light back that is generated externally, the markers themselves are powered to emit their own light. Since inverse square law provides one quarter the power at two times the distance, this can increase the distances and volume for capture. This also enables high signal-to-noise ratio, resulting in very low marker jitter and a resulting high measurement resolution (often down to 0.1 mm within the calibrated volume). | Active optical systems triangulate positions by illuminating one LED at a time very quickly or multiple LEDs with software to identify them by their relative positions, somewhat akin to celestial navigation. Rather than reflecting light back that is generated externally, the markers themselves are powered to emit their own light. Since the inverse square law provides one quarter of the power at two times the distance, this can increase the distances and volume for capture. This also enables a high signal-to-noise ratio, resulting in very low marker jitter and a resulting high measurement resolution (often down to 0.1 mm within the calibrated volume). | ||
The TV series ''[[Stargate SG1]]'' produced episodes using an active optical system for the VFX allowing the actor to walk around props that would make motion capture difficult for other non-active optical systems.{{citation needed|date=August 2016}} | The TV series ''[[Stargate SG1]]'' produced episodes using an active optical system for the VFX allowing the actor to walk around props that would make motion capture difficult for other non-active optical systems.{{citation needed|date=August 2016}} | ||
ILM used active markers in ''[[Van Helsing (film)|Van Helsing]]'' to allow capture of Dracula's flying brides on very large sets similar to Weta's use of active markers in ''[[Rise of the Planet of the Apes]]''. The power to each marker can be provided sequentially in phase with the capture system providing a unique identification of each marker for a given capture frame at a cost to the resultant frame rate. The ability to identify each marker in this manner is useful in | ILM used active markers in ''[[Van Helsing (film)|Van Helsing]]'' to allow capture of Dracula's flying brides on very large sets similar to Weta's use of active markers in ''[[Rise of the Planet of the Apes]]''. The power to each marker can be provided sequentially in phase with the capture system providing a unique identification of each marker for a given capture frame at a cost to the resultant frame rate. The ability to identify each marker in this manner is useful in real-time applications. The alternative method of identifying markers is to do it algorithmically requiring extra processing of the data. | ||
There are also possibilities to find the position by using | There are also possibilities to find the position by using colored LED markers. In these systems, each color is assigned to a specific point of the body. | ||
One of the earliest active marker systems in the 1980s was a hybrid passive-active mocap system with rotating mirrors and colored glass reflective markers and which used masked linear array detectors. | One of the earliest active marker systems in the 1980s was a hybrid passive-active mocap system with rotating mirrors and colored glass reflective markers and which used masked linear array detectors. | ||
| Line 100: | Line 104: | ||
===Time modulated active marker=== | ===Time modulated active marker=== | ||
[[Image:Activemarker2.PNG|thumb|300px|A high-resolution uniquely identified active marker system with 3,600 × 3,600 resolution at 960 hertz providing real time submillimeter positions]] | [[Image:Activemarker2.PNG|thumb|300px|A high-resolution uniquely identified active marker system with 3,600 × 3,600 resolution at 960 hertz providing real time submillimeter positions]] | ||
Active marker systems can further be refined by strobing one marker on at a time, or tracking multiple markers over time and modulating the amplitude or pulse width to provide marker ID. 12 megapixel spatial resolution modulated systems show more subtle movements than 4 megapixel optical systems by having both higher spatial and temporal resolution. Directors can see the | Active marker systems can further be refined by strobing one marker on at a time, or tracking multiple markers over time and modulating the amplitude or pulse width to provide marker ID. 12-megapixel spatial resolution modulated systems show more subtle movements than 4-megapixel optical systems by having both higher spatial and temporal resolution. Directors can see the actor's performance in real-time, and watch the results on the motion capture-driven CG character. The unique marker IDs reduce the turnaround, by eliminating marker swapping and providing much cleaner data than other technologies. LEDs with onboard processing and radio synchronization allow motion capture outdoors in direct sunlight while capturing at 120 to 960 frames per second due to a high-speed electronic shutter. Computer processing of modulated IDs allows less hand cleanup or filtered results for lower operational costs. This higher accuracy and resolution requires more processing than passive technologies, but the additional processing is done at the camera to improve resolution via subpixel or centroid processing, providing both high resolution and high speed. These motion capture systems typically cost $20,000 for an eight-camera, 12-megapixel spatial resolution 120-hertz system with one actor. | ||
[[Image:PrakashOutdoorMotionCapture.jpg|thumb|300px| [[Infrared|IR]] sensors can compute their location when lit by mobile multi-LED emitters, e.g. in a moving car. With Id per marker, these sensor tags can be worn under clothing and tracked at 500 Hz in broad daylight.]] | [[Image:PrakashOutdoorMotionCapture.jpg|thumb|300px| [[Infrared|IR]] sensors can compute their location when lit by mobile multi-LED emitters, e.g. in a moving car. With Id per marker, these sensor tags can be worn under clothing and tracked at 500 Hz in broad daylight.]] | ||
===Semi-passive imperceptible marker=== | ===Semi-passive imperceptible marker=== | ||
One can reverse the traditional approach based on high speed cameras. Systems such as [http://web.media.mit.edu/~raskar/LumiNetra/ Prakash] use inexpensive multi-LED high speed projectors. The specially built multi-LED IR projectors optically encode the space. Instead of retro-reflective or active light emitting diode (LED) markers, the system uses photosensitive marker tags to decode the optical signals. By attaching tags with photo sensors to scene points, the tags can compute not only their own locations of each point, but also their own orientation, incident illumination, and reflectance. | One can reverse the traditional approach based on high-speed cameras. Systems such as [http://web.media.mit.edu/~raskar/LumiNetra/ Prakash] use inexpensive multi-LED high-speed projectors. The specially built multi-LED IR projectors optically encode the space. Instead of retro-reflective or active light emitting diode (LED) markers, the system uses photosensitive marker tags to decode the optical signals. By attaching tags with photo sensors to scene points, the tags can compute not only their own locations of each point, but also their own orientation, incident illumination, and reflectance. | ||
These tracking tags work in natural lighting conditions and can be imperceptibly embedded in attire or other objects. The system supports an unlimited number of tags in a scene, with each tag uniquely identified to eliminate marker reacquisition issues. Since the system eliminates a high speed camera and the corresponding high-speed image stream, it requires significantly lower data bandwidth. The tags also provide incident illumination data which can be used to match scene lighting when inserting synthetic elements. The technique appears ideal for on-set motion capture or real-time broadcasting of virtual sets but has yet to be proven. | These tracking tags work in natural lighting conditions and can be imperceptibly embedded in attire or other objects. The system supports an unlimited number of tags in a scene, with each tag uniquely identified to eliminate marker reacquisition issues. Since the system eliminates a high-speed camera and the corresponding high-speed image stream, it requires significantly lower data bandwidth. The tags also provide incident illumination data which can be used to match scene lighting when inserting synthetic elements. The technique appears ideal for on-set motion capture or real-time broadcasting of virtual sets but has yet to be proven. | ||
===Underwater motion capture system=== | ===Underwater motion capture system=== | ||
| Line 114: | Line 118: | ||
====Underwater cameras==== | ====Underwater cameras==== | ||
The vital part of the system, the underwater camera, has a waterproof housing. The housing has a finish that withstands corrosion and chlorine which makes it perfect for use in basins and swimming pools. There are two types of cameras. Industrial high-speed | The vital part of the system, the underwater camera, has a waterproof housing. The housing has a finish that withstands corrosion and chlorine which makes it perfect for use in basins and swimming pools. There are two types of cameras. Industrial high-speed cameras can also be used as infrared cameras. Infrared underwater cameras come with a cyan light strobe instead of the typical IR light for minimum fall-off underwater and high-speed cameras with an LED light or with the option of using image processing. [[File:Oqus underwater.jpg|thumb|Underwater motion capture camera]] | ||
[[File:Motion tacking by using image processing.PNG|thumb|Motion tracking in swimming by using image processing]] | [[File:Motion tacking by using image processing.PNG|thumb|Motion tracking in swimming by using image processing]] | ||
| Line 123: | Line 127: | ||
=====Tailored===== | =====Tailored===== | ||
Different pools require different mountings and fixtures. Therefore, all underwater motion capture systems are uniquely tailored to suit each specific pool | Different pools require different mountings and fixtures. Therefore, all underwater motion capture systems are uniquely tailored to suit each specific pool instalment. For cameras placed in the center of the pool, specially designed tripods, using suction cups, are provided. | ||
===Markerless=== | ===Markerless=== | ||
Emerging techniques and research in [[computer vision]] are leading to the rapid development of the markerless approach to motion capture. Markerless systems such as those developed at [[Stanford University]], the [[University of Maryland]], [[MIT]], and the [[Max Planck Institute]], do not require subjects to wear special equipment for tracking. Special computer algorithms are designed to allow the system to analyze multiple streams of optical input and identify human forms, breaking them down into constituent parts for tracking. [[ESC entertainment]], a subsidiary of [[Warner Brothers Pictures]] created | Emerging techniques and research in [[computer vision]] are leading to the rapid development of the markerless approach to motion capture. Markerless systems such as those developed at [[Stanford University]], the [[University of Maryland]], [[MIT]], and the [[Max Planck Institute]], do not require subjects to wear special equipment for tracking. Special computer algorithms are designed to allow the system to analyze multiple streams of optical input and identify human forms, breaking them down into constituent parts for tracking. [[ESC entertainment]], a subsidiary of [[Warner Brothers Pictures]] created especially to enable [[virtual cinematography]], including [[photorealistic]] [[digital look-alike]]s for filming ''[[The Matrix Reloaded]]'' and ''[[The Matrix Revolutions]]'' movies, used a technique called Universal Capture that utilized [[multi-camera setup|7 camera setup]] and the tracking the [[optical flow]] of all [[pixel]]s over all the 2-D planes of the cameras for motion, [[gesture]] and [[facial expression]] capture leading to photorealistic results. | ||
====Traditional systems==== | ====Traditional systems==== | ||
Traditionally markerless optical motion tracking is used to keep track | Traditionally markerless optical motion tracking is used to keep track of various objects, including airplanes, launch vehicles, missiles and satellites. Many such optical motion tracking applications occur outdoors, requiring differing lens and camera configurations. High-resolution images of the target being tracked can thereby provide more information than just motion data. The image obtained from NASA's long-range tracking system on the space shuttle Challenger's fatal launch provided crucial evidence about the cause of the accident. Optical tracking systems are also used to identify known spacecraft and space debris despite the fact that it has a disadvantage compared to radar in that the objects must be reflecting or emitting sufficient light.<ref>{{Cite journal| doi = 10.1007/BF00216781| title = Optical tracking of artificial satellites| year = 1963| last1 = Veis | first1 = G.| journal = Space Science Reviews| volume = 2| issue = 2| pages = 250–296| bibcode=1963SSRv....2..250V| s2cid = 121533715}}</ref> | ||
An optical tracking system typically consists of three subsystems: the optical imaging system, the mechanical tracking platform and the tracking computer. | An optical tracking system typically consists of three subsystems: the optical imaging system, the mechanical tracking platform and the tracking computer. | ||
The optical imaging system is responsible for converting the light from the target area into digital image that the tracking computer can process. Depending on the design of the optical tracking system, the optical imaging system can vary from as simple as a standard digital camera to as specialized as an astronomical telescope on the top of a mountain. The specification of the optical imaging system determines the upper | The optical imaging system is responsible for converting the light from the target area into a digital image that the tracking computer can process. Depending on the design of the optical tracking system, the optical imaging system can vary from as simple as a standard digital camera to as specialized as an astronomical telescope on the top of a mountain. The specification of the optical imaging system determines the upper limit of the effective range of the tracking system. | ||
The mechanical tracking platform holds the optical imaging system and is responsible for manipulating the optical imaging system in such a way that it always points to the target being tracked. The dynamics of the mechanical tracking platform combined with the optical imaging system determines the tracking system's ability to keep the lock on a target that changes speed rapidly. | The mechanical tracking platform holds the optical imaging system and is responsible for manipulating the optical imaging system in such a way that it always points to the target being tracked. The dynamics of the mechanical tracking platform combined with the optical imaging system determines the tracking system's ability to keep the lock on a target that changes speed rapidly. | ||
The tracking computer is responsible for capturing the images from the optical imaging system, analyzing the image to extract target position and controlling the mechanical tracking platform to follow the target. There are several challenges. First the tracking computer has to be able to capture the image at a relatively high frame rate. This posts a requirement on the bandwidth of the image capturing hardware. The second challenge is that the image processing software has to be able to extract the target image from its background and calculate its position. Several textbook image processing algorithms are designed for this task. This problem can be simplified if the tracking system can expect certain characteristics that is common in all the targets it will track. The next problem down the line is | The tracking computer is responsible for capturing the images from the optical imaging system, analyzing the image to extract the target position and controlling the mechanical tracking platform to follow the target. There are several challenges. First, the tracking computer has to be able to capture the image at a relatively high frame rate. This posts a requirement on the bandwidth of the image-capturing hardware. The second challenge is that the image processing software has to be able to extract the target image from its background and calculate its position. Several textbook image-processing algorithms are designed for this task. This problem can be simplified if the tracking system can expect certain characteristics that is common in all the targets it will track. The next problem down the line is controlling the tracking platform to follow the target. This is a typical control system design problem rather than a challenge, which involves modeling the system dynamics and designing [[motion controller|controllers]] to control it. This will however become a challenge if the tracking platform the system has to work with is not designed for real-time. | ||
The software that runs such systems | The software that runs such systems is also customized for the corresponding hardware components. One example of such software is OpticTracker, which controls computerized telescopes to track moving objects at great distances, such as planes and satellites. Another option is the software SimiShape, which can also be used hybrid in combination with markers. | ||
====RGB-D | ====RGB-D cameras==== | ||
RGB-D cameras such as [[kinect]] | RGB-D cameras such as [[kinect]] capture both the color and depth images. By fusing the two images, 3D colored [[voxel]] can be captured, allowing motion capture of 3D human motion and human surface in real-time. | ||
Because of the use of a single-view camera, motions captured are usually noisy. Machine learning techniques have been proposed to automatically reconstruct such noisy motions into higher quality ones, using methods such as [[lazy learning]]<ref>{{cite journal |last1=Shum |first1=Hubert P. H. |last2=Ho |first2=Edmond S. L. |last3=Jiang |first3=Yang |last4=Takagi |first4=Shu |title=Real-Time Posture Reconstruction for Microsoft Kinect |journal=IEEE Transactions on Cybernetics |date=2013 |volume=43 |issue=5 |pages=1357–1369 |doi=10.1109/TCYB.2013.2275945|pmid=23981562 |s2cid=14124193 }}</ref> and [[Gaussian]] models.<ref>{{cite journal |last1=Liu |first1=Zhiguang |last2=Zhou |first2=Liuyang |last3=Leung |first3=Howard |last4=Shum |first4=Hubert P. H. |title=Kinect Posture Reconstruction based on a Local Mixture of Gaussian Process Models |journal=IEEE Transactions on Visualization and Computer Graphics |date=2016 |volume=22 |issue=11 |pages=2437–2450 |doi=10.1109/TVCG.2015.2510000|pmid=26701789 |s2cid=216076607 |url=http://nrl.northumbria.ac.uk/id/eprint/25559/1/07360215.pdf }}</ref> Such method | Because of the use of a single-view camera, motions captured are usually noisy. Machine learning techniques have been proposed to automatically reconstruct such noisy motions into higher quality ones, using methods such as [[lazy learning]]<ref>{{cite journal |last1=Shum |first1=Hubert P. H. |last2=Ho |first2=Edmond S. L. |last3=Jiang |first3=Yang |last4=Takagi |first4=Shu |title=Real-Time Posture Reconstruction for Microsoft Kinect |journal=IEEE Transactions on Cybernetics |date=2013 |volume=43 |issue=5 |pages=1357–1369 |doi=10.1109/TCYB.2013.2275945|pmid=23981562 |s2cid=14124193 }}</ref> and [[Gaussian]] models.<ref>{{cite journal |last1=Liu |first1=Zhiguang |last2=Zhou |first2=Liuyang |last3=Leung |first3=Howard |last4=Shum |first4=Hubert P. H. |title=Kinect Posture Reconstruction based on a Local Mixture of Gaussian Process Models |journal=IEEE Transactions on Visualization and Computer Graphics |date=2016 |volume=22 |issue=11 |pages=2437–2450 |doi=10.1109/TVCG.2015.2510000|pmid=26701789 |s2cid=216076607 |url=http://nrl.northumbria.ac.uk/id/eprint/25559/1/07360215.pdf }}</ref> Such method generates accurate enough motion for serious applications like ergonomic assessment.<ref>{{cite journal |last1=Plantard |first1=Pierre |last2=Shum |first2=Hubert P. H. |last3=Pierres |first3=Anne-Sophie Le |last4=Multon |first4=Franck |title=Validation of an Ergonomic Assessment Method using Kinect Data in Real Workplace Conditions |journal=Applied Ergonomics |date=2017 |volume=65 |pages=562–569 |doi=10.1016/j.apergo.2016.10.015|pmid=27823772 |s2cid=13658487 |doi-access=free }}</ref> | ||
==Non-optical systems== | ==Non-optical systems== | ||
===Inertial systems=== | ===Inertial systems=== | ||
Inertial motion capture<ref>{{Cite web|url=http://www.xsens.com/images/stories/PDF/MVN_white_paper.pdf|title=Full 6DOF Human Motion Tracking Using Miniature Inertial Sensors}}</ref> technology is based on miniature inertial sensors, biomechanical models and [[sensor fusion]] algorithms.<ref>{{Cite web|url=https://www.xsens.com/fascination-motion-capture/|title=A history of motion capture|website=Xsens 3D motion tracking|language=en-US|access-date=2019-01-22}}</ref> The motion data of the inertial sensors ([[inertial guidance system]]) is often transmitted wirelessly to a computer, where the motion is recorded or viewed. Most inertial systems use inertial measurement units (IMUs) containing a combination of gyroscope, magnetometer, and accelerometer, to measure rotational rates. These rotations are translated to a skeleton in the software. Much like optical markers, the more IMU sensors the more natural the data. No external cameras, emitters or markers are needed for relative motions, although they are required to give the absolute position of the user if desired. Inertial motion capture systems capture the full six degrees of freedom body motion of a human in real-time and can give limited direction information if they include a magnetic bearing sensor, although these are much lower resolution and susceptible to electromagnetic noise. Benefits of using Inertial systems include: capturing in a variety of environments including tight spaces, no solving, portability, and large capture areas. Disadvantages include lower positional accuracy and positional drift which can compound over time. These systems are similar to the Wii controllers but are more sensitive and have greater resolution and update rates. They can accurately measure the direction to the ground to within a degree. The popularity of inertial systems is rising amongst game developers,<ref name="Xsens MVN Animate - Products"/> mainly because of the quick and easy | Inertial motion capture<ref>{{Cite web|url=http://www.xsens.com/images/stories/PDF/MVN_white_paper.pdf|title=Full 6DOF Human Motion Tracking Using Miniature Inertial Sensors|access-date=2013-04-03|archive-date=2013-04-04|archive-url=https://web.archive.org/web/20130404224049/http://www.xsens.com/images/stories/PDF/MVN_white_paper.pdf|url-status=dead}}</ref> technology is based on miniature inertial sensors, biomechanical models and [[sensor fusion]] algorithms.<ref>{{Cite web|url=https://www.xsens.com/fascination-motion-capture/|title=A history of motion capture|website=Xsens 3D motion tracking|language=en-US|access-date=2019-01-22|archive-date=2019-01-22|archive-url=https://web.archive.org/web/20190122195540/https://www.xsens.com/fascination-motion-capture/|url-status=dead}}</ref> The motion data of the inertial sensors ([[inertial guidance system]]) is often transmitted wirelessly to a computer, where the motion is recorded or viewed. Most inertial systems use inertial measurement units (IMUs) containing a combination of gyroscope, magnetometer, and accelerometer, to measure rotational rates. These rotations are translated to a skeleton in the software. Much like optical markers, the more IMU sensors the more natural the data. No external cameras, emitters or markers are needed for relative motions, although they are required to give the absolute position of the user if desired. Inertial motion capture systems capture the full six degrees of freedom body motion of a human in real-time and can give limited direction information if they include a magnetic bearing sensor, although these are much lower resolution and susceptible to electromagnetic noise. Benefits of using Inertial systems include: capturing in a variety of environments including tight spaces, no solving, portability, and large capture areas. Disadvantages include lower positional accuracy and positional drift which can compound over time. These systems are similar to the Wii controllers but are more sensitive and have greater resolution and update rates. They can accurately measure the direction to the ground to within a degree. The popularity of inertial systems is rising amongst game developers,<ref name="Xsens MVN Animate - Products"/> mainly because of the quick and easy setup resulting in a fast pipeline. A range of suits are now available from various manufacturers and base prices range from $600 (like [https://huskysense.com/ Husky Sense]) to US$80,000. | ||
===Mechanical motion=== | ===Mechanical motion=== | ||
| Line 165: | Line 169: | ||
{{Main|Facial motion capture}} | {{Main|Facial motion capture}} | ||
Most traditional motion capture hardware vendors provide for some type of low resolution facial capture utilizing anywhere from 32 to 300 markers with either an active or passive marker system. All of these solutions are limited by the time it takes to apply the markers, calibrate the positions and process the data. Ultimately the technology also limits their resolution and raw output quality levels. | Most traditional motion capture hardware vendors provide for some type of low-resolution facial capture utilizing anywhere from 32 to 300 markers with either an active or passive marker system. All of these solutions are limited by the time it takes to apply the markers, calibrate the positions and process the data. Ultimately the technology also limits their resolution and raw output quality levels. | ||
High fidelity facial motion capture, also known as '''performance capture''', is the next generation of fidelity and is utilized to record the more complex movements in a human face in order to capture higher degrees of emotion. Facial capture is currently arranging itself in several distinct camps, including traditional motion capture data, blend shaped based solutions, capturing the actual topology of an actor's face, and proprietary systems. | High-fidelity facial motion capture, also known as '''performance capture''', is the next generation of fidelity and is utilized to record the more complex movements in a human face in order to capture higher degrees of emotion. Facial capture is currently arranging itself in several distinct camps, including traditional motion capture data, blend-shaped based solutions, capturing the actual topology of an actor's face, and proprietary systems. | ||
The two main techniques are stationary systems with an array of cameras capturing the facial expressions from multiple angles and using software such as the stereo mesh solver from OpenCV to create a 3D surface mesh, or to use light arrays as well to calculate the surface normals from the variance in brightness as the light source, camera position or both are changed. These techniques tend to be only limited in feature resolution by the camera resolution, apparent object size and number of cameras. If the users face is 50 percent of the working area of the camera and a camera has megapixel resolution, then sub millimeter facial motions can be detected by comparing frames. Recent work is focusing on increasing the frame rates and doing optical flow to allow the motions to be retargeted to other computer generated faces, rather than just making a 3D Mesh of the actor and their expressions. | The two main techniques are stationary systems with an array of cameras capturing the facial expressions from multiple angles and using software such as the stereo mesh solver from OpenCV to create a 3D surface mesh, or to use light arrays as well to calculate the surface normals from the variance in brightness as the light source, camera position or both are changed. These techniques tend to be only limited in feature resolution by the camera resolution, apparent object size and number of cameras. If the users face is 50 percent of the working area of the camera and a camera has megapixel resolution, then sub millimeter facial motions can be detected by comparing frames. Recent work is focusing on increasing the frame rates and doing optical flow to allow the motions to be retargeted to other computer generated faces, rather than just making a 3D Mesh of the actor and their expressions. | ||
| Line 192: | Line 196: | ||
* [[Kinect]] (created by Microsoft Corporation) | * [[Kinect]] (created by Microsoft Corporation) | ||
* [[List of motion and gesture file formats]] | * [[List of motion and gesture file formats]] | ||
* [[Motion capture acting]] | * [[Motion-capture acting]] | ||
* [[Video tracking]] | * [[Video tracking]] | ||
* [[VR positional tracking]] | * [[VR positional tracking]] | ||
| Line 201: | Line 205: | ||
==External links== | ==External links== | ||
{{Library resources box|by=no|onlinebooks=no|others=no|about=yes|label=Motion capture}} | {{Library resources box|by=no|onlinebooks=no|others=no|about=yes|label=Motion capture}} | ||
*[https://www.xsens.com/fascination-motion-capture/ The fascination for motion capture], an introduction to the history of motion capture technology | *[https://www.xsens.com/fascination-motion-capture/ The fascination for motion capture] {{Webarchive|url=https://web.archive.org/web/20160304025241/https://www.xsens.com/fascination-motion-capture/ |date=2016-03-04 }}, an introduction to the history of motion capture technology | ||
{{Computer vision footer}} | {{Computer vision footer}} | ||