Table of Contents
Comments
Commenters1
Time-to-Adoption Horizon: Four to Five Years
For nearly forty years, the keyboard and mouse have been the primary means to interact with computers. The Nintendo Wii in 2006 and the Apple iPhone in 2007 signaled the beginning of widespread consumer interest in — and acceptance of — interfaces based on natural human gestures. Now, new devices are appearing on the market that take advantage of motions that are easy and intuitive to make, allowing us an unprecedented level of control over the devices around us. Cameras and sensors pick up the movements of our bodies without the need of remotes or handheld tracking tools. The full realization of the potential of gesture-based computing is still several years away, especially for education; but we are moving ever closer to a time when our gestures will speak for us, even to our machines.
Time-to-Adoption Horizon: Four to Five YearsFor nearly forty years, the keyboard and mouse have been the primary means to interact with computers. The Nintendo Wii in 2006 and the Apple iPhone in 2007 signaled the beginning of widespread consumer interest in — and acceptance of — interfaces based on natural human gestures. Now, new devices are appearing on the market that take advantage of motions that are easy and intuitive to make, allowing us an unprecedented level of control over the devices around us. Cameras and sensors pick up the movements of our bodies without the need of remotes or handheld tracking tools. The full realization of the potential of gesture-based computing is still several years away, especially for education; but we are moving ever closer to a time when our gestures will speak for us, even to our machines.
3
It is already common to interact with a new class of devices entirely through the use of natural gestures. The Microsoft Surface, the iPhone and iPod Touch, the Nintendo Wii, and other gesture-based systems accept input in the form of taps, swipes, and other ways of touching, hand and arm motions, gentle shakes, or body movements. These are the first in a growing array of alternative input devices that allow computers to recognize and interpret natural physical gestures as a means of control and the start of a gradual shift towards interfaces that make sense of common human movements. Gestural interfaces allow users to engage in virtual activities with motion and movement similar to what they would use in the real world, manipulating content intuitively. The idea that natural, comfortable motions can be used to control computers is opening the way to a host of input devices that look and feel very different from the keyboard and mouse.
It is already common to interact with a new class of devices entirely through the use of natural gestures. The Microsoft Surface, the iPhone and iPod Touch, the Nintendo Wii, and other gesture-based systems accept input in the form of taps, swipes, and other ways of touching, hand and arm motions, gentle shakes, or body movements. These are the first in a growing array of alternative input devices that allow computers to recognize and interpret natural physical gestures as a means of control and the start of a gradual shift towards interfaces that make sense of common human movements. Gestural interfaces allow users to engage in virtual activities with motion and movement similar to what they would use in the real world, manipulating content intuitively. The idea that natural, comfortable motions can be used to control computers is opening the way to a host of input devices that look and feel very different from the keyboard and mouse.4
As the underlying technologies evolve, a variety of approaches to gesture-based input are being explored. The screens of the iPhone and the Surface, for instance, react to pressure, motion, and the number of fingers touching the devices. The iPhone additionally can react to manipulation of the device itself — shaking, rotating, tilting, or moving the device in space. The Wii and other emerging gaming systems use a combination of a handheld, accelerometer-based controller and stationary infrared sensor to determine position, acceleration, and direction. The technology to detect gestural movement and to display its results is improving very rapidly, increasing the opportunities for this kind of interaction. Two new gaming systems — the Sony PlayStation 3 Motion Controller and the Microsoft Kinect system — take a step closer to stripping the gesture-based interface of anything beyond the gesture and the machine, at least in terms of how it is experienced by the user.
As the underlying technologies evolve, a variety of approaches to gesture-based input are being explored. The screens of the iPhone and the Surface, for instance, react to pressure, motion, and the number of fingers touching the devices. The iPhone additionally can react to manipulation of the device itself — shaking, rotating, tilting, or moving the device in space. The Wii and other emerging gaming systems use a combination of a handheld, accelerometer-based controller and stationary infrared sensor to determine position, acceleration, and direction. The technology to detect gestural movement and to display its results is improving very rapidly, increasing the opportunities for this kind of interaction. Two new gaming systems — the Sony PlayStation 3 Motion Controller and the Microsoft Kinect system — take a step closer to stripping the gesture-based interface of anything beyond the gesture and the machine, at least in terms of how it is experienced by the user.5
Gesture-based interfaces are increasingly built into things we can already use; Logitech and Apple have brought gesture-based mice to market, and Microsoft is developing several similar models. Smart phones, remote controls, and touch-screen computers all accept gesture input. As more of these devices are developed and released, our options for controlling a host of electronic devices are expanding. We can make music louder or softer by moving a hand, or skip a track with the flick of a finger. Apple’s Remote application for the iPhone turns the mobile device into a remote control for the Apple TV; users can search, play, pause, rewind, and so on, just by gliding a finger over the iPhone’s surface. Instead of learning where to point and click and how to type, we are beginning to be able to expect our computers to respond to natural movements that make sense to us.
Gesture-based interfaces are increasingly built into things we can already use; Logitech and Apple have brought gesture-based mice to market, and Microsoft is developing several similar models. Smart phones, remote controls, and touch-screen computers all accept gesture input. As more of these devices are developed and released, our options for controlling a host of electronic devices are expanding. We can make music louder or softer by moving a hand, or skip a track with the flick of a finger. Apple’s Remote application for the iPhone turns the mobile device into a remote control for the Apple TV; users can search, play, pause, rewind, and so on, just by gliding a finger over the iPhone’s surface. Instead of learning where to point and click and how to type, we are beginning to be able to expect our computers to respond to natural movements that make sense to us.6
Currently, the most common applications of gesture-based computing are for computer games, file and media browsing, and simulation and training. A number of simple mobile applications use gestures. Mover lets users “flick” photos and files from one phone to another; Shut Up from Nokia silences the phone when the user turns it upside down; nAlertme, an anti-theft tool, sounds an alarm if the phone isn’t shaken in a specific, preset way when it is switched on. Some companies are exploring further possibilities; for instance, Softkinetic (http://www.softkinetic.net) develops platforms that support gesture-based technology, as well as designing custom applications for clients, including interactive marketing and consumer electronics as well as games and entertainment.
Currently, the most common applications of gesture-based computing are for computer games, file and media browsing, and simulation and training. A number of simple mobile applications use gestures. Mover lets users “flick” photos and files from one phone to another; Shut Up from Nokia silences the phone when the user turns it upside down; nAlertme, an anti-theft tool, sounds an alarm if the phone isn’t shaken in a specific, preset way when it is switched on. Some companies are exploring further possibilities; for instance, Softkinetic (http://www.softkinetic.net) develops platforms that support gesture-based technology, as well as designing custom applications for clients, including interactive marketing and consumer electronics as well as games and entertainment. 7
Because gesture-based computing changes not only the physical and mechanical aspects of interacting with computers, but also our perception of what it means to work with a computer, it is a transformative technology.
Because gesture-based computing changes not only the physical and mechanical aspects of interacting with computers, but also our perception of what it means to work with a computer, it is a transformative technology. 9
In certain respects, gesture-based computing is simply an extension of learning activities museums have always embraced; museums clearly understand the value of kinesthetic learning. It is not uncommon to find reproductions of manuscripts or scrolls in galleries that allow visitors to experience what it might be like to handle the real object. While gesture-based computing goes far beyond this, the experiences are related.
In certain respects, gesture-based computing is simply an extension of learning activities museums have always embraced; museums clearly understand the value of kinesthetic learning. It is not uncommon to find reproductions of manuscripts or scrolls in galleries that allow visitors to experience what it might be like to handle the real object. While gesture-based computing goes far beyond this, the experiences are related. 10
Gesture-based computing is important for museums particularly because they have created, for good reasons, an environment in which an audience is prevented from touching and manipulating objects — many of which could teach or be better understood if visitors could actually touch or manipulate them. What is missing in too many museum experiences is the direct and satisfying personal connection of an individual with the object.
Gesture-based computing is important for museums particularly because they have created, for good reasons, an environment in which an audience is prevented from touching and manipulating objects — many of which could teach or be better understood if visitors could actually touch or manipulate them. What is missing in too many museum experiences is the direct and satisfying personal connection of an individual with the object. 11
The kinesthetic nature of gesture-based computing will very likely lead to new kinds of teaching or training simulations that look, feel, and operate almost exactly like their real-world counterparts. The very ease and intuitiveness of a gestural interface makes the experience seem very natural, and even fun. Already, medical students benefit from simulations that teach the use of specific tools through gesture-based interfaces, and it is easy to see how such interfaces could be applied in the visual arts and other fields where fine motor skills come into play. When combined with haptic (touch or motion-based) feedback, the overall effect is very compelling.
The kinesthetic nature of gesture-based computing will very likely lead to new kinds of teaching or training simulations that look, feel, and operate almost exactly like their real-world counterparts. The very ease and intuitiveness of a gestural interface makes the experience seem very natural, and even fun. Already, medical students benefit from simulations that teach the use of specific tools through gesture-based interfaces, and it is easy to see how such interfaces could be applied in the visual arts and other fields where fine motor skills come into play. When combined with haptic (touch or motion-based) feedback, the overall effect is very compelling.12
Gesture-based computing opens up unparalleled avenues of accessibility and interaction for visitors. Larger multi-touch displays support collaborative work, allowing multiple users to interact with content simultaneously, unlike a single-user mouse. Imagine an interface that allows the visitor to seamlessly explore a delicate silk scroll using the gestures of a Chinese scholar, determine or change the DNA of a fruit fly by piecing it together by hand, mix wine with water in an ancient Greek krater, page through an illuminated manuscript, or embroider a medieval altar frontal — with gestural interfaces, discovery-based learning opportunities like these become possible.
Gesture-based computing opens up unparalleled avenues of accessibility and interaction for visitors. Larger multi-touch displays support collaborative work, allowing multiple users to interact with content simultaneously, unlike a single-user mouse. Imagine an interface that allows the visitor to seamlessly explore a delicate silk scroll using the gestures of a Chinese scholar, determine or change the DNA of a fruit fly by piecing it together by hand, mix wine with water in an ancient Greek krater, page through an illuminated manuscript, or embroider a medieval altar frontal — with gestural interfaces, discovery-based learning opportunities like these become possible.14
- Education and Interpretation. Educational games built on gesture-based platforms can allow visitors to explore relevant techniques, actions, and interactions in a playful manner.
- Exhibitions and Collections. Gesture-based interfaces and tools may help visitors to more intuitively understand the operational aesthetic of objects, including how they function, how they are created, and related information. An example of how this might be done can be seen in the iPad application Elements, which allows users to manipulate and “touch” chemical compounds, metals, and even radioactive substances in a fun, engaging way.
- Visitor Services and Accessibility. Gesture-based tools are especially easy for the very young, who need almost no instruction in their use. On the other end of the spectrum, gesture-based tools could allow the elderly and infirm to participate in activities and increase their ability to access collections.
17
The Hybridiser
http://vimeo.com/6580702
This innovative project at the Auckland Museum uses touch-screen interfaces to allow visitors to create custom virtual orchids in lifelike detail. Such screens can provide an immersive, interactive experience that directly engages the visitor.
The Hybridiserhttp://vimeo.com/6580702
This innovative project at the Auckland Museum uses touch-screen interfaces to allow visitors to create custom virtual orchids in lifelike detail. Such screens can provide an immersive, interactive experience that directly engages the visitor.
18
Lisbon Oceanarium Sea Monster Exhibit
http://www.ydreams.com/#/en/projects/eventsexhibitions/interactivechildrensexhibitionlisbonaceanario/
In this immersive, touch-screen exhibit, visitors use both their hands and their feet to interact with — and control — the exhibit.
Lisbon Oceanarium Sea Monster Exhibithttp://www.ydreams.com/#/en/projects/eventsexhibitions/interactivechildrensexhibitionlisbonaceanario/
In this immersive, touch-screen exhibit, visitors use both their hands and their feet to interact with — and control — the exhibit.
19
Mapping Application Magnifies California’s Rich History
http://www.ideum.com/interactive-exhibits/omca-mapping-app/
Installed in the Oakland Museum in California, this exhibit offers visitors a rich, immersive experience with the maps of California in order to see how land distribution and changes occurred through time.
Mapping Application Magnifies California’s Rich Historyhttp://www.ideum.com/interactive-exhibits/omca-mapping-app/
Installed in the Oakland Museum in California, this exhibit offers visitors a rich, immersive experience with the maps of California in order to see how land distribution and changes occurred through time.
20
Oil Spill Multitouch Map Mashup
http://www.ideum.com/blog/2010/06/bp-oil-spill-multitouch-mashup/
This application, created for display at natural science museums and aquariums, highlights the June 2010 oil spill disaster in the Gulf of Mexico. While engaging visitors with its touch-based controls, the display helps them understand how the event has unfolded.
Oil Spill Multitouch Map Mashuphttp://www.ideum.com/blog/2010/06/bp-oil-spill-multitouch-mashup/
This application, created for display at natural science museums and aquariums, highlights the June 2010 oil spill disaster in the Gulf of Mexico. While engaging visitors with its touch-based controls, the display helps them understand how the event has unfolded.
21
Uffizi Touch Project
http://www.uffizitouch.it/?cat=3
The Uffizi Gallery in Florence has incorporated gesture-based interfaces in their space to allow visitors to explore centuries of art using immersive touch screen imaging technology similar in some respects to Apple’s CoverFlow interface.
Uffizi Touch Project http://www.uffizitouch.it/?cat=3
The Uffizi Gallery in Florence has incorporated gesture-based interfaces in their space to allow visitors to explore centuries of art using immersive touch screen imaging technology similar in some respects to Apple’s CoverFlow interface.
22
Virtual Ercolano (Video)
http://www.youtube.com/user/MuseoMavTV#p/a/u/1/muDBWr0AP8g
The Museo Archeologico Virtuale, in Ercolano, Italy, has integrated several gesture-based technologies in its galleries. This video shows an exhibit covered with virtual dust that scatters, revealing mosaics, when visitors disturb the dust with their hands or feet.
Virtual Ercolano (Video)http://www.youtube.com/user/MuseoMavTV#p/a/u/1/muDBWr0AP8g
The Museo Archeologico Virtuale, in Ercolano, Italy, has integrated several gesture-based technologies in its galleries. This video shows an exhibit covered with virtual dust that scatters, revealing mosaics, when visitors disturb the dust with their hands or feet.
24
The following articles and resources are recommended for those who wish to learn more about gesture-based computing.
The following articles and resources are recommended for those who wish to learn more about gesture-based computing.25
The Best Computer Interfaces: Past, Present, and Future
http://www.technologyreview.com/computing/22393
(Duncan Graham-Rowe, Technology Review, 6 April 2009.) This article discusses a variety of interfaces, including gesture-sensing, voice recognition, and multi-touch surfaces.
The Best Computer Interfaces: Past, Present, and Futurehttp://www.technologyreview.com/computing/22393
(Duncan Graham-Rowe, Technology Review, 6 April 2009.) This article discusses a variety of interfaces, including gesture-sensing, voice recognition, and multi-touch surfaces.
26
Haptics Brings A Personal Touch To Technology
http://news.bbc.co.uk/2/hi/technology/10373923.stm
(Michael Fitzpatrick, BBC News, 5 July 2010.) This article outlines the types of haptic interfaces and technologies that will be forthcoming as touch-based interfaces and flexible displays become more common.
Haptics Brings A Personal Touch To Technologyhttp://news.bbc.co.uk/2/hi/technology/10373923.stm
(Michael Fitzpatrick, BBC News, 5 July 2010.) This article outlines the types of haptic interfaces and technologies that will be forthcoming as touch-based interfaces and flexible displays become more common.
27
‘Imaginary’ Interface Could Replace Real Thing
http://www.msnbc.msn.com/id/37580233/ns/technology_and_science-innovation/
(Adam Hadhazy, MSNBC, 8 June 2010.) This post outlines some of the emerging technologies that will have impact on gesture-based computing as it continues to develop.
‘Imaginary’ Interface Could Replace Real Thinghttp://www.msnbc.msn.com/id/37580233/ns/technology_and_science-innovation/
(Adam Hadhazy, MSNBC, 8 June 2010.) This post outlines some of the emerging technologies that will have impact on gesture-based computing as it continues to develop.
28
Towards Interactive Museum: Mapping Cultural Contexts to Historical Objects (PDF)
http://mirw09.offis.de/paper/Towards%20Interactive%20Museum%20Mapping%20Cultural%20Contexts%20to%20Historical%20Objects.pdf
(Ki-Woong Park, Sung Kyu Park, Jong-Woon Yoo, Kyu Ho Park, MIRW Workshop ’09, Bonn, Germany, 15 September 2009.) This paper outlines technologies that could be used by visitors to interact with exhibits and collections, including a wearable computer system that is controlled largely by the gestures of the user.
Towards Interactive Museum: Mapping Cultural Contexts to Historical Objects (PDF)http://mirw09.offis.de/paper/Towards%20Interactive%20Museum%20Mapping%20Cultural%20Contexts%20to%20Historical%20Objects.pdf
(Ki-Woong Park, Sung Kyu Park, Jong-Woon Yoo, Kyu Ho Park, MIRW Workshop ’09, Bonn, Germany, 15 September 2009.) This paper outlines technologies that could be used by visitors to interact with exhibits and collections, including a wearable computer system that is controlled largely by the gestures of the user.
29
Why Desktop Touch Screens Don’t Really Work Well For Humans
http://www.washingtonpost.com/wp-dyn/content/article/2009/10/13/AR2009101300113.html
(Michael Arrington, The Washington Post, 12 October 2009.) Desktop touch screens are available but difficult to use over long periods. This article suggests another design approach.
Why Desktop Touch Screens Don’t Really Work Well For Humanshttp://www.washingtonpost.com/wp-dyn/content/article/2009/10/13/AR2009101300113.html
(Michael Arrington, The Washington Post, 12 October 2009.) Desktop touch screens are available but difficult to use over long periods. This article suggests another design approach.
30
Delicious: Gesture-Based Computing
http://delicious.com/tag/hz10mu+gesturecomputing
(Tagged by Horizon Advisory Board and friends, 2010). Follow this link to find additional resources tagged for this topic and this edition of the Horizon Report. To add to this list, simply tag resources with “hz10mu” and “gesturecomputing” when you save them to Delicious.
Delicious: Gesture-Based Computinghttp://delicious.com/tag/hz10mu+gesturecomputing
(Tagged by Horizon Advisory Board and friends, 2010). Follow this link to find additional resources tagged for this topic and this edition of the Horizon Report. To add to this list, simply tag resources with “hz10mu” and “gesturecomputing” when you save them to Delicious.
