Table of Contents
Comments
Commenters1
Time-to-Adoption Horizon: Four to Five Years
For nearly forty years, the keyboard and mouse have been the primary means to interact with computers. The Nintendo Wii in 2006 and the Apple iPhone in 2007 signalled the beginning of widespread consumer interest in — and acceptance of — interfaces based on natural human gestures. Now, new devices are appearing on the market that take advantage of motions that are easy and intuitive to make, allowing us an unprecedented level of control over the devices around us. Cameras and sensors pick up the movements of our bodies without the need of remotes or handheld tracking tools. The full realization of the potential of gesture-based computing is still several years away, especially for education; but we are daily moving closer to a time when our gestures will speak for us, even to our machines.
Time-to-Adoption Horizon: Four to Five YearsFor nearly forty years, the keyboard and mouse have been the primary means to interact with computers. The Nintendo Wii in 2006 and the Apple iPhone in 2007 signalled the beginning of widespread consumer interest in — and acceptance of — interfaces based on natural human gestures. Now, new devices are appearing on the market that take advantage of motions that are easy and intuitive to make, allowing us an unprecedented level of control over the devices around us. Cameras and sensors pick up the movements of our bodies without the need of remotes or handheld tracking tools. The full realization of the potential of gesture-based computing is still several years away, especially for education; but we are daily moving closer to a time when our gestures will speak for us, even to our machines.
3
It is already common to interact with a new class of devices entirely by using natural gestures. The Microsoft Surface, the iPhone, iPad, and iPod Touch, the Nintendo Wii, the Kinect system for Xbox, and other gesture-based systems accept input in the form of taps, swipes, and other ways of touching, hand and arm motions, or body movement. These are the first in a growing array of alternative input devices that allow computers to recognize and interpret natural physical gestures as a means of control. We are seeing a gradual shift towards interfaces that adapt to — or are built for — humans and human movements.
It is already common to interact with a new class of devices entirely by using natural gestures. The Microsoft Surface, the iPhone, iPad, and iPod Touch, the Nintendo Wii, the Kinect system for Xbox, and other gesture-based systems accept input in the form of taps, swipes, and other ways of touching, hand and arm motions, or body movement. These are the first in a growing array of alternative input devices that allow computers to recognize and interpret natural physical gestures as a means of control. We are seeing a gradual shift towards interfaces that adapt to — or are built for — humans and human movements.4
As the underlying technologies evolve, a variety of approaches to gesture-based input are being explored. The screens of the iPhone, the iPad, and the Surface, for instance, react to pressure, motion, and the number of fingers touching the devices. The smaller devices additionally can react to manipulation — shaking, rotating, tilting, or moving the device in space. The Wii and similar gaming systems use a combination of a handheld, accelerometer-based controller and a stationary infrared sensor to determine position, acceleration, and direction. The technology to detect gestural movement and to display its results is improving very rapidly, increasing the opportunities for this kind of interaction. Two new gaming systems — the Sony PlayStation 3 Motion Controller and the Microsoft Kinect system — take a step closer to stripping the gesture-based interface of anything beyond the gesture and the machine, at least in terms of how it is experienced by the user.
As the underlying technologies evolve, a variety of approaches to gesture-based input are being explored. The screens of the iPhone, the iPad, and the Surface, for instance, react to pressure, motion, and the number of fingers touching the devices. The smaller devices additionally can react to manipulation — shaking, rotating, tilting, or moving the device in space. The Wii and similar gaming systems use a combination of a handheld, accelerometer-based controller and a stationary infrared sensor to determine position, acceleration, and direction. The technology to detect gestural movement and to display its results is improving very rapidly, increasing the opportunities for this kind of interaction. Two new gaming systems — the Sony PlayStation 3 Motion Controller and the Microsoft Kinect system — take a step closer to stripping the gesture-based interface of anything beyond the gesture and the machine, at least in terms of how it is experienced by the user.5
Gesture-based interfaces are increasingly built into things we can already use; Logitech and Apple have brought gesture-based mice to market, and Microsoft is developing several similar models. Smart phones, remote controls, and touch-screen computers all accept gesture input. As more of these devices are developed and released, our options for controlling a host of electronic devices are expanding. We can make music louder or softer by moving a hand, or skip a track with the flick of a finger. Apple’s Remote application for the iPhone turns the mobile device into a remote control for the Apple TV; users can search, play, pause, rewind, and so on, just by gliding a finger over the iPhone’s surface. Instead of learning where to point and click and what commands to type, we are beginning to be able to expect our computers to respond to natural movements that make sense to us.
Gesture-based interfaces are increasingly built into things we can already use; Logitech and Apple have brought gesture-based mice to market, and Microsoft is developing several similar models. Smart phones, remote controls, and touch-screen computers all accept gesture input. As more of these devices are developed and released, our options for controlling a host of electronic devices are expanding. We can make music louder or softer by moving a hand, or skip a track with the flick of a finger. Apple’s Remote application for the iPhone turns the mobile device into a remote control for the Apple TV; users can search, play, pause, rewind, and so on, just by gliding a finger over the iPhone’s surface. Instead of learning where to point and click and what commands to type, we are beginning to be able to expect our computers to respond to natural movements that make sense to us.6
Currently, the most common applications of gesture-based computing are for computer games, file and media browsing, and simulation and training. A number of simple mobile applications use gestures. The iPhone application Mover lets users “flick” photos and files from one phone to another; Shut Up from Nokia silences the phone when the user turns it upside down; nAlertme, an anti-theft tool, sounds an alarm if the phone isn’t shaken in a specific, preset way when it is switched on. Some companies are exploring further possibilities; for instance, Softkinetic develops platforms that support gesture-based technology, as well as designing custom applications for clients, including interactive marketing and consumer electronics as well as games and entertainment.
Currently, the most common applications of gesture-based computing are for computer games, file and media browsing, and simulation and training. A number of simple mobile applications use gestures. The iPhone application Mover lets users “flick” photos and files from one phone to another; Shut Up from Nokia silences the phone when the user turns it upside down; nAlertme, an anti-theft tool, sounds an alarm if the phone isn’t shaken in a specific, preset way when it is switched on. Some companies are exploring further possibilities; for instance, Softkinetic develops platforms that support gesture-based technology, as well as designing custom applications for clients, including interactive marketing and consumer electronics as well as games and entertainment. 7
Because it is changing not only the physical and mechanical aspects of interacting with computers, but also our very notions of what it means to work with a computer, gesture-based computing is a transformative and potentially disruptive technology. The sense of distance between the user and the machine decreases and the sensation of power and control over it increases when the machine responds to movements that feel intuitive and natural. Unlike a keyboard and mouse, gestural interfaces can often be used by more than one person at a time, making it possible to engage in truly collaborative activities and games. Our perception of the kinds of activities that computers are good for is also altered by gestural interaction — for instance, activities that require sweeping movements, such as many sports or exercises, are suited to gestural interfaces, as are many augmented reality interfaces.
Because it is changing not only the physical and mechanical aspects of interacting with computers, but also our very notions of what it means to work with a computer, gesture-based computing is a transformative and potentially disruptive technology. The sense of distance between the user and the machine decreases and the sensation of power and control over it increases when the machine responds to movements that feel intuitive and natural. Unlike a keyboard and mouse, gestural interfaces can often be used by more than one person at a time, making it possible to engage in truly collaborative activities and games. Our perception of the kinds of activities that computers are good for is also altered by gestural interaction — for instance, activities that require sweeping movements, such as many sports or exercises, are suited to gestural interfaces, as are many augmented reality interfaces.9
The kinaesthetic nature of gesture-based computing will very likely lead to new kinds of teaching or training simulations that look, feel, and operate almost exactly like their real-world counterparts. The ease and intuitiveness of a gestural interface makes the experience seem very natural, and even fun, making them relevant for learning at any age or level of study. Already, medical students benefit from simulations that teach the use of specific tools through gesture-based interfaces, and it is easy to see how such interfaces could be applied in the visual arts and other fields where fine motor skills come into play. Gestural interfaces allow users to easily perform precise manipulations that can be difficult with a mouse, as the prototypical video editing system Tamper makes plain (see the demonstration video at http://www.youtube.com/user/oblongtamper). When combined with haptic (touch or motion-based) feedback and augmented reality, the overall effect is very compelling.
The kinaesthetic nature of gesture-based computing will very likely lead to new kinds of teaching or training simulations that look, feel, and operate almost exactly like their real-world counterparts. The ease and intuitiveness of a gestural interface makes the experience seem very natural, and even fun, making them relevant for learning at any age or level of study. Already, medical students benefit from simulations that teach the use of specific tools through gesture-based interfaces, and it is easy to see how such interfaces could be applied in the visual arts and other fields where fine motor skills come into play. Gestural interfaces allow users to easily perform precise manipulations that can be difficult with a mouse, as the prototypical video editing system Tamper makes plain (see the demonstration video at http://www.youtube.com/user/oblongtamper). When combined with haptic (touch or motion-based) feedback and augmented reality, the overall effect is very compelling.10
Gesture-based computing opens up unparalleled avenues of accessibility and interaction for learners. Larger multi-touch displays support collaborative work, allowing multiple users to interact with content simultaneously. In schools where the Microsoft Surface has been installed in study areas, staff report that students naturally gravitate to the devices when they want to work together to study collaboratively. Gestural interfaces go beyond multi-touch displays, of course.
Gesture-based computing opens up unparalleled avenues of accessibility and interaction for learners. Larger multi-touch displays support collaborative work, allowing multiple users to interact with content simultaneously. In schools where the Microsoft Surface has been installed in study areas, staff report that students naturally gravitate to the devices when they want to work together to study collaboratively. Gestural interfaces go beyond multi-touch displays, of course. 11
Imagine an interface that allows students to determine or change the DNA of a fruit fly by piecing it together by hand, page through a fragile text from the Middle Ages, or practice surgical operations using the same movements a surgeon would — with gestural interfaces, discovery-based learning opportunities like these are likely to be common scenarios. Although these examples are hypothetical, research in the field of gesture-based computing is expanding rapidly and early results show that applications like these are not far-fetched.
Imagine an interface that allows students to determine or change the DNA of a fruit fly by piecing it together by hand, page through a fragile text from the Middle Ages, or practice surgical operations using the same movements a surgeon would — with gestural interfaces, discovery-based learning opportunities like these are likely to be common scenarios. Although these examples are hypothetical, research in the field of gesture-based computing is expanding rapidly and early results show that applications like these are not far-fetched. 12
Pranav Mistry, while at the MIT Media Lab, developed a gesture-based system called SixthSense that uses visual markers and gesture recognition to allow interaction with all sorts of real-time information and data in extremely intuitive ways. The platform was recently released as open source, which is likely to stimulate a raft of new ideas. Another gesture-based system by Mgestyk uses a 3-dimensional camera to capture user movements. The system has been demonstrated with Microsoft Flight Simulator, and allows players to fly a simulated plane simply by moving their hands — without any joystick or remote. It is not difficult to picture similar applications, a little further down the road, that could be used to simulate many kinds of experiences.
Pranav Mistry, while at the MIT Media Lab, developed a gesture-based system called SixthSense that uses visual markers and gesture recognition to allow interaction with all sorts of real-time information and data in extremely intuitive ways. The platform was recently released as open source, which is likely to stimulate a raft of new ideas. Another gesture-based system by Mgestyk uses a 3-dimensional camera to capture user movements. The system has been demonstrated with Microsoft Flight Simulator, and allows players to fly a simulated plane simply by moving their hands — without any joystick or remote. It is not difficult to picture similar applications, a little further down the road, that could be used to simulate many kinds of experiences.14
- Kinesiology. Dutch company Silverfit uses a gesture-based system to deliver fitness games designed for the elderly. Used in elder care organizations, the games provide gentle exercise and “activity of daily life” practice.
- Medicine. Digital Lightbox by BrainLAB is a multi-touch screen that allows doctors and surgeons to view and manipulate data from MRI, CT, x-ray, and other scan images. The system integrates with hospital data sources to enable health professionals to collaborate throughout the cycle of treatment.
- Sign Language. Researchers at Georgia Tech University have developed gesture-based games designed to help deaf children learn sign language. Deaf children of hearing parents often lack opportunities to pick up language serendipitously in the way hearing children do; the game provides an opportunity for incidental learning. The iPhone application Sign 4 Me allows easy manipulation of the point of view so that a sign can be viewed from any angle.
17
Digital Foam
http://wearables.unisa.edu.au/projects/digital-foam
Researchers at The University of South Australia’s Wearable Computer Lab (WCL) are developing a thin, flexible material that responds to touch (and multi-touch) and can be wrapped around objects of any shape.
Digital Foamhttp://wearables.unisa.edu.au/projects/digital-foam
Researchers at The University of South Australia’s Wearable Computer Lab (WCL) are developing a thin, flexible material that responds to touch (and multi-touch) and can be wrapped around objects of any shape.
18
Gesture-Based Computing on the Cheap
http://web.mit.edu/newsoffice/2010/gesture-computing-0520.html
MIT researchers have developed a gestural system that requires only a computer webcam and an inexpensive pair of multicoloured Lycra gloves.
Gesture-Based Computing on the Cheaphttp://web.mit.edu/newsoffice/2010/gesture-computing-0520.html
MIT researchers have developed a gestural system that requires only a computer webcam and an inexpensive pair of multicoloured Lycra gloves.
19
The Hybridiser
http://vimeo.com/6580702
This innovative project at the Auckland Museum uses touch-screen interfaces to allow visitors to create custom virtual orchids in lifelike detail. Such screens can provide an immersive, interactive experience that directly engages the visitor.
Kinect for XBox 360
http://www.xbox.com/kinect
Microsoft’s Kinect system works with the XBox 360 platform to deliver immersive, gesture-based gaming and interactive experiences.
The Hybridiserhttp://vimeo.com/6580702
This innovative project at the Auckland Museum uses touch-screen interfaces to allow visitors to create custom virtual orchids in lifelike detail. Such screens can provide an immersive, interactive experience that directly engages the visitor.
Kinect for XBox 360
http://www.xbox.com/kinect
Microsoft’s Kinect system works with the XBox 360 platform to deliver immersive, gesture-based gaming and interactive experiences.
20
SixthSense
http://www.pranavmistry.com/projects/sixthsense
MIT student Pranav Mistry created a wearable gesture-based computing system to interact with everyday objects and people. This site describes the project and links to demonstration videos.
SixthSensehttp://www.pranavmistry.com/projects/sixthsense
MIT student Pranav Mistry created a wearable gesture-based computing system to interact with everyday objects and people. This site describes the project and links to demonstration videos.
22
The following articles and resources are recommended for those who wish to learn more about gesture-based computing.
The following articles and resources are recommended for those who wish to learn more about gesture-based computing.23
A Future Full of Touchscreens? It’s All in the Software
http://venturebeat.com/2010/05/25/a-future-full-of-touchscreens-its-all-in-the-software
(Andrew Hsu, VentureBeat.com, 25 May 2010.) Software will be one of the main drivers of the technology behind touch screens, and innovators in this space will be the ones who entice consumers to use their products.
A Future Full of Touchscreens? It’s All in the Softwarehttp://venturebeat.com/2010/05/25/a-future-full-of-touchscreens-its-all-in-the-software
(Andrew Hsu, VentureBeat.com, 25 May 2010.) Software will be one of the main drivers of the technology behind touch screens, and innovators in this space will be the ones who entice consumers to use their products.
24
IDENT Technology’s Near Field Electrical Sensing Interfaces
http://www.ident-technology.com
(Accessed 21 October 2010.) IDENT Technology has developed interfaces that use near field electrical sensing to allow mobiles to respond to grip and proximity sensing. A ringing mobile will put the call through if it is picked up and held but will send it to voice mail if it is picked up and quickly put down again.
IDENT Technology’s Near Field Electrical Sensing Interfaces http://www.ident-technology.com
(Accessed 21 October 2010.) IDENT Technology has developed interfaces that use near field electrical sensing to allow mobiles to respond to grip and proximity sensing. A ringing mobile will put the call through if it is picked up and held but will send it to voice mail if it is picked up and quickly put down again.
25
Interactive Instrumental Performance and Gesture Sonification
http://scan.net.au/scan/journal/display.php?journal_id=122
(Kirsty Beilharz, SCAN: Journal of Media Arts Culture, Macquarie University, Vol 5 Number 3 December 2008.) This ongoing research by Kirsty Beilharz of University of Technology Sydney explores gestural interfaces for allowing artists and musicians to augment art installations and musical performances in real time.
Interactive Instrumental Performance and Gesture Sonification http://scan.net.au/scan/journal/display.php?journal_id=122
(Kirsty Beilharz, SCAN: Journal of Media Arts Culture, Macquarie University, Vol 5 Number 3 December 2008.) This ongoing research by Kirsty Beilharz of University of Technology Sydney explores gestural interfaces for allowing artists and musicians to augment art installations and musical performances in real time.
26
Point, Click: A Review of Gesture Control Technologies
http://games.venturebeat.com/2010/02/09/point-click-a-review-of-gesture-control-technologies
(Damian Rollison, VentureBeat.com, 9 February 2010.) This article discusses the key developers and platforms working with gesture-based technologies.
Point, Click: A Review of Gesture Control Technologies http://games.venturebeat.com/2010/02/09/point-click-a-review-of-gesture-control-technologies
(Damian Rollison, VentureBeat.com, 9 February 2010.) This article discusses the key developers and platforms working with gesture-based technologies.
27
Touching the Future: The Rise of Multitouch Interfaces
http://www.perada-magazine.eu/pdf/002864/002864.pdf
(Johannes Schöning, PerAda Magazine, 1 April 2010.) This short research synopsis discusses the technologies behind multi-touch and suggests ways it will be used, particularly for geospatial applications. An extensive list of references is provided.
Touching the Future: The Rise of Multitouch Interfaceshttp://www.perada-magazine.eu/pdf/002864/002864.pdf
(Johannes Schöning, PerAda Magazine, 1 April 2010.) This short research synopsis discusses the technologies behind multi-touch and suggests ways it will be used, particularly for geospatial applications. An extensive list of references is provided.
28
Delicious: Gesture-Based Computing
http://delicious.com/tag/hz10anz+gesturecomputing
Follow this link to find additional resources tagged for this topic and this edition of the Horizon Report. To add to this list, simply tag resources with “hz10anz” and “gesturecomputing” when you save them to Delicious.
Delicious: Gesture-Based Computinghttp://delicious.com/tag/hz10anz+gesturecomputing
Follow this link to find additional resources tagged for this topic and this edition of the Horizon Report. To add to this list, simply tag resources with “hz10anz” and “gesturecomputing” when you save them to Delicious.
