Time-to-Adoption Horizon: Two to Three Years
The way we expect to interact with computers is changing as movement and gesture-based interfaces accompany popular games, mobile phones, and new computers and peripherals. Two advancements in particular — the accelerometer and the multi-touch screen — have already begun to show us more natural, intuitive ways to communicate with computers and work with electronic content. As more devices with these capabilities enter the marketplace, interface designers and software engineers are developing novel ways to use these interfaces that are ever more comfortable physically and more engaging and satisfying emotionally.

Overview

All around us, new interfaces are changing the way we work with computers and other technology. First appearing in the gaming world, the Nintendo Wii and its “wand” controller broke away from the traditional handheld controller and keyboard/mouse models. Along with other innovative interfaces like Activision’s Guitar Hero, these devices allow players to engage in virtual activities with motion and movement similar to what they would use in the real world. Apple’s iPhone and the table-sized Microsoft Surface broke similar barriers in the realm of interactive displays with their multi-touch screen-based controls. These two innovations — accelerometer-based devices and multi-touch screens — allow users to manipulate content intuitively, using natural gestures like flicking the wrist or sweeping the fingertips over a display.

Accelerometer-based devices use speed, direction, and momentum as inputs. The Wii-mote (the controller for the Wii) acts like an extension of the body, conveying movements of the wrist, fingers, and arm to the device’s sensors. Other devices like the Wii Balance Board detect body posture, allowing the system to react to the user’s entire body and enabling the simulation of physics concepts and principles of motion through kinaesthetic experience. Mobile devices like the iPhone use an accelerometer to determine the device’s orientation, causing text and photos to rotate when the mobile is turned on its side, and enabling it to detect when it is being shaken so applications can react. For instance, dice games are available on the iPhone that allow the player to roll the dice by shaking the phone back and forth; a driving game for the iPod Touch is played by turning the device as if it were a steering wheel.

Large multi-touch displays also open up possibilities for collaborative activity; only one person can use a mouse at a time, but the Surface responds to several users at once. Perceptive Pixel’s multi-touch system (www.perceptivepixel.com) reacts to multiple areas of contact (i.e. all the fingers on both hands, or several people’s hands at one time), allowing a more natural, gestural movement than single-touch interfaces. Forty years after the computer mouse was first demonstrated, these devices and other alternatives are just beginning to suggest new ways to interact with computers — ways that are based on movements that are more closely associated with the task at hand, easier to do in a hurry, or more natural.

The Wii remote is not irrevocably locked to the Wii console as its only computing device. Some have experimented with connecting the Wii with a personal computer to use as an interface for basic operations and gaming (see www.youtube.com/watch?v=jkX1oQsvNe0). At Carnegie Mellon University, for example, a graduate student developed and demonstrated an interactive whiteboard created with a laptop computer, a standard projector and screen, and a homemade infrared pen (see a video of Johnny Lee’s talk at TED 2008 at http://www.youtube.com/watch?v=0H1zrLZwPjQ). Others have implemented his model, providing interactive, multi-touch whiteboards in schools where the cost of purchasing one is simply too prohibitive.

Experiments like these are increasing our understanding of how alternative input devices can be designed and used. Combined with increasingly affordable large screens and video hardware capable of large-scale, multiscreen projection, alternative input devices will lead to new interfaces and displays that take advantage of natural motion, increased resolution, and larger display size. The result, still several years away, will be interaction devices that are as big a leap over the mouse as the mouse was over typing at the command line.

Relevance for Teaching, Learning, and Creative Expression

New applications that use the iPhone’s accelerometer and multi-touch input screen are appearing daily, but most of them are designed for entertainment and gaming, and few are targeted specifically for education. One recent example is Grafly (http://grafly.com), a graphing calculator for the iPhone that uses the touch interface to zoom, and also allows the user to view graphs from any angle by simply moving the phone “around” the graph to change the view. Music and language applications that capture sound or written words are emerging, but these seem still to be in their infancy.

The Microsoft Surface is currently too expensive for most schools, but devices like the Wii remote show promise for both their flexibility and affordability. Homemade smartboards created with the Wii remote are already being used in K-12 and college classes around the world. Accelerometers in mobile devices and in the Wii remote are sophisticated enough to be used for science experiments and are affordable as teacher tools, and they are already being adapted for use in schools.

Many other potential applications for these devices exist. Portable, motion sensitive devices, can be used for all sorts of data capture collection, and especially lend themselves to field work. Multi-sensory input devices that combine touch with visual, aural, and tactile feedback engage the user on multiple levels. Such devices have the potential to create simple but powerfully engaging learning experiences. Input devices that behave or respond like instruments (Ocarina) or artists’ materials (Trace) enhance the transfer of real-world skills to computer-assisted music and art, enabling artists to experiment with a wider range of forms for creative expression — or simply to create.

The alternative input devices already widely available are driving development of increasingly sophisticated applications, and as this technology matures, we can expect a great deal more. Tools continue to emerge that take greater advantage of the capabilities for gestural input and multiple streams of feedback that mobiles, gaming systems, and even computers increasingly possess. Over time, innovative educational uses for these devices will certainly look to take advantage of them.

A sampling of applications of alternative input devices across disciplines includes the following:

  • Physics Two physicists in Italy designed an experiment for secondary students using a Wii remote to measure acceleration of a pendulum. The remote provided an affordable accelerometer that could be easily obtained by secondary school teachers for use in science classes.
  • Language Multi-touch and touch-sensitive screens or interactive whiteboard setups can capture a student’s effort to write characters in Arabic, Chinese, Cyrillic, Hebrew, Japanese, or other languages, giving feedback on whether the character is correctly written.
  • Art Touch sensitive devices create new ways for artists to express themselves using digital tools and techniques — and to open creative doors to new audiences as well. For example, the Wii remote has been used to control a painting program via the movement of a wheelchair.

Examples of Alternative Input Devices

The following links provide examples of educational applications of alternative input devices.

au Design Project x Yamaha
http://www.kilian-nakamura.com/blog-english/index.php/au-design-project-x-yamaha-merges-music-with-mobile-in-new-ways/
Au, the mobile division of Japanese company KDDI, develops a series of projects intended to push the envelope of mobile device design. One project, in collaboration with Yamaha, has resulted in mobile phones that double as musical instruments.

gMote
http://techie-buzz.com/featured/control-pc-with-mouse-gestures.html
gMote is software that allows the user to capture gestures — custom mouse motions — and assign them to various computer tasks, such as copying and pasting, text formatting, opening programs, and more.

iKana Touch
http://www.thinkmac.co.uk/ikanatouch/
iKana Touch is a language application for the iPhone that allows Japanese language learners to review Hiragana and Katakana, see an animation of how each character is written, and practice forming a character by drawing it directly on the touch screen.

Microsoft Surface
http://www.microsoft.com/surface/index.html
The Microsoft Surface is a tabletop, multi-touch interactive display. Designed for businesses like hotels or conference centres, it includes applications for viewing photographs, creating maps, and listening to music.

Nintendo Wii Fit
http://www.nintendo.com/wiifit
The Wii Fit combines the Wii Balance Board, the Wii remote, and the Wii console into a full-body input package. It includes sports games, strength and yoga exercises, and balance games. Special remotes shaped like golf clubs, tennis racquets, and baseball bats are available.

QuickGraph
http://www.colombiamug.com/EN/QuickGraph.html
QuickGraph is a graphing calculator for the iPhone. Users can zoom in and out or rotate the graphs using the touch interface, or shake the phone to reset the view.

Wiimote in My Classrooms
http://heyjude.wordpress.com/2008/09/02/wiimote-in-my-classrooms/
In her blog, Judy O’Connell reports on classes at St. Joseph’s College, Hunters Hill that are using techniques developed by Carnegie Mellon graduate student Johnny Lee to adapt the Wii remote as an interactive whiteboard.

Wiimote Wheelchair Helps Disabled People Paint (Roughly Speaking)
http://gizmodo.com/5013758/wiimote-wheelchair-helps-disabled-people-paint-roughly-speaking A Wii remote was used to control a painting program by allowing people to move the brush with a wheelchair.

For Further Reading

The following articles and resources are recommended for those who wish to learn more about alternative input devices.

Accelerometer
http://en.wikipedia.org/wiki/Accelerometer
(Wikipedia.) This article describes accelerometers and lists some applications for them, including the Wii and several mobile phones that use the technology.

Hacking the Wii Remote for Physics Class
http://www.physorg.com/news104502773.html
Physicists in Italy designed experiments for secondary school science students using the accelerometer in a Wii remote.

Johnny Chung Lee: Projects
http://www.cs.cmu.edu/~johnny/projects/wii/
Johnny Chung Lee’s projects include interactive whiteboard surfaces and other tracking applications that use the Wii remote. Videos demonstrating the projects and the source code that make them work are available on the site.

Multi-touch
http://en.wikipedia.org/wiki/Multi-touch
(Wikipedia.) This article discusses the history of multi-touch technology and describes recent developments.

Steven Levy on Melding the Digital and Physical Realms
http://www.wired.com/gadgets/gadgetreviews/magazine/16-11/ts_levy
This brief article describes the Wii, Guitar Hero, and the iPhone as the long-promised merging of the virtual and the real.

Top 15 Interactive Display Technologies
http://www.gizmowatch.com/entry/top-15-interactive-display-technologies/
(Naveen, Gizmo Watch, 15 May 2007.) This article lists fifteen interactive display technologies, including tabletop and wall-mounted multi-touch screens.

del.icio.us: Alternative Input Devices
http://del.icio.us/tag/hzau08+altinteraction (Australia–New Zealand Horizon Advisory Board and Friends, 2008.) Follow this link to find resources tagged for this topic and this edition of the Horizon Report, including the ones listed here. To add to this list, simply tag resources with “hzau08” and “altinteraction” when you save them to del.icio.us.

Related Tags: alternative interaction devices, accelerometer, multi-touch interface, alternative interfaces

Posted by NMC on November 30, 2008
Tags: Section

Total comments on this page: 0

How to read/write comments

Comments on specific paragraphs:

Click the icon to the right of a paragraph

  • If there are no prior comments there, a comment entry form will appear automatically
  • If there are already comments, you will see them and the form will be at the bottom of the thread

Comments on the page as a whole:

Click the icon to the right of the page title (works the same as paragraphs)

Comments

No comments yet.

Name (required)
E-mail (required - never shown publicly)
URI