When Apple’s iPhone hit the streets last year, it introduced so-called multi-touch screens to the general public. Images on the screen can be moved around with a fingertip and made bigger or smaller by placing two fingertips on the image’s edges and then either spreading those fingers apart or bringing them closer together. The tactile pleasure the interface provides beyond its utility quickly brought it accolades. The operations felt intuitive, even sensuous. But in laboratories around the world at the time of the iPhone’s launch, multi-touch screens had vastly outgrown two-finger commands. Engineers have developed much larger screens that respond to 10 fingers at once, even to multiple hands from multiple people.
It is easy to imagine how photographers, graphic designers or architects—professionals who must manipulate lots of visual material and who often work in teams—would welcome this multi-touch computing. Yet the technology is already being applied in more far-flung situations in which anyone without any training can reach out during a brainstorming session and move or mark up objects and plans.
Perceptive Pixels
Jeff Han, a consulting computer scientist at New York University and founder of Perceptive Pixel in New York City, is at the forefront of multi-touch technology. Walking into his company’s lobby, one is greeted by a three-by-eight-foot flat screen. Han steps up to the electronic wall and unleashes a world of images using nothing but the touch of his fingers. As many as 10 or more video feeds can run simultaneously, and there is no toolbar in sight. When Han wants the display to access different files he taps it twice, bringing up charts or menus that can also be tapped.
Several early adopters have purchased complete systems, including intelligence agencies that need to quickly compare geographically coordinated surveillance images in their war rooms. News anchors on CNN used a big Perceptive Pixel system during coverage of the presidential primaries that boldly displayed all 50 U.S. states; to depict voting results, the anchors, standing in front of the screen, dramatically zoomed in and out of states, even counties, simply by moving their fingers across the map. Looking ahead, Han expects the technology to find a home in graphically intense businesses such as energy trading and medical imaging.
Rudimentary work on multi-touch interfaces dates to the early 1980s, according to Bill Buxton, a principal researcher at Microsoft Research. But around 2000, at N.Y.U., Han began a journey to overcome one of the technology’s toughest hurdles: achieving fine-resolution fingertip sensing. The solution required both hardware and software innovations.
Perhaps most fundamental was exploiting an optical effect known as frustrated total internal reflection (FTIR), which is also used in fingerprint-recognition equipment. Han, who describes himself as “a very tactile person,” became aware of the effect one day when he was looking through a full glass of water. He noticed how crisply his fingerprint on the outside of the glass appeared when viewed through the water at a steep angle. He imagined that an electronic system could optically track fingertips placed on the face of a clear computer monitor. Thus began his six-year absorption with multi-touch interfaces.
He first considered building a very high resolution version of the single-touch screens used in automated teller machines and kiosks, which typically sense the electrical capacitance of a finger touching predefined points on the screen. But tracking a randomly moving finger would have required an insane amount of wiring behind the screen, which also would have limited the screen’s functionality. Han ultimately devised a rectangular sheet of clear acrylic that acts like a waveguide, essentially a pipe for light waves. Light-emitting diodes (LEDs) around the edges pump infrared light into the sheet. The light streams through, reflecting internally off the sheet walls, much as light flows though an optical fiber. No light leaks out. But when someone places a finger on one face of the sheet, some of the internally reflecting light beams hit it and scatter off, bouncing through the sheet and out the opposite face. Cameras behind the screen sense this leaking light, or FTIR, revealing the location being touched. The cameras can track this leakage from many points at once.
Han soon discovered that the acrylic panel could also serve as a diffusion screen; a projector behind the panel, linked to a computer, could beam images toward it, and they would diffuse through to the other side. The screen could therefore serve as both an output of imagery and an input of touches made on that imagery.
Sensing the exact location of fingers was one challenge. Devising software routines that could track the finger movements and convert them to instructions for what should be happening with images on the screen was tougher. The half a dozen software developers working with Han had to first write software that would function as a high-performance graphics engine, in part to give the display low latency, or ghosting, when fingers dragged objects quickly across the screen. Then they had to deal with the screen’s unorthodox FTIR light output from fingertips sweeping around in random directions.
Deep in the architecture of a computer’s operating system is an assumption that a user’s inputs will come either from a keyboard or a mouse. Keystrokes are unambiguous; a “q” means “q.” The movement of a mouse is expressed as Cartesian coordinates—x and y locations on a two-dimensional grid. Such methods for representing inputs belong to a general discipline known as the graphical user interface, or GUI. Han’s multi-touch screen generates 10 or more streams of x and y coordinates at the same time, and “the traditional GUIs are really not designed for that much simultaneity,” he notes. The current operating systems—Windows, Macintosh, Linux—are so predicated on the single mouse cursor that “we had to tear up a lot of plumbing to make a new multi-touch graphical framework,” Han says.
During all this work, Han found that pressure sensing could be accomplished, too, by applying to the front of the acrylic screen a thin layer of polymer with microscopic ridges engineered into its surface. When a user presses harder or more softly on any spot on the polymer, it flexes slightly, and the fingerprint area becomes larger or smaller, causing the scattered light to become brighter or darker, which the camera can sense. By maintaining firm pressure on an object on the screen, a user can slide it behind an adjacent object.
Han’s Perceptive Pixel team, formed in 2006, put all the elements together and demonstrated the system at the TED (for technology, entertainment and design) conference that year to an enthusiastic audience. Since then, orders for the system have steadily increased. Perceptive Pixel is not disclosing prices.
Microsoft Scratches the Surface
While Han was perfecting his setup, engineers elsewhere were pursuing similar goals by different means. Software giant Microsoft is now rolling out a smaller multi-touch computer called Surface and is trying to brand this category of hardware as “surface computers.” The initiative dates back to 2001, when Stevie Bathiche of Microsoft Hardware and Andy Wilson of Microsoft Research began developing an interactive tabletop that could recognize certain physical objects placed on it. The two innovators envisioned that the tabletop could function as an electronic pinball machine, a video puzzle or a photo browser.
More than 85 prototypes later, the pair ended up with a table that has a clear acrylic top and houses a projector on the floor below. The projector sends imagery up onto the horizontal, 30-inch screen. An infrared LED shines light up to the tabletop as well, which bounces off fingertips or objects on the other side, thus allowing the device to recognize commands from people’s fingers. A Windows Vista computer provides the processing.
Microsoft is shipping Surface table computers to four partners in the leisure, retail and entertainment industries, which it believes are most likely to apply the technology. Starwood Hotels’ Sheraton chain, for example, will try installing surface computers in hotel lobbies that will let guests browse and listen to music, send home digital photographs, or order food and drinks. Customers in T-Mobile USA’s retail stores will be able to compare different cell phone models by simply placing them atop a surface screen; black-dotted “domino” tags on the undersides of the phones will cue the system to display price, feature and phone plan details. Other Microsoft software will allow a wireless-enabled digital camera, when placed on a surface computer, to upload its photographic content to the computer without a cable.
First-generation surface systems are priced from $5,000 to $10,000. As with most electronic items, the company expects the price to decline as production volume increases. Microsoft says Surface computers should be available at consumer prices in three to five years.
Mitsubishi Wired In, Too
Technology developers might be interested in the DiamondTouch table from a start-up company called Circle Twelve in Framingham, Mass., that was recently spun off from Mitsubishi Electric Research Laboratories. The table, developed at Mitsubishi, is configured so that outside parties can write software for applications they envision; several dozen tables are already in the hands of academic researchers and commercial customers.
The purpose of DiamondTouch “is to support small-group collaboration,” says Adam Bogue, Mitsubishi’s vice president of marketing. “Multiple people can interact, and the system knows who’s who.” Several people sit in chairs that are positioned around the table and are linked to a computer below. When one of them touches the tabletop, an array of antennas embedded in the screen sends an extremely small amount of radio-frequency energy through the person’s body and chair to a receiver in the computer, a scheme known as capacitive coupling. Alternatively, a special floor mat can be used to complete the circuit. The antennas that are coupled indicate the spot on the screen that the person is touching.
Though seemingly restrictive, this setup can keep track of who makes what inputs, and it can give control to whoever touches the screen first. In that case it will ignore other touches, sensed through the assigned seating, until the first user has completed his or her inputs. The system can also track who makes which annotations to images, such as blueprints.
Parsons Brinckerhoff, a global engineering firm headquartered in New York City, has been experimenting with the tables and plans to acquire more. “We have thousands of meetings during the course of a big project,” says Timothy Case, the company’s visualization department regional manager. “We could have multiple tables in multiple locations, and everybody can be looking at the same thing.”
Both the DiamondTouch and Perceptive Pixel systems feature keyboard “emulators” that shine a virtual keyboard onto the screen so that people can type. But it seems unlikely that enthusiasts would prefer to use the dynamic systems for this mundane activity. The great strength of multi-touch is letting multiple people work together on a complex activity. It is hard to remember how liberating the mouse seemed when it freed people from keyboard arrow keys some 25 years ago. Soon the multi-touch interface could help untether us from the ubiquitous mouse. “It’s very rare that you come upon a really new user interface,” Han says. “We’re just at the beginning of this whole thing.”
Note: This story was originally published with the title, "Hands-On Computing".
