At Perceptive Pixel's offices on Manhattan's West Side, Jefferson Han stands in front of a megasize multi-touch screen and runs his fingertips across the display. Each finger leaves a trail of colored pixels in its wake, causing the display to look, briefly, like it has been scratched by a set of digital claws.

Han, who founded Perceptive Pixel in 2006 and serves as the company's chief scientist, next uses his index finger to draw a loop on the 100-inch display. It causes a menu of options to appear on the screen, not all that different from using a mouse to click on a drop-down menu. Of course, Han doesn't need a mouse, or a keyboard for that matter. He selects one of his menu options, which are arranged in a loop resembling the one he drew to pull up the menu, and away he goes.

Perceptive Pixel's technology is best known for the role it played in helping cable news network CNN create its "Magic Wall," a dynamic, on-air graphical representation of voting results during the network's coverage of the 2008 presidential election. Although Han acknowledges that multi-touch is "interesting" at the level of gadgets such as Apple's iPhone, the technology comes into its own only when you're able to manipulate objects on a display screen using both hands.

Creating software to make multi-touch work properly is the hardest part of developing these systems. "The interface is such a paradigm shift," he says. "You can't just bolt the software onto existing hardware." For this reason, Han and his colleagues have essentially created new operating system for touch-sensitive technology that recognizes an infinite number of touch points—ideally, allowing multiple users to manipulate the same display simultaneously.

We sat down with Han to talk about his approach to multi-touch, how it differs from offerings by consumer electronic device-makers, and where the technology is headed.

[An edited transcript of the interview follows.]

What prompted you to start researching and developing multi-touch sensing technology?
The user's interface with the computer, not the back-end processing horsepower, has become the bottleneck to the user, who accesses information via a straw, in the form of a mouse and keyboard. The input needs more attention, but it's very difficult to impose a new interface on people. Still, as you see computing in more places people realize you can ask for more than just the keyboard. As much as I hate [the movie] Minority Report, it had some impact on people realizing that there's more to computing that simply doing one thing at a time, you can bring your other hand around and use that too. One way to approach a new interface is to ask, "What would a child do with this?"
 
How does your multi-touch work build on your previous research in the areas of computer graphics, machine learning, real-time computer vision and human-computer interfaces?
My interest is in the visual, that's really why I got into programming and in particular, computer graphics. Graphics has lately made a great shift towards machine learning, which itself is about understanding data. That's where the computer vision work came from. Computer vision is the inverse problem of computer graphics. Instead of telling the computer what the state of the world is, the computer has to figure it out [using e.g. visual cues]. How do you get computers to understand a scene? How do you clean up the sensor information [that the computers are receiving]? Doing this involves using strong engineering and math components.

7 Comments

1. 1. hotblack 09:37 PM 3/5/10

   I like to think that infrared sensors will be able to maintain a lock on individual users in front of a device.
   
   Actually, I like to imagine not having to touch the screen at all, but have the system identify your movements and gestures via your infrared signature... but that is yet another step into the future.

   Reply | Report Abuse | Link to this

2. 2. FellOnEarth 04:08 AM 3/6/10

   I'm really pleased to know Han continues to be a pioneer in touch technology. I imagine the solution to improving a touchscreen's ability to differentiate between various users' gestures lies in biometric recognition. Ultimately, what would require a screen that not only displays, but also sees. IR solutions exist (Han has actually shown how easy it is to implement using a Nintendo Wii paddle in reverse to simulate hands free responsiveness for an individual user, a user would need IR reflective gloves and headgear), the use of an accelerometer also expands "hands free" usage and interface, but that still requires a peripheral component. I think the point of the touchscreen though is to get away from using such peripherals in preference of a tactile, easy to use and immediately responsive interface.

   Reply | Report Abuse | Link to this

3. 3. JamesDavis 07:15 AM 3/6/10

   Fantastic work Han. I hope you, and some other very intelligent people, rapidly develope this computer and get it on the market soon.

   Reply | Report Abuse | Link to this

4. 4. sapbucket 10:03 AM 3/6/10

   "...on people realizing that there's more to computing that simply doing one thing at a time, you can bring your other hand around and use that too."
   
   ...I think we all know what the other hand is doing

   Reply | Report Abuse | Link to this

5. 5. dr@gonaut 08:18 PM 3/6/10

   I wonder if Han worked on this project with a team, or was he... Solo?

   Reply | Report Abuse | Link to this

6. 6. cloverstreet 12:40 PM 3/7/10

   Important to distinguish - near IR from true IR. All of these systems, wiis, Han's table, remotes, etc. use near-IR (860nm-940nm) range. There is no "IR signature" in this range. Going to true IR is very expensive. Han is one of the first people on the multi-touch scene. He got his idea for using FTIR from fingerprint scanner devices. It is basically a macro version of the same thing. He was (is?) a professor at NYU, and had quite a bit of help from his students in the beginning. I don't know if it is true but I've heard rumors that he got a large amount of his funding for perceptive pixel by selling patents for gestures (a ridiculas idea in my opinion) to apple.

   Reply | Report Abuse | Link to this

7. 7. robert schmidt 03:07 PM 3/8/10

   It's cool looking but fingers make a mess of a computer screen pretty quickly. Having the computer interact by watching the user's movements resolves that but there needs to be some type of feedback and a way to know when the user is interacting. A pair of input gloves that contained feedback devices combined with shutter goggles for 3D interaction would be powerful. Ultimately, no one input device will fit all.

   Reply | Report Abuse | Link to this