Rich Jenkins opens a child’s picture book and aims a camera phone at a page depicting a cartoon panda bear that is gesturing toward a set of Chinese characters. As Jenkins and I view the page through the cell phone screen, the printed panda suddenly erupts into a 3-D video version that points at the first symbol, pronounces it in Mandarin and then defines it in English.

Jenkins, who leads Media Power, a New York City–based firm that develops mobile communications applications, smiles at my rather startled reaction. “A software application that we’ve downloaded into this phone reads cues that the book designers have embedded into the graphics,” he explains. “It then calls up the video segment appropriate for that page from the network server. The result is like a pop-up book on steroids.” Jenkins notes that this new kind of animated content could help kids learn and that these “magic books” could become available by the end of this year. The company will also be introducing cell phone–enabled museum exhibit tours based on the same technology, as well as the means by which consumers can trigger delivery of targeted advertising by directing camera phones at brand logos.

Media Power is part of a vanguard of organizations that is working to commercialize augmented-reality (AR) technology, which can be characterized as the timely overlay of useful virtual information onto the real world. According to Mark Billinghurst, director of the Human Interface Technology Laboratory at the University of Canterbury in New Zealand, AR incorporates three key features: virtual information that is tightly registered, or aligned, with the real world; the ability to deliver information and interactivity in real time; and seamless mixing of real-world and virtual information.

When explaining AR technology, Blair MacIntyre, who directs the Augmented Environments Laboratory at the Georgia Institute of Technology, often invokes the virtual first-down marker seen as a yellow stripe in televised football games. “The technical challenge of AR is to do something similar but more complex with the live video feed from a cell phone camera and without the 10-second delay required to generate the virtual marker.”

Although AR has mostly lived in the lab (except, notably, in the form of head-up displays for fighter jets), the recent emergence of highly capable mobile devices is fueling a surge in interest. (Much of the AR technology on cell phones is based on work done at Graz University of Technology in Austria.) “I think that we’re on the cusp of widespread application of AR technology, perhaps in a year or two,” Billinghurst says, pointing to the Eye of Judgment, a video game for Sony’s PlayStation 3, as the most prominent example of the trend. Players look at cards through a camera and watch animated versions of the game characters on the cards fight one another. The ability is based on identifying real-world objects and estimating their locations in space.

AR-like technology is also finding its way into industrial manufacturing. Inter­Sense, a Bedford, Mass.–based company, offers process-verification systems that use sensors and cameras to track the positions and motions of tools as workers do their jobs. Computers then compare the actual tool movements with ideal procedures to detect errors or confirm correct completion, information that is then provided graphically to the workers in real time.

If today’s trends hold true, more AR-based products will arrive before long. Commercial entities that have entered the field are split among suppliers of AR-authoring and development tools, including ARToolWorks in Seattle, Germany’s Metaio and France’s Total Immersion, and large companies such as Sony, Canon, Qualcomm, Motorola and Nokia. Billing­hurst estimates that around 40 academic labs shell out a combined $50 million to $60 million every year on AR research and that commercial firms spend two to three times that. Progress in AR depends on advances in display technologies (“virtual” eyeglasses, for example), tracking systems, cameras, and processors and graphics chips for mobile devices, as well as the means to deliver AR services wirelessly where and when users need them.

Widespread use of AR, though, will probably depend on integrating AR with social networking, Billinghurst states. Such a mix, he says, would, for example, “allow users to leave annotation notes—advice or opinions—for their friends on the network at sites such as restaurants or scenic spots all over the world.” Reality would take on a whole new meaning.

Note: This article was originally printed with the title, "Annotating the Real World".