The end result was a huge database consisting of frames of video with people's joints marked. Twenty percent of the data was used to train the system's brain to recognize movements. Engineers kept the rest in a "ground truth" database used to test the Natal's accuracy.

Choosing the best algorithm and sifting out the essential data are central to the art of machine learning. To test Natal's ability to recognize poses, engineers show it an image from the ground truth and then generate a digital pixel map in which the greater the computer's certainty is of a pixel being correctly placed on the body, the brighter the pixel  is. Engineers test hypotheses about how to improve the performance, trying to fine-tune weak areas without regressing strong ones. The more accurately the system can recognize gestures, the more fun it will be to play the game.

Of course, Microsoft is not the only gaming company exploring gestural interfaces. Rival console-maker Sony in May demonstrated a prototype Interactive Communication Unit (ICU) with stereo video cameras and depth sensors at the Vision 2009 trade fair in Stuttgart, Germany, according to New Scientist. Sony developed ICU with the help of Atracsys, LLC, a Swiss firm that specializes in optical-tracking technology. Although Sony makes the popular PlayStation game console, the company says it is planning to promote its new technology only within the advertising industry rather than in the gaming market at this time. Rival Nintendo has not revealed any plans to allow its Wii system to function without the need for Wiimote.

Still, the controller should not disappear altogether, says Hiroshi Ishii, head of the Tangible Media Group at the Massachusetts Institute of Technology Media Lab. "I'm a strong believer of having something tangible in your hand," he says. Wiimote devices, moreover, provide haptic feedback, such as vibration or resistance that makes the action more realistic. Even for activities like Natal's soccerlike Ricochet game demo, he points out, a player might miss the simulated feeling of connecting with a physical object that a controller provides.

But Peter Molyneux, creative director of Microsoft Game Studios Europe, looks forward to a new breed of computer entertainment, because eliminating game controllers opens up more creative possibilities. "Natal is forcing me as a designer to think of this as a relationship between the player and a piece of technology," he says. "We're trying to make something that feels as if it's alive."

9 Comments

1. 1. eco-steve 06:34 PM 1/7/10

   I just realised the foolproof way of getting rid of creationist and anti-climate-change junk comments : I won't bother to read any comments at all in future!
   This may be a pity for those who post useful comments, but just reading the articles is good enough for me.

   Reply | Report Abuse | Link to this

2. 2. Frame_Dragger 07:43 PM 1/7/10

   As with speech recognition software, the issue that in some areas of life people require VAST freedom and (quite naturally) have low tolerance for error. Speech, and I suspect "movement" recognition will be such a matter, and I don't think current technology can deliver as needed.

   Reply | Report Abuse | Link to this

3. 3. Nathaniel 01:24 PM 1/8/10

   This tech also has implications in the computing field. People are already working on tech that will allow us to operate computers merely with our own gestures. If the system can recognize fingers and facial expressions and not just limbs, it could be used to work on computers that project images on glasses and are manipulated purely by our own gestures. Imagine the computer in minority report being projected onto the inside of your glasses. That's the future.

   Reply | Report Abuse | Link to this

4. 4. Frame_Dragger 02:17 PM 1/8/10

   Nathaniel, this doesn't have much to do with gestural interfaces. Using sonic transducers or various wavelengths of light seems to be far more presice where our fingers and hands are concerned. The need to model the rest of the body is just wasteful in terms of R&D (see the amount of biometric data collected) and implementation. Lets not forget that a crucial part of this whole equation lies in the ability of the computer (360 or otherwise) to process this data rapidly enough to satisfy consumers. I would refer again to tolerence for error (a lack thereof) in certain arenas as a major stumbling block. There is also the matter that this for GAMING, and there is simply no way that this will be a true 1:1 motion experience once its integrated into gameplay. The calculations that would need to be made on the fly for collison-detection aloine are massive, and playing with balls is probably already stretching the limits. You're not going to see authentic "sim fencing" as a result of this tech.
   
   non 1:1 motion control is just a gimmick, and seperate from the need for near 100% precise gestural interfaces.

   Reply | Report Abuse | Link to this

5. 5. Pet2001 05:51 PM 1/8/10

   Nothing new here. I remember visiting an VR expo in the mid 80's, where in one exhibit you could interract and play with a virtual ball using your body, arms, legs to punch, kick the ball. The system used a camera to include your real image in the virtual game. position detection used infra-red devices. I suspect Microsoft using brute force on a problem solved for years. Well, if they can make it cheap, could be interesting.

   Reply | Report Abuse | Link to this

6. 6. joebob99 in reply to Pet2001 11:37 AM 1/13/10

   Pet2001 is an idiot. If this tech was available since the 80's then it would have already been integrated into a gaming product. You clearly have no concept of what it takes to create, develop and produce a product in the real world.

   Reply | Report Abuse | Link to this

7. 7. Pet2001 08:32 PM 1/13/10

   ref.: joebob99
   
   My mistake, i think the VR expo was in late 80's / early 90's. And no, no one tried seriously to integrate the tech on gaming products for number of reasons. It was too expensive at the time, electronics where far less advanced, and these techs where for rich VR freaks and army, not exactly gamers. Albeit there was some sort of consumer VR products wave (VR helmets with position sensors, 3D shutter glasses, 3D LCD screen glasses, haptic gloves...) that followed and failed.

   Reply | Report Abuse | Link to this

8. 8. Fish 04:43 PM 1/16/10

   I'm estimating failure and dissapointment just like most Microsoft innovations. They just don't seem to be able to bring practical innovation to the market like Apple. The company is littered with too many acedemics and not enough practical thinkers. Too bad...

   Reply | Report Abuse | Link to this

9. 9. Major Plonquer 03:44 AM 1/17/10

   Dear Mr Fish,
   
   Just out of curiosity, exactly what are these amazingly great innovations that Apple have produced? We all know they ripped off Xerox for their user interface. They even stole their name from the Beatles.
   
   And, funny enough, the 'multitouch' aspects of the iPhone were first shown by that non-innovative company Microsoft in their 'Surface' product four years ago.
   
   I assume you mean the exploding batteries. Nobody else has thought of that yet and Apple are still the leaders in that market.
   

   Reply | Report Abuse | Link to this