Last year, as has been widely reported, the Pentagon started a program called Total Information Awareness to link databases of personal information and scan them for signs of terrorist threats. Officials there say that every credit-card purchase you make, every prescription you fill, every phone call you place could go into a government computer. The Transportation Security Administration has similar goals for version 2.0 of its Computer Assisted Passenger Prescreening System (CAPPS). Leaving aside the possible implications for civil liberties, would such systems really make us more secure?

Homeland Security officials and private contractors gush about the potential for "data mining." But for scientists--unlike, say, marketers--data mining is something of a dirty word. It connotes a blind search through data, an effort that tends to confuse real patterns with mere coincidences. In the past decade, many statisticians have rehabilitated the word and tried to inject more rigor into the procedure. The government programs, however, are bumping up against fundamental limitations.

This article was originally published with the title Total Information Overload.

Comments