By Sharon Weinberger of Nature magazine
It is every government's dream: a system that can predict future events such as riots, political upheavals and the outbreak of wars. Last week, a collection of academics and private businesses was scrambling to meet the deadline for proposals for research aiming to do just that.
The Intelligence Advanced Research Projects Activity (IARPA), a research arm of the US intelligence community, is sponsoring the work under the Open Source Indicators (OSI) program. The three-year project, with an unspecified budget, is designed to gather digital data from a range of sources, from traffic webcams to television to Twitter. The goal, according to IARPA, is to provide the intelligence community with predictions of social and political events that can "beat the news".
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Initially, the OSI project will focus on Latin America, which has abundant publicly available data and offers a convenient test bed for researchers' models. Those models will build on strategies that have already shown promise for predicting disease outbreaks and consumer behavior, and which are becoming increasingly popular with US national security agencies (see Nature471, 566-568; 2011).
Indeed, the OSI project is one of many being sponsored by the US national security community, which seeks to meld mathematics, computer science and economics with the social sciences, creating a new field of social and political forecasting that has often been compared to Isaac Asimov's concept of 'psychohistory'.
At the Center for Collective Intelligence at the Massachusetts Institute of Technology in Cambridge, computer scientist Peter Gloor has been working with colleagues to build models that can predict consumer behavior, such as ticket sales for Hollywood films, using a range of online sources including social media. "We're up to 90% accuracy" for predicting box-office returns, says Gloor, who is part of a team applying for OSI funding.
John Brownstein, an epidemiologist at Harvard Medical School in Boston, Massachusetts, is working with a group that analyses international news sources, government data and social media to provide an early warning of disease outbreaks. He is also applying to work on the IARPA project. "In many cases, what we are searching for are patterns of activity that would not only apply to disease events but to conflicts, environmental disasters and other forms of social disruption," he says.
However, there is disagreement about how effective such biosurveillance programs have been. Picking up hints of an H1N1 outbreak is less useful if it is swamped by false alarms, says James Wilson, a medical doctor and the co-founder of Ascel Bio in New York. "There is a huge difference between a computer harvesting data and beginning to move communications so that people begin to contemplate action," says Wilson, whose company is applying for OSI funding.
Data crunching
The approach is nevertheless catching on in the financial sector. Christopher Ahlberg, the chief executive of Recorded Future in Boston, also applying to the OSI project, says his company has proprietary software that can crunch through 300,000 sources an hour for clues to future stock movements. The company has already received investment from In-Q-Tel, a venture-capital firm in Arlington, Virginia, founded by the US Central Intelligence Agency, and is mining online data sources, such as Twitter, in an effort to predict cyber attacks and developments in the ongoing 'Occupy Wall Street' protests.
There is already plenty of published literature on predicting stock market volatility based on open sources, says Kalev Leetaru, a computer scientist at the University of Illinois at Urbana-Champaign. But that's a long way from being able to predict a riot next Wednesday. Leetaru, who is also part of a team applying to be part of the OSI project, says he asked IARPA whether it would consider a broader approach that involved predicting trends, such as the Arab Spring movement, rather than focusing on specific events (see Nature; 2011). The answer, he says, was "no".
"In my mind," Leetaru says,"this would be akin to the National Institutes of Health announcing it was terminating all funding for work on cancer drugs that targeted individual cancers or that slowed cancer down, and instead announcing that they would only fund a single miracle pill that you take and your cancer is cured the next morning."
Robert Albro, an anthropologist at American University in Washington DC, and an expert in Latin American social movements, believes that IARPA is mistakenly presuming that social media will provide high-quality data. "Just because data are available doesn't make them good, or the key to the kingdom," he says.
A bigger question is whether models used to measure consumer preferences and disease outbreaks are applicable to the complex world of social change and political events. Albro says that such models make faulty assumptions about what motivates humans, and he worries "that companies concerned about consumer behavior are now driving how IARPA thinks. That's a leap of faith.
This article is reproduced with permission from the magazine Nature. The article was first published on October 17, 2011.
