And actually that's typical in the history of experimental psychology and behavioral economics—you recruit a bunch of people for your experiment, they come in and participate and then they leave and you never hear from them again.
How can you use Mechanical Turk and other Web tools to improve that situation?
We'd like to build up much more persistent subject pools, or what we call panels, of online research subjects. One idea would be to pay them retainers so they're available when we need them for experiments. Then you could send them a request to participate at any point in time, although they're not obligated to. The advantage of doing it that way is—rather than grabbing whoever happens to be around, which is what we do currently—we could specify in advance that we’re interested in a particular type of person. It seems like an obvious thing to do but it requires building a bit of an infrastructure so that you can ask these sorts of questions. Once you’ve created these online panels, and they’ve participated in a number of experiments, you could create customized samples based not just on their demographic information but also on how they’ve behaved in past experiments.
In addition to improving research methods, how might the Web be used to deliver timely, meaningful research results?
I'm interested in collective problem solving—that, how groups of people, or even groups of organizations, solve complex problems. One example that I’m particularly interested is response to crisis situations, such as natural disasters. This is timely, of course, because in the aftermath of Sandy in New York you have a whole suite of organizations—local agencies like the New York Police Department and the fire department as well as national agencies like FEMA and the American Red Cross—swarming in to try to help. Immediately after a disaster you have this massive problem of uncertainty about what's happening on the ground. First responders are generally highly motivated and well meaning, and may even have a lot of experience with previous disasters, but crisis situations also have a way of differing from the past in unanticipated ways, so invariably you have a situation where nobody knows exactly who needs what, where the relevant resources are located, or how to coordinate the relief effort.
Recently, a handful of volunteer “crisis mapping” organizations such as The Standby Task Force [SBTF] have begun to make a difference in crisis situations by performing real-time monitoring of information sources such as Facebook, Twitter and other social media, news reports and so on and then superposing these reports on a map interface, which then can be used by relief agencies and affected populations alike to improve their understanding of the situation. Their efforts are truly inspiring, and they have learned a lot from experience. We want to build off that real-world model through Web-based crisis-response drills that test the best ways to communicate and coordinate resources during and after a disaster.
How might you improve upon existing crisis-mapping efforts?
The efforts of these crisis mappers are truly inspiring, and groups like the SBTF have learned a lot about how to operate more effectively, most from hard-won experience. At the same time, they’ve encountered some limitations to their model, which depends critically on a relatively small number of dedicated individuals, who can easily get overwhelmed or burned out. We’d like to help them by trying to understand in a more scientific manner how to scale up information processing organizations like the SBTF without overloading any part of the system.
How would you do this in the kind of virtual lab environment you’ve been describing?
The basic idea is to put groups of subjects into simulated crisis-mapping drills, systematically vary different ways of organizing them, and measure how quickly and accurately they collectively process the corresponding information. So for any given drill, the organizer would create a particular disaster scenario, including downed power lines, fallen trees, fires and flooded streets and homes. The simulation would then generate a flow of information, like a live tweet stream that resembles the kind of on-the-ground reporting that occurs in real events, but in a controllable way.
As a participant in this drill, imagine you’re monitoring a Twitter feed, or some other stream of reports, and that your job is to try to accurately recreate the organizer’s disaster map based on what you’re reading. So for example, you’re looking at Twitter feeds for everything during hurricane Sandy that has “#sandy” associated with it. From that information, you want to build a map of New York and the tri-state region that shows everywhere there’s been lost power, everywhere there’s a downed tree, everywhere where there’s a fire.
You could of course try to do this on your own, but as the rate of information flow increased, any one person would get overwhelmed; so it would be necessary to have a group of people working on it together. But depending on how the group is organized, you could imagine that they’d do a better or worse job, collectively. The goal of the experiment then would be to measure the performance of different types of organizations—say with different divisions of labor or different hierarchies of management—and discover which work better as a function of the complexity of the scenario you’ve presented and the rate of information being generated. This is something that we’re trying to build right now.
What's the time frame for implementing such crowd-sourced disaster mapping drills?
We’re months away from doing something like this. We still need to set up the logistics and are talking to a colleague who works as a crisis mapper to get a better understanding of how they do things so that we can design the experiment in a way that is motivated by a real problem.
How will you know when your experiments have created something valuable for better managing disaster responses?
There’s no theory that says, here’s the best way to organize n people to process the maximum amount of information reliably. So ideally we would like to design an experiment that is close enough to realistic crisis-mapping scenarios that it could yield some actionable insights. But the experiment would also need to be sufficiently simple and abstract so that we learn something about how groups of people process information that generalizes beyond the very specific case of crisis mapping.
As a scientist, I want to identify causal mechanisms in a nice, clean way and reduce the problem to its essence. But as someone who cares about making a difference in the real world, I would also like to be able to go back to my friend who’s a crisis mapper and say we did the experiment, and here’s what the science says you should do to be more effective.
Subscription Center
Rights & Permissions
See what we're tweeting about
2 Comments
Add CommentDear Scientific American,
Reply | Report Abuse | Link to thisI have an idea on how to share research over the web in a secure way. Maybe a sort of social network, such as Facebook or Twitter, could be initiated, but only with scientists and researchers (accounts established through universities and institutes). Therefore, you can easily share whatever research among each other, and have reliable, updated information. Just like a social network, but more direct and selective to those people or those interested in the science field.
Hi How are you,I have a question if you can help meHow can I recycle used motor oil to make base oil?
Reply | Report Abuse | Link to this