Word 'Bursts' Could Help Refine Web Searches

Join Our Community of Science Lovers!

On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

Searching the Internet can sometimes feel like looking for a shrinking needle in an expanding haystack. To that end, a new approach to sorting information that relies on scanning documents for abrupt "bursts" in the usage of particular words may help. Jon Kleinberg of Cornell University described the technique yesterday at the annual meeting of the American Association for the Advancement of Science in Denver, Colo.

A besieged e-mail inbox prompted Kleinberg to design the new system. While trying to filter his mail, he theorized that whenever an important topic arose, keywords related to it would show up in messages with increasing frequency. As a result, searching for words whose usage increased dramatically and quickly--or "burst"--could help identify significant topics and provide a way to categorize messages. Kleinberg devised a search algorithm that analyzes both the number of times words appear and the rate of increase in their frequency over time.

To test his approach, Kleinberg used the algorithm to scrutinize the full text of all the State of the Union addresses given since 1790. The "bursts" that the program identified matched important events occurring at the time certain speeches were delivered. For instance, in the aftermath of the American Revolution, "militia" and "British" were among the flagged words, whereas "atomic" displayed substantial "burst" between 1947 and 1959. Such trends are intuitive to people, Kleinberg notes, but a computer, which lacks historical context, still successfully identified them solely by scanning raw text. He posits that the new approach could help narrow web searches by better recognizing the time context of a query. In addition, sociologists or marketers might be able to identify emerging trends more adeptly by monitoring the "burstiness" of words on weblogs or e-mails to consumer web sites.

Subscribe to Support Independent Journalism

Great science journalism requires human expertise, time, effort and creativity. And it costs money. That’s why I and the journalists here at Scientific American hope you’ll join our community.

When you subscribe, you are supporting staff and freelance journalists who are passionate about telling science stories that are true, important and compelling. Our editors and reporters are often experts in their fields, which means they understand the nuances of big discoveries and can untangle the breakthroughs from the hype. With a subscription, you are also supporting rigorous fact-checking to ensure the words we publish are precise and accurate. And you’re supporting original illustrations, graphics and photos that bring you closer to an advanced laboratory, an ice sheet in Antarctica or a space mission in orbit. You’re helping us craft other types of high-quality journalism as well: Our newsletters are carefully written, edited and curated by staffers you have or will come to know and love. Our Science Quickly podcast is based on original reporting, collaboration with editors and scientists and exacting production.

Subscriptions keep this engine running so we can continue to deliver thoughtful, rigorous and independent science journalism to you. In an era of viral misinformation, this work is crucial. If you value what we do, I hope you’ll consider joining us as a subscriber

Thank you,

Jeanna Bryner, Editor in Chief, Scientific American