New Issue: Orbital Catastrophe Ahead? Read Now

Inpaint by Numbers

An algorithm to automate the repair of moving images

Join Our Community of Science Lovers!


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


Astute moviegoers may have noticed a traveling car and its trail of exhaust in a scene from Peter Jackson's The Lord of the Rings: The Fellowship of the Ring. Current tools, such as Adobe Systems's After Effects "healing brush," can readily camouflage, or inpaint, small blemishes. To fix larger gaffes, though, the user must tirelessly cut small patches of image and paste them over the unwanted object. The effort typically yields mediocre results for all but the smallest repairs: on the DVD of Fellowship, a blurry spot is visible where the car was.

But new software using advanced mathematics may soon enable video editors to automatically inpaint a moving object quickly and seamlessly. It can even massage away large moving objects that hide other action, according to the software's principal developer, computer scientist Guillermo Sapiro of the University of Minnesota.

Subscribe to Support Independent Journalism

Great science journalism requires human expertise, time, effort and creativity. And it costs money. That’s why I and the journalists here at Scientific American hope you’ll join our community.

When you subscribe, you are supporting staff and freelance journalists who are passionate about telling science stories that are true, important and compelling. Our editors and reporters are often experts in their fields, which means they understand the nuances of big discoveries and can untangle the breakthroughs from the hype. With a subscription, you are also supporting rigorous fact-checking to ensure the words we publish are precise and accurate. And you’re supporting original illustrations, graphics and photos that bring you closer to an advanced laboratory, an ice sheet in Antarctica or a space mission in orbit. You’re helping us craft other types of high-quality journalism as well: Our newsletters are carefully written, edited and curated by staffers you have or will come to know and love. Our Science Quickly podcast is based on original reporting, collaboration with editors and scientists and exacting production.

Subscriptions keep this engine running so we can continue to deliver thoughtful, rigorous and independent science journalism to you. In an era of viral misinformation, this work is crucial. If you value what we do, I hope you’ll consider joining us as a subscriber

Thank you,

Jeanna Bryner, Editor in Chief, Scientific American

Subscribe