Inpaint by Numbers

An algorithm to automate the repair of moving images

Join Our Community of Science Lovers!


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


Astute moviegoers may have noticed a traveling car and its trail of exhaust in a scene from Peter Jackson's The Lord of the Rings: The Fellowship of the Ring. Current tools, such as Adobe Systems's After Effects "healing brush," can readily camouflage, or inpaint, small blemishes. To fix larger gaffes, though, the user must tirelessly cut small patches of image and paste them over the unwanted object. The effort typically yields mediocre results for all but the smallest repairs: on the DVD of Fellowship, a blurry spot is visible where the car was.

But new software using advanced mathematics may soon enable video editors to automatically inpaint a moving object quickly and seamlessly. It can even massage away large moving objects that hide other action, according to the software's principal developer, computer scientist Guillermo Sapiro of the University of Minnesota.

It’s Time to Stand Up for Science

If you enjoyed this article, I’d like to ask for your support. Scientific American has served as an advocate for science and industry for 180 years, and right now may be the most critical moment in that two-century history.

I’ve been a Scientific American subscriber since I was 12 years old, and it helped shape the way I look at the world. SciAm always educates and delights me, and inspires a sense of awe for our vast, beautiful universe. I hope it does that for you, too.

If you subscribe to Scientific American, you help ensure that our coverage is centered on meaningful research and discovery; that we have the resources to report on the decisions that threaten labs across the U.S.; and that we support both budding and working scientists at a time when the value of science itself too often goes unrecognized.

In return, you get essential news, captivating podcasts, brilliant infographics, can't-miss newsletters, must-watch videos, challenging games, and the science world's best writing and reporting. You can even gift someone a subscription.

There has never been a more important time for us to stand up and show why science matters. I hope you’ll support us in that mission.

Thank you,

David M. Ewalt, Editor in Chief, Scientific American

Subscribe