<p>The closest thing I know of to the "<em>computer: enhance!</em>" trope in real life is the family of <a href="http://www.wisdom.weizmann.ac.il/~vision/SingleImageSR.html" rel="nofollow">Single Image Super-Resolution</a> techniques. That page shows a number of examples of the results on various images. You can see that while it improves the visual quality of the enlarged images, it's a long way from what you see on TV where they can read the text on a letter that's reflected in a wine glass, or recognize the face of someone standing in the shadows, etc.</p>

<p>The technique basically works by observing that a single image often contains recurring instances of the same patterns or structures. When you have multiple copies of the same pattern, they often occur at different sub-pixel offsets, i.e. aligned differently to the pixel grid, which means that each copy contains slightly different information about the underlying pattern, and by putting them together you can recover (really, guess at) a higher-resolution version of that pattern. This can then be used to "fill in the missing detail" wherever that pattern occurs. This is a very handwavy explanation, but you can see the paper (linked from the above site) for more details. Most of it is automated, but I'd guess it still requires a fair amount of human parameter-tuning to get good results.</p>

<p>The paper is from 2009, and it looks like there have been a few follow-ups since then, but only incremental improvements.</p>

<p>Also note that if you have a specific use case such as reading vehicle license plates, I'd guess it's possible to use machine-learning techniques to get much better results than you can in general. In that case, you have a concrete, known set of possible shapes (the characters that can appear on a license plate, in the particular font used for license plates), and rather than "enhancing" an image of literally anything, you're just trying to find the set of characters that best matches the image you're looking at. That's not my area of expertise, though.</p>

CC BY-SA 3.0
2016-01-09T02:18:06.283Z
I haven't heard of anyone doing it yet, but I really think temporal techniques could be used to get sub pixel accuracy from video streams.
Alan Wolfe
CC BY-SA 3.0
2016-01-09T04:53:26.357Z
@AlanWolfe Definitely. Googling "video super-resolution" turns up a number of papers on that idea.
Nathan Reed
CC BY-SA 3.0
2016-01-09T17:40:34.587Z
I did toyed with SISR a while ago, but my mind cleverly decided to erase that information from the brain. If I can recall though one important bit (mentioned in the paper) that I would explicit in your excellent answer is that the patterns or substructures are to be searched not only in the original image but also in multiple scales of it. That IIRC lead to significantly better results, although as I said I don't remember that much details :(
cifz
CC BY-SA 3.0
2016-01-08T22:50:50.167Z
2016-01-08T22:50:50.167Z
Nathan Reed
<p>TWO PART QUESTION - We've all seen the movies/TV shows where the police/feds/spies use computer software to take a grainy photo and do a "clean-up" to see a better picture and more details. I assume the concept is executed by some sort of <strong>uber-power</strong> pixel-smoothing or anti-aliasing type of algorithms to fill in the blanks based on deductive processing.</p>

<p>PART 1: How <em>real</em> is this technology in the public/commercial software world? I'm not asking about any <em>speculation</em> on alleged secret gov software or such, I just want to know where we actually are with this concept today? How much is fully automated vs human-assisted. </p>

<p>PART 2: Assuming there actually is <em>some</em> reality with this technology for photographs the second part of this question is how (if at all) has this been applied to videos? Again the issue of fully automated vs human assisted is of interest here.</p>

<p>At the heart of this post is the ultimate question of how viable is today's software for being able to take an old VHS or DVD recording and process the frames to create a new HD-resolution remaster. Considering that doing this would mean cleaning up tens-of-thousands of frames for even a simple wedding video I am <strong>not</strong> expecting this technology to be fast of course.</p>

<hr>

<hr>

<p>NOTE: Per in-topic and meta discussions I went ahead and cross-posted this in two other suggested SE forums to acquire their viewpoints and expertise on this matter. So far (it is still a little "early in the day" so to speak) I have received some pretty interesting information in 2 of the 3.</p>

<p>When this is all done (when good answers have been selected) I would appreciate a way to merge these <strong>for the benefit of all three SE communities</strong>:</p>

<ul>
<li>Computer Graphics: <a href="https://computergraphics.stackexchange.com/questions/1880">What is the state-of-the-art on using computers to &quot;clean-up&quot; images?</a></li>
<li>Signal Processing: <a href="https://dsp.stackexchange.com/questions/28168">https://dsp.stackexchange.com/questions/28168</a></li>
<li>Video Production: <a href="https://video.stackexchange.com/questions/17363">https://video.stackexchange.com/questions/17363</a></li>
</ul>

CC BY-SA 3.0
2016-01-08T20:41:33.267Z
This question is about image / signal processing, which is not within the scope of this site. However, there is a SE for signal processing here: http://dsp.stackexchange.com/
RichieSams
CC BY-SA 3.0
2016-01-08T21:19:06.850Z
increase contrast add more frames from video feed. Most of it remains guesswork though
ratchet freak
CC BY-SA 3.0
2016-01-08T23:15:37.607Z
[Meta discussion](http://meta.computergraphics.stackexchange.com/questions/201/is-this-graphics-or-image-processing-is-there-an-overlap) on whether image processing is on topic.
trichoplax is on Codidact now
CC BY-SA 3.0
2016-01-09T07:13:44.533Z
Is there a way to move this question to the other SE or should I just cut/paste it over there ?
O.M.Y.
CC BY-SA 3.0
2016-01-09T15:55:40.803Z
Per these comments and @trichoplax 's meta discussion I went ahead and cross-posted this in two other SE forums. Here are the links to those discussions: http://dsp.stackexchange.com/questions/28168/what-is-the-state-of-the-art-on-using-computers-to-clean-up-videos?noredirect=1#comment52603_28168 and http://video.stackexchange.com/questions/17363/what-is-the-state-of-the-art-on-using-computers-to-clean-up-videos
O.M.Y.
CC BY-SA 3.0
2016-01-09T16:00:30.437Z
Although this was with good intentions, note that [cross posting on several SE sites is not recommended](http://meta.stackexchange.com/questions/64068/is-cross-posting-a-question-on-multiple-stack-exchange-sites-permitted-if-the-qu).
trichoplax is on Codidact now
CC BY-SA 3.0
2016-01-09T16:02:31.760Z
@O.M.Y. note that despite the comment about being off topic, the only response so far on meta is to say that this question is definitely on topic here. It will take time to see what other responses come in, but I don't see any reason to move this question at present unless you have your own reasons to.
trichoplax is on Codidact now
CC BY-SA 3.0
2016-01-09T16:16:52.227Z
Also cross posting is considered rude.
joojaa
CC BY-SA 3.0
2016-01-09T16:17:25.070Z
@trichoplax While it is likely that some of the people in the three SE communities will be joined in all three, there is no guarantee that experts in VIDEO will be part of DSP or CG and vice-versa. As the question seems to be on topic for all three I would like to *gather wool* from each community, give credit where credit is due to each expert, then merge the information into a really good answer for all three SE communities. I am a futurist and it seems reasonable to me that these three fields will simultaneously become both more *integrated* and more *specialized* as we progress.
O.M.Y.
CC BY-SA 3.0
2016-01-09T16:19:46.647Z
@joojaa I asked about that in a meta discussion and was told to go ahead and post it in the other forum. If I was given bad information then I apologize, but others have not complained and one even helped me to add the cross post links into the comments.
O.M.Y.
CC BY-SA 3.0
2016-01-09T16:22:05.063Z
You were not asked to post you were told there are other avenues by one user. Cross posting is not the stackexhange way never. If you have a question you need to make it worthy of the site your asking. So you cant ask the same question on video for example.
joojaa
CC BY-SA 3.0
2016-01-09T16:23:34.663Z
@O.M.Y. the meta post I linked to about cross posting shows community consensus is against cross posting, but there are also answers in favour of cross posting in certain rare circumstances, which are worth reading. Even then they recommend tailoring your question to different sites rather than just copy and pasting.
trichoplax is on Codidact now
CC BY-SA 3.0
2016-01-09T16:25:40.903Z
The reason that meta post exists is because it is not immediately obvious why cross posting would be a bad thing. It's well worth reading through. If there is going to be ill feeling on our site due to this it might be worth having a local meta discussion to see how our particular community feels about when and whether to cross post.
trichoplax is on Codidact now
CC BY-SA 3.0
2016-01-09T16:28:00.320Z
@joojaa This is where I was told to "go ahead and post" in VIDEO ... http://meta.video.stackexchange.com/questions/1467/about-asking-a-question-in-two-different-se-communities/1468#1468
O.M.Y.
CC BY-SA 3.0
2016-01-09T16:28:56.570Z
Still i think you should re word your question anyway. Video will get you a different answer.
joojaa
CC BY-SA 3.0
2016-01-09T16:31:39.993Z
Also user @OlliNiemitalo (2000+ rep) helped me add the cross links in DSP without complain... again if I was provided bad information I apologize but I did ask and I did feel it was okay based on those responses. Now I feel confused and wonder if I should just delete it all and shut up.
O.M.Y.
CC BY-SA 3.0
2016-01-09T16:35:06.137Z
@O.M.Y. you certainly shouldn't feel bad about posting as you have sought advice in advance. There will always be conflicting opinions about things. If you want to talk about it further I recommend [chat] rather than trying to fit it into comments here.
trichoplax is on Codidact now
<image-processing><photo-realistic><video>
What is the state-of-the-art on using computers to "clean-up" images?
O.M.Y.
# What is the state-of-the-art on using computers to "clean-up" images?

(Asked by: username_0 on 2016-01-08)

TWO PART QUESTION - We've all seen the movies/TV shows where the police/feds/spies use computer software to take a grainy photo and do a "clean-up" to see a better picture and more details. I assume the concept is executed by some sort of **uber-power** pixel-smoothing or anti-aliasing type of algorithms to fill in the blanks based on deductive processing.

PART 1: How *real* is this technology in the public/commercial software world? I'm not asking about any *speculation* on alleged secret gov software or such, I just want to know where we actually are with this concept today? How much is fully automated vs human-assisted.

PART 2: Assuming there actually is *some* reality with this technology for photographs the second part of this question is how (if at all) has this been applied to videos? Again the issue of fully automated vs human assisted is of interest here.

At the heart of this post is the ultimate question of how viable is today's software for being able to take an old VHS or DVD recording and process the frames to create a new HD-resolution remaster. Considering that doing this would mean cleaning up tens-of-thousands of frames for even a simple wedding video I am **not** expecting this technology to be fast of course.

---

---

NOTE: Per in-topic and meta discussions I went ahead and cross-posted this in two other suggested SE forums to acquire their viewpoints and expertise on this matter. So far (it is still a little "early in the day" so to speak) I have received some pretty interesting information in 2 of the 3.

When this is all done (when good answers have been selected) I would appreciate a way to merge these **for the benefit of all three SE communities**:

* Computer Graphics: [What is the state-of-the-art on using computers to "clean-up" images?](https://computergraphics.stackexchange.com/questions/1880)
* Signal Processing: <https://dsp.stackexchange.com/questions/28168>
* Video Production: <https://video.stackexchange.com/questions/17363>

**Tags:** image-processing, photo-realistic, video

> **Comments:**
> **username_1**: This question is about image / signal processing, which is not within the scope of this site. However, there is a SE for signal processing here: http://dsp.stackexchange.com/  
> **username_2**: increase contrast add more frames from video feed. Most of it remains guesswork though  
> **username_3**: [Meta discussion](http://meta.computergraphics.stackexchange.com/questions/201/is-this-graphics-or-image-processing-is-there-an-overlap) on whether image processing is on topic.  
> **username_0**: Is there a way to move this question to the other SE or should I just cut/paste it over there ?  
> **username_0**: Per these comments and @username_3 's meta discussion I went ahead and cross-posted this in two other SE forums. Here are the links to those discussions: http://dsp.stackexchange.com/questions/28168/what-is-the-state-of-the-art-on-using-computers-to-clean-up-videos?noredirect=1#comment52603_28168 and http://video.stackexchange.com/questions/17363/what-is-the-state-of-the-art-on-using-computers-to-clean-up-videos  
> **username_3**: Although this was with good intentions, note that [cross posting on several SE sites is not recommended](http://meta.stackexchange.com/questions/64068/is-cross-posting-a-question-on-multiple-stack-exchange-sites-permitted-if-the-qu).  
> **username_3**: @O.M.Y. note that despite the comment about being off topic, the only response so far on meta is to say that this question is definitely on topic here. It will take time to see what other responses come in, but I don't see any reason to move this question at present unless you have your own reasons to.  
> **username_4**: Also cross posting is considered rude.  
> **username_0**: @username_3 While it is likely that some of the people in the three SE communities will be joined in all three, there is no guarantee that experts in VIDEO will be part of DSP or CG and vice-versa. As the question seems to be on topic for all three I would like to *gather wool* from each community, give credit where credit is due to each expert, then merge the information into a really good answer for all three SE communities. I am a futurist and it seems reasonable to me that these three fields will simultaneously become both more *integrated* and more *specialized* as we progress.  
> **username_0**: @username_4 I asked about that in a meta discussion and was told to go ahead and post it in the other forum. If I was given bad information then I apologize, but others have not complained and one even helped me to add the cross post links into the comments.  
> **username_4**: You were not asked to post you were told there are other avenues by one user. Cross posting is not the stackexhange way never. If you have a question you need to make it worthy of the site your asking. So you cant ask the same question on video for example.  
> **username_3**: @O.M.Y. the meta post I linked to about cross posting shows community consensus is against cross posting, but there are also answers in favour of cross posting in certain rare circumstances, which are worth reading. Even then they recommend tailoring your question to different sites rather than just copy and pasting.  
> **username_3**: The reason that meta post exists is because it is not immediately obvious why cross posting would be a bad thing. It's well worth reading through. If there is going to be ill feeling on our site due to this it might be worth having a local meta discussion to see how our particular community feels about when and whether to cross post.  
> **username_0**: @username_4 This is where I was told to "go ahead and post" in VIDEO ... http://meta.video.stackexchange.com/questions/1467/about-asking-a-question-in-two-different-se-communities/1468#1468  
> **username_4**: Still i think you should re word your question anyway. Video will get you a different answer.  
> **username_0**: Also user @OlliNiemitalo (2000+ rep) helped me add the cross links in DSP without complain... again if I was provided bad information I apologize but I did ask and I did feel it was okay based on those responses. Now I feel confused and wonder if I should just delete it all and shut up.  
> **username_3**: @O.M.Y. you certainly shouldn't feel bad about posting as you have sought advice in advance. There will always be conflicting opinions about things. If you want to talk about it further I recommend [chat] rather than trying to fit it into comments here.  

---

## 1 answers

### Answer by username_5 on 2016-01-08. Score: 2

The closest thing I know of to the "*computer: enhance!*" trope in real life is the family of [Single Image Super-Resolution](http://www.wisdom.weizmann.ac.il/~vision/SingleImageSR.html) techniques. That page shows a number of examples of the results on various images. You can see that while it improves the visual quality of the enlarged images, it's a long way from what you see on TV where they can read the text on a letter that's reflected in a wine glass, or recognize the face of someone standing in the shadows, etc.

The technique basically works by observing that a single image often contains recurring instances of the same patterns or structures. When you have multiple copies of the same pattern, they often occur at different sub-pixel offsets, i.e. aligned differently to the pixel grid, which means that each copy contains slightly different information about the underlying pattern, and by putting them together you can recover (really, guess at) a higher-resolution version of that pattern. This can then be used to "fill in the missing detail" wherever that pattern occurs. This is a very handwavy explanation, but you can see the paper (linked from the above site) for more details. Most of it is automated, but I'd guess it still requires a fair amount of human parameter-tuning to get good results.

The paper is from 2009, and it looks like there have been a few follow-ups since then, but only incremental improvements.

Also note that if you have a specific use case such as reading vehicle license plates, I'd guess it's possible to use machine-learning techniques to get much better results than you can in general. In that case, you have a concrete, known set of possible shapes (the characters that can appear on a license plate, in the particular font used for license plates), and rather than "enhancing" an image of literally anything, you're just trying to find the set of characters that best matches the image you're looking at. That's not my area of expertise, though.

> **Comments:**
> **username_6**: I haven't heard of anyone doing it yet, but I really think temporal techniques could be used to get sub pixel accuracy from video streams.  
> **username_5**: @username_6 Definitely. Googling "video super-resolution" turns up a number of papers on that idea.  
> **username_7**: I did toyed with SISR a while ago, but my mind cleverly decided to erase that information from the brain. If I can recall though one important bit (mentioned in the paper) that I would explicit in your excellent answer is that the patterns or substructures are to be searched not only in the original image but also in multiple scales of it. That IIRC lead to significantly better results, although as I said I don't remember that much details :(  

---

