<p>This probably does not explore the full depth of the term, but the first thing that comes to my mind when I hear "bilateral upsampling" is depth-aware blending of low-resolution images onto high-resolution ones; for instance, when you render alpha-blended geometry to a half-resolution buffer (for performance savings) and then composite it back onto the main render target. Angelo Pesce <a href="http://c0de517e.blogspot.com/2016/02/downsampled-effects-with-depth-aware.html" rel="nofollow">wrote a post</a> that explains it well.</p>

CC BY-SA 3.0
2016-06-21T08:50:29.780Z
2016-06-21T08:50:29.780Z
IneQuation
<p>Thanks @IneQuation, I'm also wondering what is bilateral upsampling recently, and the post by Angelo Pesce you provided explained his practices quite well, but one thing should be made clear ahead —— 'Bilateral Filtering' usually means adding neigboring pixel similarity factor to Guassian Filtering, which can be used to denoise and preserve sharp edges, but 'Bilateral Upsampling' Angelo talked about has nothing to do with Gaussian, it's Bilinear Upsampling added with depth and normal continuity factor, and so also named with 'Bilateral'. <a href="https://developer.amd.com/wordpress/media/2012/10/ShopfMixedResolutionRendering.pdf" rel="nofollow noreferrer">This presentation by AMD's Shopf</a> explained it better for beginners I think.</p>

CC BY-SA 4.0
2022-08-04T06:32:55.550Z
2022-08-04T06:32:55.550Z
Emily Galahad
<p>It seems like you're asking two things.  I can't really speak technically about JBU, but I can give an overview of the necessary concepts and bilateral filtering generally.  You'll probably need to find more details yourself, but this should give a coherent structure to start from.</p>
<hr />
<h2>Fixing &quot;Image&quot;s</h2>
<p>Many image-processing people view filtering as either something to be done as a post-process, or as a way to scale images.  The mathematician's perspective is more accurate, descriptive, and complex.</p>
<p>When you have an image file, you don't have a real image.  You have pixel &quot;values&quot;, which are strictly valid <em>only</em> at the center of pixels.  Roughly speaking, each pixel has an <em>area</em> that takes on that color.  Mathematically, what you're doing is using a <em>nearest-neighbor reconstruction filter</em> to reconstruct the true image from pixel values.  So <strong>an &quot;image&quot; is really a set of samples and a reconstruction filter, typically nearest-neighbor</strong>.</p>
<h2>Resampling</h2>
<p>When you upscale an image, what you're really doing is <em>resampling</em>.  How do you do this?  You take your signal, and generate a new set of samples from it.  But for an image, I just said we don't <em>have</em> the real image; we have to reconstruct it first.  So a typical resampling procedure looks like this:</p>
<ol>
<li>Take your pixel values.</li>
<li>Reconstruct an image between the values, which live at pixel centers, by using reconstruction filtering.</li>
<li>Measure a new set of samples from the reconstructed image, possibly spaced differently.</li>
<li>Throw away the old samples and use the new ones.</li>
</ol>
<p><strong>The key point is that the reconstruction filter used is the &quot;filter&quot; used for resampling.</strong>  For example, a &quot;bicubic upsample&quot; would use a bicubic reconstruction filter.</p>
<p>It turns out that if you use a sinc function for the reconstruction filter, and you don't shrink the image, you get (theoretically) the same image back.  In fact, if you upscale the image and then downscale it again, you'll get the same answer.  This isn't generally true for any other filter.</p>
<p>Most filters approximate a sinc filter to more or lesser degrees.  Deviations result in image loss.</p>
<h2>Filtering</h2>
<p>If you use the same before/after sampling rate, you do something called &quot;filtering&quot;.  Formally, <strong>filtering is resampling without a change in sample rate.  Filtering is a special case of resampling.</strong></p>
<p>The only possible purpose to filtering it to throw away information (e.g. a Gaussian) or possibly also to distort or add spurious information (e.g. a Sobel).</p>
<h1>Bilateral Filtering</h1>
<p>You can use many different filters for filtering.  Bilateral filtering is one of them.  I don't believe it's considered state-of-the-art for any purpose now (though I don't know what is; probably some newfangled machine-learning thing I expect).</p>
<p>A bilateral filter is a non-linear filter, meaning it doesn't have a nice Fourier representation.  Conceptually, it is like a Gaussian, but the source colors themselves also factor into a pixel's &quot;distance&quot; from others.  This tends to smooth flat areas while preserving edges.</p>
<h1>Bilateral Upscaling</h1>
<p>By now, I hope you see where I'm going with this: any filter you can use for filtering, you can use for upscaling or downscaling, which are really the same thing (because all of it is resampling).</p>
<p>If you've implemented a bilateral filter, you should be able to rather simply implement a bilateral filter resampler - just as, if you've implemented a Gaussian filter, you should be able to implement a Gaussian filter resampler.</p>
<p>The only difficulty may be the definitions.  Unfortunately, the distinction between filtering and resampling is systemically muddied to occasional non-existence in the image-processing literature.  For implementing a bilateral-type filter, I would rely heavily on its similarities to Gaussian resampling.  IIRC (I haven't read the paper in years) bilateral filtering is posed in terms of pixels, not samples.</p>

CC BY-SA 3.0
2016-06-20T21:01:51.810Z
How bilateral filtering specifically works is what I am trying to get information about. Can you elaborate on that?
Alan Wolfe
CC BY-SA 3.0
2016-06-20T21:10:43.230Z
One can lend images for the explanations in the beginning from [here](http://computergraphics.stackexchange.com/questions/1862/how-does-a-computer-upscale-1024x768-resolution-to-1920x1080/1870#1870)
joojaa
CC BY-SA 3.0
2016-06-21T03:49:44.197Z
@AlanWolfe I assumed you were asking after broader context, since implementing basic bilateral filtering is highly Google-able and rather simple. As I wrote, the basic idea is to make values factor into the weight, not just distance.
geometrian
CC BY-SA 3.0
2016-06-21T03:49:58.150Z
Also @joolaa's answer is quite nice and may be useful as a second explanation of my overview.
geometrian
CC BY-SA 3.0
2016-06-21T03:53:37.333Z
Now that I have the term "bilateral filter" I am having more luck on Google. Thanks for that.
Alan Wolfe
CC BY-SA 3.0
2016-06-20T20:56:30.480Z
2016-06-21T08:35:13.940Z
2020-06-18T08:32:12.800Z
geometrian
<p>I've worked with various interpolation algorithms including linear, cubic (and bilinear and bicubic), other bezier (and nurbs, etc) based interpolations, lagrange interpolation and some others, but I haven't been able to find any basic info that explains what bilateral upsampling is.</p>

<p>I have been able to find information on <a href="http://johanneskopf.de/publications/jbu/">Joint Bilateral Upsampling</a> which as best as I can tell interpolates multiple pieces of data in parallel, using hints from each to help the other interpolate better, but am having trouble separating the info of BU and JBU.</p>

<p>Is anyone able to explain the basic ideas of what bilateral upsampling is about?</p>

CC BY-SA 3.0
2016-07-26T05:30:40.837Z
The original paper on bilateral filtering, _"Bilateral Filtering for Gray and Color Images"_, by Tomasi and Manduchi: https://users.cs.duke.edu/~tomasi/papers/tomasi/tomasiIccv98.pdf
Julien Guertault
<image-processing><interpolation><image>
What is bilateral upsampling?
Alan Wolfe
# What is bilateral upsampling?

(Asked by: username_0 on 2016-06-20)

I've worked with various interpolation algorithms including linear, cubic (and bilinear and bicubic), other bezier (and nurbs, etc) based interpolations, lagrange interpolation and some others, but I haven't been able to find any basic info that explains what bilateral upsampling is.

I have been able to find information on [Joint Bilateral Upsampling](http://johanneskopf.de/publications/jbu/) which as best as I can tell interpolates multiple pieces of data in parallel, using hints from each to help the other interpolate better, but am having trouble separating the info of BU and JBU.

Is anyone able to explain the basic ideas of what bilateral upsampling is about?

**Tags:** image-processing, interpolation, image

> **Comments:**
> **username_1**: The original paper on bilateral filtering, _"Bilateral Filtering for Gray and Color Images"_, by Tomasi and Manduchi: https://users.cs.duke.edu/~tomasi/papers/tomasi/tomasiIccv98.pdf  

---

## 3 answers

### Answer by username_4 on 2016-06-20. Score: 11 | Accepted answer

It seems like you're asking two things. I can't really speak technically about JBU, but I can give an overview of the necessary concepts and bilateral filtering generally. You'll probably need to find more details yourself, but this should give a coherent structure to start from.

---

## Fixing "Image"s

Many image-processing people view filtering as either something to be done as a post-process, or as a way to scale images. The mathematician's perspective is more accurate, descriptive, and complex.

When you have an image file, you don't have a real image. You have pixel "values", which are strictly valid *only* at the center of pixels. Roughly speaking, each pixel has an *area* that takes on that color. Mathematically, what you're doing is using a *nearest-neighbor reconstruction filter* to reconstruct the true image from pixel values. So **an "image" is really a set of samples and a reconstruction filter, typically nearest-neighbor**.

## Resampling

When you upscale an image, what you're really doing is *resampling*. How do you do this? You take your signal, and generate a new set of samples from it. But for an image, I just said we don't *have* the real image; we have to reconstruct it first. So a typical resampling procedure looks like this:

1. Take your pixel values.
2. Reconstruct an image between the values, which live at pixel centers, by using reconstruction filtering.
3. Measure a new set of samples from the reconstructed image, possibly spaced differently.
4. Throw away the old samples and use the new ones.

**The key point is that the reconstruction filter used is the "filter" used for resampling.** For example, a "bicubic upsample" would use a bicubic reconstruction filter.

It turns out that if you use a sinc function for the reconstruction filter, and you don't shrink the image, you get (theoretically) the same image back. In fact, if you upscale the image and then downscale it again, you'll get the same answer. This isn't generally true for any other filter.

Most filters approximate a sinc filter to more or lesser degrees. Deviations result in image loss.

## Filtering

If you use the same before/after sampling rate, you do something called "filtering". Formally, **filtering is resampling without a change in sample rate. Filtering is a special case of resampling.**

The only possible purpose to filtering it to throw away information (e.g. a Gaussian) or possibly also to distort or add spurious information (e.g. a Sobel).

# Bilateral Filtering

You can use many different filters for filtering. Bilateral filtering is one of them. I don't believe it's considered state-of-the-art for any purpose now (though I don't know what is; probably some newfangled machine-learning thing I expect).

A bilateral filter is a non-linear filter, meaning it doesn't have a nice Fourier representation. Conceptually, it is like a Gaussian, but the source colors themselves also factor into a pixel's "distance" from others. This tends to smooth flat areas while preserving edges.

# Bilateral Upscaling

By now, I hope you see where I'm going with this: any filter you can use for filtering, you can use for upscaling or downscaling, which are really the same thing (because all of it is resampling).

If you've implemented a bilateral filter, you should be able to rather simply implement a bilateral filter resampler - just as, if you've implemented a Gaussian filter, you should be able to implement a Gaussian filter resampler.

The only difficulty may be the definitions. Unfortunately, the distinction between filtering and resampling is systemically muddied to occasional non-existence in the image-processing literature. For implementing a bilateral-type filter, I would rely heavily on its similarities to Gaussian resampling. IIRC (I haven't read the paper in years) bilateral filtering is posed in terms of pixels, not samples.

> **Comments:**
> **username_0**: How bilateral filtering specifically works is what I am trying to get information about. Can you elaborate on that?  
> **username_5**: One can lend images for the explanations in the beginning from [here](http://computergraphics.stackexchange.com/questions/1862/how-does-a-computer-upscale-1024x768-resolution-to-1920x1080/1870#1870)  
> **username_4**: @username_0 I assumed you were asking after broader context, since implementing basic bilateral filtering is highly Google-able and rather simple. As I wrote, the basic idea is to make values factor into the weight, not just distance.  
> **username_4**: Also @joolaa's answer is quite nice and may be useful as a second explanation of my overview.  
> **username_0**: Now that I have the term "bilateral filter" I am having more luck on Google. Thanks for that.  

---

### Answer by username_2 on 2016-06-21. Score: 3

This probably does not explore the full depth of the term, but the first thing that comes to my mind when I hear "bilateral upsampling" is depth-aware blending of low-resolution images onto high-resolution ones; for instance, when you render alpha-blended geometry to a half-resolution buffer (for performance savings) and then composite it back onto the main render target. Angelo Pesce [wrote a post](http://c0de517e.blogspot.com/2016/02/downsampled-effects-with-depth-aware.html) that explains it well.


---

### Answer by username_3 on 2022-08-04. Score: 0

Thanks @username_2, I'm also wondering what is bilateral upsampling recently, and the post by Angelo Pesce you provided explained his practices quite well, but one thing should be made clear ahead —— 'Bilateral Filtering' usually means adding neigboring pixel similarity factor to Guassian Filtering, which can be used to denoise and preserve sharp edges, but 'Bilateral Upsampling' Angelo talked about has nothing to do with Gaussian, it's Bilinear Upsampling added with depth and normal continuity factor, and so also named with 'Bilateral'. [This presentation by AMD's Shopf](https://developer.amd.com/wordpress/media/2012/10/ShopfMixedResolutionRendering.pdf) explained it better for beginners I think.


---

