<p>Thanks for @Kwinkunks's answer. I have read this <a href="http://www.atmos-meas-tech.net/4/1905/2011/amt-4-1905-2011.pdf" rel="noreferrer">paper</a>. And this figure may explain something important.<br>
<a href="https://i.sstatic.net/8K68c.png" rel="noreferrer"><img src="https://i.sstatic.net/8K68c.png" alt="enter image description here"></a></p>

<p>What I have already done is plotting the original data like the first subplot.<br>
From <strong>Destriped -  v1.02</strong>, the noise data can be shown clearly.     </p>

<p>Adding another figure to indicate the big importance of induced noise.  </p>

<p><a href="https://i.sstatic.net/Cya9e.png" rel="noreferrer"><img src="https://i.sstatic.net/Cya9e.png" alt="enter image description here"></a>  </p>

<p>Huge difference there!</p>

<p>So, I'll learn some technology to deal with the destriped problem. I'm familiar with <em>Python</em>. Anyone who has similar project experience could post your method here.</p>

CC BY-SA 3.0
2016-06-18T02:40:05.533Z
2016-06-19T02:56:29.433Z
2016-06-19T02:56:29.433Z
Han Zhengzu
<p>You ask, <em>How to remove these abnormal value for more precise data?</em></p>

<p>The short answer is that you might be able to make your data look more representative of the natural 'truth', but you will probably have to pay for this with some precision. The problem is that the measurement itself is imperfect: this looks like an artifact of the data collection method. The general name for unwanted energy that interferes with a signal you are interested in is <a href="https://en.wikipedia.org/wiki/Noise_(signal_processing)" rel="noreferrer">noise</a>.</p>

<p>In your case, there is some random noise, and there is this coherent (i.e. spatially organized) noise. It is likely impossible to substantially attack either type of noise without touching the signal. In other words, you will probably have to pay for the noise reduction with some loss of precision. This may be a trade-off you are willing to make.</p>

<h3>Reducing noise</h3>

<p>There are many, many ways to <a href="https://en.wikipedia.org/wiki/Noise_reduction" rel="noreferrer">reduce noise</a> in data. A very common way to reduce random noise is to apply a <a href="https://en.wikipedia.org/wiki/Smoothing" rel="noreferrer">smoothing filter</a> such as a simple mean in some small, moving window (often expressed as a convolution operation). The problem with this is that it just mixes the noise into the image.</p>

<p>You can also think of a smoothing high-cut filter in the spatial frequency (aka <a href="https://en.wikipedia.org/wiki/Wavenumber" rel="noreferrer">wavenumber</a>) domain. A common approach to removing the sort of spatially organized noise you have is to use a 'notch' (narrow) cut filter in this domain, which has the effect of removing noise with that spatial frequency. In other words, you can remove stripes.</p>

<p>Filters that remove stripes can be found in various image processing software, or could be implemented with various scientific computing languages (e.g. MATLAB's <a href="http://www.mathworks.com/products/image/" rel="noreferrer">Image Processing Toolbox</a> or <a href="http://scikit-image.org/" rel="noreferrer">scikit-image</a> for Python). Here are some options:</p>

<ul>
<li><a href="https://imagej.nih.gov/ij/docs/guide/146-29.html#toc-Subsection-29.10" rel="noreferrer">FFT in ImageJ/FIJI</a> — I would start here, unless you're already into MATLAB or Python.</li>
<li><a href="https://docs.gimp.org/en/plug-in-destripe.html" rel="noreferrer">Destripe in GIMP</a> — another good option.</li>
<li><a href="https://isis.astrogeology.usgs.gov/IsisWorkshop/index.php/Removing_Striping_Noise_from_Image_Data" rel="noreferrer">Destriping in ISIS</a> — an image processing package from USGS.</li>
</ul>

CC BY-SA 3.0
2016-06-18T00:10:20.090Z
2016-06-18T00:10:20.090Z
Matt Hall
<p>Satellite remote sensing data is a powerful tool for researching the spatial distribution and temporal variation of atmospheric species.   </p>

<p><strong>Aura</strong> is one satellite owing to NASA which meaning atmosphere in Latin literally.  </p>

<p><strong>OMI</strong> which means Ozone Monitoring Instrument has the global observation data for $\ce{O3}$, $\ce{NO2}$, $\ce{BrO}$ etc.  </p>

<p>Getting the $\ce{NO2}$ monthly mean data from <a href="http://www.temis.nl/airpollution/no2col/no2regioomimonth_v2.php" rel="noreferrer">TEMIS</a>, the ESRI grid format file was plotted by me showing here:   </p>

<p><a href="https://i.sstatic.net/1CtFh.png" rel="noreferrer"><img src="https://i.sstatic.net/1CtFh.png" alt="A satellite image"></a>  </p>

<p>I have noticed that the figure are covered by discrete silk braid (in my plot, they are plotted in green color liking raining top->down) like this:  </p>

<p><a href="https://i.sstatic.net/RppOR.png" rel="noreferrer"><img src="https://i.sstatic.net/RppOR.png" alt="An excerpt from the same image"></a>  </p>

<p>IMHO, these braid must be not related to the anthropogenic emission but something related to the satellite orbit. </p>

<p>For quantify research of certain grid box, these braid would influence the finite column concentration.   </p>

<p>Besides, from NASA Giovanni online visualization tools, the figure below is also the $\ce{NO2}$ tropospheric column in the same area.</p>

<p><a href="https://i.sstatic.net/kVqw6.png" rel="noreferrer"><img src="https://i.sstatic.net/kVqw6.png" alt="More satellite data"></a>    </p>

<p>There are no silk braid like my figure.</p>

<p>So, my question is here: <strong>How to remove these abnormal value for more precise data</strong>. </p>

<p>That would be nice if anyone familiar with this area give me some advice.       </p>

<h3>Update</h3>

<p>Thanks for @deneil's attention. I presented two data source here for distinguishing.      </p>

<ol>
<li><p>TEMIS data was called <em>Dutch OMI NO<sub>2</sub> (DOMINO) data product v2.0</em>.<br>
In its official manual <a href="http://www.temis.nl/docs/OMI_NO2_HE5_2.0_2011.pdf" rel="noreferrer">here</a>, it said, I quoted:</p>

<ul>
<li><strong>DOMINO = Level 2 product</strong><br>
The DOMINO data contains geolocated column integrated $\ce{NO2}$ concentrations, or<br>
$\ce{NO2}$ columns (in units of molecules/cm2). DOMINO data constitute a pure Level 2 product, i.e.       </li>
<li>it provides geophysical information for each and every ground pixel observed by the instrument, without the additional binning, averaging or gridding typically applied for Level 3 data.      </li>
</ul></li>
<li><p>The Giovanni visualization tool use OMI/Aura $\ce{NO2}$ Cloud-Screened Total and Tropospheric Column Daily L3 Global 0.25deg Lat/Lon Grid which means it's level 3 data.    </p></li>
</ol>

<p>I think the difference sitting there. But I have also tried OMI $\ce{NO2}$ level2 data from NASA (OMI has two independent data source in Dutch and NASA), it has 15 candidate scenes which was plotted different from the TEMIS DUMNO2 data.</p>

<p>Here is an example. The figure below plot the level 2 data in tilted grid due to the satellite orbits. Level 3 data must resample the tilted grid data into regular grid network.  So, I think my first figure shows the right result as level 2 data's character. </p>

<p><a href="https://i.sstatic.net/mn83h.png" rel="noreferrer"><img src="https://i.sstatic.net/mn83h.png" alt="A satellite image"></a></p>

<h3>Updated II</h3>

<p>There are three different source for tropospheric $\ce{NO2}$ column. It seems that Berkely's dataset is the best choice.
<a href="https://i.sstatic.net/o9bm7.png" rel="noreferrer"><img src="https://i.sstatic.net/o9bm7.png" alt="enter image description here"></a></p>

CC BY-SA 3.0
2016-06-17T11:19:24.270Z
Both Plots are based on exactly the same data? Or might it be that the NAS Giovanni Plot is based on further processed data?
daniel.heydebreck
CC BY-SA 3.0
2016-06-17T15:33:20.623Z
Sorry for replying late. I check the both datasource which are `Dutch OMI NO2 (DOMINO) data product v2.0` and `GES_DISC_OMNO2d_V003(Tropospheric Column Daily L3 Global 0.25deg Lat/Lon Grid)`
Han Zhengzu
<atmosphere><remote-sensing><air-pollution><satellites>
How to remove 'silk braid' noise in satellite data?
Han Zhengzu
# How to remove 'silk braid' noise in satellite data?

(Asked by: username_0 on 2016-06-17)

Satellite remote sensing data is a powerful tool for researching the spatial distribution and temporal variation of atmospheric species.

**Aura** is one satellite owing to NASA which meaning atmosphere in Latin literally.

**OMI** which means Ozone Monitoring Instrument has the global observation data for $\ce{O3}$, $\ce{NO2}$, $\ce{BrO}$ etc.

Getting the $\ce{NO2}$ monthly mean data from [TEMIS](http://www.temis.nl/airpollution/no2col/no2regioomimonth_v2.php), the ESRI grid format file was plotted by me showing here:

[![A satellite image](https://i.sstatic.net/1CtFh.png)](https://i.sstatic.net/1CtFh.png)

I have noticed that the figure are covered by discrete silk braid (in my plot, they are plotted in green color liking raining top->down) like this:

[![An excerpt from the same image](https://i.sstatic.net/RppOR.png)](https://i.sstatic.net/RppOR.png)

IMHO, these braid must be not related to the anthropogenic emission but something related to the satellite orbit.

For quantify research of certain grid box, these braid would influence the finite column concentration.

Besides, from NASA Giovanni online visualization tools, the figure below is also the $\ce{NO2}$ tropospheric column in the same area.

[![More satellite data](https://i.sstatic.net/kVqw6.png)](https://i.sstatic.net/kVqw6.png)

There are no silk braid like my figure.

So, my question is here: **How to remove these abnormal value for more precise data**.

That would be nice if anyone familiar with this area give me some advice.

### Update

Thanks for @deneil's attention. I presented two data source here for distinguishing.

1. TEMIS data was called *Dutch OMI NO2 (DOMINO) data product v2.0*.  
   In its official manual [here](http://www.temis.nl/docs/OMI_NO2_HE5_2.0_2011.pdf), it said, I quoted:

   * **DOMINO = Level 2 product**  
     The DOMINO data contains geolocated column integrated $\ce{NO2}$ concentrations, or  
     $\ce{NO2}$ columns (in units of molecules/cm2). DOMINO data constitute a pure Level 2 product, i.e.
   * it provides geophysical information for each and every ground pixel observed by the instrument, without the additional binning, averaging or gridding typically applied for Level 3 data.
2. The Giovanni visualization tool use OMI/Aura $\ce{NO2}$ Cloud-Screened Total and Tropospheric Column Daily L3 Global 0.25deg Lat/Lon Grid which means it's level 3 data.

I think the difference sitting there. But I have also tried OMI $\ce{NO2}$ level2 data from NASA (OMI has two independent data source in Dutch and NASA), it has 15 candidate scenes which was plotted different from the TEMIS DUMNO2 data.

Here is an example. The figure below plot the level 2 data in tilted grid due to the satellite orbits. Level 3 data must resample the tilted grid data into regular grid network. So, I think my first figure shows the right result as level 2 data's character.

[![A satellite image](https://i.sstatic.net/mn83h.png)](https://i.sstatic.net/mn83h.png)

### Updated II

There are three different source for tropospheric $\ce{NO2}$ column. It seems that Berkely's dataset is the best choice.
[![enter image description here](https://i.sstatic.net/o9bm7.png)](https://i.sstatic.net/o9bm7.png)

**Tags:** atmosphere, remote-sensing, air-pollution, satellites

> **Comments:**
> **username_1**: Both Plots are based on exactly the same data? Or might it be that the NAS Giovanni Plot is based on further processed data?  
> **username_0**: Sorry for replying late. I check the both datasource which are `Dutch OMI NO2 (DOMINO) data product v2.0` and `GES_DISC_OMNO2d_V003(Tropospheric Column Daily L3 Global 0.25deg Lat/Lon Grid)`  

---

## 2 answers

### Answer by username_2 on 2016-06-18. Score: 7 | Accepted answer

You ask, *How to remove these abnormal value for more precise data?*

The short answer is that you might be able to make your data look more representative of the natural 'truth', but you will probably have to pay for this with some precision. The problem is that the measurement itself is imperfect: this looks like an artifact of the data collection method. The general name for unwanted energy that interferes with a signal you are interested in is [noise](https://en.wikipedia.org/wiki/Noise_(signal_processing)).

In your case, there is some random noise, and there is this coherent (i.e. spatially organized) noise. It is likely impossible to substantially attack either type of noise without touching the signal. In other words, you will probably have to pay for the noise reduction with some loss of precision. This may be a trade-off you are willing to make.

### Reducing noise

There are many, many ways to [reduce noise](https://en.wikipedia.org/wiki/Noise_reduction) in data. A very common way to reduce random noise is to apply a [smoothing filter](https://en.wikipedia.org/wiki/Smoothing) such as a simple mean in some small, moving window (often expressed as a convolution operation). The problem with this is that it just mixes the noise into the image.

You can also think of a smoothing high-cut filter in the spatial frequency (aka [wavenumber](https://en.wikipedia.org/wiki/Wavenumber)) domain. A common approach to removing the sort of spatially organized noise you have is to use a 'notch' (narrow) cut filter in this domain, which has the effect of removing noise with that spatial frequency. In other words, you can remove stripes.

Filters that remove stripes can be found in various image processing software, or could be implemented with various scientific computing languages (e.g. MATLAB's [Image Processing Toolbox](http://www.mathworks.com/products/image/) or [scikit-image](http://scikit-image.org/) for Python). Here are some options:

* [FFT in ImageJ/FIJI](https://imagej.nih.gov/ij/docs/guide/146-29.html#toc-Subsection-29.10) — I would start here, unless you're already into MATLAB or Python.
* [Destripe in GIMP](https://docs.gimp.org/en/plug-in-destripe.html) — another good option.
* [Destriping in ISIS](https://isis.astrogeology.usgs.gov/IsisWorkshop/index.php/Removing_Striping_Noise_from_Image_Data) — an image processing package from USGS.


---

### Answer by username_0 on 2016-06-18. Score: 5

Thanks for @Kwinkunks's answer. I have read this [paper](http://www.atmos-meas-tech.net/4/1905/2011/amt-4-1905-2011.pdf). And this figure may explain something important.  
[![enter image description here](https://i.sstatic.net/8K68c.png)](https://i.sstatic.net/8K68c.png)

What I have already done is plotting the original data like the first subplot.  
From **Destriped - v1.02**, the noise data can be shown clearly.

Adding another figure to indicate the big importance of induced noise.

[![enter image description here](https://i.sstatic.net/Cya9e.png)](https://i.sstatic.net/Cya9e.png)

Huge difference there!

So, I'll learn some technology to deal with the destriped problem. I'm familiar with *Python*. Anyone who has similar project experience could post your method here.


---

