<p>I have 250 or so soil sensors that I need to QC. They monitor various parameters such as soil temp, soil water content, soil co2, and soil heat flux. It isn't unusual for a sensor to behave poorly due to elk chewing up wires a little bit, electrical noise, and what not.</p>
<p>I already use some basic scripts to filter out based on moving averages, standard deviation, and physical constraints. Sometimes there are sensors that still have bad data, such as this soil water. It looks like there is still a viable signal in that noise that is traced out by a bunch of local minima, and I've confirmed this pattern with nearby sensors.</p>
<p>I've played around with some finite-impulse response filters in MATLAB to try to remove the noise, but I don't really know how to design one of these at all.</p>
<p>Any suggestions, resources, or other filters that could help with QCing this?</p>
<p><a href="https://i.sstatic.net/u88SS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/u88SS.png" alt="Soil water content at 5 cm" /></a></p>

CC BY-SA 3.0
2017-09-24T22:44:43.470Z
Maybe the simplest thing would be to do something like a moving average but use the minimum value rather than average. Do you understand the reason why the sources of error lead to unreasonably high values? That would help justify your filter. You might also check the literature on processing eddy covariance data - they have to filter aggressively to retrieve the usable signal.
haresfur
CC BY-SA 3.0
2017-09-25T17:02:28.880Z
We were unable to figure out why there were months of high values. The probes are controlled by a CR23x datalogger, which had some program changes around the time when the data got noisy. This probe is alright in 2017.
Lou-gazi
<hydrology><soil>
Designing FIR or other filters for soil water content
Lou-gazi
# Designing FIR or other filters for soil water content

(Asked by: username_0 on 2017-09-22)

I have 250 or so soil sensors that I need to QC. They monitor various parameters such as soil temp, soil water content, soil co2, and soil heat flux. It isn't unusual for a sensor to behave poorly due to elk chewing up wires a little bit, electrical noise, and what not.

I already use some basic scripts to filter out based on moving averages, standard deviation, and physical constraints. Sometimes there are sensors that still have bad data, such as this soil water. It looks like there is still a viable signal in that noise that is traced out by a bunch of local minima, and I've confirmed this pattern with nearby sensors.

I've played around with some finite-impulse response filters in MATLAB to try to remove the noise, but I don't really know how to design one of these at all.

Any suggestions, resources, or other filters that could help with QCing this?

[![Soil water content at 5 cm](https://i.sstatic.net/u88SS.png)](https://i.sstatic.net/u88SS.png)

**Tags:** hydrology, soil

> **Comments:**
> **username_1**: Maybe the simplest thing would be to do something like a moving average but use the minimum value rather than average. Do you understand the reason why the sources of error lead to unreasonably high values? That would help justify your filter. You might also check the literature on processing eddy covariance data - they have to filter aggressively to retrieve the usable signal.  
> **username_0**: We were unable to figure out why there were months of high values. The probes are controlled by a CR23x datalogger, which had some program changes around the time when the data got noisy. This probe is alright in 2017.  

---

## 0 answers

