<p>Looks like you've got some flaky electronics attached to your CCD if the bias level is changing by several units in consecutive exposures. However, all is not lost if your CCD has an <a href="http://www.mirametrics.com/tech_note_overscan_bias.htm" rel="nofollow">overscan region</a>.</p>

<p>This is normally several columns of "pixels" that appear appended to each row of your data, but which do not correspond to physical pixels on the CCD. They are actually dummy readouts to assess the actual bias level in each particular row.</p>

<p>The overscan region should be clearly visible in any real exposure of a light source, since the overscan region will be un-illuminated.</p>

<p>You can use the overscan in two ways. The conventional method is to overscan-correct your images <em>before</em> doing a bias subtraction. You should also do this to your bias frames <em>before</em> combining them into your "master bias" frame.</p>

<p>So, take your overscan region and either (i) calculate a median (or clipped mean) within it and subtract this value from every pixel in the image before use (including the bias frames); or (ii) fit something more complicated as a function of row number and then subtract this from your actual data on a row-by-row basis.</p>

<p>Method (ii) is required if you notice that there is any structure or drift/trend in the overscan values as a function of row number.</p>

<p>This <a href="http://www.mirametrics.com/tech_note_overscan_bias.htm" rel="nofollow">link</a> gives plenty of details, some pictures and a recipe.</p>

CC BY-SA 3.0
2015-05-26T13:19:08.413Z
Thanks, Rob. Unfortunately, from what I can tell, the SBIG ST-7 does not support reading the overscan region. I'll get some more time on the telescope next month and hopefully I'll find some way to make that happen.
user181339
CC BY-SA 3.0
2015-05-26T13:32:24.323Z
I've actually noticed the same behaviour with our other camera (STL-6303), so this is probably due to electronics further down the line. In the meantime it seems I'll just have to make due with the data I have. I guess the best thing for now would be to normalize the biases by shifting to a central value, and then add the uncertainty of this central value to the noise estimation/propagation. If the small shift per frame is normally distributed this should be appropriate given the limited data at my disposal at the moment.
user181339
CC BY-SA 3.0
2015-05-26T13:32:39.177Z
@user181339 You're stuck then. All you can do with your sequence of bias frames is characterise how bad it could be. Every *astronomical* CCD instrument I've used has had an overscan strip. Looking at some documentation http://www.astro.louisville.edu/software/sbig/archive/xmccd-4.1/xmccd-4.1e/docs/sbig/sbigudrv4r70.pdf it looks like you ought to be able to do dummy reads on every row to produce an overscan on SBIG CCDs.
ProfRob
CC BY-SA 3.0
2015-05-26T13:34:36.837Z
@user181339 Note that the overscan is (usually) just a simple numerical offset to the bias level that applies to all the pixels.
ProfRob
CC BY-SA 3.0
2015-05-26T13:41:57.897Z
Thanks for the link to the driver manual. I see there might be an option for reading overscan via a registry key. I'll give it a shot as soon as I get my hands on the telescope again.
user181339
CC BY-SA 3.0
2015-05-26T08:32:02.997Z
2015-05-26T08:32:02.997Z
ProfRob
<p>The median values of the bias frames coming from our teaching telescope shift up or down with each capture. For example:</p>
<pre><code>import glob, numpy as np, astropy.io.fits as fio, matplotlib.pyplot as plt

data = np.zeros((10,255,765), dtype=np.int)
for i, f in enumerate(glob.glob('bias_*.fit')):
    data[i,:,:] = fio.getdata(f)

print(np.median(data, axis=(1,2)))
</code></pre>
<p><strong>[ 106.  106.  108.  108.  110.  108.  105.  106.  107.  107.]</strong></p>
<p>Now, during data reduction we average combine our bias frames. We also reject outliers (either minmax or sigmaclip). But if some of our frames are &quot;shifted&quot; higher than others, wouldn't this skew the averaging (and especially the outlier rejection) so that we might be introducing more noise during averaging?</p>
<p>To give you an idea of the nature of this fluctuation, I did a quick analysis: read 10 bias frames, then iterated through them, subtracting each frame from the median combination of all other frames. This gives me something like &quot;noise frames&quot;, showing only how much each frame deviates from the median of the rest. Below is the code and the distributions of two of these noise frames.</p>
<pre><code>noiseframes = np.zeros((10,255,765), dtype=np.int)
otherframes = np.ones(10, dtype=np.bool) # just a mask
for i in xrange(10):
    otherframes[:] = True
    otherframes[i] = False # exclude current frame from mask
    median_of_others = np.median(data[otherframes,:,:], axis=0)
    noiseframes[i,:,:] = data[i,:,:] - median_of_others

def plotdist(i):
    frame = noiseframes[i].flatten()
    x = np.arange(np.min(frame),np.max(frame)+1)
    y = np.bincount(frame-np.min(frame))
    plt.step(x, y, label=str(np.median(data[i,:,:])))

plotdist(4)
plotdist(3)
plt.xlabel('Difference from median [ADU]')
plt.legend()
plt.show()
</code></pre>
<p><a href="https://i.sstatic.net/qFZYm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qFZYm.png" alt="enter image description here" /></a></p>
<p>All of the noise frames have near-perfect gaussian distributions with pretty much constant FWHM, it's just the central value that shifts around with each frame.</p>
<p>This leads me to a couple of conclusions: firstly, this shift truely is a shift in median value and not just a result of something like cosmic rays; secondly, the fluctuation is an additive effect and so not due to a change in gain due to a difference in voltage or something like that.</p>
<p>I also could not find any correlation with temperature or time since last readout, both of which I suspected for a moment.</p>
<p>Perhaps it's slight changes in readout time?</p>
<p>I couldn't find any mention of this kind of artefact in my own course notes. It's also not mentioned anywhere online that I can find.</p>
<p>Can any of you help me to a) identify the possible sources of such a fluctuation and/or b) provide a proven strategy for dealing with this during data reduction?</p>

<photography><observational-astronomy>
How to deal with shifting CCD bias (zero exposure) levels?
user181339
# How to deal with shifting CCD bias (zero exposure) levels?

(Asked by: username_0 on 2015-05-25)

The median values of the bias frames coming from our teaching telescope shift up or down with each capture. For example:
import glob, numpy as np, astropy.io.fits as fio, matplotlib.pyplot as plt

data = np.zeros((10,255,765), dtype=np.int)
for i, f in enumerate(glob.glob('bias_*.fit')):
    data[i,:,:] = fio.getdata(f)

print(np.median(data, axis=(1,2)))

[ 106.  106.  108.  108.  110.  108.  105.  106.  107.  107.]
Now, during data reduction we average combine our bias frames. We also reject outliers (either minmax or sigmaclip). But if some of our frames are "shifted" higher than others, wouldn't this skew the averaging (and especially the outlier rejection) so that we might be introducing more noise during averaging?
To give you an idea of the nature of this fluctuation, I did a quick analysis: read 10 bias frames, then iterated through them, subtracting each frame from the median combination of all other frames. This gives me something like "noise frames", showing only how much each frame deviates from the median of the rest. Below is the code and the distributions of two of these noise frames.
noiseframes = np.zeros((10,255,765), dtype=np.int)
otherframes = np.ones(10, dtype=np.bool) # just a mask
for i in xrange(10):
    otherframes[:] = True
    otherframes[i] = False # exclude current frame from mask
    median_of_others = np.median(data[otherframes,:,:], axis=0)
    noiseframes[i,:,:] = data[i,:,:] - median_of_others

def plotdist(i):
    frame = noiseframes[i].flatten()
    x = np.arange(np.min(frame),np.max(frame)+1)
    y = np.bincount(frame-np.min(frame))
    plt.step(x, y, label=str(np.median(data[i,:,:])))

plotdist(4)
plotdist(3)
plt.xlabel('Difference from median [ADU]')
plt.legend()
plt.show()


All of the noise frames have near-perfect gaussian distributions with pretty much constant FWHM, it's just the central value that shifts around with each frame.
This leads me to a couple of conclusions: firstly, this shift truely is a shift in median value and not just a result of something like cosmic rays; secondly, the fluctuation is an additive effect and so not due to a change in gain due to a difference in voltage or something like that.
I also could not find any correlation with temperature or time since last readout, both of which I suspected for a moment.
Perhaps it's slight changes in readout time?
I couldn't find any mention of this kind of artefact in my own course notes. It's also not mentioned anywhere online that I can find.
Can any of you help me to a) identify the possible sources of such a fluctuation and/or b) provide a proven strategy for dealing with this during data reduction?

**Tags:** photography, observational-astronomy


---

## 1 answers

### Answer by username_1 on 2015-05-26. Score: 4 | Accepted answer

Looks like you've got some flaky electronics attached to your CCD if the bias level is changing by several units in consecutive exposures. However, all is not lost if your CCD has an [overscan region](http://www.mirametrics.com/tech_note_overscan_bias.htm).

This is normally several columns of "pixels" that appear appended to each row of your data, but which do not correspond to physical pixels on the CCD. They are actually dummy readouts to assess the actual bias level in each particular row.

The overscan region should be clearly visible in any real exposure of a light source, since the overscan region will be un-illuminated.

You can use the overscan in two ways. The conventional method is to overscan-correct your images *before* doing a bias subtraction. You should also do this to your bias frames *before* combining them into your "master bias" frame.

So, take your overscan region and either (i) calculate a median (or clipped mean) within it and subtract this value from every pixel in the image before use (including the bias frames); or (ii) fit something more complicated as a function of row number and then subtract this from your actual data on a row-by-row basis.

Method (ii) is required if you notice that there is any structure or drift/trend in the overscan values as a function of row number.

This [link](http://www.mirametrics.com/tech_note_overscan_bias.htm) gives plenty of details, some pictures and a recipe.

> **Comments:**
> **username_0**: Thanks, Rob. Unfortunately, from what I can tell, the SBIG ST-7 does not support reading the overscan region. I'll get some more time on the telescope next month and hopefully I'll find some way to make that happen.  
> **username_0**: I've actually noticed the same behaviour with our other camera (STL-6303), so this is probably due to electronics further down the line. In the meantime it seems I'll just have to make due with the data I have. I guess the best thing for now would be to normalize the biases by shifting to a central value, and then add the uncertainty of this central value to the noise estimation/propagation. If the small shift per frame is normally distributed this should be appropriate given the limited data at my disposal at the moment.  
> **username_1**: @username_0 You're stuck then. All you can do with your sequence of bias frames is characterise how bad it could be. Every *astronomical* CCD instrument I've used has had an overscan strip. Looking at some documentation http://www.astro.louisville.edu/software/sbig/archive/xmccd-4.1/xmccd-4.1e/docs/sbig/sbigudrv4r70.pdf it looks like you ought to be able to do dummy reads on every row to produce an overscan on SBIG CCDs.  
> **username_1**: @username_0 Note that the overscan is (usually) just a simple numerical offset to the bias level that applies to all the pixels.  
> **username_0**: Thanks for the link to the driver manual. I see there might be an option for reading overscan via a registry key. I'll give it a shot as soon as I get my hands on the telescope again.  

---

