<p>After contacting the author of the chart, I believe the following defines the process behind the chart.</p>

<ol>
<li>Data source: NOAA's <a href="https://www.ncdc.noaa.gov/ghcnd-data-access" rel="nofollow noreferrer">Global Historical Climatology Network Daily</a> station data for "PCPT".</li>
<li>Use only stations in the Southeast states which have data for 90% of days from 1895 to 2011, to match stations used previous edition of the climate report.</li>
<li>Use only years with at least 300 days of data.</li>
<li>Remove data values that have been flagged in GHND for quality issues.</li>
<li>Count the number of days of heavy rainfall (3 or more inches) for each station and year.</li>
<li>Using a 1° by 1° lat/lon grid, average counts of stations in the same grid cell.</li>
<li>Report the average of all grid cell-years for a given decade.</li>
</ol>

<p>I have still not been able to reproduce the chart because I have had trouble getting clean data. Will update with a chart when that happens.</p>

CC BY-SA 4.0
2018-12-21T19:28:26.633Z
The 1 by 1 seems a bit quirky... for example a lot of partial cells in places like Florida counting equitable to whole cells, so it would over-represent Florida perhaps (depending on how many cells even have stations with that length of years!)  And I wonder "only use years with 300 days"... do they normalize the final result numbers if some grid cells end up empty because a year was thrown out?  Lots of little questions... but probably not that crazy of differences I guess.
JeopardyTempest
CC BY-SA 4.0
2018-12-16T20:50:14.210Z
2018-12-16T20:50:14.210Z
xan
<p>I'm trying to understand/reproduce/revise this chart in the <a href="https://nca2018.globalchange.gov/chapter/19/" rel="nofollow noreferrer">Southeast section</a> of the United States <a href="https://nca2018.globalchange.gov/" rel="nofollow noreferrer">National Climate Assessment</a>. The truncated scale for the bars is what caught by eye as a classic <a href="https://callingbullshit.org/tools/tools_misleading_axes.html" rel="nofollow noreferrer">chart lie</a>, but I'm also wondering about other features: different thresholds (besides 3 inches) and different aggregations (besides decades).</p>

<p><a href="https://i.sstatic.net/PkFz1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PkFz1.png" alt="enter image description here"></a></p>

<p>Original caption for this chart plus a map next to it:</p>

<blockquote>
  <p>Figure 19.3: The figure shows variability and change in (left) the
  annual number of days with precipitation greater than 3 inches
  (1900–2016) averaged over the Southeast by decade and (right)
  individual station trends (1950–2016). The number of days with heavy
  precipitation has increased at most stations, particularly since the
  1980s. Sources: NOAA NCEI and CICS-NC.</p>
</blockquote>

<p>The vague source citations are not particularly helpful, but I did find NOAA's <a href="https://www.ncdc.noaa.gov/ghcnd-data-access" rel="nofollow noreferrer">Global Historical Climatology Network Daily</a> downloadable data. That has daily precipitation data for 100,000 stations around the world, with about 12,000 in the Southeast. However, in a given year, I'm seeing 100 to 200 days with precipitation above 3 inches, but the chart indicates a different order of magnitude.</p>

<p><strong>What does "days with precipitation greater than 3 inches" mean in this context?</strong></p>

<p>I'm guessing there must be a "per station" part to it, but I still can't guess the exact math. The simplest treatment, a weighted average of each stations rate of heavy rain, produces the chart below.</p>

<p><a href="https://i.sstatic.net/1LWf4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1LWf4.png" alt="enter image description here"></a></p>

<p>It's close (same order of magnitude and peaks in 40s, 70s and recent), but not quite the same. Is there a different data source? Or a different treatment for irregular samples? Or a way to account for increasing number of stations?</p>

<p><strong>Update after comments:</strong>
If I filter on stations with at least 110 years of data and use only years with 200+ days of data, I get the following, which is closer. (Using the same scale as the original, but avoiding bar truncation by just showing the tops.)</p>

<p><a href="https://i.sstatic.net/pSi3N.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pSi3N.png" alt="enter image description here"></a></p>

<p>Most decades have lower values than the original.</p>

CC BY-SA 4.0
2018-12-11T10:42:33.000Z
I certainly haven't spent that much time closely monitoring or analyzing their results (glad to see others doing so!) and so any input is rather uninformed and just thoughts... by weighted average, you're meaning you're taking into account the spatial distribution, yes?  Wonder if perhaps they used a more limited set of sites.... one thing that would seem somewhat important to such a graph would be station existence... it'd seem best to only use stations existing since the start of the graph's timeperiod to ensure the change in weighting/data availability doesn't alter the graph over time?
JeopardyTempest
CC BY-SA 4.0
2018-12-11T13:18:58.580Z
My weighting was based on the number of days of the year that the site was reporting. I doubt many sites have a 100 yr run, but I should check that. If I instead remove sites with incomplete years, the last 3 decades look similar to the report, but the 40s are well below 1.0. The report mentions a 5 degree grid for some national analyses but that's too coarse for SE.
xan
CC BY-SA 4.0
2018-12-12T00:16:13.293Z
I think a fair portion of the official climate sites (the ones they specifically publish daily highs and lows on and state long term records) go back around that far... the siting may change (like from one airport location to another or such)... but I just happened to be looking at Orlando records last night and it goes back to 1892, and I expect there's a few sites in each state that do likewise?
JeopardyTempest
CC BY-SA 4.0
2018-12-12T02:22:22.303Z
Thanks. There are about 425 stations out of 12,000 with 100+ years of data. Looking only at those (or higher thresholds), my spike in the 2010s goes down to be closer to the original (a lot of stations are relatively new), but the whole chart is still not a great match -- 1940s is still barely above 1.0. Progress...
xan
CC BY-SA 4.0
2018-12-12T11:39:02.887Z
Good to hear.  Do think another possible factor could be unequal representation... perhaps more of your sites are from one area than another or such, which given the southeast has such a range of regimes (coastal impacts/seabreezes, Appalachian Mountains, etc) I'd think would be important.  Indeed I'd be surprised if they used a 5 degree grid because it'd seem fairly poor resolution given 425 stations. I'd think GrADS software could do grid interpolation with some effort, or perhaps like R or GIS packages.
JeopardyTempest
CC BY-SA 4.0
2018-12-12T11:42:29.907Z
Or I wonder if there's some specific factor in the 1940s.  I happened to notice when looking through that Orlando data yesterday that there was a fair bit of missing data around the war.  Figured maybe they were just changing sites, or too busy with the war (since it's before automation).  But maybe there's a more notable factor across multiple sites that you'll need to uncover :-/  I'd certainly be interested to see the graph now, as it sort of seems that if your scales were adjusted to be equal, most other early decades would be quite close.
JeopardyTempest
CC BY-SA 4.0
2018-12-12T23:28:52.607Z
I added the latest chart to the question body. The number of stations per year doesn't vary much once I filter on stations with so many years present. I have emailed the author of the chart and am in the mean time collecting data from another source (ACIS) to compare. I have worked on my own gridding, but I have to figure out a way to fairly compare grid cells with different station counts.
xan
CC BY-SA 4.0
2018-12-13T05:19:00.740Z
Indeed, still work to be done: [combined overlay of your lines with the original graphic to recreate](https://imgur.com/UJ02ne6)
JeopardyTempest
CC BY-SA 4.0
2018-12-14T02:07:18.340Z
I got some more details from the author and am [even closer](https://pbs.twimg.com/media/DuV5Gf2XQAcGqXm.jpg:large). Will post his response as an answer if I get permission, and hopefully a couple clarifications. In short: stations with 90% nonmissing data and years with 300+ nonmissing days.
xan
CC BY-SA 4.0
2018-12-14T08:46:07.387Z
Makes fair sense, throw out the ones with the bigger danger of missing data altering the result robustness.  As long as the criteria set was blind, seems reasonable enough.  Glad to keep seeing progress!!!
JeopardyTempest
<climate><precipitation>
How is heavy rainfall determined in National Climate Assessment
xan
# How is heavy rainfall determined in National Climate Assessment

(Asked by: username_0 on 2018-12-09)

I'm trying to understand/reproduce/revise this chart in the [Southeast section](https://nca2018.globalchange.gov/chapter/19/) of the United States [National Climate Assessment](https://nca2018.globalchange.gov/). The truncated scale for the bars is what caught by eye as a classic [chart lie](https://callingbullshit.org/tools/tools_misleading_axes.html), but I'm also wondering about other features: different thresholds (besides 3 inches) and different aggregations (besides decades).

[![enter image description here](https://i.sstatic.net/PkFz1.png)](https://i.sstatic.net/PkFz1.png)

Original caption for this chart plus a map next to it:

> Figure 19.3: The figure shows variability and change in (left) the
> annual number of days with precipitation greater than 3 inches
> (1900–2016) averaged over the Southeast by decade and (right)
> individual station trends (1950–2016). The number of days with heavy
> precipitation has increased at most stations, particularly since the
> 1980s. Sources: NOAA NCEI and CICS-NC.

The vague source citations are not particularly helpful, but I did find NOAA's [Global Historical Climatology Network Daily](https://www.ncdc.noaa.gov/ghcnd-data-access) downloadable data. That has daily precipitation data for 100,000 stations around the world, with about 12,000 in the Southeast. However, in a given year, I'm seeing 100 to 200 days with precipitation above 3 inches, but the chart indicates a different order of magnitude.

**What does "days with precipitation greater than 3 inches" mean in this context?**

I'm guessing there must be a "per station" part to it, but I still can't guess the exact math. The simplest treatment, a weighted average of each stations rate of heavy rain, produces the chart below.

[![enter image description here](https://i.sstatic.net/1LWf4.png)](https://i.sstatic.net/1LWf4.png)

It's close (same order of magnitude and peaks in 40s, 70s and recent), but not quite the same. Is there a different data source? Or a different treatment for irregular samples? Or a way to account for increasing number of stations?

**Update after comments:**
If I filter on stations with at least 110 years of data and use only years with 200+ days of data, I get the following, which is closer. (Using the same scale as the original, but avoiding bar truncation by just showing the tops.)

[![enter image description here](https://i.sstatic.net/pSi3N.png)](https://i.sstatic.net/pSi3N.png)

Most decades have lower values than the original.

**Tags:** climate, precipitation

> **Comments:**
> **username_1**: I certainly haven't spent that much time closely monitoring or analyzing their results (glad to see others doing so!) and so any input is rather uninformed and just thoughts... by weighted average, you're meaning you're taking into account the spatial distribution, yes?  Wonder if perhaps they used a more limited set of sites.... one thing that would seem somewhat important to such a graph would be station existence... it'd seem best to only use stations existing since the start of the graph's timeperiod to ensure the change in weighting/data availability doesn't alter the graph over time?  
> **username_0**: My weighting was based on the number of days of the year that the site was reporting. I doubt many sites have a 100 yr run, but I should check that. If I instead remove sites with incomplete years, the last 3 decades look similar to the report, but the 40s are well below 1.0. The report mentions a 5 degree grid for some national analyses but that's too coarse for SE.  
> **username_1**: I think a fair portion of the official climate sites (the ones they specifically publish daily highs and lows on and state long term records) go back around that far... the siting may change (like from one airport location to another or such)... but I just happened to be looking at Orlando records last night and it goes back to 1892, and I expect there's a few sites in each state that do likewise?  
> **username_0**: Thanks. There are about 425 stations out of 12,000 with 100+ years of data. Looking only at those (or higher thresholds), my spike in the 2010s goes down to be closer to the original (a lot of stations are relatively new), but the whole chart is still not a great match -- 1940s is still barely above 1.0. Progress...  
> **username_1**: Good to hear.  Do think another possible factor could be unequal representation... perhaps more of your sites are from one area than another or such, which given the southeast has such a range of regimes (coastal impacts/seabreezes, Appalachian Mountains, etc) I'd think would be important.  Indeed I'd be surprised if they used a 5 degree grid because it'd seem fairly poor resolution given 425 stations. I'd think GrADS software could do grid interpolation with some effort, or perhaps like R or GIS packages.  
> **username_1**: Or I wonder if there's some specific factor in the 1940s.  I happened to notice when looking through that Orlando data yesterday that there was a fair bit of missing data around the war.  Figured maybe they were just changing sites, or too busy with the war (since it's before automation).  But maybe there's a more notable factor across multiple sites that you'll need to uncover :-/  I'd certainly be interested to see the graph now, as it sort of seems that if your scales were adjusted to be equal, most other early decades would be quite close.  
> **username_0**: I added the latest chart to the question body. The number of stations per year doesn't vary much once I filter on stations with so many years present. I have emailed the author of the chart and am in the mean time collecting data from another source (ACIS) to compare. I have worked on my own gridding, but I have to figure out a way to fairly compare grid cells with different station counts.  
> **username_1**: Indeed, still work to be done: [combined overlay of your lines with the original graphic to recreate](https://imgur.com/UJ02ne6)  
> **username_0**: I got some more details from the author and am [even closer](https://pbs.twimg.com/media/DuV5Gf2XQAcGqXm.jpg:large). Will post his response as an answer if I get permission, and hopefully a couple clarifications. In short: stations with 90% nonmissing data and years with 300+ nonmissing days.  
> **username_1**: Makes fair sense, throw out the ones with the bigger danger of missing data altering the result robustness.  As long as the criteria set was blind, seems reasonable enough.  Glad to keep seeing progress!!!  

---

## 1 answers

### Answer by username_0 on 2018-12-16. Score: 2

After contacting the author of the chart, I believe the following defines the process behind the chart.

1. Data source: NOAA's [Global Historical Climatology Network Daily](https://www.ncdc.noaa.gov/ghcnd-data-access) station data for "PCPT".
2. Use only stations in the Southeast states which have data for 90% of days from 1895 to 2011, to match stations used previous edition of the climate report.
3. Use only years with at least 300 days of data.
4. Remove data values that have been flagged in GHND for quality issues.
5. Count the number of days of heavy rainfall (3 or more inches) for each station and year.
6. Using a 1° by 1° lat/lon grid, average counts of stations in the same grid cell.
7. Report the average of all grid cell-years for a given decade.

I have still not been able to reproduce the chart because I have had trouble getting clean data. Will update with a chart when that happens.

> **Comments:**
> **username_1**: The 1 by 1 seems a bit quirky... for example a lot of partial cells in places like Florida counting equitable to whole cells, so it would over-represent Florida perhaps (depending on how many cells even have stations with that length of years!)  And I wonder "only use years with 300 days"... do they normalize the final result numbers if some grid cells end up empty because a year was thrown out?  Lots of little questions... but probably not that crazy of differences I guess.  

---

