<p>I'm going to hazard a guess that you're using the <a href="https://psl.noaa.gov/repository/entry/show?entryid=ca90a190-f6b1-475f-bbe0-53cb0a308520" rel="nofollow noreferrer">NOAA OISST data</a>, although I don't think that makes a difference for the answer here.</p>
<p>Note that your loops,</p>
<pre><code>for i in range(len(lat)):
    for j in range(len(lon)):
        for m in range(len(lat)):
            for n in range(len(lon)):
</code></pre>
<p>will just generate the indexes,</p>
<pre><code>i = 0, ..., 8
    j = 0, ..., 31
        m = 0, ..., 8
            n = 0, ..., 31
</code></pre>
<p>When you then subset the global SST array using <code>sst5[59:90,i,j]</code> and <code>sst5[59:90,m,n]</code> you're not extracting data from the tropical points in your <code>lat</code> and <code>lon</code> variables, you're just extracting data from the corner of the full, global array, i.e., <code>sst5[:, 0:8, 0:31]</code>. You can see this by printing out <code>sst5.shape</code> in your loop, which will show,</p>
<pre><code>...
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
...
</code></pre>
<p>This corner is in Antarctica, hence no SST data and <code>numpy.corrcoef</code> will be seeing only missing data values and trying to divide by a zero covariance, yielding NaN values.  My Numpy issues the following warning when this happens:</p>
<pre><code>RuntimeWarning: invalid value encountered in true_divide
c /= stddev[:, None]
</code></pre>
<p>You could modify your program to subset the <code>sst5</code> array before the loops:</p>
<pre><code>sst5_subset = sst5[59:90, 210:510:35, ::45]
</code></pre>
<p>and then refer to this array inside the loops:</p>
<pre><code>mar_05.append(np.corrcoef(sst5_subset[:,i,j],sst5_subset[:, m,n])[0,1])
</code></pre>
<p>But, personally, I would get rid of the loops and do this as a whole-array operation:</p>
<pre><code>sst_array = sst5_subset.reshape(sst5_subset.shape[0], -1).T
corr = np.ma.corrcoef(sst_array)
</code></pre>
<p>A quick picture of the output for one point of interest:
<a href="https://i.sstatic.net/bzCFL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bzCFL.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2023-03-09T10:44:55.550Z
thanks for the convenient reply, and it saved me from the nested loop. Earlier, I calculated the correlation for the globe and did not realize lat = [ ..] and lon = [] did not fulfill the purpose in the nested loop conditions. Thanks for making me realize the mistake I made.
Ruby
CC BY-SA 4.0
2023-03-07T13:00:19.393Z
2023-03-08T16:57:37.630Z
2023-03-08T16:57:37.630Z
Deditos
<p>I have a sea surface temperature time series, and I want to calculate the (Pearson) correlation coefficients among the nodes in the tropics (30.0N to 30S). In the time series, the land information is present as masked. I do not know how to handle masked data in correlation calculation. Please help. The data I used here is in this link <a href="https://drive.google.com/file/d/1SVKQ4uBDEZOuN7_ftd5tqpF9_NGKY3pZ/view?usp=sharing" rel="nofollow noreferrer">https://drive.google.com/file/d/1SVKQ4uBDEZOuN7_ftd5tqpF9_NGKY3pZ/view?usp=sharing</a></p>
<p>I tried the following code, which did not work:</p>
<pre><code>      temp5 = 'sst.day.mean.1983.nc'
      fh5 = Dataset(temp5, mode = 'r')
      sst5 = fh5.variables['sst'][:365]
      time = fh5.variables['time'][:]
      lat = fh5.variables['lat'][210:510][::35] #tropics latitude
      lon = fh5.variables['lon'][::45]

      mar_05=[]
      for i in range(len(lat)):
        for j in range(len(lon)):
           for m in range(len(lat)):
              for n in range(len(lon)): 
               mar_05.append(np.corrcoef(sst5[59:90,i,j],sst5[59:90, m,n][0,1]))
      df = pd.DataFrame(data = mar_05)
</code></pre>

CC BY-SA 4.0
2023-03-05T19:52:17.093Z
I have tried np.ma.corrcoeff instead of np.corrcoef, but still, the correlation list is NAN. Any plausible solution?
Ruby
CC BY-SA 4.0
2023-03-07T09:31:07.130Z
Are any of your data in `sst5` NaN values, including the masked elements?
Deditos
CC BY-SA 4.0
2023-03-07T11:48:16.240Z
@Deditos There are no NAN values in the data; only masked values exist.
Ruby
CC BY-SA 4.0
2023-03-08T13:12:03.293Z
@Deditos thanks for the suggestion, but my problem is still there. I used the NOAA SST, and I defined the latitude of interest (lat = fh5.variables['lat'][210:510][::35]). The latitude range is from [-37.375 -28.625 -19.875 -11.125  -2.375   6.375  15.125  23.875  32.625], the latitude of the tropic. Over the latitude range, the land area is masked, and the ocean data are available. I tried calculating all possible correlation values among the nodes, so the nested loop. I also use ma.corrcoef instead np.corrcoef,  but the correlation list contains NAN values only.
Ruby
CC BY-SA 4.0
2023-03-08T16:26:55.287Z
Your lines `lat=` and `lon=` don't do what you seem to think they do.  They don't set which parts of the axes are active/visible when you reference `sst5`, they just make lists of the lats and lons that you're interested in, but `sst5` knows nothing about their contents.  I'll edit my answer.
Deditos
<netcdf><correlation>
Correlation calculation of masked array:
Ruby
# Correlation calculation of masked array:

(Asked by: username_0 on 2023-03-03)

I have a sea surface temperature time series, and I want to calculate the (Pearson) correlation coefficients among the nodes in the tropics (30.0N to 30S). In the time series, the land information is present as masked. I do not know how to handle masked data in correlation calculation. Please help. The data I used here is in this link <https://drive.google.com/file/d/1SVKQ4uBDEZOuN7_ftd5tqpF9_NGKY3pZ/view?usp=sharing>

I tried the following code, which did not work:

```
      temp5 = 'sst.day.mean.1983.nc'
      fh5 = Dataset(temp5, mode = 'r')
      sst5 = fh5.variables['sst'][:365]
      time = fh5.variables['time'][:]
      lat = fh5.variables['lat'][210:510][::35] #tropics latitude
      lon = fh5.variables['lon'][::45]

      mar_05=[]
      for i in range(len(lat)):
        for j in range(len(lon)):
           for m in range(len(lat)):
              for n in range(len(lon)): 
               mar_05.append(np.corrcoef(sst5[59:90,i,j],sst5[59:90, m,n][0,1]))
      df = pd.DataFrame(data = mar_05)

```

**Tags:** netcdf, correlation

> **Comments:**
> **username_0**: I have tried np.ma.corrcoeff instead of np.corrcoef, but still, the correlation list is NAN. Any plausible solution?  
> **username_1**: Are any of your data in `sst5` NaN values, including the masked elements?  
> **username_0**: @username_1 There are no NAN values in the data; only masked values exist.  
> **username_0**: @username_1 thanks for the suggestion, but my problem is still there. I used the NOAA SST, and I defined the latitude of interest (lat = fh5.variables['lat'][210:510][::35]). The latitude range is from [-37.375 -28.625 -19.875 -11.125  -2.375   6.375  15.125  23.875  32.625], the latitude of the tropic. Over the latitude range, the land area is masked, and the ocean data are available. I tried calculating all possible correlation values among the nodes, so the nested loop. I also use ma.corrcoef instead np.corrcoef,  but the correlation list contains NAN values only.  
> **username_1**: Your lines `lat=` and `lon=` don't do what you seem to think they do.  They don't set which parts of the axes are active/visible when you reference `sst5`, they just make lists of the lats and lons that you're interested in, but `sst5` knows nothing about their contents.  I'll edit my answer.  

---

## 1 answers

### Answer by username_1 on 2023-03-07. Score: 1 | Accepted answer

I'm going to hazard a guess that you're using the [NOAA OISST data](https://psl.noaa.gov/repository/entry/show?entryid=ca90a190-f6b1-475f-bbe0-53cb0a308520), although I don't think that makes a difference for the answer here.

Note that your loops,

```
for i in range(len(lat)):
    for j in range(len(lon)):
        for m in range(len(lat)):
            for n in range(len(lon)):

```

will just generate the indexes,

```
i = 0, ..., 8
    j = 0, ..., 31
        m = 0, ..., 8
            n = 0, ..., 31

```

When you then subset the global SST array using `sst5[59:90,i,j]` and `sst5[59:90,m,n]` you're not extracting data from the tropical points in your `lat` and `lon` variables, you're just extracting data from the corner of the full, global array, i.e., `sst5[:, 0:8, 0:31]`. You can see this by printing out `sst5.shape` in your loop, which will show,

```
...
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
...

```

This corner is in Antarctica, hence no SST data and `numpy.corrcoef` will be seeing only missing data values and trying to divide by a zero covariance, yielding NaN values. My Numpy issues the following warning when this happens:

```
RuntimeWarning: invalid value encountered in true_divide
c /= stddev[:, None]

```

You could modify your program to subset the `sst5` array before the loops:

```
sst5_subset = sst5[59:90, 210:510:35, ::45]

```

and then refer to this array inside the loops:

```
mar_05.append(np.corrcoef(sst5_subset[:,i,j],sst5_subset[:, m,n])[0,1])

```

But, personally, I would get rid of the loops and do this as a whole-array operation:

```
sst_array = sst5_subset.reshape(sst5_subset.shape[0], -1).T
corr = np.ma.corrcoef(sst_array)

```

A quick picture of the output for one point of interest:
[![enter image description here](https://i.sstatic.net/bzCFL.png)](https://i.sstatic.net/bzCFL.png)

> **Comments:**
> **username_0**: thanks for the convenient reply, and it saved me from the nested loop. Earlier, I calculated the correlation for the globe and did not realize lat = [ ..] and lon = [] did not fulfill the purpose in the nested loop conditions. Thanks for making me realize the mistake I made.  

---

