<p>On line 3, <code>rain_clim=xr.open_dataset('Rainfall-'+str(i)+'.nc')</code>, every time the loop iterates, <code>rain_clim</code> gets set to the value of the current data set. Follow user arkaia's suggestion, along the lines of either <code>rain_clim += xr.open_dataset('My_File')</code> or</p>
<p><code>rain_clim = []</code> <br/>
<code>for i in range(n_flood_years):</code> <br/>
<code>    tmp = xr.open_dataset(files[i])</code> <br/>
<code>    rain_clim += tmp</code> <br/>
<code>rain_clim = rain_clim / (i+1)  # sensitive to 0-based or 1-based indexing</code></p>
<p>Apologies for the poor formatting</p>

CC BY-SA 4.0
2024-01-25T16:11:02.143Z
There is also the ncra command to make netcdf record averages
John Osborne
CC BY-SA 4.0
2024-01-30T08:31:02.840Z
Hi John, I tried your way but it gives "ERROR: operands could not be broadcast together with shapes (0,) (12,2) ". FYI, my input files have dimension=>
time: 12 longitude: 360 latitude: 180; and the n_flood_years is an `array([1977, 1988, 1990])`
Bidyut Bikash Goswami
CC BY-SA 4.0
2024-01-30T08:32:22.120Z
Thanks. I know about ncra or cdo. I just need to do it in Python.
Bidyut Bikash Goswami
CC BY-SA 4.0
2024-01-31T17:34:13.567Z
`n_flood_years` should be an integer and it should be 3. Need to confirm that rain_clim and tmp have the same dimension - I am not very familiar with xarray, so I'm sure there are some syntax details I am overlooking.
John Osborne
CC BY-SA 4.0
2024-01-25T16:06:49.977Z
2024-01-25T16:09:21.483Z
2024-01-25T16:09:21.483Z
John Osborne
<p>I am trying to compute daily climatology of ERA5 data, but without using <strong>append</strong> and <strong>concat</strong> so that I do not consume a lot of memory.</p>
<p>My idea was to read every file and update it by adding the next file and then dividing it by 2 in a loop. To do this, I tried to use the += operator. I am writing it in the code-snippet below. Since, I am reading inside the loop, I am not really updating it. Can anyone please tell me, how I can update <strong>rain_clim</strong> as I read along. Many thanks.</p>
<pre><code>rain_clim=[]
for i in flood_years:
        rain_clim=xr.open_dataset('Rainfall-'+str(i)+'.nc')
        rain_clim+=rain_clim
        rain_clim=rain_clim/2```

</code></pre>

CC BY-SA 4.0
2024-01-25T13:38:29.123Z
Don't divide by 2 each time! Simple add all of them, create a counter that add 1 per time and then divide by that counter at the end
arkaia
CC BY-SA 4.0
2024-01-25T15:16:11.277Z
Thanks for your comment. But my main problem is, **rain_clim+=rain_clim** is not updating **rain_clim**. Any suggestions?
Bidyut Bikash Goswami
CC BY-SA 4.0
2024-01-26T13:12:02.107Z
Have you tried to use dask?
gerrit
CC BY-SA 4.0
2024-01-30T08:36:43.383Z
@gerrit: thanks. I have not tried dask. I am trying to learn how to use it. Meanwhile, if you do not mind, it would be great if you can give me a quick pointer using my snippet as example.
Bidyut Bikash Goswami
CC BY-SA 4.0
2024-01-30T11:10:57.303Z
@BidyutBikashGoswami I don't have a snippet, but you can google documentation for xarray and dask in combination.  I think you can also open all datasets together, without a loop, using zarr or xarray mfdataset.  Sorry, no time for a detailed answer.
gerrit
<climatology><era>
Compute ERA5 daily climatology without consuming lot of memory
Bidyut Bikash Goswami
# Compute ERA5 daily climatology without consuming lot of memory

(Asked by: username_0 on 2024-01-25)

I am trying to compute daily climatology of ERA5 data, but without using **append** and **concat** so that I do not consume a lot of memory.

My idea was to read every file and update it by adding the next file and then dividing it by 2 in a loop. To do this, I tried to use the += operator. I am writing it in the code-snippet below. Since, I am reading inside the loop, I am not really updating it. Can anyone please tell me, how I can update **rain\_clim** as I read along. Many thanks.

```
rain_clim=[]
for i in flood_years:
        rain_clim=xr.open_dataset('Rainfall-'+str(i)+'.nc')
        rain_clim+=rain_clim
        rain_clim=rain_clim/2```

```

**Tags:** climatology, era

> **Comments:**
> **username_1**: Don't divide by 2 each time! Simple add all of them, create a counter that add 1 per time and then divide by that counter at the end  
> **username_0**: Thanks for your comment. But my main problem is, **rain_clim+=rain_clim** is not updating **rain_clim**. Any suggestions?  
> **username_2**: Have you tried to use dask?  
> **username_0**: @username_2: thanks. I have not tried dask. I am trying to learn how to use it. Meanwhile, if you do not mind, it would be great if you can give me a quick pointer using my snippet as example.  
> **username_2**: @username_0 I don't have a snippet, but you can google documentation for xarray and dask in combination.  I think you can also open all datasets together, without a loop, using zarr or xarray mfdataset.  Sorry, no time for a detailed answer.  

---

## 1 answers

### Answer by username_3 on 2024-01-25. Score: 3

On line 3, `rain_clim=xr.open_dataset('Rainfall-'+str(i)+'.nc')`, every time the loop iterates, `rain_clim` gets set to the value of the current data set. Follow user username_1's suggestion, along the lines of either `rain_clim += xr.open_dataset('My_File')` or

`rain_clim = []`   
`for i in range(n_flood_years):`   
 `tmp = xr.open_dataset(files[i])`   
 `rain_clim += tmp`   
`rain_clim = rain_clim / (i+1) # sensitive to 0-based or 1-based indexing`

Apologies for the poor formatting

> **Comments:**
> **username_3**: There is also the ncra command to make netcdf record averages  
> **username_0**: Hi username_3, I tried your way but it gives "ERROR: operands could not be broadcast together with shapes (0,) (12,2) ". FYI, my input files have dimension=> time: 12 longitude: 360 latitude: 180; and the n_flood_years is an `array([1977, 1988, 1990])`  
> **username_0**: Thanks. I know about ncra or cdo. I just need to do it in Python.  
> **username_3**: `n_flood_years` should be an integer and it should be 3. Need to confirm that rain_clim and tmp have the same dimension - I am not very familiar with xarray, so I'm sure there are some syntax details I am overlooking.  

---

