<p>I am analyzing ERA5 climate data spanning from 1950 to 2023, covering 73 years. The dataset has a daily temporal resolution and a spatial resolution of 0.25° (1440 x 720).</p>
<p>Each yearly data is stored in a NetCDF file, containing approximately 365 layers.</p>
<p>Attempting to combine these into a single stack results in around 27000 layers, which is overwhelming for my computer, and even the workplace cluster struggles with this volume.</p>
<p>As I'm new to handling large-scale datasets, I'm seeking advice on best practices and effective solutions for managing and processing this type of data.</p>
<p>Note: I have a preference for working with dataframes over NetCDF files as I will perform time series analysis.</p>
<p>Note 2: different types of analysis will be performed in the future. It's very wide: from simple statistics, to indices computations etc. On a grid cell level. So far, I am handling precipitation Prec and PET data to calculate a cumulative water deficit for each grid cell, followed by identifying the maximum deficit values annually, then conducting an extreme value analysis. But I would do other type of analysis in the future.</p>

CC BY-SA 4.0
2023-11-21T07:37:04.890Z
Do you plan to do time series analysis for a certain horizontal location or certain vertical layer? It would be important to read only parts of the data at once and divide the analysis into separate steps. E.g. read only data at one grid point (or a set of grid points) over the whole time span and do the analysis for this grid point (s). Then, this could be repeated for other grid point.
daniel.heydebreck
CC BY-SA 4.0
2023-11-21T07:40:58.147Z
Warning: consider the "chunking" of data, which means in which pattern they are stored. Commonly, the horizontal data of one vertical layer of one time stamp is stored in a row. Then the whole data of the next vertical layer (same time stamp) is stored ... and, then, the next vertical layer ... . After the last layer of time stamp `n`, the first layer of time stamp `n+1` is stored. This is means that accessing all time stamps of at one spatial location is very inefficient and slow. If you wish to do temporal analysis on your data, you should "re-chunk" your data (change the storage pattern).
daniel.heydebreck
CC BY-SA 4.0
2023-11-25T11:36:48.163Z
I'm not an R user, but it sounds like you're looking for a combination of lazy loading and possibly re-chunking the original data, which can change how lazy the loading can be.  Also, dataframes and netCDF aren't alternatives to each other; the former is in-memory storage and the latter is on-disk storage.
Deditos
CC BY-SA 4.0
2023-12-01T15:29:05.777Z
@daniel.heydebreck thank you very much. Yes, I aim to perform time series analysis. The thing is I would need to map or analysis specific regions. Could you redirect me to practicals in R referring to what you suggested?
Shunrei
CC BY-SA 4.0
2024-04-03T09:44:14.220Z
@Shunrei: Have you already tried `{terra}` (with GDAL underneath) for raster data handling and `{xts}` for handling of time series? Maybe even `{rts}` combining both approaches?
dimfalk
<climate><netcdf><r>
Handling large-scale climate dataset in R
Shunrei
# Handling large-scale climate dataset in R

(Asked by: username_0 on 2023-11-17)

I am analyzing ERA5 climate data spanning from 1950 to 2023, covering 73 years. The dataset has a daily temporal resolution and a spatial resolution of 0.25° (1440 x 720).

Each yearly data is stored in a NetCDF file, containing approximately 365 layers.

Attempting to combine these into a single stack results in around 27000 layers, which is overwhelming for my computer, and even the workplace cluster struggles with this volume.

As I'm new to handling large-scale datasets, I'm seeking advice on best practices and effective solutions for managing and processing this type of data.

Note: I have a preference for working with dataframes over NetCDF files as I will perform time series analysis.

Note 2: different types of analysis will be performed in the future. It's very wide: from simple statistics, to indices computations etc. On a grid cell level. So far, I am handling precipitation Prec and PET data to calculate a cumulative water deficit for each grid cell, followed by identifying the maximum deficit values annually, then conducting an extreme value analysis. But I would do other type of analysis in the future.

**Tags:** climate, netcdf, r

> **Comments:**
> **username_1**: Do you plan to do time series analysis for a certain horizontal location or certain vertical layer? It would be important to read only parts of the data at once and divide the analysis into separate steps. E.g. read only data at one grid point (or a set of grid points) over the whole time span and do the analysis for this grid point (s). Then, this could be repeated for other grid point.  
> **username_1**: Warning: consider the "chunking" of data, which means in which pattern they are stored. Commonly, the horizontal data of one vertical layer of one time stamp is stored in a row. Then the whole data of the next vertical layer (same time stamp) is stored ... and, then, the next vertical layer ... . After the last layer of time stamp `n`, the first layer of time stamp `n+1` is stored. This is means that accessing all time stamps of at one spatial location is very inefficient and slow. If you wish to do temporal analysis on your data, you should "re-chunk" your data (change the storage pattern).  
> **username_2**: I'm not an R user, but it sounds like you're looking for a combination of lazy loading and possibly re-chunking the original data, which can change how lazy the loading can be.  Also, dataframes and netCDF aren't alternatives to each other; the former is in-memory storage and the latter is on-disk storage.  
> **username_0**: @username_1 thank you very much. Yes, I aim to perform time series analysis. The thing is I would need to map or analysis specific regions. Could you redirect me to practicals in R referring to what you suggested?  
> **username_3**: @username_0: Have you already tried `{terra}` (with GDAL underneath) for raster data handling and `{xts}` for handling of time series? Maybe even `{rts}` combining both approaches?  

---

## 0 answers

