<p>For processing large data it is a good practice to call data into RAM in slices(Either by spliting time axis or spatial domain). In the interest of earth sciences python packages <strong>Xarray</strong>, <strong>iris</strong>, <strong>netCDF4</strong> and <strong>h5py</strong> are few of the great tools for handling huge hierarchical data. For handling data in a labeled fashion Xarray and Iris will be useful while netCDf4 and h5py are good to process in a gridded way.</p>

<p>My personal suggestion is <strong>h5py</strong> which is meant for processing and archiving large datasets. Documentation <a href="http://docs.h5py.org/en/latest/" rel="nofollow noreferrer">here</a> explains it. If you have netcdf files and want them in hdf5 format a <a href="https://stackoverflow.com/questions/17332353/what-is-the-easiest-way-to-convert-netcdf-to-hdf5-on-windows">question</a> at Stack Overflow might help.</p>

CC BY-SA 3.0
2017-12-26T13:27:34.373Z
2017-12-26T13:27:34.373Z
Jithu
<p>One option is to not load the entire file at a time. You can use <a href="https://code.google.com/archive/p/nctoolbox/wikis/ncgeodataset.wiki" rel="nofollow noreferrer">ncgeodataset</a>. The routine allows for the extraction of a subset of data without having to load the entire file or even an array into Matlab. It is great for large datasets.</p>

CC BY-SA 3.0
2017-12-24T16:21:52.087Z
2017-12-24T16:21:52.087Z
arkaia
<p>Can any one help me to handle huge netcdf files each of 1gb memory in loop and at least 2 files at a time in ncl or linux or python or matlab.</p>

<p>for example era_interim daily pressure level datasets.</p>

CC BY-SA 3.0
2017-12-24T16:22:29.680Z
We have a set of related questions in this site. For instance: https://earthscience.stackexchange.com/questions/5259/how-to-read-a-netcdf4-file-in-matlab
arkaia
CC BY-SA 3.0
2017-12-24T16:40:17.273Z
pretty vague question.  "handle" them how?  Do you have an error of some type or sample code you are trying to use?
f.thorpe
CC BY-SA 3.0
2017-12-24T19:44:57.427Z
What exactly do you want to do?
daniel.heydebreck
CC BY-SA 3.0
2017-12-25T01:37:13.950Z
I've handled large data files (over 1.5 GB) before. Never really noticed much of difference from using smaller datasets. In python, try using the netCDF4 module.
BarocliniCplusplus
CC BY-SA 3.0
2017-12-25T22:05:52.293Z
1Gb shouldn’t be a problem, but you should have a look at xarray and dask. Rather good tools for netcdf and large files.
user2821
CC BY-SA 3.0
2017-12-29T11:27:54.813Z
The answers given here are perhaps useful, but if you can be more specific about what's in the file and what you are trying to do, you may get a more specific answer.
Semidiurnal Simon
<climatology><netcdf>
Huge netcdf files handling
msarun
# Huge netcdf files handling

(Asked by: username_0 on 2017-12-24)

Can any one help me to handle huge netcdf files each of 1gb memory in loop and at least 2 files at a time in ncl or linux or python or matlab.

for example era\_interim daily pressure level datasets.

**Tags:** climatology, netcdf

> **Comments:**
> **username_1**: We have a set of related questions in this site. For instance: https://earthscience.stackexchange.com/questions/5259/how-to-read-a-netcdf4-file-in-matlab  
> **username_2**: pretty vague question.  "handle" them how?  Do you have an error of some type or sample code you are trying to use?  
> **username_3**: What exactly do you want to do?  
> **username_4**: I've handled large data files (over 1.5 GB) before. Never really noticed much of difference from using smaller datasets. In python, try using the netCDF4 module.  
> **username_5**: 1Gb shouldn’t be a problem, but you should have a look at xarray and dask. Rather good tools for netcdf and large files.  
> **username_6**: The answers given here are perhaps useful, but if you can be more specific about what's in the file and what you are trying to do, you may get a more specific answer.  

---

## 2 answers

### Answer by username_7 on 2017-12-26. Score: 3

For processing large data it is a good practice to call data into RAM in slices(Either by spliting time axis or spatial domain). In the interest of earth sciences python packages **Xarray**, **iris**, **netCDF4** and **h5py** are few of the great tools for handling huge hierarchical data. For handling data in a labeled fashion Xarray and Iris will be useful while netCDf4 and h5py are good to process in a gridded way.

My personal suggestion is **h5py** which is meant for processing and archiving large datasets. Documentation [here](http://docs.h5py.org/en/latest/) explains it. If you have netcdf files and want them in hdf5 format a [question](https://stackoverflow.com/questions/17332353/what-is-the-easiest-way-to-convert-netcdf-to-hdf5-on-windows) at Stack Overflow might help.


---

### Answer by username_1 on 2017-12-24. Score: 1

One option is to not load the entire file at a time. You can use [ncgeodataset](https://code.google.com/archive/p/nctoolbox/wikis/ncgeodataset.wiki). The routine allows for the extraction of a subset of data without having to load the entire file or even an array into Matlab. It is great for large datasets.


---

