Q: Best way to concatenate large list of xarrays I have a large list of xarrays (named results_list which is about 620000 of xarrays (each xarray has the following configuration: shape=(73, 41, 1), dtype=np.float32).
I can work on a cluster of 72 cores (30Go RAM each one).
To concatenate xarrays into only one dask array, I tried the following steps:
import dask.array as da
rho = [da.from_array(result[0].data.squeeze()) for result in results_list]
It's not as quick as I thought (CPU time = 1min 39s) but it's not a bottleneck.
Then:
da_rho = da.concatenate(rho, axis=0)
This calculation takes too much time (CPU time = 14min 5s) because I'll have to do that kind of operations dozens of times but I'm time limited.
Is it possible to reduce time computation significantly using dask ?
