<p>I have a data file from a Monte Carlo simulation of fifteen protein chains. The file contains 10 million <code>r_end_to_end</code> 3D vectors as rows and 3 x 15 = 45 columns.</p>
<p>My task is to find the exact row where the simulation stabilized. In other words, I need to find autocorrelation time. In other words, I need to find the <code>Tau0</code> time.</p>
<p>Now, the problem is that the data file is huge.</p>
<p>So, reading the entire 10 million rows is not practical for frequent testing, plot generation, curve-fitting, etc.</p>
<p>Is there any technique that I can use to achieve the required computation without reading all 10 million rows?</p>

CC BY-SA 4.0
2024-06-07T14:25:46.663Z
I don't think we can help without seeing an example input and the expected output. Of course I understand you can't show us 10 million lines, but could you give a toy example so we can understand the kind of data you are working with? And how slow? I made a toy example file with 10 million rows and 45 columns with `perl -le 'for my $i (1..10000000){ print join("\t", (1..45));  }' > file`. A simple operation (`awk '($2 > $3){k++}' file`) on the resulting 1.2G file only took ~1 second. Can you explain the kind of operations you will be running?
terdon
CC BY-SA 4.0
2024-06-08T08:11:09.840Z
For the frequent data access stay away from CSV/TSV text formats and switch to i.e `parquet` or `numpy arrays`.
darked89
<data-preprocessing>
Is there any technique that I can use to achieve the required computation without reading all 10 million rows?
# Is there any technique that I can use to achieve the required computation without reading all 10 million rows?

(Asked by: username_0 on 2024-06-05)

I have a data file from a Monte Carlo simulation of fifteen protein chains. The file contains 10 million `r_end_to_end` 3D vectors as rows and 3 x 15 = 45 columns.

My task is to find the exact row where the simulation stabilized. In other words, I need to find autocorrelation time. In other words, I need to find the `Tau0` time.

Now, the problem is that the data file is huge.

So, reading the entire 10 million rows is not practical for frequent testing, plot generation, curve-fitting, etc.

Is there any technique that I can use to achieve the required computation without reading all 10 million rows?

**Tags:** data-preprocessing
> **Comments:**
> **username_1**: I don't think we can help without seeing an example input and the expected output. Of course I understand you can't show us 10 million lines, but could you give a toy example so we can understand the kind of data you are working with? And how slow? I made a toy example file with 10 million rows and 45 columns with `perl -le 'for my $i (1..10000000){ print join("\t", (1..45));  }' > file`. A simple operation (`awk '($2 > $3){k++}' file`) on the resulting 1.2G file only took ~1 second. Can you explain the kind of operations you will be running?  
> **username_2**: For the frequent data access stay away from CSV/TSV text formats and switch to i.e `parquet` or `numpy arrays`.  

---

## 0 answers

