<p>You've hit the RAM bottleneck. Its very common in de novo assemblies and resolved using template assemblies. I don't know this program but suspect it is performing a pairwise analysis for every piece of data in a genomics data set.</p>
<p>The solution is to either:</p>
<blockquote>
<ol>
<li><p>find a bigger machine, e.g. a big multi-core machine and make sure the program distributes across every core.</p>
</li>
<li><p>parallelise the calculation (an excellent way to overcome RAM problems), or,</p>
</li>
<li><p>find an approximation to the calculation that is less computationally demanding, templates are a good way to circumvent this.</p>
</li>
</ol>
</blockquote>

CC BY-SA 4.0
2018-12-23T22:56:35.087Z
2018-12-23T22:56:35.087Z
2020-06-18T08:30:41.743Z
M__
<p>I have <a href="https://github.com/tbenavi1/smudgeplot/blob/master/hetkmers.py" rel="nofollow noreferrer">a script</a> that extracts the set of unique kmer pairs from a kmer dump file (&amp; then using them downstream to guess ploidy).</p>

<p>The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.</p>

<p>I don't really get it; the script actually prints the last line </p>

<blockquote>
  <p>*_families_2.tsv and *_coverages_2.tsv files saved.</p>
</blockquote>

<p>, but then it runs further (&amp; forever). I thought it could be related to longer time to free the memory as discussed <a href="https://stackoverflow.com/questions/2412715/python-program-doesnt-quit-when-finished">here</a>, but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).</p>

<p>Does anybody an idea where the problem could be? Or how to figure out?</p>

<p>The reported problem on <a href="https://github.com/tbenavi1/smudgeplot/issues/18" rel="nofollow noreferrer">GitHub</a>.</p>

CC BY-SA 4.0
2018-12-20T15:01:21.390Z
Can you trace the memory usage of the progress to see if it's related to freeing memory? we had a similar issue some time ago, tried a lot of things like disabling garbage collection etc. but never resolved it. Also have you tried running on another system? our issue was linked to a particular system with an old OS/software
Chris_Rands
CC BY-SA 4.0
2018-12-20T15:02:10.457Z
Also this might fit better on Stack Overflow (although they might complain about it being no small and reproducible)
Chris_Rands
CC BY-SA 4.0
2018-12-20T15:02:31.480Z
After the last line is printed the program seems to occupy exactly the same amount of memory over time.
Kamil S Jaron
CC BY-SA 4.0
2018-12-20T15:04:05.343Z
I was thinking about posting it to Stack Overflow, but since it's kmer related problem and since I prefer to interact with this community I have decided to post it here.
Kamil S Jaron
CC BY-SA 4.0
2018-12-20T15:33:17.710Z
As the author [suggested](https://github.com/tbenavi1/smudgeplot/issues/18#issuecomment-426609963), this is probably caused by a deadlock in multi-threading.
user172818
CC BY-SA 4.0
2018-12-24T02:07:17.993Z
you probably need to do "from multiprocessing import Pool" and not "Queue" or "spawn"
user3479780
<python><k-mer>
python does not quit when the input file is too big
# python does not quit when the input file is too big

(Asked by: username_0 on 2018-12-20)

I have [a script](https://github.com/tbenavi1/smudgeplot/blob/master/hetkmers.py) that extracts the set of unique kmer pairs from a kmer dump file (& then using them downstream to guess ploidy).

The script seems to work correctly (it's reporting a meaningful set of kmer pairs), but since we have added a parallelization it does not quit when the job is done for really big data sets. The problem is that I never managed to reproduce the behavior with a small reproducible example and it takes around 400G of memory and a day of computation to get to the problem. My imagination is quite limited, I have no idea how should I debug such a problem.

I don't really get it; the script actually prints the last line

> \*\_families\_2.tsv and \*\_coverages\_2.tsv files saved.

, but then it runs further (& forever). I thought it could be related to longer time to free the memory as discussed [here](https://stackoverflow.com/questions/2412715/python-program-doesnt-quit-when-finished), but it does not seem like the case because it does not quit even after a really long time (I tried to just leave it for a day or so and when i kill it the memory is freed in like 20s).

Does anybody an idea where the problem could be? Or how to figure out?

The reported problem on [GitHub](https://github.com/tbenavi1/smudgeplot/issues/18).

**Tags:** python, k-mer
> **Comments:**
> **username_1**: Can you trace the memory usage of the progress to see if it's related to freeing memory? we had a similar issue some time ago, tried a lot of things like disabling garbage collection etc. but never resolved it. Also have you tried running on another system? our issue was linked to a particular system with an old OS/software  
> **username_1**: Also this might fit better on Stack Overflow (although they might complain about it being no small and reproducible)  
> **username_0**: After the last line is printed the program seems to occupy exactly the same amount of memory over time.  
> **username_0**: I was thinking about posting it to Stack Overflow, but since it's kmer related problem and since I prefer to interact with this community I have decided to post it here.  
> **username_2**: As the author [suggested](https://github.com/tbenavi1/smudgeplot/issues/18#issuecomment-426609963), this is probably caused by a deadlock in multi-threading.  
> **username_3**: you probably need to do "from multiprocessing import Pool" and not "Queue" or "spawn"  

---

## 1 answers

### Answer by username_4 on 2018-12-23. Score: 2

You've hit the RAM bottleneck. Its very common in de novo assemblies and resolved using template assemblies. I don't know this program but suspect it is performing a pairwise analysis for every piece of data in a genomics data set.

The solution is to either:

> 1. find a bigger machine, e.g. a big multi-core machine and make sure the program distributes across every core.
> 2. parallelise the calculation (an excellent way to overcome RAM problems), or,
> 3. find an approximation to the calculation that is less computationally demanding, templates are a good way to circumvent this.


---

