<p>Note, its a while since I've used <code>scipy</code> and I don't fancy loading an deleting for this test ... so in the absence of <code>pdist()</code> the solution would be ...</p>
<pre class="lang-py prettyprint-override"><code>import numpy as np
import pandas as pd
from pathlib import Path

data = [[1,2,3],[0,5,6],[7,0,9],[10,11,0]]
arr = np.tril([data]).astype('float')
arr[arr == 0] = 'nan' 
arr = np.where(np.isnan(arr), '', arr)
columns = ['a','b','c', 'd']
file = Path('/Volumes/Data2/test/test1.meg')
if file.is_file():
    file.unlink()
with open(file, 'a') as fout:
    fout.write(&quot;#MEGA\n&quot;)
    for col in columns:
        fout.write(col + &quot;\n&quot;)
    fout.write(&quot;\n0\n&quot;)
    np.savetxt(fout, arr.reshape(3,-1), fmt='%s')
</code></pre>
<p><strong>output</strong></p>
<pre><code>#MEGA
a
b
c
d

0
1.0 0.0
5.0 7.0 0.0
9.0 10.0 11.0 0.0
</code></pre>
<p>That works. Keep in mind where it says <code>arr.reshape(3,-1)</code> the <code>3</code> should be the number of columns -1 in your matrix: <strong>in your case this appears to be 177 (178 - 1)</strong>.</p>
<p>The gist is to use <code>np.tril</code>, which doesn't exists in <code>pandas</code>. Hence don't use <code>pandas</code>.</p>
<p>At a guess with <code>scipy</code> then it would be ...</p>
<pre class="lang-py prettyprint-override"><code>import numpy as np
import pandas as pd
from pathlib import Path
from scipy import pdist

tmp = [[1,2,3],[4,5,6],[7,8,9],[10,11,12]]

data = pdist(tmp, 'jaccard')
arr = np.tril([data]).astype('float')
arr[arr == 0] = 'nan' 
arr = np.where(np.isnan(arr), '', arr)
columns = ['a','b','c', 'd']
file =Path('/Volumes/Data2/test/test1.meg')
if file.is_file():
    file.unlink()
with open(file, 'a') as fout:
    fout.write(&quot;#MEGA\n&quot;)
    for col in columns:
        fout.write(col + &quot;\n&quot;)
    fout.write(&quot;\n0\n&quot;)
    np.savetxt(fout, arr.reshape(3,-1), fmt='%s')
</code></pre>
<p>You might need a quick <code>perl -p -i -e 's/[&quot; &quot;]+/ /g' test1.meg</code> or even <code>perl -p -i -e 's/[&quot; &quot;]+/\t/g' test1.meg</code> to set the file for MEGA. I dunno why but <code>\s</code> or <code>&quot; &quot;</code> in the transform part of perl pie isn't working for me.</p>

CC BY-SA 4.0
2024-01-13T04:07:01.783Z
2024-01-15T10:41:51.923Z
2024-01-15T10:41:51.923Z
M__
<p>This is an add-on to the previous question I asked <a href="https://bioinformatics.stackexchange.com/questions/22005/process-gets-killed-before-completing/22006?noredirect=1#comment34017_22006">here</a>.</p>
<p>I was able to deploy a &quot;chunking&quot; strategy to my kmer matrix and was able to get a file with the distance measurements! From what I could tell by using the <code>head</code> command was that the data looks like what it did before when I was using this approach on a smaller dataset so now I'm going to use this data to generate a tree using MEGAX.</p>
<p>Normally when I'd convert the <code>tsv</code> file to a <code>meg</code> file I'd manually add in the meg headers/requirements for conversion. However, I can't open the distance <code>tsv</code> file (its ~23 GB) to make these changes.</p>
<p>Is there an easy solution to this problem? How do you convert a <code>tsv</code> file to a <code>meg</code> file without opening the file?</p>
<p>EDIT:</p>
<p>For more context here is the format for each file.</p>
<p>Currently, my <code>tsv</code> file looks like this:</p>
<pre><code>0 1 2 3 4 5 6 7 8 9 10 ... 178
1 0 # # # # # # # # #  ... #
2 # 0 # # # # # # # #  ... #
3 # # 0 # # # # # # #  ... #
4 # # # 0 # # # # # #  ... #
5 # # # # 0 # # # # #  ... #
6 # # # # # 0 # # # #  ... #
7 # # # # # # 0 # # #  ... #
8 # # # # # # # 0 # #  ... #
9 # # # # # # # # 0 #  ... #
10 # # # # # # # # # 0 ... #
...
178 # # # # # # # # # #  ... 0
</code></pre>
<p>This is a jaccard distance matrix of 178 samples. The <code>#</code> represent the distance values for each sample compared to the other. I was able to calcualte the distance using pandas and scipy: <code>matrix = pd.DataFrame(squareform(pdist(data, 'jaccard')))</code></p>
<p>And I would like it to look like this:</p>
<pre><code>#mega
TITLE: Jaccard distance of 178 samples

sample_1
sample_2
sample_3
...
sample_178

distance_1
distance_1 distance_2
distance_1 distance_2 distance_3
distance_1 distance_2 distance_3 distance_4
...
distance_1 distance_2 distance_3 distance_4...distance_178
</code></pre>

CC BY-SA 4.0
2024-01-12T20:47:39.617Z
I added some extra information to give more context to this issue! Let me know if you'd like me to clarify something further or if you'd like more information. I appreciate the help!
rimo
CC BY-SA 4.0
2024-01-12T23:41:48.303Z
Thanks @rimo .. okay this ain't the data set I imagined. I'll need to reproduce it. Are you using SciPy, i.e. where is the pdist() method coming from?
M__
<format-conversion><k-mer><megax>
Converting .tsv to .meg for visualization
# Converting .tsv to .meg for visualization

(Asked by: username_0 on 2024-01-12)

This is an add-on to the previous question I asked here.
I was able to deploy a "chunking" strategy to my kmer matrix and was able to get a file with the distance measurements! From what I could tell by using the head command was that the data looks like what it did before when I was using this approach on a smaller dataset so now I'm going to use this data to generate a tree using MEGAX.
Normally when I'd convert the tsv file to a meg file I'd manually add in the meg headers/requirements for conversion. However, I can't open the distance tsv file (its ~23 GB) to make these changes.
Is there an easy solution to this problem? How do you convert a tsv file to a meg file without opening the file?
EDIT:
For more context here is the format for each file.
Currently, my tsv file looks like this:
0 1 2 3 4 5 6 7 8 9 10 ... 178
1 0 # # # # # # # # #  ... #
2 # 0 # # # # # # # #  ... #
3 # # 0 # # # # # # #  ... #
4 # # # 0 # # # # # #  ... #
5 # # # # 0 # # # # #  ... #
6 # # # # # 0 # # # #  ... #
7 # # # # # # 0 # # #  ... #
8 # # # # # # # 0 # #  ... #
9 # # # # # # # # 0 #  ... #
10 # # # # # # # # # 0 ... #
...
178 # # # # # # # # # #  ... 0

This is a jaccard distance matrix of 178 samples. The # represent the distance values for each sample compared to the other. I was able to calcualte the distance using pandas and scipy: matrix = pd.DataFrame(squareform(pdist(data, 'jaccard')))
And I would like it to look like this:
#mega
TITLE: Jaccard distance of 178 samples

sample_1
sample_2
sample_3
...
sample_178

distance_1
distance_1 distance_2
distance_1 distance_2 distance_3
distance_1 distance_2 distance_3 distance_4
...
distance_1 distance_2 distance_3 distance_4...distance_178

**Tags:** format-conversion, k-mer, megax
> **Comments:**
> **username_0**: I added some extra information to give more context to this issue! Let me know if you'd like me to clarify something further or if you'd like more information. I appreciate the help!  
> **username_1**: Thanks @rimo .. okay this ain't the data set I imagined. I'll need to reproduce it. Are you using SciPy, i.e. where is the pdist() method coming from?  

---

## 1 answers

### Answer by username_1 on 2024-01-13. Score: 1

Note, its a while since I've used scipy and I don't fancy loading an deleting for this test ... so in the absence of pdist() the solution would be ...
import numpy as np
import pandas as pd
from pathlib import Path

data = [[1,2,3],[0,5,6],[7,0,9],[10,11,0]]
arr = np.tril([data]).astype('float')
arr[arr == 0] = 'nan' 
arr = np.where(np.isnan(arr), '', arr)
columns = ['a','b','c', 'd']
file = Path('/Volumes/Data2/test/test1.meg')
if file.is_file():
    file.unlink()
with open(file, 'a') as fout:
    fout.write("#MEGA\n")
    for col in columns:
        fout.write(col + "\n")
    fout.write("\n0\n")
    np.savetxt(fout, arr.reshape(3,-1), fmt='%s')

output
#MEGA
a
b
c
d

0
1.0 0.0
5.0 7.0 0.0
9.0 10.0 11.0 0.0

That works. Keep in mind where it says arr.reshape(3,-1) the 3 should be the number of columns -1 in your matrix: in your case this appears to be 177 (178 - 1).
The gist is to use np.tril, which doesn't exists in pandas. Hence don't use pandas.
At a guess with scipy then it would be ...
import numpy as np
import pandas as pd
from pathlib import Path
from scipy import pdist

tmp = [[1,2,3],[4,5,6],[7,8,9],[10,11,12]]

data = pdist(tmp, 'jaccard')
arr = np.tril([data]).astype('float')
arr[arr == 0] = 'nan' 
arr = np.where(np.isnan(arr), '', arr)
columns = ['a','b','c', 'd']
file =Path('/Volumes/Data2/test/test1.meg')
if file.is_file():
    file.unlink()
with open(file, 'a') as fout:
    fout.write("#MEGA\n")
    for col in columns:
        fout.write(col + "\n")
    fout.write("\n0\n")
    np.savetxt(fout, arr.reshape(3,-1), fmt='%s')

You might need a quick perl -p -i -e 's/[" "]+/ /g' test1.meg or even perl -p -i -e 's/[" "]+/\t/g' test1.meg to set the file for MEGA. I dunno why but \s or " " in the transform part of perl pie isn't working for me.


---

