<p>You can do this with the <code>sort</code> command-line tool. This command pipe sequence sorts numerically by the second field, then chooses unique elements based on the first field</p>
<pre><code>sort -r -k 2n,2 read_lengths.txt | sort -r -u -k 1,1 &gt; unique_lengths.txt
</code></pre>
<p><em>[the additional <code>-r</code> bits are not strictly necessary; they help make sure that the headings stay at the top]</em></p>

CC BY-SA 4.0
2023-07-11T22:12:41.737Z
2023-07-12T16:16:09.527Z
2023-07-12T16:16:09.527Z
gringer
<p><s>Not sure how you'd do this in python but</s> you need a group-by operation here. Group by the read name and pick the min length per group.</p>
<p>EDIT: With pandas, you can do:</p>
<pre class="lang-py prettyprint-override"><code>df.groupby(['read'])['length'].agg('min')
</code></pre>

CC BY-SA 4.0
2023-07-11T18:45:50.543Z
That's so much simpler and it worked! Thank you!
rimo
CC BY-SA 4.0
2023-07-11T17:48:56.983Z
2023-07-11T17:48:56.983Z
Ram RS
<p>I have a tsv file that lists the reads and read lengths from a FASTA file but some reads are duplicated - that's just from the analysis I did previously - but I want to only take one instance of the read and have it be the SHORTER length than the other duplicate.</p>
<p>For instance, here is the beginning of the tsv file:</p>
<pre><code>read                                    length
91290b97-5ca1-489d-b488-17afc1d217e7    3584
dad08436-3cd6-40a1-9af1-b846bd24a815    3787
f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
f4a7bbdf-de62-48a8-a8b0-06076e251820    4717
21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520
</code></pre>
<p>You can see that <code>f4a7bbdf-de62-48a8-a8b0-06076e251820</code> is found twice with different lengths <code>3637</code> and <code>4717</code>. I want to go through and take the shorter length of that read and include it in the final tsv file, resulting in this structure:</p>
<pre><code>read                                    length
91290b97-5ca1-489d-b488-17afc1d217e7    3584
dad08436-3cd6-40a1-9af1-b846bd24a815    3787
f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520
</code></pre>
<p>I'm trying to do this in python and have started with making the tsv file a list:</p>
<pre class="lang-py prettyprint-override"><code>newlist = []
duplist = []

with open(tsv_path) as file:
    tsv_file = csv.reader(file, delimiter=&quot;\t&quot;)
    
    for line in tsv_file:
        if line[0] not in newlist:
            newlist.append(line[0])
        else:
            duplist.append(line)
            
    print(&quot;All the duplicate reads: &quot;, duplist)
    print(&quot;The normal reads: &quot;, newlist)
</code></pre>
<p>Which works to contain the first instance of a read in one list and the second/third instance of the read to go in a separate list with their lengths. However, the shorter length of a read is not always the first instance of a read. <strong>Do I need to include another loop that looks at the lengths and compares each instance of a read? Is there an easier way to do this?</strong></p>

<python><file-formats><pandas><duplications>
Getting rid of duplicates in a dictionary
# Getting rid of duplicates in a dictionary

(Asked by: username_0 on 2023-07-11)

I have a tsv file that lists the reads and read lengths from a FASTA file but some reads are duplicated - that's just from the analysis I did previously - but I want to only take one instance of the read and have it be the SHORTER length than the other duplicate.
For instance, here is the beginning of the tsv file:
read                                    length
91290b97-5ca1-489d-b488-17afc1d217e7    3584
dad08436-3cd6-40a1-9af1-b846bd24a815    3787
f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
f4a7bbdf-de62-48a8-a8b0-06076e251820    4717
21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520

You can see that f4a7bbdf-de62-48a8-a8b0-06076e251820 is found twice with different lengths 3637 and 4717. I want to go through and take the shorter length of that read and include it in the final tsv file, resulting in this structure:
read                                    length
91290b97-5ca1-489d-b488-17afc1d217e7    3584
dad08436-3cd6-40a1-9af1-b846bd24a815    3787
f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520

I'm trying to do this in python and have started with making the tsv file a list:
newlist = []
duplist = []

with open(tsv_path) as file:
    tsv_file = csv.reader(file, delimiter="\t")
    
    for line in tsv_file:
        if line[0] not in newlist:
            newlist.append(line[0])
        else:
            duplist.append(line)
            
    print("All the duplicate reads: ", duplist)
    print("The normal reads: ", newlist)

Which works to contain the first instance of a read in one list and the second/third instance of the read to go in a separate list with their lengths. However, the shorter length of a read is not always the first instance of a read. Do I need to include another loop that looks at the lengths and compares each instance of a read? Is there an easier way to do this?

**Tags:** python, file-formats, pandas, duplications

---

## 2 answers

### Answer by username_2 on 2023-07-11. Score: 3 | Accepted answer

Not sure how you'd do this in python but you need a group-by operation here. Group by the read name and pick the min length per group.
EDIT: With pandas, you can do:
df.groupby(['read'])['length'].agg('min')

> **Comments:**
> **username_0**: That's so much simpler and it worked! Thank you!  

---

### Answer by username_1 on 2023-07-11. Score: 2

You can do this with the sort command-line tool. This command pipe sequence sorts numerically by the second field, then chooses unique elements based on the first field
sort -r -k 2n,2 read_lengths.txt | sort -r -u -k 1,1 > unique_lengths.txt

[the additional -r bits are not strictly necessary; they help make sure that the headings stay at the top]


---

