<p>I wouldn't use Python for this, myself. Instead you can use the FastaToTbl and TblToFasta scripts <a href="https://bioinformatics.stackexchange.com/a/2651/298">I have posted before</a>, and pipe to standard *nix utilities:</p>
<pre><code>FastaToTbl file.fa | sort -u | TblToFasta &gt; file.uniq.fa
</code></pre>
<p>Or, if you don't have GNU <code>sort</code>:</p>
<pre><code>FastaToTbl file.fa | sort | uniq | TblToFasta &gt; file.uniq.fa
</code></pre>
<p>Alternatively, you can also do something like this:</p>
<pre><code>awk -v RS='&gt;' -v ORS='&gt;' '++a[$0]==1' file.fa | sed 's/&gt;$//'
</code></pre>
<p>For example, given this input file:</p>
<pre><code>$ cat file.fa
&gt;seq1
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
&gt;seq2
ATGGCGCGCTTAGGAGCGCTAGGACT
&gt;seq2
ATGGCGCGCTTAGGAGCGCTAGGACT
&gt;seq1
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
&gt;seq1
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
</code></pre>
<p>You can do:</p>
<pre><code>$ awk -v RS='&gt;' -v ORS='&gt;' '++a[$0]==1' file.fa | sed 's/&gt;$//'
&gt;seq1
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
&gt;seq2
ATGGCGCGCTTAGGAGCGCTAGGACT
</code></pre>
<p>Both of the above approaches have the advantage of not needing to store the entire file in memory.</p>

CC BY-SA 4.0
2022-02-24T21:19:46.530Z
There is also an application at UCSC utilities that converts Fasta to Tab separated file. It's called `faToTab` and can be downloaded from http://hgdownload.soe.ucsc.edu/admin/exe/
Supertech
CC BY-SA 4.0
2022-02-24T11:23:35.720Z
2022-02-24T12:09:38.957Z
2022-02-24T12:09:38.957Z
terdon
<p>I would use a <code>dict</code> so that the lookup time is constant. If you are creating an array with all the unique gene IDs, then checking if a gene is in that array, your runtime is O(N<sup>2</sup>). When you create an entry in the <code>dict</code>, the time to access is constant, and if the entry is already present, it will just overwrite it. Something like this should do it for you:</p>
<pre><code>from Bio import SeqIO
    
genes = {}
fasta_file = sys.argv[1]
for entry in SeqIO.parse( fasta_file, &quot;fasta&quot; ):
    genes[ entry.id ] = entry.seq
    
</code></pre>
<p>Then you can write a new, unique fasta file:</p>
<pre><code>with open('unique_genes.fasta', 'w') as out:
    for key,value in genes.items():
        out.write( '&gt;%s\n%s\n'%(key,value) )
</code></pre>

CC BY-SA 4.0
2021-01-28T19:29:37.060Z
2021-01-28T19:29:37.060Z
Sevy
<pre><code>from Bio import SeqIO
import sys
import re 

unique = []
fasta_file = (sys.argv[1])
for myfile in SeqIO.parse(fasta_file, &quot;fasta&quot;):
    if len(myfile) &gt; 250:
        gene_id = myfile.id
        if gene.id not in unique:
            unique.append(gene.id)
            mylist = re.match(r&quot;H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+&quot;, gene_id)
            print (&quot;&gt;&quot;+mylist.group(1))
</code></pre>

CC BY-SA 4.0
2021-01-27T15:55:26.173Z
Yeah, I would do something along these lines too. Though, I would use a set or dict insead of list if the fasta file is big...
Kamil S Jaron
CC BY-SA 4.0
2020-12-28T11:46:17.773Z
2022-02-23T21:36:32.557Z
2022-02-23T21:36:32.557Z
pippo1980
<p>Perhaps itertools might be an option? From the itertools page / moreitertools project
<a href="https://docs.python.org/3/library/itertools.html#itertools-recipes" rel="nofollow noreferrer">https://docs.python.org/3/library/itertools.html#itertools-recipes</a></p>
<pre class="lang-py prettyprint-override"><code>def unique_everseen(iterable, key=None):
    &quot;List unique elements, preserving order. Remember all elements ever seen.&quot;
    # unique_everseen('AAAABBBCCDAABBB') --&gt; A B C D
    # unique_everseen('ABBCcAD', str.lower) --&gt; A B C D
    seen = set()
    seen_add = seen.add
    if key is None:
        for element in filterfalse(seen.__contains__, iterable):
            seen_add(element)
            yield element
    else:
        for element in iterable:
            k = key(element)
            if k not in seen:
                seen_add(k)
                yield element
</code></pre>

CC BY-SA 4.0
2021-01-28T08:57:04.337Z
2021-01-28T08:57:04.337Z
Mladen Rasic
<p>I use the following code. It doesn't only remove repeated sequences but similar sequences (with a quite simplistic approach).
I use it just sporadically and I guess it can be largely depurated. Remove the debug prints if these annoy you.</p>
<pre><code>import sys
import requests 
import shlex, subprocess
import random
from Bio import SeqIO
import editdistance

if len(sys.argv)&lt;3:
    print(&quot;Uso: &quot;+sys.argv[0]+&quot; input_fasta output_fasta&quot;)
    sys.exit(0)

savefile = sys.argv[2]
content_file = sys.argv[1]

class Sequence:
    id=&quot;&quot;
    sequence=&quot;&quot;

def are_similar(seq1, seq2):
    l1 = len(seq1.sequence)
    l2 = len(seq2.sequence)
    if abs(l1-l2) &lt; l1/40:
        d = editdistance.eval(seq1.sequence[0:250], seq2.sequence[0:250])
        #d = editdistance.eval(seq1.sequence, seq2.sequence)
        print(seq1.id+&quot; &quot;+seq2.id+&quot; -&gt; &quot;+str(d))
        #if d &lt; l1/20:
        if d &lt; 50:
            return True
    return False

def find_repeats(sequences, seq, repeated):
    b=False
    for seq2 in sequences:
        if(b and not seq2.id in repeated):
            if are_similar(seq, seq2):
                repeated.add(seq2.id)        
        if seq2.id==seq.id:
            b=True

sequences=[]

res=&quot;&quot;

for record in SeqIO.parse(content_file, &quot;fasta&quot;):
    seq=Sequence()
    seq.id=record.id
    seq.sequence=record.seq
    sequences.append(seq)

repeated_sequences=set()
for seq in sequences:
    find_repeats(sequences, seq, repeated_sequences)

print(repeated_sequences)

def mySortFunc(e):
    return len(e.sequence)

sequences.sort(key=mySortFunc)

for seq in sequences:
    b= seq.id in repeated_sequences
    if not b:
        res+=&quot;&gt;&quot;+seq.id+&quot;\n&quot;
        res+=str(seq.sequence)+&quot;\n&quot;

output_file = open(savefile, &quot;w&quot;)

output_file.write(res)

output_file.close()
</code></pre>
<p>EDIT: I see that's not exactly what OP is asking ...</p>

CC BY-SA 4.0
2021-01-28T11:45:42.850Z
2021-01-29T18:24:15.830Z
2021-01-29T18:24:15.830Z
juanjo75es
<p>Your question isn't well formatted, for example list is used as a variable name where it should be reserved as a keyword...</p>
<pre><code>print (&quot;&gt;&quot;+list.group(1))
</code></pre>
<p>However, the answer you're after is going to be something like:</p>
<pre><code>print (&quot;&gt;&quot;+set(name_of_list.group(1)))
</code></pre>
<p>Anytime you want to get the unique elements of a list, just wrap it in set()</p>

CC BY-SA 4.0
2021-01-28T03:33:54.757Z
2021-01-28T03:33:54.757Z
Liam McIntyre
<p>I am using the following command:</p>
<pre><code>from Bio import SeqIO
import sys
import re 


fasta_file = (sys.argv[1])
for myfile in SeqIO.parse(fasta_file, &quot;fasta&quot;):
if len(myfile.seq) &gt; 250:
    gene_id = myfile.id
    mylist = re.match(r&quot;H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+&quot;, gene_id)
    print (&quot;&gt;&quot;+mylist.group(1)) 
</code></pre>
<p>And this is providing me with duplicates of the same gene:</p>
<pre><code>&gt;H149xcV_Fge342_r3_h2
&gt;H149xcV_bTr423_r3_h2
&gt;H149xcV_kN893_r3_h2
&gt;H149xcV_DNp021_r3_h2
&gt;H149xcV_JEP3324_r3_h2
&gt;H149xcV_JEP3324_r3_h2
&gt;H149xcV_JEP3324_r3_h2
&gt;H149xcV_SRt424234_r3_h2
&gt;H149xcV_Fge342_r3_h2
&gt;H149xcV_Fge342_r3_h2
</code></pre>
<p>How can I reformat my command so that I only receive unique gene id's:</p>
<pre><code>&gt;H149xcV_Fge342_r3_h2
&gt;H149xcV_bTr423_r3_h2
&gt;H149xcV_kN893_r3_h2
&gt;H149xcV_DNp021_r3_h2
&gt;H149xcV_JEP3324_r3_h2
&gt;H149xcV_SRt424234_r3_h2
</code></pre>

CC BY-SA 4.0
2020-11-24T00:21:36.387Z
Why do you need to use python for this?
Ram RS
CC BY-SA 4.0
2020-11-24T00:23:16.810Z
I am using `nano` within commandline to write my python command. @RamRS
andnowmywatchbegins
CC BY-SA 4.0
2020-11-24T02:28:16.413Z
I'm asking **why**, not **how**. There are command line tools that de-dup fasta, so why reinvent the wheel?
Ram RS
CC BY-SA 4.0
2020-11-24T02:46:29.597Z
@RamRS The only command-line tools that I am aware of are `sort`|`uniq` but I am not sure as to how I would use it? I must apologize, I am kind of new to bioinformatics, so I am still getting the hang of it. I know I can output my python into another file, but is there a way to obtain the unique id's afterwards? Thanks!
andnowmywatchbegins
CC BY-SA 4.0
2020-11-24T09:47:51.473Z
You can use  `seqkit rmdup` instead reinvent the wheel. https://bioinf.shenwei.me/seqkit/usage/#rmdup
zorbax
CC BY-SA 4.0
2021-01-27T15:47:01.317Z
@zorbax I think that would be a good answer...
Kamil S Jaron
<fasta><python>
How to remove duplicates from a fasta file using python
# How to remove duplicates from a fasta file using python

(Asked by: username_0 on 2020-11-23)

I am using the following command:
from Bio import SeqIO
import sys
import re 


fasta_file = (sys.argv[1])
for myfile in SeqIO.parse(fasta_file, "fasta"):
if len(myfile.seq) > 250:
    gene_id = myfile.id
    mylist = re.match(r"H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+", gene_id)
    print (">"+mylist.group(1)) 

And this is providing me with duplicates of the same gene:
>H149xcV_Fge342_r3_h2
>H149xcV_bTr423_r3_h2
>H149xcV_kN893_r3_h2
>H149xcV_DNp021_r3_h2
>H149xcV_JEP3324_r3_h2
>H149xcV_JEP3324_r3_h2
>H149xcV_JEP3324_r3_h2
>H149xcV_SRt424234_r3_h2
>H149xcV_Fge342_r3_h2
>H149xcV_Fge342_r3_h2

How can I reformat my command so that I only receive unique gene id's:
>H149xcV_Fge342_r3_h2
>H149xcV_bTr423_r3_h2
>H149xcV_kN893_r3_h2
>H149xcV_DNp021_r3_h2
>H149xcV_JEP3324_r3_h2
>H149xcV_SRt424234_r3_h2

**Tags:** fasta, python
> **Comments:**
> **username_1**: Why do you need to use python for this?  
> **username_0**: I am using `nano` within commandline to write my python command. @RamRS  
> **username_1**: I'm asking **why**, not **how**. There are command line tools that de-dup fasta, so why reinvent the wheel?  
> **username_0**: @RamRS The only command-line tools that I am aware of are `sort`|`uniq` but I am not sure as to how I would use it? I must apologize, I am kind of new to bioinformatics, so I am still getting the hang of it. I know I can output my python into another file, but is there a way to obtain the unique id's afterwards? Thanks!  
> **username_2**: You can use  `seqkit rmdup` instead reinvent the wheel. https://bioinf.shenwei.me/seqkit/usage/#rmdup  
> **username_3**: @zorbax I think that would be a good answer...  

---

## 6 answers

### Answer by username_4 on 2022-02-24. Score: 2

I wouldn't use Python for this, myself. Instead you can use the FastaToTbl and TblToFasta scripts I have posted before, and pipe to standard *nix utilities:
FastaToTbl file.fa | sort -u | TblToFasta > file.uniq.fa

Or, if you don't have GNU sort:
FastaToTbl file.fa | sort | uniq | TblToFasta > file.uniq.fa

Alternatively, you can also do something like this:
awk -v RS='>' -v ORS='>' '++a[$0]==1' file.fa | sed 's/>$//'

For example, given this input file:
$ cat file.fa
>seq1
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
>seq2
ATGGCGCGCTTAGGAGCGCTAGGACT
>seq2
ATGGCGCGCTTAGGAGCGCTAGGACT
>seq1
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
>seq1
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA

You can do:
$ awk -v RS='>' -v ORS='>' '++a[$0]==1' file.fa | sed 's/>$//'
>seq1
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
ACTTCGCAGAGGCTTCGGAGAGA
>seq2
ATGGCGCGCTTAGGAGCGCTAGGACT

Both of the above approaches have the advantage of not needing to store the entire file in memory.

> **Comments:**
> **username_5**: There is also an application at UCSC utilities that converts Fasta to Tab separated file. It's called `faToTab` and can be downloaded from http://hgdownload.soe.ucsc.edu/admin/exe/  

---

### Answer by username_6 on 2021-01-28. Score: 1

I would use a dict so that the lookup time is constant. If you are creating an array with all the unique gene IDs, then checking if a gene is in that array, your runtime is O(N2). When you create an entry in the dict, the time to access is constant, and if the entry is already present, it will just overwrite it. Something like this should do it for you:
from Bio import SeqIO
    
genes = {}
fasta_file = sys.argv[1]
for entry in SeqIO.parse( fasta_file, "fasta" ):
    genes[ entry.id ] = entry.seq
    

Then you can write a new, unique fasta file:
with open('unique_genes.fasta', 'w') as out:
    for key,value in genes.items():
        out.write( '>%s\n%s\n'%(key,value) )


---

### Answer by username_7 on 2020-12-28. Score: 1

from Bio import SeqIO
import sys
import re 

unique = []
fasta_file = (sys.argv[1])
for myfile in SeqIO.parse(fasta_file, "fasta"):
    if len(myfile) > 250:
        gene_id = myfile.id
        if gene.id not in unique:
            unique.append(gene.id)
            mylist = re.match(r"H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+", gene_id)
            print (">"+mylist.group(1))

> **Comments:**
> **username_3**: Yeah, I would do something along these lines too. Though, I would use a set or dict insead of list if the fasta file is big...  

---

### Answer by username_8 on 2021-01-28. Score: 1

Perhaps itertools might be an option? From the itertools page / moreitertools project
https://docs.python.org/3/library/itertools.html#itertools-recipes
def unique_everseen(iterable, key=None):
    "List unique elements, preserving order. Remember all elements ever seen."
    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
    # unique_everseen('ABBCcAD', str.lower) --> A B C D
    seen = set()
    seen_add = seen.add
    if key is None:
        for element in filterfalse(seen.__contains__, iterable):
            seen_add(element)
            yield element
    else:
        for element in iterable:
            k = key(element)
            if k not in seen:
                seen_add(k)
                yield element


---

### Answer by username_9 on 2021-01-28. Score: 0

I use the following code. It doesn't only remove repeated sequences but similar sequences (with a quite simplistic approach).
I use it just sporadically and I guess it can be largely depurated. Remove the debug prints if these annoy you.
import sys
import requests 
import shlex, subprocess
import random
from Bio import SeqIO
import editdistance

if len(sys.argv)<3:
    print("Uso: "+sys.argv[0]+" input_fasta output_fasta")
    sys.exit(0)

savefile = sys.argv[2]
content_file = sys.argv[1]

class Sequence:
    id=""
    sequence=""

def are_similar(seq1, seq2):
    l1 = len(seq1.sequence)
    l2 = len(seq2.sequence)
    if abs(l1-l2) < l1/40:
        d = editdistance.eval(seq1.sequence[0:250], seq2.sequence[0:250])
        #d = editdistance.eval(seq1.sequence, seq2.sequence)
        print(seq1.id+" "+seq2.id+" -> "+str(d))
        #if d < l1/20:
        if d < 50:
            return True
    return False

def find_repeats(sequences, seq, repeated):
    b=False
    for seq2 in sequences:
        if(b and not seq2.id in repeated):
            if are_similar(seq, seq2):
                repeated.add(seq2.id)        
        if seq2.id==seq.id:
            b=True

sequences=[]

res=""

for record in SeqIO.parse(content_file, "fasta"):
    seq=Sequence()
    seq.id=record.id
    seq.sequence=record.seq
    sequences.append(seq)

repeated_sequences=set()
for seq in sequences:
    find_repeats(sequences, seq, repeated_sequences)

print(repeated_sequences)

def mySortFunc(e):
    return len(e.sequence)

sequences.sort(key=mySortFunc)

for seq in sequences:
    b= seq.id in repeated_sequences
    if not b:
        res+=">"+seq.id+"\n"
        res+=str(seq.sequence)+"\n"

output_file = open(savefile, "w")

output_file.write(res)

output_file.close()

EDIT: I see that's not exactly what OP is asking ...


---

### Answer by username_10 on 2021-01-28. Score: 0

Your question isn't well formatted, for example list is used as a variable name where it should be reserved as a keyword...
print (">"+list.group(1))

However, the answer you're after is going to be something like:
print (">"+set(name_of_list.group(1)))

Anytime you want to get the unique elements of a list, just wrap it in set()


---

