<p>And here’s a solution using R (with the Bioconductor):</p>

<pre class="lang-r prettyprint-override"><code>fa = ShortRead::readFasta(infile)
ids = as.character(ShortRead::id(fa))
fa@id = Biostrings::BStringSet(make.unique(ids, sep = '_'))
ShortRead::writeFasta(fa, outfile)
</code></pre>

CC BY-SA 3.0
2017-07-04T13:10:58.163Z
2017-07-04T17:14:57.440Z
2017-07-04T17:14:57.440Z
Konrad Rudolph
<p>Sure, using biopython:</p>

<pre><code>#!/usr/bin/env python
from Bio import SeqIO

records = set()
of = open("output.fa", "w")
for record in SeqIO.parse("foo.fa", "fasta"):
    ID = record.id
    num = 1
    while ID in records:
        ID = "{}_{}".format(record.id, num)
        num += 1
    records.add(ID)
    record.id = ID
    record.name = ID
    record.description = ID
    SeqIO.write(record, of, "fasta")
of.close()
</code></pre>

<p>Change <code>output.fa</code> and <code>foo.fa</code>. One doesn't need to explicitly change the <code>.name</code> and <code>.description</code>, but that's handy to prevent the original ID from not appearing still (after a space).</p>

<h1>Explanation</h1>

<ul>
<li><code>records = set()</code> This will create a lookup table of all IDs <strong>written</strong></li>
<li><code>for record in SeqIO.parse("foo.fa", "fasta"):</code> Open <code>foo.fa</code> as a fasta file and iterate over the <strong>entries</strong> in it. These are objects with an ID, name, description, and sequence attribute (the name and description are the same as the ID if not present).</li>
<li><code>ID = record.id</code> <a href="https://en.wikipedia.org/wiki/Memoization" rel="nofollow noreferrer">Memoize</a> the entry ID (e.g., <code>1_duplicateName</code>)</li>
<li><code>while ID in records:</code> As long as the ID has already been seen, keep looping.</li>
<li><code>ID = "{}_{}".format(record.id, num)</code> Start adding increasing numbers after the ID, such as <code>1_duplicateName_1</code> and <code>1_duplicateName_2</code>. This will continue until the ID has not been seen.</li>
<li><code>records.add(ID)</code> Add the unseen ID to the set.</li>
<li><code>record.id = ID</code> Update the ID, the <code>.name</code> and <code>.description</code> are the same. If you don't do that, then you get output like <code>&gt;1_duplicateName_1 1_duplicateName</code>. That's not really a problem, but it's excessive.</li>
<li><code>SeqIO.write(record, of, "fasta")</code> Write the record to the output file in fasta format.</li>
</ul>

<p>A benefit of biopython is the easy ability to change formats, so instead of fasta one could have substituted Genbank or another format if needed.</p>

CC BY-SA 3.0
2017-07-04T12:02:17.610Z
2017-07-05T15:49:58.337Z
2017-07-05T15:49:58.337Z
Devon Ryan
<p>Sure, this little Perl snippet should do it:</p>

<pre><code>$ perl -pe 's/$/_$seen{$_}/ if ++$seen{$_}&gt;1 and /^&gt;/; ' file.fa 
&gt;1_uniqueGeneName
atgc
&gt;1_anotherUniqueGeneName
atgc_2
&gt;1_duplicateName
atgc_3
&gt;1_duplicateName_2
atgc_4
</code></pre>

<p>Or, to make the changes in the original file, use <code>-i</code>:    </p>

<pre><code>perl -i.bak -pe 's/$/_$seen{$_}/ if ++$seen{$_}&gt;1 and /^&gt;/; ' file.fa 
</code></pre>

<p>Note that the first occurrence of a duplicate name isn't changed, the second will become <code>_2</code>, the third <code>_3</code> etc. </p>

<h3>Explanation</h3>

<ul>
<li><code>perl -pe</code> : <strong>p</strong>rint each input line after applying the script given by <code>-e</code> to it. </li>
<li><code>++$seen{$_}&gt;1</code> : increment the current value stored in the hash <code>%seen</code> for this line (<code>$_</code>) by 1 and compare it to <code>1</code>. </li>
<li><code>s/$/_$seen{$_}/ if ++$seen{$_}&gt;1 and /^&gt;/</code> : if the current line starts with a <code>&gt;</code> and the  value stored in the hash <code>%seen</code> for this line is greater than 1 (if this isn't the first time we see this line), replace the end of the line (<code>$</code>) with a <code>_</code> and the current value in the hash</li>
</ul>

<hr>

<p>Alternatively, here's the same idea in <code>awk</code>:</p>

<pre><code>$ awk '(/^&gt;/ &amp;&amp; s[$0]++){$0=$0"_"s[$0]}1;' file.fa 
&gt;1_uniqueGeneName
atgc
&gt;1_anotherUniqueGeneName
atgc
&gt;1_duplicateName
atgc
&gt;1_duplicateName_2
atgc
</code></pre>

<p>To make the changes in the original file (assuming you are using GNU <code>awk</code> which is the default on most Linux versions), use <code>-i inplace</code>:</p>

<pre><code>awk -iinplace '(/^&gt;/ &amp;&amp; s[$0]++){$0=$0"_"s[$0]}1;' file.fa 
</code></pre>

<h3>Explanation</h3>

<p>In awk, the special variable <code>$0</code> is the current line. </p>

<ul>
<li><code>(/^&gt;/ &amp;&amp; s[$0]++)</code> : if this line starts with a <code>&gt;</code> and incrementing the value stored in the array <code>s</code> for this line by 1 evaluates to true (is greater than 0).</li>
<li><code>$0=$0"_"s[$0]</code> :  make the current line be itself with a <code>_</code> and the value from <code>s</code> appended.</li>
<li><code>1;</code> : this is just shorthand for "print this line". If an expression evaluates to true, awk will print the current line. Since <code>1</code> is always true, this will print every line. </li>
</ul>

<hr>

<p>If you want all of the duplicates to be marked, you need to read the file twice. Once to collect the names and a second to mark them:</p>

<pre><code>$ awk '{
    if (NR==FNR){
        if(/^&gt;/){
            s[$0]++
        }
        next;
    }
    if(/^&gt;/){
        k[$0]++;
        if(s[$0]&gt;1){
            $0=$0"_"k[$0]
        }
    }
    print
}' file.fa file.fa
&gt;1_uniqueGeneName
atgc
&gt;1_anotherUniqueGeneName
atgc
&gt;1_duplicateName_1
atgc
&gt;1_duplicateName_2
atgc
</code></pre>

<hr>

<p><strong>IMPORTANT</strong>: note that all of these approaches assume you don't already have sequence names ending with <code>_N</code> where <code>N</code> is a number. If your input file has 2 sequences called <code>foo</code> and one called <code>foo_2</code>, then you will end up with two <code>foo_2</code>:</p>

<pre><code>$ cat test.fa
&gt;foo_2
actg
&gt;foo
actg
&gt;foo
actg
$ perl -pe 's/$/_$seen{$_}/ if ++$seen{$_}&gt;1 and /^&gt;/; ' test.fa 
&gt;foo_2
actg
&gt;foo
actg
&gt;foo_2
actg
</code></pre>

<p>If this can be an issue for you, use one of the more sophisticated approaches suggested by the other answers.</p>

CC BY-SA 3.0
2017-07-04T13:37:43.387Z
In the `awk` example I worry that duplicate IDs could be created if you have contigs like `>foo` and `>foo_1` in the original file. `foo_1` wouldn't be in `s`, but might already have been printed out if `foo` had already been encountered.
Devon Ryan
CC BY-SA 3.0
2017-07-04T13:56:06.430Z
@DevonRyan ah, yes that's a very valid point. Not just the awk one either, all approaches here break if a duplicate sequence already exists with a `_N`. In that case, the extra verbiage added by python is actually worth it :P. I added a disclaimer, thanks.
terdon
CC BY-SA 3.0
2017-07-04T11:58:43.597Z
2017-07-04T13:59:59.517Z
2017-07-04T13:59:59.517Z
terdon
<p>I have a reference database with contains 100s of sequences in fasta format. Some of these sequences have duplicate names like so:</p>

<pre><code>&gt;1_uniqueGeneName
atgc
&gt;1_anotherUniqueGeneName
atgc
&gt;1_duplicateName
atgc
&gt;1_duplicateName
atgc
</code></pre>

<p>Is is possible to run through a large file like this and change the names of only the duplicates?</p>

<pre><code>&gt;1_uniqueGeneName
atgc
&gt;1_anotherUniqueGeneName
atgc
&gt;1_duplicateName_1
atgc
&gt;1_duplicateName_2
atgc
</code></pre>

CC BY-SA 3.0
2017-07-04T11:57:44.083Z
Do you want to only change the names of the duplicates, I mean of the second, third etc seqs with the same name or also of the 1st as you have done? I mean, do you really need `duplicate_1`, `duplicate_2` or is `duplicate`, `duplicate_1` enough? The latter is far simpler since the former will require you to read the file twice.
terdon
CC BY-SA 3.0
2017-07-04T12:32:35.593Z
Cool. Id prefer the `duplicate_1 duplicate_2`format but if it is much more complex then I am fine with the `duplicate duplicate_1`format.
AudileF
CC BY-SA 3.0
2017-07-04T16:08:06.887Z
@AudileF: I'm having a hard time coming up with a good reason why I would want the first duplicate to be changed as well.  Can you give us a bit more insight into the reason for this request?
winni2k
<fasta><text>
How to append numbers only on duplicates sequence names?
# How to append numbers only on duplicates sequence names?

(Asked by: username_0 on 2017-07-04)

I have a reference database with contains 100s of sequences in fasta format. Some of these sequences have duplicate names like so:
>1_uniqueGeneName
atgc
>1_anotherUniqueGeneName
atgc
>1_duplicateName
atgc
>1_duplicateName
atgc

Is is possible to run through a large file like this and change the names of only the duplicates?
>1_uniqueGeneName
atgc
>1_anotherUniqueGeneName
atgc
>1_duplicateName_1
atgc
>1_duplicateName_2
atgc

**Tags:** fasta, text
> **Comments:**
> **username_1**: Do you want to only change the names of the duplicates, I mean of the second, third etc seqs with the same name or also of the 1st as you have done? I mean, do you really need `duplicate_1`, `duplicate_2` or is `duplicate`, `duplicate_1` enough? The latter is far simpler since the former will require you to read the file twice.  
> **username_0**: Cool. Id prefer the `duplicate_1 duplicate_2`format but if it is much more complex then I am fine with the `duplicate duplicate_1`format.  
> **username_2**: @AudileF: I'm having a hard time coming up with a good reason why I would want the first duplicate to be changed as well.  Can you give us a bit more insight into the reason for this request?  

---

## 3 answers

### Answer by username_1 on 2017-07-04. Score: 5 | Accepted answer

Sure, this little Perl snippet should do it:
$ perl -pe 's/$/_$seen{$_}/ if ++$seen{$_}>1 and /^>/; ' file.fa 
>1_uniqueGeneName
atgc
>1_anotherUniqueGeneName
atgc_2
>1_duplicateName
atgc_3
>1_duplicateName_2
atgc_4

Or, to make the changes in the original file, use -i:    
perl -i.bak -pe 's/$/_$seen{$_}/ if ++$seen{$_}>1 and /^>/; ' file.fa 

Note that the first occurrence of a duplicate name isn't changed, the second will become _2, the third _3 etc. 
Explanation

perl -pe : print each input line after applying the script given by -e to it. 
++$seen{$_}>1 : increment the current value stored in the hash %seen for this line ($_) by 1 and compare it to 1. 
s/$/_$seen{$_}/ if ++$seen{$_}>1 and /^>/ : if the current line starts with a > and the  value stored in the hash %seen for this line is greater than 1 (if this isn't the first time we see this line), replace the end of the line ($) with a _ and the current value in the hash


Alternatively, here's the same idea in awk:
$ awk '(/^>/ && s[$0]++){$0=$0"_"s[$0]}1;' file.fa 
>1_uniqueGeneName
atgc
>1_anotherUniqueGeneName
atgc
>1_duplicateName
atgc
>1_duplicateName_2
atgc

To make the changes in the original file (assuming you are using GNU awk which is the default on most Linux versions), use -i inplace:
awk -iinplace '(/^>/ && s[$0]++){$0=$0"_"s[$0]}1;' file.fa 

Explanation
In awk, the special variable $0 is the current line. 

(/^>/ && s[$0]++) : if this line starts with a > and incrementing the value stored in the array s for this line by 1 evaluates to true (is greater than 0).
$0=$0"_"s[$0] :  make the current line be itself with a _ and the value from s appended.
1; : this is just shorthand for "print this line". If an expression evaluates to true, awk will print the current line. Since 1 is always true, this will print every line. 


If you want all of the duplicates to be marked, you need to read the file twice. Once to collect the names and a second to mark them:
$ awk '{
    if (NR==FNR){
        if(/^>/){
            s[$0]++
        }
        next;
    }
    if(/^>/){
        k[$0]++;
        if(s[$0]>1){
            $0=$0"_"k[$0]
        }
    }
    print
}' file.fa file.fa
>1_uniqueGeneName
atgc
>1_anotherUniqueGeneName
atgc
>1_duplicateName_1
atgc
>1_duplicateName_2
atgc


IMPORTANT: note that all of these approaches assume you don't already have sequence names ending with _N where N is a number. If your input file has 2 sequences called foo and one called foo_2, then you will end up with two foo_2:
$ cat test.fa
>foo_2
actg
>foo
actg
>foo
actg
$ perl -pe 's/$/_$seen{$_}/ if ++$seen{$_}>1 and /^>/; ' test.fa 
>foo_2
actg
>foo
actg
>foo_2
actg

If this can be an issue for you, use one of the more sophisticated approaches suggested by the other answers.

> **Comments:**
> **username_4**: In the `awk` example I worry that duplicate IDs could be created if you have contigs like `>foo` and `>foo_1` in the original file. `foo_1` wouldn't be in `s`, but might already have been printed out if `foo` had already been encountered.  
> **username_1**: @DevonRyan ah, yes that's a very valid point. Not just the awk one either, all approaches here break if a duplicate sequence already exists with a `_N`. In that case, the extra verbiage added by python is actually worth it :P. I added a disclaimer, thanks.  

---

### Answer by username_3 on 2017-07-04. Score: 7

And here’s a solution using R (with the Bioconductor):
fa = ShortRead::readFasta(infile)
ids = as.character(ShortRead::id(fa))
fa@id = Biostrings::BStringSet(make.unique(ids, sep = '_'))
ShortRead::writeFasta(fa, outfile)


---

### Answer by username_4 on 2017-07-04. Score: 6

Sure, using biopython:
#!/usr/bin/env python
from Bio import SeqIO

records = set()
of = open("output.fa", "w")
for record in SeqIO.parse("foo.fa", "fasta"):
    ID = record.id
    num = 1
    while ID in records:
        ID = "{}_{}".format(record.id, num)
        num += 1
    records.add(ID)
    record.id = ID
    record.name = ID
    record.description = ID
    SeqIO.write(record, of, "fasta")
of.close()

Change output.fa and foo.fa. One doesn't need to explicitly change the .name and .description, but that's handy to prevent the original ID from not appearing still (after a space).
Explanation

records = set() This will create a lookup table of all IDs written
for record in SeqIO.parse("foo.fa", "fasta"): Open foo.fa as a fasta file and iterate over the entries in it. These are objects with an ID, name, description, and sequence attribute (the name and description are the same as the ID if not present).
ID = record.id Memoize the entry ID (e.g., 1_duplicateName)
while ID in records: As long as the ID has already been seen, keep looping.
ID = "{}_{}".format(record.id, num) Start adding increasing numbers after the ID, such as 1_duplicateName_1 and 1_duplicateName_2. This will continue until the ID has not been seen.
records.add(ID) Add the unseen ID to the set.
record.id = ID Update the ID, the .name and .description are the same. If you don't do that, then you get output like >1_duplicateName_1 1_duplicateName. That's not really a problem, but it's excessive.
SeqIO.write(record, of, "fasta") Write the record to the output file in fasta format.

A benefit of biopython is the easy ability to change formats, so instead of fasta one could have substituted Genbank or another format if needed.


---

