<p>If both the ID lines and the sequences are duplicated, you can do this using the FastaToTbl and TblToFasta scripts <a href="https://bioinformatics.stackexchange.com/a/2651/298">I have posted before</a> and just normal Linux utilities:</p>

<pre><code>FastaToTbl file1.fa file2.fa ... file6.fa | 
    sort | uniq -dc | awk '$1==6' | sed 's/^  *6 //' | TblToFasta &gt; dupes.fa
</code></pre>

<p>To understand this, let's look at the output of each step. The <code>FastaToTbl</code> script 
just converts fasta to a tab-delimited format where the ID is the first field and the sequence is the second, all on one line:</p>

<pre><code>$ FastaToTbl file*fa | head -n1
Haladaptatus sp. R4 MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAESPGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTDAERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGRGTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTESALEEVADR
</code></pre>

<p>Then, we pass this through <code>sort</code> and <code>uniq -dc</code> which prints out the number of times a line was repeated (only for duplicated lines):</p>

<pre><code>$ FastaToTbl file*fa | sort | uniq -dc
      4 some other sequence ALASOOPAKASKAPPPQQCDDREETAYSSLDAS
      6 Haladaptatus sp. R4 MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAESPGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTDAERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGRGTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTESALEEVADR
</code></pre>

<p>Since we only want those that were present in all 6 files, we can use <code>awk</code> telling it to only print lines whose first field was 6:</p>

<pre><code><span class="math-container">$ FastaToTbl file*fa | sort | uniq -dc | awk '$</span>1==6'
      6 Haladaptatus sp. R4 MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAESPGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTDAERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGRGTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTESALEEVADR
</code></pre>

<p>But then we don't want the 6 as part of the final output, so we need to remove it:</p>

<pre><code><span class="math-container">$ FastaToTbl file*fa | sort | uniq -dc | awk '$</span>1==6' | sed 's/^  *6 //'
Haladaptatus sp. R4 MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAESPGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTDAERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGRGTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTESALEEVADR
</code></pre>

<p>Finally, convert to fasta again:</p>

<pre><code><span class="math-container">$ FastaToTbl file*fa | sort | uniq -dc | awk '$</span>1==6' | sed 's/^  *6 //' | TblToFasta 
&gt;Haladaptatus sp. R4 
MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAES
PGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTD
AERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGR
GTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTES
ALEEVADR
</code></pre>

<hr>

<p>Alternatively, if your files are small enough to be loaded into memory, you could do this instead:</p>

<pre><code>FastaToTbl file*fa | 
    awk '{a[$0]++}END{for(s in a){if(a[s]==6){print s}}}' | 
        TblToFasta &gt; dupes.fa
</code></pre>

<p>Or you could just do the whole thing in a single awk operation:</p>

<pre><code>awk '{ 
        if(/^&gt;/){
            a[seq]++; 
            seq=<span class="math-container">$0; 
        }
        else{
            seq=seq"\n"$</span>0
        }
      }
      END{
        a[seq]++;
        for(s in a){
            if(a[s]==6){
                print s
            }
        }
       }' file*.fa 
</code></pre>

CC BY-SA 4.0
2019-08-19T12:35:49.787Z
2019-08-19T12:44:47.937Z
2019-08-19T12:44:47.937Z
terdon
<p>For pandas you simply load the fasta names and fasta sequences into separate arrays. The "fasta names" array becomes the index in a pandas Series, whilst the sequence are the values. The dataframe will correctly re-associated the names with each sequence. You load all the files into a single dataframe and use the duplicated() function then write out. </p>

<p>The easiest way to do this is concatenate all your files into one via shell ...</p>

<pre><code>cat file*.fasta &gt; concat_file.fa
</code></pre>

<p>Python script (biopython + pandas) ...</p>

<pre><code>from Bio import SeqIO
import pandas as pd

path = "/Users/username/location/concat_file.fa"
outpath = "/Users/username/Desktop/duplicates.fasta"
fasta_name = []
fasta_seq = []

for record in SeqIO.parse(path, "fasta"):
    fasta_name.append('&gt;' + record.id)
    fasta_seq.append(record.seq)
df = pd.Series(fasta_seq, index = fasta_name)
df_dups = df.index.duplicated()
mydups = df[df_dups]
mydups.to_csv(outpath, sep="\n", index=True, header=False)
</code></pre>

<p>... </p>

<p>I checked it on a ~4000 sequence fasta file of unique sequence ids (dengue), duplicated the last sequence id + sequence and the only sequence output was the very last sequence. </p>

CC BY-SA 4.0
2019-08-19T13:29:28.540Z
2019-08-20T00:08:57.383Z
2019-08-20T00:08:57.383Z
M__
<p>I have <strong>6 multi-fasta files</strong>, every of them contains ca 1500 sequences like that: </p>

<pre><code>&gt;Haladaptatus sp. R4
MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAES
PGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTD
AERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGR
GTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTES
ALEEVADR
&gt;Methanosphaera sp. WGK6
MAKVNVYSLKGDITEEIELPEIFEEEYRPDVIKRAVISTQTARIQPWGANPMAGKRTTAE
SFGSGRGAAMVPRVKGSSKAAFVPQAIGGRKAHPPRVNTIYHEKINKKERILAIRSAIAA
TANKEIVEQRGHAVANLEQVPFVVDDELETIKTTKETREIFKDLGIMDDILRAKKGRKIK
SGKGKLRGRKYRTPKGPLVVVGNDRGISLGARNHAGVEVVEVNNINAELLAPGTHAGRLT
IYTKSAVEKLADLFQQNRS
</code></pre>

<p>and so on. 
I need to create a new result file containing sequences that are common in these multi-fasta files. 
Example:
If sequence</p>

<pre><code>&gt;Haladaptatus sp. R4
    MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAES
    PGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTD
    AERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGR
    GTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTES
    ALEEVADR
</code></pre>

<p>occures in 1 <strong>and</strong> 2 <strong>and</strong> 3 <strong>and</strong> 4 <strong>and</strong> 5 <strong>and</strong> 6 file it shoud be written to the result file. 
May anoyone help?</p>

<p>Edited: this is my Python script which should work on 2 files (not 6 for now to make it simplier):</p>

<pre><code>for z in f1:
if z[0] == '&gt;': 
    znaleziony = False
    gatunek1 = z[1:len(z)]
    #print(gatunek1)
    for x in g1:
        #print(x)
        if x[0] == '&gt;':

            gatunek2 = x[1:len(x)]
            if gatunek1 == gatunek2:
                wynik.write('&gt;')                    
                wynik.write(gatunek1)
                znaleziony = True
                continue



elif znaleziony == True:
    wynik.write(z)
</code></pre>

<p>wynik.close()</p>

<p>Unfortunately it repeats some identifiers of sequence and does compare only identifiers (not whole sequence).</p>

CC BY-SA 4.0
2019-08-17T15:58:34.157Z
Are you looking for duplicates?
M__
CC BY-SA 4.0
2019-08-17T16:31:33.703Z
Welcome to the site. What approach have you tried to do this? In which step are you unsure how to continue? Could you post your code so that we can advice how to continue?
llrs
CC BY-SA 4.0
2019-08-17T17:36:51.450Z
@MichaelG. I woudn`t say so because there should`n be any duplicates among one separate file
Patrycja
CC BY-SA 4.0
2019-08-17T17:44:27.603Z
@Ilrs I tried to run a script in Python. I`ve begun with finding common sequences between two files (not six for now) to make it simplier. It finds common sequences but also writes only names of several sequences many times in the file like this: >Natrialbaceae
>Natrialbaceae
>Natrialbaceae
MEATVRDLDGSDAGSTELPAVFETTYRPDLIARAVRVAQANRKQDYGADEFAGMRTPAES
FGSGRGMAHVPRQEGRGRRVPQTIKGRKAHPPKAEKDQSESINTKEKKLAVRSAIAATTD
AELVADRGHQFDDDAEIPVVVSDEFEDLVKTKEVVEFLEAAGLEADVERADEGRSIRSGR
GKTRGRKYKQPKSILFVTSSESGPSRAARNLAGADVTTAAEVNAEDLAPGAQAGRLTVWT
ESALEEVADR
Patrycja
CC BY-SA 4.0
2019-08-17T17:57:40.740Z
`for z in f1:
 
 if z[0] == '>': 
  znaleziony = False
  gatunek1 = z[1:len(z)]
  #print(gatunek1)
  for x in g1:
   #print(x)
   if x[0] == '>':
    
    gatunek2 = x[1:len(x)]
    if gatunek1 == gatunek2:
     wynik.write('>')     
     wynik.write(gatunek1)
     znaleziony = True
     continue
     


 elif znaleziony == True:
  wynik.write(z)
  
wynik.close()`
Patrycja
CC BY-SA 4.0
2019-08-17T19:26:56.213Z
Please edit your post and add the code there - comments don't allow for good code formatting and that makes your code difficult to understand.
Ram RS
CC BY-SA 4.0
2019-08-18T17:05:36.793Z
Based on your question it is not entirely clear if you are looking for common header + sequence or also sequences that are identical too each other but have been mislabeled. "Unfortunately it repeats some identifiers of sequence and does compare only identifiers (not whole sequence)." I would assume that identical identifiers have the same sequence. Can you elaborate on what exactly you are looking to extract?
Mack123456
CC BY-SA 4.0
2019-08-18T18:55:39.317Z
Hello @Patrycja, your code is not well indented. Please edit the post and indent the code properly as Python blocks are identified by indentation and bad indentation makes the code impossible to read.
Ram RS
CC BY-SA 4.0
2019-08-19T11:41:01.293Z
Hi @Patryvia if I get.a.minute I'll do it in python pandas. The duplicated function will make short work of the problem
M__
<python><fasta><ngs><phylogenetics><phylogeny>
How to find common sequences among 6 multi-fasta files
# How to find common sequences among 6 multi-fasta files

(Asked by: username_0 on 2019-08-17)

I have 6 multi-fasta files, every of them contains ca 1500 sequences like that: 
>Haladaptatus sp. R4
MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAES
PGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTD
AERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGR
GTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTES
ALEEVADR
>Methanosphaera sp. WGK6
MAKVNVYSLKGDITEEIELPEIFEEEYRPDVIKRAVISTQTARIQPWGANPMAGKRTTAE
SFGSGRGAAMVPRVKGSSKAAFVPQAIGGRKAHPPRVNTIYHEKINKKERILAIRSAIAA
TANKEIVEQRGHAVANLEQVPFVVDDELETIKTTKETREIFKDLGIMDDILRAKKGRKIK
SGKGKLRGRKYRTPKGPLVVVGNDRGISLGARNHAGVEVVEVNNINAELLAPGTHAGRLT
IYTKSAVEKLADLFQQNRS

and so on. 
I need to create a new result file containing sequences that are common in these multi-fasta files. 
Example:
If sequence
>Haladaptatus sp. R4
    MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAES
    PGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTD
    AERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGR
    GTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTES
    ALEEVADR

occures in 1 and 2 and 3 and 4 and 5 and 6 file it shoud be written to the result file. 
May anoyone help?
Edited: this is my Python script which should work on 2 files (not 6 for now to make it simplier):
for z in f1:
if z[0] == '>': 
    znaleziony = False
    gatunek1 = z[1:len(z)]
    #print(gatunek1)
    for x in g1:
        #print(x)
        if x[0] == '>':

            gatunek2 = x[1:len(x)]
            if gatunek1 == gatunek2:
                wynik.write('>')                    
                wynik.write(gatunek1)
                znaleziony = True
                continue



elif znaleziony == True:
    wynik.write(z)

wynik.close()
Unfortunately it repeats some identifiers of sequence and does compare only identifiers (not whole sequence).

**Tags:** python, fasta, ngs, phylogenetics, phylogeny
> **Comments:**
> **username_1**: Are you looking for duplicates?  
> **username_2**: Welcome to the site. What approach have you tried to do this? In which step are you unsure how to continue? Could you post your code so that we can advice how to continue?  
> **username_0**: @MichaelG. I woudn`t say so because there should`n be any duplicates among one separate file  
> **username_0**: @Ilrs I tried to run a script in Python. I`ve begun with finding common sequences between two files (not six for now) to make it simplier. It finds common sequences but also writes only names of several sequences many times in the file like this: >Natrialbaceae >Natrialbaceae >Natrialbaceae MEATVRDLDGSDAGSTELPAVFETTYRPDLIARAVRVAQANRKQDYGADEFAGMRTPAES FGSGRGMAHVPRQEGRGRRVPQTIKGRKAHPPKAEKDQSESINTKEKKLAVRSAIAATTD AELVADRGHQFDDDAEIPVVVSDEFEDLVKTKEVVEFLEAAGLEADVERADEGRSIRSGR GKTRGRKYKQPKSILFVTSSESGPSRAARNLAGADVTTAAEVNAEDLAPGAQAGRLTVWT ESALEEVADR  
> **username_0**: `for z in f1:    if z[0] == '>':    znaleziony = False   gatunek1 = z[1:len(z)]   #print(gatunek1)   for x in g1:    #print(x)    if x[0] == '>':          gatunek2 = x[1:len(x)]     if gatunek1 == gatunek2:      wynik.write('>')           wynik.write(gatunek1)      znaleziony = True      continue          elif znaleziony == True:   wynik.write(z)    wynik.close()`  
> **username_3**: Please edit your post and add the code there - comments don't allow for good code formatting and that makes your code difficult to understand.  
> **username_4**: Based on your question it is not entirely clear if you are looking for common header + sequence or also sequences that are identical too each other but have been mislabeled. "Unfortunately it repeats some identifiers of sequence and does compare only identifiers (not whole sequence)." I would assume that identical identifiers have the same sequence. Can you elaborate on what exactly you are looking to extract?  
> **username_3**: Hello @Patrycja, your code is not well indented. Please edit the post and indent the code properly as Python blocks are identified by indentation and bad indentation makes the code impossible to read.  
> **username_1**: Hi @Patryvia if I get.a.minute I'll do it in python pandas. The duplicated function will make short work of the problem  

---

## 2 answers

### Answer by username_5 on 2019-08-19. Score: 3

If both the ID lines and the sequences are duplicated, you can do this using the FastaToTbl and TblToFasta scripts I have posted before and just normal Linux utilities:
FastaToTbl file1.fa file2.fa ... file6.fa | 
    sort | uniq -dc | awk '$1==6' | sed 's/^  *6 //' | TblToFasta > dupes.fa

To understand this, let's look at the output of each step. The FastaToTbl script 
just converts fasta to a tab-delimited format where the ID is the first field and the sequence is the second, all on one line:
$ FastaToTbl file*fa | head -n1
Haladaptatus sp. R4 MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAESPGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTDAERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGRGTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTESALEEVADR

Then, we pass this through sort and uniq -dc which prints out the number of times a line was repeated (only for duplicated lines):
$ FastaToTbl file*fa | sort | uniq -dc
      4 some other sequence ALASOOPAKASKAPPPQQCDDREETAYSSLDAS
      6 Haladaptatus sp. R4 MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAESPGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTDAERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGRGTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTESALEEVADR

Since we only want those that were present in all 6 files, we can use awk telling it to only print lines whose first field was 6:
$ FastaToTbl file*fa | sort | uniq -dc | awk '$1==6'
      6 Haladaptatus sp. R4 MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAESPGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTDAERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGRGTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTESALEEVADR

But then we don't want the 6 as part of the final output, so we need to remove it:
$ FastaToTbl file*fa | sort | uniq -dc | awk '$1==6' | sed 's/^  *6 //'
Haladaptatus sp. R4 MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAESPGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTDAERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGRGTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTESALEEVADR

Finally, convert to fasta again:
$ FastaToTbl file*fa | sort | uniq -dc | awk '$1==6' | sed 's/^  *6 //' | TblToFasta 
>Haladaptatus sp. R4 
MQATVRDLNGEDADTVDLPDVFETTVRTDLIKRAVLAAQANRKQDYGTDPHAGMRTSAES
PGSGRGMAHVPQTNGRGARVPFTVGGRVAHPPKAEKDRSRSINKKERKLAVRSAIAATTD
AERVSERGHRFDEDTELPLVVSDDFEDLVKTQEVVSFLEAVGIDADIARAEDNKKVRAGR
GTTRGRKYKTPKSILFVTSEEPSRAARNLAGADVATAREVNTEDLAPGTQAGRLTVWTES
ALEEVADR


Alternatively, if your files are small enough to be loaded into memory, you could do this instead:
FastaToTbl file*fa | 
    awk '{a[$0]++}END{for(s in a){if(a[s]==6){print s}}}' | 
        TblToFasta > dupes.fa

Or you could just do the whole thing in a single awk operation:
awk '{ 
        if(/^>/){
            a[seq]++; 
            seq=$0; 
        }
        else{
            seq=seq"\n"$0
        }
      }
      END{
        a[seq]++;
        for(s in a){
            if(a[s]==6){
                print s
            }
        }
       }' file*.fa


---

### Answer by username_1 on 2019-08-19. Score: 0

For pandas you simply load the fasta names and fasta sequences into separate arrays. The "fasta names" array becomes the index in a pandas Series, whilst the sequence are the values. The dataframe will correctly re-associated the names with each sequence. You load all the files into a single dataframe and use the duplicated() function then write out. 
The easiest way to do this is concatenate all your files into one via shell ...
cat file*.fasta > concat_file.fa

Python script (biopython + pandas) ...
from Bio import SeqIO
import pandas as pd

path = "/Users/username/location/concat_file.fa"
outpath = "/Users/username/Desktop/duplicates.fasta"
fasta_name = []
fasta_seq = []

for record in SeqIO.parse(path, "fasta"):
    fasta_name.append('>' + record.id)
    fasta_seq.append(record.seq)
df = pd.Series(fasta_seq, index = fasta_name)
df_dups = df.index.duplicated()
mydups = df[df_dups]
mydups.to_csv(outpath, sep="\n", index=True, header=False)

... 
I checked it on a ~4000 sequence fasta file of unique sequence ids (dengue), duplicated the last sequence id + sequence and the only sequence output was the very last sequence.


---

