<p>Using the TblToFasta and FastaToTbl scripts <a href="https://bioinformatics.stackexchange.com/a/2651/298">I have posted before</a>, combined with classic *nix utils, you can do:</p>

<pre><code>$ join -a1 -a2 &lt;(FastaToTbl dna1.fa | sort) &lt;(FastaToTbl dna2.fa | sort) | 
    sed 's/ /=/2' | TblToFasta | tr = '\n'
&gt;chr1 
ACTGACTAGCTAGCTAACTG
GCATCGTAGCTAGCTACGAT
&gt;chr2 
GCATCGTAGCTAGCTACGAT
&gt;chr3 
CATCGATCGTACGTACGTAG
CATCGATCGTACGTACGTAG
&gt;chr4 
ATCGATCGATCGTACGATCG
ATCGATCGATCGTACGATCG
&gt;chr5 
CATCGATCGTACGTACGTAG
</code></pre>

<h3>Explanation</h3>

<p>The <code>FastaToTbl</code> will put everything on the same line, with the sequence's ID at the beginning of the line, then a tab and then the sequence (no matter how many lines of sequence you have). So, we do this to both files, sort them and pass that as input to <code>join</code>, which will join its input on lines where the 1st field is the same (by default). The <code>-a1 -a2</code> means "also print non-matching lines from both files" so we don't miss cases where a sequence is in only one of the two files. </p>

<p>Since the <code>join</code> will add a space between the two joined lines, we deal with that by replacing the second space on each line with a <code>=</code> (<code>sed 's/ /=/2'</code>) and then pass the whole thing to <code>TblToFasta</code> to revert to fasta format. Finally, we use <code>tr</code> to replace the <code>=</code> with a newline so you can see where each sequence came from as you show in your example.</p>

<p>For longer sequences where you want to avoid sorting the input, you can do:</p>

<pre><code>$ FastaToTbl dna1.fa dna2.fa | 
    awk '{
            seq[$1] ? seq[$1]=seq[$1]"\n"$2 : seq[$1]=$2 
         }
         END{
              for(chr in seq){
                printf "&gt;%s\n%s\n", chr, seq[chr]
              }
         }'
&gt;chr1
ACTGACTAGCTAGCTAACTG
GCATCGTAGCTAGCTACGAT
&gt;chr2
GCATCGTAGCTAGCTACGAT
&gt;chr3
CATCGATCGTACGTACGTAG
CATCGATCGTACGTACGTAG
&gt;chr4
ATCGATCGATCGTACGATCG
ATCGATCGATCGTACGATCG
&gt;chr5
CATCGATCGTACGTACGTAG
</code></pre>

<h3>Explanation</h3>

<p>Here, the tbl format sequences (see above) are passed to an awk script which does the work:</p>

<ul>
<li><code>seq[$1] ? seq[$1]=seq[$1]"\n"$2 : seq[$1]=$2</code> : if the 1st field (<code>$1</code>, the sequence identifier) is already in the array <code>seq</code>, then set the value of <code>seq[$1]</code> to the current value (so whatever sequence has been already seen for this ID), a newline and the 2nd field (<code>$2</code>, the sequence). If it hasn't been seen before, then set the value of <code>seq[$1]</code> to the 2nd field, the sequence. </li>
<li><code>END{ for(chr in seq){ printf "&gt;%s\n%s\n", chr, seq[chr]}</code> : after processing everything, go through each element in the array <code>seq</code> and print out the data in fasta format.</li>
</ul>

<p>However, that will mean keeping both files in memory which may be a problem for huge sequences. You can avoid that by using temp files:</p>

<pre><code>$ FastaToTbl dna1.fa dna2.fa | 
    awk '{ 
            if(!seen[$1]++){
                print "&gt;"$1 &gt; $1".tmp"
            } 
            print $2 &gt;&gt; $1".tmp"
        }' &amp;&amp; cat *.tmp &gt; out.fa &amp;&amp; rm *.tmp
$ cat out.fa 
&gt;chr1
ACTGACTAGCTAGCTAACTG
GCATCGTAGCTAGCTACGAT
&gt;chr2
GCATCGTAGCTAGCTACGAT
&gt;chr3
CATCGATCGTACGTACGTAG
CATCGATCGTACGTACGTAG
&gt;chr4
ATCGATCGATCGTACGATCG
ATCGATCGATCGTACGATCG
&gt;chr5
CATCGATCGTACGTACGTAG
</code></pre>

<h3>Explanation</h3>

<p>This is the same basic idea as the script above, only instead of storing the information in an array, we print it as it comes into temp files. </p>

<ul>
<li><code>if(!seen[$1]++){print "&gt;"$1 &gt; $1".tmp"}</code> : if this is the first time we've seen this ID, print it as a fasta header into the file <code>$1".tmp"</code> (e.g. <code>chr1.tmp</code>).</li>
<li><code>print $2 &gt;&gt; $1".tmp"</code> append the current sequence to the corresponding temp file.</li>
</ul>

<p>Once the temp files have been created, we concatenate them (<code>cat *.tmp &gt; out.fa</code>) and delete them (<code>rm *tmp</code>).</p>

<hr>

<p>As for why you would want to do this, an obvious example is reconstructing a reference genome from short sequences. You would want all the sequences of the same schromosome together. Of course, you would also need to assemble them in the right order, which you aren't doing here.</p>

<p>Alternatively, you might want to concatenate them this way so as to avoid having the extra, unneeded lines (<code>&gt;chrN</code>) which would make your file larger. Or you would want to be able to retrieve all sequences with the same ID at once. </p>

<p>But I don't really know why you would want to do it. You're the one asking for it, after all! :)</p>

CC BY-SA 4.0
2018-06-27T14:10:52.973Z
Can you answer their question 2, the why?
Chris_Rands
CC BY-SA 4.0
2018-06-27T11:08:11.157Z
2018-07-02T07:42:38.807Z
2018-07-02T07:42:38.807Z
terdon
<p>I have the following two small FASTAs, <code>dna1.fa</code> and <code>dna2.fa</code>:</p>

<pre><code># dna1.fa
&gt;chr1
ACTGACTAGCTAGCTAACTG
&gt;chr2
GCATCGTAGCTAGCTACGAT
&gt;chr3
CATCGATCGTACGTACGTAG
&gt;chr4
ATCGATCGATCGTACGATCG

# dna2.fa
&gt;chr1
GCATCGTAGCTAGCTACGAT
&gt;chr3
CATCGATCGTACGTACGTAG
&gt;chr4
ATCGATCGATCGTACGATCG
&gt;chr5
CATCGATCGTACGTACGTAG
</code></pre>

<p>If I concatenate these two files using <code>cat</code>, the piped result is as follows:</p>

<pre><code>$ cat dna1.fa dna2.fa &gt; total1.fa
$ cat total1.fa
&gt;chr1
ACTGACTAGCTAGCTAACTG
&gt;chr2
GCATCGTAGCTAGCTACGAT
&gt;chr3
CATCGATCGTACGTACGTAG
&gt;chr4
ATCGATCGATCGTACGATCG
&gt;chr1
GCATCGTAGCTAGCTACGAT
&gt;chr3
CATCGATCGTACGTACGTAG
&gt;chr4
ATCGATCGATCGTACGATCG
&gt;chr5
CATCGATCGTACGTACGTAG
</code></pre>

<p>Here, as expected, these are two files which have concatenated together. It seems this is usually what is done when one concatenates several FASTAs together in bioinformatics.</p>

<p>Question1: Is it possible to concatenate these "based on the chromosome"? Here is the desired result:</p>

<pre><code>&gt;chr1
ACTGACTAGCTAGCTAACTG
GCATCGTAGCTAGCTACGAT
&gt;chr2
GCATCGTAGCTAGCTACGAT
&gt;chr3
CATCGATCGTACGTACGTAG
CATCGATCGTACGTACGTAG
&gt;chr4
ATCGATCGATCGTACGATCG
ATCGATCGATCGTACGATCG
&gt;chr5
CATCGATCGTACGTACGTAG
</code></pre>

<p>Note that these are exceptionally small FASTAs---usually one could be dealing with files of several GB in size. </p>

<p>Question2: The meta question here is, are there any standard bioinformatic analyses whereby the "concatenated by chromosome" FASTA would be more useful than the straightforward concatenated FASTA via <code>cat</code> alone? </p>

CC BY-SA 4.0
2018-06-25T14:28:20.473Z
Why would you do this? You're creating contiguous sequences out of (presumably) non-contiguous sequence?
Chris_Rands
CC BY-SA 4.0
2018-06-25T14:28:55.290Z
Also you should not assume FASTA format is one sequence per line
Chris_Rands
CC BY-SA 4.0
2018-06-25T14:29:46.563Z
What's the actual context here? Are these sequence alignments? Aligned FASTA would not follow the format of your desired output
Chris_Rands
<fasta>
How to concatenate FASTAs with respect to chromosome?
# How to concatenate FASTAs with respect to chromosome?

(Asked by: username_0 on 2018-06-25)

I have the following two small FASTAs, dna1.fa and dna2.fa:
# dna1.fa
>chr1
ACTGACTAGCTAGCTAACTG
>chr2
GCATCGTAGCTAGCTACGAT
>chr3
CATCGATCGTACGTACGTAG
>chr4
ATCGATCGATCGTACGATCG

# dna2.fa
>chr1
GCATCGTAGCTAGCTACGAT
>chr3
CATCGATCGTACGTACGTAG
>chr4
ATCGATCGATCGTACGATCG
>chr5
CATCGATCGTACGTACGTAG

If I concatenate these two files using cat, the piped result is as follows:
$ cat dna1.fa dna2.fa > total1.fa
$ cat total1.fa
>chr1
ACTGACTAGCTAGCTAACTG
>chr2
GCATCGTAGCTAGCTACGAT
>chr3
CATCGATCGTACGTACGTAG
>chr4
ATCGATCGATCGTACGATCG
>chr1
GCATCGTAGCTAGCTACGAT
>chr3
CATCGATCGTACGTACGTAG
>chr4
ATCGATCGATCGTACGATCG
>chr5
CATCGATCGTACGTACGTAG

Here, as expected, these are two files which have concatenated together. It seems this is usually what is done when one concatenates several FASTAs together in bioinformatics.
Question1: Is it possible to concatenate these "based on the chromosome"? Here is the desired result:
>chr1
ACTGACTAGCTAGCTAACTG
GCATCGTAGCTAGCTACGAT
>chr2
GCATCGTAGCTAGCTACGAT
>chr3
CATCGATCGTACGTACGTAG
CATCGATCGTACGTACGTAG
>chr4
ATCGATCGATCGTACGATCG
ATCGATCGATCGTACGATCG
>chr5
CATCGATCGTACGTACGTAG

Note that these are exceptionally small FASTAs---usually one could be dealing with files of several GB in size. 
Question2: The meta question here is, are there any standard bioinformatic analyses whereby the "concatenated by chromosome" FASTA would be more useful than the straightforward concatenated FASTA via cat alone?

**Tags:** fasta
> **Comments:**
> **username_1**: Why would you do this? You're creating contiguous sequences out of (presumably) non-contiguous sequence?  
> **username_1**: Also you should not assume FASTA format is one sequence per line  
> **username_1**: What's the actual context here? Are these sequence alignments? Aligned FASTA would not follow the format of your desired output  

---

## 1 answers

### Answer by username_2 on 2018-06-27. Score: 2 | Accepted answer

Using the TblToFasta and FastaToTbl scripts I have posted before, combined with classic *nix utils, you can do:
$ join -a1 -a2 <(FastaToTbl dna1.fa | sort) <(FastaToTbl dna2.fa | sort) | 
    sed 's/ /=/2' | TblToFasta | tr = '\n'
>chr1 
ACTGACTAGCTAGCTAACTG
GCATCGTAGCTAGCTACGAT
>chr2 
GCATCGTAGCTAGCTACGAT
>chr3 
CATCGATCGTACGTACGTAG
CATCGATCGTACGTACGTAG
>chr4 
ATCGATCGATCGTACGATCG
ATCGATCGATCGTACGATCG
>chr5 
CATCGATCGTACGTACGTAG

Explanation
The FastaToTbl will put everything on the same line, with the sequence's ID at the beginning of the line, then a tab and then the sequence (no matter how many lines of sequence you have). So, we do this to both files, sort them and pass that as input to join, which will join its input on lines where the 1st field is the same (by default). The -a1 -a2 means "also print non-matching lines from both files" so we don't miss cases where a sequence is in only one of the two files. 
Since the join will add a space between the two joined lines, we deal with that by replacing the second space on each line with a = (sed 's/ /=/2') and then pass the whole thing to TblToFasta to revert to fasta format. Finally, we use tr to replace the = with a newline so you can see where each sequence came from as you show in your example.
For longer sequences where you want to avoid sorting the input, you can do:
$ FastaToTbl dna1.fa dna2.fa | 
    awk '{
            seq[$1] ? seq[$1]=seq[$1]"\n"$2 : seq[$1]=$2 
         }
         END{
              for(chr in seq){
                printf ">%s\n%s\n", chr, seq[chr]
              }
         }'
>chr1
ACTGACTAGCTAGCTAACTG
GCATCGTAGCTAGCTACGAT
>chr2
GCATCGTAGCTAGCTACGAT
>chr3
CATCGATCGTACGTACGTAG
CATCGATCGTACGTACGTAG
>chr4
ATCGATCGATCGTACGATCG
ATCGATCGATCGTACGATCG
>chr5
CATCGATCGTACGTACGTAG

Explanation
Here, the tbl format sequences (see above) are passed to an awk script which does the work:

seq[$1] ? seq[$1]=seq[$1]"\n"$2 : seq[$1]=$2 : if the 1st field ($1, the sequence identifier) is already in the array seq, then set the value of seq[$1] to the current value (so whatever sequence has been already seen for this ID), a newline and the 2nd field ($2, the sequence). If it hasn't been seen before, then set the value of seq[$1] to the 2nd field, the sequence. 
END{ for(chr in seq){ printf ">%s\n%s\n", chr, seq[chr]} : after processing everything, go through each element in the array seq and print out the data in fasta format.

However, that will mean keeping both files in memory which may be a problem for huge sequences. You can avoid that by using temp files:
$ FastaToTbl dna1.fa dna2.fa | 
    awk '{ 
            if(!seen[$1]++){
                print ">"$1 > $1".tmp"
            } 
            print $2 >> $1".tmp"
        }' && cat *.tmp > out.fa && rm *.tmp
$ cat out.fa 
>chr1
ACTGACTAGCTAGCTAACTG
GCATCGTAGCTAGCTACGAT
>chr2
GCATCGTAGCTAGCTACGAT
>chr3
CATCGATCGTACGTACGTAG
CATCGATCGTACGTACGTAG
>chr4
ATCGATCGATCGTACGATCG
ATCGATCGATCGTACGATCG
>chr5
CATCGATCGTACGTACGTAG

Explanation
This is the same basic idea as the script above, only instead of storing the information in an array, we print it as it comes into temp files. 

if(!seen[$1]++){print ">"$1 > $1".tmp"} : if this is the first time we've seen this ID, print it as a fasta header into the file $1".tmp" (e.g. chr1.tmp).
print $2 >> $1".tmp" append the current sequence to the corresponding temp file.

Once the temp files have been created, we concatenate them (cat *.tmp > out.fa) and delete them (rm *tmp).

As for why you would want to do this, an obvious example is reconstructing a reference genome from short sequences. You would want all the sequences of the same schromosome together. Of course, you would also need to assemble them in the right order, which you aren't doing here.
Alternatively, you might want to concatenate them this way so as to avoid having the extra, unneeded lines (>chrN) which would make your file larger. Or you would want to be able to retrieve all sequences with the same ID at once. 
But I don't really know why you would want to do it. You're the one asking for it, after all! :)

> **Comments:**
> **username_1**: Can you answer their question 2, the why?  

---

