<p>I had a similar problem and found a solution using awk. Since this is the first time I use awk, the syntax might not be perfect, but you should get an idea of what is possible.</p>

<pre><code>awk -F' ' '{if(/^&gt;/){sub("strain ", ""); name=$2" "$3" "$4; print &gt; name".fa"}else{print &gt; name".fa"}}' yourfile.fa
</code></pre>

<p><em>(more information below)</em></p>

<p>This should transform a file like this:</p>

<p>yourfile.fa</p>

<pre><code>&gt;NZ_AESB01000237.1 Clostridium botulinum BKT028387 contig00237 ATGC
ATGCATGC
ATGCATGC
&gt;NZ_AESB01000236.1 Clostridium botulinum BKT028387 contig00236 ATGC
ATGCATGC
&gt;NZ_ALNK01000035.1 Peptoanaerobacter stomatis strain OBRC8 ctg120007290634 ATGC
ATGCATGC
ATGCATGC
&gt;NZ_ALNK01000031.1 Peptoanaerobacter stomatis strain OBRC8 ctg120007290635 ATGC
ATGCATGC
</code></pre>

<p>Into two files like this:</p>

<p>'Clostridium botulinum BKT028387.fa'</p>

<pre><code>&gt;NZ_AESB01000237.1 Clostridium botulinum BKT028387 contig00237 ATGC
ATGCATGC
ATGCATGC
&gt;NZ_AESB01000236.1 Clostridium botulinum BKT028387 contig00236 ATGC
ATGCATGC
</code></pre>

<p>'Peptoanaerobacter stomatis OBRC8.fa'</p>

<pre><code>&gt;NZ_ALNK01000035.1 Peptoanaerobacter stomatis OBRC8 ctg120007290634 ATGC
ATGCATGC
ATGCATGC
&gt;NZ_ALNK01000031.1 Peptoanaerobacter stomatis OBRC8 ctg120007290635 ATGC
ATGCATGC
</code></pre>

<p><strong>How it works</strong></p>

<ul>
<li><code>-F' '</code> tells awk to split the line by space (accessible with <code>$1</code>, <code>$2</code>, ...)</li>
<li><code>if(/^&gt;/)</code> looks for line starting with the <code>&gt;</code> character, then <code>sub("strain ", "")</code> remove the word <code>strain</code> and <code>name=$2" "$3" "$4; print &gt; name".fa"</code> print the line into a new file with a name based on what you asked for.</li>
<li>else if the line does not start with <code>&gt;</code>, <code>{print &gt; name".fa"}</code> appends it to a file using the current value of <code>name</code>.</li>
</ul>

<p>This solution will work regardless of the number of lines between the fasta headers.
Note that I won't recommend using spaces in filenames, so I would replace <code>name=$2" "$3" "$4</code> with something like <code>name=$2"_"$3"_"$4</code>.
I used this solution to split a large fasta file (>5 GB) containing >5k genomes with >25k scaffolds and I found it fast and rather memory efficient.</p>

CC BY-SA 4.0
2021-09-03T17:45:57.547Z
Thank you so much for the code and explanation. However, when I tried it, it gave me the following error. Any ideas? 
`awk: syntax error at source line 1
 context is
 {if(/^>/){ name=$2" "$3" "$4; print > >>>  name".fa" <<< 
awk: illegal statement at source line 1
awk: syntax error at source line 1`
user1995
CC BY-SA 4.0
2021-09-04T19:15:05.987Z
I'm not sure about your error but you can probably put the code on several lines for easier debugging, and execute it with the awk -f option like this: `awk -f myscript.awk mygenomes.fasta`.
Nils Giordano
CC BY-SA 3.0
2018-03-29T15:50:22.970Z
2018-03-29T15:50:22.970Z
Nils Giordano
<p>The <a href="https://github.com/biologyguy/BuddySuite/wiki/SB-Pull-records" rel="nofollow noreferrer">SeqBuddy --pull_records</a> command can grab sequences based on a regular expression against record IDs and/or metadata. Given your example:</p>

<pre><code>$: seqbuddy.py sequences.fa --pull_records "Clostridium botulinum.*BKT028387" full &gt; Clostridium_botulinum_BKT028387.fas
$: seqbuddy.py sequences.fa --pull_records "Peptoanaerobacter stomatis.*OBRC8" full &gt; Peptoanaerobacter_stomatis_OBRC8.fas
</code></pre>

CC BY-SA 3.0
2017-12-15T11:54:27.037Z
This looks great. Is there a way to set this up to loop using an input list of names?
AudileF
CC BY-SA 3.0
2017-12-21T15:18:01.413Z
Yep, you just pass in the file name.
Steve Bond
CC BY-SA 3.0
2017-12-14T04:00:24.687Z
2017-12-14T04:00:24.687Z
Steve Bond
<p>I would convert the fasta file to a format with one line per record (I have <a href="https://bioinformatics.stackexchange.com/a/2651/298">already posted</a> the FastaToTbl and TblToFasta scripts I use for this before), and then look for a specific pattern:</p>

<pre><code>FastaToTbl file.fa | grep 'BKT028387' | TblToFasta &gt; clostridium.fa
FastaToTbl file.fa | grep 'OBRC8' | TblToFasta &gt; peptoanaerobacter.fa
</code></pre>

<p>I have downloaded a few sequences using the same command you described in your previous question, and it looks like the headers are not standardized. For example, while you have many that are in the format of <code>$species strain $strain</code>, like these:</p>

<pre><code>Clostridioides difficile strain 7496-D/ST9
Clostridioides difficile strain 5616-DH/ST42,
Peptoanaerobacter stomatis strain OBRC8
Romboutsia maritimum strain CCRI-22766
Tepidibacter mesophilus strain JCM
</code></pre>

<p>Others don't have the word <code>strain</code> and just give the strain:</p>

<pre><code>Peptostreptococcus anaerobius 653-L contig00088,
Peptoclostridium acidaminophilum DSM 3953
</code></pre>

<p>So, as a first step, you could deal with the ones that do have the word <code>strain</code> as the 4th word of the header line (the 1st is the seqID which I have not included in the examples above) and the actual strain as the 5th and get those out of the way:</p>

<pre><code>FastaToTbl foo.fa | 
    awk '($4=="strain"){gsub("/","_"); print $0 &gt;&gt; $2"_"$3"_"$5".tbl"}'
</code></pre>

<p>That will look for lines whose 4th field is <code>strain</code>, then it will replace any <code>/</code> in the line with <code>_</code> (there's a strain called <code>CF/ST37</code> and that cannot be in a filename) and print into a file whose name is the 2nd, 3rd and 5th fields of the line concatenated. On my test file, this produced the following files (as you can see, some are wrong since there is a comma after the strain, but that's because they're not standardized):</p>

<pre><code>$ ls *tbl
Clostridioides_difficile_5479-NonSp_ST229,.tbl  Clostridioides_difficile_6589-DH_ST42.tbl         Clostridioides_difficile_7457-NonSp_novelST,.tbl
Clostridioides_difficile_5479-NonSp_ST229.tbl   Clostridioides_difficile_6592-G_ST8,.tbl          Clostridioides_difficile_7457-NonSp_novelST.tbl
Clostridioides_difficile_5480-E_ST6,.tbl        Clostridioides_difficile_6592-G_ST8.tbl           Clostridioides_difficile_7458-N_ST10,.tbl
Clostridioides_difficile_5480-E_ST6.tbl         Clostridioides_difficile_6593-Y_ST110,.tbl        Clostridioides_difficile_7458-N_ST10.tbl
Clostridioides_difficile_5482-L_ST55,.tbl       Clostridioides_difficile_6593-Y_ST110.tbl         Clostridioides_difficile_7459-G_ST8,.tbl
Clostridioides_difficile_5482-L_ST55.tbl        Clostridioides_difficile_6595-AL_ST58,.tbl        Clostridioides_difficile_7459-G_ST8.tbl
Clostridioides_difficile_5485-Y_ST2,.tbl        Clostridioides_difficile_6595-AL_ST58.tbl         Clostridioides_difficile_7462-DH_ST42,.tbl
Clostridioides_difficile_5485-Y_ST2.tbl         Clostridioides_difficile_6596-D_ST9,.tbl          Clostridioides_difficile_7462-DH_ST42.tbl
Clostridioides_difficile_5488-DH_ST42,.tbl      Clostridioides_difficile_6596-D_ST9.tbl           Clostridioides_difficile_7463-E_ST6,.tbl
Clostridioides_difficile_5488-DH_ST42.tbl       Clostridioides_difficile_6599-NonSp_ST103,.tbl    Clostridioides_difficile_7463-E_ST6.tbl
Clostridioides_difficile_5489-M_ST15,.tbl       Clostridioides_difficile_6599-NonSp_ST103.tbl     Clostridioides_difficile_7464-Y_ST110,.tbl
Clostridioides_difficile_5489-M_ST15.tbl        Clostridioides_difficile_6600-G_ST8,.tbl          Clostridioides_difficile_7464-Y_ST110.tbl
Clostridioides_difficile_5490-CF_ST37,.tbl      Clostridioides_difficile_6600-G_ST8.tbl           Clostridioides_difficile_7466-DH_ST42,.tbl
Clostridioides_difficile_5490-CF_ST37.tbl       Clostridioides_difficile_6606-BK_ST11,.tbl        Clostridioides_difficile_7466-DH_ST42.tbl
Clostridioides_difficile_5493-DH_ST42,.tbl      Clostridioides_difficile_6606-BK_ST11.tbl         Clostridioides_difficile_7467-DH_ST42,.tbl
Clostridioides_difficile_5493-DH_ST42.tbl       Clostridioides_difficile_6607-DH_ST42,.tbl        Clostridioides_difficile_7467-DH_ST42.tbl
Clostridioides_difficile_5495-AF_ST41,.tbl      Clostridioides_difficile_6607-DH_ST42.tbl         Clostridioides_difficile_7468-NonSp_ST97,.tbl
Clostridioides_difficile_5495-AF_ST41.tbl       Clostridioides_difficile_6610-N_ST10,.tbl         Clostridioides_difficile_7468-NonSp_ST97.tbl
Clostridioides_difficile_5497-Y_ST2,.tbl        Clostridioides_difficile_6610-N_ST10.tbl          Clostridioides_difficile_7471-BI_ST1,.tbl
Clostridioides_difficile_5497-Y_ST2.tbl         Clostridioides_difficile_6612-DH_ST28,.tbl        Clostridioides_difficile_7471-BI_ST1.tbl
Clostridioides_difficile_5500-DH_ST42,.tbl      Clostridioides_difficile_6612-DH_ST28.tbl         Clostridioides_difficile_7472-Y_ST2,.tbl
Clostridioides_difficile_5500-DH_ST42.tbl       Clostridioides_difficile_6613-NonSp_ST3,.tbl      Clostridioides_difficile_7472-Y_ST2.tbl
Clostridioides_difficile_5504-L_ST55,.tbl       Clostridioides_difficile_6613-NonSp_ST3.tbl       Clostridioides_difficile_7473-DH_ST42,.tbl
Clostridioides_difficile_5504-L_ST55.tbl        Clostridioides_difficile_6614-AL_ST34,.tbl        Clostridioides_difficile_7473-DH_ST42.tbl
Clostridioides_difficile_5505-Y_ST2,.tbl        Clostridioides_difficile_6614-AL_ST34.tbl         Clostridioides_difficile_7476-Y_ST2,.tbl
Clostridioides_difficile_5505-Y_ST2.tbl         Clostridioides_difficile_6615-NonSp_ST11,.tbl     Clostridioides_difficile_7476-Y_ST2.tbl
Clostridioides_difficile_5537-D_ST9,.tbl        Clostridioides_difficile_6615-NonSp_ST11.tbl      Clostridioides_difficile_7478-NonSp_novelST,.tbl
Clostridioides_difficile_5537-D_ST9.tbl         Clostridioides_difficile_6616-NonSp_novelST,.tbl  Clostridioides_difficile_7478-NonSp_novelST.tbl
Clostridioides_difficile_5538-BK_ST11,.tbl      Clostridioides_difficile_6616-NonSp_novelST.tbl   Clostridioides_difficile_7479-NonSp_ST3,.tbl
Clostridioides_difficile_5538-BK_ST11.tbl       Clostridioides_difficile_6617-A_ST43,.tbl         Clostridioides_difficile_7479-NonSp_ST3.tbl
Clostridioides_difficile_5539-Y_ST2,.tbl        Clostridioides_difficile_6617-A_ST43.tbl          Clostridioides_difficile_7480-M_ST15,.tbl
Clostridioides_difficile_5539-Y_ST2.tbl         Clostridioides_difficile_6618-NonSp_ST53,.tbl     Clostridioides_difficile_7480-M_ST15.tbl
Clostridioides_difficile_5541-CF_ST37,.tbl      Clostridioides_difficile_6618-NonSp_ST53.tbl      Clostridioides_difficile_7481-NonSp_ST59,.tbl
Clostridioides_difficile_5541-CF_ST37.tbl       Clostridioides_difficile_6619-CF_ST37,.tbl        Clostridioides_difficile_7481-NonSp_ST59.tbl
Clostridioides_difficile_5542-Y_ST2,.tbl        Clostridioides_difficile_6619-CF_ST37.tbl         Clostridioides_difficile_7482-DH_ST42,.tbl
Clostridioides_difficile_5542-Y_ST2.tbl         Clostridioides_difficile_6621-NonSp_ST10,.tbl     Clostridioides_difficile_7482-DH_ST42.tbl
Clostridioides_difficile_5547-DH_ST42,.tbl      Clostridioides_difficile_6621-NonSp_ST10.tbl      Clostridioides_difficile_7486-A_ST43,.tbl
Clostridioides_difficile_5547-DH_ST42.tbl       Clostridioides_difficile_6622-CF_ST37,.tbl        Clostridioides_difficile_7486-A_ST43.tbl
Clostridioides_difficile_5548-Y_ST2,.tbl        Clostridioides_difficile_6622-CF_ST37.tbl         Clostridioides_difficile_7488-NonSp_ST103,.tbl
Clostridioides_difficile_5548-Y_ST2.tbl         Clostridioides_difficile_6623-NonSp_ST54,.tbl     Clostridioides_difficile_7488-NonSp_ST103.tbl
Clostridioides_difficile_5549-NonSp_ST21,.tbl   Clostridioides_difficile_6623-NonSp_ST54.tbl      Clostridioides_difficile_7491-DH_ST42,.tbl
Clostridioides_difficile_5549-NonSp_ST21.tbl    Clostridioides_difficile_6625-Y_ST2,.tbl          Clostridioides_difficile_7491-DH_ST42.tbl
Clostridioides_difficile_5550-AH_ST67,.tbl      Clostridioides_difficile_6625-Y_ST2.tbl           Clostridioides_difficile_7492-CF_ST37,.tbl
Clostridioides_difficile_5550-AH_ST67.tbl       Clostridioides_difficile_6626-Y_ST110,.tbl        Clostridioides_difficile_7492-CF_ST37.tbl
Clostridioides_difficile_5551-CF_ST37,.tbl      Clostridioides_difficile_6626-Y_ST110.tbl         Clostridioides_difficile_7494-E_ST6,.tbl
Clostridioides_difficile_5551-CF_ST37.tbl       Clostridioides_difficile_6628-AL_ST58,.tbl        Clostridioides_difficile_7494-E_ST6.tbl
Clostridioides_difficile_5552-DH_ST42,.tbl      Clostridioides_difficile_6628-AL_ST58.tbl         Clostridioides_difficile_7496-D_ST9,.tbl
Clostridioides_difficile_5552-DH_ST42.tbl       Clostridioides_difficile_6629-NonSp_novelST,.tbl  Clostridioides_difficile_7496-D_ST9.tbl
Clostridioides_difficile_5553-L_ST55,.tbl       Clostridioides_difficile_6629-NonSp_novelST.tbl   Clostridioides_difficile_7497-NonSp_novelST,.tbl
Clostridioides_difficile_5553-L_ST55.tbl        Clostridioides_difficile_6631-DH_ST42,.tbl        Clostridioides_difficile_7497-NonSp_novelST.tbl
Clostridioides_difficile_5555-DH_ST42,.tbl      Clostridioides_difficile_6631-DH_ST42.tbl         Clostridioides_difficile_7498-M_ST15,.tbl
Clostridioides_difficile_5555-DH_ST42.tbl       Clostridioides_difficile_6632-NonSp_ST3,.tbl      Clostridioides_difficile_7498-M_ST15.tbl
Clostridioides_difficile_5557-DH_ST42,.tbl      Clostridioides_difficile_6632-NonSp_ST3.tbl       Clostridioides_difficile_7499-CF_ST37,.tbl
Clostridioides_difficile_5557-DH_ST42.tbl       Clostridioides_difficile_6634-DH_ST42,.tbl        Clostridioides_difficile_7499-CF_ST37.tbl
Clostridioides_difficile_5558-AH_ST67,.tbl      Clostridioides_difficile_6634-DH_ST42.tbl         Clostridioides_difficile_7501-BI_ST1,.tbl
Clostridioides_difficile_5558-AH_ST67.tbl       Clostridioides_difficile_6635-BI_ST1,.tbl         Clostridioides_difficile_7501-BI_ST1.tbl
Clostridioides_difficile_5559-DH_ST42,.tbl      Clostridioides_difficile_6635-BI_ST1.tbl          Clostridioides_difficile_7503-Y_ST2,.tbl
Clostridioides_difficile_5559-DH_ST42.tbl       Clostridioides_difficile_6637-AL_ST58,.tbl        Clostridioides_difficile_7503-Y_ST2.tbl
Clostridioides_difficile_5573-CF_ST37,.tbl      Clostridioides_difficile_6637-AL_ST58.tbl         Clostridioides_difficile_7505-Y_ST2,.tbl
Clostridioides_difficile_5573-CF_ST37.tbl       Clostridioides_difficile_7450-DH_ST42,.tbl        Clostridioides_difficile_7505-Y_ST2.tbl
Clostridioides_difficile_5616-DH_ST42,.tbl      Clostridioides_difficile_7450-DH_ST42.tbl         Clostridioides_difficile_NCKUH-21,.tbl
Clostridioides_difficile_5616-DH_ST42.tbl       Clostridioides_difficile_7451-NonSp_ST2,.tbl      Clostridioides_difficile_RC10,.tbl
Clostridioides_difficile_5754,.tbl              Clostridioides_difficile_7451-NonSp_ST2.tbl       Clostridioides_difficile_RC10.tbl
Clostridioides_difficile_5754.tbl               Clostridioides_difficile_7452-Y_ST110,.tbl        Clostridioides_difficile_RF17,.tbl
Clostridioides_difficile_6583-AH_ST67,.tbl      Clostridioides_difficile_7452-Y_ST110.tbl         Clostridioides_difficile_RF17.tbl
Clostridioides_difficile_6583-AH_ST67.tbl       Clostridioides_difficile_7453-Y_ST2,.tbl          Peptoanaerobacter_stomatis_OBRC8,.tbl
Clostridioides_difficile_6584-AQ_ST26,.tbl      Clostridioides_difficile_7453-Y_ST2.tbl           Peptoanaerobacter_stomatis_OBRC8.tbl
Clostridioides_difficile_6584-AQ_ST26.tbl       Clostridioides_difficile_7455-A_ST43,.tbl         Romboutsia_maritimum_CCRI-22766,.tbl
Clostridioides_difficile_6588-G_ST8,.tbl        Clostridioides_difficile_7455-A_ST43.tbl          Romboutsia_maritimum_CCRI-22766.tbl
Clostridioides_difficile_6588-G_ST8.tbl         Clostridioides_difficile_7456-DH_ST42,.tbl        Tepidibacter_mesophilus_JCM.tbl
Clostridioides_difficile_6589-DH_ST42,.tbl      Clostridioides_difficile_7456-DH_ST42.tbl
</code></pre>

<p>Finally, you can convert these back to fasta with:</p>

<pre><code>for f in *tbl; do TblToFasta $f &gt; ${f/.tbl/.fa} &amp;&amp; rm $f; done
</code></pre>

<p>This will only get those sequences whose naming scheme is as described above. But that's a start. You can then see what else you have and extract the rest manually or by finding a similar approach that matches those.     </p>

CC BY-SA 3.0
2017-12-08T10:06:59.213Z
@AudileF we can't know that unless we know the details of your sequences. it all depends on how the strains are defined. I assume this is about the sequences you downloaded using edirect from your other question *but that is the type of thing you should mention in your question*. It's only luck that I happen to have read the other one. I am trying to give you a more general solution but the names of the few sequences I downloaded are not standard so it will not be easy.
terdon
CC BY-SA 3.0
2017-12-08T09:58:02.083Z
2017-12-08T10:24:54.633Z
2017-12-08T10:24:54.633Z
terdon
<p>I have download a batch of refseq fasta files and want to split them based on strain. this is complicated by accession numbers, project IDs etc. I downloaded the genomes <a href="https://bioinformatics.stackexchange.com/questions/3012/ncbi-edirect-suite-to-download-all-genome-sequences-associated-with-a-query">via eDirect from NCBI</a>. I know I can split the output but I want them together for some preliminary parsing (remove ambiguous and plasmid sequences etc.)</p>

<p>See example below. </p>

<pre><code>&gt;NZ_AESB01000237.1 Clostridium botulinum BKT028387 contig00237 ATGC
&gt;NZ_AESB01000236.1 Clostridium botulinum BKT028387 contig00236 ATGC
&gt;NZ_ALNK01000035.1 Peptoanaerobacter stomatis strain OBRC8 ctg120007290634 ATGC
&gt;NZ_ALNK01000031.1 Peptoanaerobacter stomatis strain OBRC8 ctg120007290635 ATGC
</code></pre>

<p>I want two files:
Clostridium botulinum BKT028387.fas and Peptoanaerobacter stomatis OBRC8.fas</p>

<p>My actual dataset is much larger (~70,000 genomes) and not too sure about the headers in each case.</p>

<hr>

<p>Im thinking the following could work but Im not sure how to execute it:</p>

<p>Using a <a href="https://unix.stackexchange.com/questions/361616/extract-lines-from-large-tab-delimited-file-using-a-list">grep command</a> to extract the lines but split the output.</p>

<p>If I download the summary list from refseq and use the names as the query example:</p>

<pre><code>grep -Fwf refseq.list sequences.fa
</code></pre>

<blockquote>
  <p>Summary file</p>
  
  <ol>
  <li><p>Peptoanaerobacter stomatis strain OBRC8, whole genome shotgun sequencing project 7,810,226 rc other DNA  NZ_AJXR00000000.1 GI:938474759</p>
  
  <ol start="2">
  <li>Clostridium botulinum BKT028387, whole genome shotgun sequencing project 4,134,593 rc other DNA  NZ_NOJY00000000.1 GI:1231817869</li>
  </ol></li>
  </ol>
  
  <p>But I could edit it down to </p>
  
  <p>Peptoanaerobacter stomatis strain OBRC8</p>
  
  <p>Clostridium botulinum BKT028387</p>
</blockquote>

<p>Am I one the right track. How do I finish this?</p>

CC BY-SA 3.0
2017-12-08T10:14:54.657Z
What operating system are you using? I asked you in your previous question as well but you didn't reply. The tools available to you and their behavior change depending on your OS.
terdon
CC BY-SA 3.0
2017-12-08T10:18:55.020Z
Unix? Do you mean an actual Unix or macOS or Linux? Please tell us your actual OS, a unix-like system can be assumed if you're doing bioinformatics usually.
terdon
CC BY-SA 3.0
2017-12-08T10:23:33.803Z
Yes Its is a linux system .
AudileF
CC BY-SA 3.0
2017-12-08T10:25:10.003Z
Great, then you should have the GNU tools. Good. Have a look at the updated answer. What does the summary list look like?
terdon
CC BY-SA 3.0
2017-12-08T10:38:27.087Z
Ah! That summary file can simplify things enormously! Can you link to the entire file? Does it have all the strains and does it have each strain in *exactly* the same format as it appears in the fasta file header? I am afraid it won't since I've already found inconsistencies. It might be simpler to start from the beginning and download each strain separately.
terdon
CC BY-SA 3.0
2017-12-08T12:14:37.057Z
For the files you have terdon, use the link and use 'send to: > complete record > file > summary > create file` https://www.ncbi.nlm.nih.gov/nuccore/?term=peptostreptococcaceae Im pretty confident they match any left behind I can curate manually.
AudileF
CC BY-SA 3.0
2017-12-08T12:27:49.920Z
OK, no, that summary is useless. It is unstructured and non-standard. For example, it has `Peptostreptococcaceae bacterium` and `Uncultured Peptostreptococcaceae bacterium`. I'm afraid this isn't the right way of doing what you need. It would probably be best to use a more sophisticated approach and save each sequence to the right file while downloading by also downloading the meta data for the sequence entry.
terdon
<fasta><data-management>
How to split multifasta based on partial fasta header
# How to split multifasta based on partial fasta header

(Asked by: username_0 on 2017-12-08)

I have download a batch of refseq fasta files and want to split them based on strain. this is complicated by accession numbers, project IDs etc. I downloaded the genomes via eDirect from NCBI. I know I can split the output but I want them together for some preliminary parsing (remove ambiguous and plasmid sequences etc.)
See example below. 
>NZ_AESB01000237.1 Clostridium botulinum BKT028387 contig00237 ATGC
>NZ_AESB01000236.1 Clostridium botulinum BKT028387 contig00236 ATGC
>NZ_ALNK01000035.1 Peptoanaerobacter stomatis strain OBRC8 ctg120007290634 ATGC
>NZ_ALNK01000031.1 Peptoanaerobacter stomatis strain OBRC8 ctg120007290635 ATGC

I want two files:
Clostridium botulinum BKT028387.fas and Peptoanaerobacter stomatis OBRC8.fas
My actual dataset is much larger (~70,000 genomes) and not too sure about the headers in each case.

Im thinking the following could work but Im not sure how to execute it:
Using a grep command to extract the lines but split the output.
If I download the summary list from refseq and use the names as the query example:
grep -Fwf refseq.list sequences.fa


Summary file

Peptoanaerobacter stomatis strain OBRC8, whole genome shotgun sequencing project 7,810,226 rc other DNA  NZ_AJXR00000000.1 GI:938474759

Clostridium botulinum BKT028387, whole genome shotgun sequencing project 4,134,593 rc other DNA  NZ_NOJY00000000.1 GI:1231817869


But I could edit it down to 
Peptoanaerobacter stomatis strain OBRC8
Clostridium botulinum BKT028387

Am I one the right track. How do I finish this?

**Tags:** fasta, data-management
> **Comments:**
> **username_1**: What operating system are you using? I asked you in your previous question as well but you didn't reply. The tools available to you and their behavior change depending on your OS.  
> **username_1**: Unix? Do you mean an actual Unix or macOS or Linux? Please tell us your actual OS, a unix-like system can be assumed if you're doing bioinformatics usually.  
> **username_0**: Yes Its is a linux system .  
> **username_1**: Great, then you should have the GNU tools. Good. Have a look at the updated answer. What does the summary list look like?  
> **username_1**: Ah! That summary file can simplify things enormously! Can you link to the entire file? Does it have all the strains and does it have each strain in *exactly* the same format as it appears in the fasta file header? I am afraid it won't since I've already found inconsistencies. It might be simpler to start from the beginning and download each strain separately.  
> **username_0**: For the files you have terdon, use the link and use 'send to: > complete record > file > summary > create file` https://www.ncbi.nlm.nih.gov/nuccore/?term=peptostreptococcaceae Im pretty confident they match any left behind I can curate manually.  
> **username_1**: OK, no, that summary is useless. It is unstructured and non-standard. For example, it has `Peptostreptococcaceae bacterium` and `Uncultured Peptostreptococcaceae bacterium`. I'm afraid this isn't the right way of doing what you need. It would probably be best to use a more sophisticated approach and save each sequence to the right file while downloading by also downloading the meta data for the sequence entry.  

---

## 3 answers

### Answer by username_2 on 2018-03-29. Score: 1

I had a similar problem and found a solution using awk. Since this is the first time I use awk, the syntax might not be perfect, but you should get an idea of what is possible.
awk -F' ' '{if(/^>/){sub("strain ", ""); name=$2" "$3" "$4; print > name".fa"}else{print > name".fa"}}' yourfile.fa

(more information below)
This should transform a file like this:
yourfile.fa
>NZ_AESB01000237.1 Clostridium botulinum BKT028387 contig00237 ATGC
ATGCATGC
ATGCATGC
>NZ_AESB01000236.1 Clostridium botulinum BKT028387 contig00236 ATGC
ATGCATGC
>NZ_ALNK01000035.1 Peptoanaerobacter stomatis strain OBRC8 ctg120007290634 ATGC
ATGCATGC
ATGCATGC
>NZ_ALNK01000031.1 Peptoanaerobacter stomatis strain OBRC8 ctg120007290635 ATGC
ATGCATGC

Into two files like this:
'Clostridium botulinum BKT028387.fa'
>NZ_AESB01000237.1 Clostridium botulinum BKT028387 contig00237 ATGC
ATGCATGC
ATGCATGC
>NZ_AESB01000236.1 Clostridium botulinum BKT028387 contig00236 ATGC
ATGCATGC

'Peptoanaerobacter stomatis OBRC8.fa'
>NZ_ALNK01000035.1 Peptoanaerobacter stomatis OBRC8 ctg120007290634 ATGC
ATGCATGC
ATGCATGC
>NZ_ALNK01000031.1 Peptoanaerobacter stomatis OBRC8 ctg120007290635 ATGC
ATGCATGC

How it works

-F' ' tells awk to split the line by space (accessible with $1, $2, ...)
if(/^>/) looks for line starting with the > character, then sub("strain ", "") remove the word strain and name=$2" "$3" "$4; print > name".fa" print the line into a new file with a name based on what you asked for.
else if the line does not start with >, {print > name".fa"} appends it to a file using the current value of name.

This solution will work regardless of the number of lines between the fasta headers.
Note that I won't recommend using spaces in filenames, so I would replace name=$2" "$3" "$4 with something like name=$2"_"$3"_"$4.
I used this solution to split a large fasta file (>5 GB) containing >5k genomes with >25k scaffolds and I found it fast and rather memory efficient.

> **Comments:**
> **username_3**: Thank you so much for the code and explanation. However, when I tried it, it gave me the following error. Any ideas?  `awk: syntax error at source line 1  context is  {if(/^>/){ name=$2" "$3" "$4; print > >>>  name".fa" <<<  awk: illegal statement at source line 1 awk: syntax error at source line 1`  
> **username_2**: I'm not sure about your error but you can probably put the code on several lines for easier debugging, and execute it with the awk -f option like this: `awk -f myscript.awk mygenomes.fasta`.  

---

### Answer by username_4 on 2017-12-14. Score: 1

The SeqBuddy --pull_records command can grab sequences based on a regular expression against record IDs and/or metadata. Given your example:
$: seqbuddy.py sequences.fa --pull_records "Clostridium botulinum.*BKT028387" full > Clostridium_botulinum_BKT028387.fas
$: seqbuddy.py sequences.fa --pull_records "Peptoanaerobacter stomatis.*OBRC8" full > Peptoanaerobacter_stomatis_OBRC8.fas

> **Comments:**
> **username_0**: This looks great. Is there a way to set this up to loop using an input list of names?  
> **username_4**: Yep, you just pass in the file name.  

---

### Answer by username_1 on 2017-12-08. Score: 0

I would convert the fasta file to a format with one line per record (I have already posted the FastaToTbl and TblToFasta scripts I use for this before), and then look for a specific pattern:
FastaToTbl file.fa | grep 'BKT028387' | TblToFasta > clostridium.fa
FastaToTbl file.fa | grep 'OBRC8' | TblToFasta > peptoanaerobacter.fa

I have downloaded a few sequences using the same command you described in your previous question, and it looks like the headers are not standardized. For example, while you have many that are in the format of $species strain $strain, like these:
Clostridioides difficile strain 7496-D/ST9
Clostridioides difficile strain 5616-DH/ST42,
Peptoanaerobacter stomatis strain OBRC8
Romboutsia maritimum strain CCRI-22766
Tepidibacter mesophilus strain JCM

Others don't have the word strain and just give the strain:
Peptostreptococcus anaerobius 653-L contig00088,
Peptoclostridium acidaminophilum DSM 3953

So, as a first step, you could deal with the ones that do have the word strain as the 4th word of the header line (the 1st is the seqID which I have not included in the examples above) and the actual strain as the 5th and get those out of the way:
FastaToTbl foo.fa | 
    awk '($4=="strain"){gsub("/","_"); print $0 >> $2"_"$3"_"$5".tbl"}'

That will look for lines whose 4th field is strain, then it will replace any / in the line with _ (there's a strain called CF/ST37 and that cannot be in a filename) and print into a file whose name is the 2nd, 3rd and 5th fields of the line concatenated. On my test file, this produced the following files (as you can see, some are wrong since there is a comma after the strain, but that's because they're not standardized):
$ ls *tbl
Clostridioides_difficile_5479-NonSp_ST229,.tbl  Clostridioides_difficile_6589-DH_ST42.tbl         Clostridioides_difficile_7457-NonSp_novelST,.tbl
Clostridioides_difficile_5479-NonSp_ST229.tbl   Clostridioides_difficile_6592-G_ST8,.tbl          Clostridioides_difficile_7457-NonSp_novelST.tbl
Clostridioides_difficile_5480-E_ST6,.tbl        Clostridioides_difficile_6592-G_ST8.tbl           Clostridioides_difficile_7458-N_ST10,.tbl
Clostridioides_difficile_5480-E_ST6.tbl         Clostridioides_difficile_6593-Y_ST110,.tbl        Clostridioides_difficile_7458-N_ST10.tbl
Clostridioides_difficile_5482-L_ST55,.tbl       Clostridioides_difficile_6593-Y_ST110.tbl         Clostridioides_difficile_7459-G_ST8,.tbl
Clostridioides_difficile_5482-L_ST55.tbl        Clostridioides_difficile_6595-AL_ST58,.tbl        Clostridioides_difficile_7459-G_ST8.tbl
Clostridioides_difficile_5485-Y_ST2,.tbl        Clostridioides_difficile_6595-AL_ST58.tbl         Clostridioides_difficile_7462-DH_ST42,.tbl
Clostridioides_difficile_5485-Y_ST2.tbl         Clostridioides_difficile_6596-D_ST9,.tbl          Clostridioides_difficile_7462-DH_ST42.tbl
Clostridioides_difficile_5488-DH_ST42,.tbl      Clostridioides_difficile_6596-D_ST9.tbl           Clostridioides_difficile_7463-E_ST6,.tbl
Clostridioides_difficile_5488-DH_ST42.tbl       Clostridioides_difficile_6599-NonSp_ST103,.tbl    Clostridioides_difficile_7463-E_ST6.tbl
Clostridioides_difficile_5489-M_ST15,.tbl       Clostridioides_difficile_6599-NonSp_ST103.tbl     Clostridioides_difficile_7464-Y_ST110,.tbl
Clostridioides_difficile_5489-M_ST15.tbl        Clostridioides_difficile_6600-G_ST8,.tbl          Clostridioides_difficile_7464-Y_ST110.tbl
Clostridioides_difficile_5490-CF_ST37,.tbl      Clostridioides_difficile_6600-G_ST8.tbl           Clostridioides_difficile_7466-DH_ST42,.tbl
Clostridioides_difficile_5490-CF_ST37.tbl       Clostridioides_difficile_6606-BK_ST11,.tbl        Clostridioides_difficile_7466-DH_ST42.tbl
Clostridioides_difficile_5493-DH_ST42,.tbl      Clostridioides_difficile_6606-BK_ST11.tbl         Clostridioides_difficile_7467-DH_ST42,.tbl
Clostridioides_difficile_5493-DH_ST42.tbl       Clostridioides_difficile_6607-DH_ST42,.tbl        Clostridioides_difficile_7467-DH_ST42.tbl
Clostridioides_difficile_5495-AF_ST41,.tbl      Clostridioides_difficile_6607-DH_ST42.tbl         Clostridioides_difficile_7468-NonSp_ST97,.tbl
Clostridioides_difficile_5495-AF_ST41.tbl       Clostridioides_difficile_6610-N_ST10,.tbl         Clostridioides_difficile_7468-NonSp_ST97.tbl
Clostridioides_difficile_5497-Y_ST2,.tbl        Clostridioides_difficile_6610-N_ST10.tbl          Clostridioides_difficile_7471-BI_ST1,.tbl
Clostridioides_difficile_5497-Y_ST2.tbl         Clostridioides_difficile_6612-DH_ST28,.tbl        Clostridioides_difficile_7471-BI_ST1.tbl
Clostridioides_difficile_5500-DH_ST42,.tbl      Clostridioides_difficile_6612-DH_ST28.tbl         Clostridioides_difficile_7472-Y_ST2,.tbl
Clostridioides_difficile_5500-DH_ST42.tbl       Clostridioides_difficile_6613-NonSp_ST3,.tbl      Clostridioides_difficile_7472-Y_ST2.tbl
Clostridioides_difficile_5504-L_ST55,.tbl       Clostridioides_difficile_6613-NonSp_ST3.tbl       Clostridioides_difficile_7473-DH_ST42,.tbl
Clostridioides_difficile_5504-L_ST55.tbl        Clostridioides_difficile_6614-AL_ST34,.tbl        Clostridioides_difficile_7473-DH_ST42.tbl
Clostridioides_difficile_5505-Y_ST2,.tbl        Clostridioides_difficile_6614-AL_ST34.tbl         Clostridioides_difficile_7476-Y_ST2,.tbl
Clostridioides_difficile_5505-Y_ST2.tbl         Clostridioides_difficile_6615-NonSp_ST11,.tbl     Clostridioides_difficile_7476-Y_ST2.tbl
Clostridioides_difficile_5537-D_ST9,.tbl        Clostridioides_difficile_6615-NonSp_ST11.tbl      Clostridioides_difficile_7478-NonSp_novelST,.tbl
Clostridioides_difficile_5537-D_ST9.tbl         Clostridioides_difficile_6616-NonSp_novelST,.tbl  Clostridioides_difficile_7478-NonSp_novelST.tbl
Clostridioides_difficile_5538-BK_ST11,.tbl      Clostridioides_difficile_6616-NonSp_novelST.tbl   Clostridioides_difficile_7479-NonSp_ST3,.tbl
Clostridioides_difficile_5538-BK_ST11.tbl       Clostridioides_difficile_6617-A_ST43,.tbl         Clostridioides_difficile_7479-NonSp_ST3.tbl
Clostridioides_difficile_5539-Y_ST2,.tbl        Clostridioides_difficile_6617-A_ST43.tbl          Clostridioides_difficile_7480-M_ST15,.tbl
Clostridioides_difficile_5539-Y_ST2.tbl         Clostridioides_difficile_6618-NonSp_ST53,.tbl     Clostridioides_difficile_7480-M_ST15.tbl
Clostridioides_difficile_5541-CF_ST37,.tbl      Clostridioides_difficile_6618-NonSp_ST53.tbl      Clostridioides_difficile_7481-NonSp_ST59,.tbl
Clostridioides_difficile_5541-CF_ST37.tbl       Clostridioides_difficile_6619-CF_ST37,.tbl        Clostridioides_difficile_7481-NonSp_ST59.tbl
Clostridioides_difficile_5542-Y_ST2,.tbl        Clostridioides_difficile_6619-CF_ST37.tbl         Clostridioides_difficile_7482-DH_ST42,.tbl
Clostridioides_difficile_5542-Y_ST2.tbl         Clostridioides_difficile_6621-NonSp_ST10,.tbl     Clostridioides_difficile_7482-DH_ST42.tbl
Clostridioides_difficile_5547-DH_ST42,.tbl      Clostridioides_difficile_6621-NonSp_ST10.tbl      Clostridioides_difficile_7486-A_ST43,.tbl
Clostridioides_difficile_5547-DH_ST42.tbl       Clostridioides_difficile_6622-CF_ST37,.tbl        Clostridioides_difficile_7486-A_ST43.tbl
Clostridioides_difficile_5548-Y_ST2,.tbl        Clostridioides_difficile_6622-CF_ST37.tbl         Clostridioides_difficile_7488-NonSp_ST103,.tbl
Clostridioides_difficile_5548-Y_ST2.tbl         Clostridioides_difficile_6623-NonSp_ST54,.tbl     Clostridioides_difficile_7488-NonSp_ST103.tbl
Clostridioides_difficile_5549-NonSp_ST21,.tbl   Clostridioides_difficile_6623-NonSp_ST54.tbl      Clostridioides_difficile_7491-DH_ST42,.tbl
Clostridioides_difficile_5549-NonSp_ST21.tbl    Clostridioides_difficile_6625-Y_ST2,.tbl          Clostridioides_difficile_7491-DH_ST42.tbl
Clostridioides_difficile_5550-AH_ST67,.tbl      Clostridioides_difficile_6625-Y_ST2.tbl           Clostridioides_difficile_7492-CF_ST37,.tbl
Clostridioides_difficile_5550-AH_ST67.tbl       Clostridioides_difficile_6626-Y_ST110,.tbl        Clostridioides_difficile_7492-CF_ST37.tbl
Clostridioides_difficile_5551-CF_ST37,.tbl      Clostridioides_difficile_6626-Y_ST110.tbl         Clostridioides_difficile_7494-E_ST6,.tbl
Clostridioides_difficile_5551-CF_ST37.tbl       Clostridioides_difficile_6628-AL_ST58,.tbl        Clostridioides_difficile_7494-E_ST6.tbl
Clostridioides_difficile_5552-DH_ST42,.tbl      Clostridioides_difficile_6628-AL_ST58.tbl         Clostridioides_difficile_7496-D_ST9,.tbl
Clostridioides_difficile_5552-DH_ST42.tbl       Clostridioides_difficile_6629-NonSp_novelST,.tbl  Clostridioides_difficile_7496-D_ST9.tbl
Clostridioides_difficile_5553-L_ST55,.tbl       Clostridioides_difficile_6629-NonSp_novelST.tbl   Clostridioides_difficile_7497-NonSp_novelST,.tbl
Clostridioides_difficile_5553-L_ST55.tbl        Clostridioides_difficile_6631-DH_ST42,.tbl        Clostridioides_difficile_7497-NonSp_novelST.tbl
Clostridioides_difficile_5555-DH_ST42,.tbl      Clostridioides_difficile_6631-DH_ST42.tbl         Clostridioides_difficile_7498-M_ST15,.tbl
Clostridioides_difficile_5555-DH_ST42.tbl       Clostridioides_difficile_6632-NonSp_ST3,.tbl      Clostridioides_difficile_7498-M_ST15.tbl
Clostridioides_difficile_5557-DH_ST42,.tbl      Clostridioides_difficile_6632-NonSp_ST3.tbl       Clostridioides_difficile_7499-CF_ST37,.tbl
Clostridioides_difficile_5557-DH_ST42.tbl       Clostridioides_difficile_6634-DH_ST42,.tbl        Clostridioides_difficile_7499-CF_ST37.tbl
Clostridioides_difficile_5558-AH_ST67,.tbl      Clostridioides_difficile_6634-DH_ST42.tbl         Clostridioides_difficile_7501-BI_ST1,.tbl
Clostridioides_difficile_5558-AH_ST67.tbl       Clostridioides_difficile_6635-BI_ST1,.tbl         Clostridioides_difficile_7501-BI_ST1.tbl
Clostridioides_difficile_5559-DH_ST42,.tbl      Clostridioides_difficile_6635-BI_ST1.tbl          Clostridioides_difficile_7503-Y_ST2,.tbl
Clostridioides_difficile_5559-DH_ST42.tbl       Clostridioides_difficile_6637-AL_ST58,.tbl        Clostridioides_difficile_7503-Y_ST2.tbl
Clostridioides_difficile_5573-CF_ST37,.tbl      Clostridioides_difficile_6637-AL_ST58.tbl         Clostridioides_difficile_7505-Y_ST2,.tbl
Clostridioides_difficile_5573-CF_ST37.tbl       Clostridioides_difficile_7450-DH_ST42,.tbl        Clostridioides_difficile_7505-Y_ST2.tbl
Clostridioides_difficile_5616-DH_ST42,.tbl      Clostridioides_difficile_7450-DH_ST42.tbl         Clostridioides_difficile_NCKUH-21,.tbl
Clostridioides_difficile_5616-DH_ST42.tbl       Clostridioides_difficile_7451-NonSp_ST2,.tbl      Clostridioides_difficile_RC10,.tbl
Clostridioides_difficile_5754,.tbl              Clostridioides_difficile_7451-NonSp_ST2.tbl       Clostridioides_difficile_RC10.tbl
Clostridioides_difficile_5754.tbl               Clostridioides_difficile_7452-Y_ST110,.tbl        Clostridioides_difficile_RF17,.tbl
Clostridioides_difficile_6583-AH_ST67,.tbl      Clostridioides_difficile_7452-Y_ST110.tbl         Clostridioides_difficile_RF17.tbl
Clostridioides_difficile_6583-AH_ST67.tbl       Clostridioides_difficile_7453-Y_ST2,.tbl          Peptoanaerobacter_stomatis_OBRC8,.tbl
Clostridioides_difficile_6584-AQ_ST26,.tbl      Clostridioides_difficile_7453-Y_ST2.tbl           Peptoanaerobacter_stomatis_OBRC8.tbl
Clostridioides_difficile_6584-AQ_ST26.tbl       Clostridioides_difficile_7455-A_ST43,.tbl         Romboutsia_maritimum_CCRI-22766,.tbl
Clostridioides_difficile_6588-G_ST8,.tbl        Clostridioides_difficile_7455-A_ST43.tbl          Romboutsia_maritimum_CCRI-22766.tbl
Clostridioides_difficile_6588-G_ST8.tbl         Clostridioides_difficile_7456-DH_ST42,.tbl        Tepidibacter_mesophilus_JCM.tbl
Clostridioides_difficile_6589-DH_ST42,.tbl      Clostridioides_difficile_7456-DH_ST42.tbl

Finally, you can convert these back to fasta with:
for f in *tbl; do TblToFasta $f > ${f/.tbl/.fa} && rm $f; done

This will only get those sequences whose naming scheme is as described above. But that's a start. You can then see what else you have and extract the rest manually or by finding a similar approach that matches those.

> **Comments:**
> **username_1**: @AudileF we can't know that unless we know the details of your sequences. it all depends on how the strains are defined. I assume this is about the sequences you downloaded using edirect from your other question *but that is the type of thing you should mention in your question*. It's only luck that I happen to have read the other one. I am trying to give you a more general solution but the names of the few sequences I downloaded are not standard so it will not be easy.  

---

