<p>You can use <code>awk</code> here, by setting the input (<code>RS</code>) and output (<code>ORS</code>) record separator, this is what defines a &quot;line&quot; in awk, to <code>&gt;</code>. Then, tell it to print records that do not match <code>Sequence unavailable</code>. For instance, using this input file:</p>
<pre><code>$ cat file
&gt;foo
AGAGTAGATAGAAGATA
&gt;bar
AGAGATAGAGAGATATA
&gt;baz
Sequence unavailable
&gt;bar1
AGAGATAGAGAGATATA
&gt;asld
Sequence unavailable
&gt;asd
ACTTC
</code></pre>
<p>You can do:</p>
<pre><code>$ awk  'BEGIN{RS=ORS=&quot;&gt;&quot;}!/Sequence unavailable/' file 
&gt;foo
AGAGTAGATAGAAGATA
&gt;bar
AGAGATAGAGAGATATA
&gt;bar1
AGAGATAGAGAGATATA
&gt;asd
ACTTC
&gt;
</code></pre>
<p>That will add a trailing <code>&gt;</code>, so you need one more step to remove it:</p>
<pre><code>$ awk  'BEGIN{RS=ORS=&quot;&gt;&quot;}!/Sequence unavailable/' file | sed 's/&gt;$//'
&gt;foo
AGAGTAGATAGAAGATA
&gt;bar
AGAGATAGAGAGATATA
&gt;bar1
AGAGATAGAGAGATATA
&gt;asd
ACTTC
</code></pre>
<hr />
<p>Alternatively, using the <code>FastaToTbl</code> and <code>TblToFasta</code> scripts I have posted <a href="https://bioinformatics.stackexchange.com/a/2651/298">here</a>, you can do:</p>
<pre><code>$ FastaToTbl file | grep -v 'Sequence unavailable' | TblToFasta 
&gt;foo 
AGAGTAGATAGAAGATA
&gt;bar 
AGAGATAGAGAGATATA
&gt;bar1 
AGAGATAGAGAGATATA
&gt;asd 
ACTTC
</code></pre>
<p>Both approaches have the advantage of not needing to read the entire file to memory so can be used on files of any size.</p>

CC BY-SA 4.0
2023-01-11T23:40:45.067Z
FWIW with GNU awk you could do `awk  'BEGIN{RS=">"} {ORS=RT} !/Sequence unavailable/' file` instead of piping to `sed`.
Ed Morton
CC BY-SA 4.0
2022-11-05T13:08:12.123Z
2022-11-05T13:08:12.123Z
terdon
<p>Try this, if you have GNU <code>grep</code> (the default on Linux systems):</p>
<pre><code>grep --no-group-separator -i -B 1 '^[ACGTN]\+$' input.fasta &gt; filtered.fasta
</code></pre>
<p>Explanation: find all lines that are composed of only A/C/G/T/N (case insensitive search), and show that line plus the preceding line. Don't put any breaks between matches that are not consecutive.</p>
<p>This should work on both multi-line and single line fasta files.</p>

CC BY-SA 4.0
2022-11-05T13:06:34.450Z
2022-11-05T13:14:16.047Z
2022-11-05T13:14:16.047Z
gringer
<p>If there are no line breaks within the sequence data (the format I use) then the following will work.</p>
<pre><code>perl -0777 -nle 'while (/^(&gt;.+\n^[AGCT-]+)\n/gm){print $1,&quot;\n&quot;}' myfile.fa
</code></pre>
<hr />
<p>Test</p>
<pre><code>&gt;foo
AGAGTAGATAGAAGATA
&gt;bar
AGAGATAGAGAGATATA
&gt;baz
Sequence unavailable
</code></pre>
<p>Output</p>
<pre><code>&gt;foo
AGAGTAGATAGAAGATA
&gt;bar
AGAGATAGAGAGATATA
</code></pre>
<hr />
<p><strong>However</strong> If there are line breaks within the sequence data, please use the following ...</p>
<pre><code>perl -0777 -nle 'while (/^(&gt;.+\n[AGCT-]{10}.+\n[^&gt;]+)/gm){print $1}' myfile.fa
</code></pre>
<p>Then pipe to a file,</p>
<pre><code>perl -0777 -nle 'while (/^(&gt;.+\n[AGCT-]{10}.+\n[^&gt;]+)/gm){print $1}' myfile.fa &gt; myfile_minus_blanks.fa
</code></pre>
<p>This works due to a multi-line <code>m</code></p>
<p>Test ...</p>
<pre><code>&gt;foo
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
&gt;bar
AGAGATAGAGAGATATA
AGAGATAGAGAGATATA
&gt;baz
Sequence unavailable
</code></pre>
<p>Out ...</p>
<pre><code>&gt;foo
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
&gt;bar
AGAGATAGAGAGATATA
AGAGATAGAGAGATATA
</code></pre>
<p><strong>Note</strong>
If you've mixed bases, degenerate nucleotides,</p>
<p><code>AGCT-</code> should be replaced with <code>AGCTRYWSMKNBDHV</code> (omitting <code>-</code> [not needed])</p>

CC BY-SA 4.0
2022-11-05T13:00:39.350Z
Note that this will require loading the entire file into memory which might be an issue with large files.
terdon
CC BY-SA 4.0
2022-11-05T22:26:51.810Z
I just processed a 22Gig file with a slightly more complicated command (removing loci). It worked fine.
M__
CC BY-SA 4.0
2022-11-06T09:42:42.350Z
Thank you very much for your kind response. This is work fro me.
Dr Sud
CC BY-SA 4.0
2022-11-06T13:30:20.100Z
Hi @DrSud great! I use it because its versatile (use it for different stuff). Please do remember to mark an answer "accepted" if you find it helpful and you can also now upvote all the answers given here.
M__
CC BY-SA 4.0
2022-11-04T18:51:03.250Z
2022-11-05T12:58:19.177Z
2022-11-05T12:58:19.177Z
M__
<p>One way using <a href="/questions/tagged/sed" class="post-tag" title="show questions tagged &#39;sed&#39;" aria-label="show questions tagged &#39;sed&#39;" rel="tag" aria-labelledby="sed-container">sed</a>:</p>
<pre><code>sed -e '/^&gt;/ { N; }; /Sequence unavailable/ { d; }' file.fa
</code></pre>
<p>For each header line, append the next line of input into the pattern space. Then, if the pattern space contains &quot;Sequence unavailable&quot; delete the pattern space and start a new cycle.</p>
<hr />
<p>Or using <a href="/questions/tagged/awk" class="post-tag" title="show questions tagged &#39;awk&#39;" aria-label="show questions tagged &#39;awk&#39;" rel="tag" aria-labelledby="awk-container">awk</a>:</p>
<pre><code>awk '/^&gt;/ { f=$0; next } /^Sequence unavailable/ { next } f { print f; f=&quot;&quot; }1' file.fa
</code></pre>
<p>For each header line, use a variable to store the current record. We can use <a href="https://www.gnu.org/software/gawk/manual/html_node/Next-Statement.html" rel="nofollow noreferrer"><em>next</em></a> here to force awk to immediately stop processing the current record and move on to the next record, which is just the next line, since, by default, awk's <a href="https://www.gnu.org/software/gawk/manual/html_node/awk-split-records.html" rel="nofollow noreferrer"><em>record separator</em></a> is a newline character. Then, stop processing the current record and move on to the next record if that line begins with &quot;Sequence unavailable&quot;. Finally, print the header line if it exists and unset the variable that we used to store it if it does. Once we've printed the header, we can then just print the current record. We can do this by appending <em>1</em> to our command string. The <em>1</em> on the end just returns true, and by doing so, it performs the default action, which is to print the current record.</p>

CC BY-SA 4.0
2022-11-05T18:22:51.380Z
There's a bug in the sed code, `sed: 1: "/^>/ { N }; /Sequence u ...": extra characters at the end of N command`. OSX Darwin
M__
CC BY-SA 4.0
2022-11-05T21:54:48.893Z
@M__ ah... my mistake. Should be good now I think. Thanks for testing it on OSX
Steve
CC BY-SA 4.0
2022-11-05T22:29:39.493Z
Thanks but same error but as I'm not feeling motivated to test on Linux ...  lets say it an OSX thing, most work on Linux. The awk works fine.
M__
CC BY-SA 4.0
2022-11-06T14:20:15.060Z
@M__ I don't think I read your error message properly before. Sorry. Hopefully the above will work now..
Steve
CC BY-SA 4.0
2022-11-05T13:43:35.993Z
2022-11-06T14:22:43.730Z
2022-11-06T14:22:43.730Z
Steve
<p>I have a fasta file of 3'UTR sequences of the genes. Some of the gene do not have a 3'UTR sequence and reported with <code>sequence unavailable</code> but there is FASTA header.</p>
<p>I need to delete or remove all the &quot;sequence unavailable&quot; lines along with their header, i.e. all blank fasta entries.</p>
<p>Please suggest a command with <code>awk</code> or <code>sed</code> command.</p>
<p><a href="https://i.sstatic.net/5orlh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5orlh.png" alt="Image of FASTA file" /></a></p>

CC BY-SA 4.0
2022-11-05T13:31:07.230Z
Please [don't post images of text](https://bioinformatics.meta.stackexchange.com/q/326/298). Instead, copy/paste the text into your question and use the [formatting tools](https://bioinformatics.stackexchange.com/help/formatting) to format it as code. Also, what operating system are you using? We need to know so we can know what tools and what versions of the tools will be available.
terdon
<phylogenetics><phylogeny><fasta><awk><grep>
Removing particular line along with Fasta header in fasta file
# Removing particular line along with Fasta header in fasta file

(Asked by: username_0 on 2022-11-04)

I have a fasta file of 3'UTR sequences of the genes. Some of the gene do not have a 3'UTR sequence and reported with `sequence unavailable` but there is FASTA header.

I need to delete or remove all the "sequence unavailable" lines along with their header, i.e. all blank fasta entries.

Please suggest a command with `awk` or `sed` command.

[![Image of FASTA file](https://i.sstatic.net/5orlh.png)](https://i.sstatic.net/5orlh.png)

**Tags:** phylogenetics, phylogeny, fasta, awk, grep
> **Comments:**
> **username_1**: Please [don't post images of text](https://bioinformatics.meta.stackexchange.com/q/326/298). Instead, copy/paste the text into your question and use the [formatting tools](https://bioinformatics.stackexchange.com/help/formatting) to format it as code. Also, what operating system are you using? We need to know so we can know what tools and what versions of the tools will be available.  

---

## 4 answers

### Answer by username_1 on 2022-11-05. Score: 4

You can use awk here, by setting the input (RS) and output (ORS) record separator, this is what defines a "line" in awk, to >. Then, tell it to print records that do not match Sequence unavailable. For instance, using this input file:
$ cat file
>foo
AGAGTAGATAGAAGATA
>bar
AGAGATAGAGAGATATA
>baz
Sequence unavailable
>bar1
AGAGATAGAGAGATATA
>asld
Sequence unavailable
>asd
ACTTC

You can do:
$ awk  'BEGIN{RS=ORS=">"}!/Sequence unavailable/' file 
>foo
AGAGTAGATAGAAGATA
>bar
AGAGATAGAGAGATATA
>bar1
AGAGATAGAGAGATATA
>asd
ACTTC
>

That will add a trailing >, so you need one more step to remove it:
$ awk  'BEGIN{RS=ORS=">"}!/Sequence unavailable/' file | sed 's/>$//'
>foo
AGAGTAGATAGAAGATA
>bar
AGAGATAGAGAGATATA
>bar1
AGAGATAGAGAGATATA
>asd
ACTTC


Alternatively, using the FastaToTbl and TblToFasta scripts I have posted here, you can do:
$ FastaToTbl file | grep -v 'Sequence unavailable' | TblToFasta 
>foo 
AGAGTAGATAGAAGATA
>bar 
AGAGATAGAGAGATATA
>bar1 
AGAGATAGAGAGATATA
>asd 
ACTTC

Both approaches have the advantage of not needing to read the entire file to memory so can be used on files of any size.

> **Comments:**
> **username_2**: FWIW with GNU awk you could do `awk  'BEGIN{RS=">"} {ORS=RT} !/Sequence unavailable/' file` instead of piping to `sed`.  

---

### Answer by username_3 on 2022-11-05. Score: 3

Try this, if you have GNU grep (the default on Linux systems):
grep --no-group-separator -i -B 1 '^[ACGTN]\+$' input.fasta > filtered.fasta

Explanation: find all lines that are composed of only A/C/G/T/N (case insensitive search), and show that line plus the preceding line. Don't put any breaks between matches that are not consecutive.
This should work on both multi-line and single line fasta files.


---

### Answer by username_4 on 2022-11-04. Score: 2

If there are no line breaks within the sequence data (the format I use) then the following will work.
perl -0777 -nle 'while (/^(>.+\n^[AGCT-]+)\n/gm){print $1,"\n"}' myfile.fa


Test
>foo
AGAGTAGATAGAAGATA
>bar
AGAGATAGAGAGATATA
>baz
Sequence unavailable

Output
>foo
AGAGTAGATAGAAGATA
>bar
AGAGATAGAGAGATATA


However If there are line breaks within the sequence data, please use the following ...
perl -0777 -nle 'while (/^(>.+\n[AGCT-]{10}.+\n[^>]+)/gm){print $1}' myfile.fa

Then pipe to a file,
perl -0777 -nle 'while (/^(>.+\n[AGCT-]{10}.+\n[^>]+)/gm){print $1}' myfile.fa > myfile_minus_blanks.fa

This works due to a multi-line m
Test ...
>foo
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
>bar
AGAGATAGAGAGATATA
AGAGATAGAGAGATATA
>baz
Sequence unavailable

Out ...
>foo
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
AGAGTAGATAGAAGATA
>bar
AGAGATAGAGAGATATA
AGAGATAGAGAGATATA

Note
If you've mixed bases, degenerate nucleotides,
AGCT- should be replaced with AGCTRYWSMKNBDHV (omitting - [not needed])

> **Comments:**
> **username_1**: Note that this will require loading the entire file into memory which might be an issue with large files.  
> **username_4**: I just processed a 22Gig file with a slightly more complicated command (removing loci). It worked fine.  
> **username_0**: Thank you very much for your kind response. This is work fro me.  
> **username_4**: Hi @DrSud great! I use it because its versatile (use it for different stuff). Please do remember to mark an answer "accepted" if you find it helpful and you can also now upvote all the answers given here.  

---

### Answer by username_5 on 2022-11-05. Score: 2

One way using sed:
sed -e '/^>/ { N; }; /Sequence unavailable/ { d; }' file.fa

For each header line, append the next line of input into the pattern space. Then, if the pattern space contains "Sequence unavailable" delete the pattern space and start a new cycle.

Or using awk:
awk '/^>/ { f=$0; next } /^Sequence unavailable/ { next } f { print f; f="" }1' file.fa

For each header line, use a variable to store the current record. We can use next here to force awk to immediately stop processing the current record and move on to the next record, which is just the next line, since, by default, awk's record separator is a newline character. Then, stop processing the current record and move on to the next record if that line begins with "Sequence unavailable". Finally, print the header line if it exists and unset the variable that we used to store it if it does. Once we've printed the header, we can then just print the current record. We can do this by appending 1 to our command string. The 1 on the end just returns true, and by doing so, it performs the default action, which is to print the current record.

> **Comments:**
> **username_4**: There's a bug in the sed code, `sed: 1: "/^>/ { N }; /Sequence u ...": extra characters at the end of N command`. OSX Darwin  
> **username_5**: @M__ ah... my mistake. Should be good now I think. Thanks for testing it on OSX  
> **username_4**: Thanks but same error but as I'm not feeling motivated to test on Linux ...  lets say it an OSX thing, most work on Linux. The awk works fine.  
> **username_5**: @M__ I don't think I read your error message properly before. Sorry. Hopefully the above will work now..  

---

