<p><strong>Don't do it:</strong> Your FASTQ file is either malformed or your FASTQ record spans more than four lines, which is allowed in FASTQ.  For a detailed description of what can go wrong in FASTQ parsing see for example <a href="http://biopython.org/DIST/docs/api/Bio.SeqIO.QualityIO-module.html#FastqGeneralIterator" rel="noreferrer">http://biopython.org/DIST/docs/api/Bio.SeqIO.QualityIO-module.html#FastqGeneralIterator</a>.</p>

<p>If the FASTQ is malformed, then you should really ask yourself how this happened in the first place and fix the source of the problem. If the record is valid FASTQ, then I suggest parsing the read with for example <code>FastqGeneralIterator</code> and dumping the parsed result back to FASTQ in a 4-line-per-record form.</p>

CC BY-SA 4.0
2018-11-09T09:23:58.110Z
2018-11-09T09:23:58.110Z
winni2k
<p>If you are 100% sure the read only has 4 lines (they can have more), you can use this <code>sed</code> command:</p>

<pre><code>sed -i.bak '/^@HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG/,+3d'
</code></pre>

<p>The <code>-i.bak</code> makes <code>sed</code> modify the original file and create a backup copy with the same name and the extension <code>.bak</code>. The command just means "delete the line matching the pattern and the next three lines".</p>

CC BY-SA 4.0
2018-10-31T12:44:02.447Z
Maybe the "@" should be added to the pattern. What would happen if the read name was also on the "+" line?
bli
CC BY-SA 4.0
2018-10-30T09:40:39.210Z
2018-11-21T19:17:19.677Z
2018-11-21T19:17:19.677Z
terdon
<p>When I can I like to do file processing line-by-line in the spirit of UNIX tools. You can read 4 lines from a Fastq file into 4 tab-separated values on a single line using <code>paste</code>, and then use <code>grep</code> to filter out the record in question. (Then you just have to turn the tabs back into line breaks.)</p>

<pre><code>paste - - - - &lt; reads.fastq \
    | grep -v 'HWI-D00466:116:CC62WANXX:3:1102:7363:63646' \
    | tr '\t' '\n' \
    &gt; reads-fixed.fastq
</code></pre>

<p>However, it's likely that other reads in your Fastq file are corrupted as well, in which case it's probably better to write a script in Python or some other language that discards all reads whose read length doesn't match the quality string length.</p>

<p>But of course the most important question is: how did the data become corrupted in the first place? Where did this file come from, and what pre-processing steps were performed on it? Is getting rid of this read (or all reads with mismatched sequence/quality length) going to solve the problem for you? These aren't questions that we can answer for you, but probably deserve good answers. :-)</p>

CC BY-SA 4.0
2018-11-01T15:20:17.277Z
2018-11-01T15:20:17.277Z
Daniel Standage
<p>Here's an <code>awk</code> one-liner that works as intended, unlike the <code>bioawk</code> answer.</p>

<p>This removes all instances of fastq records in which the length of the seq and qual fields do not match. Again, append <code>| gzip &gt; filtered.fastq.gz</code> to gzip the output.</p>

<pre><code>zcat bad_file.fq.gz | awk '{ pos = NR%4; if ( pos == 1) {h=<span class="math-container">$0} else if (pos == 2) \
  {s=$0} else if (pos==3){c=$0} else if ( pos==0 ) \
  {q=$</span>0; if ( length(q) == length(s) ){printf("%s\n%s\n%s\n%s\n", h, s, c, q)}}}'
</code></pre>

<p>Delete the <code>\</code> characters at the end of the first two lines to make it easy to paste into your terminal.</p>

<p>Here is a shorter version of the same thing thanks to @terdon</p>

<pre><code>zcat file.fastq.gz | awk '{ record[++k]=<span class="math-container">$0; if(NR%4==2){slen=length($0)} else if(NR%4==0){if(length($</span>0) == slen){ for(i in record){print record[i]}} k=0; }}'
</code></pre>

CC BY-SA 4.0
2018-10-31T15:51:28.393Z
Heh, I was just writing one too :) Here's what I came up with, it's a bit shorter than yours: `zcat file.fastq.gz | awk '{ record[++k]=$0; if(NR%4==2){slen=length($0)} else if(NR%4==0){if(length($0) == slen){ for(i in record){print record[i]}} k=0; }}'` feel free to add it to your answer if you want to.
terdon
CC BY-SA 4.0
2018-10-31T15:41:27.400Z
2018-11-01T15:51:00.017Z
2018-11-01T15:51:00.017Z
conchoecia
<p>To generalize this to remove all reads that do not have the same length of read and sequence quality, you can use this bioawk one-liner:</p>

<pre><code>bioawk -cfastx '{ if (length(<span class="math-container">$seq) == length($</span>qual) ){printf("@%s %s\n%s\n+\n%s\n", <span class="math-container">$name, $</span>comment, <span class="math-container">$seq, $</span>qual)} }' my_fastq.gz
</code></pre>

<p>This strips anything in the "fastq comment field", <code>+</code>, but that almost never has anything there anyway.</p>

<p>Append <code>| gzip &gt; filtered.fastq.gz</code> to the above command to gzip the output.</p>

CC BY-SA 4.0
2018-10-31T14:27:54.283Z
This doesn't seem to work for me (Arch Linux, bioawk version 20110810). Even after correcting the syntax error (corrected in your answer), it seems to exit after the first record it finds where the lengths are not the same. Can you confirm it works on your machine?
terdon
CC BY-SA 4.0
2018-10-31T15:15:02.677Z
Thanks for the edit! Yep, I just confirmed that it works for me on all records. I'm using Ubuntu 16.04. `>>bioawk --version` returns `awk version 20110810`. It looks like the last time that I compiled bioawk was from the github repo in 2016.
conchoecia
CC BY-SA 4.0
2018-10-31T15:18:35.250Z
That's really weird. I have the same version. Can you try a case where the record with different lengths is the second one in the file? Does it not just print the first record there?
terdon
CC BY-SA 4.0
2018-10-31T15:23:56.647Z
I got the same error as you - I made the second entry in the fastq file have a different length of seq and qual fields and `bioawk` just exited before printing that record. I wonder if this is intended behavior? I have to add that in my above comment, I just tried the bioawk one-liner on a normal fastq file without any length mismatches. Oops!
conchoecia
CC BY-SA 4.0
2018-10-31T15:35:30.507Z
I don't see how it can be intended. Sounds like a bug in bioawk.
terdon
CC BY-SA 4.0
2018-10-30T19:20:55.403Z
2018-10-31T13:50:33.817Z
2018-10-31T13:50:33.817Z
conchoecia
<p>I have the following error when running bowtie2:</p>

<blockquote>
  <p>Error: Read HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG
  has more read characters than quality values.</p>
</blockquote>

<p>I now want to remove all 4 lines of this specific read from the fastq file.</p>

<p>How can I use awk or sed to do this?</p>

CC BY-SA 4.0
2018-11-23T07:34:01.323Z
Run `tail` on your FASTQ file. If a job aborted while writing to disk, this read will be the last line. This error message likely means the file is truncated and missing reads. That read is not as much of a concern as the rest of the data. It’s best to run your data processing again to make sure the file is complete.
Tom Kelly ケリー・トム
<ngs><fastq><filtering><bioawk><text-processing>
Delete all 4 lines of a fastq read from a fastq file using read ID
# Delete all 4 lines of a fastq read from a fastq file using read ID

(Asked by: username_0 on 2018-10-30)

I have the following error when running bowtie2:

> Error: Read HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG
> has more read characters than quality values.

I now want to remove all 4 lines of this specific read from the fastq file.

How can I use awk or sed to do this?

**Tags:** ngs, fastq, filtering, bioawk, text-processing
> **Comments:**
> **username_1**: Run `tail` on your FASTQ file. If a job aborted while writing to disk, this read will be the last line. This error message likely means the file is truncated and missing reads. That read is not as much of a concern as the rest of the data. It’s best to run your data processing again to make sure the file is complete.  

---

## 5 answers

### Answer by username_2 on 2018-11-09. Score: 9

**Don't do it:** Your FASTQ file is either malformed or your FASTQ record spans more than four lines, which is allowed in FASTQ. For a detailed description of what can go wrong in FASTQ parsing see for example <http://biopython.org/DIST/docs/api/Bio.SeqIO.QualityIO-module.html#FastqGeneralIterator>.

If the FASTQ is malformed, then you should really ask yourself how this happened in the first place and fix the source of the problem. If the record is valid FASTQ, then I suggest parsing the read with for example `FastqGeneralIterator` and dumping the parsed result back to FASTQ in a 4-line-per-record form.


---

### Answer by username_3 on 2018-10-30. Score: 6

If you are 100% sure the read only has 4 lines (they can have more), you can use this sed command:
sed -i.bak '/^@HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG/,+3d'

The -i.bak makes sed modify the original file and create a backup copy with the same name and the extension .bak. The command just means "delete the line matching the pattern and the next three lines".

> **Comments:**
> **username_4**: Maybe the "@" should be added to the pattern. What would happen if the read name was also on the "+" line?  

---

### Answer by username_5 on 2018-11-01. Score: 3

When I can I like to do file processing line-by-line in the spirit of UNIX tools. You can read 4 lines from a Fastq file into 4 tab-separated values on a single line using paste, and then use grep to filter out the record in question. (Then you just have to turn the tabs back into line breaks.)
paste - - - - < reads.fastq \
    | grep -v 'HWI-D00466:116:CC62WANXX:3:1102:7363:63646' \
    | tr '\t' '\n' \
    > reads-fixed.fastq

However, it's likely that other reads in your Fastq file are corrupted as well, in which case it's probably better to write a script in Python or some other language that discards all reads whose read length doesn't match the quality string length.
But of course the most important question is: how did the data become corrupted in the first place? Where did this file come from, and what pre-processing steps were performed on it? Is getting rid of this read (or all reads with mismatched sequence/quality length) going to solve the problem for you? These aren't questions that we can answer for you, but probably deserve good answers. :-)


---

### Answer by username_6 on 2018-10-31. Score: 1

Here's an awk one-liner that works as intended, unlike the bioawk answer.
This removes all instances of fastq records in which the length of the seq and qual fields do not match. Again, append | gzip > filtered.fastq.gz to gzip the output.
zcat bad_file.fq.gz | awk '{ pos = NR%4; if ( pos == 1) {h=$0} else if (pos == 2) \
  {s=$0} else if (pos==3){c=$0} else if ( pos==0 ) \
  {q=$0; if ( length(q) == length(s) ){printf("%s\n%s\n%s\n%s\n", h, s, c, q)}}}'

Delete the \ characters at the end of the first two lines to make it easy to paste into your terminal.
Here is a shorter version of the same thing thanks to @terdon
zcat file.fastq.gz | awk '{ record[++k]=$0; if(NR%4==2){slen=length($0)} else if(NR%4==0){if(length($0) == slen){ for(i in record){print record[i]}} k=0; }}'

> **Comments:**
> **username_3**: Heh, I was just writing one too :) Here's what I came up with, it's a bit shorter than yours: `zcat file.fastq.gz | awk '{ record[++k]=$0; if(NR%4==2){slen=length($0)} else if(NR%4==0){if(length($0) == slen){ for(i in record){print record[i]}} k=0; }}'` feel free to add it to your answer if you want to.  

---

### Answer by username_6 on 2018-10-30. Score: 0

To generalize this to remove all reads that do not have the same length of read and sequence quality, you can use this bioawk one-liner:
bioawk -cfastx '{ if (length($seq) == length($qual) ){printf("@%s %s\n%s\n+\n%s\n", $name, $comment, $seq, $qual)} }' my_fastq.gz

This strips anything in the "fastq comment field", +, but that almost never has anything there anyway.
Append | gzip > filtered.fastq.gz to the above command to gzip the output.

> **Comments:**
> **username_3**: This doesn't seem to work for me (Arch Linux, bioawk version 20110810). Even after correcting the syntax error (corrected in your answer), it seems to exit after the first record it finds where the lengths are not the same. Can you confirm it works on your machine?  
> **username_6**: Thanks for the edit! Yep, I just confirmed that it works for me on all records. I'm using Ubuntu 16.04. `>>bioawk --version` returns `awk version 20110810`. It looks like the last time that I compiled bioawk was from the github repo in 2016.  
> **username_3**: That's really weird. I have the same version. Can you try a case where the record with different lengths is the second one in the file? Does it not just print the first record there?  
> **username_6**: I got the same error as you - I made the second entry in the fastq file have a different length of seq and qual fields and `bioawk` just exited before printing that record. I wonder if this is intended behavior? I have to add that in my above comment, I just tried the bioawk one-liner on a normal fastq file without any length mismatches. Oops!  
> **username_3**: I don't see how it can be intended. Sounds like a bug in bioawk.  

---

