<p>Ok, I wrote a bit brute pysam / BioPython parser that uses index of bam to get proper order of read pair for R1 / R2 files and bitwise flag. It should not be too difficult to add more sophisticated filtering rules now.</p>

<pre><code>#!/usr/bin/env python3
# 1. arg - indexed bam file
# 2. arg - list of headers to filter
# 3. arg - name pattern for the output reads

import os
import sys
import pysam
from Bio import SeqIO, Seq, SeqRecord

samfile = pysam.AlignmentFile(sys.argv[1], "rb")
header_set = set(line.strip() for line in open(sys.argv[2]))
base = sys.argv[3]

out_R1 = base + 'R1_filtered.fq'
out_R2 = base + 'R2_filtered.fq'

with open(out_R1, mode='w') as R1, open(out_R2, mode='w') as R2:
    for entry in samfile:
        if entry.is_read1 and not entry.reference_name in header_set:
            # pait pair
            entry_R2 = samfile.mate(entry)
            # do some sequence gymnastics with R1
            seq_R1 = Seq.Seq(entry.seq)
            if entry.is_reverse :
                seq_R1 = seq_R1.reverse_complement()
            # do some sequence gymnastics with R2
            seq_R2 = Seq.Seq(entry_R2.seq)
            if entry_R2.is_reverse :
                seq_R2 = seq_R2.reverse_complement()
            R1.write('@' + entry.qname + '\n' + str(seq_R1) + '\n+\n' + entry.qqual + '\n')
            R2.write('@' + entry_R2.qname + '\n' + str(seq_R2) + '\n+\n' + entry_R2.qqual + '\n')


samfile.close()
</code></pre>

<p>There are some ugly bits, please feel free to make it nicer.</p>

CC BY-SA 3.0
2017-07-26T13:17:23.220Z
2017-07-26T13:17:23.220Z
Kamil S Jaron
<p>I'm not aware of any pre-made program to do this, so I <a href="https://github.com/dpryan79/Answers/tree/master/bioinfoSE_2149" rel="noreferrer">wrote one for you</a>. This will take a BAM file with any ordering and produce properly ordered gzipped fastq files with the filtering as you requested. Internally, this iterates over all of the entries in the BAM file (ignoring secondary/supplemental entries and those where both mates map to your filter list), store the properly oriented sequence/quality/read name in a buffer, and then dumps that buffer entry to disk once the mate is found. This should be reasonably performant (hey, it's python, to don't expect too much), though if you happen to have indexed BAM files then one could think of ways to make this run faster.</p>

<p>Do check the output, since I've only run one test.</p>

CC BY-SA 3.0
2017-07-26T13:21:35.363Z
You were 3 minutes faster and your code looks way nicer. I will try it out... Thanks a lot.
Kamil S Jaron
CC BY-SA 3.0
2017-07-26T13:29:10.907Z
The only real benefit to my code is that it avoids jumping around in the BAM file looking for mates and that the BAM file doesn't have to be sorted. If you DO have sorted files, then I think it'd be faster to make a list of contigs that aren't excluded, add `*` to that, and then iterate over it as in my program. That'd save more time.
Devon Ryan
CC BY-SA 3.0
2017-07-26T13:13:32.230Z
2017-07-26T13:13:32.230Z
Devon Ryan
<h3>Question</h3>

<p>How can I extract reads from a <code>bam</code> file (produced by <code>bwa-mem</code>) to <code>fastq</code> given a list of reference sequences to filter out?</p>

<h3>Potential difficulties</h3>

<ul>
<li>maintaining FR orientation of pair end reads (in <code>bam</code> all the sequences are reference sequences)</li>
<li>keeping R1 and R2 reads </li>
<li>keeping quality scores in the same encoding as original fastq (default illumina phred scores in my case)</li>
<li>bam can be (ana usually is) sorted by coordinates</li>
</ul>

<h3>Almost there solutions</h3>

<ol>
<li><p><a href="https://github.com/blaxterlab/blobology/blob/master/bowtie2_extract_reads_mapped_to_specific_contigs.pl" rel="noreferrer">Sujai's perl solution</a> in <a href="https://github.com/blaxterlab/blobology" rel="noreferrer">blobology</a> does exact opposite - getting reads from list of references (so I could just reverse the list). The disadvantage is that the script outputs an interleaved <code>fq</code> file; requires unique names of mates, otherwise R1/R2 information is lost.</p></li>
<li><p>samtools + grep them all from fastq files</p></li>
</ol>

<p>create a list of read names that do not map to filtered scaffolds. (cut will extract just read names, uniq will collapse pair end read names if they are the same). Then grep read names from fastq files and remove -- separator between hits</p>

<pre><code>samtools view foo.bam | grep -vf list_of_scaffols_filter \
  | cut -f 1 | uniq &gt; list_of_reads_to_keep

grep -A 3 -f list_of_reads_to_keep foo_R1.fq | grep -v "^--$" &gt; foo_R1_filtered_bash.fq
grep -A 3 -f list_of_reads_to_keep foo_R2.fq | grep -v "^--$" &gt; foo_R1_filtered_bash.fq
</code></pre>

<ol start="3">
<li>filter bam &amp; picard tools</li>
</ol>

<p>Or I could do just the filtering part and use <a href="http://broadinstitute.github.io/picard/command-line-overview.html" rel="noreferrer">Picard-tools</a> (Picard.SamToFastq), but as usual I am avoiding java as much as I can. I guess</p>

<pre><code>samtools view foo.bam | grep -vf list_of_scaffols_filter \
  | java -jar picard.jar SamToFastq INPUT=/dev/stdin \
  FASTQ=foo_R1_filtered_bash.fq SECOND_END_FASTQ=foo_R2_filtered_bash.fq
</code></pre>

<hr>

<p>The first solution does not really work for me since I do not want to rename all the reads in bam file and I want to keep the R1/R2 information (since R1 and R2 have different error profiles). Both solutions 2 and 3 I find bit clumsy and I am not sure if they are general, I might get some unexpected behaviour if one of reads is mapped second is not... They both relay on the same filtering step.</p>

<p>I was wondering about some pysam solution. I guess it will be much slower, but at least it will be much clearer and perhaps more general. Something like in <a href="https://www.biostars.org/p/6970/" rel="noreferrer">Convert Bam File To Fasta File</a> - there is pysam solution for fasta (not fastq), almost there...</p>

<h3>Story</h3>

<p>I have very fragmented reference genome. Some of scaffolds them are too small to works with, and some of them are contaminants (identified using blobtools).
I want to separate reads that are mapping to different groups to separate contaminants, short scaffolds and scaffold that will be used for downstream analysis. The reason is that if we remap all the reads to filtered reference (0.7 - 0.8 of original genome), the most of them (0.95 - 0.99) will still find a place where they map, therefore there is 0.2 - 0.3 of misplaced reads that will obviously have to affect downstream analysis, like variant calling.</p>

<p>This filtering idea is based logic that if the filtered duplicated genomic region will contain some small differences, they will attract their reads (and if I filter them I will improve variant calling) and if they will be exactly same, they will get reads assigned at random, so there is no harm in doing that.</p>

CC BY-SA 3.0
2017-07-26T11:58:46.840Z
What do you want to do if only one read in a pair maps to a scaffold/contig that you want excluded? Honestly, I'd just name sort the BAM file and write a bit of python to do the conversion and filtering, but that's just me being lazy.
Devon Ryan
CC BY-SA 3.0
2017-07-26T12:07:04.980Z
I am not sure yet, but I guess better will be to keep all pair where at least one of the pair maps to any non-filtered scaffold.
--
In fact if R1/2 have same names, this is exactly how solution 2 is going to behave. If the names will be different I will filter different number of R1 and R2 reads.
Kamil S Jaron
<sam><fastq><format-conversion><filtering>
How to safely and efficiently convert subset of bam to fastq?
# How to safely and efficiently convert subset of bam to fastq?

(Asked by: username_0 on 2017-07-26)

Question
How can I extract reads from a bam file (produced by bwa-mem) to fastq given a list of reference sequences to filter out?
Potential difficulties

maintaining FR orientation of pair end reads (in bam all the sequences are reference sequences)
keeping R1 and R2 reads 
keeping quality scores in the same encoding as original fastq (default illumina phred scores in my case)
bam can be (ana usually is) sorted by coordinates

Almost there solutions

Sujai's perl solution in blobology does exact opposite - getting reads from list of references (so I could just reverse the list). The disadvantage is that the script outputs an interleaved fq file; requires unique names of mates, otherwise R1/R2 information is lost.
samtools + grep them all from fastq files

create a list of read names that do not map to filtered scaffolds. (cut will extract just read names, uniq will collapse pair end read names if they are the same). Then grep read names from fastq files and remove -- separator between hits
samtools view foo.bam | grep -vf list_of_scaffols_filter \
  | cut -f 1 | uniq > list_of_reads_to_keep

grep -A 3 -f list_of_reads_to_keep foo_R1.fq | grep -v "^--$" > foo_R1_filtered_bash.fq
grep -A 3 -f list_of_reads_to_keep foo_R2.fq | grep -v "^--$" > foo_R1_filtered_bash.fq


filter bam & picard tools

Or I could do just the filtering part and use Picard-tools (Picard.SamToFastq), but as usual I am avoiding java as much as I can. I guess
samtools view foo.bam | grep -vf list_of_scaffols_filter \
  | java -jar picard.jar SamToFastq INPUT=/dev/stdin \
  FASTQ=foo_R1_filtered_bash.fq SECOND_END_FASTQ=foo_R2_filtered_bash.fq


The first solution does not really work for me since I do not want to rename all the reads in bam file and I want to keep the R1/R2 information (since R1 and R2 have different error profiles). Both solutions 2 and 3 I find bit clumsy and I am not sure if they are general, I might get some unexpected behaviour if one of reads is mapped second is not... They both relay on the same filtering step.
I was wondering about some pysam solution. I guess it will be much slower, but at least it will be much clearer and perhaps more general. Something like in Convert Bam File To Fasta File - there is pysam solution for fasta (not fastq), almost there...
Story
I have very fragmented reference genome. Some of scaffolds them are too small to works with, and some of them are contaminants (identified using blobtools).
I want to separate reads that are mapping to different groups to separate contaminants, short scaffolds and scaffold that will be used for downstream analysis. The reason is that if we remap all the reads to filtered reference (0.7 - 0.8 of original genome), the most of them (0.95 - 0.99) will still find a place where they map, therefore there is 0.2 - 0.3 of misplaced reads that will obviously have to affect downstream analysis, like variant calling.
This filtering idea is based logic that if the filtered duplicated genomic region will contain some small differences, they will attract their reads (and if I filter them I will improve variant calling) and if they will be exactly same, they will get reads assigned at random, so there is no harm in doing that.

**Tags:** sam, fastq, format-conversion, filtering
> **Comments:**
> **username_1**: What do you want to do if only one read in a pair maps to a scaffold/contig that you want excluded? Honestly, I'd just name sort the BAM file and write a bit of python to do the conversion and filtering, but that's just me being lazy.  
> **username_0**: I am not sure yet, but I guess better will be to keep all pair where at least one of the pair maps to any non-filtered scaffold. -- In fact if R1/2 have same names, this is exactly how solution 2 is going to behave. If the names will be different I will filter different number of R1 and R2 reads.  

---

## 2 answers

### Answer by username_1 on 2017-07-26. Score: 5 | Accepted answer

I'm not aware of any pre-made program to do this, so I [wrote one for you](https://github.com/dpryan79/Answers/tree/master/bioinfoSE_2149). This will take a BAM file with any ordering and produce properly ordered gzipped fastq files with the filtering as you requested. Internally, this iterates over all of the entries in the BAM file (ignoring secondary/supplemental entries and those where both mates map to your filter list), store the properly oriented sequence/quality/read name in a buffer, and then dumps that buffer entry to disk once the mate is found. This should be reasonably performant (hey, it's python, to don't expect too much), though if you happen to have indexed BAM files then one could think of ways to make this run faster.

Do check the output, since I've only run one test.

> **Comments:**
> **username_0**: You were 3 minutes faster and your code looks way nicer. I will try it out... Thanks a lot.  
> **username_1**: The only real benefit to my code is that it avoids jumping around in the BAM file looking for mates and that the BAM file doesn't have to be sorted. If you DO have sorted files, then I think it'd be faster to make a list of contigs that aren't excluded, add `*` to that, and then iterate over it as in my program. That'd save more time.  

---

### Answer by username_0 on 2017-07-26. Score: 2

Ok, I wrote a bit brute pysam / BioPython parser that uses index of bam to get proper order of read pair for R1 / R2 files and bitwise flag. It should not be too difficult to add more sophisticated filtering rules now.
#!/usr/bin/env python3
# 1. arg - indexed bam file
# 2. arg - list of headers to filter
# 3. arg - name pattern for the output reads

import os
import sys
import pysam
from Bio import SeqIO, Seq, SeqRecord

samfile = pysam.AlignmentFile(sys.argv[1], "rb")
header_set = set(line.strip() for line in open(sys.argv[2]))
base = sys.argv[3]

out_R1 = base + 'R1_filtered.fq'
out_R2 = base + 'R2_filtered.fq'

with open(out_R1, mode='w') as R1, open(out_R2, mode='w') as R2:
    for entry in samfile:
        if entry.is_read1 and not entry.reference_name in header_set:
            # pait pair
            entry_R2 = samfile.mate(entry)
            # do some sequence gymnastics with R1
            seq_R1 = Seq.Seq(entry.seq)
            if entry.is_reverse :
                seq_R1 = seq_R1.reverse_complement()
            # do some sequence gymnastics with R2
            seq_R2 = Seq.Seq(entry_R2.seq)
            if entry_R2.is_reverse :
                seq_R2 = seq_R2.reverse_complement()
            R1.write('@' + entry.qname + '\n' + str(seq_R1) + '\n+\n' + entry.qqual + '\n')
            R2.write('@' + entry_R2.qname + '\n' + str(seq_R2) + '\n+\n' + entry_R2.qqual + '\n')


samfile.close()

There are some ugly bits, please feel free to make it nicer.


---

