<p>@StupidWolf's answer is correct -- that first number in the flagstat output is what you want to look at to see the number of reads marked as duplicates. I wanted to add that the number given in the Picard metrics file is in fact 6%, not 0.06%. This is due to an ambiguity that is widespread in Picard metrics files; in many places the program emits a fraction even when the column name is "percent of something". Soo Hee Lee wrote a very short GATK blog post about this way back in the day, see the archived version <a href="https://github.com/broadinstitute/gatk-docs/blob/master/blog-2012-to-2019/2016-03-18-Metrics_say_percent_%E2%80%93_Double-check_those_decimals_%E2%80%93_Fractions_everywhere.md" rel="nofollow noreferrer">here</a>.</p>

CC BY-SA 4.0
2020-04-07T13:55:54.143Z
Good to know that the percentage of duplicated reads from picard and from samtools match
BCArg
CC BY-SA 4.0
2020-04-07T02:26:59.310Z
2020-04-07T02:26:59.310Z
Geraldine_VdAuwera
<p>There are duplicates, in this line:</p>

<p>1636809 + 0 duplicates, gives 1636809/26595942 = 0.06154356</p>

<p>According to <a href="http://www.htslib.org/doc/samtools-flagstat.html" rel="noreferrer">samtools documentation for flagstat</a>:</p>

<blockquote>
  <p>Provides counts for each of 13 categories based primarily on bit flags
  in the FLAG field. Each category in the output is broken down into QC
  pass and QC fail. In the default output format, these are presented as
  "#PASS + #FAIL" followed by a description of the category.</p>
</blockquote>

<p>So you can usually take whatever is in the first column</p>

CC BY-SA 4.0
2020-03-26T12:20:40.610Z
2020-03-26T12:20:40.610Z
StupidWolf
<p>I am trying to identify the best library prep method for noninvasive prenatal test samples, to be sequenced with Illumina Novaseq.</p>

<p>One metric that I am evaluating is the number/ percentage of PCR or optical duplicates. In order to achieve that I have done the following steps:</p>

<ol>
<li>aligned the fastq reads against the reference genome with <code>bwa mem</code></li>
<li>the <code>.sam</code> output from <code>bwa mem</code> was converted to <code>bam</code>, sorted and indexed</li>
<li>the duplicates were marked with the command <code>MarkDuplicates</code> from picard</li>
</ol>

<p>Then if I call <code>samtools flagstat</code> on the sorted <code>bam</code> file which had the duplicates marked with picard I get:</p>

<pre><code>26595942 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 secondary
466 + 0 supplementary
1636809 + 0 duplicates
24969064 + 0 mapped (93.88% : N/A)
26595476 + 0 paired in sequencing
13297738 + 0 read1
13297738 + 0 read2
21102678 + 0 properly paired (79.35% : N/A)
24049752 + 0 with itself and mate mapped
918846 + 0 singletons (3.45% : N/A)
462212 + 0 with mate mapped to a different chr
182138 + 0 with mate mapped to a different chr (mapQ&gt;=5)
</code></pre>

<p>Which sounds a bit strange to me, as no duplicates were found.</p>

<p>However, the <code>MarkDuplicates</code> command also outputted a metrics file (<code>dups/dupsMetrics.txt</code>) that did identify some duplicates. This was achieved with the following command:</p>

<pre><code>java -jar picard.jar MarkDuplicates I=alignment/sample.sorted.bam O=alignment/sample_markDup.bam M=dups/dupsMetrics.txt
</code></pre>

<p>I see that the value under the column <code>PERCENT_DUPLICATION</code> in <code>dups/dupsMetrics.txt</code> is 0.065555 (given the column name, I would expect that the value is indeed 0.06%, and not 6.55%)</p>

<p>What would be the explanation between the discrepancy in the output of <code>samtools flagstat</code> and <code>MarkDuplicates</code>. Is it reasonable to expect that the value displayed by <code>samtools flagstat</code> is <strong>rounded</strong> to zero, given the low percentage which was observed by <code>MarkDuplicates</code> (0.06%)?</p>

<sam><ngs><samtools><gatk><picard>
Marking optical or PCR duplicates with picard vs. samtools flagstat
# Marking optical or PCR duplicates with picard vs. samtools flagstat

(Asked by: username_0 on 2020-03-26)

I am trying to identify the best library prep method for noninvasive prenatal test samples, to be sequenced with Illumina Novaseq.
One metric that I am evaluating is the number/ percentage of PCR or optical duplicates. In order to achieve that I have done the following steps:

aligned the fastq reads against the reference genome with bwa mem
the .sam output from bwa mem was converted to bam, sorted and indexed
the duplicates were marked with the command MarkDuplicates from picard

Then if I call samtools flagstat on the sorted bam file which had the duplicates marked with picard I get:
26595942 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 secondary
466 + 0 supplementary
1636809 + 0 duplicates
24969064 + 0 mapped (93.88% : N/A)
26595476 + 0 paired in sequencing
13297738 + 0 read1
13297738 + 0 read2
21102678 + 0 properly paired (79.35% : N/A)
24049752 + 0 with itself and mate mapped
918846 + 0 singletons (3.45% : N/A)
462212 + 0 with mate mapped to a different chr
182138 + 0 with mate mapped to a different chr (mapQ>=5)

Which sounds a bit strange to me, as no duplicates were found.
However, the MarkDuplicates command also outputted a metrics file (dups/dupsMetrics.txt) that did identify some duplicates. This was achieved with the following command:
java -jar picard.jar MarkDuplicates I=alignment/sample.sorted.bam O=alignment/sample_markDup.bam M=dups/dupsMetrics.txt

I see that the value under the column PERCENT_DUPLICATION in dups/dupsMetrics.txt is 0.065555 (given the column name, I would expect that the value is indeed 0.06%, and not 6.55%)
What would be the explanation between the discrepancy in the output of samtools flagstat and MarkDuplicates. Is it reasonable to expect that the value displayed by samtools flagstat is rounded to zero, given the low percentage which was observed by MarkDuplicates (0.06%)?

**Tags:** sam, ngs, samtools, gatk, picard

---

## 2 answers

### Answer by username_2 on 2020-03-26. Score: 5 | Accepted answer

There are duplicates, in this line:

1636809 + 0 duplicates, gives 1636809/26595942 = 0.06154356

According to [samtools documentation for flagstat](http://www.htslib.org/doc/samtools-flagstat.html):

> Provides counts for each of 13 categories based primarily on bit flags
> in the FLAG field. Each category in the output is broken down into QC
> pass and QC fail. In the default output format, these are presented as
> "#PASS + #FAIL" followed by a description of the category.

So you can usually take whatever is in the first column


---

### Answer by username_1 on 2020-04-07. Score: 2

@StupidWolf's answer is correct -- that first number in the flagstat output is what you want to look at to see the number of reads marked as duplicates. I wanted to add that the number given in the Picard metrics file is in fact 6%, not 0.06%. This is due to an ambiguity that is widespread in Picard metrics files; in many places the program emits a fraction even when the column name is "percent of something". Soo Hee Lee wrote a very short GATK blog post about this way back in the day, see the archived version [here](https://github.com/broadinstitute/gatk-docs/blob/master/blog-2012-to-2019/2016-03-18-Metrics_say_percent_%E2%80%93_Double-check_those_decimals_%E2%80%93_Fractions_everywhere.md).

> **Comments:**
> **username_0**: Good to know that the percentage of duplicated reads from picard and from samtools match  

---

