<p>As you suggested, sambamba is faster at marking duplicates than picard (it's also multithreaded). Recent versions of samtools have a rewritten duplicate marking algorithm, though I doubt it'll be as quick as sambamba. 46GB of RAM seems excessive for marking duplicates unless you're having it store the whole file in memory.</p>

CC BY-SA 3.0
2017-12-12T21:50:50.313Z
Does sambamba with default parameters equate to Picard MarkDuplicates?
719016
CC BY-SA 3.0
2017-12-12T23:55:43.347Z
I'm not sure that any tool exactly equates (there are a lot of edge cases in duplicate marking and picard misses some things), but it's a sufficient replacement. Sambamba is also used for this purpose by a number of international consortia (I think either DEEP or IHEC, if I remember correctly).
Devon Ryan
CC BY-SA 3.0
2017-12-11T20:05:44.793Z
2017-12-11T20:05:44.793Z
Devon Ryan
<p>I am running Picard MarkDuplicates with the following parameters below. On the file described, it takes about 41.6Gb of RAM memory and about 20-25 minutes to compute (only uses 1 core AFAICS).</p>

<pre><code>java -Xmx54890m -jar picard.jar MarkDuplicates \
    I=/home/dnanexus/in/sorted_bam/TST73-86-2IC_S16_L00.bml.GRCh38.karyo.bam \
    O=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.deduplicated.bam \
    M=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.duplication_metrics \
    CREATE_INDEX=true VALIDATION_STRINGENCY=SILENT REMOVE_DUPLICATES=true
</code></pre>

<p>What equivalent free/open source software (samtools?, sambamba?) can I use to obtain the same deduplicated bam file but at a lower cost, considering the instance used here has 8 cores but only one of them is being used, and has about 68Gb of RAM out of which it's using 46Gb odd at peak.</p>

<p>I think previous versions of samtools/sambamba didn't behave the same way as Picard MarkDuplicates with the parameters above, but I would like to know which recent versions would give me the same output.</p>

CC BY-SA 3.0
2017-12-11T15:58:04.443Z
I assume you only want free (in terms of price, at least) software, right? [Sentieon](http://sentieon.com/) have a tool for this that is orders of magnitude faster (and I would guess uses less RAM but I'd have to check) but you need to pay for it.
terdon
CC BY-SA 3.0
2017-12-11T16:05:59.533Z
Correct, open source.
719016
CC BY-SA 3.0
2017-12-11T20:24:48.547Z
I done no benchmark, but I guess [samblaster](https://github.com/GregoryFaust/samblaster) is bit faster (by my subjective perception of time).
Kamil S Jaron
CC BY-SA 3.0
2017-12-12T15:21:54.107Z
You should specify a smaller "-Xmx". MarkDuplicate doesn't need that much memory, but seeing a large "-Xmx", java will use extra.
user172818
<sam><samtools><sambamba><picard>
low-memory high-speed replacement for Picard MarkDuplicates
# low-memory high-speed replacement for Picard MarkDuplicates

(Asked by: username_0 on 2017-12-11)

I am running Picard MarkDuplicates with the following parameters below. On the file described, it takes about 41.6Gb of RAM memory and about 20-25 minutes to compute (only uses 1 core AFAICS).
java -Xmx54890m -jar picard.jar MarkDuplicates \
    I=/home/dnanexus/in/sorted_bam/TST73-86-2IC_S16_L00.bml.GRCh38.karyo.bam \
    O=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.deduplicated.bam \
    M=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.duplication_metrics \
    CREATE_INDEX=true VALIDATION_STRINGENCY=SILENT REMOVE_DUPLICATES=true

What equivalent free/open source software (samtools?, sambamba?) can I use to obtain the same deduplicated bam file but at a lower cost, considering the instance used here has 8 cores but only one of them is being used, and has about 68Gb of RAM out of which it's using 46Gb odd at peak.
I think previous versions of samtools/sambamba didn't behave the same way as Picard MarkDuplicates with the parameters above, but I would like to know which recent versions would give me the same output.

**Tags:** sam, samtools, sambamba, picard
> **Comments:**
> **username_1**: I assume you only want free (in terms of price, at least) software, right? [Sentieon](http://sentieon.com/) have a tool for this that is orders of magnitude faster (and I would guess uses less RAM but I'd have to check) but you need to pay for it.  
> **username_0**: Correct, open source.  
> **username_2**: I done no benchmark, but I guess [samblaster](https://github.com/GregoryFaust/samblaster) is bit faster (by my subjective perception of time).  
> **username_3**: You should specify a smaller "-Xmx". MarkDuplicate doesn't need that much memory, but seeing a large "-Xmx", java will use extra.  

---

## 1 answers

### Answer by username_4 on 2017-12-11. Score: 2

As you suggested, sambamba is faster at marking duplicates than picard (it's also multithreaded). Recent versions of samtools have a rewritten duplicate marking algorithm, though I doubt it'll be as quick as sambamba. 46GB of RAM seems excessive for marking duplicates unless you're having it store the whole file in memory.

> **Comments:**
> **username_0**: Does sambamba with default parameters equate to Picard MarkDuplicates?  
> **username_4**: I'm not sure that any tool exactly equates (there are a lot of edge cases in duplicate marking and picard misses some things), but it's a sufficient replacement. Sambamba is also used for this purpose by a number of international consortia (I think either DEEP or IHEC, if I remember correctly).  

---

