<p>Another library you can use for this purpose is the <a href="https://github.com/samtools/htsjdk" rel="nofollow noreferrer">htsjdk</a>, which is written in java. Using htsjdk with java is analogous to using htslib with C; the BAM format is already handled by the library and you can manipulate fields in your own code. The same basic analysis applies to java as C; you do not need to convert the BAM file to text and then parse it. Generally, a program implemented in java will be slower than C but still significantly faster than scripting. You do get language features of java: portability, array bounds checking, garbage collection, etc. </p>

CC BY-SA 3.0
2017-08-25T14:27:47.847Z
2017-08-25T14:27:47.847Z
mattm
<p>I modified your original question: as you are extracting 4 fields, you are not outputting BAM. The answer to the modified question is: yes, you can write a C program with htslib (or with bamtools, bioD, bioGo or rust-bio). Formatting an entire SAM is fairly expensive. You can see this by comparing <code>samtools view aln.bam &gt; /dev/null</code> and <code>samtools view -u aln.bam &gt; /dev/null</code>. With a C program, you can select fields to output. This will give you a noticeable performance boost, depending on the fields you extract.</p>

<p>In general, if you really care about performance, don't use script. The time you save from optimizing <code>samtools view|cut -f</code> will be quickly neutralized by the inefficiency of most scripting languages.</p>

CC BY-SA 3.0
2017-08-29T10:05:14.110Z
> With a C program, you can select fields to output.

Can you detail a bit more how this would be done? The BAM format is in binary, so one would have to convert to text first and then extract the fields desired?
ShanZhengYang
CC BY-SA 3.0
2017-08-29T15:36:51.683Z
@ShanZhengYang The library parses the file record for you and places it in memory. From there, you print (or otherwise process) from the in-memory data only your desired fields in whatever format you want. There is no text parsing required.
mattm
CC BY-SA 3.0
2017-08-25T12:17:12.407Z
2017-08-25T12:17:12.407Z
user172818
<p>The BAM file format is not a text-based format. It has a specific binary structure, specified in reasonable detail in the <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="noreferrer">SAM file format specification</a>. Whenever this information is displayed on a screen as text, it needs to be converted from the binary format to a text format, which takes a bit of time and processing power.</p>

<p>As this question suggests, if only a specific field from alignments is needed (or fields), then it will probably be better to extract just those fields and do any necessary conversion only on those fields. While this <em>can</em> be done by writing a BAM parser from scratch, many developers have already written software libraries to process BAM files in this way.</p>

<p>Devon Ryan has suggested <a href="http://www.htslib.org/" rel="noreferrer">htslib</a>, which is a C library written by a group that includes the people who wrote the SAM/BAM/CRAM file format specifications. There's also <a href="https://github.com/pysam-developers/pysam" rel="noreferrer">pysam</a>, which is a python wrapper around htslib.</p>

<p>The particular tool that is used will depend on your familiarity with programming and the specific thing that you want to do. If you just want to "output a BAM file", then cat is the quickest:</p>

<pre><code>cat file1.bam
</code></pre>

<p>... but you probably don't want to do that, because it seems like you want to process a text-based representation with a script. Because you haven't specified which fields you are interested in, it's not possible to suggest the best thing to use. In the end, I expect that an "efficient" solution to your problem would involve <code>htslib</code> in some form. This is not really hacking <em>samtools</em>, it's using the backend of samtools to process BAM data.</p>

<p>However, it's useful to also think about the time cost of coding. What application do you have which means that the text-based processing method is not sufficient? It takes time to write code, and a lot more time to debug that code to make sure it's doing the right thing in all situations. If this is for a one-off thing, then <code>samtools view</code> output fed into your script may be the quickest solution.</p>

CC BY-SA 3.0
2017-08-23T04:02:43.557Z
I'm a bit familiar with pysam. It seems like this would be very inefficient, as I would need to input the bam into a python data structure, e.g. ` pysam.AlignmentFile()`. As for the script, yes, I currently am parsing the text-format of the outputted bam file. I have done nothing with compression/binary yet.
ShanZhengYang
CC BY-SA 3.0
2017-08-23T21:49:41.357Z
Whether or not it is inefficient depends on the way the library is implemented. It's possible for data structures to contain pointers to files (or places within files) and only retrieve information from the files when requested. There's no need to load the entire file into memory to explore just one aspect of it.
gringer
CC BY-SA 3.0
2017-08-23T03:14:30.797Z
2017-08-23T03:14:30.797Z
gringer
<p>I am constructing a bit of software which pipes the outputs of the bam file via <code>samtools view</code> into a script for parsing. My goal is to (somehow) make this process more efficient, and faster than <code>samtools view</code>. </p>

<p>I am only using 3-4 fields in the bam. Naturally, if I only wanted those fields, I could do something like <code>samtools view file1.bam | cut -f ##</code>. </p>

<p>However, I'm trying to be faster than this approach. More specifically: </p>

<p>(1) Given the bam file format, is it theoretically possible to output several requested SAM fields faster than <code>samtools view|cut -f</code>? </p>

<p>(2) Is there software available to achieve (1), or would one need to hack samtools? </p>

CC BY-SA 3.0
2017-08-22T21:41:36.073Z
Which fields do you need exactly? Using `htslib` is relatively easy if you know C, but if you can get by with something like [sambamba](http://lomereiter.github.io/sambamba/) then that'd be vastly easier still.
Devon Ryan
CC BY-SA 3.0
2017-08-22T22:22:07.930Z
@DevonRyan Outputting the bam could be faster with sambambda? These are "mandatory" fields, so they're present in each bam "row"
ShanZhengYang
CC BY-SA 3.0
2017-08-22T22:39:07.903Z
`samtools view`, especially in the way it is used here, produces a text representation of a BAM file. Using a library (or tool that taps into a library) allows faster access at the binary level by avoiding full text conversion of each alignment prior to filtering.
gringer
CC BY-SA 3.0
2017-08-22T22:52:41.050Z
@gringer I'm not following this exactly. Could you provide more details/an explanation?
ShanZhengYang
CC BY-SA 3.0
2017-08-23T04:53:07.897Z
Sambamba is generally faster than samtools, or at least that used to be the case. Samtools is a reference implementation, it doesn't always have to be the fastest.
Devon Ryan
<sam><samtools>
Is there a way to retrieve several SAM fields faster than `samtools view | cut -f`?
# Is there a way to retrieve several SAM fields faster than `samtools view | cut -f`?

(Asked by: username_0 on 2017-08-22)

I am constructing a bit of software which pipes the outputs of the bam file via `samtools view` into a script for parsing. My goal is to (somehow) make this process more efficient, and faster than `samtools view`.

I am only using 3-4 fields in the bam. Naturally, if I only wanted those fields, I could do something like `samtools view file1.bam | cut -f ##`.

However, I'm trying to be faster than this approach. More specifically:

(1) Given the bam file format, is it theoretically possible to output several requested SAM fields faster than `samtools view|cut -f`?

(2) Is there software available to achieve (1), or would one need to hack samtools?

**Tags:** sam, samtools
> **Comments:**
> **username_1**: Which fields do you need exactly? Using `htslib` is relatively easy if you know C, but if you can get by with something like [sambamba](http://lomereiter.github.io/sambamba/) then that'd be vastly easier still.  
> **username_0**: @DevonRyan Outputting the bam could be faster with sambambda? These are "mandatory" fields, so they're present in each bam "row"  
> **username_2**: `samtools view`, especially in the way it is used here, produces a text representation of a BAM file. Using a library (or tool that taps into a library) allows faster access at the binary level by avoiding full text conversion of each alignment prior to filtering.  
> **username_0**: @gringer I'm not following this exactly. Could you provide more details/an explanation?  
> **username_1**: Sambamba is generally faster than samtools, or at least that used to be the case. Samtools is a reference implementation, it doesn't always have to be the fastest.  

---

## 3 answers

### Answer by username_2 on 2017-08-23. Score: 6 | Accepted answer

The BAM file format is not a text-based format. It has a specific binary structure, specified in reasonable detail in the SAM file format specification. Whenever this information is displayed on a screen as text, it needs to be converted from the binary format to a text format, which takes a bit of time and processing power.
As this question suggests, if only a specific field from alignments is needed (or fields), then it will probably be better to extract just those fields and do any necessary conversion only on those fields. While this can be done by writing a BAM parser from scratch, many developers have already written software libraries to process BAM files in this way.
Devon Ryan has suggested htslib, which is a C library written by a group that includes the people who wrote the SAM/BAM/CRAM file format specifications. There's also pysam, which is a python wrapper around htslib.
The particular tool that is used will depend on your familiarity with programming and the specific thing that you want to do. If you just want to "output a BAM file", then cat is the quickest:
cat file1.bam

... but you probably don't want to do that, because it seems like you want to process a text-based representation with a script. Because you haven't specified which fields you are interested in, it's not possible to suggest the best thing to use. In the end, I expect that an "efficient" solution to your problem would involve htslib in some form. This is not really hacking samtools, it's using the backend of samtools to process BAM data.
However, it's useful to also think about the time cost of coding. What application do you have which means that the text-based processing method is not sufficient? It takes time to write code, and a lot more time to debug that code to make sure it's doing the right thing in all situations. If this is for a one-off thing, then samtools view output fed into your script may be the quickest solution.

> **Comments:**
> **username_0**: I'm a bit familiar with pysam. It seems like this would be very inefficient, as I would need to input the bam into a python data structure, e.g. ` pysam.AlignmentFile()`. As for the script, yes, I currently am parsing the text-format of the outputted bam file. I have done nothing with compression/binary yet.  
> **username_2**: Whether or not it is inefficient depends on the way the library is implemented. It's possible for data structures to contain pointers to files (or places within files) and only retrieve information from the files when requested. There's no need to load the entire file into memory to explore just one aspect of it.  

---

### Answer by username_4 on 2017-08-25. Score: 6

I modified your original question: as you are extracting 4 fields, you are not outputting BAM. The answer to the modified question is: yes, you can write a C program with htslib (or with bamtools, bioD, bioGo or rust-bio). Formatting an entire SAM is fairly expensive. You can see this by comparing `samtools view aln.bam > /dev/null` and `samtools view -u aln.bam > /dev/null`. With a C program, you can select fields to output. This will give you a noticeable performance boost, depending on the fields you extract.

In general, if you really care about performance, don't use script. The time you save from optimizing `samtools view|cut -f` will be quickly neutralized by the inefficiency of most scripting languages.

> **Comments:**
> **username_0**: > With a C program, you can select fields to output.  Can you detail a bit more how this would be done? The BAM format is in binary, so one would have to convert to text first and then extract the fields desired?  
> **username_3**: @ShanZhengYang The library parses the file record for you and places it in memory. From there, you print (or otherwise process) from the in-memory data only your desired fields in whatever format you want. There is no text parsing required.  

---

### Answer by username_3 on 2017-08-25. Score: 3

Another library you can use for this purpose is the [htsjdk](https://github.com/samtools/htsjdk), which is written in java. Using htsjdk with java is analogous to using htslib with C; the BAM format is already handled by the library and you can manipulate fields in your own code. The same basic analysis applies to java as C; you do not need to convert the BAM file to text and then parse it. Generally, a program implemented in java will be slower than C but still significantly faster than scripting. You do get language features of java: portability, array bounds checking, garbage collection, etc.


---

