<ol>
<li>In a BAM file, all of the auxiliary tags (the optional fields you refer to) are held what is effectively a <code>void *</code> field, one after the other. Each entry within this field is structured as a two <code>char</code>s storing the tag name (e.g., <code>NM</code>, followed by a <code>char</code> storing the type (e.g., <code>Z</code> or <code>i</code>), followed by the data in whatever format is appropriate (<code>char</code>, <code>float</code>, etc). Strings in the aux tags are <code>NULL</code> terminated, so their lengths aren't explicitly stored.</li>
<li>To access a given field, htslib has to iterate over all of them, in whatever order they happen to be stored, until it finds the correct one. The output order with <code>samtools view</code> is the order in which they're stored.</li>
<li>The size of <code>*data</code> is stored as <code>l_data</code>, so once it hits that then it knows that the record is finished.</li>
</ol>

<p>Regarding point 1, this is a standard C trick of casting between data types. So, while <code>*data</code> is actually a <code>uint8_t *</code> (aka, a byte array), the first two bytes of each tag are <code>char</code>s and whatever is after them (well, after the byte specifying the type) is then cast to the appropriate type (be it <code>float</code> or <code>char*</code> or what not). This is all probably clearer if you <a href="https://github.com/samtools/htslib/blob/develop/sam.c#L1727-L1758" rel="nofollow noreferrer">look at the code</a>.</p>

CC BY-SA 3.0
2018-04-03T17:39:39.773Z
Thanks for the help; this was very useful. Could you detail #1 a bit more? That is, how the optional fields are stored with a generic pointer type?
EB2127
CC BY-SA 3.0
2018-04-03T18:05:34.433Z
@EB2127 I've added a bit more detail as requested. It's best to just look at the source code at this point though. Ignore the stuff on the `B` type, since that will likely just be confusing at first.
Devon Ryan
CC BY-SA 3.0
2018-04-03T09:42:05.730Z
2018-04-03T18:04:13.697Z
2018-04-03T18:04:13.697Z
Devon Ryan
<blockquote>
  <p>How does htslib access each of the optional fields, given this structure?</p>
</blockquote>

<p>htslib uses a macro, <code>bam_get_aux</code>, for this purpose. You can check the source code of this macro — it essentially just advances the data pointer of the current record to the start of the optional data, which is after all other data. But you shouldn’t care about this, and just use the relevant exported functions for reading optional (“auxiliary”) data:</p>

<ul>
<li><code>bam_aux_get</code> returns a pointer to an optional record, given by its two-character tag name.</li>
<li><code>bam_aux2‹C›</code> converts the pointer returned by the previous function to the actual type of the optional tag; here, <code>‹C›</code> is a type identifier: <code>i</code> for an integer, <code>Z</code> for a zero-terminated character string, etc. Refer to the documentation and the <code>htslib/sam.h</code> file for a complete list.</li>
</ul>

<blockquote>
  <p>Does it keep the order of the optional fields, or is this information unavailable?</p>
</blockquote>

<p>As long as you don’t modify optional fields, the order is kept, but unspecified. <em>Do not rely on the order</em>!</p>

<blockquote>
  <p>When a user runs htslib or even <code>samtools view</code>, how does the BAM row "end"</p>
</blockquote>

<p>BAM records store a header that contains the record size.</p>

CC BY-SA 3.0
2018-04-03T17:23:50.667Z
@EB2127 No, it won’t necessarily be uniform. And if there are algorithms that depend on the order, then these algorithms are broken because they are breaking the BAM specification. As such, there *is no way* to set the order. If you need ordered tags (but what for?!) then you need to store the order inside the tag data itself (e.g. by prefixing the tag data with ordinals).
Konrad Rudolph
CC BY-SA 3.0
2018-04-03T17:30:01.373Z
Thanks, I appreciate the help. Another question: "You can check the source code of this macro — it essentially just advances the data pointer of the current record to the start of the optional data, which is after all other data.". Could you flesh out the details for this? This may help me better understand how these macros work. e.g. `bam_get_qname()`  looks like it's accessing the 'data' member of  struct bam1_t, which is of the form "qname-cigar-seq-qual-aux". Then 'bam_get_cigar()' is advancing the data pointer of the current record to the length of the query name, `core.l_qname`?
EB2127
CC BY-SA 3.0
2018-04-03T10:53:37.280Z
2018-04-03T10:53:37.280Z
Konrad Rudolph
<p>I am confused regarding how various software libraries deal with optional fields in a BAM:</p>

<p>Based upon the <a href="https://github.com/samtools/hts-specs" rel="noreferrer">BAM specification</a>, there are 11 mandatory fields to a BAM:</p>

<pre><code>QNAME, FLAG, RNAME, POS, MAPQ, CIGAR, RNEXT, PNEXT, TLEN, SEQ, QUAL
</code></pre>

<p>The BAM files of today normally have many optional fields as well, which must be of the format <code>TAG:TYPE:VALUE</code>, whereby <code>TAG</code> is a two-character string that matches <code>/[A-Za-z][A-Za-z0-9]/</code> and <code>TYPE</code> is a single case-sensitive letter which defines the format of <code>VALUE</code>. Each <code>TAG</code> can only appear once in one alignment line.  </p>

<p>Certain BAMs may have no optional fields, while others have several. </p>

<p>My understanding is that optional tags may be in a different order from row to row, and then sometimes they simply do not exist. </p>

<p>My questions:</p>

<ol>
<li><p>How does <a href="https://github.com/samtools/htslib" rel="noreferrer">htslib</a>/<a href="https://github.com/samtools/samtools" rel="noreferrer">samtools</a> (e.g. <code>samtools view</code>) access these optional fields, which differ in number from BAM to BAM (possible row to row)?</p>

<p>In <code>htslib/sam.h</code>, a BAM alignment struct is defined as follows:</p>

<pre><code>/*! @typedef
 @abstract Structure for one alignment.
 @field  core       core information about the alignment
 @field  l_data     current length of bam1_t::data
 @field  m_data     maximum length of bam1_t::data
 @field  data       all variable-length data, concatenated; structure: qname-cigar-seq-qual-aux
typedef struct {
    bam1_core_t core;
    int l_data;
    uint32_t m_data;
    uint8_t *data;
#ifndef BAM_NO_ID
    uint64_t id;
#endif
} bam1_t;
</code></pre>

<p>and followed by several methods which access mandatory BAM fields, e.g. </p>

<pre><code>/*! @function
 @abstract  Get the name of the query
 @param  b  pointer to an alignment
 @return    pointer to the name string, null terminated
 */
#define bam_get_qname(b) ((char*)(b)-&gt;data)
</code></pre></li>
<li><p>How does htslib access each of the optional fields, given this <code>structure</code>? Does it keep the order of the optional fields, or is this information unavailable? </p></li>
<li><p>When a user runs htslib or even <code>samtools view</code>, how does the BAM row "end", i.e. what data structure exists to show that the BAM row has no more optional fields, and that the algorithm should "move on" to the next BAM row. </p></li>
</ol>

<p>Any clarification for how the source code works in this case is much appreciated. </p>

CC BY-SA 3.0
2018-04-03T10:52:18.890Z
[Cross-posted on Reddit](https://www.reddit.com/r/bioinformatics/comments/89946w/how_does_htslibsamtools_access_optional_bam_fields)
Konrad Rudolph
<sam><samtools><algorithms><htslib>
How does htslib/samtools access optional BAM fields?
# How does htslib/samtools access optional BAM fields?

(Asked by: username_0 on 2018-04-03)

I am confused regarding how various software libraries deal with optional fields in a BAM:
Based upon the BAM specification, there are 11 mandatory fields to a BAM:
QNAME, FLAG, RNAME, POS, MAPQ, CIGAR, RNEXT, PNEXT, TLEN, SEQ, QUAL

The BAM files of today normally have many optional fields as well, which must be of the format TAG:TYPE:VALUE, whereby TAG is a two-character string that matches /[A-Za-z][A-Za-z0-9]/ and TYPE is a single case-sensitive letter which defines the format of VALUE. Each TAG can only appear once in one alignment line.  
Certain BAMs may have no optional fields, while others have several. 
My understanding is that optional tags may be in a different order from row to row, and then sometimes they simply do not exist. 
My questions:

How does htslib/samtools (e.g. samtools view) access these optional fields, which differ in number from BAM to BAM (possible row to row)?
In htslib/sam.h, a BAM alignment struct is defined as follows:
/*! @typedef
 @abstract Structure for one alignment.
 @field  core       core information about the alignment
 @field  l_data     current length of bam1_t::data
 @field  m_data     maximum length of bam1_t::data
 @field  data       all variable-length data, concatenated; structure: qname-cigar-seq-qual-aux
typedef struct {
    bam1_core_t core;
    int l_data;
    uint32_t m_data;
    uint8_t *data;
#ifndef BAM_NO_ID
    uint64_t id;
#endif
} bam1_t;

and followed by several methods which access mandatory BAM fields, e.g. 
/*! @function
 @abstract  Get the name of the query
 @param  b  pointer to an alignment
 @return    pointer to the name string, null terminated
 */
#define bam_get_qname(b) ((char*)(b)->data)

How does htslib access each of the optional fields, given this structure? Does it keep the order of the optional fields, or is this information unavailable? 
When a user runs htslib or even samtools view, how does the BAM row "end", i.e. what data structure exists to show that the BAM row has no more optional fields, and that the algorithm should "move on" to the next BAM row. 

Any clarification for how the source code works in this case is much appreciated.

**Tags:** sam, samtools, algorithms, htslib
> **Comments:**
> **username_1**: [Cross-posted on Reddit](https://www.reddit.com/r/bioinformatics/comments/89946w/how_does_htslibsamtools_access_optional_bam_fields)  

---

## 2 answers

### Answer by username_1 on 2018-04-03. Score: 4 | Accepted answer

> How does htslib access each of the optional fields, given this structure?

htslib uses a macro, `bam_get_aux`, for this purpose. You can check the source code of this macro — it essentially just advances the data pointer of the current record to the start of the optional data, which is after all other data. But you shouldn’t care about this, and just use the relevant exported functions for reading optional (“auxiliary”) data:

* `bam_aux_get` returns a pointer to an optional record, given by its two-character tag name.
* `bam_aux2‹C›` converts the pointer returned by the previous function to the actual type of the optional tag; here, `‹C›` is a type identifier: `i` for an integer, `Z` for a zero-terminated character string, etc. Refer to the documentation and the `htslib/sam.h` file for a complete list.

> Does it keep the order of the optional fields, or is this information unavailable?

As long as you don’t modify optional fields, the order is kept, but unspecified. *Do not rely on the order*!

> When a user runs htslib or even `samtools view`, how does the BAM row "end"

BAM records store a header that contains the record size.

> **Comments:**
> **username_1**: @EB2127 No, it won’t necessarily be uniform. And if there are algorithms that depend on the order, then these algorithms are broken because they are breaking the BAM specification. As such, there *is no way* to set the order. If you need ordered tags (but what for?!) then you need to store the order inside the tag data itself (e.g. by prefixing the tag data with ordinals).  
> **username_0**: Thanks, I appreciate the help. Another question: "You can check the source code of this macro — it essentially just advances the data pointer of the current record to the start of the optional data, which is after all other data.". Could you flesh out the details for this? This may help me better understand how these macros work. e.g. `bam_get_qname()`  looks like it's accessing the 'data' member of  struct bam1_t, which is of the form "qname-cigar-seq-qual-aux". Then 'bam_get_cigar()' is advancing the data pointer of the current record to the length of the query name, `core.l_qname`?  

---

### Answer by username_2 on 2018-04-03. Score: 3

1. In a BAM file, all of the auxiliary tags (the optional fields you refer to) are held what is effectively a `void *` field, one after the other. Each entry within this field is structured as a two `char`s storing the tag name (e.g., `NM`, followed by a `char` storing the type (e.g., `Z` or `i`), followed by the data in whatever format is appropriate (`char`, `float`, etc). Strings in the aux tags are `NULL` terminated, so their lengths aren't explicitly stored.
2. To access a given field, htslib has to iterate over all of them, in whatever order they happen to be stored, until it finds the correct one. The output order with `samtools view` is the order in which they're stored.
3. The size of `*data` is stored as `l_data`, so once it hits that then it knows that the record is finished.

Regarding point 1, this is a standard C trick of casting between data types. So, while `*data` is actually a `uint8_t *` (aka, a byte array), the first two bytes of each tag are `char`s and whatever is after them (well, after the byte specifying the type) is then cast to the appropriate type (be it `float` or `char*` or what not). This is all probably clearer if you [look at the code](https://github.com/samtools/htslib/blob/develop/sam.c#L1727-L1758).

> **Comments:**
> **username_0**: Thanks for the help; this was very useful. Could you detail #1 a bit more? That is, how the optional fields are stored with a generic pointer type?  
> **username_2**: @EB2127 I've added a bit more detail as requested. It's best to just look at the source code at this point though. Ignore the stuff on the `B` type, since that will likely just be confusing at first.  

---

