<p>Why make this so complicated with the tee command?</p>
<p>I don't think you even can index an unsorted bam, so you have to wait until sorting is done before you index.  Indexing is really short compared to mapping and sorting, you aren't saving much by trying to do both at once.</p>

CC BY-SA 4.0
2020-10-22T23:17:17.553Z
2020-10-22T23:17:17.553Z
swbarnes2
<p>This is an error from <code>bwa</code> saying that it can't find your input files:</p>
<p>Here's a checklist:</p>
<ol>
<li>Check that you have no whitespace after your <code>\</code> newline breaks. I.e. <code>\</code> is the <em>very last character</em></li>
<li>Check that you have no whitespace in any path names</li>
<li><em>Quote your variables</em>: <code>$REF_PATH</code> -&gt; <code>&quot;$REF_PATH&quot;</code></li>
<li>Use <code>set -x</code> at the top of your bash script to debug.</li>
<li>In general, read <a href="https://kvz.io/bash-best-practices.html" rel="noreferrer">this</a></li>
</ol>

CC BY-SA 4.0
2020-10-22T08:54:09.107Z
2020-10-22T08:54:09.107Z
Bastian Schiffthaler
<p>I'm trying to use the code specified in <a href="https://biology.stackexchange.com/questions/59493/how-to-convert-bwa-mem-output-to-bam-format-without-saving-sam-file?fbclid=IwAR1oClXdwhtsqvQqkmmc8a1af-dTaALbtkQ2kKW3cGnIrLZCaa_6HxjvrfY">this link</a> to create a .bam file without the need for a .sam file.</p>
<p>Here is the code I'm using:</p>
<pre><code>bwa mem -R &quot;@RG\tID:<span class="math-container">$SAMPLE\tPL:illumina\tSM:$</span>SAMPLE&quot; \
    <span class="math-container">$REF_PATH/hg38.fasta \
    mapping/R1.fastq.gz \
    mapping/R2.fastq.gz | \
    samtools sort - | \
    tee mapping/$</span>SAMPLE\.bam | \
    samtools index - mapping/$SAMPLE\.bam.bai \
    2&gt;&gt; log
</code></pre>
<p>and I get the following error:</p>
<pre><code>[M::bwa_idx_load_from_disk] read 0 ALT contigs
`[E::main_mem] fail to open file ` '.
</code></pre>
<p>The ref path and file locations work when creating and converting a .sam to a .bam file.
Any ideas?</p>

CC BY-SA 4.0
2020-10-22T10:40:29.120Z
The problem is not in skipping the intermediate `.sam` format, but already the mapping step.
Kamil S Jaron
CC BY-SA 4.0
2020-10-22T11:08:45.163Z
That link does not contain that code — is there a different link it does come from? (Whatever tutorial it's from would benefit from an update. Writing the BAM file via `tee` so that you can pipe to `samtools index` is such a spectacular anti-pattern. Though it's unfortunate that `sort` is one of the subcommands that does not yet support samtools's new `--write-index` option.)
John Marshall
CC BY-SA 4.0
2020-10-22T12:12:30.337Z
Ha, I was wrong, the index is indeed loaded. The reads are not...  The expected log should look like this:
`mapping
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 1006360 sequences (150000291 bp)...
...`
Then I would follow @Bastian Schiffthaler's answer to find what's the problem with your reads.
Kamil S Jaron
CC BY-SA 4.0
2020-11-19T09:31:05.820Z
It turns out [`samtools sort` does support `--write-index`](https://github.com/samtools/samtools/pull/1342) since 1.10 after all. So this can be written more simply as `bwa … | samtools sort --write-index -o mapping/$SAMPLE.bam -` (plus a little more if you want to use a BAI index rather than `‑‑write-index`'s default CSI).
John Marshall
<samtools><bash><bwa>
Trying to create a .bam file without the need for a .sam file
# Trying to create a .bam file without the need for a .sam file

(Asked by: username_0 on 2020-10-22)

I'm trying to use the code specified in this link to create a .bam file without the need for a .sam file.
Here is the code I'm using:
bwa mem -R "@RG\tID:$SAMPLE\tPL:illumina\tSM:$SAMPLE" \
    $REF_PATH/hg38.fasta \
    mapping/R1.fastq.gz \
    mapping/R2.fastq.gz | \
    samtools sort - | \
    tee mapping/$SAMPLE\.bam | \
    samtools index - mapping/$SAMPLE\.bam.bai \
    2>> log

and I get the following error:
[M::bwa_idx_load_from_disk] read 0 ALT contigs
`[E::main_mem] fail to open file ` '.

The ref path and file locations work when creating and converting a .sam to a .bam file.
Any ideas?

**Tags:** samtools, bash, bwa
> **Comments:**
> **username_1**: The problem is not in skipping the intermediate `.sam` format, but already the mapping step.  
> **username_2**: That link does not contain that code — is there a different link it does come from? (Whatever tutorial it's from would benefit from an update. Writing the BAM file via `tee` so that you can pipe to `samtools index` is such a spectacular anti-pattern. Though it's unfortunate that `sort` is one of the subcommands that does not yet support samtools's new `--write-index` option.)  
> **username_1**: Ha, I was wrong, the index is indeed loaded. The reads are not...  The expected log should look like this: `mapping [M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 1006360 sequences (150000291 bp)... ...` Then I would follow @Bastian Schiffthaler's answer to find what's the problem with your reads.  
> **username_2**: It turns out [`samtools sort` does support `--write-index`](https://github.com/samtools/samtools/pull/1342) since 1.10 after all. So this can be written more simply as `bwa … | samtools sort --write-index -o mapping/$SAMPLE.bam -` (plus a little more if you want to use a BAI index rather than `‑‑write-index`'s default CSI).  

---

## 2 answers

### Answer by username_4 on 2020-10-22. Score: 5 | Accepted answer

This is an error from `bwa` saying that it can't find your input files:

Here's a checklist:

1. Check that you have no whitespace after your `\` newline breaks. I.e. `\` is the *very last character*
2. Check that you have no whitespace in any path names
3. *Quote your variables*: `$REF_PATH` -> `"$REF_PATH"`
4. Use `set -x` at the top of your bash script to debug.
5. In general, read [this](https://kvz.io/bash-best-practices.html)


---

### Answer by username_3 on 2020-10-22. Score: 0

Why make this so complicated with the tee command?

I don't think you even can index an unsorted bam, so you have to wait until sorting is done before you index. Indexing is really short compared to mapping and sorting, you aren't saving much by trying to do both at once.


---

