<p>It's possible you're looking at instructions for a recent version of samtools / bcftools, whereas your computer has an older version. Can you try <code>samtools mpileup -g -f ...</code> to output the pileup as BCF format instead of <code>bcftools mpileup -f ...</code>?</p>

<p>Update: it's good to know that the samtools / bcftools pipelines are at least producing consistent results, which suggests that the error is not associated with the output format on the mpileup side of the pipeline, and also good to see that you've identified that this was a memory issue.</p>

<p>The most frequent cause of memory issues I've had with sorted BAM files has been trying to figure out variants in extremely high coverage data (e.g. over 1000X), which would make sense for a bacterial isolate sequenced on a GridION. I use <a href="https://gitlab.com/gringer/bioinfscripts/blob/master/samNormalise.pl" rel="nofollow noreferrer">my own script</a> to carry out a reservoir sampling on mapped BAM format files, which reduces coverage down to a more manageable level. Here's an example command to normalise to 100X coverage where possible:</p>

<pre><code>samtools view reads.bam | ./samNormalise.pl -coverage 100 | \
   samtools view -b &gt; reads_normalised.bam
</code></pre>

CC BY-SA 4.0
2019-07-09T07:22:19.383Z
did not work unfortunately, got the following messages after calling:
`samtools mpileup -g -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz`

`[warning] samtools mpileup option `g` is functional, but deprecated. Please switch to using bcftools mpileup in future.`

`Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid`

`[mpileup] 1 samples in 1 input files`

`Failed to open -: unknown file type`
BCArg
CC BY-SA 4.0
2019-07-09T07:40:07.633Z
The VCF/BCF-outputting `mpileup` command was transferred from samtools to bcftools relatively recently. So in fact it is @gringer's suggestion that is instructions for an earlier version of samtools / bcftools.
John Marshall
CC BY-SA 4.0
2019-07-09T05:43:51.213Z
2019-07-09T14:35:02.110Z
2019-07-09T14:35:02.110Z
gringer
<p>The <code>-</code> is usually used to mean standard input when reading data. So the message would suggest that the second <code>bcftools</code> command fails to read from stdin, so it fails to read the output of the first <code>bcftools</code> command:</p>

<pre><code>$ echo "foo bar" | /ref/software/bcftools-1.8/bcftools call -mv -Oz -o calls.vcf.gz
Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
Failed to open -: unknown file type
</code></pre>

<p>So there is something wrong with either your input file or with the output produced when running the first <code>bcftools</code> command on that input file. We can't really help more unless you share the file. </p>

CC BY-SA 4.0
2019-07-09T07:53:07.547Z
I doubt there's any problem with the input files, as I can visualise the alignment on table or IGV, so I guess the `bcftoools mpileup` is not generating any output (please see my comment to @John Marshall). I guess I will report the problem on their github page.
BCArg
CC BY-SA 4.0
2019-07-09T07:56:48.707Z
@BCArg yes, precisely. If you got `Killed` in the terminal, then the process is being killed by the operating system, most likely because it is taking up too much memory. Sounds like you need to run this on a more powerful machine.
terdon
CC BY-SA 4.0
2019-07-09T08:12:59.330Z
yes mystery solved, I monitored the the performance of the command `bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp` with `htop` and the program got killed right after I reached the maximum of my RAM memory (16GB), so in fact running the command on the cluster appears to be the solution.
BCArg
CC BY-SA 4.0
2019-07-08T16:15:40.243Z
2019-07-08T16:15:40.243Z
terdon
<p>I have sequenced a bacterial genome with a GridIon from ONT. Basically what I want to check is whether or not trimming 50 bps at the beginning of the reads will improve alignment against the reference genome and ultimately the call of a consensus sequence.</p>

<p>For that what I have done so far is:</p>

<ol>
<li>concatenated all the fastqs generated from the Gridion into a single (very large) fastq file: <code>ref494_cat_all.fastq</code></li>
<li>aligned this concatenated fastq against the reference genome <code>ref_genbank.fasta</code> with <code>minimap2</code>: <code>~/Downloads/minimap2-2.17_x64-linux/minimap2 -ax map-ont ref_genbank/ref_genbank.fasta ref494_cat_all.fastq &gt; ref494_aligned_trim50.sam</code></li>
<li>Performed the general <code>samtools</code> steps to covert sam to bam, index and sort</li>
</ol>

<p>Then, in order to get the consensus sequence I first try to generate a <code>.vcf.gz</code> file, according to <a href="https://samtools.github.io/bcftools/howtos/variant-calling.html" rel="nofollow noreferrer">this tutorial</a></p>

<pre><code>bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz
</code></pre>

<p>Immediatelly after calling the command above I got:</p>

<pre><code>Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
[mpileup] 1 samples in 1 input files
</code></pre>

<p>Then, after a minute or two, I got:</p>

<pre><code>Failed to open -: unknown file type
</code></pre>

<p>I'm pretty sure that there's no issue with <code>.bam</code> file, as I could visualise the alignment with IGV or tablet.</p>

<p>Does anyone know what the issue can be?</p>

CC BY-SA 4.0
2019-07-09T07:28:17.813Z
Please capture the output of the first command and show us the first few lines (via `bcftools view` if it's not text): `bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp`
John Marshall
CC BY-SA 4.0
2019-07-09T07:49:30.853Z
I got `Killed` output to the terminal after a minute or two. Also this command generates an empty `output.tmp` file.
BCArg
CC BY-SA 4.0
2019-07-09T09:12:31.937Z
As noted below, killed due to memory exhaustion. However: you deserve an error message. Mpileup typically doesn't use that much memory and I have been unable to reproduce this on (non-ONT) data here. If you are able to, it would be interesting to try current **develop** bcftools to see if this still happens and/or to run it under the debugger and see where it crashes.
John Marshall
CC BY-SA 4.0
2019-07-09T09:17:38.730Z
The latter is quite easy: run `gdb --args bcftools mpileup -f ref_genbank/ref_genbank.fasta -o /dev/null ref494_aligned.sorted.bam`, then type `run` and wait for it to crash. Then type `backtrace`; the output from this will be instructive.
John Marshall
CC BY-SA 4.0
2019-07-09T09:58:42.097Z
Just before the program got killed I got `Program terminated with signal SIGKILL, Killed.` `The program no longer exists.`

The output of `backtrace` was `No stack`

That you cannot reproduce this error on non-ONT data, you think it can be related to the size of my input files? My bam is 14GB (coverage is about 2.000, we ran for about 48h, which we won't do routinely)
BCArg
CC BY-SA 4.0
2019-07-09T10:07:31.197Z
Oh yeah, SIGKILL. Perhaps you were hit by the OOM Killer (cf https://stackoverflow.com/questions/12288550/c-linux-binary-terminated-with-signal-sigkill-why-loaded-in-gdb) so the lack of error message maybe isn't bcftools's fault. 14G isn't a big BAM file, but the level of coverage and nature of ONT data (long reads with maybe interesting CIGARs) would be things that could make mpileup take more than the usual amount of memory.
John Marshall
CC BY-SA 4.0
2019-07-09T13:42:36.107Z
Just tried to run the original command (as in my question) on a cluster (24 cores, 48GB of RAM). It ran for an hour or so, then I got the same error `Failed to open -: unknown file type`, so I guess I will try some commands without piping to `bcftools` to see what output I will get
BCArg
<phylogenetics><sam><vcf><samtools><nanopore>
Error while calling bcftools mpileup - Failed to open -: unknown file type
# Error while calling bcftools mpileup - Failed to open -: unknown file type

(Asked by: username_0 on 2019-07-08)

I have sequenced a bacterial genome with a GridIon from ONT. Basically what I want to check is whether or not trimming 50 bps at the beginning of the reads will improve alignment against the reference genome and ultimately the call of a consensus sequence.
For that what I have done so far is:

concatenated all the fastqs generated from the Gridion into a single (very large) fastq file: ref494_cat_all.fastq
aligned this concatenated fastq against the reference genome ref_genbank.fasta with minimap2: ~/Downloads/minimap2-2.17_x64-linux/minimap2 -ax map-ont ref_genbank/ref_genbank.fasta ref494_cat_all.fastq > ref494_aligned_trim50.sam
Performed the general samtools steps to covert sam to bam, index and sort

Then, in order to get the consensus sequence I first try to generate a .vcf.gz file, according to this tutorial
bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz

Immediatelly after calling the command above I got:
Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
[mpileup] 1 samples in 1 input files

Then, after a minute or two, I got:
Failed to open -: unknown file type

I'm pretty sure that there's no issue with .bam file, as I could visualise the alignment with IGV or tablet.
Does anyone know what the issue can be?

**Tags:** phylogenetics, sam, vcf, samtools, nanopore
> **Comments:**
> **username_1**: Please capture the output of the first command and show us the first few lines (via `bcftools view` if it's not text): `bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp`  
> **username_0**: I got `Killed` output to the terminal after a minute or two. Also this command generates an empty `output.tmp` file.  
> **username_1**: As noted below, killed due to memory exhaustion. However: you deserve an error message. Mpileup typically doesn't use that much memory and I have been unable to reproduce this on (non-ONT) data here. If you are able to, it would be interesting to try current **develop** bcftools to see if this still happens and/or to run it under the debugger and see where it crashes.  
> **username_1**: The latter is quite easy: run `gdb --args bcftools mpileup -f ref_genbank/ref_genbank.fasta -o /dev/null ref494_aligned.sorted.bam`, then type `run` and wait for it to crash. Then type `backtrace`; the output from this will be instructive.  
> **username_0**: Just before the program got killed I got `Program terminated with signal SIGKILL, Killed.` `The program no longer exists.`  The output of `backtrace` was `No stack`  That you cannot reproduce this error on non-ONT data, you think it can be related to the size of my input files? My bam is 14GB (coverage is about 2.000, we ran for about 48h, which we won't do routinely)  
> **username_1**: Oh yeah, SIGKILL. Perhaps you were hit by the OOM Killer (cf https://stackoverflow.com/questions/12288550/c-linux-binary-terminated-with-signal-sigkill-why-loaded-in-gdb) so the lack of error message maybe isn't bcftools's fault. 14G isn't a big BAM file, but the level of coverage and nature of ONT data (long reads with maybe interesting CIGARs) would be things that could make mpileup take more than the usual amount of memory.  
> **username_0**: Just tried to run the original command (as in my question) on a cluster (24 cores, 48GB of RAM). It ran for an hour or so, then I got the same error `Failed to open -: unknown file type`, so I guess I will try some commands without piping to `bcftools` to see what output I will get  

---

## 2 answers

### Answer by username_2 on 2019-07-09. Score: 1

It's possible you're looking at instructions for a recent version of samtools / bcftools, whereas your computer has an older version. Can you try samtools mpileup -g -f ... to output the pileup as BCF format instead of bcftools mpileup -f ...?
Update: it's good to know that the samtools / bcftools pipelines are at least producing consistent results, which suggests that the error is not associated with the output format on the mpileup side of the pipeline, and also good to see that you've identified that this was a memory issue.
The most frequent cause of memory issues I've had with sorted BAM files has been trying to figure out variants in extremely high coverage data (e.g. over 1000X), which would make sense for a bacterial isolate sequenced on a GridION. I use my own script to carry out a reservoir sampling on mapped BAM format files, which reduces coverage down to a more manageable level. Here's an example command to normalise to 100X coverage where possible:
samtools view reads.bam | ./samNormalise.pl -coverage 100 | \
   samtools view -b > reads_normalised.bam

> **Comments:**
> **username_0**: did not work unfortunately, got the following messages after calling: `samtools mpileup -g -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz`  `[warning] samtools mpileup option `g` is functional, but deprecated. Please switch to using bcftools mpileup in future.`  `Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid`  `[mpileup] 1 samples in 1 input files`  `Failed to open -: unknown file type`  
> **username_1**: The VCF/BCF-outputting `mpileup` command was transferred from samtools to bcftools relatively recently. So in fact it is @gringer's suggestion that is instructions for an earlier version of samtools / bcftools.  

---

### Answer by username_3 on 2019-07-08. Score: 1

The - is usually used to mean standard input when reading data. So the message would suggest that the second bcftools command fails to read from stdin, so it fails to read the output of the first bcftools command:
$ echo "foo bar" | /ref/software/bcftools-1.8/bcftools call -mv -Oz -o calls.vcf.gz
Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
Failed to open -: unknown file type

So there is something wrong with either your input file or with the output produced when running the first bcftools command on that input file. We can't really help more unless you share the file.

> **Comments:**
> **username_0**: I doubt there's any problem with the input files, as I can visualise the alignment on table or IGV, so I guess the `bcftoools mpileup` is not generating any output (please see my comment to @John Marshall). I guess I will report the problem on their github page.  
> **username_3**: @BCArg yes, precisely. If you got `Killed` in the terminal, then the process is being killed by the operating system, most likely because it is taking up too much memory. Sounds like you need to run this on a more powerful machine.  
> **username_0**: yes mystery solved, I monitored the the performance of the command `bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp` with `htop` and the program got killed right after I reached the maximum of my RAM memory (16GB), so in fact running the command on the cluster appears to be the solution.  

---

