<p>Anyone can declare anything to be a &quot;gold-standard&quot;, the term is meaningless. All if actually means is, &quot;we think this is the test that everything else should be benchmarked against&quot;.</p>

CC BY-SA 4.0
2021-01-07T12:14:03.937Z
2021-01-07T12:14:03.937Z
Devon Ryan
<p>Thanks for raising the issue, it is certainly topical. The question is whether the data set they unit tested is reliable.</p>
<p>I think the issue they are trying get to is mosaic artefacts in the sequencing process and assembly thereon. This would be particularly true in a situation of <em>de novo</em> assembly which is what the authors are targeting. There are some slick methodologies for virus sequencing, which have been around for a while now, so in any case a 'gold-standard' would be outdated at present.</p>
<p>I do have to agree with you that the language particularly for the journal certainly could have phrased better. The data set was 2014 and genome circularisation (its a virus small genomes) to remove the otherwise high SNP error rate would have been around in 2017 when this was published.</p>
<p>I fully understand why the authors choose this language. A professional editor, not affiliated with the journal (without up to date technical insight), would likely recommend the same language.</p>
<p>I guess the question would be whether verification is achieved on other virus data sets. There is no shortage of them at present and its certainly an interesting tool.</p>

CC BY-SA 4.0
2022-02-28T21:27:19.557Z
Well, I was not asking that because of any suspicion about their results. My software ([s-aligner](https://contignant.com/benchmarks)) gets better results for this benchmark set and I just wanted to know how relevant is that. It also outperforms other software in many other benchmarks.
juanjo75es
CC BY-SA 4.0
2022-03-01T15:20:31.617Z
Thanks @juanjo75es this is a good subject to explore in a question. The algorithms are pivotal to understanding viral diversity. I put this as a separate question and notify you. I'll explain what I understand and why within the question.
M__
CC BY-SA 4.0
2022-03-09T01:04:50.793Z
@juanjo75es I have raised the matter in the following question, https://bioinformatics.stackexchange.com/questions/18747/metagenomics-pipeline-recommendations
M__
CC BY-SA 4.0
2022-02-28T14:42:33.053Z
2022-02-28T15:09:13.157Z
2022-02-28T15:09:13.157Z
M__
<p>This page is claimed to contain a gold standard benchmark for viral genome assembly.
<a href="https://github.com/cbg-ethz/5-virus-mix" rel="nofollow noreferrer">https://github.com/cbg-ethz/5-virus-mix</a></p>
<p>The claim is here:
<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5411778/" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5411778/</a></p>
<blockquote>
<p>Baaijens <em>et al</em>. De novo assembly of viral quasispecies using overlap graphs
Genome Research 2017 May; 27: 835–848.</p>
</blockquote>
<p>&quot;<em>This data set was recently presented as a gold standard benchmark (Di Giallonardo et al. 2014)</em>&quot;</p>
<p>But at (Di Giallonardo et al. 2014) I can only find that mention:</p>
<p>&quot;We previously estimated the frequencies of the virus strains in the 5-virus-mix by amplifying the protease gene using single-genome amplification (SGA), the current gold standard for studying diversity of virus populations&quot;.</p>
<p>That doesn't seem to mean precisely that the dataset is a gold standard benchmark for genome assembly but, as I understand, that SGA is a gold standard for something else.</p>
<p>My question is: is it? And who decided it is? Is still valid? Is there an official list of results somewhere for this benchmark set?</p>
<p>I tried to contact the owner of the page but still no answer.</p>

<genome><assembly><covid-19><benchmarking><virus>
Gold standard benchmark
# Gold standard benchmark

(Asked by: username_0 on 2021-01-06)

This page is claimed to contain a gold standard benchmark for viral genome assembly.
<https://github.com/cbg-ethz/5-virus-mix>

The claim is here:
<https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5411778/>

> Baaijens *et al*. De novo assembly of viral quasispecies using overlap graphs
> Genome Research 2017 May; 27: 835–848.

"*This data set was recently presented as a gold standard benchmark (Di Giallonardo et al. 2014)*"

But at (Di Giallonardo et al. 2014) I can only find that mention:

"We previously estimated the frequencies of the virus strains in the 5-virus-mix by amplifying the protease gene using single-genome amplification (SGA), the current gold standard for studying diversity of virus populations".

That doesn't seem to mean precisely that the dataset is a gold standard benchmark for genome assembly but, as I understand, that SGA is a gold standard for something else.

My question is: is it? And who decided it is? Is still valid? Is there an official list of results somewhere for this benchmark set?

I tried to contact the owner of the page but still no answer.

**Tags:** genome, assembly, covid-19, benchmarking, virus

---

## 2 answers

### Answer by username_1 on 2021-01-07. Score: 1 | Accepted answer

Anyone can declare anything to be a "gold-standard", the term is meaningless. All if actually means is, "we think this is the test that everything else should be benchmarked against".


---

### Answer by username_2 on 2022-02-28. Score: 1

Thanks for raising the issue, it is certainly topical. The question is whether the data set they unit tested is reliable.

I think the issue they are trying get to is mosaic artefacts in the sequencing process and assembly thereon. This would be particularly true in a situation of *de novo* assembly which is what the authors are targeting. There are some slick methodologies for virus sequencing, which have been around for a while now, so in any case a 'gold-standard' would be outdated at present.

I do have to agree with you that the language particularly for the journal certainly could have phrased better. The data set was 2014 and genome circularisation (its a virus small genomes) to remove the otherwise high SNP error rate would have been around in 2017 when this was published.

I fully understand why the authors choose this language. A professional editor, not affiliated with the journal (without up to date technical insight), would likely recommend the same language.

I guess the question would be whether verification is achieved on other virus data sets. There is no shortage of them at present and its certainly an interesting tool.

> **Comments:**
> **username_0**: Well, I was not asking that because of any suspicion about their results. My software ([s-aligner](https://contignant.com/benchmarks)) gets better results for this benchmark set and I just wanted to know how relevant is that. It also outperforms other software in many other benchmarks.  
> **username_2**: Thanks @juanjo75es this is a good subject to explore in a question. The algorithms are pivotal to understanding viral diversity. I put this as a separate question and notify you. I'll explain what I understand and why within the question.  
> **username_2**: @juanjo75es I have raised the matter in the following question, https://bioinformatics.stackexchange.com/questions/18747/metagenomics-pipeline-recommendations  

---

