<p>Partial answer, but I want to discuss something that is only rarely mentioned in any read simulation / ground truth studies.</p>
<p>Given that you're working with Illumina reads, you need to be very careful with evaluating that ground truth and identifying repetitive sequences. In particular, don't assume that a particular high-complexity sequence will have a unique match to the ground-truth genome; you need to either ignore repetitive sequences (which would lead to biases), or have some way of incorporating repetition into your model.</p>
<p>As a demonstration of the issue, I did a 100bp repeat survey of the Telomere-to-Telomere CHM13 assembly a few years ago:</p>
<p><a href="https://zenodo.org/doi/10.5281/zenodo.4762023" rel="nofollow noreferrer">https://zenodo.org/doi/10.5281/zenodo.4762023</a></p>
<p>One of these plots (from chr1) is shown here:</p>
<p><a href="https://i.sstatic.net/jHqzcEFd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jHqzcEFd.png" alt="REPAVER plot for T2T/Chr1" /></a></p>
<p>Without getting too deep into the details, each point on this plot represents a location where there is a sequence of at least 100bp that is found <em>identically</em> in at least two different locations in the chromosome. Because most of these locations are sparsely-scattered high-complexity sequences, many repeat-finder algorithms (e.g. see <a href="https://bioinformatics.stackexchange.com/questions/8759/looking-for-a-good-tool-for-repeat-masking-in-long-sequences">here</a>) will ignore them. Any program that maps sequences to these locations and tries to evaluate ground truth as &quot;sequence X <em>must</em> be in location Y&quot; will produce false results.</p>
<p>Bear in mind that this is only looking for identical sequences; slight variations (e.g. 1 SNP differences) are not represented on this plot.</p>

CC BY-SA 4.0
2024-08-14T19:36:24.553Z
Some good information here for building out an error correction program. However, focusing on the main question being asked, the ground truth data is generated along with the "working/real data". Alignments and exact sequences are known and there is probably no program out there that could correct them perfectly. The thing I am wondering is if there exists a program that can use the ground truth data, original data, and the output from an error corrector and give an analysis of how well it did as well as maybe stats on what regions or sequences it is having particular trouble with.
Gaston19
CC BY-SA 4.0
2024-08-14T22:17:55.107Z
Yes, that's why I prefaced it by calling it a partial answer. I can't provide a full/appropriate answer to what you have asked.
gringer
CC BY-SA 4.0
2024-08-12T23:51:04.383Z
2024-08-12T23:51:04.383Z
gringer
<h2>The main idea:</h2>
<p>Given fastq/fasta files containing reads(simulated from ref. genome, with errors introduced) and another file with the ground truth(simulated but without errors) and then the resulting fastq/fasta files from error correction program ABC, is there an existing program that will give some kind of score and analysis on how well some error correction program ABC did.</p>
<h2>In depth:</h2>
<p>I have 2 tools I am using to synthetically generate reads from Illumina machines. They both take in a reference genome and then produce 2 fastq files that are paired end reads with errors. One of these programs produces a second pair of fastq files that are the same reads but without errors(ground truth). The other program produces a bam/sam file for ground truth.</p>
<p>The fastq files with errors are passed to error correction programs which then produce a new set of fasta files. I am looking to compare the quality of the error correction between these programs, as well as my own method. I am not bias towards one method or the other and could generate alignments or sequences from alignments. I am just looking for a program that exists that is scalable and fast at assesing the quality of the ouputs of error correction programs. If there are multiple programs I have to chain together, that would work too, I just don't want to write my own if one already exists.</p>

<benchmarking><read-correction>
Assessing Quality of Error Correction
# Assessing Quality of Error Correction

(Asked by: username_0 on 2024-08-12)

## The main idea:

Given fastq/fasta files containing reads(simulated from ref. genome, with errors introduced) and another file with the ground truth(simulated but without errors) and then the resulting fastq/fasta files from error correction program ABC, is there an existing program that will give some kind of score and analysis on how well some error correction program ABC did.

## In depth:

I have 2 tools I am using to synthetically generate reads from Illumina machines. They both take in a reference genome and then produce 2 fastq files that are paired end reads with errors. One of these programs produces a second pair of fastq files that are the same reads but without errors(ground truth). The other program produces a bam/sam file for ground truth.

The fastq files with errors are passed to error correction programs which then produce a new set of fasta files. I am looking to compare the quality of the error correction between these programs, as well as my own method. I am not bias towards one method or the other and could generate alignments or sequences from alignments. I am just looking for a program that exists that is scalable and fast at assesing the quality of the ouputs of error correction programs. If there are multiple programs I have to chain together, that would work too, I just don't want to write my own if one already exists.

**Tags:** benchmarking, read-correction

---

## 1 answers

### Answer by username_1 on 2024-08-12. Score: 0

Partial answer, but I want to discuss something that is only rarely mentioned in any read simulation / ground truth studies.

Given that you're working with Illumina reads, you need to be very careful with evaluating that ground truth and identifying repetitive sequences. In particular, don't assume that a particular high-complexity sequence will have a unique match to the ground-truth genome; you need to either ignore repetitive sequences (which would lead to biases), or have some way of incorporating repetition into your model.

As a demonstration of the issue, I did a 100bp repeat survey of the Telomere-to-Telomere CHM13 assembly a few years ago:

<https://zenodo.org/doi/10.5281/zenodo.4762023>

One of these plots (from chr1) is shown here:

[![REPAVER plot for T2T/Chr1](https://i.sstatic.net/jHqzcEFd.png)](https://i.sstatic.net/jHqzcEFd.png)

Without getting too deep into the details, each point on this plot represents a location where there is a sequence of at least 100bp that is found *identically* in at least two different locations in the chromosome. Because most of these locations are sparsely-scattered high-complexity sequences, many repeat-finder algorithms (e.g. see [here](https://bioinformatics.stackexchange.com/questions/8759/looking-for-a-good-tool-for-repeat-masking-in-long-sequences)) will ignore them. Any program that maps sequences to these locations and tries to evaluate ground truth as "sequence X *must* be in location Y" will produce false results.

Bear in mind that this is only looking for identical sequences; slight variations (e.g. 1 SNP differences) are not represented on this plot.

> **Comments:**
> **username_0**: Some good information here for building out an error correction program. However, focusing on the main question being asked, the ground truth data is generated along with the "working/real data". Alignments and exact sequences are known and there is probably no program out there that could correct them perfectly. The thing I am wondering is if there exists a program that can use the ground truth data, original data, and the output from an error corrector and give an analysis of how well it did as well as maybe stats on what regions or sequences it is having particular trouble with.  
> **username_1**: Yes, that's why I prefaced it by calling it a partial answer. I can't provide a full/appropriate answer to what you have asked.  

---

