<p>I would suggest subsampling an existing dataset such as the <a href="https://www.nist.gov/programs-projects/genome-bottle" rel="nofollow noreferrer">GIAB</a> resource that you've already indicated.</p>
<p>Use a utility such as <a href="https://github.com/lh3/seqtk" rel="nofollow noreferrer">seqtk</a> to subsample the appropriate number of reads from the dataset, and use that.</p>
<p>Example with <code>seqtk</code> to get 1 million reads each from paired fastqs while preserving pairing:</p>
<pre><code>  # note -s100 random seed which should be used for both to keep pairs
  seqtk sample -s100 giab_read1.fq 1000000 &gt; giab_sub_r1.fq
  seqtk sample -s100 giab_read2.fq 1000000 &gt; giab_sub_r2.fq
</code></pre>

CC BY-SA 4.0
2023-10-30T02:46:42.517Z
Ok, that's actually a good suggestion. However, how to subsample gold standard fastq, .bam, and .vcf? So, that I know my pipeline steps/changes are okay in all three steps?
Shafayet Rahat
CC BY-SA 4.0
2023-10-30T09:31:23.533Z
@ShafayetRahat just subsample the fastq, and then run the small fastq files through your pipeline to get bam abd vcf. I use a similar approach, and it only takes about 5 minutes to run through the pipeline with these small files.
terdon
CC BY-SA 4.0
2023-10-30T19:42:33.407Z
@ShafayetRahat terdon is right, unless you're doing something odd mapping should be unaffected by depth. And you will not be able to subsample a VCF to only represent certain reads, I am not sure how you would do that. (though in principle you can do it with a BAM file, it would just be a lot of work compared to re-mapping).
Maximilian Press
CC BY-SA 4.0
2023-10-29T21:08:03.470Z
2023-10-29T21:08:03.470Z
Maximilian Press
<p>I am trying to check my variant generation pipeline for humans. The pipeline is for NGS. I am searching for any dataset(.fastq+.bam+.vcf) for any specific sample. Now, GIAB has a nice dataset for that. However, the problem is that it's not a tiny dataset. And it's not feasible to test my pipeline with this data every time. I tried to generate a synthetic dataset with some read simulators(NEAT, MASON2). However, their aligned read files(.bam) files seem not very good. You can check my Issue for NEAT in <a href="https://github.com/ncsa/NEAT/issues/88" rel="nofollow noreferrer">link</a>.
So, my question is, <strong>Is there any publicly available tiny dataset(.fastq,.bam, .vcf) with which I can validate my pipeline fast? Or, is there any simulator with which I can simulate .bam, .fastq files from a given input .vcf? I am using fastqc,bwa,samtools,GATK4 mainly and GRCH38 reference in my pipeline.</strong></p>

<ngs><public-dataset><simulated-data><validation>
Generating simulated gold standard NGS dataset/ Any publicly available smal gold standard NGS dataset
# Generating simulated gold standard NGS dataset/ Any publicly available smal gold standard NGS dataset

(Asked by: username_0 on 2023-10-29)

I am trying to check my variant generation pipeline for humans. The pipeline is for NGS. I am searching for any dataset(.fastq+.bam+.vcf) for any specific sample. Now, GIAB has a nice dataset for that. However, the problem is that it's not a tiny dataset. And it's not feasible to test my pipeline with this data every time. I tried to generate a synthetic dataset with some read simulators(NEAT, MASON2). However, their aligned read files(.bam) files seem not very good. You can check my Issue for NEAT in [link](https://github.com/ncsa/NEAT/issues/88).
So, my question is, **Is there any publicly available tiny dataset(.fastq,.bam, .vcf) with which I can validate my pipeline fast? Or, is there any simulator with which I can simulate .bam, .fastq files from a given input .vcf? I am using fastqc,bwa,samtools,GATK4 mainly and GRCH38 reference in my pipeline.**

**Tags:** ngs, public-dataset, simulated-data, validation

---

## 1 answers

### Answer by username_1 on 2023-10-29. Score: 2

I would suggest subsampling an existing dataset such as the GIAB resource that you've already indicated.
Use a utility such as seqtk to subsample the appropriate number of reads from the dataset, and use that.
Example with seqtk to get 1 million reads each from paired fastqs while preserving pairing:
  # note -s100 random seed which should be used for both to keep pairs
  seqtk sample -s100 giab_read1.fq 1000000 > giab_sub_r1.fq
  seqtk sample -s100 giab_read2.fq 1000000 > giab_sub_r2.fq

> **Comments:**
> **username_0**: Ok, that's actually a good suggestion. However, how to subsample gold standard fastq, .bam, and .vcf? So, that I know my pipeline steps/changes are okay in all three steps?  
> **username_2**: @ShafayetRahat just subsample the fastq, and then run the small fastq files through your pipeline to get bam abd vcf. I use a similar approach, and it only takes about 5 minutes to run through the pipeline with these small files.  
> **username_1**: @ShafayetRahat terdon is right, unless you're doing something odd mapping should be unaffected by depth. And you will not be able to subsample a VCF to only represent certain reads, I am not sure how you would do that. (though in principle you can do it with a BAM file, it would just be a lot of work compared to re-mapping).  

---

