<p>Not sure why you would want to do this, but if the aim is to repeat <strong>all</strong> records in the fastq/fasta files <code>n</code> times, the most efficient way I can think of is just to concatenate the orginal file <code>n</code> times, similar to <a href="https://superuser.com/questions/338616/linux-command-to-concatenate-a-file-to-itself-n-times">this thread</a>.</p>
<p>i.e adapting the accepted answer would look something like this (I have not tested this):</p>
<pre><code>for i in {1..n};do cat file.fq &gt;&gt; file_ntimes.fq ; done
</code></pre>
<p><strong>Update based on comments:</strong></p>
<p>To replicate each fastq entry back to back, maybe this hacky unix command can be adjusted (say for n=4):</p>
<pre><code>n=10
paste - - - - &lt;file.fq | awk '{for(i=0;i&lt;$n;i++)print}' | tr '\t' '\n' &gt; file_10times.fq
</code></pre>
<p>The idea is to <code>paste</code> blocks of 4 lines (i.e. 1 fastq entry) into one line and replicate this line n times. <code>tr</code> is reversing the paste action (since the separation by paste is a tab character). This won't work if fastq file for some reason has tabs in it which I don't expect but didn't test either.
Note, for fasta file this would be <code>paste - -</code> rather than <code>paste - - - -</code></p>
<p>I haven't tested this, but if it works for you I could polish it up if needed.</p>
<p><strong>Update2</strong></p>
<p>Based on Xioafeis comment this is solution seems to work as well:</p>
<pre><code>bioawk -cfastx '{for(i=1; i&lt;=10; i++) {print &quot;@&quot;$name&quot;\n&quot;$seq&quot;\n+\n&quot;$qual}}' file.fq &gt; file_10times.fq
</code></pre>

CC BY-SA 4.0
2021-01-17T13:28:38.767Z
Thank you so much for your reply!
But I don't think I made my question clear. I want to repeat rd_1 for 100 times, rd_2 for 100 times, ... in this order, not (rd_1, rd_2, ...) x 100 times.
Xiaofei Wang
CC BY-SA 4.0
2021-01-17T13:36:54.300Z
My goal is to pairwise each of read record in file_1.fq with each of record in file_2.fq into 2 new fastq files and get overlap between them. Actually, for the one that you suggested that is what I want to for my file_new_2.fq, which I have already figured out. But not the one that I am asking. I will edit my question with a figure. Really appreciate for your reply!
Xiaofei Wang
CC BY-SA 4.0
2021-01-17T13:43:15.863Z
I did wrote a python code using biopython, but it is very slow I guess it slurped whole file into dictionary, so very very slow and seems running forever
Xiaofei Wang
CC BY-SA 4.0
2021-01-18T12:05:35.623Z
Thank you so much! I figured it out by using bioawk.
Xiaofei Wang
CC BY-SA 4.0
2021-01-18T12:08:58.673Z
`bioawk -cfastx '{for(i=1; i<=10; i++) {print "@"$name"\n"$seq"\n+\n"$qual}}' test_repeat.fq > test_repeat_10times.fq`
Xiaofei Wang
CC BY-SA 4.0
2021-01-18T12:59:51.157Z
I see, did my suggested solution work as well? Feel free to upvote or accept the answer in that case! Also Happy to add you solution as 3rd option if you want to have it as offical answer?
Sebastian Müller
CC BY-SA 4.0
2021-01-17T12:30:12.000Z
2021-01-20T08:07:29.723Z
2021-01-20T08:07:29.723Z
Sebastian Müller
<p><a href="https://i.sstatic.net/Qdrm0.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Qdrm0.jpg" alt="enter image description here" /></a>How to iterate/repeat a record n times in a fastq file using bioawk? I wrote a python code using biopython, but it is very very slow. So, I am wondering if I can get some help by using bioawk. Thank you so much!</p>
<p>for example, in the file.fq file, I'd like to repeat read_id1 for 100times, read_id2 for 100times, ..., and write them to a new file.</p>

CC BY-SA 4.0
2021-01-17T12:43:05.820Z
Hi and welcom to BS, could you please a little bit more what you problem you want to solve? Do you want only specific record or all records of the file? Maybe you can show your python code for clarity.
Mr_Z
CC BY-SA 4.0
2021-01-17T18:29:25.290Z
You might want to show your biopython implementation. While biopython can be slow, your solution may be sub-optimal
Chris_Rands
CC BY-SA 4.0
2021-01-22T19:03:27.927Z
There are quite a few people complaining about the speed of Biopython... I think [pysam](https://pysam.readthedocs.io/en/latest/api.html#fastq-files) is much faster.
Kamil S Jaron
<fastq><shell><perl><bioawk><repeat>
how to repeat records in fastq n times efficiently?
# how to repeat records in fastq n times efficiently?

(Asked by: username_0 on 2021-01-17)

[![enter image description here](https://i.sstatic.net/Qdrm0.jpg)](https://i.sstatic.net/Qdrm0.jpg)How to iterate/repeat a record n times in a fastq file using bioawk? I wrote a python code using biopython, but it is very very slow. So, I am wondering if I can get some help by using bioawk. Thank you so much!

for example, in the file.fq file, I'd like to repeat read\_id1 for 100times, read\_id2 for 100times, ..., and write them to a new file.

**Tags:** fastq, shell, perl, bioawk, repeat
> **Comments:**
> **username_1**: Hi and welcom to BS, could you please a little bit more what you problem you want to solve? Do you want only specific record or all records of the file? Maybe you can show your python code for clarity.  
> **username_2**: You might want to show your biopython implementation. While biopython can be slow, your solution may be sub-optimal  
> **username_3**: There are quite a few people complaining about the speed of Biopython... I think [pysam](https://pysam.readthedocs.io/en/latest/api.html#fastq-files) is much faster.  

---

## 1 answers

### Answer by username_4 on 2021-01-17. Score: 0 | Accepted answer

Not sure why you would want to do this, but if the aim is to repeat all records in the fastq/fasta files n times, the most efficient way I can think of is just to concatenate the orginal file n times, similar to this thread.
i.e adapting the accepted answer would look something like this (I have not tested this):
for i in {1..n};do cat file.fq >> file_ntimes.fq ; done

Update based on comments:
To replicate each fastq entry back to back, maybe this hacky unix command can be adjusted (say for n=4):
n=10
paste - - - -  file_10times.fq

The idea is to paste blocks of 4 lines (i.e. 1 fastq entry) into one line and replicate this line n times. tr is reversing the paste action (since the separation by paste is a tab character). This won't work if fastq file for some reason has tabs in it which I don't expect but didn't test either.
Note, for fasta file this would be paste - - rather than paste - - - -
I haven't tested this, but if it works for you I could polish it up if needed.
Update2
Based on Xioafeis comment this is solution seems to work as well:
bioawk -cfastx '{for(i=1; i<=10; i++) {print "@"$name"\n"$seq"\n+\n"$qual}}' file.fq > file_10times.fq

> **Comments:**
> **username_0**: Thank you so much for your reply! But I don't think I made my question clear. I want to repeat rd_1 for 100 times, rd_2 for 100 times, ... in this order, not (rd_1, rd_2, ...) x 100 times.  
> **username_0**: My goal is to pairwise each of read record in file_1.fq with each of record in file_2.fq into 2 new fastq files and get overlap between them. Actually, for the one that you suggested that is what I want to for my file_new_2.fq, which I have already figured out. But not the one that I am asking. I will edit my question with a figure. Really appreciate for your reply!  
> **username_0**: I did wrote a python code using biopython, but it is very slow I guess it slurped whole file into dictionary, so very very slow and seems running forever  
> **username_0**: Thank you so much! I figured it out by using bioawk.  
> **username_0**: `bioawk -cfastx '{for(i=1; i<=10; i++) {print "@"$name"\n"$seq"\n+\n"$qual}}' test_repeat.fq > test_repeat_10times.fq`  
> **username_4**: I see, did my suggested solution work as well? Feel free to upvote or accept the answer in that case! Also Happy to add you solution as 3rd option if you want to have it as offical answer?  

---

