<p>Your question can be rewritten as, &quot;how do I get the union of reads in two files?&quot; Since you have already subset the mates, the steps are:</p>
<ol>
<li>Process each file, storing the list of reads names in each.</li>
<li>Take the union of these.</li>
<li>Iterate through the original/non-subset files and write only the reads with names in the union to the output.</li>
</ol>
<p>The files will be both subset correctly and synced properly such that bowtie2 or any other aligner can correctly process them.</p>

CC BY-SA 4.0
2021-01-14T12:47:51.937Z
Hi Devon....is there a tool which can do this ? Kindly let me know.
Amit
CC BY-SA 4.0
2021-01-14T13:40:26.237Z
Probably, but in the time it would take to search for it it'd be faster to write the 10-20 lines of python (or whatever you prefer) to just do it.
Devon Ryan
CC BY-SA 4.0
2021-01-14T17:21:44.220Z
If you want to stick with shell scripting: `comm -12 file1ids file2ds` should give the union, followed by a similar grep like you used but with `-f union.txt` to give the patterns in a file. The undocumented `--no-group-separator` argument to grep may help you too if you're using GNU grep.
Jesse
CC BY-SA 4.0
2021-01-14T21:38:30.623Z
Alternately, you could go through both files simultaneously, and output both reads if either of them matches.
swbarnes2
CC BY-SA 4.0
2021-01-17T01:38:36.200Z
Based on my reading of the question, I'd guess they're interested in the *intersection* of reads, rather than the union
gringer
CC BY-SA 4.0
2021-01-17T09:43:20.760Z
@gringer Could be, I had used the intersection originally in my reply but then figured OP wanted reads with the pattern in either mate. We'd need clarification I guess.
Devon Ryan
CC BY-SA 4.0
2021-01-14T11:02:08.837Z
2021-01-14T13:41:16.293Z
2021-01-14T13:41:16.293Z
Devon Ryan
<p>It's a little bit unclear what you're asking, but if you want to repair paired-end files that have become disordered (or the linking has been lost), then the repair function of bbtools should work. This will pair up any coincident read IDs, and create a separate file for any reads that have no pairs in the other file:</p>
<p><a href="https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/repair-guide/" rel="nofollow noreferrer">https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/repair-guide/</a></p>
<pre><code>repair.sh in1=broken1.fq in2=broken2 out1=fixed1.fq out2=fixed2.fq outs=singletons.fq repair
</code></pre>
<p>If grep is used on one file, and repair used on that file in combination with the original file from the matching read, then the relevant read pairs should appear in the output:</p>
<pre><code>repair.sh in1=filtered_1.fq in2=file_2 out1=fixed_1.fq out2=fixed_2.fq outs=unpaired_2.fq repair
</code></pre>

CC BY-SA 4.0
2021-01-15T05:07:21.160Z
It's not clear to me that both halves of any pairs will be caught by that grep, so there won't be anything repairable.
swbarnes2
CC BY-SA 4.0
2021-01-17T01:35:53.987Z
That's the "unclear what you're asking" bit.
gringer
CC BY-SA 4.0
2021-01-15T03:08:16.860Z
2021-01-17T01:37:24.090Z
2021-01-17T01:37:24.090Z
gringer
<p>I have paired end reads and want to extract only those which have this sequence within them:</p>
<pre><code>TGTATGTAAACTTCCGACTTCAACTGTA
</code></pre>
<p>I tried using <code>grep -B1 -A2 &quot;TGTATGTAAACTTCCGACTTCAACTGTA&quot; file_1.fq | grep -v &quot;^--$&quot;</code>
and the same from the mate pair. After this I tried aligning them , but they dont align because the reads headers are different.
Can someone help in extracting the mate pair of one read from the other file (2.fq), so that they can be aligned ?
regards.</p>

CC BY-SA 4.0
2021-01-14T00:55:29.683Z
Can you sit down and write out **exactly** what you want to do?  I think it's pretty clear why what you did is not returning matched pairs of reads, but it's not clear to me if you see why.
swbarnes2
CC BY-SA 4.0
2021-01-14T08:33:55.697Z
As I wrote, that I have paired end reads. I want to extract reads from both pairs which have in inserted sequence TGTATGTAAACTTCCGACTTCAACTGTA in them. It belongs to a transposon along with guideDNA. The fastq files are stored as forward and reverse reads. When I executed the script it brings out the reads which have insertion within them. However, they dont align with Bowtie2 as the reads are not from the same pairs.
Amit
CC BY-SA 4.0
2021-01-14T21:37:08.167Z
What I'm trying to get at is, do you want reads which lack the sequence if their mate has it?  Because if the answer is "no", why are you trying to align paired reads when you are throwing away a member of a pair?  And if the answer is "yes", you are clearly throwing those reads away.
swbarnes2
<grep>
FASTQ file reads extraction paired ends
# FASTQ file reads extraction paired ends

(Asked by: username_0 on 2021-01-13)

I have paired end reads and want to extract only those which have this sequence within them:
TGTATGTAAACTTCCGACTTCAACTGTA

I tried using grep -B1 -A2 "TGTATGTAAACTTCCGACTTCAACTGTA" file_1.fq | grep -v "^--$"
and the same from the mate pair. After this I tried aligning them , but they dont align because the reads headers are different.
Can someone help in extracting the mate pair of one read from the other file (2.fq), so that they can be aligned ?
regards.

**Tags:** grep
> **Comments:**
> **username_1**: Can you sit down and write out **exactly** what you want to do?  I think it's pretty clear why what you did is not returning matched pairs of reads, but it's not clear to me if you see why.  
> **username_0**: As I wrote, that I have paired end reads. I want to extract reads from both pairs which have in inserted sequence TGTATGTAAACTTCCGACTTCAACTGTA in them. It belongs to a transposon along with guideDNA. The fastq files are stored as forward and reverse reads. When I executed the script it brings out the reads which have insertion within them. However, they dont align with Bowtie2 as the reads are not from the same pairs.  
> **username_1**: What I'm trying to get at is, do you want reads which lack the sequence if their mate has it?  Because if the answer is "no", why are you trying to align paired reads when you are throwing away a member of a pair?  And if the answer is "yes", you are clearly throwing those reads away.  

---

## 2 answers

### Answer by username_2 on 2021-01-14. Score: 3

Your question can be rewritten as, "how do I get the union of reads in two files?" Since you have already subset the mates, the steps are:

1. Process each file, storing the list of reads names in each.
2. Take the union of these.
3. Iterate through the original/non-subset files and write only the reads with names in the union to the output.

The files will be both subset correctly and synced properly such that bowtie2 or any other aligner can correctly process them.

> **Comments:**
> **username_0**: Hi Devon....is there a tool which can do this ? Kindly let me know.  
> **username_2**: Probably, but in the time it would take to search for it it'd be faster to write the 10-20 lines of python (or whatever you prefer) to just do it.  
> **username_3**: If you want to stick with shell scripting: `comm -12 file1ids file2ds` should give the union, followed by a similar grep like you used but with `-f union.txt` to give the patterns in a file. The undocumented `--no-group-separator` argument to grep may help you too if you're using GNU grep.  
> **username_1**: Alternately, you could go through both files simultaneously, and output both reads if either of them matches.  
> **username_4**: Based on my reading of the question, I'd guess they're interested in the *intersection* of reads, rather than the union  
> **username_2**: @gringer Could be, I had used the intersection originally in my reply but then figured OP wanted reads with the pattern in either mate. We'd need clarification I guess.  

---

### Answer by username_4 on 2021-01-15. Score: 1

It's a little bit unclear what you're asking, but if you want to repair paired-end files that have become disordered (or the linking has been lost), then the repair function of bbtools should work. This will pair up any coincident read IDs, and create a separate file for any reads that have no pairs in the other file:
https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/repair-guide/
repair.sh in1=broken1.fq in2=broken2 out1=fixed1.fq out2=fixed2.fq outs=singletons.fq repair

If grep is used on one file, and repair used on that file in combination with the original file from the matching read, then the relevant read pairs should appear in the output:
repair.sh in1=filtered_1.fq in2=file_2 out1=fixed_1.fq out2=fixed_2.fq outs=unpaired_2.fq repair

> **Comments:**
> **username_1**: It's not clear to me that both halves of any pairs will be caught by that grep, so there won't be anything repairable.  
> **username_4**: That's the "unclear what you're asking" bit.  

---

