<p>EDIT: <strong><a href="https://bioinformatics.stackexchange.com/a/11311/4980">Chris_Rands's answer</a> is more authoritative and clarifies that this behavior is essentially a bug.</strong>  (I had only tested my example on Python 3.6.8 and hadn't seen PEP 479.)</p>

<p>When <code>next</code> runs out of lines, it raises <code>StopIteration</code>, and apparently that stops the while loop similarly to calling <code>break</code> without you needing to catch the exception.  I couldn't see a definitive source on that but you can see it in action if you catch and re-raise the exception:</p>

<pre><code>def ingest2(stream1, stream2):
    while True:
        try:
            record1, record2 = list(), list()
            for _ in range(4):
                record1.append(next(stream1))
                record2.append(next(stream2))
            yield record1, record2
        except StopIteration:
            print("StopIteration raised")
            raise
</code></pre>

<p>You can also try to catch it outside the loop, but it doesn't get re-raised at that point.  (If you try raising one yourself in a while loop, though, it <em>doesn't</em> catch it in the same way, so there must be some extra Python magic going on.  I can't see any difference between the two exception objects so I'd be curious what's going on there.)  Neat example.  You could also make a pair of FASTQ parsers using Biopython's <a href="https://biopython.org/wiki/SeqIO#sequence-input" rel="nofollow noreferrer">SeqIO.parse</a> for more complex/permanent situations if needed.</p>

CC BY-SA 4.0
2020-02-04T21:42:24.807Z
2020-02-05T15:46:30.730Z
2020-02-05T15:46:30.730Z
Jesse
<p>Your code only runs prior to Python 3.7, now it raises <code>RuntimeError: generator raised StopIteration</code>. This is essentially a bug fix described in <a href="https://www.python.org/dev/peps/pep-0479/" rel="nofollow noreferrer">PEP 479 -- Change StopIteration handling inside generators</a>. You can enable the new behaviour from Python 3.5 via <code>from __future__ import generator_stop</code>.</p>

<p>Note, it's not related to file-handles per se, you can reproduce it with a smaller example with a simple iterator:</p>

<pre><code>it = iter([1,2])

def f(it):
    while True:
        yield next(it)

print(list(f(it)))
</code></pre>

<p>You should not write code relying on the bug</p>

CC BY-SA 4.0
2020-02-04T21:54:48.543Z
2020-02-04T21:54:48.543Z
Chris_Rands
<p>I wrote some throwaway code for ingesting paired-end reads from a Fastq file into memory.</p>

<pre class="lang-py prettyprint-override"><code>def ingest(stream1, stream2):
    while True:
        record1, record2 = list(), list()
        for _ in range(4):
            record1.append(next(stream1))
            record2.append(next(stream2))
        yield record1, record2
</code></pre>

<p>I had initially put the <code>for</code> loop in a <code>try/Except</code> block, anticipating I'd need to explicitly break out of the endless <code>while</code> loop. However, the code seems to work correctly even after I removed it. I even "corrupted" the final read in one of the paired files, and the <code>ingest</code> function performs reasonably (the last read pair is never <code>yield</code>ed).</p>

<p>My question is...<strong>why does this code work</strong>, since I never explicitly break out of the loop?</p>

<p>Here's an example of how the code is invoked.</p>

<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; with open('seqR1.fastq', 'r') as fh1, open('seqR2.fastq', 'r') as fh2:
...     counter = 0
...     for r1, r2 in interleave(fh1, fh2):
...         counter += 1
&gt;&gt;&gt; counter
5000
&gt;&gt;&gt; r1
['@read5000/1\n', 'AGGTCTGGGGACAGGTGTTTTACAGGCTCCTTTAGGGGTGGCAAGTCTCCCGGAAACATGATTAGTGAAGACACGTTAACAGCGAAACTGACAGAGAACCTCAACATTCTCTGGTAGGAAAAGCCAGGATCCTGGCAGAGGAAGCAACGATTCGAGCAGGGAACTGTGCAGGACAGAGAACAATGCGGCCCCAATTTTGATCCAGGGGTAGTTAGCTTAGCTAAGGAGTTCCCCAGTCCCTAAGTGGAGAGGGGATCCAGAACCCTACAATTTATTATTATATCTTAGCGGATATATCTAT\n', '+\n', 'CCCCCGGGEGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@GGGGGGGFAGGGCGGGGGFGGGGGGGGGCGGGGGDGGGGFGGGGGGGGGCGGGGGGGGGGGGGGGCGGGGGGGGGGGGGGGGG@G&lt;GGGGGGGGGGGGFGGFGFF:DGGFFCGGGGCGGFEGGGGGGEGGGFB:GGGDG=FGGGG,GAGGGDGGFGGGCD3GFG+G?G*FGGG&lt;GCGGGGC@CD&lt;FFGF*:+G=FE&lt;;GG,8C7GF=;9D=G&lt;C5C66G6C&gt;7)FD*6G7D)9;*C:)4D5+9&gt;@G*=0):)*;1*504\n']
&gt;&gt;&gt; r2
['@read5000/2\n', 'TGTGGGGTTCTGGATCCCCTCTCCACTTAGGGACTGGGGAACTCCTTAGCTAAGCTAATTACCCCTGGATCAAAATTGGGGCCGCATTGTTCTCTGTCCTGCACAGTTCCCTGCTCGAATCGTTGCTTCCTCTGCCAGGATCCTGGCTTTTCCTACCAGAGAATGTTGAGGTTCTCTGTCATTTTCGCTGCTAATGTGTCTTCACTAATCGTGTTTCCGGGAGACTTGCCACCCCTAAAGGAGCTTGTTGAACACTTGTCACCAGACCTAAGTCCTTAGGGATGAACCTGTGTCTCGGCAC\n', '+\n', 'CCCCBGC9FGCD-GGGFGGGEFEFF@CCGFGGGGFF+DG6,F69+FGG+FGF&lt;FCC;GGEBFGGEDECFGGG6GDG7BFG9GCG9G9GDGGGG&lt;GDFFFGEGC,&lt;GC=GFG:,:A:EE+GD&lt;:E,FGF=FCAGEFF++G,G,&lt;EGGFGFFA&gt;*FG9EGFCFCGC,B&gt;43GFF@E@2&gt;9ED@,568D=A;C+=5@E/F53D&gt;,6F*9@+,E,C=*0+*@56&lt;*9@:/F,1,:*)*14*0CC=*1)*0*,*)0;+)*))/**)**;0;//9)1+)10.()5/)070*)7*C**3/)(2*****\n']
</code></pre>

<python><fastq>
Why does this naive Python paired Fastq reader work?
# Why does this naive Python paired Fastq reader work?

(Asked by: username_0 on 2020-02-04)

I wrote some throwaway code for ingesting paired-end reads from a Fastq file into memory.
def ingest(stream1, stream2):
    while True:
        record1, record2 = list(), list()
        for _ in range(4):
            record1.append(next(stream1))
            record2.append(next(stream2))
        yield record1, record2

I had initially put the for loop in a try/Except block, anticipating I'd need to explicitly break out of the endless while loop. However, the code seems to work correctly even after I removed it. I even "corrupted" the final read in one of the paired files, and the ingest function performs reasonably (the last read pair is never yielded).
My question is...why does this code work, since I never explicitly break out of the loop?
Here's an example of how the code is invoked.
>>> with open('seqR1.fastq', 'r') as fh1, open('seqR2.fastq', 'r') as fh2:
...     counter = 0
...     for r1, r2 in interleave(fh1, fh2):
...         counter += 1
>>> counter
5000
>>> r1
['@read5000/1\n', 'AGGTCTGGGGACAGGTGTTTTACAGGCTCCTTTAGGGGTGGCAAGTCTCCCGGAAACATGATTAGTGAAGACACGTTAACAGCGAAACTGACAGAGAACCTCAACATTCTCTGGTAGGAAAAGCCAGGATCCTGGCAGAGGAAGCAACGATTCGAGCAGGGAACTGTGCAGGACAGAGAACAATGCGGCCCCAATTTTGATCCAGGGGTAGTTAGCTTAGCTAAGGAGTTCCCCAGTCCCTAAGTGGAGAGGGGATCCAGAACCCTACAATTTATTATTATATCTTAGCGGATATATCTAT\n', '+\n', 'CCCCCGGGEGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@GGGGGGGFAGGGCGGGGGFGGGGGGGGGCGGGGGDGGGGFGGGGGGGGGCGGGGGGGGGGGGGGGCGGGGGGGGGGGGGGGGG@G7)FD*6G7D)9;*C:)4D5+9>@G*=0):)*;1*504\n']
>>> r2
['@read5000/2\n', 'TGTGGGGTTCTGGATCCCCTCTCCACTTAGGGACTGGGGAACTCCTTAGCTAAGCTAATTACCCCTGGATCAAAATTGGGGCCGCATTGTTCTCTGTCCTGCACAGTTCCCTGCTCGAATCGTTGCTTCCTCTGCCAGGATCCTGGCTTTTCCTACCAGAGAATGTTGAGGTTCTCTGTCATTTTCGCTGCTAATGTGTCTTCACTAATCGTGTTTCCGGGAGACTTGCCACCCCTAAAGGAGCTTGTTGAACACTTGTCACCAGACCTAAGTCCTTAGGGATGAACCTGTGTCTCGGCAC\n', '+\n', 'CCCCBGC9FGCD-GGGFGGGEFEFF@CCGFGGGGFF+DG6,F69+FGG+FGF*FG9EGFCFCGC,B>43GFF@E@2>9ED@,568D=A;C+=5@E/F53D>,6F*9@+,E,C=*0+*@56<*9@:/F,1,:*)*14*0CC=*1)*0*,*)0;+)*))/**)**;0;//9)1+)10.()5/)070*)7*C**3/)(2*****\n']

**Tags:** python, fastq

---

## 2 answers

### Answer by username_2 on 2020-02-04. Score: 4 | Accepted answer

Your code only runs prior to Python 3.7, now it raises RuntimeError: generator raised StopIteration. This is essentially a bug fix described in PEP 479 -- Change StopIteration handling inside generators. You can enable the new behaviour from Python 3.5 via from __future__ import generator_stop.
Note, it's not related to file-handles per se, you can reproduce it with a smaller example with a simple iterator:
it = iter([1,2])

def f(it):
    while True:
        yield next(it)

print(list(f(it)))

You should not write code relying on the bug


---

### Answer by username_1 on 2020-02-04. Score: 2

EDIT: Chris_Rands's answer is more authoritative and clarifies that this behavior is essentially a bug.  (I had only tested my example on Python 3.6.8 and hadn't seen PEP 479.)
When next runs out of lines, it raises StopIteration, and apparently that stops the while loop similarly to calling break without you needing to catch the exception.  I couldn't see a definitive source on that but you can see it in action if you catch and re-raise the exception:
def ingest2(stream1, stream2):
    while True:
        try:
            record1, record2 = list(), list()
            for _ in range(4):
                record1.append(next(stream1))
                record2.append(next(stream2))
            yield record1, record2
        except StopIteration:
            print("StopIteration raised")
            raise

You can also try to catch it outside the loop, but it doesn't get re-raised at that point.  (If you try raising one yourself in a while loop, though, it doesn't catch it in the same way, so there must be some extra Python magic going on.  I can't see any difference between the two exception objects so I'd be curious what's going on there.)  Neat example.  You could also make a pair of FASTQ parsers using Biopython's SeqIO.parse for more complex/permanent situations if needed.


---

