<p>I have a simple python app running through the blockchain as follows...</p>
<pre><code>rpc_connection = AuthServiceProxy(&quot;http://%s:%s@127.0.0.1:8332&quot;%(rpc_user, rpc_password))
for b in range(blockStart, blockEnd): 
    block = rpc_connection.getblock(rpc_connection.getblockhash(b)) 
    for txid in block['tx']:
        raw_tx = rpc_connection.getrawtransaction(txid)
        decoded_tx = rpc_connection.decoderawtransaction(raw_tx)
        for output in decoded_tx['vout']:
            print(rpc_connection.deriveaddresses(output['scriptPubKey']['desc']))
</code></pre>
<p>It successfully runs through all blocks from block 1 until block 115,257 where it encounters &quot;BrokenPipeError: [Errno 32] Broken pipe&quot;.  It errors on transaction 11 (with txid <code>1c67abba71d570d0f0643b3d4ed93f528a68ef1c6a21e6d1f72535a15b0ce030</code>).  Sometimes it errors on the rpc_connection.getrawtransaction line and sometimes it errors on the rpc_connection.deriveaddresses line but both times with a Broken pipe.</p>
<p>What's strange is that if I just run the script for the single transaction it works fine and I notice that the transaction before transaction 11 (i.e. transaction 10 with txid <code>c45b2940e403a1598faaae33a24e385c92b4f051929a49fe3b079ef04b1e2dbc</code>) returns an enormous result for rpc_connection.getrawtransaction() (pages and pages of console ouput) whereas all the others are about 3 lines of console output, and so I'm wondering if perhaps this is related to memory usage or timeouts or some conf setting that I don't have?</p>
<p>Has anyone run into a similar problem before?  Also, I'm guessing everyone gets the same enormous result for <code>rpc_connection.getrawtransaction('c45b2940e403a1598faaae33a24e385c92b4f051929a49fe3b079ef04b1e2dbc')</code>?  Really appreciate any thoughts on the best way around this problem.</p>

CC BY-SA 4.0
2023-02-22T11:14:53.460Z
What are you doing with the output of your script? Normally, `BrokenPipeError`s occur when the process reading your script terminates early, e.g `python myscript.py | head -n 100`. `head` will terminate after 100 lines, but your script will still try to write to the pipe, which causes the `BrokenPipeError`. See https://stackoverflow.com/questions/26692284/how-to-prevent-brokenpipeerror-when-doing-a-flush-in-python for more info
josie
CC BY-SA 4.0
2023-02-22T11:46:03.863Z
thanks @josie - instead of the final "print" line in my example, I'm actually just appending to a list of dicts which once I exit the loop for a given block gets written to a file.  The list is about 600k lines large so far which seems relatively small.  And I'm just running directly like python3 myscript.py.
d3wannabe
CC BY-SA 4.0
2023-02-22T11:49:25.573Z
p.s. @josie - I noticed this morning it's actually throwing a new error (but at the exact same line): ConnectionResetError: [Errno 54] Connection reset by peer
d3wannabe
CC BY-SA 4.0
2023-02-22T14:12:22.330Z
It's difficult to tell what is a python error and what is a `bitcoind` error. perhaps you can try first running your script on `signet`. This will have fewer transactions and blocks and should at least tell you if the issue is related to the amount of data.
josie
CC BY-SA 4.0
2023-02-22T17:33:22.037Z
thanks @josie I'm pretty sure it's related to the volume data - I manually inputted the decoded transaction json and continued processing and ran into the same problem at block 118,597 right after another transaction which contains 438 vins.  hmmmm, not sure where to go with the troubleshooting now
d3wannabe
CC BY-SA 4.0
2023-02-22T17:55:46.747Z
just found a hacky fix: if I encounter a transaction with greater than a certain number of vins (around 200 in my experience), then I just reinstantiate the rpc_connection with `rpc_connection = AuthServiceProxy("http://%s:%s@127.0.0.1:8332"%(rpc_user, rpc_password))` and it works.  Not sure if there's a more elegant way to flush/restart the rpc_connection or whether I'm spawning multiple sessions, but at least there's no more connection errors
d3wannabe
<bitcoin-core><json-rpc><python>
Bitcoin RPC BrokenPipeError on specific block/transaction
d3wannabe
# Bitcoin RPC BrokenPipeError on specific block/transaction

(Asked by: username_0 on 2023-02-21)

I have a simple python app running through the blockchain as follows...

```
rpc_connection = AuthServiceProxy("http://%s:%s@127.0.0.1:8332"%(rpc_user, rpc_password))
for b in range(blockStart, blockEnd): 
    block = rpc_connection.getblock(rpc_connection.getblockhash(b)) 
    for txid in block['tx']:
        raw_tx = rpc_connection.getrawtransaction(txid)
        decoded_tx = rpc_connection.decoderawtransaction(raw_tx)
        for output in decoded_tx['vout']:
            print(rpc_connection.deriveaddresses(output['scriptPubKey']['desc']))

```

It successfully runs through all blocks from block 1 until block 115,257 where it encounters "BrokenPipeError: [Errno 32] Broken pipe". It errors on transaction 11 (with txid `1c67abba71d570d0f0643b3d4ed93f528a68ef1c6a21e6d1f72535a15b0ce030`). Sometimes it errors on the rpc\_connection.getrawtransaction line and sometimes it errors on the rpc\_connection.deriveaddresses line but both times with a Broken pipe.

What's strange is that if I just run the script for the single transaction it works fine and I notice that the transaction before transaction 11 (i.e. transaction 10 with txid `c45b2940e403a1598faaae33a24e385c92b4f051929a49fe3b079ef04b1e2dbc`) returns an enormous result for rpc\_connection.getrawtransaction() (pages and pages of console ouput) whereas all the others are about 3 lines of console output, and so I'm wondering if perhaps this is related to memory usage or timeouts or some conf setting that I don't have?

Has anyone run into a similar problem before? Also, I'm guessing everyone gets the same enormous result for `rpc_connection.getrawtransaction('c45b2940e403a1598faaae33a24e385c92b4f051929a49fe3b079ef04b1e2dbc')`? Really appreciate any thoughts on the best way around this problem.

**Tags:** bitcoin-core, json-rpc, python

> **Comments:**
> **username_1**: What are you doing with the output of your script? Normally, `BrokenPipeError`s occur when the process reading your script terminates early, e.g `python myscript.py | head -n 100`. `head` will terminate after 100 lines, but your script will still try to write to the pipe, which causes the `BrokenPipeError`. See https://stackoverflow.com/questions/26692284/how-to-prevent-brokenpipeerror-when-doing-a-flush-in-python for more info  
> **username_0**: thanks @username_1 - instead of the final "print" line in my example, I'm actually just appending to a list of dicts which once I exit the loop for a given block gets written to a file.  The list is about 600k lines large so far which seems relatively small.  And I'm just running directly like python3 myscript.py.  
> **username_0**: p.s. @username_1 - I noticed this morning it's actually throwing a new error (but at the exact same line): ConnectionResetError: [Errno 54] Connection reset by peer  
> **username_1**: It's difficult to tell what is a python error and what is a `bitcoind` error. perhaps you can try first running your script on `signet`. This will have fewer transactions and blocks and should at least tell you if the issue is related to the amount of data.  
> **username_0**: thanks @username_1 I'm pretty sure it's related to the volume data - I manually inputted the decoded transaction json and continued processing and ran into the same problem at block 118,597 right after another transaction which contains 438 vins.  hmmmm, not sure where to go with the troubleshooting now  
> **username_0**: just found a hacky fix: if I encounter a transaction with greater than a certain number of vins (around 200 in my experience), then I just reinstantiate the rpc_connection with `rpc_connection = AuthServiceProxy("http://%s:%s@127.0.0.1:8332"%(rpc_user, rpc_password))` and it works.  Not sure if there's a more elegant way to flush/restart the rpc_connection or whether I'm spawning multiple sessions, but at least there's no more connection errors  

---

## 0 answers

