<p>You're going to see much better performance by doing the parsing of block data yourself directly off disk, though that involves a decent amount of complexity.</p>
<p>If you want to stick with <code>bitcoind</code> RPC calls, I suggest using <code>getblock(blockhash, 2)</code>, which will decode all transactions in the block for you in a single result, removing the need for <code>decoderawtransaction</code>.</p>
<p>Lastly, why the roundtrip through <code>deriveaddresses</code>? <code>output['scriptPubKey']['address']</code> should give you the address directly.</p>

CC BY-SA 4.0
2023-02-24T16:29:18.870Z
wow - thanks so much for the tip on `getblock(blockhash, 2)` - didn't know that was possible but definitely sounds more efficient.  On your second question, sometimes the ['address'] key simply isn't present - in fact normally it isn't - if I could reliably access that field for every transaction I would definitely use it.  Not sure if there's any well-known non-RPC methods (or python tools) to derive an address from a descriptor?
d3wannabe
CC BY-SA 4.0
2023-02-24T17:13:06.183Z
You can just check whether the field exists, before accessing it.
Pieter Wuille
CC BY-SA 4.0
2023-02-24T15:58:32.643Z
2023-02-24T15:58:32.643Z
Pieter Wuille
<p>I have a python app to track and store utxo in a simple/flat structure that looks like this...</p>
<pre class="lang-python prettyprint-override"><code>from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException 

# track the utxo
trackUtxoList = []

# rpc call 1
rpc_connection = AuthServiceProxy(&quot;http://%s:%s@127.0.0.1:8332&quot;%(rpc_user, rpc_password))

for b in range(blockStart, blockLimit): 

    # rpc call 2 (per block)
    blockhash = rpc_connection.getblockhash(b)

    # rpc call 3 (per block)
    block = rpc_connection.getblock(blockhash)

    for txid in block['tx']:
        
        # rpc call 4 (per tx)
        raw_tx = rpc_connection.getrawtransaction(txid,False,blockhash) 

        # rpc call 5 (per tx)
        decoded_tx = rpc_connection.decoderawtransaction(raw_tx)

        for output in decoded_tx['vout']:

            # rpc call 6 (per vout)
            for l in rpc_connection.deriveaddresses(output['scriptPubKey']['desc']):
                trackUtxoList.append({'address':str(l),'value':output['value'],'block':b, 'txid':txid,'isUtxo':True})

        for output in decoded_tx['vin']:

            vinSearch = list(filter(lambda item: item['t'] == output['txid'], trackUtxoList))
            vinSearch[output['vout']]['isUtxo'] = False
</code></pre>
<p>The problem is that it runs very slowly for 2 reasons that relate to my 2 questions...</p>
<ol>
<li><p>My blocks directory is on an external hard drive because it's too big for my laptop - pretty sure this will impact read-speeds and so I'm wondering if it's possible to somehow split the blocks directory so that for example, I could pull 100k blocks onto my laptop, process that, then delete and copy the next 100k blocks onto my laptop etc.  I've already split the blocks directory (on external hdd) from the rest of the data directory (on my laptop) but can't find any examples of people temporarily splitting the blocks directory itself.</p>
</li>
<li><p>I'm making 2 rpc calls per block, 2 per transaction and 1 per vout.  To my eye it looks like calls 5 + 6 (as commented in my code example) don't really need to use my rpc_connection since they are manipulations on data already pulled from the blockchain?  Whereas the others are essentials/unavoidable?  If that's correct, are there any python libraries that are typically used to decode transactions and derive addresses without using an rpc call?  I played a little with Buterin's <a href="https://pypi.org/project/bitcoin/" rel="nofollow noreferrer">https://pypi.org/project/bitcoin/</a> but since it's no longer maintained I wasn't sure whether it's my best bet.</p>
</li>
</ol>
<p>Thanks for any thoughts at all on how to optimise my code to parse the entire blockchain</p>

<bitcoin-core><json-rpc><python><utxo>
Optimising blockchain parsing with rpc + python
d3wannabe
# Optimising blockchain parsing with rpc + python

(Asked by: username_0 on 2023-02-24)

I have a python app to track and store utxo in a simple/flat structure that looks like this...

```python
from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException 

# track the utxo
trackUtxoList = []

# rpc call 1
rpc_connection = AuthServiceProxy("http://%s:%s@127.0.0.1:8332"%(rpc_user, rpc_password))

for b in range(blockStart, blockLimit): 

    # rpc call 2 (per block)
    blockhash = rpc_connection.getblockhash(b)

    # rpc call 3 (per block)
    block = rpc_connection.getblock(blockhash)

    for txid in block['tx']:
        
        # rpc call 4 (per tx)
        raw_tx = rpc_connection.getrawtransaction(txid,False,blockhash) 

        # rpc call 5 (per tx)
        decoded_tx = rpc_connection.decoderawtransaction(raw_tx)

        for output in decoded_tx['vout']:

            # rpc call 6 (per vout)
            for l in rpc_connection.deriveaddresses(output['scriptPubKey']['desc']):
                trackUtxoList.append({'address':str(l),'value':output['value'],'block':b, 'txid':txid,'isUtxo':True})

        for output in decoded_tx['vin']:

            vinSearch = list(filter(lambda item: item['t'] == output['txid'], trackUtxoList))
            vinSearch[output['vout']]['isUtxo'] = False

```

The problem is that it runs very slowly for 2 reasons that relate to my 2 questions...

1. My blocks directory is on an external hard drive because it's too big for my laptop - pretty sure this will impact read-speeds and so I'm wondering if it's possible to somehow split the blocks directory so that for example, I could pull 100k blocks onto my laptop, process that, then delete and copy the next 100k blocks onto my laptop etc. I've already split the blocks directory (on external hdd) from the rest of the data directory (on my laptop) but can't find any examples of people temporarily splitting the blocks directory itself.
2. I'm making 2 rpc calls per block, 2 per transaction and 1 per vout. To my eye it looks like calls 5 + 6 (as commented in my code example) don't really need to use my rpc\_connection since they are manipulations on data already pulled from the blockchain? Whereas the others are essentials/unavoidable? If that's correct, are there any python libraries that are typically used to decode transactions and derive addresses without using an rpc call? I played a little with Buterin's <https://pypi.org/project/bitcoin/> but since it's no longer maintained I wasn't sure whether it's my best bet.

Thanks for any thoughts at all on how to optimise my code to parse the entire blockchain

**Tags:** bitcoin-core, json-rpc, python, utxo


---

## 1 answers

### Answer by username_1 on 2023-02-24. Score: 5 | Accepted answer

You're going to see much better performance by doing the parsing of block data yourself directly off disk, though that involves a decent amount of complexity.

If you want to stick with `bitcoind` RPC calls, I suggest using `getblock(blockhash, 2)`, which will decode all transactions in the block for you in a single result, removing the need for `decoderawtransaction`.

Lastly, why the roundtrip through `deriveaddresses`? `output['scriptPubKey']['address']` should give you the address directly.

> **Comments:**
> **username_0**: wow - thanks so much for the tip on `getblock(blockhash, 2)` - didn't know that was possible but definitely sounds more efficient.  On your second question, sometimes the ['address'] key simply isn't present - in fact normally it isn't - if I could reliably access that field for every transaction I would definitely use it.  Not sure if there's any well-known non-RPC methods (or python tools) to derive an address from a descriptor?  
> **username_1**: You can just check whether the field exists, before accessing it.  

---

