<p>output is spent (e.g. if appear in &quot;vin&quot; to some other transaction), I store an indicator.</p>
<p>Now single block is processed for 1-2 sec.</p>
<p>Implication of this is,<br />
if you want to check address balance, you will need to check each &quot;input&quot; and to compile list of unspent inputs.</p>
<p>Still unsure if this is best way to go.</p>
<h3>Update 1</h3>
<p>I finally upload the software in github here:<br />
<a href="https://github.com/nmmmnu/BlockchainWalker" rel="nofollow noreferrer">https://github.com/nmmmnu/BlockchainWalker</a></p>
<h3>Update 2</h3>
<p>Here are how keys looks like if we want to find the balance for address &quot;1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6&quot;:</p>
<pre><code>127.0.0.1:2000&gt; HGETALL a:1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6
 1) &quot;a:1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0&quot;
 2) &quot;0.01270199&quot;
</code></pre>
<p>This is funding transaction:</p>
<ul>
<li>28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0, output 0</li>
</ul>
<p>This is value:</p>
<ul>
<li>0.01270199</li>
</ul>
<p>Then we need to check if this is spent, so we check each input as follows:</p>
<pre><code>127.0.0.1:2000&gt; get t:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0:s
&quot;055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8&quot;
</code></pre>
<p>Transaction 055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8, is the transaction where this output is spent.</p>
<p>As each output is spent this means the address 1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6 have balance of zero.</p>
<hr />
<p>In case we want to check / list a transaction &quot;055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8&quot;, we can do:</p>
<pre><code>127.0.0.1:2000&gt; HGETALL t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8
 1) &quot;t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.-:i:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0&quot;
 2) &quot;1&quot;
 3) &quot;t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.-:i:39edb8741b701b6da2dbc4e02290e8e78cba244bdbad96da203e41ee2704c525.0&quot;
 4) &quot;1&quot;
 5) &quot;t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.0:o&quot;
 6) &quot;1Kj76Sxe8c3UK85RAQwwdqScAxaBwAY2eb:0.00500000&quot;
 7) &quot;t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.0:s&quot;
 8) &quot;49f8bd582439a3b2351f92e0fdb5fcb1032acd42e3ab469c16d805627889ce14&quot;
 9) &quot;t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.1:o&quot;
10) &quot;14yAJga4ZkULbaMz4LUW5vj8GhQYyzCPoW:0.01118727&quot;
11) &quot;t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.1:s&quot;
12) &quot;edf55ace1396229c7a91a4b21cd63293c631ae2ddcc5987402b04ed540b9e5d8&quot;
</code></pre>
<p>Those &quot;t:xxxxx.-:i:xxxx&quot; are inputs.</p>
<p>Those &quot;t:xxxxx.N:o&quot; are outputs</p>
<p>Those &quot;t:xxxxx.N:s&quot; are indicators, if output is spent or not.</p>
<hr />
<p>Please note this is not Redis, it is database I developing that have Redis interface. You can check it here:</p>
<p><a href="https://github.com/nmmmnu/HM4" rel="nofollow noreferrer">https://github.com/nmmmnu/HM4</a></p>

CC BY-SA 3.0
2017-09-14T18:37:32.003Z
How do you handle blockchain reorgs?
Jakub Fedyczak
CC BY-SA 3.0
2017-09-14T18:47:59.563Z
honestly, no idea what this is. By try and error, I did this code: https://github.com/nmmmnu/BlockchainWalker/blob/master/blockchainwalker.php#L121 , it handles all transactions so far, tried 07-2017 to 08-2017, then beginning to end of 2016
Nick
CC BY-SA 3.0
2017-09-14T18:48:32.677Z
I also handle transactions with wrong addresses like 71bbaef28e09d8d6fadd41f053db7768dbb5fa4570f06b961dfc29db3dc00b1d
Nick
CC BY-SA 3.0
2017-09-10T11:56:28.370Z
2017-09-14T15:19:18.670Z
2020-06-16T11:10:03.350Z
Nick
<p>I am using <a href="http://rocksdb.org" rel="nofollow noreferrer">RocksDB</a> and <a href="https://nodejs.org/en/" rel="nofollow noreferrer">node.js</a>. My database is optimized for tracking address balances and payment history. It needs around 143GB of storage, 4 days to load. Current blocks import below 2s.</p>

<p>I've tried very hard to achieve this with PostgreSQL. It is doable, but requires a lot of hacking and post-import index creating. I've tried levelDB but it is roughly 4 times slower than rocksDB on SSD drive.</p>

<p>I am using custom block parsing library and half-baked full-node implementation.</p>

CC BY-SA 3.0
2017-09-14T14:58:40.477Z
When I said redis, I meant my own database server similar to LevelDB. Is now 4th day of loading, but I believe tomorrow evening will be ready. However question was how to store data in best way. I am using RAID 5 on spinning HDD's. Currently my database use 260 GB, but it have create and expiration time on each of its keys
Nick
CC BY-SA 3.0
2017-09-14T15:19:56.823Z
I post big update, please check and let me know if you have thoughts about it :)
Nick
CC BY-SA 3.0
2017-09-16T13:40:04.437Z
total time, less than 6 days. size on disk almost 400 GB.
Nick
CC BY-SA 3.0
2017-09-14T14:27:31.487Z
2017-09-14T14:27:31.487Z
Jakub Fedyczak
<p>I am trying to develop something similar to <code>bitcoin-abe</code>, e.g. import whole blockchain into database.</p>

<p>For database I currently using Redis - I know it will took lots of RAM, but lets suppose that is OK.</p>

<p>Another possibilities could be MySQL, Postgress or even CSV file.</p>

<p>I have <code>bitcoind</code> and I am importing block after block.</p>

<p>For each transaction, I need to look for output of the transactions that "fund" current transaction - I hope you understand what I mean.</p>

<p>I am currently checking with blocks from 2017. Most of blocks have 2000+ transactions. If we assume each transaction have single input, this means I need to look at 2000 more transactions.</p>

<p>At the moment those 2000 lookups are done back to <code>bitcoind</code>, because output database could be csv file.</p>

<p>I can do all this in "lazy" way, but then I will not be able to find if some address is spent.</p>

<p>Is there any faster way to import?</p>

<p>I am using PHP and single block from 2017 is imported for about 1 min.</p>

<bitcoind><blockchain.info><php><import>
development of bitcoin-abe like software
Nick
# development of bitcoin-abe like software

(Asked by: username_0 on 2017-09-10)

I am trying to develop something similar to `bitcoin-abe`, e.g. import whole blockchain into database.

For database I currently using Redis - I know it will took lots of RAM, but lets suppose that is OK.

Another possibilities could be MySQL, Postgress or even CSV file.

I have `bitcoind` and I am importing block after block.

For each transaction, I need to look for output of the transactions that "fund" current transaction - I hope you understand what I mean.

I am currently checking with blocks from 2017. Most of blocks have 2000+ transactions. If we assume each transaction have single input, this means I need to look at 2000 more transactions.

At the moment those 2000 lookups are done back to `bitcoind`, because output database could be csv file.

I can do all this in "lazy" way, but then I will not be able to find if some address is spent.

Is there any faster way to import?

I am using PHP and single block from 2017 is imported for about 1 min.

**Tags:** bitcoind, blockchain.info, php, import


---

## 2 answers

### Answer by username_0 on 2017-09-10. Score: 1

output is spent (e.g. if appear in "vin" to some other transaction), I store an indicator.

Now single block is processed for 1-2 sec.

Implication of this is,  
if you want to check address balance, you will need to check each "input" and to compile list of unspent inputs.

Still unsure if this is best way to go.

### Update 1

I finally upload the software in github here:  
<https://github.com/nmmmnu/BlockchainWalker>

### Update 2

Here are how keys looks like if we want to find the balance for address "1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6":

```
127.0.0.1:2000> HGETALL a:1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6
 1) "a:1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0"
 2) "0.01270199"

```

This is funding transaction:

* 28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0, output 0

This is value:

* 0.01270199

Then we need to check if this is spent, so we check each input as follows:

```
127.0.0.1:2000> get t:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0:s
"055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8"

```

Transaction 055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8, is the transaction where this output is spent.

As each output is spent this means the address 1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6 have balance of zero.

---

In case we want to check / list a transaction "055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8", we can do:

```
127.0.0.1:2000> HGETALL t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8
 1) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.-:i:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0"
 2) "1"
 3) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.-:i:39edb8741b701b6da2dbc4e02290e8e78cba244bdbad96da203e41ee2704c525.0"
 4) "1"
 5) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.0:o"
 6) "1Kj76Sxe8c3UK85RAQwwdqScAxaBwAY2eb:0.00500000"
 7) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.0:s"
 8) "49f8bd582439a3b2351f92e0fdb5fcb1032acd42e3ab469c16d805627889ce14"
 9) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.1:o"
10) "14yAJga4ZkULbaMz4LUW5vj8GhQYyzCPoW:0.01118727"
11) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.1:s"
12) "edf55ace1396229c7a91a4b21cd63293c631ae2ddcc5987402b04ed540b9e5d8"

```

Those "t:xxxxx.-:i:xxxx" are inputs.

Those "t:xxxxx.N:o" are outputs

Those "t:xxxxx.N:s" are indicators, if output is spent or not.

---

Please note this is not Redis, it is database I developing that have Redis interface. You can check it here:

<https://github.com/nmmmnu/HM4>

> **Comments:**
> **username_1**: How do you handle blockchain reorgs?  
> **username_0**: honestly, no idea what this is. By try and error, I did this code: https://github.com/nmmmnu/BlockchainWalker/blob/master/blockchainwalker.php#L121 , it handles all transactions so far, tried 07-2017 to 08-2017, then beginning to end of 2016  
> **username_0**: I also handle transactions with wrong addresses like 71bbaef28e09d8d6fadd41f053db7768dbb5fa4570f06b961dfc29db3dc00b1d  

---

### Answer by username_1 on 2017-09-14. Score: 0

I am using [RocksDB](http://rocksdb.org) and [node.js](https://nodejs.org/en/). My database is optimized for tracking address balances and payment history. It needs around 143GB of storage, 4 days to load. Current blocks import below 2s.

I've tried very hard to achieve this with PostgreSQL. It is doable, but requires a lot of hacking and post-import index creating. I've tried levelDB but it is roughly 4 times slower than rocksDB on SSD drive.

I am using custom block parsing library and half-baked full-node implementation.

> **Comments:**
> **username_0**: When I said redis, I meant my own database server similar to LevelDB. Is now 4th day of loading, but I believe tomorrow evening will be ready. However question was how to store data in best way. I am using RAID 5 on spinning HDD's. Currently my database use 260 GB, but it have create and expiration time on each of its keys  
> **username_0**: I post big update, please check and let me know if you have thoughts about it :)  
> **username_0**: total time, less than 6 days. size on disk almost 400 GB.  

---

