<p>I held a research period on bitcoin (thanks to this project I became fond of Bitcoin) with the release of the alpha software at December, I had to extract information for the construction of an information graph.</p>
<p>I believe in my opinion that there are really many ways to index information but I believe that the fastest ones are to use the same technique that uses bitcoins.</p>
<ul>
<li>Parser blk files</li>
<li>Deserialize the information in a format of your choice and index it in DB</li>
<li>query the DB and make a read in the appropriate file.</li>
</ul>
<p>There are two very difficult things</p>
<ul>
<li>the construction of this indexing system</li>
<li>if you want to access the id wallet you need to build a script decompiler or use the bitcoin node</li>
</ul>
<p>ps: this is my experience, there can be other ways to build this information, which I don't know (mine is just a three-year thesis, my path has just begun)</p>
<p>This is a good <a href="https://github.com/alecalve/python-bitcoin-blockchain-parser" rel="nofollow noreferrer">parser</a> for blk, one of the few compatible with Segregated Witness.</p>
<p>The <a href="https://github.com/bitcoin-abe/bitcoin-abe" rel="nofollow noreferrer">abe</a> project still works but his work times seem unacceptable with the current bitcoin dimension</p>
<p>If you have a big pc, this <a href="https://github.com/citp/BlockSci" rel="nofollow noreferrer">project</a> is very fast (at least they say so)</p>
<h2>Update 2021</h2>
<p>The blockchain of Bitcoin grows over time and I think that the result of my work is <a href="https://github.com/vincenzopalazzo/SpyCblock" rel="nofollow noreferrer">SpyCBlock</a> can be an efficient solution in terms of memory RAM, I don't know in terms of space.</p>
<p>SpyCBlock offers different deserialization methods like JSON, transactions graph, and a deprecated address graph that use Bitcoin Core to make some work. In addition, a method to deserialize in an efficient way the Bitcoin mempool is under development.</p>
<p>Please note that the software is an academics software, developed by only me, maybe can have some mistake and can be difficult to install all the dependencies, but if it will compile, it should be work fine with the integral bitcoin data.</p>
<p>P.S: The development of BlockSci is stopped in 2020, I don't know why, but I think because it is difficult to scale with the actual dimension of Bitcoin blockchain.</p>

CC BY-SA 4.0
2019-09-03T20:45:35.800Z
2021-02-27T21:20:57.550Z
2021-02-27T21:20:57.550Z
vincenzopalazzo
<p>The fastest (simpliest) way is to use my <a href="https://github.com/ragestack/blockchain-parser" rel="nofollow noreferrer">blockchain parser</a> which has just 240 lines of code and able to extract all information from blkXXXXX.dat files to the simple text view. It parse whole the raw bitcoin blockchain database that is stored in those files.</p>

<p>Then you can easily index the data as you need and then make any data manipulations for your academic research. For example, you can build UTXO set by yourself, or make some ECDSA signatures or keys research and so on.</p>

<p>Also, you can learn how the data is stored in blockchain if you check how my parser works with the sequences of bytes.</p>

<p>I'am sure it's the fastest, simpliest and most usefull way to learn the blockchain functionality.</p>

CC BY-SA 4.0
2020-05-31T02:26:34.737Z
2020-06-01T16:47:32.000Z
2020-06-01T16:47:32.000Z
Denis Leonov
<p>If you want to read the raw <code>blkXXXX</code>s, <a href="https://github.com/rustyrussell/bitcoin-iterate" rel="nofollow noreferrer">bitcoin-iterate</a> by Rusty Russel is probably the best option.</p>
<p>For more advanced queries, you'll need an indexer. <a href="https://github.com/dpc/rust-bitcoin-indexer" rel="nofollow noreferrer">Rust-bitcoin-indexer</a> by dbc
and <a href="https://github.com/Blockstream/electrs" rel="nofollow noreferrer">Blockstream's revamped <code>electrs</code></a> are two very different yet powerful alternatives.</p>

CC BY-SA 4.0
2021-02-27T12:22:28.787Z
2021-02-27T12:22:28.787Z
Antoine Poinsot
<p>Depends on what you feel comfortable with. </p>

<p>You can go low level and query the lebeldb directly. The block index (information for every block) is in <code>$DATADIR/blocks/index</code> and the chainstate (information about the currently best known chain) in <code>$DATADIR/chainstate</code>.</p>

<p>An easier way is to have a fully synced bitcoind running locally and query over jsonrpc. From there you slice and dice the resulting JSON to fit your needs.</p>

<p>I haven't done leveldb querying, but I'd expect there to be a significant amount of work involved since it's just a key value store.</p>

CC BY-SA 4.0
2019-04-07T14:05:56.230Z
The idea is to try and track coins over long periods of time with an innovative approach. So even bitcoind won't really cut it I think.
DDecoene
CC BY-SA 4.0
2019-04-07T21:07:10.233Z
Neither of those LevelDB databases maintained by bitcoind actually contain the information you want. Maybe some more modern indexing software like electrs is more appropriate.
Pieter Wuille
CC BY-SA 4.0
2019-04-06T18:11:01.493Z
2019-04-06T18:11:01.493Z
thalisk
<p>I would like to do some research of the Bitcoin blockchain. Because i would like to do massive amounts of processing and lookups, I need a fast way to search the blockchain.</p>

<p>Http requests to insight.io just won't cut it...</p>

<p>I know of ABE but it seems no longer maintained and I don't know if it is up to par with the current implementation of the blockchain.</p>

<p>The environment I'm programming in is python.</p>

<p>Any ideas?</p>

<blockchain><academic-research><abe>
What is the fastest way to research Bitcoin block chain?
DDecoene
# What is the fastest way to research Bitcoin block chain?

(Asked by: username_0 on 2019-04-06)

I would like to do some research of the Bitcoin blockchain. Because i would like to do massive amounts of processing and lookups, I need a fast way to search the blockchain.

Http requests to insight.io just won't cut it...

I know of ABE but it seems no longer maintained and I don't know if it is up to par with the current implementation of the blockchain.

The environment I'm programming in is python.

Any ideas?

**Tags:** blockchain, academic-research, abe


---

## 4 answers

### Answer by username_1 on 2019-09-03. Score: 2

I held a research period on bitcoin (thanks to this project I became fond of Bitcoin) with the release of the alpha software at December, I had to extract information for the construction of an information graph.

I believe in my opinion that there are really many ways to index information but I believe that the fastest ones are to use the same technique that uses bitcoins.

* Parser blk files
* Deserialize the information in a format of your choice and index it in DB
* query the DB and make a read in the appropriate file.

There are two very difficult things

* the construction of this indexing system
* if you want to access the id wallet you need to build a script decompiler or use the bitcoin node

ps: this is my experience, there can be other ways to build this information, which I don't know (mine is just a three-year thesis, my path has just begun)

This is a good [parser](https://github.com/alecalve/python-bitcoin-blockchain-parser) for blk, one of the few compatible with Segregated Witness.

The [abe](https://github.com/bitcoin-abe/bitcoin-abe) project still works but his work times seem unacceptable with the current bitcoin dimension

If you have a big pc, this [project](https://github.com/citp/BlockSci) is very fast (at least they say so)

## Update 2021

The blockchain of Bitcoin grows over time and I think that the result of my work is [SpyCBlock](https://github.com/username_1/SpyCblock) can be an efficient solution in terms of memory RAM, I don't know in terms of space.

SpyCBlock offers different deserialization methods like JSON, transactions graph, and a deprecated address graph that use Bitcoin Core to make some work. In addition, a method to deserialize in an efficient way the Bitcoin mempool is under development.

Please note that the software is an academics software, developed by only me, maybe can have some mistake and can be difficult to install all the dependencies, but if it will compile, it should be work fine with the integral bitcoin data.

P.S: The development of BlockSci is stopped in 2020, I don't know why, but I think because it is difficult to scale with the actual dimension of Bitcoin blockchain.


---

### Answer by username_2 on 2020-05-31. Score: 0

The fastest (simpliest) way is to use my [blockchain parser](https://github.com/ragestack/blockchain-parser) which has just 240 lines of code and able to extract all information from blkXXXXX.dat files to the simple text view. It parse whole the raw bitcoin blockchain database that is stored in those files.

Then you can easily index the data as you need and then make any data manipulations for your academic research. For example, you can build UTXO set by yourself, or make some ECDSA signatures or keys research and so on.

Also, you can learn how the data is stored in blockchain if you check how my parser works with the sequences of bytes.

I'am sure it's the fastest, simpliest and most usefull way to learn the blockchain functionality.


---

### Answer by username_3 on 2021-02-27. Score: 0

If you want to read the raw `blkXXXX`s, [bitcoin-iterate](https://github.com/rustyrussell/bitcoin-iterate) by Rusty Russel is probably the best option.

For more advanced queries, you'll need an indexer. [Rust-bitcoin-indexer](https://github.com/dpc/rust-bitcoin-indexer) by dbc
and [Blockstream's revamped `electrs`](https://github.com/Blockstream/electrs) are two very different yet powerful alternatives.


---

### Answer by username_4 on 2019-04-06. Score: 0

Depends on what you feel comfortable with.

You can go low level and query the lebeldb directly. The block index (information for every block) is in `$DATADIR/blocks/index` and the chainstate (information about the currently best known chain) in `$DATADIR/chainstate`.

An easier way is to have a fully synced bitcoind running locally and query over jsonrpc. From there you slice and dice the resulting JSON to fit your needs.

I haven't done leveldb querying, but I'd expect there to be a significant amount of work involved since it's just a key value store.

> **Comments:**
> **username_0**: The idea is to try and track coins over long periods of time with an innovative approach. So even bitcoind won't really cut it I think.  
> **username_5**: Neither of those LevelDB databases maintained by bitcoind actually contain the information you want. Maybe some more modern indexing software like electrs is more appropriate.  

---

