<p>Possibly checkout ABE - Alternative Block Parser. It uses python and a SQL-based database but you can hook up other databases based on the driver.</p>

<p><a href="https://github.com/jtobey/bitcoin-abe" rel="nofollow">https://github.com/jtobey/bitcoin-abe</a></p>

CC BY-SA 3.0
2013-06-18T17:59:49.473Z
This is a legitimate option. However, it's getting to be unrealistically slow as the blockchain grows. With SQLite3, it now takes days, from what I can tell.
Mike
CC BY-SA 3.0
2013-06-18T21:03:11.030Z
Well start it, then have a client that keeps it up to date. Yes, it'll grow, as you're asking for a database that will encompass the blockchain. There's no immediate solution to that unless you want a database that has a reference to points in the chain. Did you want something "fast"? Use LevelDB.
Dominic Tancredi
CC BY-SA 3.0
2013-06-27T22:39:54.993Z
Or mcdee's blockparser.
Mike
CC BY-SA 3.0
2013-06-18T17:20:58.077Z
2013-06-18T17:20:58.077Z
Dominic Tancredi
<p>You can always use <a href="http://blockchainsql.io" rel="nofollow">BlockchainSQL Server</a> for SQL querying the Bitcoin blockchain.</p>

<p>The web version is free, the server is commercial.</p>

<p><b>Disclaimer:</b> I am the developer.</p>

CC BY-SA 3.0
2017-12-21T06:15:34.907Z
Website seems down
Shayan
CC BY-SA 3.0
2016-06-28T04:42:02.243Z
2016-06-28T04:42:02.243Z
Herman Schoenfeld
<p>To transfer to a SQL Server database you can try BitcoinDatabaseGenerator.<br>
You can download it from <a href="https://github.com/ladimolnar/BitcoinDatabaseGenerator/releases" rel="nofollow">https://github.com/ladimolnar/BitcoinDatabaseGenerator/releases</a><br>
Sources and wiki: <a href="https://github.com/ladimolnar/BitcoinDatabaseGenerator" rel="nofollow">https://github.com/ladimolnar/BitcoinDatabaseGenerator</a></p>

<p>The transfer is very fast. At the time of writing this, the transfer of 266 blockchain files totaling 33 GB took less than one hour and 30 minutes on my development machine. Of course, the time will vary depending on hardware and configuration. </p>

CC BY-SA 3.0
2015-06-08T09:53:04.023Z
is it working under linux?
jangorecki
CC BY-SA 3.0
2015-06-09T21:05:23.973Z
BitcoinDatabaseGenerator is Windows only.
Ladi
CC BY-SA 3.0
2015-05-08T05:03:24.800Z
2015-05-08T05:03:24.800Z
Ladi
<p>I created a <a href="https://github.com/mcdee/blockparser" rel="noreferrer">fork of blockparser</a> with a 'csvdump' command that will generate a details dump of the blockchain in CSV format.  There are also some support scripts in that repository for creation, population and periodic update of a PostgreSQL database.</p>

<p>Be aware that the resultant database is pretty large; you might want to consider PostgreSQL over SQLite for this one.</p>

CC BY-SA 3.0
2017-06-18T18:18:26.223Z
Would h2 work? I am interested in storing only the UTXO set, indexed by addresses.
Jus12
CC BY-SA 3.0
2013-06-18T15:44:10.327Z
I remember blockparser being slow as sin for single transaction and account data when I first tried it, but I'll give the CSV dump a shot and see how long it takes. Do you remember how long it took for you? If so, what were the specs of the machine you ran it on?
Mike
CC BY-SA 3.0
2013-06-18T16:38:33.133Z
blockparser is relatively CPU and memory intensive.  I just carried out a full csvdump on my machine and it took 9 mins, maxing out at just under 9GB memory used.  It will definitely help if you have a decent CPU and SSDs for storage.
jgm
CC BY-SA 3.0
2013-06-18T18:01:15.183Z
I'm currently running the SQL dump. It's much faster than Bitcoin-Abe; it looks like it's only going to take about five hours. However, I wonder whether it parses the transaction data or just leaves it in raw hex.
Mike
CC BY-SA 3.0
2013-06-18T18:03:05.160Z
Depends what you mean by 'data'.  Information is supplied on individual transaction inputs and outputs, but there is no work carried out to parse the signatures.
jgm
CC BY-SA 3.0
2013-06-21T23:43:26.310Z
This works great, but I have one qualm: why don't the transactions have timestamps?
Mike
CC BY-SA 3.0
2013-06-22T08:12:58.723Z
Individual transactions do not have timestamps in the blockchain.  See https://en.bitcoin.it/wiki/Transaction for details
jgm
CC BY-SA 3.0
2013-06-24T22:28:22.583Z
I can't find another way to contact you, but you should consider changing the sql.cpp file in that fork to just make temporary CSV dumps and then parse them into a database using the MariaDB (or equivalent) LOAD DATA INFILE command. That would take less than an hour on a weak machine, where as sql.cpp takes days. I'd help, but I don't know C++.
Mike
CC BY-SA 3.0
2013-06-24T23:26:54.987Z
Haven't looked at the existing sql.cpp much because it didn't gather all the info that I needed.  Feel free to fork znort's code and send him a pull request.
jgm
CC BY-SA 3.0
2013-06-27T21:46:15.793Z
Also, have you noticed that InputTxHash and InputTxIndex are always null in outputs.csv?
Mike
CC BY-SA 3.0
2013-06-27T22:07:53.110Z
Yes they will be.  You'll need to post-process the data to link the inputs to the outputs.  There is some example SQL to do this for PostgreSQL somewhere in the database load files.
jgm
CC BY-SA 3.0
2013-06-18T08:29:38.367Z
2013-06-18T08:29:38.367Z
jgm
<p>I need to generate a SQL database (preferably SQLite3) of every transaction that has occurred in the blockchain. Is there an efficient way to do this using open-source software? I'm attempting to do it myself using a Haskell script that makes system calls to bitcoind but I'm having trouble making it sufficiently fast.</p>

CC BY-SA 3.0
2013-06-18T02:07:07.983Z
Does it have to be SQL or can you use LevelDB?
Dominic Tancredi
CC BY-SA 3.0
2013-06-18T15:40:24.777Z
I know that bitcoind (if you changed the settings, which I have) creates a LevelDB database for its own use. However, there is very little documentation on LevelDB in general, and even less in my language of choice (Haskell). I want something that I can easily use in a clearly correct way in code that analyzes the data, and SQL seems like the only option for that.
Mike
<blockchain><raw-data>
Reliable, efficient way to parse the blockchain into a SQL database
Mike
# Reliable, efficient way to parse the blockchain into a SQL database

(Asked by: username_0 on 2013-06-17)

I need to generate a SQL database (preferably SQLite3) of every transaction that has occurred in the blockchain. Is there an efficient way to do this using open-source software? I'm attempting to do it myself using a Haskell script that makes system calls to bitcoind but I'm having trouble making it sufficiently fast.

**Tags:** blockchain, raw-data

> **Comments:**
> **username_1**: Does it have to be SQL or can you use LevelDB?  
> **username_0**: I know that bitcoind (if you changed the settings, which I have) creates a LevelDB database for its own use. However, there is very little documentation on LevelDB in general, and even less in my language of choice (Haskell). I want something that I can easily use in a clearly correct way in code that analyzes the data, and SQL seems like the only option for that.  

---

## 4 answers

### Answer by username_6 on 2013-06-18. Score: 7 | Accepted answer

I created a [fork of blockparser](https://github.com/mcdee/blockparser) with a 'csvdump' command that will generate a details dump of the blockchain in CSV format. There are also some support scripts in that repository for creation, population and periodic update of a PostgreSQL database.

Be aware that the resultant database is pretty large; you might want to consider PostgreSQL over SQLite for this one.

> **Comments:**
> **username_0**: I remember blockparser being slow as sin for single transaction and account data when I first tried it, but I'll give the CSV dump a shot and see how long it takes. Do you remember how long it took for you? If so, what were the specs of the machine you ran it on?  
> **username_6**: blockparser is relatively CPU and memory intensive.  I just carried out a full csvdump on my machine and it took 9 mins, maxing out at just under 9GB memory used.  It will definitely help if you have a decent CPU and SSDs for storage.  
> **username_0**: I'm currently running the SQL dump. It's much faster than Bitcoin-Abe; it looks like it's only going to take about five hours. However, I wonder whether it parses the transaction data or just leaves it in raw hex.  
> **username_6**: Depends what you mean by 'data'.  Information is supplied on individual transaction inputs and outputs, but there is no work carried out to parse the signatures.  
> **username_0**: This works great, but I have one qualm: why don't the transactions have timestamps?  
> **username_6**: Individual transactions do not have timestamps in the blockchain.  See https://en.bitcoin.it/wiki/Transaction for details  
> **username_0**: I can't find another way to contact you, but you should consider changing the sql.cpp file in that fork to just make temporary CSV dumps and then parse them into a database using the MariaDB (or equivalent) LOAD DATA INFILE command. That would take less than an hour on a weak machine, where as sql.cpp takes days. I'd help, but I don't know C++.  
> **username_6**: Haven't looked at the existing sql.cpp much because it didn't gather all the info that I needed.  Feel free to fork znort's code and send him a pull request.  
> **username_0**: Also, have you noticed that InputTxHash and InputTxIndex are always null in outputs.csv?  
> **username_6**: Yes they will be.  You'll need to post-process the data to link the inputs to the outputs.  There is some example SQL to do this for PostgreSQL somewhere in the database load files.  
> **username_7**: Would h2 work? I am interested in storing only the UTXO set, indexed by addresses.  

---

### Answer by username_1 on 2013-06-18. Score: 3

Possibly checkout ABE - Alternative Block Parser. It uses python and a SQL-based database but you can hook up other databases based on the driver.

<https://github.com/jtobey/bitcoin-abe>

> **Comments:**
> **username_0**: This is a legitimate option. However, it's getting to be unrealistically slow as the blockchain grows. With SQLite3, it now takes days, from what I can tell.  
> **username_1**: Well start it, then have a client that keeps it up to date. Yes, it'll grow, as you're asking for a database that will encompass the blockchain. There's no immediate solution to that unless you want a database that has a reference to points in the chain. Did you want something "fast"? Use LevelDB.  
> **username_0**: Or mcdee's blockparser.  

---

### Answer by username_2 on 2016-06-28. Score: 3

You can always use [BlockchainSQL Server](http://blockchainsql.io) for SQL querying the Bitcoin blockchain.

The web version is free, the server is commercial.

**Disclaimer:** I am the developer.

> **Comments:**
> **username_3**: Website seems down  

---

### Answer by username_4 on 2015-05-08. Score: 1

To transfer to a SQL Server database you can try BitcoinDatabaseGenerator.  
You can download it from <https://github.com/ladimolnar/BitcoinDatabaseGenerator/releases>  
Sources and wiki: <https://github.com/ladimolnar/BitcoinDatabaseGenerator>

The transfer is very fast. At the time of writing this, the transfer of 266 blockchain files totaling 33 GB took less than one hour and 30 minutes on my development machine. Of course, the time will vary depending on hardware and configuration.

> **Comments:**
> **username_5**: is it working under linux?  
> **username_4**: BitcoinDatabaseGenerator is Windows only.  

---

