<p>This topic had me searching quite a lot as this post is almost all there is as form of documentation (thank Pieter for that). The missing piece, at least from my perspective, is that those fields are encoded using <a href="https://developers.google.com/protocol-buffers/docs/encoding" rel="nofollow noreferrer">base 128 / Varint</a>. To make things more difficult, this is <em>not</em> the same <code>Varint</code> as in sizes representations in serialized Bitcoin block (as explained <a href="https://learnmeabitcoin.com/guide/varint" rel="nofollow noreferrer">here</a>).</p>

<p>So, in an attempt to bring some light on this topic, here are 2 pieces of code, one in python, and the other in golang, that will decode and read those fields using a <code>base128</code> decoder function. The first one is adapted from the book "The Undocumented Internals of Bitcoin, Ethereum and blockchains".<br>
The golang version is my own adaptation from this python code.</p>

<p>python version:</p>



<pre><code>import leveldb
import sys

def reverse(hash):
    return "".join(reversed([hash[i:i+2] for i in range(0, len(hash), 2)]))

def base128(ans, off):
    n = 0
    while True:
        ch = ord(ans[off:off+1])
        print ch
        off = off + 1
        n = (n &lt;&lt; 7) | (ch &amp; 0x7f)
        if ch &amp; 0x80 == 128:
            n = n + 1
        else:
            return(n, off)

hash = "62" + reverse(sys.argv[1])
hash = hash.decode('hex')
db = leveldb.LevelDB('./testindex')
ans = db.Get(hash)

v, o = base128(ans, 0)
print v, o
v, o = base128(ans, o)
print v, o
# etc ...
</code></pre>

<p>golang version:</p>



<pre><code>package main

import (
    "encoding/hex"
    "fmt"
    "log"
    "os"

    "github.com/syndtr/goleveldb/leveldb"
)

func reverse(s []byte) []byte {
    for i, j := 0, len(s)-1; i &lt; j; i, j = i+1, j-1 {
        s[i], s[j] = s[j], s[i]
    }
    return s
}

// binary.Uvarint is supposed to do this, except it doesn't.
// And it is not the same varint as in bitcoin serialized varints.
// this is ported from "The Undocumented Internals of The Bitcoin
// and Ethereum Blockchains" python code
func base128(b []byte, offset uint64) (uint64, uint64) {
    for n := 0; ; n++ {
        ch := int(b[offset])
        offset++
        n = (n &lt;&lt; 7) | (ch &amp; 0x7f)
        if ch&amp;0x80 != 128 {
            return uint64(n), offset
        }
    }
    return 0, 0
}

func main() {
    db, err := leveldb.OpenFile("./testindex", nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    h, err := hex.DecodeString(os.Args[1])
    if err != nil {
        log.Fatal(err)
    }

    b := append([]byte("b"), reverse(h)...)

    data, err := db.Get(b, nil)
    if err != nil {
        log.Fatal(err)
    }

    v, o := base128(data, 0)
    fmt.Printf("%d %d\n", v, o)
    v, o = base128(data, o)
    fmt.Printf("%d %d\n", v, o)
    // etc...
}
</code></pre>

<p>HTH</p>

CC BY-SA 4.0
2019-12-03T15:30:52.697Z
2019-12-03T15:30:52.697Z
iMil
<p>Ok, I have produced a standalone code snippet that can decode the block index records stored in the levelDB database. The challenge here, is that bitcoin core appears to use an extremely non-standard variable length integer representation. It does <em>not</em> match 'varint128' which is used by Google protobufs and other projects. It's very similar, but there is a real difference.</p>
<p>In this code snippet you pass the raw data stored in the levelDB record and it will populate a data structure with the results.</p>
<p>You can find the code reference here:</p>
<p><a href="https://github.com/jratcliff63367/bitcoinstats/blob/main/include/CBlockIndex.h" rel="nofollow noreferrer">https://github.com/jratcliff63367/bitcoinstats/blob/main/include/CBlockIndex.h</a></p>

CC BY-SA 4.0
2022-04-15T15:44:43.840Z
2022-04-15T15:44:43.840Z
John Ratcliff
<p><a href="https://github.com/bitcoin/bitcoin/blob/v0.15.1/src/chain.h#L387L407" rel="noreferrer">This</a> is the Bitcoin Core v0.15.1 source code snippet that defines the serialization of <code>CDiskBlockIndex</code>. The various fields are:</p>

<ul>
<li>A varint version number of the client who wrote the record (for future extensibility). This is the <code>87c628</code> you're seeing.</li>
<li>A varint block height.</li>
<li>A varint status with various <a href="https://github.com/bitcoin/bitcoin/blob/v0.15.1/src/chain.h#L126L162" rel="noreferrer">flags</a>.</li>
<li>A varint determining the number of transactions in the block.</li>
<li>If the status field has bits 8 or 16 set, a varint determining which file (blk*.dat and rev*.dat) the block is stored in.</li>
<li>If the status field has bit 8 set, a varint determining the offset into the blk*.dat file where the start of the block is.</li>
<li>if the status field has bit 16 set, a varint determining the offset into the rev*.dat file where the start of the block's undo data is.</li>
<li>The block header</li>
</ul>

CC BY-SA 3.0
2018-01-06T14:06:58.913Z
I've now managed to decode the first five fields: version, height, status, num of txs and block number. I'm getting incorrect values for the data position, but I now know that I'm on the right track. Thank you for taking the time to answer.
Arjen
CC BY-SA 3.0
2018-01-06T14:39:07.543Z
I was a bit too fast: I had a bug in my VarInt parser. All is good now. Thanks again!
Arjen
CC BY-SA 3.0
2018-01-05T11:01:11.877Z
2018-01-05T11:01:11.877Z
Pieter Wuille
<p>I am trying to use bitcoind's LevelDB block index files (that are stored in <code>blocks/index</code>) to find the file (e.g. <code>blk00029.dat</code>) and byte position of specfic blocks. </p>

<p>Thanks to questions <a href="https://bitcoin.stackexchange.com/questions/11104/what-is-the-database-for">What is the database for?</a> and <a href="https://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair">What are the keys used in the blockchain levelDB (ie what are the key:value pairs)?</a> I know where to look, however the contents of the database key for a block puzzles me. Even with the help of the <a href="https://github.com/bitcoin/bitcoin/blob/0.15/src/chain.h#L386" rel="nofollow noreferrer">source code</a> I cannot figure out it's format.</p>

<p>For example for the block at height 100,000, which resides under database key <code>b\x06\xe53\xfd\x1a\xda\x869\x1f?l42\x04\xb0\xd2x\xd4\xaa\xec\x1c\x0b \xaa'\xba\x03\x00\x00\x00\x00\x00</code> (block's hash string converted to bytes in  little endian order), I am getting the following content which I hexlified for readability:</p>

<p><code>87c628858c201d04009be99e5883d1e92c0100000050120119172a610421a6c3011dd330d9df07b63616c2cc1f1cd00200000000006657a9252aacd5c0b2940996ecff952228c3067cc38d4885efb5a4ac4247e9f337221b4d4c86041b0f2b5710</code></p>

<p>The first 3 bytes (<code>87c628</code>) seem to be the same for every block that I get from the database. I'm guessing it is some version identifier?</p>

<p>The last 80 bytes (or 160 characters in hex) are the block header: <code>010000005 ... f2b5710</code>. They match what you get by executing <code>bitcoin-cli getblock 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 false</code>.</p>

<p>The rest is a mystery to me. I know that block 100,000 had 4 transactions, but the byte that holds the value <code>4</code> has <code>129</code> for block #1, which had only a single transaction. </p>

<p>Where can I find out about the format of, at least, the block keys in LevelDB?</p>

<p>Could it be that the format of each key depends on the version of bitcoind that serialized the contents? If so, how do I figure out which format to use?</p>

CC BY-SA 3.0
2018-01-04T22:42:11.217Z
LevelDB has its own database format and compression stuff that may make reading the data directly from disk as binary data to be difficult. You're probably running into some of that stuff.
Ava Chow
CC BY-SA 3.0
2018-01-05T08:55:20.933Z
You're right, that's why I am reading the LevelDB using a Python library (`plyvel`). That seems to be working correctly because I can retrieve records for blocks (`bX*32` records) and files (`fXXXX`). I want to use the info stored in these records to seek through the .blk files directly.
Arjen
CC BY-SA 4.0
2018-08-18T18:59:41.867Z
Arjen can you please publish your code for retrieving the records using plyvel?
Adam Rosenthal
<bitcoind><leveldb>
Format of a block key's contents in bitcoind's LevelDB?
Arjen
# Format of a block key's contents in bitcoind's LevelDB?

(Asked by: username_0 on 2018-01-04)

I am trying to use bitcoind's LevelDB block index files (that are stored in `blocks/index`) to find the file (e.g. `blk00029.dat`) and byte position of specfic blocks.

Thanks to questions [What is the database for?](https://bitcoin.stackexchange.com/questions/11104/what-is-the-database-for) and [What are the keys used in the blockchain levelDB (ie what are the key:value pairs)?](https://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair) I know where to look, however the contents of the database key for a block puzzles me. Even with the help of the [source code](https://github.com/bitcoin/bitcoin/blob/0.15/src/chain.h#L386) I cannot figure out it's format.

For example for the block at height 100,000, which resides under database key `b\x06\xe53\xfd\x1a\xda\x869\x1f?l42\x04\xb0\xd2x\xd4\xaa\xec\x1c\x0b \xaa'\xba\x03\x00\x00\x00\x00\x00` (block's hash string converted to bytes in little endian order), I am getting the following content which I hexlified for readability:

`87c628858c201d04009be99e5883d1e92c0100000050120119172a610421a6c3011dd330d9df07b63616c2cc1f1cd00200000000006657a9252aacd5c0b2940996ecff952228c3067cc38d4885efb5a4ac4247e9f337221b4d4c86041b0f2b5710`

The first 3 bytes (`87c628`) seem to be the same for every block that I get from the database. I'm guessing it is some version identifier?

The last 80 bytes (or 160 characters in hex) are the block header: `010000005 ... f2b5710`. They match what you get by executing `bitcoin-cli getblock 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 false`.

The rest is a mystery to me. I know that block 100,000 had 4 transactions, but the byte that holds the value `4` has `129` for block #1, which had only a single transaction.

Where can I find out about the format of, at least, the block keys in LevelDB?

Could it be that the format of each key depends on the version of bitcoind that serialized the contents? If so, how do I figure out which format to use?

**Tags:** bitcoind, leveldb

> **Comments:**
> **username_1**: LevelDB has its own database format and compression stuff that may make reading the data directly from disk as binary data to be difficult. You're probably running into some of that stuff.  
> **username_0**: You're right, that's why I am reading the LevelDB using a Python library (`plyvel`). That seems to be working correctly because I can retrieve records for blocks (`bX*32` records) and files (`fXXXX`). I want to use the info stored in these records to seek through the .blk files directly.  
> **username_2**: username_0 can you please publish your code for retrieving the records using plyvel?  

---

## 3 answers

### Answer by username_5 on 2018-01-05. Score: 5 | Accepted answer

[This](https://github.com/bitcoin/bitcoin/blob/v0.15.1/src/chain.h#L387L407) is the Bitcoin Core v0.15.1 source code snippet that defines the serialization of `CDiskBlockIndex`. The various fields are:

* A varint version number of the client who wrote the record (for future extensibility). This is the `87c628` you're seeing.
* A varint block height.
* A varint status with various [flags](https://github.com/bitcoin/bitcoin/blob/v0.15.1/src/chain.h#L126L162).
* A varint determining the number of transactions in the block.
* If the status field has bits 8 or 16 set, a varint determining which file (blk\*.dat and rev\*.dat) the block is stored in.
* If the status field has bit 8 set, a varint determining the offset into the blk\*.dat file where the start of the block is.
* if the status field has bit 16 set, a varint determining the offset into the rev\*.dat file where the start of the block's undo data is.
* The block header

> **Comments:**
> **username_0**: I've now managed to decode the first five fields: version, height, status, num of txs and block number. I'm getting incorrect values for the data position, but I now know that I'm on the right track. Thank you for taking the time to answer.  
> **username_0**: I was a bit too fast: I had a bug in my VarInt parser. All is good now. Thanks again!  

---

### Answer by username_3 on 2019-12-03. Score: 0

This topic had me searching quite a lot as this post is almost all there is as form of documentation (thank username_5 for that). The missing piece, at least from my perspective, is that those fields are encoded using [base 128 / Varint](https://developers.google.com/protocol-buffers/docs/encoding). To make things more difficult, this is *not* the same `Varint` as in sizes representations in serialized Bitcoin block (as explained [here](https://learnmeabitcoin.com/guide/varint)).

So, in an attempt to bring some light on this topic, here are 2 pieces of code, one in python, and the other in golang, that will decode and read those fields using a `base128` decoder function. The first one is adapted from the book "The Undocumented Internals of Bitcoin, Ethereum and blockchains".  
The golang version is my own adaptation from this python code.

python version:

```
import leveldb
import sys

def reverse(hash):
    return "".join(reversed([hash[i:i+2] for i in range(0, len(hash), 2)]))

def base128(ans, off):
    n = 0
    while True:
        ch = ord(ans[off:off+1])
        print ch
        off = off + 1
        n = (n << 7) | (ch & 0x7f)
        if ch & 0x80 == 128:
            n = n + 1
        else:
            return(n, off)

hash = "62" + reverse(sys.argv[1])
hash = hash.decode('hex')
db = leveldb.LevelDB('./testindex')
ans = db.Get(hash)

v, o = base128(ans, 0)
print v, o
v, o = base128(ans, o)
print v, o
# etc ...

```

golang version:

```
package main

import (
    "encoding/hex"
    "fmt"
    "log"
    "os"

    "github.com/syndtr/goleveldb/leveldb"
)

func reverse(s []byte) []byte {
    for i, j := 0, len(s)-1; i < j; i, j = i+1, j-1 {
        s[i], s[j] = s[j], s[i]
    }
    return s
}

// binary.Uvarint is supposed to do this, except it doesn't.
// And it is not the same varint as in bitcoin serialized varints.
// this is ported from "The Undocumented Internals of The Bitcoin
// and Ethereum Blockchains" python code
func base128(b []byte, offset uint64) (uint64, uint64) {
    for n := 0; ; n++ {
        ch := int(b[offset])
        offset++
        n = (n << 7) | (ch & 0x7f)
        if ch&0x80 != 128 {
            return uint64(n), offset
        }
    }
    return 0, 0
}

func main() {
    db, err := leveldb.OpenFile("./testindex", nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    h, err := hex.DecodeString(os.Args[1])
    if err != nil {
        log.Fatal(err)
    }

    b := append([]byte("b"), reverse(h)...)

    data, err := db.Get(b, nil)
    if err != nil {
        log.Fatal(err)
    }

    v, o := base128(data, 0)
    fmt.Printf("%d %d\n", v, o)
    v, o = base128(data, o)
    fmt.Printf("%d %d\n", v, o)
    // etc...
}

```

HTH


---

### Answer by username_4 on 2022-04-15. Score: 0

Ok, I have produced a standalone code snippet that can decode the block index records stored in the levelDB database. The challenge here, is that bitcoin core appears to use an extremely non-standard variable length integer representation. It does *not* match 'varint128' which is used by Google protobufs and other projects. It's very similar, but there is a real difference.

In this code snippet you pass the raw data stored in the levelDB record and it will populate a data structure with the results.

You can find the code reference here:

<https://github.com/jratcliff63367/bitcoinstats/blob/main/include/CBlockIndex.h>


---

