<p>You are confusing the <code>VarInt</code> and <code>CompactSize</code> encoding. <code>CompactSize</code> encoding (also confusingly referred to as &quot;varint&quot; outside Bitcoin Core) is used in the Bitcoin protocol. <code>VarInt</code> encoding is internal to Bitcoin Core and used as a more compact encoding in the internal database.</p>
<p>See <a href="https://bitcoin.stackexchange.com/a/114585/101498">this answer</a> for details about the <code>VarInt</code> format:</p>
<blockquote>
<p>Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make sure the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number:</p>
<pre><code>(a[len-1] &amp; 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] &amp; 0x7F)+1))
</code></pre>
</blockquote>

CC BY-SA 4.0
2024-02-16T09:03:57.597Z
Thanks for this. I've updated my code on the question, and it does seem to extract 3 integers correctly from every value as expected. However, I was not able to actually find the transactions at those offsets as explained in the question now, I wonder what I'm doing wrong.
Ciro Santilli OurBigBook.com
CC BY-SA 4.0
2024-02-15T09:53:28.627Z
2024-02-15T09:53:28.627Z
Antoine Poinsot
<p>The keys I understand, <code>t</code> + 32-byte hash.</p>
<p>But my problem are the values. I understand from sources such as <a href="https://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair">What are the keys used in the blockchain levelDB (ie what are the key:value pairs)?</a> that the values should encode three values: dat file number, block offset, and tx offset within block.</p>
<p>But I've noticed that each value has a different sizes between 5 and 10 on the first thousand entries, so I'm not sure how to decode the values into those three fields. Are those fields simply 3 varint values?</p>
<p>Here's my Plyvel code that prints out the lengths using plyvel==1.5.1, Bitcoin Core v26.0.0 on Ubuntu 23.10:</p>
<pre><code>#!/usr/bin/env python3

import struct

import plyvel

def decode_varint(data):
    &quot;&quot;&quot;
    https://github.com/alecalve/python-bitcoin-blockchain-parser/blob/c06f420995b345c9a193c8be6e0916eb70335863/blockchain_parser/utils.py#L41
    &quot;&quot;&quot;
    assert(len(data) &gt; 0)
    size = int(data[0])
    assert(size &lt;= 255)

    if size &lt; 253:
        return size, 1

    if size == 253:
        format_ = '&lt;H'
    elif size == 254:
        format_ = '&lt;I'
    elif size == 255:
        format_ = '&lt;Q'
    else:
        # Should never be reached
        assert 0, &quot;unknown format_ for size : %s&quot; % size

    size = struct.calcsize(format_)
    return struct.unpack(format_, data[1:size+1])[0], size + 1

ldb = plyvel.DB('/home/ciro/snap/bitcoin-core/common/.bitcoin/indexes/txindex/', compression=None)
i = 0
for key, value in ldb:
    if key[0:1] == b't':
        txid = bytes(reversed(key[1:])).hex()
        print(i)
        print(txid)
        print(len(value))
        print(value.hex(' '))
        value = bytes(reversed(value))
        file, off = decode_varint(value)
        blk_off, off = decode_varint(value[off:])
        tx_off, off = decode_varint(value[off:])
        print((txid, file, blk_off, tx_off))
        print()
        i += 1
</code></pre>
<p>but it eventually blows up at:</p>
<pre><code>131344
ec4de461b0dd1350b7596f95c0d7576aa825214d9af0e8c54de567ab0ce70800
7
42 ff c0 43 8b 94 35
Traceback (most recent call last):
  File &quot;/home/ciro/bak/git/bitcoin-strings-with-txids/./tmp.py&quot;, line 39, in &lt;module&gt;
    blk_off, off = decode_varint(value[off:])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File &quot;/home/ciro/bak/git/bitcoin-strings-with-txids/./tmp.py&quot;, line 29, in decode_varint
    return struct.unpack(format_, data[1:size+1])[0], size + 1
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack requires a buffer of 8 bytes
</code></pre>
<p>So I wonder if I guessed the format wrong, or if it's just a bug in my code.</p>
<p>Comparing to: <a href="https://en.bitcoin.it/wiki/Protocol_documentation#Variable_length_integer" rel="nofollow noreferrer">https://en.bitcoin.it/wiki/Protocol_documentation#Variable_length_integer</a> I would decode:</p>
<pre><code>42 ff c0 43 8b 94 35
</code></pre>
<p>manually as:</p>
<ul>
<li>42</li>
<li>ff: expect 8 bytes next
<ul>
<li>c0 43 8b 94 35: only 5 bytes left, blowup</li>
</ul>
</li>
</ul>
<p>I also tried to inverse value:</p>
<pre><code>value = bytes(reversed(value))
</code></pre>
<p>but then it blows up very early, definitely wrong.</p>
<p>I also tried to ignore the error to see if there are others, but there were hundreds them, so something is definitely wrong with my method.</p>
<p>Related:</p>
<ul>
<li><a href="https://bitcointalk.org/index.php?topic=1068721.0" rel="nofollow noreferrer">https://bitcointalk.org/index.php?topic=1068721.0</a></li>
</ul>
<p><strong>Update</strong></p>
<p>Given Antoine's answer, I've updated my code to:</p>
<pre><code>#!/usr/bin/env python3

import struct

import plyvel

def decode_varint(data):
    i = 0
    ret = 0
    while True:
        b = data[i]
        ret += b &amp; 0x7F
        if b &amp; 0x80:
            b += 1
        else:
            return (ret, i + 1)
        ret &lt;&lt;= 7
        i += 1

ldb = plyvel.DB('/home/ciro/snap/bitcoin-core/common/.bitcoin/indexes/txindex/', compression=None)
i = 0
nerrs = 0
for key, value in ldb:
    if key[0:1] == b't':
        if i % 1000000 == 0:
            print(i//1000000)
        txid = bytes(reversed(key[1:])).hex()
        print(value.hex(' '))
        total_off = 0
        file, off = decode_varint(value)
        total_off += off
        blk_off, off = decode_varint(value[total_off:])
        total_off += off
        tx_off, off = decode_varint(value[total_off:])
        total_off += off
        assert total_off == len(value)
        print((txid, file, blk_off, tx_off))
        i += 1
print(f'nerrs: {nerrs}')
</code></pre>
<p>and all length asserts seem to pass which is promising. However, when I tried to verify the first few values:</p>
<pre><code>93 1a 96 8c e7 74 a7 bb 0a
('d176d4960a78b41971f9d19207b59af6584b16ef323de55e983aec0100000000', 2458, 46347252, 646538)
93 1a aa d0 f0 72 86 90 01
('4d19a5ba6a5455a6d4d46d6999fb0a4a7b29e603fe229cd4f1a34d0300000000', 2458, 89405554, 100353)
93 1e d4 cf 18 82 be 76
('ba75d6ac075959632d9e012f0321c367a54e9f2bde2e2bebce6baa0300000000', 2462, 1386392, 40822)
93 18 a8 96 8e 5a 8a d3 14
('7df98bab9d89fc14d0dec365fa124b3e112017c6b8f56f64cf159e0600000000', 2456, 84248410, 174484)
</code></pre>
<p>I didn't find a match between:</p>
<pre><code>bitcoin-core.cli getrawtransaction d176d4960a78b41971f9d19207b59af6584b16ef323de55e983aec0100000000
tail -c+$((46347252 + 646538 - 1)) ~/snap/bitcoin-core/common/.bitcoin/blocks/blk02458.dat  | head -c 2000 | xxd -p | tr -d '\n'

bitcoin-core.cli getrawtransaction 4d19a5ba6a5455a6d4d46d6999fb0a4a7b29e603fe229cd4f1a34d0300000000
tail -c+$((89405554 + 100353 - 1)) ~/snap/bitcoin-core/common/.bitcoin/blocks/blk02458.dat  | head -c 2000 | xxd -p | tr -d '\n'

bitcoin-core.cli getrawtransaction ba75d6ac075959632d9e012f0321c367a54e9f2bde2e2bebce6baa0300000000
tail -c+$((89405554 + 100353 - 1)) ~/snap/bitcoin-core/common/.bitcoin/blocks/blk02462.dat  | head -c 2000 | xxd -p | tr -d '\n'
</code></pre>
<p>so either my parsing or my checks are wrong somehow.</p>
<p>I've searched for the bytes that <code>getrawtransactions</code> gave me for d176d4960a78b41971f9d19207b59af6584b16ef323de55e983aec0100000000 using <code>bgrep</code>: <a href="https://unix.stackexchange.com/questions/223078/best-way-to-grep-a-big-binary-file/758528#758528">https://unix.stackexchange.com/questions/223078/best-way-to-grep-a-big-binary-file/758528#758528</a> and found it at:</p>
<pre><code>~/snap/bitcoin-core/common/.bitcoin/blocks/blk02586.dat: 02ed92ce
</code></pre>
<p>and offset 0x02ed92ce = 49124046, so these would be the expected correct values for this one.</p>

<bitcoin-core-development><database><txindex>
What is the data format layout for txindex LevelDB values?
Ciro Santilli OurBigBook.com
# What is the data format layout for txindex LevelDB values?

(Asked by: username_0 on 2024-02-15)

The keys I understand, `t` + 32-byte hash.

But my problem are the values. I understand from sources such as [What are the keys used in the blockchain levelDB (ie what are the key:value pairs)?](https://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair) that the values should encode three values: dat file number, block offset, and tx offset within block.

But I've noticed that each value has a different sizes between 5 and 10 on the first thousand entries, so I'm not sure how to decode the values into those three fields. Are those fields simply 3 varint values?

Here's my Plyvel code that prints out the lengths using plyvel==1.5.1, Bitcoin Core v26.0.0 on Ubuntu 23.10:

```
#!/usr/bin/env python3

import struct

import plyvel

def decode_varint(data):
    """
    https://github.com/alecalve/python-bitcoin-blockchain-parser/blob/c06f420995b345c9a193c8be6e0916eb70335863/blockchain_parser/utils.py#L41
    """
    assert(len(data) > 0)
    size = int(data[0])
    assert(size <= 255)

    if size < 253:
        return size, 1

    if size == 253:
        format_ = '<H'
    elif size == 254:
        format_ = '<I'
    elif size == 255:
        format_ = '<Q'
    else:
        # Should never be reached
        assert 0, "unknown format_ for size : %s" % size

    size = struct.calcsize(format_)
    return struct.unpack(format_, data[1:size+1])[0], size + 1

ldb = plyvel.DB('/home/username_0/snap/bitcoin-core/common/.bitcoin/indexes/txindex/', compression=None)
i = 0
for key, value in ldb:
    if key[0:1] == b't':
        txid = bytes(reversed(key[1:])).hex()
        print(i)
        print(txid)
        print(len(value))
        print(value.hex(' '))
        value = bytes(reversed(value))
        file, off = decode_varint(value)
        blk_off, off = decode_varint(value[off:])
        tx_off, off = decode_varint(value[off:])
        print((txid, file, blk_off, tx_off))
        print()
        i += 1

```

but it eventually blows up at:

```
131344
ec4de461b0dd1350b7596f95c0d7576aa825214d9af0e8c54de567ab0ce70800
7
42 ff c0 43 8b 94 35
Traceback (most recent call last):
  File "/home/username_0/bak/git/bitcoin-strings-with-txids/./tmp.py", line 39, in <module>
    blk_off, off = decode_varint(value[off:])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username_0/bak/git/bitcoin-strings-with-txids/./tmp.py", line 29, in decode_varint
    return struct.unpack(format_, data[1:size+1])[0], size + 1
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack requires a buffer of 8 bytes

```

So I wonder if I guessed the format wrong, or if it's just a bug in my code.

Comparing to: <https://en.bitcoin.it/wiki/Protocol_documentation#Variable_length_integer> I would decode:

```
42 ff c0 43 8b 94 35

```

manually as:

* 42
* ff: expect 8 bytes next
  + c0 43 8b 94 35: only 5 bytes left, blowup

I also tried to inverse value:

```
value = bytes(reversed(value))

```

but then it blows up very early, definitely wrong.

I also tried to ignore the error to see if there are others, but there were hundreds them, so something is definitely wrong with my method.

Related:

* <https://bitcointalk.org/index.php?topic=1068721.0>

**Update**

Given username_1's answer, I've updated my code to:

```
#!/usr/bin/env python3

import struct

import plyvel

def decode_varint(data):
    i = 0
    ret = 0
    while True:
        b = data[i]
        ret += b & 0x7F
        if b & 0x80:
            b += 1
        else:
            return (ret, i + 1)
        ret <<= 7
        i += 1

ldb = plyvel.DB('/home/username_0/snap/bitcoin-core/common/.bitcoin/indexes/txindex/', compression=None)
i = 0
nerrs = 0
for key, value in ldb:
    if key[0:1] == b't':
        if i % 1000000 == 0:
            print(i//1000000)
        txid = bytes(reversed(key[1:])).hex()
        print(value.hex(' '))
        total_off = 0
        file, off = decode_varint(value)
        total_off += off
        blk_off, off = decode_varint(value[total_off:])
        total_off += off
        tx_off, off = decode_varint(value[total_off:])
        total_off += off
        assert total_off == len(value)
        print((txid, file, blk_off, tx_off))
        i += 1
print(f'nerrs: {nerrs}')

```

and all length asserts seem to pass which is promising. However, when I tried to verify the first few values:

```
93 1a 96 8c e7 74 a7 bb 0a
('d176d4960a78b41971f9d19207b59af6584b16ef323de55e983aec0100000000', 2458, 46347252, 646538)
93 1a aa d0 f0 72 86 90 01
('4d19a5ba6a5455a6d4d46d6999fb0a4a7b29e603fe229cd4f1a34d0300000000', 2458, 89405554, 100353)
93 1e d4 cf 18 82 be 76
('ba75d6ac075959632d9e012f0321c367a54e9f2bde2e2bebce6baa0300000000', 2462, 1386392, 40822)
93 18 a8 96 8e 5a 8a d3 14
('7df98bab9d89fc14d0dec365fa124b3e112017c6b8f56f64cf159e0600000000', 2456, 84248410, 174484)

```

I didn't find a match between:

```
bitcoin-core.cli getrawtransaction d176d4960a78b41971f9d19207b59af6584b16ef323de55e983aec0100000000
tail -c+$((46347252 + 646538 - 1)) ~/snap/bitcoin-core/common/.bitcoin/blocks/blk02458.dat  | head -c 2000 | xxd -p | tr -d '\n'

bitcoin-core.cli getrawtransaction 4d19a5ba6a5455a6d4d46d6999fb0a4a7b29e603fe229cd4f1a34d0300000000
tail -c+$((89405554 + 100353 - 1)) ~/snap/bitcoin-core/common/.bitcoin/blocks/blk02458.dat  | head -c 2000 | xxd -p | tr -d '\n'

bitcoin-core.cli getrawtransaction ba75d6ac075959632d9e012f0321c367a54e9f2bde2e2bebce6baa0300000000
tail -c+$((89405554 + 100353 - 1)) ~/snap/bitcoin-core/common/.bitcoin/blocks/blk02462.dat  | head -c 2000 | xxd -p | tr -d '\n'

```

so either my parsing or my checks are wrong somehow.

I've searched for the bytes that `getrawtransactions` gave me for d176d4960a78b41971f9d19207b59af6584b16ef323de55e983aec0100000000 using `bgrep`: <https://unix.stackexchange.com/questions/223078/best-way-to-grep-a-big-binary-file/758528#758528> and found it at:

```
~/snap/bitcoin-core/common/.bitcoin/blocks/blk02586.dat: 02ed92ce

```

and offset 0x02ed92ce = 49124046, so these would be the expected correct values for this one.

**Tags:** bitcoin-core-development, database, txindex


---

## 1 answers

### Answer by username_1 on 2024-02-15. Score: 3

You are confusing the `VarInt` and `CompactSize` encoding. `CompactSize` encoding (also confusingly referred to as "varint" outside Bitcoin Core) is used in the Bitcoin protocol. `VarInt` encoding is internal to Bitcoin Core and used as a more compact encoding in the internal database.

See [this answer](https://bitcoin.stackexchange.com/a/114585/101498) for details about the `VarInt` format:

> Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make sure the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number:
>
> ```
> (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1))
>
> ```

> **Comments:**
> **username_0**: Thanks for this. I've updated my code on the question, and it does seem to extract 3 integers correctly from every value as expected. However, I was not able to actually find the transactions at those offsets as explained in the question now, I wonder what I'm doing wrong.  

---

