<p>I was able to read chainstate database.</p>
<p>Getting the key and de-obfuscating to get a <strong>valid last block</strong>.</p>
<p>However, when it comes to transactions that follow after that,</p>
<p>I cannot de-serialize them no matter what I'm doing.</p>
<pre><code>obtained the key: 1e9ad7d35416f693
last block:
ind:  42
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  1e9ad7d35416f6931e9d0500458046bf32a7a4316479e1d99a33ea499237f9c3
xor:  00000000000000000007d2d31196b02c2c3d73e2306f174a84a93d9ac6210f50 &lt;- this is good
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d100
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e
val:  badbd53e2b2b9a9893a09c755e97af385069c874d3520a3641
xor:  a44102ed7f3d6c0b8d3a4ba60a8159ab4ef31fa78744fca55f      &lt;- looks like wrong? idk
version:  164  nCode:  65  fCoinBase:  1  vAvail 1&amp;2:  false false  N:  7
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d102
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  01c855e08c6c63663c160665a736b7977b4d3f1049a47bc6ee9ad7d35416fb853e9afff7d50a3641
xor:  1f528233d87a95f5228cd1b6f320410465d7e8c31db28d55f000000000000d1620002824811cc0d2
version:  31  nCode:  82  fCoinBase:  0  vAvail 1&amp;2:  true false  N:  10
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d103
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  76a2bac0ed1819a6ce9bd2da2eaff013b1f584bf68e82e1b8d145941e1aca4ac3e9afff7d50a3641
xor:  68386d13b90eef35d00105097ab90680af6f536c3cfed888938e8e92b5ba523f20002824811cc0d2
version:  104  nCode:  56  fCoinBase:  0  vAvail 1&amp;2:  false false  N:  6
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d104
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  17528449f55864088b11ccf31519fb6f26ab84223f38016b613fd83116fe6e853e9afff7d50a3641
xor:  09c8539aa14e929b958b1b20410f0dfc383153f16b2ef7f87fa50fe242e8981620002824811cc0d2
version:  9  nCode:  200  fCoinBase:  0  vAvail 1&amp;2:  false false  N:  24
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d105
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  1db5122772cdab7b79e9d625261bc618652539fa4250aeb3efde1c950f6c588a3e9afff7d50a3641
xor:  032fc5f426db5de8677301f6720d308b7bbfee2916465820f144cb465b7aae1920002824811cc0d2
version:  3  nCode:  47  fCoinBase:  1  vAvail 1&amp;2:  true true  N:  5
</code></pre>
<p>They just don't seem to follow the rules. When I take the first two bytes of the transition value, they produce arbitrary version and unreasonably large N. I believe, it should be zero or a couple of bytes at best.</p>
<blockquote>
<ul>
<li>The higher bits encode N, the number of non-zero bytes in the following bitvector.</li>
<li>In case both bit 2 and bit 4 are unset, they encode N-1, as there must be at least one non-spent output).</li>
</ul>
</blockquote>
<p><strong>Don't know how to make the right unspentness bitvector out of it. Also it's unclear to me when one vout ends and the other one starts.
Likewise, no idea how to identify the height, especially because it's in a c++ varint format.</strong></p>
<p>Here is code, am I getting the right XORs for transactions right?
I am changing endianness immediately after retrieving each value.</p>
<pre><code>var level = require('level')
var db = level('kaka-read', {  valueEncoding: 'hex', keyEncoding: 'hex'})

db.createReadStream({ limit:1000 })
    .on('data', function (data) {
        var value= reverseBytes(data.value);                // changing endiannes for all values
        var index= data.key;
        
        if (!this.hasOwnProperty('key')){
            var key_val = value.slice(0,-2);                    // removing excessive byte from the obfuscation key
            console.log('obtained the key: '+key_val);
            
            this.key= class {                               // setting up the obfuscation key class
                constructor(new_len=0){
                    this.pos=-1;
                    this.len= key_val.length;
                    this.verbose= key_val;
                    this.extended='';
                    this.extend(new_len);
                }
                generate(){
                    this.pos = (this.pos+ 1) % this.len;
                    this.extended += this.verbose[this.pos];
                }
                extend(new_len){
                    for (var i=0; i&lt;new_len; this.generate(), i++);
                    return this.extended;
                }
            }
            return;                                         // obfuscation key has been set up
        }       
        
        //// de-obfuscating
        var key = new this.key(value.length).extended;      // extending the obfuscation key
        
        var array_1 = hexToBytes(key);
        var array_2 = hexToBytes(value);                    
        var array_3 = [];
        for (var i = 0; i&lt;array_1.length; i++)              // xor resulting byte arrays
            array_3[i]= array_2[i] ^ array_1[i];
            
        var decoded= bytesToHex(array_3);                   // converting result to hex string
        
        
        //// reporting to console
        var ind_1byte=index.substring(0, 2);                // checking the first byte of the index
        console.log(ind_1byte=='42'?'last block:':ind_1byte=='43'?'transaction:':'?' );
        console.log('ind: ',index );
        console.log('key: ',key);
        console.log('val: ',value);
        console.log('xor: ',decoded);
        if (ind_1byte=='42') return;        // if block we are done, otherwise try to unserialize transaction
        
        
        //// unserializing
        var version =array_3[0];
        var nCode = array_3[1];                 // nCode~00000000-11111111
        
        var vAvail= [];                         // &amp; = bitwise AND                              
        var fCoinBase = nCode &amp; 1;              // 1 = 001 binary (bit 1)
        vAvail[0] = (nCode &amp; 2) !=0;            // 2 = 010 binary (bit 2)
        vAvail[1] = (nCode &amp; 4) !=0;            // 4 = 100 binary (bit 3)
        N = nCode&gt;&gt;3;                           // shift 3 right (bit 4+)
        if (vAvail[0]==0 &amp;&amp; vAvail[1]==0) N--;  // if both spent, N-1
        
        //// the results don't seem to look right, but I don't know
        console.log('version: ', version,' nCode: ',nCode,' fCoinBase: ', fCoinBase,' vAvail 1&amp;2: ', vAvail[0],vAvail[1],' N: ',N);
    })
    
    .on('end', function () {
        console.log('Fin')
    })

// Convert a hex string to a byte array
function hexToBytes(hex) {
    for (var bytes = [], c = 0; c &lt; hex.length; c += 2)
    bytes.push(parseInt(hex.substr(c, 2), 16));
    return bytes;
}

// Convert a byte array to a hex string
function bytesToHex(bytes) {
    for (var hex = [], i = 0; i &lt; bytes.length; i++) {
        var current = bytes[i] &lt; 0 ? bytes[i] + 256 : bytes[i];
        hex.push((current &gt;&gt;&gt; 4).toString(16));
        hex.push((current &amp; 0xF).toString(16));
    }
    return hex.join(&quot;&quot;);
}

function reverseBytes(input){
    return bytesToHex(hexToBytes(input).reverse());
}
</code></pre>

CC BY-SA 4.0
2021-06-02T07:03:13.053Z
It sounds like you're trying to read entries in the pre-0.15 format. Since 0.15 the chainstate stores individual UTXOs per key/value pair, rather than all UTXOs created by a given txid.
Pieter Wuille
CC BY-SA 4.0
2021-06-02T11:13:24.093Z
This is the document I was referring to, couldn't find anything fresher. https://github.com/bitcoin/bitcoin/blob/d4a42334d447cad48fb3996cad0fd5c945b75571/src/coins.h#L21L69 is this invalid now?
Anonymous
CC BY-SA 4.0
2021-06-02T14:29:45.007Z
Yes, that code no longer exists since version 0.15. This is what you should look at in the current codebase: https://github.com/bitcoin/bitcoin/blob/master/src/coins.h#L23L29
Pieter Wuille
CC BY-SA 4.0
2021-06-04T01:12:22.203Z
@Pieter Wuille Not sure what I'm looking at. How exactly do I extract UTXO? So far I am able to read the database value and de-obfuscate it as well. I'm only interested in a list of addresses.
Anonymous
CC BY-SA 4.0
2021-06-04T01:37:25.507Z
From version 0.8 through 0.14, the chainstate represented a map (txid -> CCoins), where CCoins was serialized according to the documentation you link to. Since 0.15, it is much simpler, as the chainstate represents a map ((txid, vout) -> Coin), where Coin is serialized according to the documentation I link to.
Pieter Wuille
CC BY-SA 4.0
2021-06-04T02:51:55.550Z
Do you mean that vout with the actual addresses is now stored in the database key, not in the value?
Anonymous
CC BY-SA 4.0
2021-06-04T02:57:03.153Z
No, there is now a key per transaction output, not per transaction. And the value contains just one UTXO, rather than all UTXOs for a given transaction.
Pieter Wuille
CC BY-SA 4.0
2021-06-04T03:00:11.037Z
And there are no addresses in there at all; there are UTXOs which consist of a height, an amount, and a scriptPubKey. Those scriptPubKeys may or may not have a corresponding address representation.
Pieter Wuille
CC BY-SA 4.0
2021-06-04T03:09:30.463Z
So I see it stores 3 values: ```out``` ```fCoinBase``` and ```nHeight```, the out must contain scriptPubKey, but what is ```TxOutCompression```?
Anonymous
CC BY-SA 4.0
2021-06-04T03:42:42.577Z
Yes, out is the unspent output (an CTxOut object with scriptPubKey and nValue), fCoinBase is whether it's a coinbase or not, and nHeight is the height it was created at. TxOutCompression is the serializer that encodes the output in compact form; search the codebase for details.
Pieter Wuille
<utxo><javascript><leveldb>
Cannot unserialize chainstate transaction outputs after de-obfuscation
Anonymous
# Cannot unserialize chainstate transaction outputs after de-obfuscation

(Asked by: username_0 on 2021-06-02)

I was able to read chainstate database.

Getting the key and de-obfuscating to get a **valid last block**.

However, when it comes to transactions that follow after that,

I cannot de-serialize them no matter what I'm doing.

```
obtained the key: 1e9ad7d35416f693
last block:
ind:  42
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  1e9ad7d35416f6931e9d0500458046bf32a7a4316479e1d99a33ea499237f9c3
xor:  00000000000000000007d2d31196b02c2c3d73e2306f174a84a93d9ac6210f50 <- this is good
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d100
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e
val:  badbd53e2b2b9a9893a09c755e97af385069c874d3520a3641
xor:  a44102ed7f3d6c0b8d3a4ba60a8159ab4ef31fa78744fca55f      <- looks like wrong? idk
version:  164  nCode:  65  fCoinBase:  1  vAvail 1&2:  false false  N:  7
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d102
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  01c855e08c6c63663c160665a736b7977b4d3f1049a47bc6ee9ad7d35416fb853e9afff7d50a3641
xor:  1f528233d87a95f5228cd1b6f320410465d7e8c31db28d55f000000000000d1620002824811cc0d2
version:  31  nCode:  82  fCoinBase:  0  vAvail 1&2:  true false  N:  10
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d103
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  76a2bac0ed1819a6ce9bd2da2eaff013b1f584bf68e82e1b8d145941e1aca4ac3e9afff7d50a3641
xor:  68386d13b90eef35d00105097ab90680af6f536c3cfed888938e8e92b5ba523f20002824811cc0d2
version:  104  nCode:  56  fCoinBase:  0  vAvail 1&2:  false false  N:  6
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d104
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  17528449f55864088b11ccf31519fb6f26ab84223f38016b613fd83116fe6e853e9afff7d50a3641
xor:  09c8539aa14e929b958b1b20410f0dfc383153f16b2ef7f87fa50fe242e8981620002824811cc0d2
version:  9  nCode:  200  fCoinBase:  0  vAvail 1&2:  false false  N:  24
transaction:
ind:  430000000001ec3a985ee53d32ef164b58f69ab50792d1f97119b4780a96d476d105
key:  1e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f6931e9ad7d35416f693
val:  1db5122772cdab7b79e9d625261bc618652539fa4250aeb3efde1c950f6c588a3e9afff7d50a3641
xor:  032fc5f426db5de8677301f6720d308b7bbfee2916465820f144cb465b7aae1920002824811cc0d2
version:  3  nCode:  47  fCoinBase:  1  vAvail 1&2:  true true  N:  5

```

They just don't seem to follow the rules. When I take the first two bytes of the transition value, they produce arbitrary version and unreasonably large N. I believe, it should be zero or a couple of bytes at best.

> * The higher bits encode N, the number of non-zero bytes in the following bitvector.
> * In case both bit 2 and bit 4 are unset, they encode N-1, as there must be at least one non-spent output).

**Don't know how to make the right unspentness bitvector out of it. Also it's unclear to me when one vout ends and the other one starts.
Likewise, no idea how to identify the height, especially because it's in a c++ varint format.**

Here is code, am I getting the right XORs for transactions right?
I am changing endianness immediately after retrieving each value.

```
var level = require('level')
var db = level('kaka-read', {  valueEncoding: 'hex', keyEncoding: 'hex'})

db.createReadStream({ limit:1000 })
    .on('data', function (data) {
        var value= reverseBytes(data.value);                // changing endiannes for all values
        var index= data.key;
        
        if (!this.hasOwnProperty('key')){
            var key_val = value.slice(0,-2);                    // removing excessive byte from the obfuscation key
            console.log('obtained the key: '+key_val);
            
            this.key= class {                               // setting up the obfuscation key class
                constructor(new_len=0){
                    this.pos=-1;
                    this.len= key_val.length;
                    this.verbose= key_val;
                    this.extended='';
                    this.extend(new_len);
                }
                generate(){
                    this.pos = (this.pos+ 1) % this.len;
                    this.extended += this.verbose[this.pos];
                }
                extend(new_len){
                    for (var i=0; i<new_len; this.generate(), i++);
                    return this.extended;
                }
            }
            return;                                         // obfuscation key has been set up
        }       
        
        //// de-obfuscating
        var key = new this.key(value.length).extended;      // extending the obfuscation key
        
        var array_1 = hexToBytes(key);
        var array_2 = hexToBytes(value);                    
        var array_3 = [];
        for (var i = 0; i<array_1.length; i++)              // xor resulting byte arrays
            array_3[i]= array_2[i] ^ array_1[i];
            
        var decoded= bytesToHex(array_3);                   // converting result to hex string
        
        
        //// reporting to console
        var ind_1byte=index.substring(0, 2);                // checking the first byte of the index
        console.log(ind_1byte=='42'?'last block:':ind_1byte=='43'?'transaction:':'?' );
        console.log('ind: ',index );
        console.log('key: ',key);
        console.log('val: ',value);
        console.log('xor: ',decoded);
        if (ind_1byte=='42') return;        // if block we are done, otherwise try to unserialize transaction
        
        
        //// unserializing
        var version =array_3[0];
        var nCode = array_3[1];                 // nCode~00000000-11111111
        
        var vAvail= [];                         // & = bitwise AND                              
        var fCoinBase = nCode & 1;              // 1 = 001 binary (bit 1)
        vAvail[0] = (nCode & 2) !=0;            // 2 = 010 binary (bit 2)
        vAvail[1] = (nCode & 4) !=0;            // 4 = 100 binary (bit 3)
        N = nCode>>3;                           // shift 3 right (bit 4+)
        if (vAvail[0]==0 && vAvail[1]==0) N--;  // if both spent, N-1
        
        //// the results don't seem to look right, but I don't know
        console.log('version: ', version,' nCode: ',nCode,' fCoinBase: ', fCoinBase,' vAvail 1&2: ', vAvail[0],vAvail[1],' N: ',N);
    })
    
    .on('end', function () {
        console.log('Fin')
    })

// Convert a hex string to a byte array
function hexToBytes(hex) {
    for (var bytes = [], c = 0; c < hex.length; c += 2)
    bytes.push(parseInt(hex.substr(c, 2), 16));
    return bytes;
}

// Convert a byte array to a hex string
function bytesToHex(bytes) {
    for (var hex = [], i = 0; i < bytes.length; i++) {
        var current = bytes[i] < 0 ? bytes[i] + 256 : bytes[i];
        hex.push((current >>> 4).toString(16));
        hex.push((current & 0xF).toString(16));
    }
    return hex.join("");
}

function reverseBytes(input){
    return bytesToHex(hexToBytes(input).reverse());
}

```

**Tags:** utxo, javascript, leveldb

> **Comments:**
> **username_1**: It sounds like you're trying to read entries in the pre-0.15 format. Since 0.15 the chainstate stores individual UTXOs per key/value pair, rather than all UTXOs created by a given txid.  
> **username_0**: This is the document I was referring to, couldn't find anything fresher. https://github.com/bitcoin/bitcoin/blob/d4a42334d447cad48fb3996cad0fd5c945b75571/src/coins.h#L21L69 is this invalid now?  
> **username_1**: Yes, that code no longer exists since version 0.15. This is what you should look at in the current codebase: https://github.com/bitcoin/bitcoin/blob/master/src/coins.h#L23L29  
> **username_0**: @username_1 Not sure what I'm looking at. How exactly do I extract UTXO? So far I am able to read the database value and de-obfuscate it as well. I'm only interested in a list of addresses.  
> **username_1**: From version 0.8 through 0.14, the chainstate represented a map (txid -> CCoins), where CCoins was serialized according to the documentation you link to. Since 0.15, it is much simpler, as the chainstate represents a map ((txid, vout) -> Coin), where Coin is serialized according to the documentation I link to.  
> **username_0**: Do you mean that vout with the actual addresses is now stored in the database key, not in the value?  
> **username_1**: No, there is now a key per transaction output, not per transaction. And the value contains just one UTXO, rather than all UTXOs for a given transaction.  
> **username_1**: And there are no addresses in there at all; there are UTXOs which consist of a height, an amount, and a scriptPubKey. Those scriptPubKeys may or may not have a corresponding address representation.  
> **username_0**: So I see it stores 3 values: ```out``` ```fCoinBase``` and ```nHeight```, the out must contain scriptPubKey, but what is ```TxOutCompression```?  
> **username_1**: Yes, out is the unspent output (an CTxOut object with scriptPubKey and nValue), fCoinBase is whether it's a coinbase or not, and nHeight is the height it was created at. TxOutCompression is the serializer that encodes the output in compact form; search the codebase for details.  

---

## 0 answers

