<p><a href="https://i.sstatic.net/aVI1a.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aVI1a.png" alt="enter image description here" /></a></p>
<p><strong>NOTES:</strong></p>
<p><strong>Heigh and Version fields</strong>
Fields height and version are needed to identify Outputs if they are last spent in the Transaction outputs. In older revxx.dat files, only if Output was last unspent in a Transaction it's height will be there together with it’s Transaction’s Version. If it's a regular Output then the height field will be 0 (00 HEX in Varint format ) and there will be no Version field</p>
<p>In newer revxx.dat files height is written for every Output and the Version field is 0 (00 in HEX) to be backward compatible. I did not check the revxx.dat files myself if it’s so, so please test this note first.</p>
<p><strong>Calculation of Height from the Field Value:</strong>
To compress the Amount of Statoshis this field uses the following logic:
If the amount is 0, output 0
First, divide the amount (in base units) by the largest power of 10 possible; call the exponent e (e is max 9)
if e &lt; 9, the last digit of the resulting number cannot be 0; store it as d, and drop it (divide by 10)
call the result n
output 1 + 10*(9<em>n + d - 1) + e
if e==9, we only know the resulting number is not zero, so output 1 + 10</em>(n - 1) + 9
(this is decodable, as d is in [1-9] and e is in [0-9])
Then store it as a VARINT</p>
<p><strong>nValue field(Amount)</strong></p>
<p>To compress the Amount of Statoshis this field uses the following logic:</p>
<p>If the amount is 0, output 0</p>
<p>First, divide the amount (in base units) by the largest power of 10 possible; call the exponent e (e is max 9)</p>
<p>if e &lt; 9, the last digit of the resulting number cannot be 0; store it as d, and drop it (divide by 10)</p>
<p>call the result n</p>
<p>output 1 + 10*(9*n + d - 1) + e</p>
<p>if e==9, we only know the resulting number is not zero, so output 1 + 10*(n - 1) + 9</p>
<p>(this is decodable, as d is in [1-9] and e is in [0-9])</p>
<p>Then store it as a VARINT</p>
<p><strong>Calculation of nValue field(Amount):</strong></p>
<p>To decode this amount go through the operation backward:</p>
<p>First, decode it from VARINT. Then use the following procedure(taken directly from the Bitcoin core implementation):</p>
<pre><code>uint64_t DecompressAmount(uint64_t x)
{
    // x = 0 OR x = 1+10\*(9\*n + d - 1) + e OR x = 1+10\*(n - 1) + 9
    if (x == 0)
        return 0;
    x--;
    // x = 10\*(9\*n + d - 1) + e
    int e = x % 10;
    x /= 10;
    uint64_t n = 0;
    if (e &lt; 9) {
        // x = 9\*n + d - 1
        int d = (x % 9) + 1;
        x /= 9;
        // x = n
        n = x\*10 + d;
    } else {
        n = x+1;
    }
    while (e) {
        n \*= 10;
        e--;
    }
    return n;
}
</code></pre>
<p><strong>ScriptPubKey size and SriptPubKey fields</strong></p>
<p>Current Bitcoin serialization is using a compressed format to store common types of scripts:</p>
<ul>
<li>Pay to Pubkey Hash (P2PKH) is encoded to – 20 bytes</li>
<li>Pay to Script Hash (P2SH) is encoded to – 20 bytes</li>
<li>Pay to Pubkey (P2PK) is encoded to – 32 bytes</li>
</ul>
<p>Other script types are serialized using the general rule shown in the table:</p>
<ul>
<li>scriptPubKey size</li>
<li>scriptPubKey data</li>
</ul>
<p>Where script size is a VARINT field of len(scriptPubKey data) + number of special cases, where the number of special cases is int(6), then goes the regular scriptPubKey data.</p>
<p>The compression algorithm is removing all OP_CODE values from the common types of scripts and leaves only the Key, KeyHash, or ScriptHash intact. Then it is using the  field to indicate the type of script in the next field.</p>
<p><a href="https://i.sstatic.net/NfiEs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NfiEs.png" alt="enter image description here" /></a></p>
<p>P2Pk cases: 0x02, 0x03 identify compressed Public Key and show how which Y coordinate to use to reconstruct the full key. 0x04, 0x05 identify uncompressed Public Key, but the value in the  is still in a compressed format and, and the size value shows which Y coordinate to use.</p>
<p>For other script types be sure to keep in mind that  is VARINT(len(scriptPubKey size) + int(6))</p>
<p>A small addition to the script compression, while you be looking through the Bitcoin core compress.cpp/.h files you may find that during compression either 20 bytes or 33/65 bytes are being manipulated that are not identified as any opcodes.</p>
<p>That's because of in bitcoin scripting if a field(4-bit hex value) is smaller than OP_PUSHDATA1 = 0x4c=76 is interpreted as an instruction to push the following number of bytes (the exact value of this field) onto the stack – this is called an immediate push.</p>
<p>As I'm not that good in modern C++, it was much easier for me to follow the compression algorithm when it was done in Golang:
<a href="https://github.com/btcsuite/btcd/blob/master/blockchain/compress.go" rel="nofollow noreferrer">https://github.com/btcsuite/btcd/blob/master/blockchain/compress.go</a></p>
<p>and in VarInt logic in Python:
<a href="https://github.com/christianb93/bitcoin/blob/1241778dc51bd39022611916f90f1bfaf07ee331/btc/serialize.py#L54" rel="nofollow noreferrer">https://github.com/christianb93/bitcoin/blob/1241778dc51bd39022611916f90f1bfaf07ee331/btc/serialize.py#L54</a></p>
<p>Greg did a really really nice explanation of Var Int, that I found so easy to follow. Thank you.</p>
<p><a href="https://learnmeabitcoin.com/guide/varint" rel="nofollow noreferrer">https://learnmeabitcoin.com/guide/varint</a></p>
<p>Yes, everything is in the code of original BitcoinCore implementation, I think maintainers and developers are doing an epic job to keep it nice and clear. It's because I was lacking some C++ skills I could not understand it properly from the beginning and had to spend some time on it. I just hope my notes will help someone to speed their learning process. Good luck.</p>

CC BY-SA 4.0
2020-07-15T02:09:34.263Z
2020-07-16T09:49:40.633Z
2020-07-16T09:49:40.633Z
Vladimir Proskurin
<p>This is a follow-up question for <a href="https://bitcoin.stackexchange.com/questions/57978/file-format-rev-dat/">rev.dat file format</a></p>

<p>Pieter Wuille gave a great explanation that is easy to follow. Since that posting there was a replacement of CTxInUndo with Coin, does the format still stands for the CBlockUndo record? </p>

<ul>
<li>4 bytes: network magic (0xf9,0xbe,0xb4,0xd9)</li>
<li>4 bytes: size of the CBlockUndo record (LE32)</li>
<li>data_size bytes: CBlockUndo record</li>
<li>32 bytes: double-SHA256 of the serialized CBlockUndo record</li>
</ul>

<p>CBlockUndo results in a serialized vector of CTxUndo records. Which is right now the serialized vector of Coin records. </p>

<p>And here is the Coin (or at least how I get it right now):</p>

<ul>
<li>varint that is serialized: 2*height (+1 if it was a coinbase output): the height of the block that created the spent UTXO. By the way, is uint32_t{2} the same as (uint32_t)2 in TxInUndoFormatter varint Serialize </li>
<li>version dummy value 0: we do not use version anymore but for compatibility using a 0000 or zero value in here</li>
<li>CompressedAmount: spent UTXO's nValue</li>
<li>CompressedScript: spent UTXO's scriptPubKey</li>
</ul>

<p>Questions:</p>

<ol>
<li>Is it correct?</li>
<li>Does anyone by any chance have a detailed mind map of undo process and a byte format map for the CBlockUndo record? </li>
</ol>

<p>Just for reference, I draw some image of current v0.19  and previous v0.13 versions 
of rev*.dat file:
<a href="https://i.sstatic.net/Qw8F1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Qw8F1.png" alt="enter image description here"></a></p>

<p><a href="https://i.sstatic.net/IltGb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IltGb.png" alt="enter image description here"></a></p>

CC BY-SA 4.0
2020-05-01T01:41:19.390Z
There have not been any format changes in 0.19 as far as I know. There last was a backward-compatible change in 0.15.
Pieter Wuille
CC BY-SA 4.0
2020-05-01T02:46:45.023Z
Thank you for responding.
So we can use the format that you described earlier and my pic of v0.19 is correct?
Vlad Proskurin
<bitcoin-core-development><chain-reorganization><learning><undofiles>
Undo files format change and reorg process question for v0.19
Vlad Proskurin
# Undo files format change and reorg process question for v0.19

(Asked by: username_0 on 2020-05-01)

This is a follow-up question for [rev.dat file format](https://bitcoin.stackexchange.com/questions/57978/file-format-rev-dat/)

username_1 username_1 gave a great explanation that is easy to follow. Since that posting there was a replacement of CTxInUndo with Coin, does the format still stands for the CBlockUndo record?

* 4 bytes: network magic (0xf9,0xbe,0xb4,0xd9)
* 4 bytes: size of the CBlockUndo record (LE32)
* data\_size bytes: CBlockUndo record
* 32 bytes: double-SHA256 of the serialized CBlockUndo record

CBlockUndo results in a serialized vector of CTxUndo records. Which is right now the serialized vector of Coin records.

And here is the Coin (or at least how I get it right now):

* varint that is serialized: 2\*height (+1 if it was a coinbase output): the height of the block that created the spent UTXO. By the way, is uint32\_t{2} the same as (uint32\_t)2 in TxInUndoFormatter varint Serialize
* version dummy value 0: we do not use version anymore but for compatibility using a 0000 or zero value in here
* CompressedAmount: spent UTXO's nValue
* CompressedScript: spent UTXO's scriptPubKey

Questions:

1. Is it correct?
2. Does anyone by any chance have a detailed mind map of undo process and a byte format map for the CBlockUndo record?

Just for reference, I draw some image of current v0.19 and previous v0.13 versions
of rev\*.dat file:
[![enter image description here](https://i.sstatic.net/Qw8F1.png)](https://i.sstatic.net/Qw8F1.png)

[![enter image description here](https://i.sstatic.net/IltGb.png)](https://i.sstatic.net/IltGb.png)

**Tags:** bitcoin-core-development, chain-reorganization, learning, undofiles

> **Comments:**
> **username_1**: There have not been any format changes in 0.19 as far as I know. There last was a backward-compatible change in 0.15.  
> **username_0**: Thank you for responding. So we can use the format that you described earlier and my pic of v0.19 is correct?  

---

## 1 answers

### Answer by username_2 on 2020-07-15. Score: 3

[![enter image description here](https://i.sstatic.net/aVI1a.png)](https://i.sstatic.net/aVI1a.png)

**NOTES:**

**Heigh and Version fields**
Fields height and version are needed to identify Outputs if they are last spent in the Transaction outputs. In older revxx.dat files, only if Output was last unspent in a Transaction it's height will be there together with it’s Transaction’s Version. If it's a regular Output then the height field will be 0 (00 HEX in Varint format ) and there will be no Version field

In newer revxx.dat files height is written for every Output and the Version field is 0 (00 in HEX) to be backward compatible. I did not check the revxx.dat files myself if it’s so, so please test this note first.

**Calculation of Height from the Field Value:**
To compress the Amount of Statoshis this field uses the following logic:
If the amount is 0, output 0
First, divide the amount (in base units) by the largest power of 10 possible; call the exponent e (e is max 9)
if e < 9, the last digit of the resulting number cannot be 0; store it as d, and drop it (divide by 10)
call the result n
output 1 + 10\*(9*n + d - 1) + e
if e==9, we only know the resulting number is not zero, so output 1 + 10*(n - 1) + 9
(this is decodable, as d is in [1-9] and e is in [0-9])
Then store it as a VARINT

**nValue field(Amount)**

To compress the Amount of Statoshis this field uses the following logic:

If the amount is 0, output 0

First, divide the amount (in base units) by the largest power of 10 possible; call the exponent e (e is max 9)

if e < 9, the last digit of the resulting number cannot be 0; store it as d, and drop it (divide by 10)

call the result n

output 1 + 10\*(9\*n + d - 1) + e

if e==9, we only know the resulting number is not zero, so output 1 + 10\*(n - 1) + 9

(this is decodable, as d is in [1-9] and e is in [0-9])

Then store it as a VARINT

**Calculation of nValue field(Amount):**

To decode this amount go through the operation backward:

First, decode it from VARINT. Then use the following procedure(taken directly from the Bitcoin core implementation):

```
uint64_t DecompressAmount(uint64_t x)
{
    // x = 0 OR x = 1+10\*(9\*n + d - 1) + e OR x = 1+10\*(n - 1) + 9
    if (x == 0)
        return 0;
    x--;
    // x = 10\*(9\*n + d - 1) + e
    int e = x % 10;
    x /= 10;
    uint64_t n = 0;
    if (e < 9) {
        // x = 9\*n + d - 1
        int d = (x % 9) + 1;
        x /= 9;
        // x = n
        n = x\*10 + d;
    } else {
        n = x+1;
    }
    while (e) {
        n \*= 10;
        e--;
    }
    return n;
}

```

**ScriptPubKey size and SriptPubKey fields**

Current Bitcoin serialization is using a compressed format to store common types of scripts:

* Pay to Pubkey Hash (P2PKH) is encoded to – 20 bytes
* Pay to Script Hash (P2SH) is encoded to – 20 bytes
* Pay to Pubkey (P2PK) is encoded to – 32 bytes

Other script types are serialized using the general rule shown in the table:

* scriptPubKey size
* scriptPubKey data

Where script size is a VARINT field of len(scriptPubKey data) + number of special cases, where the number of special cases is int(6), then goes the regular scriptPubKey data.

The compression algorithm is removing all OP\_CODE values from the common types of scripts and leaves only the Key, KeyHash, or ScriptHash intact. Then it is using the field to indicate the type of script in the next field.

[![enter image description here](https://i.sstatic.net/NfiEs.png)](https://i.sstatic.net/NfiEs.png)

P2Pk cases: 0x02, 0x03 identify compressed Public Key and show how which Y coordinate to use to reconstruct the full key. 0x04, 0x05 identify uncompressed Public Key, but the value in the is still in a compressed format and, and the size value shows which Y coordinate to use.

For other script types be sure to keep in mind that is VARINT(len(scriptPubKey size) + int(6))

A small addition to the script compression, while you be looking through the Bitcoin core compress.cpp/.h files you may find that during compression either 20 bytes or 33/65 bytes are being manipulated that are not identified as any opcodes.

That's because of in bitcoin scripting if a field(4-bit hex value) is smaller than OP\_PUSHDATA1 = 0x4c=76 is interpreted as an instruction to push the following number of bytes (the exact value of this field) onto the stack – this is called an immediate push.

As I'm not that good in modern C++, it was much easier for me to follow the compression algorithm when it was done in Golang:
<https://github.com/btcsuite/btcd/blob/master/blockchain/compress.go>

and in VarInt logic in Python:
<https://github.com/christianb93/bitcoin/blob/1241778dc51bd39022611916f90f1bfaf07ee331/btc/serialize.py#L54>

Greg did a really really nice explanation of Var Int, that I found so easy to follow. Thank you.

<https://learnmeabitcoin.com/guide/varint>

Yes, everything is in the code of original BitcoinCore implementation, I think maintainers and developers are doing an epic job to keep it nice and clear. It's because I was lacking some C++ skills I could not understand it properly from the beginning and had to spend some time on it. I just hope my notes will help someone to speed their learning process. Good luck.


---

