<p>I wrote a bitcoin core parser, for my personal project and because it helped me to start understanding the bitcoin protocol as a data structure.</p>

<p>This project mainly parsers every blk file and creates the corresponding json with the deserialized information.</p>

<p>I launched the alpha version of the project and actually parserizes the information and converts it into a json file, my application worked fine until the blk file number 976, where I get an exception from the blockchain library and precisely in the ReadCompactSize operation, the exception is of the type</p>

<pre><code>unknown file: Failure
C++ exception with description "ReadCompactSize(): size too large: iostream error" thrown in the test body
</code></pre>

<p>I used the bitcoin core libraries for example serialize.h</p>

<p>I don't understand why this error only comes up to me after blk00976 blochi and for all the subsequent ones I tried also with the file blk00977 blk01124</p>

<p>I have some posts I have consulted and others where I asked about this problem, now I was opening an issue on github for bitcoins but there it is written that for general questions you have to refer here.</p>

<p>The questions I have read but do not solve the problem
<a href="https://bitcoin.stackexchange.com/questions/70810/how-to-handle-incomplete-blocks-when-reading-blk-dat-files">How to handle incomplete blocks when reading blk*.dat files?</a></p>

<p>My question
<a href="https://bitcointalk.org/index.php?topic=5134784.msg50723554#msg50723554" rel="nofollow noreferrer">https://bitcointalk.org/index.php?topic=5134784.msg50723554#msg50723554</a></p>

<p>My source code
<a href="https://github.com/vincenzopalazzo/SpyCblock" rel="nofollow noreferrer">https://github.com/vincenzopalazzo/SpyCblock</a></p>

<p>Two words about the source code</p>

<p>I am a java programmer dealing with the bitcoin protocol and I decided to carry out the C ++ project to study the language too, so to those who linked the code I apologize if it looks like an offense because it is badly written but I am studying to refact and improve the performance of the parser but also the quality of the source code.</p>

CC BY-SA 4.0
2019-04-23T19:33:08.560Z
Does your code support segwit? (BIP144)?
Pieter Wuille
CC BY-SA 4.0
2019-04-23T19:50:49.520Z
The BIP144 is this right? [Click me](https://github.com/bitcoin/bips/blob/master/bip-0144.mediawiki)     I don't think my code supports it, I didn't know it existed, so is this the reason for the error?

Can you give me more information about this? why was a new type of serialization introduced?
This type of serialization is valid only for transactions (input and output) or even raw transactions and blocks ??
vincenzopalazzo
CC BY-SA 4.0
2019-04-23T19:53:48.677Z
It was part of the segregated witness upgrade, which was a major upgrade to Bitcoin's technology. It affects a lot (P2P protocol, signatures, transactions, ...). You can read more about the reasons for it and design here: https://bitcoincore.org/en/2016/01/26/segwit-benefits/ and https://bitcoincore.org/en/segwit_wallet_dev/ More about the history here: https://bitcoinmagazine.com/articles/long-road-segwit-how-bitcoins-biggest-protocol-upgrade-became-reality/ Currently around half of the transactions on the network use segwit features.
Pieter Wuille
CC BY-SA 4.0
2019-04-23T19:59:55.910Z
So the serialization types are mixed? or are they all fulfilled in the way you said?
vincenzopalazzo
CC BY-SA 4.0
2019-04-23T20:04:11.567Z
Segwit is designed so it uses an otherwise invalid encoding, so it can be detected. It isn't so much a mix of formats; the new format is an extension of the old format; but the blk files indeed contain blocks with witnesses and without.
Pieter Wuille
CC BY-SA 4.0
2019-04-23T20:10:27.813Z
excuse the constant questions, what can a block with a witness be re-scanned?  from the transaction version?  as soon as I'm at the pc, I'll read your references thanks again
vincenzopalazzo
CC BY-SA 4.0
2019-04-23T21:03:49.107Z
Read the references, and BIP144 specifically.
Pieter Wuille
<bitcoin-core><bitcoin-core-development>
Exception CompactSize when I parsing the blk file >= 976
vincenzopalazzo
# Exception CompactSize when I parsing the blk file >= 976

(Asked by: username_0 on 2019-04-23)

I wrote a bitcoin core parser, for my personal project and because it helped me to start understanding the bitcoin protocol as a data structure.

This project mainly parsers every blk file and creates the corresponding json with the deserialized information.

I launched the alpha version of the project and actually parserizes the information and converts it into a json file, my application worked fine until the blk file number 976, where I get an exception from the blockchain library and precisely in the ReadCompactSize operation, the exception is of the type

```
unknown file: Failure
C++ exception with description "ReadCompactSize(): size too large: iostream error" thrown in the test body

```

I used the bitcoin core libraries for example serialize.h

I don't understand why this error only comes up to me after blk00976 blochi and for all the subsequent ones I tried also with the file blk00977 blk01124

I have some posts I have consulted and others where I asked about this problem, now I was opening an issue on github for bitcoins but there it is written that for general questions you have to refer here.

The questions I have read but do not solve the problem
[How to handle incomplete blocks when reading blk\*.dat files?](https://bitcoin.stackexchange.com/questions/70810/how-to-handle-incomplete-blocks-when-reading-blk-dat-files)

My question
<https://bitcointalk.org/index.php?topic=5134784.msg50723554#msg50723554>

My source code
<https://github.com/username_0/SpyCblock>

Two words about the source code

I am a java programmer dealing with the bitcoin protocol and I decided to carry out the C ++ project to study the language too, so to those who linked the code I apologize if it looks like an offense because it is badly written but I am studying to refact and improve the performance of the parser but also the quality of the source code.

**Tags:** bitcoin-core, bitcoin-core-development

> **Comments:**
> **username_1**: Does your code support segwit? (BIP144)?  
> **username_0**: The BIP144 is this right? [Click me](https://github.com/bitcoin/bips/blob/master/bip-0144.mediawiki)     I don't think my code supports it, I didn't know it existed, so is this the reason for the error?  Can you give me more information about this? why was a new type of serialization introduced? This type of serialization is valid only for transactions (input and output) or even raw transactions and blocks ??  
> **username_1**: It was part of the segregated witness upgrade, which was a major upgrade to Bitcoin's technology. It affects a lot (P2P protocol, signatures, transactions, ...). You can read more about the reasons for it and design here: https://bitcoincore.org/en/2016/01/26/segwit-benefits/ and https://bitcoincore.org/en/segwit_wallet_dev/ More about the history here: https://bitcoinmagazine.com/articles/long-road-segwit-how-bitcoins-biggest-protocol-upgrade-became-reality/ Currently around half of the transactions on the network use segwit features.  
> **username_0**: So the serialization types are mixed? or are they all fulfilled in the way you said?  
> **username_1**: Segwit is designed so it uses an otherwise invalid encoding, so it can be detected. It isn't so much a mix of formats; the new format is an extension of the old format; but the blk files indeed contain blocks with witnesses and without.  
> **username_0**: excuse the constant questions, what can a block with a witness be re-scanned?  from the transaction version?  as soon as I'm at the pc, I'll read your references thanks again  
> **username_1**: Read the references, and BIP144 specifically.  

---

## 0 answers

