<blockquote>
<p><strong>Can anyone show me how to properly parse the blk.dat files using this library?</strong></p>
</blockquote>
<p>I can't, using that library, but the following observations might help:</p>
<pre><code>lines = f.readlines()
</code></pre>
<p>I'd expect something more like <code>bindata = f.read()</code> You opened the file for reading binary data (<code>br</code>) not for reading lines of text.</p>
<pre><code>for block in tqdm(lines):
</code></pre>
<p>Isn't <a href="https://github.com/tqdm/tqdm" rel="nofollow noreferrer">tqdm</a> a progress-bar library? I don't understand what you are trying to do with this line. It isn't clear to me that this would extract a block from blockfile data. Since you are having problems it would be appropriate to add comments to the code explaining the <em>intent</em> of each line.</p>
<pre><code>  block = block.strip().hex()
</code></pre>
<p>The string method <code>strip()</code> removes leading and trailing whitespace. Blockfiles contain binary data which shouldn't have string functions of this sort applied to them. For example 0x20 and 0x09 etc can occur as part of data without signifying whitespace (space or tab), removing these byte values could damage the data. Since you don't show your imports we don't know what data type your tqdm returns and therefore don't know what its methods do.</p>
<p>You don't show your <code>imports</code> which might help work out what this code does.</p>
<p>The method <code>.hex()</code> presumably was intended to convert the block data into a hexadecimal string. This would be OK as the the <a href="https://bitcoin.stackexchange.com/a/84352/13866">author's example</a> suggests this is what pybtc's <code>Block</code> function expects.</p>
<p>However so far as I can tell, Python strings don't have a <code>.hex()</code> method. Python integers do. This <em>might</em> explain the <code>Int too large to convert</code> error you get, I don't know Python well enough to say for sure.</p>
<pre><code>        for i in range(len(block)):
            tmp_block = block[i:]
            try:
                bt = pybtc.Block(tmp_block, format=&quot;raw&quot;)
</code></pre>
<p>This code seems to be searching for the start of a block and discarding bytes until it finds a starting point where the library succeeds in decoding a block. I think this isn't necessary. The blk*.dat files contain a fixed 4-byte <a href="https://en.bitcoin.it/wiki/Protocol_documentation#Message_structure" rel="nofollow noreferrer">network indication</a>  <code>0xD9B4BEF9</code> before each block. I would just check for that value and exit with a fatal error if it is not present. But note that the data may also contain this sequence in places other than the start of a block. You can check for it but not search for it.</p>

CC BY-SA 4.0
2023-03-23T18:21:45.233Z
2023-03-23T18:44:45.903Z
2023-03-23T18:44:45.903Z
RedGrittyBrick
<p>I am trying to parse the blk.dat files using pybtc but I can't seem to get this right.</p>
<pre><code>       f = open(block_filename, &quot;rb&quot;)
        lines = f.readlines()
        for block in tqdm(lines):
            block = block.strip().hex()
            for i in range(len(block)):
                tmp_block = block[i:]
                try:
                    bt = pybtc.Block(tmp_block, format=&quot;raw&quot;)
                    tx = bt[&quot;tx&quot;]
                    if (len(tx.keys()) &gt; 0):
                        print(&quot;keys&quot;, tx[&quot;vOut&quot;].keys())
</code></pre>
<p>Can anyone show me how to properly parse the blk.dat files using this library? Most of the time I get the following errors:</p>
<pre><code>Int too large to convert
Empty tx dictionary
</code></pre>

<transactions><block><python>
Reading blk.dat files using pybtc does not yield transactions
bitcoinluvr6969
# Reading blk.dat files using pybtc does not yield transactions

(Asked by: username_0 on 2023-03-21)

I am trying to parse the blk.dat files using pybtc but I can't seem to get this right.

```
       f = open(block_filename, "rb")
        lines = f.readlines()
        for block in tqdm(lines):
            block = block.strip().hex()
            for i in range(len(block)):
                tmp_block = block[i:]
                try:
                    bt = pybtc.Block(tmp_block, format="raw")
                    tx = bt["tx"]
                    if (len(tx.keys()) > 0):
                        print("keys", tx["vOut"].keys())

```

Can anyone show me how to properly parse the blk.dat files using this library? Most of the time I get the following errors:

```
Int too large to convert
Empty tx dictionary

```

**Tags:** transactions, block, python


---

## 1 answers

### Answer by username_1 on 2023-03-23. Score: 0 | Accepted answer

> **Can anyone show me how to properly parse the blk.dat files using this library?**

I can't, using that library, but the following observations might help:

```
lines = f.readlines()

```

I'd expect something more like `bindata = f.read()` You opened the file for reading binary data (`br`) not for reading lines of text.

```
for block in tqdm(lines):

```

Isn't [tqdm](https://github.com/tqdm/tqdm) a progress-bar library? I don't understand what you are trying to do with this line. It isn't clear to me that this would extract a block from blockfile data. Since you are having problems it would be appropriate to add comments to the code explaining the *intent* of each line.

```
  block = block.strip().hex()

```

The string method `strip()` removes leading and trailing whitespace. Blockfiles contain binary data which shouldn't have string functions of this sort applied to them. For example 0x20 and 0x09 etc can occur as part of data without signifying whitespace (space or tab), removing these byte values could damage the data. Since you don't show your imports we don't know what data type your tqdm returns and therefore don't know what its methods do.

You don't show your `imports` which might help work out what this code does.

The method `.hex()` presumably was intended to convert the block data into a hexadecimal string. This would be OK as the the [author's example](https://bitcoin.stackexchange.com/a/84352/13866) suggests this is what pybtc's `Block` function expects.

However so far as I can tell, Python strings don't have a `.hex()` method. Python integers do. This *might* explain the `Int too large to convert` error you get, I don't know Python well enough to say for sure.

```
        for i in range(len(block)):
            tmp_block = block[i:]
            try:
                bt = pybtc.Block(tmp_block, format="raw")

```

This code seems to be searching for the start of a block and discarding bytes until it finds a starting point where the library succeeds in decoding a block. I think this isn't necessary. The blk\*.dat files contain a fixed 4-byte [network indication](https://en.bitcoin.it/wiki/Protocol_documentation#Message_structure) `0xD9B4BEF9` before each block. I would just check for that value and exit with a fatal error if it is not present. But note that the data may also contain this sequence in places other than the start of a block. You can check for it but not search for it.


---

