<p>Blocks are never split across files: the whole block, from magic to length, to header, to transaction data, is always stored in a single file, continuously.</p>
<p>However you may need to take into account:</p>
<ul>
<li>Blocks in the blk*.dat files are stored in the order they are received from the network.l, which may not be the blocks' logical order.</li>
<li>It is possible that a block is in file N, and its descendant block is in file N-1.</li>
<li>The files contain all blocks ever received, including ones that are not part of the main chain but were reorged out.</li>
<li>There can be &quot;gaps&quot; of zeroes or garbage data between blocks in the blk*.dat files.</li>
</ul>

CC BY-SA 4.0
2022-08-07T03:53:27.410Z
Thanks for your answer! Just one follow up, is there anyway we can pre-sort the blk files in anyway ? Because I'd like to process them into a CSV on individual nodes on a distributed environment and then find out the input address by looking backwards if that makes sense
Rohan Maheshwari
CC BY-SA 4.0
2022-08-07T12:12:57.107Z
Bitcoin Core has a linearize script: https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize. This can be used to construct a single file with all blocks in the active chain (no reorged forks), in order, without gaps.
Pieter Wuille
CC BY-SA 4.0
2022-08-05T12:00:07.243Z
2022-08-05T12:20:48.460Z
2022-08-05T12:20:48.460Z
Pieter Wuille
<p>As a kind of addendum to the first bullet-point in <a href="https://bitcoin.stackexchange.com/a/114743/13866">Pieter's answer</a>, I had written a program to parse <code>blkxxxxx.dat</code> files and, in my case at least, the blocks are very obviously <em>not</em> in logical order. Just as Pieter asserted.</p>
<p>Notice the day number in the timestamps in the first ten blocks:</p>
<pre><code>C&gt; blockchain -file blk02100.dat -blocks -longstats
Block      1, 2020-06-03 03:48, target 171297F6,   2273 transactions
Block      2, 2020-06-03 04:29, target 171297F6,   2768 transactions
Block      3, 2020-06-04 02:00, target 171297F6,   2177 transactions
Block      4, 2020-06-05 12:37, target 17147F35,   2243 transactions
Block      5, 2020-06-01 07:15, target 171297F6,   3150 transactions
Block      6, 2020-06-02 21:48, target 171297F6,   2911 transactions
Block      7, 2020-06-02 21:49, target 171297F6,   2722 transactions
Block      8, 2020-06-05 16:23, target 17147F35,    182 transactions
Block      9, 2020-06-03 05:34, target 171297F6,   2002 transactions
Block     10, 2020-06-03 04:36, target 171297F6,   1958 transactions
...
Block    111, 2020-06-06 03:39, target 17147F35,    259 transactions

Statistics for blk02100.dat (2020-06-01 to 2020-06-06)

   111 blocks
222519 transactions (2004.7 per block)
109173 Segwit tx    (49.1%)
622026 inputs       (mean 2.8, max 1030 per transaction)
614553 outputs      (mean 2.8, max 3799 per transaction)

Processed file in 322.5756ms
</code></pre>
<p>I guess this scattered order is because my node was receiving blocks from multiple peers and concurrently receiving them from fairly disjoint ranges.</p>
<p>I can confirm that every <code>blkxxxxx.dat</code> file I tried, started and ended at a block boundary. There were no partial blocks.</p>
<hr />
<p>Footnote: My program isn't extensively tested so maybe take the above with a pinch of salt. I believe the timestamps to be parsed correctly.</p>

CC BY-SA 4.0
2022-08-07T03:54:56.467Z
Yes ! I have found the same to be true and the python parser i am currently using [parser](https://github.com/alecalve/python-bitcoin-blockchain-parser) also has the option to obtain blocks in an ordered/ unordered form.
Rohan Maheshwari
CC BY-SA 4.0
2022-08-06T19:56:09.457Z
2022-08-06T20:45:49.420Z
2022-08-06T20:45:49.420Z
RedGrittyBrick
<p>I am trying to parse out the bitcoin data using Big Data and wanted to know more about the blk files themselves. I was wondering how does bitcoin split the transactions across the files. I know there is a 128 MiB limit on the files. Does that mean it is possible for transaction data to be split across a blk ? For instance the merkle root of the transaction is in one file and the output index is in another ? Or is block data retained in a single file ?</p>

<bitcoin-core><block><blockchain-data-storage>
Splitting of data across blkxxxxx.dat files
Rohan Maheshwari
# Splitting of data across blkxxxxx.dat files

(Asked by: username_0 on 2022-08-05)

I am trying to parse out the bitcoin data using Big Data and wanted to know more about the blk files themselves. I was wondering how does bitcoin split the transactions across the files. I know there is a 128 MiB limit on the files. Does that mean it is possible for transaction data to be split across a blk ? For instance the merkle root of the transaction is in one file and the output index is in another ? Or is block data retained in a single file ?

**Tags:** bitcoin-core, block, blockchain-data-storage


---

## 2 answers

### Answer by username_1 on 2022-08-05. Score: 3 | Accepted answer

Blocks are never split across files: the whole block, from magic to length, to header, to transaction data, is always stored in a single file, continuously.

However you may need to take into account:

* Blocks in the blk\*.dat files are stored in the order they are received from the network.l, which may not be the blocks' logical order.
* It is possible that a block is in file N, and its descendant block is in file N-1.
* The files contain all blocks ever received, including ones that are not part of the main chain but were reorged out.
* There can be "gaps" of zeroes or garbage data between blocks in the blk\*.dat files.

> **Comments:**
> **username_0**: Thanks for your answer! Just one follow up, is there anyway we can pre-sort the blk files in anyway ? Because I'd like to process them into a CSV on individual nodes on a distributed environment and then find out the input address by looking backwards if that makes sense  
> **username_1**: Bitcoin Core has a linearize script: https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize. This can be used to construct a single file with all blocks in the active chain (no reorged forks), in order, without gaps.  

---

### Answer by username_2 on 2022-08-06. Score: 0

As a kind of addendum to the first bullet-point in [username_1's answer](https://bitcoin.stackexchange.com/a/114743/13866), I had written a program to parse `blkxxxxx.dat` files and, in my case at least, the blocks are very obviously *not* in logical order. Just as username_1 asserted.

Notice the day number in the timestamps in the first ten blocks:

```
C> blockchain -file blk02100.dat -blocks -longstats
Block      1, 2020-06-03 03:48, target 171297F6,   2273 transactions
Block      2, 2020-06-03 04:29, target 171297F6,   2768 transactions
Block      3, 2020-06-04 02:00, target 171297F6,   2177 transactions
Block      4, 2020-06-05 12:37, target 17147F35,   2243 transactions
Block      5, 2020-06-01 07:15, target 171297F6,   3150 transactions
Block      6, 2020-06-02 21:48, target 171297F6,   2911 transactions
Block      7, 2020-06-02 21:49, target 171297F6,   2722 transactions
Block      8, 2020-06-05 16:23, target 17147F35,    182 transactions
Block      9, 2020-06-03 05:34, target 171297F6,   2002 transactions
Block     10, 2020-06-03 04:36, target 171297F6,   1958 transactions
...
Block    111, 2020-06-06 03:39, target 17147F35,    259 transactions

Statistics for blk02100.dat (2020-06-01 to 2020-06-06)

   111 blocks
222519 transactions (2004.7 per block)
109173 Segwit tx    (49.1%)
622026 inputs       (mean 2.8, max 1030 per transaction)
614553 outputs      (mean 2.8, max 3799 per transaction)

Processed file in 322.5756ms

```

I guess this scattered order is because my node was receiving blocks from multiple peers and concurrently receiving them from fairly disjoint ranges.

I can confirm that every `blkxxxxx.dat` file I tried, started and ended at a block boundary. There were no partial blocks.

---

Footnote: My program isn't extensively tested so maybe take the above with a pinch of salt. I believe the timestamps to be parsed correctly.

> **Comments:**
> **username_0**: Yes ! I have found the same to be true and the python parser i am currently using [parser](https://github.com/alecalve/python-bitcoin-blockchain-parser) also has the option to obtain blocks in an ordered/ unordered form.  

---

