<p>If you mean whether block verification can be distributed so that one entity can participate without storing the entire blockchain, this isn't compatible with the current design, but maybe a new design can do that. Also, suggestions such as <a href="https://bitcointalk.org/index.php?topic=88208.0;topicseen" rel="nofollow">this</a> aim to improve the ability of lightweight (SPV) clients, who do not store the entire blockchain, to verify transactions concerning them specifically, without changing the core protocol.</p>

<p>If you mean whether a single entity can distribute the blockchain among several machines on the same LAN, then this is possible. The basic operation needed for block verification is to look up the details of a previous output, so the natural sharding key would be output hashes. One could also cache the newest outputs which are the most likely to be used in the following transactions.</p>

CC BY-SA 3.0
2012-06-20T14:09:30.780Z
Now that I think about it, perhaps N different (but very similar) blockchains could be created. If an easy API existed to trade between the chains in a secure, atomic way, then you could send money between any two chains by trading to the target chain first and then sending a normal tx.
ripper234
CC BY-SA 3.0
2012-06-20T15:54:17.377Z
https://bitcointalk.org/index.php?topic=88714.msg977222#msg977222
ripper234
CC BY-SA 3.0
2012-06-19T07:45:21.783Z
2012-06-19T07:45:21.783Z
Meni Rosenfeld
<p>There is nothing so special about the data in the block chain that this cannot be done, but at present I know of no client which implements such a thing. BitcoinSharp, BitcoinJ and the default (Satoshi) client all use different database types and there's no reason you couldn't make a client that used still another database type.</p>

<p>So at present, no, it's not possible with off-the-shelf software, but there's no reason it couldn't be implemented as such if scale became an issue, or even implemented today as a proof-of-concept.</p>

CC BY-SA 3.0
2012-06-19T06:27:31.120Z
OP might have meant changing the Bitcoin protocol to allow each miner to process a particular shard, easing/solving its scalability requirements. See my answer.
ripper234
CC BY-SA 3.0
2012-06-19T08:09:15.550Z
The blockchain system **IS** a distributed database with its own architecture.
Rook
CC BY-SA 3.0
2012-06-19T05:09:50.733Z
2012-06-19T05:09:50.733Z
David Perry
<p>I don't think it will work.</p>

<p>I don't understand what you mean by sharding by date. The bitcoin analogy would be sharding by block-number ... but that is already the case ... every block is worked on separately. The thing is, every block depends on the previous block, so a miner has to keep some record of the entire blockchain history.</p>

<p>I initially thought you meant just sharding by "shard ID" - splitting the blockchain into N independent chains, where a miner can decide to work on "subchain 137 out of 1000". This idea won't work because every Bitcoin address must be able to send money to every other Bitcoin address. So, in order to mine only one shard, you still have to know the state of every other shard, because there will be transactions linking these shards (there could also be tx linking multiple addresses split between multiple (>2) shards).</p>

<p>Lastly, if you aren't suggesting a fork in this question, but just asking about techniques to process the existing chain structure using shards ... yes, the blockchain's data is shardable, it could be sharded by address. A miner would still have to process each new block, but its data set could be stored on a Redis cluster instead of in the memory of a single machine.</p>

<p><strong>Update</strong> - then again, maybe I was wrong. Here are my <a href="https://bitcointalk.org/index.php?topic=88714.0" rel="nofollow">latest thoughts on the matter</a>.</p>

CC BY-SA 3.0
2012-06-19T08:10:05.987Z
maybe you should read about sharding before posting.
Rook
CC BY-SA 3.0
2012-06-19T06:23:12.747Z
2012-06-20T14:31:30.407Z
2012-06-20T14:31:30.407Z
ripper234
<p>It would be possible in a few areas:</p>

<p>Storing the block chain on a server could be done in the same manner any data is stored - be it in one places or in many. This would be trivial from Bitcoin perspective.</p>

<p>Transmitting the data could also be partitioned into shards should a small change to the protocol be made - allowing one to request a block header with the information about the full merkle tree. This way one would know exactly which transactions are stored in the block without the need to know everything about all the transactions. The rest of the data could be fetched as needed from other places. This is similar to pruning the block history.</p>

<p>Preparing the blocks for mining could also be handled in a similar way, but it would require more trust from other nodes. A pool would only request the transaction hashes from other clients, not really needing to see the entire transaction in order to include it in the block header + merkle tree structure. The only downfall of this would be that someone could sneak in a few invalid transactions in this way, as there would be no data checking. So all in all, the pools that would wish to operate on seeing only part of the data would probably have to limit themselves to including transactions they have seen. This would mean that some transactions might take longer to include than others, depending on which pool has seen them.</p>

<p>All in all, Bitcoin's data could be partitioned with only small changes to the protocol, but that would require a lot of consideration first.</p>

CC BY-SA 3.0
2012-06-19T15:21:19.047Z
2012-06-19T15:21:19.047Z
ThePiachu
<p><a href="https://en.bitcoin.it/wiki/Scalability">Bitcoin Scalability</a> is a hot topic.  In database architecture design data is segmented into <a href="http://en.wikipedia.org/wiki/Shard_%28database_architecture%29">shards</a> or the data is <a href="http://en.wikipedia.org/wiki/Partition_%28database%29">partitioned</a> such that it can be spread among large number of machines.</p>

<p>In the financial world, credit card companies like Visa have to deal with thousands of transactions per second.  To deal with this volume credit card processors will use the Date as a shard key.  This allows for the credit card processor to group all transactions made in a set period of time and a set of machines are made responsible for a month of transactions.   Is this sharding tactic compatible with Bitcoin blocks?   Or more generally what if Bitcoin was as popular as credit cards?  What kind of techniques would Bitcoin be forced to use to cope with the scale of the problem? </p>

CC BY-SA 3.0
2012-06-20T14:31:05.620Z
I gave it some more thought - this is the result: https://bitcointalk.org/index.php?topic=88714.0
ripper234
<block><scalability>
Can Bitcoin blocks be organized into shards?
Rook
# Can Bitcoin blocks be organized into shards?

(Asked by: username_0 on 2012-06-19)

[Bitcoin Scalability](https://en.bitcoin.it/wiki/Scalability) is a hot topic. In database architecture design data is segmented into [shards](http://en.wikipedia.org/wiki/Shard_%28database_architecture%29) or the data is [partitioned](http://en.wikipedia.org/wiki/Partition_%28database%29) such that it can be spread among large number of machines.

In the financial world, credit card companies like Visa have to deal with thousands of transactions per second. To deal with this volume credit card processors will use the Date as a shard key. This allows for the credit card processor to group all transactions made in a set period of time and a set of machines are made responsible for a month of transactions. Is this sharding tactic compatible with Bitcoin blocks? Or more generally what if Bitcoin was as popular as credit cards? What kind of techniques would Bitcoin be forced to use to cope with the scale of the problem?

**Tags:** block, scalability

> **Comments:**
> **username_1**: I gave it some more thought - this is the result: https://bitcointalk.org/index.php?topic=88714.0  

---

## 4 answers

### Answer by username_2 on 2012-06-19. Score: 4

If you mean whether block verification can be distributed so that one entity can participate without storing the entire blockchain, this isn't compatible with the current design, but maybe a new design can do that. Also, suggestions such as [this](https://bitcointalk.org/index.php?topic=88208.0;topicseen) aim to improve the ability of lightweight (SPV) clients, who do not store the entire blockchain, to verify transactions concerning them specifically, without changing the core protocol.

If you mean whether a single entity can distribute the blockchain among several machines on the same LAN, then this is possible. The basic operation needed for block verification is to look up the details of a previous output, so the natural sharding key would be output hashes. One could also cache the newest outputs which are the most likely to be used in the following transactions.

> **Comments:**
> **username_1**: Now that I think about it, perhaps N different (but very similar) blockchains could be created. If an easy API existed to trade between the chains in a secure, atomic way, then you could send money between any two chains by trading to the target chain first and then sending a normal tx.  
> **username_1**: https://bitcointalk.org/index.php?topic=88714.msg977222#msg977222  

---

### Answer by username_3 on 2012-06-19. Score: 3

There is nothing so special about the data in the block chain that this cannot be done, but at present I know of no client which implements such a thing. BitcoinSharp, BitcoinJ and the default (Satoshi) client all use different database types and there's no reason you couldn't make a client that used still another database type.

So at present, no, it's not possible with off-the-shelf software, but there's no reason it couldn't be implemented as such if scale became an issue, or even implemented today as a proof-of-concept.

> **Comments:**
> **username_1**: OP might have meant changing the Bitcoin protocol to allow each miner to process a particular shard, easing/solving its scalability requirements. See my answer.  
> **username_0**: The blockchain system **IS** a distributed database with its own architecture.  

---

### Answer by username_1 on 2012-06-19. Score: 3

I don't think it will work.

I don't understand what you mean by sharding by date. The bitcoin analogy would be sharding by block-number ... but that is already the case ... every block is worked on separately. The thing is, every block depends on the previous block, so a miner has to keep some record of the entire blockchain history.

I initially thought you meant just sharding by "shard ID" - splitting the blockchain into N independent chains, where a miner can decide to work on "subchain 137 out of 1000". This idea won't work because every Bitcoin address must be able to send money to every other Bitcoin address. So, in order to mine only one shard, you still have to know the state of every other shard, because there will be transactions linking these shards (there could also be tx linking multiple addresses split between multiple (>2) shards).

Lastly, if you aren't suggesting a fork in this question, but just asking about techniques to process the existing chain structure using shards ... yes, the blockchain's data is shardable, it could be sharded by address. A miner would still have to process each new block, but its data set could be stored on a Redis cluster instead of in the memory of a single machine.

**Update** - then again, maybe I was wrong. Here are my [latest thoughts on the matter](https://bitcointalk.org/index.php?topic=88714.0).

> **Comments:**
> **username_0**: maybe you should read about sharding before posting.  

---

### Answer by username_4 on 2012-06-19. Score: 1

It would be possible in a few areas:

Storing the block chain on a server could be done in the same manner any data is stored - be it in one places or in many. This would be trivial from Bitcoin perspective.

Transmitting the data could also be partitioned into shards should a small change to the protocol be made - allowing one to request a block header with the information about the full merkle tree. This way one would know exactly which transactions are stored in the block without the need to know everything about all the transactions. The rest of the data could be fetched as needed from other places. This is similar to pruning the block history.

Preparing the blocks for mining could also be handled in a similar way, but it would require more trust from other nodes. A pool would only request the transaction hashes from other clients, not really needing to see the entire transaction in order to include it in the block header + merkle tree structure. The only downfall of this would be that someone could sneak in a few invalid transactions in this way, as there would be no data checking. So all in all, the pools that would wish to operate on seeing only part of the data would probably have to limit themselves to including transactions they have seen. This would mean that some transactions might take longer to include than others, depending on which pool has seen them.

All in all, Bitcoin's data could be partitioned with only small changes to the protocol, but that would require a lot of consideration first.


---

