<p>The Bitcoin Core <a href="https://github.com/bitcoin/bitcoin/blob/0d71395848bbc2941862e7a0644f47212ec02e87/doc/reduce-memory.md#in-memory-caches" rel="nofollow noreferrer">documentation</a> describes dbcache as the cache for the UTXO database.</p>
<p><a href="https://en.wikipedia.org/wiki/Cache_(computing)" rel="nofollow noreferrer">Wikipedia</a> describes a cache as a:</p>
<blockquote>
<p>software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere.</p>
</blockquote>
<p>Greg Maxwell added on IRC:</p>
<blockquote>
<p>There are basically two separate uses: the block index which is
essentially metadata and the UTXO database.</p>
<p>The dbcache is not really a cache, it's a write buffer and it prevents
needing to sync the disk or make random writes. As a cache it doesn't
do much and isn't needed for that. It does <em>also</em> act as a cache, but
if you nullify that benefit it's only approximately a 10% slowdown
even with a huge dbcache.</p>
<p>With a fast SSD (e.g. NVMe) I think the difference between a 400MB
cache and a 5GB one is &quot;only&quot; a halving of IBD time (when syncing from
LAN peers).</p>
<p>To prevent corruption the database updates must involve synchronizing
writes.</p>
<p>We changed how dbcache flushing worked with the explicit intention of
making background flushing possible so that it could be concurrently
flushing and processing blocks constantly rather than inserting these
bubbles where it waits on the disk. Even though the consistency
requirements make that possible now (the UTXO database doesn't have to
be consistent except there is a record that states that all blocks
before it have absolutely positively been applied to the database),
actually making the change is still extremely complicated.</p>
<p>Every time you process a block, once the cache fills you could flush
out to disk the remaining dirty entries for the oldest remaining
blocks until it's back under the limit again.. So essentially flushing
every block in the background.  Then every once in a while, you do a
disk sync and update the record that says where the completely
consistent point is, without having to flush anything additional out.
So almost no latency spikes at runtime.</p>
<p>This would be good for mining but doing that requires a bunch of
machinery to efficiently track things and probably would make sense to
replace the dbcache's map with an open hash table of some kind at the
same time to cut the malloc traffic down by 10x. I expect there is a
factor of 2 IBD speedup waiting from these changes. I'm not sure if
it'll ever happen though, it's a big complicated task and any mistake
is a consensus bug.</p>
<p>You can even do neat stuff like resolve collisions in the open hash
table by displacing the oldest UTXO and once you've gone &quot;far enough&quot;
flush. The table could run extremely close to completely full at all
times.</p>
<p>I'm personally fond of cuckoo tables. Where every item has a small
number of possible locations like two random buckets which each hold 4
items and if those locations are full you pick one, insert the item
there, bump what was in that slot into one of its alternative
locations.</p>
<p>If the table has a little slack  (no more than ~95% full for 2 buckets
of 4 items for example) then it's always successful at finding an
empty slot after a few kicks. Lookup is extremely fast since you only
need to do two random memory accesses. There are a bunch of different
designs though most are best when you can keep the table &lt;50% full but
the cuckoo design is easy to take up to arbritarily high fullness.</p>
<p>The STL sets that Bitcoin uses are a hash table, but collisions are
resolved by having each entry in the hash table be a linked list, so
every access involves several pointer chases and every insertion takes
a malloc.</p>
</blockquote>

CC BY-SA 4.0
2020-09-18T15:49:22.657Z
2020-09-23T16:46:11.717Z
2020-09-23T16:46:11.717Z
Michael Folkson
<p>What is in the Bitcoin Core LevelDB <a href="https://bitcoin.stackexchange.com/questions/89100/bitcoind-node-crashes-after-using-dbcache-option">dbcache</a>? Is it full records or metadata?</p>
<p>This question was asked on IRC by Anonymous.</p>

<bitcoin-core><bitcoin-core-development><database><leveldb><cache>
What is in the Bitcoin Core LevelDB dbcache? Is it full records or metadata?
Michael Folkson
# What is in the Bitcoin Core LevelDB dbcache? Is it full records or metadata?

(Asked by: username_0 on 2020-09-18)

What is in the Bitcoin Core LevelDB [dbcache](https://bitcoin.stackexchange.com/questions/89100/bitcoind-node-crashes-after-using-dbcache-option)? Is it full records or metadata?

This question was asked on IRC by Anonymous.

**Tags:** bitcoin-core, bitcoin-core-development, database, leveldb, cache


---

## 1 answers

### Answer by username_0 on 2020-09-18. Score: 3 | Accepted answer

The Bitcoin Core [documentation](https://github.com/bitcoin/bitcoin/blob/0d71395848bbc2941862e7a0644f47212ec02e87/doc/reduce-memory.md#in-memory-caches) describes dbcache as the cache for the UTXO database.

[Wikipedia](https://en.wikipedia.org/wiki/Cache_(computing)) describes a cache as a:

> software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere.

Greg Maxwell added on IRC:

> There are basically two separate uses: the block index which is
> essentially metadata and the UTXO database.
>
> The dbcache is not really a cache, it's a write buffer and it prevents
> needing to sync the disk or make random writes. As a cache it doesn't
> do much and isn't needed for that. It does *also* act as a cache, but
> if you nullify that benefit it's only approximately a 10% slowdown
> even with a huge dbcache.
>
> With a fast SSD (e.g. NVMe) I think the difference between a 400MB
> cache and a 5GB one is "only" a halving of IBD time (when syncing from
> LAN peers).
>
> To prevent corruption the database updates must involve synchronizing
> writes.
>
> We changed how dbcache flushing worked with the explicit intention of
> making background flushing possible so that it could be concurrently
> flushing and processing blocks constantly rather than inserting these
> bubbles where it waits on the disk. Even though the consistency
> requirements make that possible now (the UTXO database doesn't have to
> be consistent except there is a record that states that all blocks
> before it have absolutely positively been applied to the database),
> actually making the change is still extremely complicated.
>
> Every time you process a block, once the cache fills you could flush
> out to disk the remaining dirty entries for the oldest remaining
> blocks until it's back under the limit again.. So essentially flushing
> every block in the background. Then every once in a while, you do a
> disk sync and update the record that says where the completely
> consistent point is, without having to flush anything additional out.
> So almost no latency spikes at runtime.
>
> This would be good for mining but doing that requires a bunch of
> machinery to efficiently track things and probably would make sense to
> replace the dbcache's map with an open hash table of some kind at the
> same time to cut the malloc traffic down by 10x. I expect there is a
> factor of 2 IBD speedup waiting from these changes. I'm not sure if
> it'll ever happen though, it's a big complicated task and any mistake
> is a consensus bug.
>
> You can even do neat stuff like resolve collisions in the open hash
> table by displacing the oldest UTXO and once you've gone "far enough"
> flush. The table could run extremely close to completely full at all
> times.
>
> I'm personally fond of cuckoo tables. Where every item has a small
> number of possible locations like two random buckets which each hold 4
> items and if those locations are full you pick one, insert the item
> there, bump what was in that slot into one of its alternative
> locations.
>
> If the table has a little slack (no more than ~95% full for 2 buckets
> of 4 items for example) then it's always successful at finding an
> empty slot after a few kicks. Lookup is extremely fast since you only
> need to do two random memory accesses. There are a bunch of different
> designs though most are best when you can keep the table <50% full but
> the cuckoo design is easy to take up to arbritarily high fullness.
>
> The STL sets that Bitcoin uses are a hash table, but collisions are
> resolved by having each entry in the hash table be a linked list, so
> every access involves several pointer chases and every insertion takes
> a malloc.


---

