<p>In general, HD Wallets use the following logic for how many addresses to query, and when to stop:</p>

<ol>
<li>Start with account 0, generate <code>gap limit</code> number of addresses (usually 20)</li>
<li>Check for any transactions in those addresses</li>
<li>If there are no transactions, stop searching for new addresses and accounts</li>
<li>If there are transactions, generator <code>gap limit</code> more from the index of the last address with transactions on it. Additionally, also do this process for account 1 (you only check for n+1 accounts if Account n has transactions</li>
</ol>

<p>The general idea is that no accounts will exist if the previous account is unused, and that no addresses in an account will be used after a single, consecutive block of <code>gap limit</code> number of unused addresses.</p>

<p>Companies like BitGo who offer HD wallet services to exchanges etc. where you can easily have thousands of unused addresses in a row, and then addresses with transactions, generally ignore the gap limit. They instead maintain a list of all generated addresses and index them as individual addresses, not as an HD wallet.</p>

<p>On the API side of things, wallets generally try to keep a running index. They will maintain a cache of known transactions, and keep appending to it. This way, when you start it up, it only scans for transactions in the new blocks since the last time it scanned. This allows even light clients such as Electrum or Ledger/Trezor wallets to serve thousands of users with just a handful of nodes, since each wallet will not requery for previous transactions it alread knows about.</p>

<p>Naturally, if you import a frequently used HD wallet to a new wallet or delete the cache, the initial sync can take very long. However, subsequent syncs of the wallet will be much faster.</p>

<p>In addition to this, HD wallets will often rely on a full node (such as Bitcoin Core) or a system such as ElectrumX or Insight. All of these are capable of maintaining a full index. Bitcoin Core maintains only a running index across known addresses (which is why you need to rescan when you import an address). ElectrumX and Insight maintain an augmented index which keeps track of all transactions, and can quickly return data for any given address.</p>

<p>HD Wallets do continue to check for transactions across all used addresses. They simply don't rescan the entire chain every time. But for every new block, every tx in that block will be checked for every address in the HD wallet. This check can be done quickly via bloom filters or preindexed data such as from Electrum or Insight, without reaching a bottleneck.</p>

CC BY-SA 3.0
2018-04-23T13:00:20.077Z
I understand how HD wallets work in theory, thanks for the clarification.

My question concerns how it works in practice. How would a HD Wallet, check if any of the previously generated addresses had funds sent to them efficiently? If they had 1,000 this would be a bottleneck for an API Wallet
Kyle Graham
CC BY-SA 3.0
2018-04-23T14:29:50.213Z
@KyleGraham Most wallets deal with this by maintaining a list of known transactions and only requesting for txs in newer blocks instead of scanning the entire chain for every address each time. I've edited the answer to include this.
Raghav Sood
CC BY-SA 3.0
2018-04-23T14:34:01.153Z
So if the wallet is maintaining 1,000 accounts, it will still scan the blockchain for each account, but it will be looking in newer blocks? If this is correct, then it does not seem efficient as users can potentially have thousands of accounts. With each account having the possibility of having funds sent to them.
Kyle Graham
CC BY-SA 3.0
2018-04-23T15:00:46.343Z
Scanning for new transactions has the same cost whether you're scanning for 1 address or 10000. Most implementations use some sort of  [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) to make it fast. Once the filter is constructed, checking each tx to see if it for your address is trivial.
Raghav Sood
CC BY-SA 3.0
2018-04-23T15:04:35.840Z
Yes exactly. It would be wasteful to scan for all 1,000 addresses(pass used addresses) each time the user opens the HD Wallet. If we do not check to see if someone has sent funds to a previously used address, the HD Wallet, will show an incorrect balance. My question is how do HD Wallets(mobile ones) deal with this?
Kyle Graham
CC BY-SA 3.0
2018-04-23T15:50:29.107Z
I've edited the answer some more. In short, the wallet *do* check every used address as well. It's just that there are ways to do it efficiently to a point where it's not a bottleneck in most cases
Raghav Sood
CC BY-SA 4.0
2020-07-29T20:32:33.377Z
@RaghavSood How is making 1,000+ HTTP requests every time you start the app not a bottleneck?
M.K. Safi
CC BY-SA 4.0
2020-07-30T01:38:16.987Z
Any place dealing with significant amounts of addresses is maintaining a constantly updated state for each address by scanning the mempool and each new block. The applications using the address balance, transactions etc. then query the monitoring applications, which are optimized to not require 1000 queries
Raghav Sood
CC BY-SA 4.0
2020-07-30T05:25:38.927Z
@RaghavSood How can a mobile wallet, which is relying on SPV or ElectrumX, be constantly scanning the mempool and new blocks when it's offline? Wouldn't it need to query the server for all the addresses in it every time it goes online?
M.K. Safi
CC BY-SA 4.0
2020-07-30T05:27:27.273Z
Using bloomf filters allows you to query reasonably large batches of addresses at a time - most electrum wallets that have been used for a while already scan thousands of addresses in a matter of minutes. That works well enough for personal wallets. At an exchange level, you have custom systems meant to deal with millions of addresses
Raghav Sood
CC BY-SA 4.0
2020-07-30T05:31:55.497Z
I'm implementing a mobile wallet right now and I'd like to learn how to do this, but I don't see any ElectrumX methods that allow batch queries https://electrumx.readthedocs.io/en/latest/protocol-methods.html I'm also using Esplora and I don't see such batch queries there either https://github.com/Blockstream/esplora/blob/master/API.md
M.K. Safi
CC BY-SA 3.0
2018-04-23T12:52:47.587Z
2018-04-23T15:49:49.233Z
2018-04-23T15:49:49.233Z
Raghav Sood
<p>If a HD Wallet generates 1,000 accounts for example. </p>

<p>How would they keep track of all of the account's balances, if someone for example, paid into the 455th account, they would want to update the wallet balance.</p>

<p>I imagine that storing all of the accounts would not be efficient, and querying the blockchain or an API for all 1,000 would not be efficient either.</p>

<p>Using an isolated use case:</p>

<p>If an API HD wallet generates 1,000 addresses, it cannot query all 1,000 each time the user starts up the phone, and it cannot do it periodically because with everyone hitting the same endpoint, it would be a bottleneck.</p>

<p>I do remember blockchain.info allowing you to query with your extended public key, I imagine that they iterate through the accounts the same way using the gap limit.</p>

CC BY-SA 3.0
2018-04-23T13:29:05.583Z
Can you elaborate what do you mean by  API HD Wallet (its not a commonly used term). Do you mean an SPV wallet like electrum  ?
dark knight
CC BY-SA 3.0
2018-04-23T13:31:00.693Z
@darkknight A wallet that uses API services like blockchain.info to get the balance and send transactions, instead of being an SPV
Kyle Graham
<api><json-rpc><bip32-hd-wallets>
How Do HD Wallets Keep Track of all accounts?
Kyle Graham
# How Do HD Wallets Keep Track of all accounts?

(Asked by: username_0 on 2018-04-23)

If a HD Wallet generates 1,000 accounts for example.

How would they keep track of all of the account's balances, if someone for example, paid into the 455th account, they would want to update the wallet balance.

I imagine that storing all of the accounts would not be efficient, and querying the blockchain or an API for all 1,000 would not be efficient either.

Using an isolated use case:

If an API HD wallet generates 1,000 addresses, it cannot query all 1,000 each time the user starts up the phone, and it cannot do it periodically because with everyone hitting the same endpoint, it would be a bottleneck.

I do remember blockchain.info allowing you to query with your extended public key, I imagine that they iterate through the accounts the same way using the gap limit.

**Tags:** api, json-rpc, bip32-hd-wallets

> **Comments:**
> **username_1**: Can you elaborate what do you mean by  API HD Wallet (its not a commonly used term). Do you mean an SPV wallet like electrum  ?  
> **username_0**: @username_1 A wallet that uses API services like blockchain.info to get the balance and send transactions, instead of being an SPV  

---

## 1 answers

### Answer by username_2 on 2018-04-23. Score: 9 | Accepted answer

In general, HD Wallets use the following logic for how many addresses to query, and when to stop:

1. Start with account 0, generate `gap limit` number of addresses (usually 20)
2. Check for any transactions in those addresses
3. If there are no transactions, stop searching for new addresses and accounts
4. If there are transactions, generator `gap limit` more from the index of the last address with transactions on it. Additionally, also do this process for account 1 (you only check for n+1 accounts if Account n has transactions

The general idea is that no accounts will exist if the previous account is unused, and that no addresses in an account will be used after a single, consecutive block of `gap limit` number of unused addresses.

Companies like BitGo who offer HD wallet services to exchanges etc. where you can easily have thousands of unused addresses in a row, and then addresses with transactions, generally ignore the gap limit. They instead maintain a list of all generated addresses and index them as individual addresses, not as an HD wallet.

On the API side of things, wallets generally try to keep a running index. They will maintain a cache of known transactions, and keep appending to it. This way, when you start it up, it only scans for transactions in the new blocks since the last time it scanned. This allows even light clients such as Electrum or Ledger/Trezor wallets to serve thousands of users with just a handful of nodes, since each wallet will not requery for previous transactions it alread knows about.

Naturally, if you import a frequently used HD wallet to a new wallet or delete the cache, the initial sync can take very long. However, subsequent syncs of the wallet will be much faster.

In addition to this, HD wallets will often rely on a full node (such as Bitcoin Core) or a system such as ElectrumX or Insight. All of these are capable of maintaining a full index. Bitcoin Core maintains only a running index across known addresses (which is why you need to rescan when you import an address). ElectrumX and Insight maintain an augmented index which keeps track of all transactions, and can quickly return data for any given address.

HD Wallets do continue to check for transactions across all used addresses. They simply don't rescan the entire chain every time. But for every new block, every tx in that block will be checked for every address in the HD wallet. This check can be done quickly via bloom filters or preindexed data such as from Electrum or Insight, without reaching a bottleneck.

> **Comments:**
> **username_0**: I understand how HD wallets work in theory, thanks for the clarification.  My question concerns how it works in practice. How would a HD Wallet, check if any of the previously generated addresses had funds sent to them efficiently? If they had 1,000 this would be a bottleneck for an API Wallet  
> **username_2**: @username_0 Most wallets deal with this by maintaining a list of known transactions and only requesting for txs in newer blocks instead of scanning the entire chain for every address each time. I've edited the answer to include this.  
> **username_0**: So if the wallet is maintaining 1,000 accounts, it will still scan the blockchain for each account, but it will be looking in newer blocks? If this is correct, then it does not seem efficient as users can potentially have thousands of accounts. With each account having the possibility of having funds sent to them.  
> **username_2**: Scanning for new transactions has the same cost whether you're scanning for 1 address or 10000. Most implementations use some sort of  [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) to make it fast. Once the filter is constructed, checking each tx to see if it for your address is trivial.  
> **username_0**: Yes exactly. It would be wasteful to scan for all 1,000 addresses(pass used addresses) each time the user opens the HD Wallet. If we do not check to see if someone has sent funds to a previously used address, the HD Wallet, will show an incorrect balance. My question is how do HD Wallets(mobile ones) deal with this?  
> **username_2**: I've edited the answer some more. In short, the wallet *do* check every used address as well. It's just that there are ways to do it efficiently to a point where it's not a bottleneck in most cases  
> **username_3**: @username_2 How is making 1,000+ HTTP requests every time you start the app not a bottleneck?  
> **username_2**: Any place dealing with significant amounts of addresses is maintaining a constantly updated state for each address by scanning the mempool and each new block. The applications using the address balance, transactions etc. then query the monitoring applications, which are optimized to not require 1000 queries  
> **username_3**: @username_2 How can a mobile wallet, which is relying on SPV or ElectrumX, be constantly scanning the mempool and new blocks when it's offline? Wouldn't it need to query the server for all the addresses in it every time it goes online?  
> **username_2**: Using bloomf filters allows you to query reasonably large batches of addresses at a time - most electrum wallets that have been used for a while already scan thousands of addresses in a matter of minutes. That works well enough for personal wallets. At an exchange level, you have custom systems meant to deal with millions of addresses  
> **username_3**: I'm implementing a mobile wallet right now and I'd like to learn how to do this, but I don't see any ElectrumX methods that allow batch queries https://electrumx.readthedocs.io/en/latest/protocol-methods.html I'm also using Esplora and I don't see such batch queries there either https://github.com/Blockstream/esplora/blob/master/API.md  

---

