<p>The only place clear is used on the mempool is after all peers are down. Presumably there are references to it that you can't just leave dangling.</p>
<p>See CTxMemPool::Expire in txmempool.cpp for how expiration works.  It wouldn't be hard to modify that existing function to just remove everything at a particular point.</p>
<p>I assume you're the author of <a href="http://people.bu.edu/staro/Characterizing_churn.pdf" rel="noreferrer">http://people.bu.edu/staro/Characterizing_churn.pdf</a> ?</p>
<p>I saw that paper back on May 22nd and I found the claims of the paper to be false to the best of my ability to determine.  If the paper was correct nodes would see large numbers of getblocktxn messages-- patching in missing transactions that showed up in blocks, yet they see fairly few.</p>
<p>To give some concrete figures, yesterday my node received 210 getblocktxn requests from ~66 peers. There were 162 blocks. So that is about 0.0196 request per block*peer. I sent 3097 compact blocks during the same interval. My own rate of having missing transactions and needing a getblocktxn was 6.7%, which is not wildly inconsistent from what I see with other peers: it's higher because I'll only see 1/N-th my peers getblocktxn, since I'm not always the first to send them a block. But if I assume that the rate that I sent compact blocks (28.9%) reflects the rate that I'm first, I estimate my peers getblocktxn rate to be almost exactly the same as mine.</p>
<p>I did more analysis with multiple nodes back when I first saw the paper, but I'm not sure where I put the figures. :)</p>
<p>I believe the paper makes an error taking stats from &quot;bitnodes&quot;:   There are enormous numbers of fake nodes on the networks which don't behave much all like real nodes-- e.g. for the purpose of spying on transactions to deanonymize bitcoin users or academic research, most of &quot;nodes&quot; displayed on bitnodes are of this sort. It's irrelevant to the network if fake 'monitoring' nodes churn a lot.</p>
<p>You might also be interested in the FIBRE protocol, which renders block propagation times close to to independent of the presence of a few missing transactions (<a href="https://bitcoin.stackexchange.com/questions/56485/can-someone-please-explain-fibre-to-me-like-im-5-and-why-is-it-useful">Can someone please explain FIBRE to me like I&#39;m 5 and why is it useful?</a>) and Erlay (<a href="https://arxiv.org/abs/1905.10518" rel="noreferrer">https://arxiv.org/abs/1905.10518</a>) which uses reconciliation for transaction relay  to reduce relay overheads.</p>

CC BY-SA 4.0
2020-07-03T16:25:44.527Z
Thank you, Gregory, for pointing to `CTxMemPool::Expire`. I'll see if I can find what I am looking for. I also appreciate you sharing your analysis. I don't mean to point fingers or offend anybody in general but if what you say is true, then this surely raises many questions towards several past research work and blog posts from well-known websites in the Bitcoin space that use data from bitnodes.
an4s
CC BY-SA 4.0
2020-07-02T18:49:04.897Z
2020-07-02T22:50:15.427Z
2020-07-02T22:50:15.427Z
G. Maxwell
<p><strong>Background:</strong></p>
<p>For a research project, I am trying to periodically clear the content of my Bitcoin node's mempool. For this purpose I have created a new thread in <code>init.cpp</code> using the <code>boost</code> <code>threadgroup</code> (I have experimented with this successfully with a different project in the past and I am sure this is not the cause of the problem) in the Bitcoin Core software. In the new thread, I periodically call a function <code>func</code> (defined below), the sole purpose of which is to clear the content of the node's mempool.</p>
<pre><code>void func()
{
    LOCK2(cs_main, mempool.cs);
    AssertLockHeld(cs_main);
    AssertLockHeld(mempool.cs);
    mempool.clear();
}
</code></pre>
<p>I check for and lock critical sections that I think I should before I modify the mempool (the code above mimics what happens when a new transaction is added to the mempool; I basically looked at what happens in the <code>NetMsgType::TX</code> branch of <code>ProcessMessage</code> in <code>net_processing.cpp</code> and then <code>AcceptToMemoryPoolWorker</code> in <code>validation.cpp</code> and mimic that).</p>
<p><strong>Issue:</strong></p>
<p>When I run Bitcoin after successfully compiling my code, I sometimes <em>immediately</em> (i.e., as soon as the GUI window pops up) and sometimes much later in the execution face a segfault that causes the program to crash.</p>
<pre><code>ubuntu@root:~$ bitcoin-qt -mpctimeout=1
Segmentation fault (core dumped)
</code></pre>
<p>In the code above, the function <code>func</code> is executed every 1 minute (represented by <code>mpctimeout=1</code>).</p>
<p><strong>Question:</strong></p>
<p>What could be causing the segfault? Could it possibly be race conditions when trying to get a lock on the mempool or execution of the main thread? It is my understanding that <code>LOCK2</code> should keep trying to get a lock on the critical sections when they become available in case they are already locked by another thread/process. Since these locks only exist within the scope of the function <code>func</code>, they <em>should</em> ideally be released when returning from the function.</p>
<p><strong>Edit # 1</strong>:</p>
<p>Some debugging shows that the segfault doesn't occur while <code>func</code> is being executed but while the thread is put in sleep for <code>mpctimeout</code> amount of time. <strong>This leaves me wondering whether clearing the mempool has some long-term effects?</strong> I have made sure that the lock is released by verifying that transactions are indeed accepted to the mempool while the new thread is asleep. The segfault still occurs even if the <code>mpctimeout</code> value is large (e.g., 1 hour).</p>
<p>FWIW, I use <code>boost::this_thread::sleep_for</code> to put the thread to sleep. I assume this works without fault and that the segfault is instead caused in the long run due to clearing the mempool. Any tips?</p>
<p><strong>Edit # 2:</strong></p>
<p>Below is the output of debugging the binary through <code>gdb</code>:</p>
<pre><code>(gdb) run
Starting program: /usr/local/bin/bitcoin-qt
[Thread debugging using libthread_db enabled]
Using host libthread_db library &quot;/lib/x86_64-linux-gnu/libthread_db.so.1&quot;.
[New Thread 0x7fffe9900700 (LWP 25643)]
[New Thread 0x7fffdf37f700 (LWP 25644)]
[New Thread 0x7fffdeb7e700 (LWP 25645)]
[New Thread 0x7fffdd0e7700 (LWP 25646)]
[New Thread 0x7fffc7cd2700 (LWP 25648)]
[New Thread 0x7fffc74d1700 (LWP 25649)]
[New Thread 0x7fffc6cd0700 (LWP 25650)]
[New Thread 0x7fffc64cf700 (LWP 25651)]
[New Thread 0x7fffc5cce700 (LWP 25652)]
[New Thread 0x7fffc54cd700 (LWP 25653)]
[New Thread 0x7fffc4ccc700 (LWP 25654)]
[New Thread 0x7fffa7fff700 (LWP 25655)]
[New Thread 0x7fffa77fe700 (LWP 25656)]
[New Thread 0x7fffa6ffd700 (LWP 25657)]
[New Thread 0x7fffa67fc700 (LWP 25658)]
[New Thread 0x7fffa5ffb700 (LWP 25659)]
[New Thread 0x7fffa57fa700 (LWP 25660)]
[New Thread 0x7fffa4f63700 (LWP 25661)]
[New Thread 0x7fff8dffd700 (LWP 25662)]
[New Thread 0x7fff8d7fc700 (LWP 25663)]
[New Thread 0x7fff8cffb700 (LWP 25664)]
[New Thread 0x7fff7bfff700 (LWP 25665)]
[New Thread 0x7fff7b7fe700 (LWP 25666)]
[New Thread 0x7fff7affd700 (LWP 25667)]
[New Thread 0x7fff7a7fc700 (LWP 25668)]
[New Thread 0x7fff79ffb700 (LWP 25669)]
[New Thread 0x7fff797fa700 (LWP 25670)]
[New Thread 0x7fff78ff9700 (LWP 25671)]
[New Thread 0x7fff53fff700 (LWP 25672)]
[New Thread 0x7fff537fe700 (LWP 25673)]
[New Thread 0x7fff4a174700 (LWP 25677)]
[New Thread 0x7ffed1a7b700 (LWP 25678)]
[Thread 0x7ffed1a7b700 (LWP 25678) exited]
[New Thread 0x7ffed127a700 (LWP 25679)]
[New Thread 0x7ffed0a79700 (LWP 25680)]
[New Thread 0x7ffecbfff700 (LWP 25681)]
[New Thread 0x7ffecb7fe700 (LWP 25682)]
[New Thread 0x7ffecaffd700 (LWP 25683)]
[New Thread 0x7ffeca7fc700 (LWP 25684)]
[New Thread 0x7ffec9ffb700 (LWP 25685)]
[New Thread 0x7ffec97fa700 (LWP 25686)]
[New Thread 0x7ffec8ff9700 (LWP 25687)]
[Thread 0x7ffed127a700 (LWP 25679) exited]
[Thread 0x7ffecb7fe700 (LWP 25682) exited]

Thread 40 &quot;bitcoin-msghand&quot; received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec9ffb700 (LWP 25685)]
__gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x1a45fe36190ee438)
    at /usr/include/c++/7/ext/atomicity.h:96
96        __atomic_add(__mem, __val);
(gdb)
</code></pre>
<p>I suppose that the segfault is caused when trying to update an <code>atomic</code> variable.</p>
<p><strong>Edit # 3:</strong></p>
<p><code>gdb</code> backtrace</p>
<pre><code>#0  0x0000555555854178 in __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0xf) at /usr/include/c++/7/ext/atomicity.h:96
#1  0x0000555555854178 in std::_Sp_counted_base&lt;(__gnu_cxx::_Lock_policy)2&gt;::_M_add_ref_copy() (this=0x7) at /usr/include/c++/7/bits/shared_ptr_base.h:138
#2  0x0000555555854178 in std::__shared_count&lt;(__gnu_cxx::_Lock_policy)2&gt;::__shared_count(std::__shared_count&lt;(__gnu_cxx::_Lock_policy)2&gt; const&amp;) (__r=..., this=0x7ffef5ff71b8)
    at /usr/include/c++/7/bits/shared_ptr_base.h:691
#3  0x0000555555854178 in std::__shared_ptr&lt;CTransaction const, (__gnu_cxx::_Lock_policy)2&gt;::__shared_ptr(std::__shared_ptr&lt;CTransaction const, (__gnu_cxx::_Lock_policy)2&gt; const&amp;)
    (this=0x7ffef5ff71b0) at /usr/include/c++/7/bits/shared_ptr_base.h:1121
#4  0x0000555555854178 in std::shared_ptr&lt;CTransaction const&gt;::shared_ptr(std::shared_ptr&lt;CTransaction const&gt; const&amp;) (this=0x7ffef5ff71b0)
    at /usr/include/c++/7/bits/shared_ptr.h:119
#5  0x0000555555854178 in CTxMemPoolEntry::GetSharedTx() const (this=0x7ffee12106a0) at ./txmempool.h:101
#6  0x0000555555854178 in PartiallyDownloadedBlock::InitData(CBlockHeaderAndShortTxIDs const&amp;, std::vector&lt;std::pair&lt;uint256, std::shared_ptr&lt;CTransaction const&gt; &gt;, std::allocator&lt;std::pair&lt;uint256, std::shared_ptr&lt;CTransaction const&gt; &gt; &gt; &gt; const&amp;) (this=this@entry=0x7ffef5ff9f30, cmpctblock=..., extra_txn=std::vector of length 100, capacity 100 = {...})
    at blockencodings.cpp:114
#7  0x00005555556a2d4a in ProcessMessage(CNode*, std::__cxx11::string const&amp;, CDataStream&amp;, int64_t, CChainParams const&amp;, CConnman*, std::atomic&lt;bool&gt; const&amp;, bool)
    (pfrom=pfrom@entry=0x7ffedc0010d0, strCommand=&quot;cmpctblock&quot;, vRecv=..., nTimeReceived=1593549860206352, chainparams=..., connman=0x7fffbc04ba20, interruptMsgProc=..., enable_bip61=false) at net_processing.cpp:3051
#8  0x00005555556a6f01 in PeerLogicValidation::ProcessMessages(CNode*, std::atomic&lt;bool&gt;&amp;) (this=&lt;optimized out&gt;, pfrom=0x7ffedc0010d0, interruptMsgProc=...) at net_processing.cpp:3629
#9  0x000055555566b19e in CConnman::ThreadMessageHandler() (this=0x7fffbc04ba20) at net.cpp:1966
</code></pre>

CC BY-SA 4.0
2020-06-28T21:41:34.240Z
Did you know that you can run with no mempool using `blocksonly`? I fail to see any reason to manually clear it every minute.
Claris
CC BY-SA 4.0
2020-06-28T22:35:58.277Z
@Anonymous I only clear the mempool every minute to make sure what I am trying to do works. In my experiment, this is going to be a larger value. My research project requires clearing the mempool so I cannot use the `blocksonly` mode.
an4s
CC BY-SA 4.0
2020-06-30T18:31:11.847Z
Have you tried running this in GDB and just waiting for it to segfault? It should tell you where that happens. What is probably going on is that something has a reference to a mempool entry but when you clear the mempool in another thread, using that reference causes a segfault.
Ava Chow
CC BY-SA 4.0
2020-06-30T18:36:48.373Z
Why do you need to periodically clear the mempool? What in particular are you trying to research? I'm sure there's a way to do this without modifying the Bitcoin software. It should be possible to emulate the mempool state by having another software connected via ZMQ.
Ava Chow
CC BY-SA 4.0
2020-06-30T20:05:38.197Z
@AndrewChow I have added the output of the running the binary through `gdb`. It seems that the fault is caused when trying to update an `atomic` variable. Do you have any tips on which variables I should specifically debug (since there are so many in the software)?
an4s
CC BY-SA 4.0
2020-06-30T20:07:26.673Z
@AndrewChow re: "What is probably going ... causes a segfault."

I assumed that the `mempool.clear()` function (implemented in `txmempool.cpp`) should have taken care of any references to the mempool whenever it is cleared. Could there be something missing in the function then?
an4s
CC BY-SA 4.0
2020-06-30T20:35:59.210Z
@an4s In gdb, when you hit the segfault, use the `bt` command to get a backtrace. This will tell you exactly where the segfault occurred. Clearing a data structure will not clear the variables that point to data that was inside of it. There are probably variables in other functions in other threads that have pointers or references to now-cleared mempool data.
Ava Chow
CC BY-SA 4.0
2020-06-30T21:30:17.727Z
@AndrewChow thanks for the tip. The backtrace and some more debugging shows that the fault is caused when a call to `GetSharedTx()` is made in `InitData()` for a partially downloaded compact block in `net_processing.cpp`. I suppose then it makes sense to delay clearing the mempool if there are still blocks in flight. AFAIK, the `mapBlocksInFlight` data structure keeps track of all blocks currently in flight. However, it is defined in a namespace defined in `net_processing.cpp` and it may be tricky to access it in my new/custom thread. Is there another way to check for blocks in flight?
an4s
CC BY-SA 4.0
2020-06-30T22:11:58.210Z
I'm still not quite sure why you want to do this. There's almost certainly an easier way to get the information you want. All this will cause is a huge amount of rejects on your node.
Claris
CC BY-SA 4.0
2020-07-01T00:33:20.450Z
Could you say why you are trying to do this? What are you trying to observe. As nonted by Anonymous, this will have a lot of side effects on your node and potentially give you erroneous or biased results.
Ava Chow
CC BY-SA 4.0
2020-07-01T00:55:59.693Z
@AndrewChow We published a paper at ICBC'20 on the topic of orphan transactions in Bitcoin and are performing additional experiments to understand when such transactions are most likely to show up. Along that line, the purpose of clearing the mempool is to see how likely it may cause transactions to become orphans.
an4s
CC BY-SA 4.0
2020-07-01T10:33:53.220Z
Instead of clearing the mempool like you are doing, could you turn off bitcoin core, delete the mempool.dat file and then start it again? This will clear the mempool...
Pedro
CC BY-SA 4.0
2020-07-01T15:02:31.130Z
@Pedro yes, I already started doing that when clearing the mempool from within the software started resulting in segfaults. I am still wondering if I can make this work (hence this question). Thanks!
an4s
CC BY-SA 4.0
2020-07-02T18:12:35.887Z
@an4s That concept doesn't really make a whole lot of sense, as a question to be asking to begin with. Transactions arriving without pre-existing inputs in your UTXO or your mempool are pretty rare as they're not relayed around the network. By "clearing" your mempool you're just going to result in seeing huge numbers of them artificially.
Claris
CC BY-SA 4.0
2020-07-02T18:19:42.787Z
@Anonymous the idea is to emulate churn.
an4s
CC BY-SA 4.0
2020-07-02T18:20:48.507Z
@an4s Can't that be achieved by just setting your mempool limit arbitrarily low?
Claris
CC BY-SA 4.0
2020-07-02T18:24:24.357Z
@an4s In any case, I'd suggest talking to some of the bitcoin domain experts about what you're doing and get some feedback. On the surface of it what you're describing sounds a little mislead and could probably do with some input to make it more relevant. You're more than welcome to ask me questions in the chat here if you want to.
Claris
CC BY-SA 4.0
2020-07-02T18:39:47.017Z
@an4s I believe I've read your paper, there's some serious issues with the conclusions you've drawn from faulty bitnodes.io data. There is not significant churn of peers on the Bitcoin network, as evidenced by my node having hundreds of extremely long lived connections, which would not be the case if the claim that there was a significant amount of churn in operational nodes was correct. There's simply no property which would cause this to be the case.
Claris
<bitcoin-core-development><mempool>
Segfault when clearing mempool. What am I doing wrong?
an4s
# Segfault when clearing mempool. What am I doing wrong?

(Asked by: username_0 on 2020-06-28)

**Background:**

For a research project, I am trying to periodically clear the content of my Bitcoin node's mempool. For this purpose I have created a new thread in `init.cpp` using the `boost` `threadgroup` (I have experimented with this successfully with a different project in the past and I am sure this is not the cause of the problem) in the Bitcoin Core software. In the new thread, I periodically call a function `func` (defined below), the sole purpose of which is to clear the content of the node's mempool.

```
void func()
{
    LOCK2(cs_main, mempool.cs);
    AssertLockHeld(cs_main);
    AssertLockHeld(mempool.cs);
    mempool.clear();
}

```

I check for and lock critical sections that I think I should before I modify the mempool (the code above mimics what happens when a new transaction is added to the mempool; I basically looked at what happens in the `NetMsgType::TX` branch of `ProcessMessage` in `net_processing.cpp` and then `AcceptToMemoryPoolWorker` in `validation.cpp` and mimic that).

**Issue:**

When I run Bitcoin after successfully compiling my code, I sometimes *immediately* (i.e., as soon as the GUI window pops up) and sometimes much later in the execution face a segfault that causes the program to crash.

```
ubuntu@root:~$ bitcoin-qt -mpctimeout=1
Segmentation fault (core dumped)

```

In the code above, the function `func` is executed every 1 minute (represented by `mpctimeout=1`).

**Question:**

What could be causing the segfault? Could it possibly be race conditions when trying to get a lock on the mempool or execution of the main thread? It is my understanding that `LOCK2` should keep trying to get a lock on the critical sections when they become available in case they are already locked by another thread/process. Since these locks only exist within the scope of the function `func`, they *should* ideally be released when returning from the function.

**Edit # 1**:

Some debugging shows that the segfault doesn't occur while `func` is being executed but while the thread is put in sleep for `mpctimeout` amount of time. **This leaves me wondering whether clearing the mempool has some long-term effects?** I have made sure that the lock is released by verifying that transactions are indeed accepted to the mempool while the new thread is asleep. The segfault still occurs even if the `mpctimeout` value is large (e.g., 1 hour).

FWIW, I use `boost::this_thread::sleep_for` to put the thread to sleep. I assume this works without fault and that the segfault is instead caused in the long run due to clearing the mempool. Any tips?

**Edit # 2:**

Below is the output of debugging the binary through `gdb`:

```
(gdb) run
Starting program: /usr/local/bin/bitcoin-qt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe9900700 (LWP 25643)]
[New Thread 0x7fffdf37f700 (LWP 25644)]
[New Thread 0x7fffdeb7e700 (LWP 25645)]
[New Thread 0x7fffdd0e7700 (LWP 25646)]
[New Thread 0x7fffc7cd2700 (LWP 25648)]
[New Thread 0x7fffc74d1700 (LWP 25649)]
[New Thread 0x7fffc6cd0700 (LWP 25650)]
[New Thread 0x7fffc64cf700 (LWP 25651)]
[New Thread 0x7fffc5cce700 (LWP 25652)]
[New Thread 0x7fffc54cd700 (LWP 25653)]
[New Thread 0x7fffc4ccc700 (LWP 25654)]
[New Thread 0x7fffa7fff700 (LWP 25655)]
[New Thread 0x7fffa77fe700 (LWP 25656)]
[New Thread 0x7fffa6ffd700 (LWP 25657)]
[New Thread 0x7fffa67fc700 (LWP 25658)]
[New Thread 0x7fffa5ffb700 (LWP 25659)]
[New Thread 0x7fffa57fa700 (LWP 25660)]
[New Thread 0x7fffa4f63700 (LWP 25661)]
[New Thread 0x7fff8dffd700 (LWP 25662)]
[New Thread 0x7fff8d7fc700 (LWP 25663)]
[New Thread 0x7fff8cffb700 (LWP 25664)]
[New Thread 0x7fff7bfff700 (LWP 25665)]
[New Thread 0x7fff7b7fe700 (LWP 25666)]
[New Thread 0x7fff7affd700 (LWP 25667)]
[New Thread 0x7fff7a7fc700 (LWP 25668)]
[New Thread 0x7fff79ffb700 (LWP 25669)]
[New Thread 0x7fff797fa700 (LWP 25670)]
[New Thread 0x7fff78ff9700 (LWP 25671)]
[New Thread 0x7fff53fff700 (LWP 25672)]
[New Thread 0x7fff537fe700 (LWP 25673)]
[New Thread 0x7fff4a174700 (LWP 25677)]
[New Thread 0x7ffed1a7b700 (LWP 25678)]
[Thread 0x7ffed1a7b700 (LWP 25678) exited]
[New Thread 0x7ffed127a700 (LWP 25679)]
[New Thread 0x7ffed0a79700 (LWP 25680)]
[New Thread 0x7ffecbfff700 (LWP 25681)]
[New Thread 0x7ffecb7fe700 (LWP 25682)]
[New Thread 0x7ffecaffd700 (LWP 25683)]
[New Thread 0x7ffeca7fc700 (LWP 25684)]
[New Thread 0x7ffec9ffb700 (LWP 25685)]
[New Thread 0x7ffec97fa700 (LWP 25686)]
[New Thread 0x7ffec8ff9700 (LWP 25687)]
[Thread 0x7ffed127a700 (LWP 25679) exited]
[Thread 0x7ffecb7fe700 (LWP 25682) exited]

Thread 40 "bitcoin-msghand" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec9ffb700 (LWP 25685)]
__gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x1a45fe36190ee438)
    at /usr/include/c++/7/ext/atomicity.h:96
96        __atomic_add(__mem, __val);
(gdb)

```

I suppose that the segfault is caused when trying to update an `atomic` variable.

**Edit # 3:**

`gdb` backtrace

```
#0  0x0000555555854178 in __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0xf) at /usr/include/c++/7/ext/atomicity.h:96
#1  0x0000555555854178 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_copy() (this=0x7) at /usr/include/c++/7/bits/shared_ptr_base.h:138
#2  0x0000555555854178 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count(std::__shared_count<(__gnu_cxx::_Lock_policy)2> const&) (__r=..., this=0x7ffef5ff71b8)
    at /usr/include/c++/7/bits/shared_ptr_base.h:691
#3  0x0000555555854178 in std::__shared_ptr<CTransaction const, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<CTransaction const, (__gnu_cxx::_Lock_policy)2> const&)
    (this=0x7ffef5ff71b0) at /usr/include/c++/7/bits/shared_ptr_base.h:1121
#4  0x0000555555854178 in std::shared_ptr<CTransaction const>::shared_ptr(std::shared_ptr<CTransaction const> const&) (this=0x7ffef5ff71b0)
    at /usr/include/c++/7/bits/shared_ptr.h:119
#5  0x0000555555854178 in CTxMemPoolEntry::GetSharedTx() const (this=0x7ffee12106a0) at ./txmempool.h:101
#6  0x0000555555854178 in PartiallyDownloadedBlock::InitData(CBlockHeaderAndShortTxIDs const&, std::vector<std::pair<uint256, std::shared_ptr<CTransaction const> >, std::allocator<std::pair<uint256, std::shared_ptr<CTransaction const> > > > const&) (this=this@entry=0x7ffef5ff9f30, cmpctblock=..., extra_txn=std::vector of length 100, capacity 100 = {...})
    at blockencodings.cpp:114
#7  0x00005555556a2d4a in ProcessMessage(CNode*, std::__cxx11::string const&, CDataStream&, int64_t, CChainParams const&, CConnman*, std::atomic<bool> const&, bool)
    (pfrom=pfrom@entry=0x7ffedc0010d0, strCommand="cmpctblock", vRecv=..., nTimeReceived=1593549860206352, chainparams=..., connman=0x7fffbc04ba20, interruptMsgProc=..., enable_bip61=false) at net_processing.cpp:3051
#8  0x00005555556a6f01 in PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) (this=<optimized out>, pfrom=0x7ffedc0010d0, interruptMsgProc=...) at net_processing.cpp:3629
#9  0x000055555566b19e in CConnman::ThreadMessageHandler() (this=0x7fffbc04ba20) at net.cpp:1966

```

**Tags:** bitcoin-core-development, mempool

> **Comments:**
> **username_1**: Did you know that you can run with no mempool using `blocksonly`? I fail to see any reason to manually clear it every minute.  
> **username_0**: @Anonymous I only clear the mempool every minute to make sure what I am trying to do works. In my experiment, this is going to be a larger value. My research project requires clearing the mempool so I cannot use the `blocksonly` mode.  
> **username_2**: Have you tried running this in GDB and just waiting for it to segfault? It should tell you where that happens. What is probably going on is that something has a reference to a mempool entry but when you clear the mempool in another thread, using that reference causes a segfault.  
> **username_2**: Why do you need to periodically clear the mempool? What in particular are you trying to research? I'm sure there's a way to do this without modifying the Bitcoin software. It should be possible to emulate the mempool state by having another software connected via ZMQ.  
> **username_0**: @AndrewChow I have added the output of the running the binary through `gdb`. It seems that the fault is caused when trying to update an `atomic` variable. Do you have any tips on which variables I should specifically debug (since there are so many in the software)?  
> **username_0**: @AndrewChow re: "What is probably going ... causes a segfault."  I assumed that the `mempool.clear()` function (implemented in `txmempool.cpp`) should have taken care of any references to the mempool whenever it is cleared. Could there be something missing in the function then?  
> **username_2**: @username_0 In gdb, when you hit the segfault, use the `bt` command to get a backtrace. This will tell you exactly where the segfault occurred. Clearing a data structure will not clear the variables that point to data that was inside of it. There are probably variables in other functions in other threads that have pointers or references to now-cleared mempool data.  
> **username_0**: @AndrewChow thanks for the tip. The backtrace and some more debugging shows that the fault is caused when a call to `GetSharedTx()` is made in `InitData()` for a partially downloaded compact block in `net_processing.cpp`. I suppose then it makes sense to delay clearing the mempool if there are still blocks in flight. AFAIK, the `mapBlocksInFlight` data structure keeps track of all blocks currently in flight. However, it is defined in a namespace defined in `net_processing.cpp` and it may be tricky to access it in my new/custom thread. Is there another way to check for blocks in flight?  
> **username_1**: I'm still not quite sure why you want to do this. There's almost certainly an easier way to get the information you want. All this will cause is a huge amount of rejects on your node.  
> **username_2**: Could you say why you are trying to do this? What are you trying to observe. As nonted by Anonymous, this will have a lot of side effects on your node and potentially give you erroneous or biased results.  
> **username_0**: @AndrewChow We published a paper at ICBC'20 on the topic of orphan transactions in Bitcoin and are performing additional experiments to understand when such transactions are most likely to show up. Along that line, the purpose of clearing the mempool is to see how likely it may cause transactions to become orphans.  
> **username_3**: Instead of clearing the mempool like you are doing, could you turn off bitcoin core, delete the mempool.dat file and then start it again? This will clear the mempool...  
> **username_0**: @username_3 yes, I already started doing that when clearing the mempool from within the software started resulting in segfaults. I am still wondering if I can make this work (hence this question). Thanks!  
> **username_1**: @username_0 That concept doesn't really make a whole lot of sense, as a question to be asking to begin with. Transactions arriving without pre-existing inputs in your UTXO or your mempool are pretty rare as they're not relayed around the network. By "clearing" your mempool you're just going to result in seeing huge numbers of them artificially.  
> **username_0**: @Anonymous the idea is to emulate churn.  
> **username_1**: @username_0 Can't that be achieved by just setting your mempool limit arbitrarily low?  
> **username_1**: @username_0 In any case, I'd suggest talking to some of the bitcoin domain experts about what you're doing and get some feedback. On the surface of it what you're describing sounds a little mislead and could probably do with some input to make it more relevant. You're more than welcome to ask me questions in the chat here if you want to.  
> **username_1**: @username_0 I believe I've read your paper, there's some serious issues with the conclusions you've drawn from faulty bitnodes.io data. There is not significant churn of peers on the Bitcoin network, as evidenced by my node having hundreds of extremely long lived connections, which would not be the case if the claim that there was a significant amount of churn in operational nodes was correct. There's simply no property which would cause this to be the case.  

---

## 1 answers

### Answer by username_4 on 2020-07-02. Score: 5

The only place clear is used on the mempool is after all peers are down. Presumably there are references to it that you can't just leave dangling.

See CTxMemPool::Expire in txmempool.cpp for how expiration works. It wouldn't be hard to modify that existing function to just remove everything at a particular point.

I assume you're the author of <http://people.bu.edu/staro/Characterizing_churn.pdf> ?

I saw that paper back on May 22nd and I found the claims of the paper to be false to the best of my ability to determine. If the paper was correct nodes would see large numbers of getblocktxn messages-- patching in missing transactions that showed up in blocks, yet they see fairly few.

To give some concrete figures, yesterday my node received 210 getblocktxn requests from ~66 peers. There were 162 blocks. So that is about 0.0196 request per block\*peer. I sent 3097 compact blocks during the same interval. My own rate of having missing transactions and needing a getblocktxn was 6.7%, which is not wildly inconsistent from what I see with other peers: it's higher because I'll only see 1/N-th my peers getblocktxn, since I'm not always the first to send them a block. But if I assume that the rate that I sent compact blocks (28.9%) reflects the rate that I'm first, I estimate my peers getblocktxn rate to be almost exactly the same as mine.

I did more analysis with multiple nodes back when I first saw the paper, but I'm not sure where I put the figures. :)

I believe the paper makes an error taking stats from "bitnodes": There are enormous numbers of fake nodes on the networks which don't behave much all like real nodes-- e.g. for the purpose of spying on transactions to deanonymize bitcoin users or academic research, most of "nodes" displayed on bitnodes are of this sort. It's irrelevant to the network if fake 'monitoring' nodes churn a lot.

You might also be interested in the FIBRE protocol, which renders block propagation times close to to independent of the presence of a few missing transactions ([Can someone please explain FIBRE to me like I'm 5 and why is it useful?](https://bitcoin.stackexchange.com/questions/56485/can-someone-please-explain-fibre-to-me-like-im-5-and-why-is-it-useful)) and Erlay (<https://arxiv.org/abs/1905.10518>) which uses reconciliation for transaction relay to reduce relay overheads.

> **Comments:**
> **username_0**: Thank you, Gregory, for pointing to `CTxMemPool::Expire`. I'll see if I can find what I am looking for. I also appreciate you sharing your analysis. I don't mean to point fingers or offend anybody in general but if what you say is true, then this surely raises many questions towards several past research work and blog posts from well-known websites in the Bitcoin space that use data from bitnodes.  

---

