<p>For a project I need to write a scraper that sends a GetAddr request to all Bitcoin nodes. I determine the reachable nodes using the bitnodes.io API. Then I go through all the IPs and send a request using the python-nmap library and the bitcoin-getaddr script from nmap.</p>
<p>The code to do this looks like the following:</p>
<pre><code>def send_get_addr_requests(ip, port):
    ip_addr = ipaddress.ip_address(ip)
    # check if it is an ipv6 address and fix arguments in this case
    arguments_get_addr = '--script bitcoin-getaddr -d -Pn'
    if isinstance(ip_addr, ipaddress.IPv6Address):
        arguments_get_addr += ' -6'
    # send get_addr request to the node    
    nm = nmap.PortScanner()
    response_get_addr = nm.scan(ip, port, arguments=arguments_get_addr)
    return response_get_addr
</code></pre>
<p>However, as I have found, the nmap script (especially the bitcoin.lua file) is relatively unreliable and in most cases does not return a result on the request. I have already modified the GetNodes function as follows:</p>
<pre><code> getNodes = function(self)
    local req = Request.GetAddr:new(
      self.host, self.port, self.lhost, self.lport
    )
    local status, err = self.socket:send(tostring(req))
    if ( not(status) ) then
      return false, &quot;Failed to send \&quot;GetAddr\&quot; request to server&quot;
    end

    local status, response = Response.recvPacket(self.socket, self.version)
    local all_addrs = {}
    -- limit is not usable because it is unclear how many packets are returned
    -- thats why we take a timer
    -- local limit = 20
    local startTime = os.time()
    local endTime = startTime + 60  -- wait 60 seconds max
    -- Usually sends an addr response with 1 address,
    -- then some other stuff like getheaders or ping,
    -- then one with hundreds of addrs.
    while #all_addrs &lt;= 1 and os.time() &lt; endTime and status do
      -- limit = limit - 1
      status, response = Response.recvPacket(self.socket, self.version)
      if status and response.cmd == &quot;addr&quot; then
        for _, addr in ipairs(response.addresses) do
          all_addrs[#all_addrs+1] = addr
        end
      end
    end
    return #all_addrs &gt; 0, all_addrs
  end,
</code></pre>
<p>I added a timer and removed the limit. Since the requested nodes send many packets before the GetAddr response, this approach works much better. I also set the replaced version to 70015 and adjusted the ping/pong messages of the script to the newest implementation.</p>
<p>However, in many cases (about 2000 out of 7000) I still do not receive a GetAddr response. The function Response.recvPacket(self.socket, self.version) from the bitcoin.lua file gets at some point only a status = false and a response = EOF ...</p>
<p>So my question is if this is normal or how I can further customize the script to get a more reliable response? When I manually type in the terminal &quot;nmap -Pn -d -script bitcoin-getaddr -p 8333 &quot; I get a response from time to time, but not reliable and rather rare. So the requested nodes don't block the GetAddr requests, which makes the whole thing even more complicated to understand.</p>
<p>Thanks in advance for any help!</p>
<p>Here is another Wireshark screenshot. I executed the command several times in a row (14x) in the terminal. In the last attempt I finally got the GetAddr response. The [unknown command] packets are &quot;sendheaders&quot; and &quot;sendcmpct&quot; packets (and in the last attempt &quot;feefilter&quot;). <a href="https://i.sstatic.net/TnAB7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TnAB7.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2022-03-07T15:55:00.100Z
Have you used a network sniffer, such as wireshark, to examine to detailed packet by packet conversation for a successful vs unsuccessful query?
RedGrittyBrick
CC BY-SA 4.0
2022-03-07T18:09:59.677Z
Yes, I have actually done that. I have added a Wireshark screenshot to the question. There I simply executed the command several times in a row (14x) in the terminal. In the last attempt I finally got the GetAddr response.
Brian
CC BY-SA 4.0
2022-03-07T21:13:20.513Z
Well, that didn't shed any light on the matter, I'm out of ideas, sorry.
RedGrittyBrick
CC BY-SA 4.0
2022-03-07T21:27:02.660Z
Okay, thanks for thinking about it anyway!
Brian
<nodes><peers><peer-discovery>
The nmap script bitcoin-getaddr often does not return a response
Brian
# The nmap script bitcoin-getaddr often does not return a response

(Asked by: username_0 on 2022-03-07)

For a project I need to write a scraper that sends a GetAddr request to all Bitcoin nodes. I determine the reachable nodes using the bitnodes.io API. Then I go through all the IPs and send a request using the python-nmap library and the bitcoin-getaddr script from nmap.

The code to do this looks like the following:

```
def send_get_addr_requests(ip, port):
    ip_addr = ipaddress.ip_address(ip)
    # check if it is an ipv6 address and fix arguments in this case
    arguments_get_addr = '--script bitcoin-getaddr -d -Pn'
    if isinstance(ip_addr, ipaddress.IPv6Address):
        arguments_get_addr += ' -6'
    # send get_addr request to the node    
    nm = nmap.PortScanner()
    response_get_addr = nm.scan(ip, port, arguments=arguments_get_addr)
    return response_get_addr

```

However, as I have found, the nmap script (especially the bitcoin.lua file) is relatively unreliable and in most cases does not return a result on the request. I have already modified the GetNodes function as follows:

```
 getNodes = function(self)
    local req = Request.GetAddr:new(
      self.host, self.port, self.lhost, self.lport
    )
    local status, err = self.socket:send(tostring(req))
    if ( not(status) ) then
      return false, "Failed to send \"GetAddr\" request to server"
    end

    local status, response = Response.recvPacket(self.socket, self.version)
    local all_addrs = {}
    -- limit is not usable because it is unclear how many packets are returned
    -- thats why we take a timer
    -- local limit = 20
    local startTime = os.time()
    local endTime = startTime + 60  -- wait 60 seconds max
    -- Usually sends an addr response with 1 address,
    -- then some other stuff like getheaders or ping,
    -- then one with hundreds of addrs.
    while #all_addrs <= 1 and os.time() < endTime and status do
      -- limit = limit - 1
      status, response = Response.recvPacket(self.socket, self.version)
      if status and response.cmd == "addr" then
        for _, addr in ipairs(response.addresses) do
          all_addrs[#all_addrs+1] = addr
        end
      end
    end
    return #all_addrs > 0, all_addrs
  end,

```

I added a timer and removed the limit. Since the requested nodes send many packets before the GetAddr response, this approach works much better. I also set the replaced version to 70015 and adjusted the ping/pong messages of the script to the newest implementation.

However, in many cases (about 2000 out of 7000) I still do not receive a GetAddr response. The function Response.recvPacket(self.socket, self.version) from the bitcoin.lua file gets at some point only a status = false and a response = EOF ...

So my question is if this is normal or how I can further customize the script to get a more reliable response? When I manually type in the terminal "nmap -Pn -d -script bitcoin-getaddr -p 8333 " I get a response from time to time, but not reliable and rather rare. So the requested nodes don't block the GetAddr requests, which makes the whole thing even more complicated to understand.

Thanks in advance for any help!

Here is another Wireshark screenshot. I executed the command several times in a row (14x) in the terminal. In the last attempt I finally got the GetAddr response. The [unknown command] packets are "sendheaders" and "sendcmpct" packets (and in the last attempt "feefilter"). [![enter image description here](https://i.sstatic.net/TnAB7.png)](https://i.sstatic.net/TnAB7.png)

**Tags:** nodes, peers, peer-discovery

> **Comments:**
> **username_1**: Have you used a network sniffer, such as wireshark, to examine to detailed packet by packet conversation for a successful vs unsuccessful query?  
> **username_0**: Yes, I have actually done that. I have added a Wireshark screenshot to the question. There I simply executed the command several times in a row (14x) in the terminal. In the last attempt I finally got the GetAddr response.  
> **username_1**: Well, that didn't shed any light on the matter, I'm out of ideas, sorry.  
> **username_0**: Okay, thanks for thinking about it anyway!  

---

## 0 answers

