<p>Usually, issues with port spawning will require admin access/logs to troubleshoot fully.</p>
<p>It's harder with DVR, but you might have to trace each port creation and see where the delay happens.</p>
<p>In Openstack, events are transiting through rabbitmq to provide the end to end orchestration. If one part of the process takes time, the overall process will be delayed.</p>

CC BY-SA 4.0
2022-02-23T02:12:28.083Z
2022-02-23T02:12:28.083Z
LaurentDumont
<p>I have created a private network called &quot;Private_Network&quot; in the range of 192.168.220.0/24 plus a virtual router called &quot;Virtual_Router&quot; inside openstack which is connected to the external network. Then I connect the default gateway of &quot;Private_Network&quot; i.e. 192.168.220.1 to the &quot;Virtual_Router&quot; so that I all VMs connected to &quot;Private_Network&quot; can access the Internet via SNAT.</p>
<p>I used vxlan as overlay network, and a flat provider network.</p>
<p>By connecting &quot;Private_Network&quot; to &quot;Virtual_Router&quot;, two ports are immediately created: <strong>router_interface_distributed</strong> with the IP address of 192.168.220.1 and <strong>router_centralized_snat</strong> with IP address of 192.168.220.45, however, both ports are in the DOWN state for a bit <strong>long and random time</strong> like 2 hours, or 45 minutes, or 20 minutes. I should mention, that rarely the ports get UP as soon as (less than a minute) I connect the &quot;Private_Network&quot; to &quot;Virtual_Router&quot;.</p>
<p><a href="https://i.sstatic.net/vX33r.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vX33r.png" alt="please see the two ports just after creation that both are in the Down state" /></a></p>
<p>I have searched a lot to find out the main reason behind this issue. I am not convinced that server configuration is wrong because I have a few cases in which the two ports gets up right after I connect the &quot;Private_Network&quot; to &quot;Virtual_Router&quot;. So, I tried to look at the log files and noticed there are three main phases to pass in order to get each of those ports to UP state: DHCP, port binding, and L2 provisioning. I changed the log level to DEBUG and investigate the log files in details.</p>
<p>I run the following process several times:</p>
<ol>
<li>create a brand new project in Horizon.</li>
<li>create a brand new virtual network (called &quot;Private_Network&quot;) in the range of 192.168.220.x/24.</li>
<li>create a brand new virtual router (called &quot;Virtual_Router&quot;) connected to external network.</li>
<li>connect port 192.168.220.1 (default gateway) of &quot;Private_Network&quot; to &quot;Virtual_Router&quot;.</li>
<li>cat /var/log/neutron/* | grep snat_port</li>
</ol>
<p>From more than ten cases I experimented, neutron either stuck at the &quot;port binding&quot; phase or &quot;L2 provisioning&quot; phase. When It stuck at &quot;port binding&quot; it takes random time to finish, like 45 minutes, 20 minutes or 10 minutes and Once &quot;port binding&quot; phase is done, &quot;L2 provisioning&quot; phase will be done in less than a minute and port state changes to &quot;UP&quot;. However, In case that &quot;L2 provisioning&quot; phase get stuck, the previous two phases finished in less that 1 minutes, but &quot;L2 provisioning&quot; get stuck for hours. Its so confusing to me why do I see this much delay in getting ports &quot;UP&quot;.</p>
<p>I would appreciate if anybody can assist me to resolve this issue.</p>

<networking><openstack>
Random delay in port binding for distributed_virtual_router and router_centralized_snat in openstack neutron
Amir0015
# Random delay in port binding for distributed_virtual_router and router_centralized_snat in openstack neutron

(Asked by: username_0 on 2022-01-08)

I have created a private network called "Private\_Network" in the range of 192.168.220.0/24 plus a virtual router called "Virtual\_Router" inside openstack which is connected to the external network. Then I connect the default gateway of "Private\_Network" i.e. 192.168.220.1 to the "Virtual\_Router" so that I all VMs connected to "Private\_Network" can access the Internet via SNAT.

I used vxlan as overlay network, and a flat provider network.

By connecting "Private\_Network" to "Virtual\_Router", two ports are immediately created: **router\_interface\_distributed** with the IP address of 192.168.220.1 and **router\_centralized\_snat** with IP address of 192.168.220.45, however, both ports are in the DOWN state for a bit **long and random time** like 2 hours, or 45 minutes, or 20 minutes. I should mention, that rarely the ports get UP as soon as (less than a minute) I connect the "Private\_Network" to "Virtual\_Router".

[![please see the two ports just after creation that both are in the Down state](https://i.sstatic.net/vX33r.png)](https://i.sstatic.net/vX33r.png)

I have searched a lot to find out the main reason behind this issue. I am not convinced that server configuration is wrong because I have a few cases in which the two ports gets up right after I connect the "Private\_Network" to "Virtual\_Router". So, I tried to look at the log files and noticed there are three main phases to pass in order to get each of those ports to UP state: DHCP, port binding, and L2 provisioning. I changed the log level to DEBUG and investigate the log files in details.

I run the following process several times:

1. create a brand new project in Horizon.
2. create a brand new virtual network (called "Private\_Network") in the range of 192.168.220.x/24.
3. create a brand new virtual router (called "Virtual\_Router") connected to external network.
4. connect port 192.168.220.1 (default gateway) of "Private\_Network" to "Virtual\_Router".
5. cat /var/log/neutron/\* | grep snat\_port

From more than ten cases I experimented, neutron either stuck at the "port binding" phase or "L2 provisioning" phase. When It stuck at "port binding" it takes random time to finish, like 45 minutes, 20 minutes or 10 minutes and Once "port binding" phase is done, "L2 provisioning" phase will be done in less than a minute and port state changes to "UP". However, In case that "L2 provisioning" phase get stuck, the previous two phases finished in less that 1 minutes, but "L2 provisioning" get stuck for hours. Its so confusing to me why do I see this much delay in getting ports "UP".

I would appreciate if anybody can assist me to resolve this issue.

**Tags:** networking, openstack


---

## 1 answers

### Answer by username_1 on 2022-02-23. Score: 0

Usually, issues with port spawning will require admin access/logs to troubleshoot fully.

It's harder with DVR, but you might have to trace each port creation and see where the delay happens.

In Openstack, events are transiting through rabbitmq to provide the end to end orchestration. If one part of the process takes time, the overall process will be delayed.


---

