<p>I was getting this because I had a pod in <code>CrashLoopBackOff</code>. In my specific case because <code>yaml</code> file had changed (through the Grafana interface) and Grafana could no longer start.</p>
<blockquote>
<p>Error: ✗ Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default</p>
</blockquote>
<p><strong>So I couldn't port forward <code>:3000</code> because nothing was running on it.</strong></p>

CC BY-SA 4.0
2024-09-23T04:19:52.707Z
2024-09-25T02:04:22.060Z
2024-09-25T02:04:22.060Z
Evan Carroll
<p>I've been doing some work with k3s, and everything was working fine. Now I'm getting the following error when I restart and I try to run the port-forward command I had previously run to get to the service I want,</p>
<blockquote>
<p>E0921 19:32:10.085091 3083215 portforward.go:400] an error occurred forwarding 3000 -&gt; 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace &quot;/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5&quot;: failed to connect to localhost:3000 inside namespace &quot;ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df&quot;, IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused</p>
</blockquote>
<p>I can see my service running,</p>
<pre><code>kubectl get svc -n monitoring prometheus-grafana
NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
prometheus-grafana   ClusterIP   10.43.34.93   &lt;none&gt;        80/TCP    45h
</code></pre>
<p>And when I run this command</p>
<pre class="lang-shell prettyprint-override"><code>kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
</code></pre>
<p>I see</p>
<pre><code>Forwarding from 127.0.0.1:3000 -&gt; 3000
Forwarding from [::1]:3000 -&gt; 3000
Handling connection for 3000
E0921 19:37:03.012976 3085961 portforward.go:400] an error occurred forwarding 3000 -&gt; 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace &quot;/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5&quot;: failed to connect to localhost:3000 inside namespace &quot;ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df&quot;, IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused
</code></pre>

<networking><k3s>
k3s: Error E0921 failed to connect inside namespace ... connect: connection refused
Evan Carroll
# k3s: Error E0921 failed to connect inside namespace ... connect: connection refused

(Asked by: username_0 on 2024-09-22)

I've been doing some work with k3s, and everything was working fine. Now I'm getting the following error when I restart and I try to run the port-forward command I had previously run to get to the service I want,

E0921 19:32:10.085091 3083215 portforward.go:400] an error occurred forwarding 3000 -> 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace "/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5": failed to connect to localhost:3000 inside namespace "ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df", IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused

I can see my service running,
kubectl get svc -n monitoring prometheus-grafana
NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
prometheus-grafana   ClusterIP   10.43.34.93           80/TCP    45h

And when I run this command
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

I see
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000
Handling connection for 3000
E0921 19:37:03.012976 3085961 portforward.go:400] an error occurred forwarding 3000 -> 3000: error forwarding port 3000 to pod ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df, uid : failed to execute portforward in network namespace "/var/run/netns/cni-00a42412-9f1f-6713-07bd-0872bbdf6de5": failed to connect to localhost:3000 inside namespace "ecc54372f959863a00a568fbfc9e0fb9f7a671a01ad26df7ce803a446e4391df", IPv4: dial tcp4 127.0.0.1:3000: connect: connection refused IPv6 dial tcp6 [::1]:3000: connect: connection refused

**Tags:** networking, k3s


---

## 1 answers

### Answer by username_0 on 2024-09-23. Score: 1 | Accepted answer

I was getting this because I had a pod in `CrashLoopBackOff`. In my specific case because `yaml` file had changed (through the Grafana interface) and Grafana could no longer start.

> Error: ✗ Datasource provisioning error: datasource.yaml config is invalid. Only one datasource per organization can be marked as default

**So I couldn't port forward `:3000` because nothing was running on it.**


---

