<p>Apparently it's software issue, the software being deployed have blocking on specific headers, it only allows specific <code>Host:</code> header, that's why it's 403 instead of timeout or something else.</p>

CC BY-SA 4.0
2023-07-07T20:28:34.673Z
2023-07-07T20:28:34.673Z
Kokizzu
<p>I have this terraform-kubernetes script <a href="https://github.com/kokizzu/terraform1" rel="nofollow noreferrer">terraform1</a></p>
<p>so prometheus scraping list of pods, and hitting <code>/metrics</code> based on custom resource <code>Prometheus</code> and <code>PodMonitor</code> rules</p>
<p>it's working fine locally using minikube,</p>
<p>but in GKE, it doesn't work,</p>
<p>podmonitor works normally:</p>
<p><a href="https://i.sstatic.net/0zP0w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0zP0w.png" alt="service discovery works" /></a></p>
<p>but scraping failed (this part is green when on local minikube):</p>
<p><a href="https://i.sstatic.net/HatyM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HatyM.png" alt="scraping targets" /></a></p>
<p>normally when I open the website on using port forwarding (32009 to random localhost port) it works normally 200, no error:</p>
<p><a href="https://i.sstatic.net/3bruF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3bruF.png" alt="metrics exists using port-forwrding" /></a></p>
<p>does GKE disallow pod to pod communication? seems not since the error is 403 instead of something else, or I need to do something so Prometheus pod that lives on 10.63.1.4 can scrape other pods in the target list (10.63.1.252, 10.63.1.5, 10.63.3.5, 10.63.4.210, 10.63.1.133, etc)</p>

<kubernetes><google-kubernetes-engine>
Pod to pod communication GKE
Kokizzu
# Pod to pod communication GKE

(Asked by: username_0 on 2023-07-07)

I have this terraform-kubernetes script [terraform1](https://github.com/kokizzu/terraform1)

so prometheus scraping list of pods, and hitting `/metrics` based on custom resource `Prometheus` and `PodMonitor` rules

it's working fine locally using minikube,

but in GKE, it doesn't work,

podmonitor works normally:

[![service discovery works](https://i.sstatic.net/0zP0w.png)](https://i.sstatic.net/0zP0w.png)

but scraping failed (this part is green when on local minikube):

[![scraping targets](https://i.sstatic.net/HatyM.png)](https://i.sstatic.net/HatyM.png)

normally when I open the website on using port forwarding (32009 to random localhost port) it works normally 200, no error:

[![metrics exists using port-forwrding](https://i.sstatic.net/3bruF.png)](https://i.sstatic.net/3bruF.png)

does GKE disallow pod to pod communication? seems not since the error is 403 instead of something else, or I need to do something so Prometheus pod that lives on 10.63.1.4 can scrape other pods in the target list (10.63.1.252, 10.63.1.5, 10.63.3.5, 10.63.4.210, 10.63.1.133, etc)

**Tags:** kubernetes, google-kubernetes-engine


---

## 1 answers

### Answer by username_0 on 2023-07-07. Score: 0

Apparently it's software issue, the software being deployed have blocking on specific headers, it only allows specific `Host:` header, that's why it's 403 instead of timeout or something else.


---

