<p>Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints.</p>
<p>You also need to define the port named <code>prometheus</code> under the ingress-nginx service or deployment.</p>
<p>ServiceMonitors can be debugged this way:-</p>
<ol>
<li><p><strong>Check if the ServiceMonitor created is being read by the Prometheus</strong>:-
Look at the /targets URL. (There should be a target in 0/0
state at least and use the old Prometheus UI) If not, that means the ServiceMonitor itself is not being picked up by Prometheus.I suggest looking into the following configuration in
your kube-prometheus-stack configuration.</p>
<pre><code>    serviceMonitorSelectorNilUsesHelmValues: false  # is true by default
    serviceMonitorSelector: {} #labels which help Prom identify which servicemonitors to pick up
    serviceMonitorNamespaceSelector: {} # If this is empty then all servicemonitors in any namespace will be picked up.
</code></pre>
<p>The default ServiceMonitor has the Helm metadata attached to it
which is used by the Prometheus Operator to filter/choose the
ServiceMonitors to monitor. Setting
<code>serviceMonitorSelectorNilUsesHelmValues:false</code> will ignore any such
selection.</p>
</li>
<li><p><strong>If the ServiceMonitor is visible in targets but there are no targets.</strong>:-
In this case the issue lies between the ServiceMonitor and
the pods it is trying to scrape.Check if the ports you mentioned are
accessible and the pods fulfill the selectors mentioned (all <code>selectors.matchLabels</code> must match).</p>
</li>
</ol>
<p>My advice would be to start another dummy ServiceMonitor by following <a href="https://sysdig.com/blog/kubernetes-monitoring-prometheus-operator-part3/" rel="nofollow noreferrer">this</a> and then modifying the ServiceMonitor one step at a time till it starts monitoring the pods</p>
<p>Edit:-</p>
<p>Moving all servicemonitors to the kube-prometheus-stack namespace is not a good practice. Multiple Helm charts generate their own ServiceMonitor objects which are applied in their own namespace.Enabling the prometheus operator to monitor all namespaces for servicemonitors is a better choice.</p>
<p>For nginx side (ports and label would vary based on your implementations, update the same in the yaml below.):-</p>
<ol>
<li><p>Add the port in the nginx deployment</p>
<pre class="lang-yaml prettyprint-override"><code> - containerPort: 10254
   name: prometheus
   protocol: TCP
</code></pre>
</li>
<li><p>Create a service to map to the monitoring port.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  labels: [] # labels of your choice
  name: {} # correct name
spec:
  clusterIP: &quot;&quot;
  ports:
    - name: prometheus
      port: 9913
      targetPort: prometheus
  selector: &lt;all labels in your pods should match&gt;
  type: ClusterIP

</code></pre>
</li>
<li><p>Create the ServiceMonitor</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    annotations: [] # as required.
    name: ingress-nginx-monitoring
spec:
  endpoints:
  - interval: 30s
    path: /metrics
    port: prometheus
  namespaceSelector:
    matchNames:
    - &lt;namespace where nginx is deployed&gt;
  selector:
    matchLabels: [] # all labels in the given in the service above should match.
</code></pre>
</li>
</ol>
<pre><code>
</code></pre>

CC BY-SA 4.0
2021-06-07T15:17:10.237Z
I do see the ServiceMonitor being discovered within the Prometheus UI, but I see 0/8 active targets. I do have a port named `prometheus` going to `9113` on the `ingress-nginx-controller`. I'm not sure if that's the right port or if `ingress-nginx` even has the metrics exposed to begin with.
chrispytoes
CC BY-SA 4.0
2021-06-07T15:49:20.963Z
I've done some more digging, check my edits.
chrispytoes
CC BY-SA 4.0
2021-06-07T17:59:28.180Z
If I move the `ServiceMonitor` out of the `kube-prometheus-stack` namespace, I can no longer see it in the Prometheus UI. I'd rather keep all my resources in a namespace if possible anyway.
chrispytoes
CC BY-SA 4.0
2021-06-07T18:07:32.550Z
That should be okay. The rest of the steps would remain the same.
rohatgisanat
CC BY-SA 4.0
2021-06-07T18:20:17.757Z
I don't see the `ServiceMonitor` showing up again in Prometheus when I move it back to the `kube-prometheus-stack` namespace, it's totally gone again. I created a `ClusterIP` service to expose the prometheus port on the controller deployment but that doesn't seem to be finding it either, and I tested that it works by port-fowarding the service and I can confirm that I do see the metrics displayed at `/metrics`.
chrispytoes
CC BY-SA 4.0
2021-06-07T18:30:30.233Z
What changes did you make to the ServiceMonitor? If the service works correctly and you can see the metrics, then the problem lies in the service monitor. Can you confirm if the matchLabels are correct? They should be equal to the labels present in the Service.
rohatgisanat
CC BY-SA 4.0
2021-06-07T18:46:51.180Z
Okay, so connecting via the service WAS working a short while ago, now port-forwarding to it is just timing out after changing the namespace back. I can port-forward directly to the controller deployment on port 10254 though and that does work. My labels all look correct to me.
chrispytoes
CC BY-SA 4.0
2021-06-07T19:06:38.093Z
Can you share the results of ‘kubectl get ep’?
rohatgisanat
CC BY-SA 4.0
2021-06-07T19:07:51.833Z
My bad, I've got the service working now, but I'm just back to where I was before. I can see the monitor in Prometheus but 0/8 active targets still.
chrispytoes
CC BY-SA 4.0
2021-06-07T19:21:35.333Z
If the serviceMonitor is the same as in the question and the matchLabels are all equal to the service and the matchNames has the name space then you can try adding the path: /metrics under the port. If that does not work you can create a new serviceMonitor from the template i provided. Or paste the one you are using along with prometheus screenshot.
rohatgisanat
CC BY-SA 4.0
2021-06-07T19:35:57.163Z
I edited my post with more details
chrispytoes
CC BY-SA 4.0
2021-06-07T20:18:18.037Z
Both the configs work on my end. Only thing I had to change was the selectors in the Service. Can you get the output of ‘kubectl get ep’ also can you check what it says under the /target UI? It is even monitoring the nginx pods?
rohatgisanat
CC BY-SA 4.0
2021-06-07T20:25:06.310Z
That command gives me one endpoint with the name `kubernetes`, and an IP with port 443. In the /target UI, nginx is not in the list.
chrispytoes
CC BY-SA 4.0
2021-06-07T20:34:21.703Z
kubectl get ep -n ingress-nginx should give you the name of the service along with Endpoint attached to it. Under the /targets UI the ingress-nginx-metrics should have 0/8. Under that you should see URL and other details related to scraping
rohatgisanat
CC BY-SA 4.0
2021-06-07T20:34:28.683Z
Okay, I didn't realize I needed to specify the namespace with the command. When using the correct namespace, I do see the `ingress-nginx-metrics` service, but it has no IP assigned to it. I only see the 0/8 targets on the /service-discovery page. nginx doesn't show at all on /targets.
chrispytoes
CC BY-SA 4.0
2021-06-07T20:37:53.530Z
Yes, If you mean under the endpoint column, that means your service in not connected to any pods. Check if the selector field is correct. Selectors should match the labels on the pods.
rohatgisanat
CC BY-SA 4.0
2021-06-07T20:43:27.703Z
I checked it over and over again, It's definitely right. I copied that YAML in my post edit directly from the dashboard. The metrics service has the `app: ingress-nginx-controller` selector, and the nginx controller deployment and service both have the label `app: ingress-nginx-controller`.
chrispytoes
CC BY-SA 4.0
2021-06-07T20:49:20.213Z
‘’’app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
 app.kubernetes.io/name: ingress-nginx’’’
Try adding these three to the service.selector
rohatgisanat
CC BY-SA 4.0
2021-06-07T20:53:13.260Z
I figured it out actually. I only had the label set on the Deployment, not on the Pod template. It's working now!
chrispytoes
CC BY-SA 4.0
2021-06-05T10:52:12.890Z
2021-06-07T17:06:05.140Z
2021-06-07T17:06:05.140Z
rohatgisanat
<p>The service monitor seems right, only thing is that the service monitor should be in the namespace of the Prometheus stack.</p>
<p>Try the below service monitor:</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx
</code></pre>
<p>Also when you do, try port-forwarding the Prometheus</p>
<pre><code> kubectl port-forward svc/&lt;SERVICE_NAME_OF_PROMETHEUS&gt; -n kube-prometheus-stack 9090:9090
</code></pre>
<p>and check in <code>Targets</code> and <code>Service Discovery</code> options in Prometheus.
<a href="https://i.sstatic.net/6vO8v.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6vO8v.png" alt="enter image description here" /></a>
You can see the service monitor in <code>Service Discovery</code> and exposed metrics in <code>Targets</code>.</p>

CC BY-SA 4.0
2021-06-07T14:24:52.257Z
Okay so I moved it to the `kube-prometheus-stack` namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the `/metrics` path in the endpoint? I've tried with and without and it doesn't seem to change anything.
chrispytoes
CC BY-SA 4.0
2021-06-07T15:49:24.417Z
I've done some more digging, check my edits.
chrispytoes
CC BY-SA 4.0
2021-06-05T18:33:34.960Z
2021-06-05T18:33:34.960Z
rksachin
<p>A few weeks ago, I installed <code>ingress-nginx</code> in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my <code>ingress-nginx-controller</code> is all set up and working, but I am unable to change settings using Helm to create the <code>ServiceMonitor</code> and scrape metrics from it.</p>
<p>I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor <code>ingress-nginx</code> without using Helm. I understand I need to make a <code>ServiceMonitor</code> resource, but I can't figure out how to make one specifically for <code>ingress-nginx</code> because every solution I find online pretty much boils down to: &quot;oh just use Helm bro&quot;.</p>
<p>I attempted to create my own <code>ServiceMonitor</code> resource like so:</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: ingress-nginx
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-controller
</code></pre>
<p>It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my <code>ServiceMonitor</code> need to be in the <code>kube-prometheus-stack</code> namespace, the namespace of the service being monitored, or does it not matter?</p>
<ul>
<li>My <code>ingress-nginx-controller</code> deployment is in the <code>ingress-nginx</code> namespace.</li>
<li><code>kube-prometheus-stack</code> is installed in the <code>kube-prometheus-stack</code> namespace.</li>
</ul>
<p>Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?</p>
<p>Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the <code>/metrics</code> path in the endpoint? I've tried with and without and it doesn't seem to change anything.</p>
<p>Edit: I created a <code>ClusterIP</code> service pointing to the <code>prometheus</code> port on my <code>ingress-nginx-controller</code> and I am able to port forward that and see the <code>/metrics</code> route which shows the Prometheus logs. Pointing my <code>ServiceMonitor</code> directly at this new <code>ClusterIP</code> service doesn't seem to be doing anything either.</p>
<p>Do I need to point the <code>ServiceMonitor</code> directly at the <code>ingress-nginx-controller</code> deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see <code>ingress-nginx</code> in the Prometheus Service Discovery screen, but 0/8 active targets.</p>
<p>Edit:
Here's what I have in my resources now:</p>
<p>NGINX Controller (relevant items only):</p>
<pre><code>kind: Deployment
apiVersion: apps/v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-controller
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
  annotations:
    prometheus.io/port: '10254'
    prometheus.io/scrape: 'true'
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
        - name: https
          containerPort: 443
          protocol: TCP
        - name: webhook
          containerPort: 8443
          protocol: TCP
        - name: prometheus
          containerPort: 10254
          protocol: TCP
</code></pre>
<p>NGINX metrics service:</p>
<pre><code>kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-metrics
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-metrics
spec:
  ports:
    - name: prometheus
      protocol: TCP
      port: 10254
      targetPort: prometheus
  selector:
    app: ingress-nginx-controller
  type: ClusterIP
</code></pre>
<p>NGINX Service Monitor:</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
  labels:
    app: ingress-nginx-metrics
    release: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      path: /metrics
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-metrics
</code></pre>
<p>I cannot port-forward to the metrics service I created, the connection times out.</p>
<p>This is what I'm seeing in Prometheus:</p>
<p><a href="https://i.sstatic.net/KEzeJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KEzeJ.png" alt="enter image description here" /></a></p>

<kubernetes><prometheus><ingress>
Enable ingress-nginx Prometheus monitoring without Helm
chrispytoes
# Enable ingress-nginx Prometheus monitoring without Helm

(Asked by: username_0 on 2021-06-04)

A few weeks ago, I installed ingress-nginx in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my ingress-nginx-controller is all set up and working, but I am unable to change settings using Helm to create the ServiceMonitor and scrape metrics from it.
I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor ingress-nginx without using Helm. I understand I need to make a ServiceMonitor resource, but I can't figure out how to make one specifically for ingress-nginx because every solution I find online pretty much boils down to: "oh just use Helm bro".
I attempted to create my own ServiceMonitor resource like so:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: ingress-nginx
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-controller

It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my ServiceMonitor need to be in the kube-prometheus-stack namespace, the namespace of the service being monitored, or does it not matter?

My ingress-nginx-controller deployment is in the ingress-nginx namespace.
kube-prometheus-stack is installed in the kube-prometheus-stack namespace.

Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?
Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the /metrics path in the endpoint? I've tried with and without and it doesn't seem to change anything.
Edit: I created a ClusterIP service pointing to the prometheus port on my ingress-nginx-controller and I am able to port forward that and see the /metrics route which shows the Prometheus logs. Pointing my ServiceMonitor directly at this new ClusterIP service doesn't seem to be doing anything either.
Do I need to point the ServiceMonitor directly at the ingress-nginx-controller deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see ingress-nginx in the Prometheus Service Discovery screen, but 0/8 active targets.
Edit:
Here's what I have in my resources now:
NGINX Controller (relevant items only):
kind: Deployment
apiVersion: apps/v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-controller
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
  annotations:
    prometheus.io/port: '10254'
    prometheus.io/scrape: 'true'
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
        - name: https
          containerPort: 443
          protocol: TCP
        - name: webhook
          containerPort: 8443
          protocol: TCP
        - name: prometheus
          containerPort: 10254
          protocol: TCP

NGINX metrics service:
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-metrics
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-metrics
spec:
  ports:
    - name: prometheus
      protocol: TCP
      port: 10254
      targetPort: prometheus
  selector:
    app: ingress-nginx-controller
  type: ClusterIP

NGINX Service Monitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
  labels:
    app: ingress-nginx-metrics
    release: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      path: /metrics
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-metrics

I cannot port-forward to the metrics service I created, the connection times out.
This is what I'm seeing in Prometheus:

**Tags:** kubernetes, prometheus, ingress


---

## 2 answers

### Answer by username_1 on 2021-06-05. Score: 2 | Accepted answer

Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints.
You also need to define the port named prometheus under the ingress-nginx service or deployment.
ServiceMonitors can be debugged this way:-

Check if the ServiceMonitor created is being read by the Prometheus:-
Look at the /targets URL. (There should be a target in 0/0
state at least and use the old Prometheus UI) If not, that means the ServiceMonitor itself is not being picked up by Prometheus.I suggest looking into the following configuration in
your kube-prometheus-stack configuration.
    serviceMonitorSelectorNilUsesHelmValues: false  # is true by default
    serviceMonitorSelector: {} #labels which help Prom identify which servicemonitors to pick up
    serviceMonitorNamespaceSelector: {} # If this is empty then all servicemonitors in any namespace will be picked up.

The default ServiceMonitor has the Helm metadata attached to it
which is used by the Prometheus Operator to filter/choose the
ServiceMonitors to monitor. Setting
serviceMonitorSelectorNilUsesHelmValues:false will ignore any such
selection.

If the ServiceMonitor is visible in targets but there are no targets.:-
In this case the issue lies between the ServiceMonitor and
the pods it is trying to scrape.Check if the ports you mentioned are
accessible and the pods fulfill the selectors mentioned (all selectors.matchLabels must match).


My advice would be to start another dummy ServiceMonitor by following this and then modifying the ServiceMonitor one step at a time till it starts monitoring the pods
Edit:-
Moving all servicemonitors to the kube-prometheus-stack namespace is not a good practice. Multiple Helm charts generate their own ServiceMonitor objects which are applied in their own namespace.Enabling the prometheus operator to monitor all namespaces for servicemonitors is a better choice.
For nginx side (ports and label would vary based on your implementations, update the same in the yaml below.):-

Add the port in the nginx deployment
 - containerPort: 10254
   name: prometheus
   protocol: TCP


Create a service to map to the monitoring port.
apiVersion: v1
kind: Service
metadata:
  labels: [] # labels of your choice
  name: {} # correct name
spec:
  clusterIP: ""
  ports:
    - name: prometheus
      port: 9913
      targetPort: prometheus
  selector: 
  type: ClusterIP



Create the ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    annotations: [] # as required.
    name: ingress-nginx-monitoring
spec:
  endpoints:
  - interval: 30s
    path: /metrics
    port: prometheus
  namespaceSelector:
    matchNames:
    - 
  selector:
    matchLabels: [] # all labels in the given in the service above should match.

> **Comments:**
> **username_0**: I do see the ServiceMonitor being discovered within the Prometheus UI, but I see 0/8 active targets. I do have a port named `prometheus` going to `9113` on the `ingress-nginx-controller`. I'm not sure if that's the right port or if `ingress-nginx` even has the metrics exposed to begin with.  
> **username_0**: I've done some more digging, check my edits.  
> **username_0**: If I move the `ServiceMonitor` out of the `kube-prometheus-stack` namespace, I can no longer see it in the Prometheus UI. I'd rather keep all my resources in a namespace if possible anyway.  
> **username_1**: That should be okay. The rest of the steps would remain the same.  
> **username_0**: I don't see the `ServiceMonitor` showing up again in Prometheus when I move it back to the `kube-prometheus-stack` namespace, it's totally gone again. I created a `ClusterIP` service to expose the prometheus port on the controller deployment but that doesn't seem to be finding it either, and I tested that it works by port-fowarding the service and I can confirm that I do see the metrics displayed at `/metrics`.  
> **username_1**: What changes did you make to the ServiceMonitor? If the service works correctly and you can see the metrics, then the problem lies in the service monitor. Can you confirm if the matchLabels are correct? They should be equal to the labels present in the Service.  
> **username_0**: Okay, so connecting via the service WAS working a short while ago, now port-forwarding to it is just timing out after changing the namespace back. I can port-forward directly to the controller deployment on port 10254 though and that does work. My labels all look correct to me.  
> **username_1**: Can you share the results of ‘kubectl get ep’?  
> **username_0**: My bad, I've got the service working now, but I'm just back to where I was before. I can see the monitor in Prometheus but 0/8 active targets still.  
> **username_1**: If the serviceMonitor is the same as in the question and the matchLabels are all equal to the service and the matchNames has the name space then you can try adding the path: /metrics under the port. If that does not work you can create a new serviceMonitor from the template i provided. Or paste the one you are using along with prometheus screenshot.  
> **username_0**: I edited my post with more details  
> **username_1**: Both the configs work on my end. Only thing I had to change was the selectors in the Service. Can you get the output of ‘kubectl get ep’ also can you check what it says under the /target UI? It is even monitoring the nginx pods?  
> **username_0**: That command gives me one endpoint with the name `kubernetes`, and an IP with port 443. In the /target UI, nginx is not in the list.  
> **username_1**: kubectl get ep -n ingress-nginx should give you the name of the service along with Endpoint attached to it. Under the /targets UI the ingress-nginx-metrics should have 0/8. Under that you should see URL and other details related to scraping  
> **username_0**: Okay, I didn't realize I needed to specify the namespace with the command. When using the correct namespace, I do see the `ingress-nginx-metrics` service, but it has no IP assigned to it. I only see the 0/8 targets on the /service-discovery page. nginx doesn't show at all on /targets.  
> **username_1**: Yes, If you mean under the endpoint column, that means your service in not connected to any pods. Check if the selector field is correct. Selectors should match the labels on the pods.  
> **username_0**: I checked it over and over again, It's definitely right. I copied that YAML in my post edit directly from the dashboard. The metrics service has the `app: ingress-nginx-controller` selector, and the nginx controller deployment and service both have the label `app: ingress-nginx-controller`.  
> **username_1**: ‘’’app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx  app.kubernetes.io/name: ingress-nginx’’’ Try adding these three to the service.selector  
> **username_0**: I figured it out actually. I only had the label set on the Deployment, not on the Pod template. It's working now!  

---

### Answer by username_2 on 2021-06-05. Score: 0

The service monitor seems right, only thing is that the service monitor should be in the namespace of the Prometheus stack.
Try the below service monitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx

Also when you do, try port-forwarding the Prometheus
 kubectl port-forward svc/ -n kube-prometheus-stack 9090:9090

and check in Targets and Service Discovery options in Prometheus.

You can see the service monitor in Service Discovery and exposed metrics in Targets.

> **Comments:**
> **username_0**: Okay so I moved it to the `kube-prometheus-stack` namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the `/metrics` path in the endpoint? I've tried with and without and it doesn't seem to change anything.  
> **username_0**: I've done some more digging, check my edits.  

---

