<p>For anyone running into the same problem:
cert-manager's issue tracker has an <a href="https://github.com/cert-manager/cert-manager/issues/466" rel="nofollow noreferrer">excellent discussion about the problem</a> dating back to 2018.</p>
<p>The solution for me was to install compumike's <a href="https://github.com/compumike/hairpin-proxy" rel="nofollow noreferrer">hairpin-proxy</a> to enable PROXY protocol support for internal-to-LoadBalancer traffic.</p>

CC BY-SA 4.0
2024-04-02T19:09:14.357Z
2024-04-02T19:09:14.357Z
cvk
<p>I am currently facing an issue with cert-manager while trying to obtain a Let's Encrypt certificate for my Kubernetes cluster running on Scaleway Kapsule.</p>
<p>I have successfully installed cert-manager from the helm chart. However, I am now encountering a problem with the challenges for domain validation.</p>
<p>I created an issuer:</p>
<pre><code>apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my@email.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - selector: {}
      http01:
        ingress:
          ingressClassName: nginx

</code></pre>
<p>and annotated the ingress:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
    nginx.ingress.kubernetes.io/use-regex: &quot;true&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /api/(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: backend
            port:
              number: 8080
      - path: /?(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: frontend
            port:
              number: 3000
  tls:
  - hosts:
    - example.com
    secretName: frontend-ingress-cert
</code></pre>
<p>This creates the cm-acme-http-solver:</p>
<pre><code>NAME                        CLASS   HOSTS                         ADDRESS          PORTS     AGE
cm-acme-http-solver-t8pb9   nginx   example.com                   51.158.131.xxx   80        11m
</code></pre>
<p>This seems to work fine, Certificate, -Request and Order are created, but the challenge is stuck performing the self check:</p>
<pre><code>Name:         frontend-ingress-cert-1-1328491808-3444115809
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2024-04-02T15:07:31Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  frontend-ingress-cert-1-1328491808
    UID:                   e85e8c53-5965-4be4-87fa-50359217539f
  Resource Version:        10398552697
  UID:                     73e25122-6340-406f-aa68-632e92adec4b
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/123456789
  Dns Name:           example.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-prod
  Key:      &lt;redacted&gt;&gt;.NaUj9lmjGqlcRba77ZLjqqNfT_qWVn2WdlTDo0K4NMI
  Solver:
    http01:
      Ingress:
        Ingress Class Name:  nginx
    Selector:
  Token:     &lt;redacted&gt;&gt;
  Type:      HTTP-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/123456789/ZaFzbg
  Wildcard:  false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://example.com/.well-known/acme-challenge/&lt;redacted&gt;&gt;': Get &quot;http://example.com/.well-known/acme-challenge/&lt;redacted&gt;&gt;&quot;: EOF
  State:       pending
Events:
  Type    Reason     Age    From                     Message
  ----    ------     ----   ----                     -------
  Normal  Started    5m40s  cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  5m39s  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism
</code></pre>
<p>Accessing the challenge url from outside the cluster works fine, but from within the cluster, the request fails:</p>
<pre><code>❯ kubectl exec -it frontend-775d68f5-7rq7x -- sh
/app $ wget http://example.com/.well-known/acme-challenge/&lt;redacted&gt;&gt;
Connecting to mydomain.com (51.158.131.xxx:80)
wget: error getting response: Resource temporarily unavailable
</code></pre>
<p>Nginx logs this:</p>
<pre><code>2024/04/02 15:16:03 [error] 3555#3555: *60767 broken header: &quot;GET /.well-known/acme-challenge/&lt;redacted&gt;&gt; HTTP/1.1&quot; while reading PROXY protocol, client: 100.64.0.110, server: 0.0.0.0:80
</code></pre>

<kubernetes><cert-manager><tls>
Cert-manager challenge failing self check
cvk
# Cert-manager challenge failing self check

(Asked by: username_0 on 2024-04-02)

I am currently facing an issue with cert-manager while trying to obtain a Let's Encrypt certificate for my Kubernetes cluster running on Scaleway Kapsule.
I have successfully installed cert-manager from the helm chart. However, I am now encountering a problem with the challenges for domain validation.
I created an issuer:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my@email.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - selector: {}
      http01:
        ingress:
          ingressClassName: nginx


and annotated the ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /api/(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: backend
            port:
              number: 8080
      - path: /?(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: frontend
            port:
              number: 3000
  tls:
  - hosts:
    - example.com
    secretName: frontend-ingress-cert

This creates the cm-acme-http-solver:
NAME                        CLASS   HOSTS                         ADDRESS          PORTS     AGE
cm-acme-http-solver-t8pb9   nginx   example.com                   51.158.131.xxx   80        11m

This seems to work fine, Certificate, -Request and Order are created, but the challenge is stuck performing the self check:
Name:         frontend-ingress-cert-1-1328491808-3444115809
Namespace:    default
Labels:       
Annotations:  
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2024-04-02T15:07:31Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  frontend-ingress-cert-1-1328491808
    UID:                   e85e8c53-5965-4be4-87fa-50359217539f
  Resource Version:        10398552697
  UID:                     73e25122-6340-406f-aa68-632e92adec4b
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/123456789
  Dns Name:           example.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-prod
  Key:      >.NaUj9lmjGqlcRba77ZLjqqNfT_qWVn2WdlTDo0K4NMI
  Solver:
    http01:
      Ingress:
        Ingress Class Name:  nginx
    Selector:
  Token:     >
  Type:      HTTP-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/123456789/ZaFzbg
  Wildcard:  false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://example.com/.well-known/acme-challenge/>': Get "http://example.com/.well-known/acme-challenge/>": EOF
  State:       pending
Events:
  Type    Reason     Age    From                     Message
  ----    ------     ----   ----                     -------
  Normal  Started    5m40s  cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  5m39s  cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

Accessing the challenge url from outside the cluster works fine, but from within the cluster, the request fails:
❯ kubectl exec -it frontend-775d68f5-7rq7x -- sh
/app $ wget http://example.com/.well-known/acme-challenge/>
Connecting to mydomain.com (51.158.131.xxx:80)
wget: error getting response: Resource temporarily unavailable

Nginx logs this:
2024/04/02 15:16:03 [error] 3555#3555: *60767 broken header: "GET /.well-known/acme-challenge/> HTTP/1.1" while reading PROXY protocol, client: 100.64.0.110, server: 0.0.0.0:80

**Tags:** kubernetes, cert-manager, tls


---

## 1 answers

### Answer by username_0 on 2024-04-02. Score: 0

For anyone running into the same problem:
cert-manager's issue tracker has an [excellent discussion about the problem](https://github.com/cert-manager/cert-manager/issues/466) dating back to 2018.

The solution for me was to install compumike's [hairpin-proxy](https://github.com/compumike/hairpin-proxy) to enable PROXY protocol support for internal-to-LoadBalancer traffic.


---

