<h1>6/30/2024</h1>
<p>I found what I posted on 6/27/2024 is misleading. Even it repaired the issue in the question, but only on the surface. The Certificate still failed to issue.</p>
<p>It took me some time to make everything including cert-manager, ExternalDNS, Traefik work well.</p>
<p>Also, it turns out <code>acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;</code> annotation is not necessary to fix the issue in the question.</p>
<p>Note for the solution in my case, I am using DNS01 challenge.</p>
<p>I also published all my deployment codes <a href="https://github.com/hongbo-miao/hongbomiao.com/issues/17463#issuecomment-2180242019" rel="nofollow noreferrer">here</a>. Hopefully it can give a bigger picture how it get deployed.</p>
<h2>cert-manager AWS IAM role <code>CertManagerRole</code></h2>
<h3>Trust relationships</h3>
<pre class="lang-json prettyprint-override"><code>{
    &quot;Version&quot;: &quot;2012-10-17&quot;,
    &quot;Statement&quot;: [
        {
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Principal&quot;: {
                &quot;Federated&quot;: &quot;arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
            },
            &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,
            &quot;Condition&quot;: {
                &quot;StringEquals&quot;: {
                    &quot;oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud&quot;: &quot;sts.amazonaws.com&quot;,
                    &quot;oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub&quot;: &quot;system:serviceaccount:production-hm-cert-manager:hm-cert-manager&quot;
                }
            }
        }
    ]
}
</code></pre>
<h3>Policy</h3>
<pre class="lang-json prettyprint-override"><code>{
    &quot;Version&quot;: &quot;2012-10-17&quot;,
    &quot;Statement&quot;: [
        {
            &quot;Action&quot;: [
                &quot;route53:GetChange&quot;
            ],
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Resource&quot;: &quot;arn:aws:route53:::change/*&quot;
        },
        {
            &quot;Action&quot;: [
                &quot;route53:ListResourceRecordSets&quot;
            ],
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Resource&quot;: &quot;arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx&quot;
        },
        {
            &quot;Action&quot;: [
                &quot;route53:ChangeResourceRecordSets&quot;
            ],
            &quot;Condition&quot;: {
                &quot;ForAllValues:StringEquals&quot;: {
                    &quot;route53:ChangeResourceRecordSetsNormalizedRecordNames&quot;: [
                        &quot;_acme-challenge.*&quot;
                    ],
                    &quot;route53:ChangeResourceRecordSetsRecordTypes&quot;: [
                        &quot;TXT&quot;
                    ]
                }
            },
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Resource&quot;: &quot;arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx&quot;
        },
        {
            &quot;Action&quot;: [
                &quot;route53:ListHostedZonesByName&quot;
            ],
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Resource&quot;: &quot;*&quot;
        }
    ]
}
</code></pre>
<h2>ExternalDNS AWS IAM role <code>ExternalDNSRole</code></h2>
<h3>Trust relationships</h3>
<pre class="lang-json prettyprint-override"><code>{
    &quot;Version&quot;: &quot;2012-10-17&quot;,
    &quot;Statement&quot;: [
        {
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Principal&quot;: {
                &quot;Federated&quot;: &quot;arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
            },
            &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,
            &quot;Condition&quot;: {
                &quot;StringEquals&quot;: {
                    &quot;oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub&quot;: &quot;system:serviceaccount:production-hm-external-dns:hm-external-dns&quot;,
                    &quot;oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud&quot;: &quot;sts.amazonaws.com&quot;
                }
            }
        }
    ]
}
</code></pre>
<h3>Policy</h3>
<pre class="lang-json prettyprint-override"><code>{
  &quot;Version&quot;: &quot;2012-10-17&quot;,
  &quot;Statement&quot;: [
    {
      &quot;Action&quot;: [
        &quot;route53:ChangeResourceRecordSets&quot;
      ],
      &quot;Effect&quot;: &quot;Allow&quot;,
      &quot;Resource&quot;: [
        &quot;arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx&quot;
      ]
    },
    {
      &quot;Action&quot;: [
        &quot;route53:ListHostedZones&quot;,
        &quot;route53:ListResourceRecordSets&quot;,
        &quot;route53:ListTagsForResource&quot;
      ],
      &quot;Effect&quot;: &quot;Allow&quot;,
      &quot;Resource&quot;: [
        &quot;*&quot;
      ]
    }
  ]
}
</code></pre>
<h2>cert-manager Helm chart custom values.yaml</h2>
<pre><code>crds:
  enabled: true
  keep: true
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/CertManagerRole
securityContext:
  fsGroup: 1001
</code></pre>
<h2>ExternalDNS Helm chart custom values.yaml</h2>
<pre class="lang-yaml prettyprint-override"><code>provider: aws
aws:
  region: us-west-2
  zoneType: public
  txtOwnerId: Z1XXXXXXXXXXXXXXXXXXX
  domainFilters:
    - internal.example.com
  registry: txt
policy: sync
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/ExternalDNSRole
</code></pre>
<h2>Traefik Helm chart custom values.yaml</h2>
<pre class="lang-yaml prettyprint-override"><code>providers:
  kubernetesIngress:
    publishedService:
      enabled: true
service:
  enabled: true
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-internal: &quot;true&quot;
</code></pre>
<h2>ClusterIssuer</h2>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: production-lets-encrypt-cluster-issuer
  namespace: production-hm-cert-manager
spec:
  acme:
    email: me@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: production-lets-encrypt-cluster-issuer-account-secret
    solvers:
      - selector:
          dnsZones:
            - internal.example.com
        dns01:
          route53:
            region: us-west-2
</code></pre>
<h2>Ingress</h2>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hm-airbyte-ingress
  namespace: production-hm-airbyte
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: &quot;true&quot;
    cert-manager.io/cluster-issuer: production-lets-encrypt-cluster-issuer
spec:
  rules:
    - host: hm-airbyte.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hm-airbyte-airbyte-webapp-svc
                port:
                  number: 80
  tls:
    - hosts:
        - hm-airbyte.internal.example.com
      secretName: production-hm-airbyte-certificate
</code></pre>
<h1>6/27/2024 (Misleading Answer)</h1>
<p>After a lot of experiments, I found if I add <code>acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;</code> annotation, it helps resolve the issue.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hm-airbyte-ingress
  namespace: production-hm-airbyte
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: &quot;true&quot;
    cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
    acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;  # &lt;- Added this
  labels:
    app.kubernetes.io/name: hm-airbyte-ingress
    app.kubernetes.io/part-of: production-hm-airbyte
spec:
  rules:
    - host: hm-airbyte.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hm-airbyte-airbyte-webapp-svc
                port:
                  number: 80
  tls:
    - hosts:
        - hm-airbyte.internal.example.com
      secretName: hm-airbyte-ingress-tls
</code></pre>
<p>So now here is what happens after I apply this Ingress file</p>
<ol>
<li>A temporary <code>Secret: hm-airbyte-ingress-tls-7w9z7</code> with random suffix will be created</li>
<li>After about half minute, a <code>Secret: hm-airbyte-ingress-tls</code> without suffix will be created</li>
<li>The one will suffix will be automatically deleted.</li>
</ol>
<p>Without this annotation, based on my experiment, it will be stuck at step 1 forever.</p>
<p>This is the <code>Secret: hm-airbyte-ingress-tls</code> without suffix, you can see the difference between the temporary secret I posted in the question.</p>
<p><a href="https://i.sstatic.net/osEg2tA4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/osEg2tA4.png" alt="enter image description here" /></a></p>
<p><a href="https://cert-manager.io/docs/usage/ingress/#supported-annotations" rel="nofollow noreferrer">Here</a> is the explanation of this annotation:</p>
<blockquote>
<p>acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;: this controls whether the ingress is modified 'in-place', or a new one is created specifically for the HTTP01 challenge. If present, and set to &quot;true&quot;, the existing ingress will be modified. Any other value, or the absence of the annotation assumes &quot;false&quot;. This annotation will also add the annotation &quot;cert-manager.io/issue-temporary-certificate&quot;: &quot;true&quot; onto created certificates which will cause a temporary certificate to be set on the resulting Secret until the final signed certificate has been returned. This is useful for keeping compatibility with the ingress-gce component.</p>
</blockquote>
<p>P.S. I cannot explain why this annotation helps resolve the issue. Because I actually expect if with this annotation, the temporary secret will not be created at all. Hope someone can further explain in future, thanks!</p>

CC BY-SA 4.0
2024-06-30T12:51:22.760Z
2024-07-01T04:40:17.777Z
2024-07-01T04:40:17.777Z
Hongbo Miao
<p>I am using &quot;Traefik&quot;, &quot;cert-mangaer&quot;, and &quot;Let's Encrypt&quot;.</p>
<p>Here is my ClusterIssuer file</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: production-lets-encrypt-issuer
  namespace: production-hm-cert-manager
spec:
  acme:
    email: me@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: production-lets-encrypt-issuer-account-secret
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik
</code></pre>
<p>Here is my Ingress file</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hm-airbyte-ingress
  namespace: production-hm-airbyte
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: &quot;true&quot;
    cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
  labels:
    app.kubernetes.io/name: hm-airbyte-ingress
    app.kubernetes.io/part-of: production-hm-airbyte
spec:
  rules:
    - host: hm-airbyte.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hm-airbyte-airbyte-webapp-svc
                port:
                  number: 80
  tls:
    - hosts:
        - hm-airbyte.example.com
      secretName: hm-airbyte-ingress-tls

</code></pre>
<p>My Traefik pod shows error</p>
<blockquote>
<p>&quot;2024-06-27T06:35:45Z ERR Error configuring TLS error=&quot;secret production-hm-airbyte/hm-airbyte-ingress-tls does not exist&quot; ingress=hm-airbyte-ingress namespace=production-hm-airbyte providerName=kubernetes&quot;</p>
</blockquote>
<p>But I found I actually have a secret <code>Secret: hm-airbyte-ingress-tls-7w9z7</code> inside namespace <code>production-hm-airbyte</code>. Note it has a random suffix.</p>
<p><a href="https://i.sstatic.net/gwoZhxcI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gwoZhxcI.png" alt="enter image description here" /></a></p>
<p>I searched online and found</p>
<ul>
<li><a href="https://github.com/cert-manager/cert-manager/issues/3300" rel="nofollow noreferrer">https://github.com/cert-manager/cert-manager/issues/3300</a></li>
<li><a href="https://github.com/cert-manager/cert-manager/issues/3283" rel="nofollow noreferrer">https://github.com/cert-manager/cert-manager/issues/3283</a></li>
</ul>
<p>This issue existed for four years, but unfortunately, no solution has been posted.</p>

<kubernetes><eks><cert-manager><traefik><letsencrypt>
Error configuring TLS error: secret xxx does not exist
Hongbo Miao
# Error configuring TLS error: secret xxx does not exist

(Asked by: username_0 on 2024-06-30)

I am using "Traefik", "cert-mangaer", and "Let's Encrypt".
Here is my ClusterIssuer file
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: production-lets-encrypt-issuer
  namespace: production-hm-cert-manager
spec:
  acme:
    email: me@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: production-lets-encrypt-issuer-account-secret
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik

Here is my Ingress file
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hm-airbyte-ingress
  namespace: production-hm-airbyte
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
  labels:
    app.kubernetes.io/name: hm-airbyte-ingress
    app.kubernetes.io/part-of: production-hm-airbyte
spec:
  rules:
    - host: hm-airbyte.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hm-airbyte-airbyte-webapp-svc
                port:
                  number: 80
  tls:
    - hosts:
        - hm-airbyte.example.com
      secretName: hm-airbyte-ingress-tls


My Traefik pod shows error

"2024-06-27T06:35:45Z ERR Error configuring TLS error="secret production-hm-airbyte/hm-airbyte-ingress-tls does not exist" ingress=hm-airbyte-ingress namespace=production-hm-airbyte providerName=kubernetes"

But I found I actually have a secret Secret: hm-airbyte-ingress-tls-7w9z7 inside namespace production-hm-airbyte. Note it has a random suffix.

I searched online and found

https://github.com/cert-manager/cert-manager/issues/3300
https://github.com/cert-manager/cert-manager/issues/3283

This issue existed for four years, but unfortunately, no solution has been posted.

**Tags:** kubernetes, eks, cert-manager, traefik, letsencrypt


---

## 1 answers

### Answer by username_0 on 2024-06-30. Score: 0 | Accepted answer

6/30/2024
I found what I posted on 6/27/2024 is misleading. Even it repaired the issue in the question, but only on the surface. The Certificate still failed to issue.
It took me some time to make everything including cert-manager, ExternalDNS, Traefik work well.
Also, it turns out acme.cert-manager.io/http01-edit-in-place: "true" annotation is not necessary to fix the issue in the question.
Note for the solution in my case, I am using DNS01 challenge.
I also published all my deployment codes here. Hopefully it can give a bigger picture how it get deployed.
cert-manager AWS IAM role CertManagerRole
Trust relationships
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud": "sts.amazonaws.com",
                    "oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub": "system:serviceaccount:production-hm-cert-manager:hm-cert-manager"
                }
            }
        }
    ]
}

Policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "route53:GetChange"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::change/*"
        },
        {
            "Action": [
                "route53:ListResourceRecordSets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
        },
        {
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Condition": {
                "ForAllValues:StringEquals": {
                    "route53:ChangeResourceRecordSetsNormalizedRecordNames": [
                        "_acme-challenge.*"
                    ],
                    "route53:ChangeResourceRecordSetsRecordTypes": [
                        "TXT"
                    ]
                }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
        },
        {
            "Action": [
                "route53:ListHostedZonesByName"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

ExternalDNS AWS IAM role ExternalDNSRole
Trust relationships
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:sub": "system:serviceaccount:production-hm-external-dns:hm-external-dns",
                    "oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "route53:ChangeResourceRecordSets"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:route53:::hostedzone/xxxxxxxxxxxxxxxxxxxxx"
      ]
    },
    {
      "Action": [
        "route53:ListHostedZones",
        "route53:ListResourceRecordSets",
        "route53:ListTagsForResource"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

cert-manager Helm chart custom values.yaml
crds:
  enabled: true
  keep: true
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/CertManagerRole
securityContext:
  fsGroup: 1001

ExternalDNS Helm chart custom values.yaml
provider: aws
aws:
  region: us-west-2
  zoneType: public
  txtOwnerId: Z1XXXXXXXXXXXXXXXXXXX
  domainFilters:
    - internal.example.com
  registry: txt
policy: sync
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/ExternalDNSRole

Traefik Helm chart custom values.yaml
providers:
  kubernetesIngress:
    publishedService:
      enabled: true
service:
  enabled: true
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: production-lets-encrypt-cluster-issuer
  namespace: production-hm-cert-manager
spec:
  acme:
    email: me@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: production-lets-encrypt-cluster-issuer-account-secret
    solvers:
      - selector:
          dnsZones:
            - internal.example.com
        dns01:
          route53:
            region: us-west-2

Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hm-airbyte-ingress
  namespace: production-hm-airbyte
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    cert-manager.io/cluster-issuer: production-lets-encrypt-cluster-issuer
spec:
  rules:
    - host: hm-airbyte.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hm-airbyte-airbyte-webapp-svc
                port:
                  number: 80
  tls:
    - hosts:
        - hm-airbyte.internal.example.com
      secretName: production-hm-airbyte-certificate

6/27/2024 (Misleading Answer)
After a lot of experiments, I found if I add acme.cert-manager.io/http01-edit-in-place: "true" annotation, it helps resolve the issue.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hm-airbyte-ingress
  namespace: production-hm-airbyte
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    cert-manager.io/cluster-issuer: production-lets-encrypt-issuer
    acme.cert-manager.io/http01-edit-in-place: "true"  # <- Added this
  labels:
    app.kubernetes.io/name: hm-airbyte-ingress
    app.kubernetes.io/part-of: production-hm-airbyte
spec:
  rules:
    - host: hm-airbyte.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hm-airbyte-airbyte-webapp-svc
                port:
                  number: 80
  tls:
    - hosts:
        - hm-airbyte.internal.example.com
      secretName: hm-airbyte-ingress-tls

So now here is what happens after I apply this Ingress file

A temporary Secret: hm-airbyte-ingress-tls-7w9z7 with random suffix will be created
After about half minute, a Secret: hm-airbyte-ingress-tls without suffix will be created
The one will suffix will be automatically deleted.

Without this annotation, based on my experiment, it will be stuck at step 1 forever.
This is the Secret: hm-airbyte-ingress-tls without suffix, you can see the difference between the temporary secret I posted in the question.

Here is the explanation of this annotation:

acme.cert-manager.io/http01-edit-in-place: "true": this controls whether the ingress is modified 'in-place', or a new one is created specifically for the HTTP01 challenge. If present, and set to "true", the existing ingress will be modified. Any other value, or the absence of the annotation assumes "false". This annotation will also add the annotation "cert-manager.io/issue-temporary-certificate": "true" onto created certificates which will cause a temporary certificate to be set on the resulting Secret until the final signed certificate has been returned. This is useful for keeping compatibility with the ingress-gce component.

P.S. I cannot explain why this annotation helps resolve the issue. Because I actually expect if with this annotation, the temporary secret will not be created at all. Hope someone can further explain in future, thanks!


---

