<p>After days and tests here's the solution :</p>
<p>Kubernetes worker and master got the wrong certificate.</p>
<p>In my case, i have to :</p>
<ul>
<li>create a .pem certificate chain composed of the intermediate + wildcard certificate</li>
<li>convert the .pem chain into a .pem fileas my tool (Nutanix Kubernetes Engine) need a .pem format.</li>
</ul>
<p>Finally, workers and master got certificate updated into /etc/containerd/certs.d/my_repo_folder.</p>
<p>This makes AWX working and successfully pull image from my private registry.</p>

CC BY-SA 4.0
2023-08-25T11:29:21.997Z
2023-08-25T11:29:21.997Z
motorbass
<p><strong>Context:</strong><br />
I got a K8s cluster where I setup AWX using the official <a href="https://github.com/ansible/awx-operator/blob/devel/docs/installation/helm-install-on-existing-cluster.md" rel="nofollow noreferrer">Helm install</a>. On this cluster, there I also setup a Sonatype Nexus repository with a docker(hosted) private registry configured in order to pull custom AWX EE image I built.</p>
<p>At the moment, the private registry works only with a HTTP listener configured on port 8080.
I'm able to login and push/pull my custom image to this registry from another server using docker cli.</p>
<p><strong>Configuration/AWX Setup for image pulling:</strong><br />
In order to pull my custom image into my AWX project, I configure AWX the following way :</p>
<ol>
<li><p>Setup a custom EE
<a href="https://i.sstatic.net/8Q1sD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8Q1sD.png" alt="enter image description here" /></a></p>
</li>
<li><p>Create a Container Registry credentials<br />
<a href="https://i.sstatic.net/A5YtC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/A5YtC.png" alt="enter image description here" /></a></p>
</li>
<li><p>And finally setup Execution Environment on my templates.</p>
</li>
</ol>
<p><strong>Problem:</strong></p>
<p>When i launch a template, it keeps running for a long time, and here's error i got from kubectl :</p>
<pre><code>PS &gt; kubectl describe pods automation-job-50-78mp5
Name:             automation-job-50-78mp5
Namespace:        awx
Priority:         0
Service Account:  default
Node:             XXXXXX
Start Time:       Tue, 22 Aug 2023 11:42:04 +0200
Labels:           ansible-awx=4b53c506-cd99-41df-a4d5-1ed0395e8906
                  ansible-awx-job-id=50
Annotations:      cni.projectcalico.org/containerID: 66b9d3f6852b82b5e1d3b37ad8fc199667170c207f5f79af2c2e5d3a67007d9e
                  cni.projectcalico.org/podIP: 172.20.5.208/32
                  cni.projectcalico.org/podIPs: 172.20.5.208/32
Status:           Pending
IP:               172.20.5.208
IPs:
  IP:  172.20.5.208
Containers:
  worker:
    Container ID:
    Image:         awx-repo.mydomain.com/awx-repo
    Image ID:
    Port:          &lt;none&gt;
    Host Port:     &lt;none&gt;
    Args:
      ansible-runner
      worker
      --private-data-dir=/runner
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        250m
      memory:     100Mi
    Environment:  &lt;none&gt;
    Mounts:       &lt;none&gt;
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:            &lt;none&gt;
QoS Class:          Burstable
Node-Selectors:     &lt;none&gt;
Tolerations:        node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  20s                default-scheduler  Successfully assigned awx/automation-job-50-78mp5 to frhd01cor003-ebeb10-worker-3
  Normal   BackOff    19s (x2 over 20s)  kubelet            Back-off pulling image &quot;awx-repo.mydomain.com/awx-repo&quot;
  Warning  Failed     19s (x2 over 20s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    5s (x2 over 20s)   kubelet            Pulling image &quot;awx-repo.mydomain.com/awx-repo&quot;
  Warning  Failed     5s (x2 over 20s)   kubelet            Failed to pull image &quot;awx-repo.mydomain.com/awx-repo&quot;: rpc error: code = Unknown desc = failed to pull and unpack image &quot;awx-repo.mydomain.com/awx-repo:latest&quot;: failed to resolve reference &quot;awx-repo.mydomain.com/awx-repo:latest&quot;: failed to do request: Head &quot;https://awx-repo.mydomain.com/v2/awx-repo/manifests/latest&quot;: x509: certificate signed by unknown authority
  Warning  Failed     5s (x2 over 20s)   kubelet            Error: ErrImagePull
</code></pre>
<p>My question are :</p>
<ul>
<li>How can I double check the pod is using credentials I set from AWX GUI ?</li>
<li>As credentials works from the server, is there a &quot;dedicated format&quot; we have to set from AWX GUI ?</li>
<li>Does the certificate x509 warning may impact the pull action ? (even if my repo seems well configured with a wildcard certificate&quot;</li>
</ul>

<kubernetes><docker-registry><nexus><awx>
AWX Failed to pull image on a private registry
motorbass
# AWX Failed to pull image on a private registry

(Asked by: username_0 on 2023-08-22)

Context:
I got a K8s cluster where I setup AWX using the official Helm install. On this cluster, there I also setup a Sonatype Nexus repository with a docker(hosted) private registry configured in order to pull custom AWX EE image I built.
At the moment, the private registry works only with a HTTP listener configured on port 8080.
I'm able to login and push/pull my custom image to this registry from another server using docker cli.
Configuration/AWX Setup for image pulling:
In order to pull my custom image into my AWX project, I configure AWX the following way :

Setup a custom EE


Create a Container Registry credentials


And finally setup Execution Environment on my templates.


Problem:
When i launch a template, it keeps running for a long time, and here's error i got from kubectl :
PS > kubectl describe pods automation-job-50-78mp5
Name:             automation-job-50-78mp5
Namespace:        awx
Priority:         0
Service Account:  default
Node:             XXXXXX
Start Time:       Tue, 22 Aug 2023 11:42:04 +0200
Labels:           ansible-awx=4b53c506-cd99-41df-a4d5-1ed0395e8906
                  ansible-awx-job-id=50
Annotations:      cni.projectcalico.org/containerID: 66b9d3f6852b82b5e1d3b37ad8fc199667170c207f5f79af2c2e5d3a67007d9e
                  cni.projectcalico.org/podIP: 172.20.5.208/32
                  cni.projectcalico.org/podIPs: 172.20.5.208/32
Status:           Pending
IP:               172.20.5.208
IPs:
  IP:  172.20.5.208
Containers:
  worker:
    Container ID:
    Image:         awx-repo.mydomain.com/awx-repo
    Image ID:
    Port:          
    Host Port:     
    Args:
      ansible-runner
      worker
      --private-data-dir=/runner
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        250m
      memory:     100Mi
    Environment:  
    Mounts:       
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:            
QoS Class:          Burstable
Node-Selectors:     
Tolerations:        node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  20s                default-scheduler  Successfully assigned awx/automation-job-50-78mp5 to frhd01cor003-ebeb10-worker-3
  Normal   BackOff    19s (x2 over 20s)  kubelet            Back-off pulling image "awx-repo.mydomain.com/awx-repo"
  Warning  Failed     19s (x2 over 20s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    5s (x2 over 20s)   kubelet            Pulling image "awx-repo.mydomain.com/awx-repo"
  Warning  Failed     5s (x2 over 20s)   kubelet            Failed to pull image "awx-repo.mydomain.com/awx-repo": rpc error: code = Unknown desc = failed to pull and unpack image "awx-repo.mydomain.com/awx-repo:latest": failed to resolve reference "awx-repo.mydomain.com/awx-repo:latest": failed to do request: Head "https://awx-repo.mydomain.com/v2/awx-repo/manifests/latest": x509: certificate signed by unknown authority
  Warning  Failed     5s (x2 over 20s)   kubelet            Error: ErrImagePull

My question are :

How can I double check the pod is using credentials I set from AWX GUI ?
As credentials works from the server, is there a "dedicated format" we have to set from AWX GUI ?
Does the certificate x509 warning may impact the pull action ? (even if my repo seems well configured with a wildcard certificate"

**Tags:** kubernetes, docker-registry, nexus, awx


---

## 1 answers

### Answer by username_0 on 2023-08-25. Score: 0 | Accepted answer

After days and tests here's the solution :

Kubernetes worker and master got the wrong certificate.

In my case, i have to :

* create a .pem certificate chain composed of the intermediate + wildcard certificate
* convert the .pem chain into a .pem fileas my tool (Nutanix Kubernetes Engine) need a .pem format.

Finally, workers and master got certificate updated into /etc/containerd/certs.d/my\_repo\_folder.

This makes AWX working and successfully pull image from my private registry.


---

