<h2>Just unregister offline runners:</h2>
<p>As per <a href="https://docs.gitlab.com/runner/commands/#gitlab-runner-verify" rel="nofollow noreferrer">GitLab Runner commands</a> manual:</p>
<blockquote>
<p>To delete the old and removed from GitLab runners, execute the following command.</p>
</blockquote>
<pre><code>    gitlab-runner verify --delete
</code></pre>
<h2>You are not alone</h2>
<p>P.S. The problem is common: there <a href="https://google.com/search?q=site%3A+https%3A%2F%2Fgitlab.com%2Fgitlab-org%2F+issue++intitle%3Arunner+intitle%3Aclean%7Cremove%7Cexpired" rel="nofollow noreferrer">are many issues</a> with questions like yours.</p>
<p>Moreover, there are several custom recipes to unregister &quot;offline&quot; runners:</p>
<ul>
<li><a href="https://gitlab.com/gitlab-org/gitlab-foss/issues/19828#note_54956232" rel="nofollow noreferrer"><code>bash</code> script</a></li>
<li><a href="https://gitlab.com/gitlab-org/gitlab-foss/issues/19828#note_132438194" rel="nofollow noreferrer">API oneliner with <code>curl</code></a></li>
<li><a href="https://gitlab.com/gitlab-org/gitlab-foss/issues/19828#note_202265652" rel="nofollow noreferrer"><code>python</code> script</a></li>
</ul>
<h2>API for &quot;zombie&quot; runners</h2>
<p>As <a href="https://devops.stackexchange.com/users/10807/rlandster">rlandster</a> noted, sometimes you'll need an <a href="https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19828#note_132438194" rel="nofollow noreferrer">API</a>
to unregister &quot;zombie&quot; runners.</p>
<pre><code>curl -S --header &quot;PRIVATE-TOKEN:&lt;token&gt;&quot; &quot;https://gitlab.example.com/api/v4/runners/all&quot; | jq '.[] | select(.status == &quot;offline&quot;) | .id' | xargs -I runner_id curl -S --request DELETE --header &quot;PRIVATE-TOKEN:&lt;token&gt;&quot; &quot;https://gitlab.example.com/api/v4/runners/runner_id&quot;`
</code></pre>
<p>This will list all runners, instead of the personal ones: <a href="https://docs.gitlab.com/ee/api/runners.html#list-all-runners" rel="nofollow noreferrer">https://docs.gitlab.com/ee/api/runners.html#list-all-runners</a></p>
<h2>P.S. note about runners</h2>
<p>Pls note that <code>Gitlab</code> itself doesn't manage runners. So, e.g. to restart or shutdown the runner, use appropriate commands on the runner's hosts.</p>

CC BY-SA 4.0
2020-04-29T10:32:12.487Z
Running `gitlab-runner verify --delete` does not unregister the "zombie" jobs; the API interface was necessary to do that.
rlandster
CC BY-SA 4.0
2020-04-21T07:44:20.903Z
2020-04-30T08:23:41.330Z
2020-06-18T08:28:10.803Z
Yasen
<p>I am running <code>GitLab Runner</code> using a <code>Helm</code> chart as described on the <a href="https://docs.gitlab.com/runner/install/kubernetes.html" rel="nofollow noreferrer">GitLab Runner Helm Chart page</a>. </p>

<p>Everything seems to be working, but after about two months of running I see on my <code>GitLab</code> group that controls the <code>GitLab runner</code> on the "CI / CD Settings|Runners" page the message</p>

<blockquote>
  <p>"Available group Runners: 54". </p>
</blockquote>

<p>Here is the first couple on that page:</p>

<p><a href="https://i.sstatic.net/zjdWOm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zjdWOm.png" alt="enter image description here"></a></p>

<p>Rather than listing all 54 here is the list run through <code>uniq -c</code>: </p>

<pre><code>      3 gitlab-runner-gitlab-runner-9fcb644b9-4f5ss
      2 gitlab-runner-gitlab-runner-9fcb644b9-4gqqq
      1 gitlab-runner-gitlab-runner-9fcb644b9-889rc
      2 gitlab-runner-gitlab-runner-9fcb644b9-grb5j
      1 gitlab-runner-gitlab-runner-9fcb644b9-hhv8f
      1 gitlab-runner-gitlab-runner-9fcb644b9-kzcdv
     13 gitlab-runner-gitlab-runner-9fcb644b9-nvrms
      7 gitlab-runner-gitlab-runner-9fcb644b9-pl2k4
      2 gitlab-runner-gitlab-runner-9fcb644b9-sxlmp
      9 gitlab-runner-gitlab-runner-9fcb644b9-vdmsj
      3 gitlab-runner-gitlab-runner-9fcb644b9-vgdnv
      8 gitlab-runner-gitlab-runner-9fcb644b9-wst8q
      2 gitlab-runner-gitlab-runner-9fcb644b9-xnzhx
</code></pre>

<p>Looking at my <code>GitLab Runner</code> <code>Kubernetes</code> container there seems to be only <em>one</em> runner:</p>

<pre><code>$ /usr/bin/gitlab-runner list
Runtime platform                                    arch=amd64 os=linux pid=4953 revision=1b659122 version=12.8.0
Listing configured runners                          ConfigFile=/home/gitlab-runner/.gitlab-runner/config.toml
gitlab-runner-gitlab-runner-9fcb644b9-grb5j         Executor=kubernetes Token=XXXXXXXXXXXX-XXXXX URL=https://myserver.example.com/ 
</code></pre>

<p>Another potentially important piece of information: the <code>Kubernetes</code> node that hosts the <code>GitLab Runner</code> container is preemptible meaning it gets killed and restarted about every 24 hours. </p>

<p>How do I convince <code>GitLab</code> that these old runners are not still around? And how do I avoid this problem in the future?</p>

<kubernetes><gitlab><gitlab-ci-runner><helm>
GitLab CI/CD page listing many gitlab runners that no longer exist
rlandster
# GitLab CI/CD page listing many gitlab runners that no longer exist

(Asked by: username_0 on 2020-04-21)

I am running GitLab Runner using a Helm chart as described on the GitLab Runner Helm Chart page. 
Everything seems to be working, but after about two months of running I see on my GitLab group that controls the GitLab runner on the "CI / CD Settings|Runners" page the message

"Available group Runners: 54". 

Here is the first couple on that page:

Rather than listing all 54 here is the list run through uniq -c: 
      3 gitlab-runner-gitlab-runner-9fcb644b9-4f5ss
      2 gitlab-runner-gitlab-runner-9fcb644b9-4gqqq
      1 gitlab-runner-gitlab-runner-9fcb644b9-889rc
      2 gitlab-runner-gitlab-runner-9fcb644b9-grb5j
      1 gitlab-runner-gitlab-runner-9fcb644b9-hhv8f
      1 gitlab-runner-gitlab-runner-9fcb644b9-kzcdv
     13 gitlab-runner-gitlab-runner-9fcb644b9-nvrms
      7 gitlab-runner-gitlab-runner-9fcb644b9-pl2k4
      2 gitlab-runner-gitlab-runner-9fcb644b9-sxlmp
      9 gitlab-runner-gitlab-runner-9fcb644b9-vdmsj
      3 gitlab-runner-gitlab-runner-9fcb644b9-vgdnv
      8 gitlab-runner-gitlab-runner-9fcb644b9-wst8q
      2 gitlab-runner-gitlab-runner-9fcb644b9-xnzhx

Looking at my GitLab Runner Kubernetes container there seems to be only one runner:
$ /usr/bin/gitlab-runner list
Runtime platform                                    arch=amd64 os=linux pid=4953 revision=1b659122 version=12.8.0
Listing configured runners                          ConfigFile=/home/gitlab-runner/.gitlab-runner/config.toml
gitlab-runner-gitlab-runner-9fcb644b9-grb5j         Executor=kubernetes Token=XXXXXXXXXXXX-XXXXX URL=https://myserver.example.com/ 

Another potentially important piece of information: the Kubernetes node that hosts the GitLab Runner container is preemptible meaning it gets killed and restarted about every 24 hours. 
How do I convince GitLab that these old runners are not still around? And how do I avoid this problem in the future?

**Tags:** kubernetes, gitlab, gitlab-ci-runner, helm


---

## 1 answers

### Answer by username_1 on 2020-04-21. Score: 3

Just unregister offline runners:
As per GitLab Runner commands manual:

To delete the old and removed from GitLab runners, execute the following command.

    gitlab-runner verify --delete

You are not alone
P.S. The problem is common: there are many issues with questions like yours.
Moreover, there are several custom recipes to unregister "offline" runners:

bash script
API oneliner with curl
python script

API for "zombie" runners
As username_0 noted, sometimes you'll need an API
to unregister "zombie" runners.
curl -S --header "PRIVATE-TOKEN:" "https://gitlab.example.com/api/v4/runners/all" | jq '.[] | select(.status == "offline") | .id' | xargs -I runner_id curl -S --request DELETE --header "PRIVATE-TOKEN:" "https://gitlab.example.com/api/v4/runners/runner_id"`

This will list all runners, instead of the personal ones: https://docs.gitlab.com/ee/api/runners.html#list-all-runners
P.S. note about runners
Pls note that Gitlab itself doesn't manage runners. So, e.g. to restart or shutdown the runner, use appropriate commands on the runner's hosts.

> **Comments:**
> **username_0**: Running `gitlab-runner verify --delete` does not unregister the "zombie" jobs; the API interface was necessary to do that.  

---

