<p>This functionality doesn't yet exist in the UI, though internally job logs are upload as a job artifact. And you can search them with a little bit of work. The backing store is configured in your, <code>/etc/gitlab/gitlab.rb</code>.</p>
<ul>
<li><p>If <code>artifacts_object_store_enabled</code> is enabled and you're using the <a href="https://docs.gitlab.com/ee/administration/object_storage.html" rel="nofollow noreferrer">Object Storage</a>, the artifacts will store in whatever the value is of <code>artifacts_object_store_remote_directory</code> under a file called <code>job.log</code>. Assuming that value is <code>artifacts</code>, and you're using something like <a href="https://min.io/" rel="nofollow noreferrer">minio</a>, you can search across jobs for a specific failure with</p>
<ul>
<li><p><code>mc find</code> and <code>mc cat</code>,</p>
<pre class="lang-sh prettyprint-override"><code># In the example below, I'm looking for failures that contain Client.Timeout
mc find minio/artifacts \
  --name 'job.log'      \
  --exec 'sh -c &quot;mc cat {} | grep Client\.Timeout | xargs -r echo {time} {}; exit 0&quot;' \
  | tee /tmp/client_timeout.log
</code></pre>
<p>In my experience, this is just too slow as <code>mc find</code> isn't indexed it's just looking for any bucket with the name <code>job.log</code>.</p>
</li>
<li><p>A faster method is to lean on the database. Gitlab ships a PostgreSQL client wrapper called <code>gitlab-psql</code> which allows you to connect to underlying PostgreSQL datastore. After you connect with <code>gitlab-psql</code> can query for all <code>file.log</code> and <a href="https://devops.stackexchange.com/q/17298/18965">get the relative path to the bucket from the result</a>. You can easily tool this by creating a batch file of commands for <a href="https://github.com/peak/s5cmd" rel="nofollow noreferrer">s5cmd</a> log,</p>
<pre><code>\t
\a
\o /tmp/pull_logs.s5cmd
SELECT FORMAT(
  'cp s3://artifacts/%s /tmp/job_%s.log',
  to_bucket_path(ci_job_artifacts),
  id
)
FROM ci_job_artifacts
WHERE file = 'job.log'
AND expire_at IS NULL
AND file_store = 2;
</code></pre>
<p>Which will generate a file you can run with something like <a href="https://github.com/peak/s5cmd" rel="nofollow noreferrer">s5cmd</a> to pull down all of your logs from the Gitlab Object store. You'd run it by setting up your <code>~/.aws/credentials</code> and running (example with minio).</p>
<pre><code>s5cmd --endpoint-url https://minio.acme.net run /tmp/pull_logs.s5cmd
</code></pre>
</li>
</ul>
</li>
<li><p>You can also use <a href="https://docs.gitlab.com/ee/administration/nfs.html" rel="nofollow noreferrer">NFS</a>. If you're using NFS for artifact storage, it's as easy as going to the mount point and searching with ripgrep.</p>
</li>
</ul>

CC BY-SA 4.0
2023-01-19T22:39:40.567Z
2023-01-26T20:08:48.770Z
2023-01-26T20:08:48.770Z
Evan Carroll
<p>I need to find all jobs that failed in a specific fashion, a few methods are viable</p>
<ul>
<li>One method that would be viable is finding where GitLab stores its log files, and searching them with something like Grep. <a href="https://devops.stackexchange.com/q/17254/18965">I asked that question here</a>.</li>
<li>Another method would be using native GitLab search functionality to search the logs. Does such functionality exist?</li>
</ul>

CC BY-SA 4.0
2023-11-01T11:45:42.843Z
The functionality seems not implemented but it was raised here https://gitlab.com/gitlab-org/gitlab/-/issues/363640 .
YAZ84
<gitlab><logging>
Does GitLab support searching logs across jobs for specific strings?
Evan Carroll
# Does GitLab support searching logs across jobs for specific strings?

(Asked by: username_0 on 2023-01-17)

I need to find all jobs that failed in a specific fashion, a few methods are viable

* One method that would be viable is finding where GitLab stores its log files, and searching them with something like Grep. [I asked that question here](https://devops.stackexchange.com/q/17254/18965).
* Another method would be using native GitLab search functionality to search the logs. Does such functionality exist?

**Tags:** gitlab, logging

> **Comments:**
> **username_1**: The functionality seems not implemented but it was raised here https://gitlab.com/gitlab-org/gitlab/-/issues/363640 .  

---

## 1 answers

### Answer by username_0 on 2023-01-19. Score: 0 | Accepted answer

This functionality doesn't yet exist in the UI, though internally job logs are upload as a job artifact. And you can search them with a little bit of work. The backing store is configured in your, /etc/gitlab/gitlab.rb.

If artifacts_object_store_enabled is enabled and you're using the Object Storage, the artifacts will store in whatever the value is of artifacts_object_store_remote_directory under a file called job.log. Assuming that value is artifacts, and you're using something like minio, you can search across jobs for a specific failure with

mc find and mc cat,
# In the example below, I'm looking for failures that contain Client.Timeout
mc find minio/artifacts \
  --name 'job.log'      \
  --exec 'sh -c "mc cat {} | grep Client\.Timeout | xargs -r echo {time} {}; exit 0"' \
  | tee /tmp/client_timeout.log

In my experience, this is just too slow as mc find isn't indexed it's just looking for any bucket with the name job.log.

A faster method is to lean on the database. Gitlab ships a PostgreSQL client wrapper called gitlab-psql which allows you to connect to underlying PostgreSQL datastore. After you connect with gitlab-psql can query for all file.log and get the relative path to the bucket from the result. You can easily tool this by creating a batch file of commands for s5cmd log,
\t
\a
\o /tmp/pull_logs.s5cmd
SELECT FORMAT(
  'cp s3://artifacts/%s /tmp/job_%s.log',
  to_bucket_path(ci_job_artifacts),
  id
)
FROM ci_job_artifacts
WHERE file = 'job.log'
AND expire_at IS NULL
AND file_store = 2;

Which will generate a file you can run with something like s5cmd to pull down all of your logs from the Gitlab Object store. You'd run it by setting up your ~/.aws/credentials and running (example with minio).
s5cmd --endpoint-url https://minio.acme.net run /tmp/pull_logs.s5cmd




You can also use NFS. If you're using NFS for artifact storage, it's as easy as going to the mount point and searching with ripgrep.


---

