<p>We're also seeing this issue on Laravel Forge. For us it mostly happens after a Craft update is deployed, so it might have something to do with the queue runner still having the previous Craft version loaded.</p>
<p>While I'm not sure what causes it, as a workaround you can ensure that the queue listener restarts periodically and/or after deployments.</p>
<p>There are some problems with this approach. For sites with a lot of traffic and queue jobs, restarting the queue runner may result in the queue getting backed up. Also, when a queue job is running while the queue runner is being forcefully restarted, it can cause the job to fail. To address this, we also have a nightly cron job that runs <code>queue/retry</code> to automatically retry failed queue jobs. But for small and medium sites that don't have tons of queue jobs coming in 24/7, this approach works quite well.</p>
<h2>Restarting the queue runner periodically</h2>
<p>Create a cron job in Forge, in the server settings under <strong>Scheduler</strong>:</p>
<ul>
<li><strong>Command:</strong> <code>sudo supervisorctl restart all</code></li>
<li><strong>User:</strong> <code>forge</code></li>
<li><strong>Frequency:</strong>  Nightly (or choose whatever you like)</li>
</ul>
<p>On newly provisioned servers, this should work without any further adjustments, because Forge now allows the <code>forge</code> user to execute <code>supervisorctl</code> commands without a password. On older servers, make sure the following snippet is in your <code>/etc/sudoers.d/supervisorctl</code>:</p>
<pre><code>forge ALL=NOPASSWD: /usr/bin/supervisorctl reload
forge ALL=NOPASSWD: /usr/bin/supervisorctl reread
forge ALL=NOPASSWD: /usr/bin/supervisorctl restart *
forge ALL=NOPASSWD: /usr/bin/supervisorctl start *
forge ALL=NOPASSWD: /usr/bin/supervisorctl status *
forge ALL=NOPASSWD: /usr/bin/supervisorctl status
forge ALL=NOPASSWD: /usr/bin/supervisorctl stop *
forge ALL=NOPASSWD: /usr/bin/supervisorctl update *
forge ALL=NOPASSWD: /usr/bin/supervisorctl update
</code></pre>
<h2>Restarting the queue runner after deployments</h2>
<p>To restart the queue runner after every deployment, add this to the end of your deployment script:</p>
<pre><code>sudo supervisorctl restart all
</code></pre>
<p>This will restart <strong>all</strong> daemons. Alternatively, you can also find the ID of your queue runner (use <code>sudo supervisorctl status</code> to list all active daemons) and only restart that in your deployment (replace <code>&lt;daemon-id&gt;</code> with the actual ID):</p>
<pre><code>sudo supervisorctl restart &lt;daemon-id&gt;
</code></pre>
<p>On newer servers, if you're not using website isolation, this should work by default. If you <em>are</em> using website isolation, you need to allow the website user to run <code>supervisorctl</code> without a password, since you can't (or rather, shouldn't) provide the password in the deployment script.</p>
<p>Add this to your <code>/etc/sudoers.d/supervisorctl</code>:</p>
<pre><code>websiteuser ALL=NOPASSWD: /usr/bin/supervisorctl *
</code></pre>
<p>Replace <code>websiteuser</code> with the website's (Ubuntu) user in Forge.</p>

CC BY-SA 4.0
2023-11-08T14:14:44.263Z
Thanks Moritz for a good and thoughtful answer.

Unfortunately in my case, deployments doesn't seem to be connected with the issue.

It has happened to pages that have been without deployments for weeks, and suddenly the daemon fails.

The tips are still very useful though. Maybe regular restarting is necessary and will help ‍♂️
Eddie
CC BY-SA 4.0
2023-11-08T14:28:45.160Z
@Eddie Yeah some people on Discord said the same. In this case I would try the nightly restart. Of course, it's only a workaround, but this should work well enough for sites without huge amounts of traffic, so those won't pile up queue jobs while nobody is looking.
MoritzLost
CC BY-SA 4.0
2023-11-11T23:53:03.953Z
Although this doesn't technically answer OPs question, I awarded the bounty to @MoritzLost as their answer at least offers some well-explained, concrete actions to mitigate the problem in the interim.
Mats Mikkel Rummelhoff
CC BY-SA 4.0
2023-11-13T09:23:06.540Z
@MatsMikkelRummelhoff Yeah, unfortunate that for now there's only a workaround – hopefully someone can figure out why this is happening in the first place! Might also be something the Craft team needs to address, not sure if it's an infrastructure problem or a bug in the queue handler.
MoritzLost
CC BY-SA 4.0
2023-11-06T10:05:31.220Z
2023-11-06T10:05:31.220Z
MoritzLost
<p>What about the <a href="https://www.kolide.com/features/checks/ubuntu-unattended-upgrades" rel="nofollow noreferrer">unattended-upgrades</a>?</p>
<p>We had a similar problem with daemons (and still do) but not with the database.
In our case, whenever an update occurs, with some chance daemons die with the same error(<code>FATAL Exited too quickly</code>). The supervisor is restarted, but somehow the daemon doesn’t seem to work.</p>
<p>To test this theory, look at when specific service (supervisor, mysql etc) crashed and compare it with the unattended-upgrade.log(everything should be in the /var/log/)</p>
<p>E.g. <a href="https://www.reddit.com/r/docker/comments/k4kk97/docker_daemon_crash_with_unattendedupgrades/" rel="nofollow noreferrer">https://www.reddit.com/r/docker/comments/k4kk97/docker_daemon_crash_with_unattendedupgrades/</a></p>

CC BY-SA 4.0
2023-11-05T22:17:10.817Z
We're experiencing the same issue as OP, and Forge's auto updates (which I believe use `unattended-upgrades`) has definitely been floated as a possible culprit. I'm going to look closer into that; though if this is somehow related I still find it strange that the problem appears to exclusively occur on boxes w/ MySQL 8 (we haven't seen it occur on any of our MariaDB boxes to date, of which we have many).
Mats Mikkel Rummelhoff
CC BY-SA 4.0
2023-11-05T16:36:42.243Z
2023-11-05T16:36:42.243Z
RomanAvr
<p>Somehow the <code>queue/listen</code> occasionally loses access to the database:</p>
<pre><code>FATAL     Exited too quickly
</code></pre>
<p>.</p>
<pre><code>Database Connection Error: Craft CMS can’t connect to the database.
</code></pre>
<p>As the connection is lost, it seems like the daemon is not able to recover, and needs to be manually restarted.</p>
<p><strong>Additional info:</strong></p>
<ul>
<li>Database: MySQL 8.0.35</li>
<li>There was no deployment being done at the time of failure.</li>
<li>I don’t currently have any automatic restarting of the daemon after deployments.</li>
<li>There are no other indications of the database being down at the time the daemon dropped out.</li>
<li>It has happened across multiple sites, and across hosts.</li>
<li>One common factor is that Laravel Forge is used for provisioning and setting up the Daemon.</li>
<li>It has happened to custom VPS’s, and Digital Ocean Droplets.</li>
<li>All affected sites as far as I know are Craft 4.5.5 or later.</li>
<li>This issue is new to me the last 3-5 months, even though I've made Craft sites with this same setup for a couple of years.</li>
<li>The issue has also been discussed in the Discord channel &quot;Devops&quot; late August 2023. <a href="https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099" rel="noreferrer">https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099</a></li>
<li>It seems to have a tendency to happen at night time. Which is also roundabout the same time doing database backups. It doesn't correlate greatly, so it seems farfetched to be linked. Often linked by +- 2 hours.</li>
</ul>
<p>Most of the time this happens, its quite critical to get up and running again, and I don't know how to really get to the bottom of the problem.</p>
<p>Daemon Supervisor Configuration</p>
<pre><code>[program:daemon-758141]
command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/www.site.com/craft queue/listen --verbose

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=2
startsecs=1
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-758141.log
stopwaitsecs=10
stopsignal=SIGTERM
stopasgroup=true
killasgroup=true
</code></pre>

CC BY-SA 4.0
2023-11-03T11:24:43.887Z
This seems like a rather complicated bug based on the lengthy Discord discussion, but I'd like to just leave these links here as they are in my documentation for an issue that I fixed which might _possibly_ be related.

https://discord.com/channels/456442477667418113/456442477667418115/940287266771832892
https://www.freedesktop.org/software/systemd/man/systemd.unit.html#StartLimitIntervalSec=interval
https://unix.stackexchange.com/questions/289629/systemd-restart-always-is-not-honored
James Smith
CC BY-SA 4.0
2023-11-03T11:25:10.897Z
Basically, with the default settings, even a 500ms DB outage would have caused the queue daemon to fail and not restart.
James Smith
CC BY-SA 4.0
2023-11-03T11:39:46.783Z
Thanks for good links and points. Unfortunately they seem to be only systemd related, but still gives some guidance.
In my case Supervisor is used, and not systemd, and I dont seem to find a similar restart-timing configuration in Supervisor ‍♂️
Eddie
<mysql><hosting><services><queue>
Craft loses contact with database, and stops daemon
# Craft loses contact with database, and stops daemon

(Asked by: username_0 on 2023-11-03)

Somehow the queue/listen occasionally loses access to the database:
FATAL     Exited too quickly

.
Database Connection Error: Craft CMS can’t connect to the database.

As the connection is lost, it seems like the daemon is not able to recover, and needs to be manually restarted.
Additional info:

Database: MySQL 8.0.35
There was no deployment being done at the time of failure.
I don’t currently have any automatic restarting of the daemon after deployments.
There are no other indications of the database being down at the time the daemon dropped out.
It has happened across multiple sites, and across hosts.
One common factor is that Laravel Forge is used for provisioning and setting up the Daemon.
It has happened to custom VPS’s, and Digital Ocean Droplets.
All affected sites as far as I know are Craft 4.5.5 or later.
This issue is new to me the last 3-5 months, even though I've made Craft sites with this same setup for a couple of years.
The issue has also been discussed in the Discord channel "Devops" late August 2023. https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099
It seems to have a tendency to happen at night time. Which is also roundabout the same time doing database backups. It doesn't correlate greatly, so it seems farfetched to be linked. Often linked by +- 2 hours.

Most of the time this happens, its quite critical to get up and running again, and I don't know how to really get to the bottom of the problem.
Daemon Supervisor Configuration
[program:daemon-758141]
command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/www.site.com/craft queue/listen --verbose

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=2
startsecs=1
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-758141.log
stopwaitsecs=10
stopsignal=SIGTERM
stopasgroup=true
killasgroup=true

**Tags:** mysql, hosting, services, queue

> **Comments:**
> **username_1**: This seems like a rather complicated bug based on the lengthy Discord discussion, but I'd like to just leave these links here as they are in my documentation for an issue that I fixed which might _possibly_ be related.  https://discord.com/channels/456442477667418113/456442477667418115/940287266771832892 https://www.freedesktop.org/software/systemd/man/systemd.unit.html#StartLimitIntervalSec=interval https://unix.stackexchange.com/questions/289629/systemd-restart-always-is-not-honored  
> **username_1**: Basically, with the default settings, even a 500ms DB outage would have caused the queue daemon to fail and not restart.  
> **username_0**: Thanks for good links and points. Unfortunately they seem to be only systemd related, but still gives some guidance. In my case Supervisor is used, and not systemd, and I dont seem to find a similar restart-timing configuration in Supervisor ‍♂️  

---

## 2 answers

### Answer by username_2 on 2023-11-06. Score: 3

We're also seeing this issue on Laravel Forge. For us it mostly happens after a Craft update is deployed, so it might have something to do with the queue runner still having the previous Craft version loaded.
While I'm not sure what causes it, as a workaround you can ensure that the queue listener restarts periodically and/or after deployments.
There are some problems with this approach. For sites with a lot of traffic and queue jobs, restarting the queue runner may result in the queue getting backed up. Also, when a queue job is running while the queue runner is being forcefully restarted, it can cause the job to fail. To address this, we also have a nightly cron job that runs queue/retry to automatically retry failed queue jobs. But for small and medium sites that don't have tons of queue jobs coming in 24/7, this approach works quite well.
Restarting the queue runner periodically
Create a cron job in Forge, in the server settings under Scheduler:

Command: sudo supervisorctl restart all
User: forge
Frequency:  Nightly (or choose whatever you like)

On newly provisioned servers, this should work without any further adjustments, because Forge now allows the forge user to execute supervisorctl commands without a password. On older servers, make sure the following snippet is in your /etc/sudoers.d/supervisorctl:
forge ALL=NOPASSWD: /usr/bin/supervisorctl reload
forge ALL=NOPASSWD: /usr/bin/supervisorctl reread
forge ALL=NOPASSWD: /usr/bin/supervisorctl restart *
forge ALL=NOPASSWD: /usr/bin/supervisorctl start *
forge ALL=NOPASSWD: /usr/bin/supervisorctl status *
forge ALL=NOPASSWD: /usr/bin/supervisorctl status
forge ALL=NOPASSWD: /usr/bin/supervisorctl stop *
forge ALL=NOPASSWD: /usr/bin/supervisorctl update *
forge ALL=NOPASSWD: /usr/bin/supervisorctl update

Restarting the queue runner after deployments
To restart the queue runner after every deployment, add this to the end of your deployment script:
sudo supervisorctl restart all

This will restart all daemons. Alternatively, you can also find the ID of your queue runner (use sudo supervisorctl status to list all active daemons) and only restart that in your deployment (replace  with the actual ID):
sudo supervisorctl restart 

On newer servers, if you're not using website isolation, this should work by default. If you are using website isolation, you need to allow the website user to run supervisorctl without a password, since you can't (or rather, shouldn't) provide the password in the deployment script.
Add this to your /etc/sudoers.d/supervisorctl:
websiteuser ALL=NOPASSWD: /usr/bin/supervisorctl *

Replace websiteuser with the website's (Ubuntu) user in Forge.

> **Comments:**
> **username_0**: Thanks Moritz for a good and thoughtful answer.  Unfortunately in my case, deployments doesn't seem to be connected with the issue.  It has happened to pages that have been without deployments for weeks, and suddenly the daemon fails.  The tips are still very useful though. Maybe regular restarting is necessary and will help ‍♂️  
> **username_2**: @username_0 Yeah some people on Discord said the same. In this case I would try the nightly restart. Of course, it's only a workaround, but this should work well enough for sites without huge amounts of traffic, so those won't pile up queue jobs while nobody is looking.  
> **username_3**: Although this doesn't technically answer OPs question, I awarded the bounty to @username_2 as their answer at least offers some well-explained, concrete actions to mitigate the problem in the interim.  
> **username_2**: @MatsMikkelRummelhoff Yeah, unfortunate that for now there's only a workaround – hopefully someone can figure out why this is happening in the first place! Might also be something the Craft team needs to address, not sure if it's an infrastructure problem or a bug in the queue handler.  

---

### Answer by username_4 on 2023-11-05. Score: 2

What about the [unattended-upgrades](https://www.kolide.com/features/checks/ubuntu-unattended-upgrades)?

We had a similar problem with daemons (and still do) but not with the database.
In our case, whenever an update occurs, with some chance daemons die with the same error(`FATAL Exited too quickly`). The supervisor is restarted, but somehow the daemon doesn’t seem to work.

To test this theory, look at when specific service (supervisor, mysql etc) crashed and compare it with the unattended-upgrade.log(everything should be in the /var/log/)

E.g. <https://www.reddit.com/r/docker/comments/k4kk97/docker_daemon_crash_with_unattendedupgrades/>

> **Comments:**
> **username_3**: We're experiencing the same issue as OP, and Forge's auto updates (which I believe use `unattended-upgrades`) has definitely been floated as a possible culprit. I'm going to look closer into that; though if this is somehow related I still find it strange that the problem appears to exclusively occur on boxes w/ MySQL 8 (we haven't seen it occur on any of our MariaDB boxes to date, of which we have many).  

---

