<p>I just upgraded to 3.7.27.2 (as an intermediate step) and since I'm always deploying with the latest docker image (craftcms/nginx:7.4) I have also gotten the latest expat upgrade.</p>
<p>I'm not sure where the root cause of my problems are, but I'm seeing a lot of weird issues right now.</p>
<p>It can't do any transforms of assets. No real error message for it, just a stack trace:</p>
<pre><code>2022-02-28 12:42:24 [94.234.41.252][-][-][error][craft\errors\AssetTransformException] craft\errors\AssetTransformException: Det gick inte att generera transformeringen med id 2028. in /app/vendor/craftcms/cms/src/services/AssetTransforms.php:669
Stack trace:
#0 /app/vendor/craftcms/cms/src/controllers/AssetsController.php(1150): craft\services\AssetTransforms-&gt;ensureTransformUrlByIndexModel(Object(craft\models\AssetTransformIndex))
#1 [internal function]: craft\controllers\AssetsController-&gt;actionGenerateTransform(2028)
#2 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#3 /app/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction-&gt;runWithParams(Array)
#4 /app/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller-&gt;runAction('generate-transf...', Array)
#5 /app/vendor/craftcms/cms/src/web/Application.php(287): yii\base\Module-&gt;runAction('assets/generate...', Array)
#6 /app/vendor/craftcms/cms/src/web/Application.php(596): craft\web\Application-&gt;runAction('assets/generate...', Array)
#7 /app/vendor/craftcms/cms/src/web/Application.php(266): craft\web\Application-&gt;_processActionRequest(Object(craft\web\Request))
#8 /app/vendor/yiisoft/yii2/base/Application.php(384): craft\web\Application-&gt;handleRequest(Object(craft\web\Request))
#9 /app/web/index.php(22): yii\base\Application-&gt;run()
#10 {main}
2022-02-28 12:42:24 [94.234.41.252][-][-][error][yii\web\HttpException:500] yii\web\ServerErrorHttpException: Image transform cannot be created. in /app/vendor/craftcms/cms/src/controllers/AssetsController.php:1153
Stack trace:
#0 [internal function]: craft\controllers\AssetsController-&gt;actionGenerateTransform(2028)
#1 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#2 /app/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction-&gt;runWithParams(Array)
#3 /app/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller-&gt;runAction('generate-transf...', Array)
#4 /app/vendor/craftcms/cms/src/web/Application.php(287): yii\base\Module-&gt;runAction('assets/generate...', Array)
#5 /app/vendor/craftcms/cms/src/web/Application.php(596): craft\web\Application-&gt;runAction('assets/generate...', Array)
#6 /app/vendor/craftcms/cms/src/web/Application.php(266): craft\web\Application-&gt;_processActionRequest(Object(craft\web\Request))
#7 /app/vendor/yiisoft/yii2/base/Application.php(384): craft\web\Application-&gt;handleRequest(Object(craft\web\Request))
#8 /app/web/index.php(22): yii\base\Application-&gt;run()
#9 {main}
</code></pre>
<p>I have verified writability to the NFS/EFS share where it keeps the thumbnails, and access to the source in S3.</p>
<p>The queue runner (I think), keeps complaining about not being able to aqcuire mutex lock</p>
<pre><code>2022-02-28 12:55:24 [pid: 15] - Worker is stopped (0:29:38)
2022-02-28 12:55:51 [-][-][-][error][yii\base\Exception] yii\base\Exception: Could not acquire a mutex lock for the queue (queue). in /app/vendor/craftcms/cms/src/queue/Queue.php:780
Stack trace:
#0 /app/vendor/craftcms/cms/src/queue/Queue.php(618): craft\queue\Queue-&gt;_lock(Object(Closure))
#1 /app/vendor/craftcms/cms/src/queue/Queue.php(130): craft\queue\Queue-&gt;reserve()
#2 [internal function]: craft\queue\Queue-&gt;craft\queue\{closure}(Object(Closure))
#3 /app/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): call_user_func(Object(Closure), Object(Closure))
#4 /app/vendor/craftcms/cms/src/queue/Queue.php(140): yii\queue\cli\Queue-&gt;runWorker(Object(Closure))
#5 /app/vendor/craftcms/cms/src/queue/Command.php(88): craft\queue\Queue-&gt;run(true, 3)
#6 [internal function]: craft\queue\Command-&gt;actionListen(3)
#7 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#8 /app/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction-&gt;runWithParams(Array)
#9 /app/vendor/yiisoft/yii2/console/Controller.php(182): yii\base\Controller-&gt;runAction('listen', Array)
#10 /app/vendor/yiisoft/yii2/base/Module.php(552): yii\console\Controller-&gt;runAction('listen', Array)
#11 /app/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module-&gt;runAction('queue/listen', Array)
#12 /app/vendor/craftcms/cms/src/console/Application.php(89): yii\console\Application-&gt;runAction('queue/listen', Array)
#13 /app/vendor/yiisoft/yii2/console/Application.php(147): craft\console\Application-&gt;runAction('queue/listen', Array)
#14 /app/vendor/yiisoft/yii2/base/Application.php(384): yii\console\Application-&gt;handleRequest(Object(craft\console\Request))
#15 /app/craft(23): yii\base\Application-&gt;run()
#16 {main}
</code></pre>
<p>I've switched to a redis backed mutex to avoid this being some mysql erratic behavior, but no difference.</p>
<p>I'm also seeing php-fpm processed killed, and core dumping:</p>
<pre><code>[28-Feb-2022 12:00:47] WARNING: [pool www] child 26 exited on signal 9 (SIGKILL) after 1012.217139 seconds from start
[28-Feb-2022 12:00:47] NOTICE: [pool www] child 35 started
2022-02-28 12:00:49,542 INFO exited: craft-queue (exit status 1; not expected)
2022-02-28 12:00:51,704 INFO spawned: 'craft-queue' with pid 36
2022-02-28 12:00:52,706 INFO success: craft-queue entered RUNNING state, process has stayed up for &gt; than 0 seconds (startsecs)
[28-Feb-2022 12:00:59] WARNING: [pool www] child 29 exited on signal 11 (SIGSEGV - core dumped) after 793.903611 seconds from start
</code></pre>
<p>for no apparent reason. The core dumps all point to php-fpm, but I haven't been able to produce a stack trace from them.</p>
<p>Is anyone else seeing the same? I'm reluctant to upgrade further since that would put me in a point of no return...</p>
<p>I'm running this in AWS ECS, running it locally on my Mac (through docker) doesn't show the same behavior at all, but it is also a different platform (ARM64, M1)</p>
<p>EDIT: Actually, the mutex error do appear locally now, might be because I started using redis instead.</p>
<p>EDIT2: my config/app.php</p>
<pre><code>return [
    'id' =&gt; App::env('APP_ID') ?: 'CraftCMS',
    'modules' =&gt; [
        'cache-purge-module' =&gt; \elefant\craft\modules\CachePurgeModule::class,
    ],
    'bootstrap' =&gt; ['cache-purge-module'],
    'components' =&gt; [
        'log' =&gt; function() {
            return Craft::createObject([
                'class' =&gt; yii\log\Dispatcher::class,
                'targets' =&gt; [
                    [
                        'class' =&gt; codemix\streamlog\Target::class,
                        'url' =&gt; 'php://stderr',
                        'levels' =&gt; Logger::LEVEL_ERROR | Logger::LEVEL_WARNING,
                        'logVars' =&gt; [],
                        'except' =&gt; [
                            'yii\web\HttpException:404',
                        ],
                    ],

                ]
            ]);
        },
        'session' =&gt; function() {
            // Get the default component config
            $config = craft\helpers\App::sessionConfig();

            // Override the class to use DB session class
            $config['class'] = yii\web\DbSession::class;

            // Set the session table name
            $config['sessionTable'] = craft\db\Table::PHPSESSIONS;

            // Instantiate and return it
            return Craft::createObject($config);
        },
        'mutex' =&gt; [
            'class' =&gt; 'yii\redis\Mutex',
        ],
        'redis' =&gt; [
            'class' =&gt; 'yii\redis\Connection',
            'hostname' =&gt; App::env('REDIS_SERVER'),
        ],
    ],
];
</code></pre>

CC BY-SA 4.0
2022-02-28T13:37:56.957Z
Hey there! Would you mind sharing your config/app.php? My hunch would be that Craft CMS php processes can't connect to redis, so can't acquire a mutex, and eventually get killed because they run for too long.
Joe Forshaw
CC BY-SA 4.0
2022-02-28T13:47:04.287Z
I'm seeing connections in the metrics for elasticache in AWS, so I assumed it was working. 

Re-indexing when saving entries does seem to queue and finish, so maybe it's a red herring... 

I'm more concerned over the assets thumbnail problems and the SIGKILL/SIGSEGV  of php-fpm. I just downgraded to my last known working version (3.7.24) locally and I'm still seeing the erratic SIGKILLS.

I can't test the older craftcms docker image either, since they don't keep previous images in docker hub... :(
jishi
CC BY-SA 4.0
2022-02-28T14:37:13.460Z
Im starting to believe that it boiled down to memory exhaustion in the end. Someone had uploaded 7-8MB assets and when scaling these it seems to have been hitting the max of 2GB on the ec2-instances, causing all kinds of trouble. Confused how that can even happen.
jishi
<assets><docker><mutex>
Upgrading Craft (and docker image) now causes SIGSEGV core dumps and problems with mutex locks
# Upgrading Craft (and docker image) now causes SIGSEGV core dumps and problems with mutex locks

(Asked by: username_0 on 2022-02-28)

I just upgraded to 3.7.27.2 (as an intermediate step) and since I'm always deploying with the latest docker image (craftcms/nginx:7.4) I have also gotten the latest expat upgrade.
I'm not sure where the root cause of my problems are, but I'm seeing a lot of weird issues right now.
It can't do any transforms of assets. No real error message for it, just a stack trace:
2022-02-28 12:42:24 [94.234.41.252][-][-][error][craft\errors\AssetTransformException] craft\errors\AssetTransformException: Det gick inte att generera transformeringen med id 2028. in /app/vendor/craftcms/cms/src/services/AssetTransforms.php:669
Stack trace:
#0 /app/vendor/craftcms/cms/src/controllers/AssetsController.php(1150): craft\services\AssetTransforms->ensureTransformUrlByIndexModel(Object(craft\models\AssetTransformIndex))
#1 [internal function]: craft\controllers\AssetsController->actionGenerateTransform(2028)
#2 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#3 /app/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#4 /app/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller->runAction('generate-transf...', Array)
#5 /app/vendor/craftcms/cms/src/web/Application.php(287): yii\base\Module->runAction('assets/generate...', Array)
#6 /app/vendor/craftcms/cms/src/web/Application.php(596): craft\web\Application->runAction('assets/generate...', Array)
#7 /app/vendor/craftcms/cms/src/web/Application.php(266): craft\web\Application->_processActionRequest(Object(craft\web\Request))
#8 /app/vendor/yiisoft/yii2/base/Application.php(384): craft\web\Application->handleRequest(Object(craft\web\Request))
#9 /app/web/index.php(22): yii\base\Application->run()
#10 {main}
2022-02-28 12:42:24 [94.234.41.252][-][-][error][yii\web\HttpException:500] yii\web\ServerErrorHttpException: Image transform cannot be created. in /app/vendor/craftcms/cms/src/controllers/AssetsController.php:1153
Stack trace:
#0 [internal function]: craft\controllers\AssetsController->actionGenerateTransform(2028)
#1 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#2 /app/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#3 /app/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller->runAction('generate-transf...', Array)
#4 /app/vendor/craftcms/cms/src/web/Application.php(287): yii\base\Module->runAction('assets/generate...', Array)
#5 /app/vendor/craftcms/cms/src/web/Application.php(596): craft\web\Application->runAction('assets/generate...', Array)
#6 /app/vendor/craftcms/cms/src/web/Application.php(266): craft\web\Application->_processActionRequest(Object(craft\web\Request))
#7 /app/vendor/yiisoft/yii2/base/Application.php(384): craft\web\Application->handleRequest(Object(craft\web\Request))
#8 /app/web/index.php(22): yii\base\Application->run()
#9 {main}

I have verified writability to the NFS/EFS share where it keeps the thumbnails, and access to the source in S3.
The queue runner (I think), keeps complaining about not being able to aqcuire mutex lock
2022-02-28 12:55:24 [pid: 15] - Worker is stopped (0:29:38)
2022-02-28 12:55:51 [-][-][-][error][yii\base\Exception] yii\base\Exception: Could not acquire a mutex lock for the queue (queue). in /app/vendor/craftcms/cms/src/queue/Queue.php:780
Stack trace:
#0 /app/vendor/craftcms/cms/src/queue/Queue.php(618): craft\queue\Queue->_lock(Object(Closure))
#1 /app/vendor/craftcms/cms/src/queue/Queue.php(130): craft\queue\Queue->reserve()
#2 [internal function]: craft\queue\Queue->craft\queue\{closure}(Object(Closure))
#3 /app/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): call_user_func(Object(Closure), Object(Closure))
#4 /app/vendor/craftcms/cms/src/queue/Queue.php(140): yii\queue\cli\Queue->runWorker(Object(Closure))
#5 /app/vendor/craftcms/cms/src/queue/Command.php(88): craft\queue\Queue->run(true, 3)
#6 [internal function]: craft\queue\Command->actionListen(3)
#7 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#8 /app/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#9 /app/vendor/yiisoft/yii2/console/Controller.php(182): yii\base\Controller->runAction('listen', Array)
#10 /app/vendor/yiisoft/yii2/base/Module.php(552): yii\console\Controller->runAction('listen', Array)
#11 /app/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('queue/listen', Array)
#12 /app/vendor/craftcms/cms/src/console/Application.php(89): yii\console\Application->runAction('queue/listen', Array)
#13 /app/vendor/yiisoft/yii2/console/Application.php(147): craft\console\Application->runAction('queue/listen', Array)
#14 /app/vendor/yiisoft/yii2/base/Application.php(384): yii\console\Application->handleRequest(Object(craft\console\Request))
#15 /app/craft(23): yii\base\Application->run()
#16 {main}

I've switched to a redis backed mutex to avoid this being some mysql erratic behavior, but no difference.
I'm also seeing php-fpm processed killed, and core dumping:
[28-Feb-2022 12:00:47] WARNING: [pool www] child 26 exited on signal 9 (SIGKILL) after 1012.217139 seconds from start
[28-Feb-2022 12:00:47] NOTICE: [pool www] child 35 started
2022-02-28 12:00:49,542 INFO exited: craft-queue (exit status 1; not expected)
2022-02-28 12:00:51,704 INFO spawned: 'craft-queue' with pid 36
2022-02-28 12:00:52,706 INFO success: craft-queue entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
[28-Feb-2022 12:00:59] WARNING: [pool www] child 29 exited on signal 11 (SIGSEGV - core dumped) after 793.903611 seconds from start

for no apparent reason. The core dumps all point to php-fpm, but I haven't been able to produce a stack trace from them.
Is anyone else seeing the same? I'm reluctant to upgrade further since that would put me in a point of no return...
I'm running this in AWS ECS, running it locally on my Mac (through docker) doesn't show the same behavior at all, but it is also a different platform (ARM64, M1)
EDIT: Actually, the mutex error do appear locally now, might be because I started using redis instead.
EDIT2: my config/app.php
return [
    'id' => App::env('APP_ID') ?: 'CraftCMS',
    'modules' => [
        'cache-purge-module' => \elefant\craft\modules\CachePurgeModule::class,
    ],
    'bootstrap' => ['cache-purge-module'],
    'components' => [
        'log' => function() {
            return Craft::createObject([
                'class' => yii\log\Dispatcher::class,
                'targets' => [
                    [
                        'class' => codemix\streamlog\Target::class,
                        'url' => 'php://stderr',
                        'levels' => Logger::LEVEL_ERROR | Logger::LEVEL_WARNING,
                        'logVars' => [],
                        'except' => [
                            'yii\web\HttpException:404',
                        ],
                    ],

                ]
            ]);
        },
        'session' => function() {
            // Get the default component config
            $config = craft\helpers\App::sessionConfig();

            // Override the class to use DB session class
            $config['class'] = yii\web\DbSession::class;

            // Set the session table name
            $config['sessionTable'] = craft\db\Table::PHPSESSIONS;

            // Instantiate and return it
            return Craft::createObject($config);
        },
        'mutex' => [
            'class' => 'yii\redis\Mutex',
        ],
        'redis' => [
            'class' => 'yii\redis\Connection',
            'hostname' => App::env('REDIS_SERVER'),
        ],
    ],
];

**Tags:** assets, docker, mutex
> **Comments:**
> **username_1**: Hey there! Would you mind sharing your config/app.php? My hunch would be that Craft CMS php processes can't connect to redis, so can't acquire a mutex, and eventually get killed because they run for too long.  
> **username_0**: I'm seeing connections in the metrics for elasticache in AWS, so I assumed it was working.   Re-indexing when saving entries does seem to queue and finish, so maybe it's a red herring...   I'm more concerned over the assets thumbnail problems and the SIGKILL/SIGSEGV  of php-fpm. I just downgraded to my last known working version (3.7.24) locally and I'm still seeing the erratic SIGKILLS.  I can't test the older craftcms docker image either, since they don't keep previous images in docker hub... :(  
> **username_0**: Im starting to believe that it boiled down to memory exhaustion in the end. Someone had uploaded 7-8MB assets and when scaling these it seems to have been hitting the max of 2GB on the ec2-instances, causing all kinds of trouble. Confused how that can even happen.  

---

## 0 answers

