<p>Solution was to put a docker pull into my launch script ... as in</p>

<pre><code>docker-compose -f ${GKE_COMPOSE_YAML} pull
</code></pre>

<p>just before issuing </p>

<pre><code>docker-compose -f ${GKE_COMPOSE_YAML} up -d
</code></pre>

<p>which seems strange since most of the time it works OK without issuing pull ... I am using a local docker registry</p>

CC BY-SA 4.0
2018-06-05T14:29:37.937Z
2018-06-05T14:29:37.937Z
Scott Stensland
<p>I have a devops container inside of which I execute <code>docker build</code> and <code>docker-compose up</code> commands which run OK then (on same server or different server with same code) sometimes incorrectly fail to <code>Recreate</code> sibling containers after their images are rebuilt.</p>

<p>My process successfully rebuilds a pair of two underlying docker images however </p>

<pre><code>docker-compose up 
</code></pre>

<p>sometimes fails to pickup that change and incorrectly says up-to-date on freshly rebuilt images when it should be saying Recreating</p>

<p>I am executing identical code across several servers each with varying OS ( ubuntu 16.04 or debian 8 or debian 9) and varying docker versions ( either  18.04.0-ce-rc1  or  18.03.0-ce ) ... am currently working to track down root cause however see good and bad behavior across each of above OS and docker versions</p>

<p>This issue does not seem related to <a href="https://github.com/docker/compose/issues/3574" rel="nofollow noreferrer">https://github.com/docker/compose/issues/3574</a></p>

<p>Yes I am tagging the images and doing a push to local docker repository</p>

<pre><code>docker build --tag localhost:5000/hygge/loudweb-admin .
</code></pre>

<p>docker push to my local docker repository is </p>

<pre><code>docker push localhost:5000/hygge/loudweb-admin
</code></pre>

<p>it also does similar to above image loudweb-enduser</p>

<p>here my my docker-compose.yaml</p>

<pre><code>version: '3'
services:

nodejs-admin:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-admin
  container_name: loud_admin
  restart: always
  depends_on:
    - loudmongo
    - loudmail
  volumes:
    - /cryptdata4/var/log/loudlog-admin:/loudlog-admin
    - /cryptdata5/var/log/blobs:/blobs
    - /cryptdata5/var:/cryptdata5/var
    - /cryptdata5/var/tools:/tools
    - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
    - $SOURCE_REPO_DIR/tests:/tmp/tests
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle:/tmp
  environment:
    - MONGO_SERVICE_HOST=loudmongo
    - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
    - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
    - METEOR_SETTINGS=${METEOR_SETTINGS}
    - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:blah@loudmail:587/
    - GKE_NOTIF_TASK_OVERDUE=$GKE_NOTIF_TASK_OVERDUE
    - GKE_NOTIF_TASK_PUSH=$GKE_NOTIF_TASK_PUSH
    - GKE_NOTIF_PLANS_RECUR=$GKE_NOTIF_PLANS_RECUR
  links:
    - loudmongo
    - loudmail
  ports:
   - 127.0.0.1:3001:3001
  working_dir: /tmp
  command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

nodejs-enduser:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-enduser
  container_name: loud_enduser
  restart: always
  depends_on:
    - nodejs-admin
    - loudmongo
    - loudmail
  volumes:
    - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
    - /cryptdata5/var/log/blobs:/blobs
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-cert.pem:/private/config/apn-cert.pem
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-key.pem:/private/config/apn-key.pem
  environment:
    - MONGO_SERVICE_HOST=loudmongo
    - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
    - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
    - METEOR_SETTINGS=${METEOR_SETTINGS}
    - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:puMDcxNmEwMDU5MmJjZjNiNzQ3M2ZlNTJjNDYyOGI1NTggIC0Ksy@loudmail:587/
  links:
    - loudmongo
    - loudmail
  ports:
   - 127.0.0.1:3000:3000
  working_dir: /tmp
  command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

loudmongo:
  image: mongo
  container_name: loud_mongo
  restart: always
  ports:
   - 127.0.0.1:$GKE_MONGO_PORT:$GKE_MONGO_PORT
  volumes:
   - /cryptdata7/var/data/db:/data/db

loudmail:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudmail
  hostname: mail
  domainname: ${GKE_DOMAIN_NAME}
  container_name: loud_mail
  restart: always
  environment:
    - DMS_DEBUG=1
    - ENABLE_MANAGESIEVE=1
    - ENABLE_FAIL2BAN=1
    - SA_TAG=2.0
    - SA_TAG2=6.31
    - SA_KILL=6.31
    - SSL_TYPE=letsencrypt
  ports:
    - 127.0.0.1:25:25
    - 127.0.0.1:110:110
    - 127.0.0.1:143:143
    - 127.0.0.1:587:587
    - 127.0.0.1:993:993
    - 127.0.0.1:995:995
    - 127.0.0.1:4190:4190


loud-devops:
  image: dind
  container_name: loud_devops
  restart: always
  ports:
    - 127.0.0.1:9000:9000
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock   
    - /usr/bin/docker:/usr/bin/docker   
    - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7  
    - /usr/lib/x86_64-linux-gnu/libgpm.so.2:/usr/lib/x86_64-linux-gnu/libgpm.so.2
    - /home/asya/src/github.com/blahblah:/inner_home/asya/src/github.com/blahblah 
    - /home/asya/.docker:/inner_home/asya/.docker
    - /cryptdata5/var/tools/inner_home/asya:/inner_home/asya
    - $GKE_DIND_SUPERVISOR_LOG_DIR:/var/log/supervisor
    - /cryptdata5/var/tools/usr/local/go:/usr/local/go
    - /cryptdata6/var/log/tmp/asya01:/cryptdata6/var/log/tmp/asya01
    - /cryptdata6/var/log/tmp/shared:/cryptdata6/var/log/tmp/shared
    - /cryptdata5/var/tools/usr/local/bin:/usr/local/bin 
    - /cryptdata:/cryptdata
    - /cryptdata4:/cryptdata4
    - /cryptdata5:/cryptdata5
    - /cryptdata6:/cryptdata6
    - /cryptdata7:/cryptdata7
    - /etc/letsencrypt/live:/etc/letsencrypt/live
    - /usr/local/ssl:/usr/local/ssl
    - /var/lib/docker:/var/lib/docker
  command: /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
</code></pre>

<p>The bottom most container in above ( loud-devops ) contains a webhook daemon which listens for a git push to trigger a rebuild cycle. While inside that container the system issues all my docker command including docker build create fresh images for two containers ( nodejs-admin and nodejs-enduser ) then while inside the container I issue the </p>

<pre><code>docker-compose -f /docker-compose.yaml up -d 
</code></pre>

<p>which then sometimes incorrectly displays</p>

<pre><code>loud_devops is up-to-date
loud_mongo is up-to-date
loud_mail is up-to-date
loud_admin is up-to-date
loud_enduser is up-to-date
</code></pre>

<p>other times it incorrectly displays</p>

<pre><code>docker-compose -f /docker-compose.yml up -d
Recreating loud_devops ...
Recreating loud_mongo ...
Recreating loud_mail ...
</code></pre>

<p>which should NEVER happen since those 3 images are never rebuilt ... to me that scenario is a blatant docker logic bug ... </p>

<p>whereas other times works OK and correctly displays</p>

<pre><code>loud_mongo is up-to-date
loud_mail is up-to-date
loud_devops is up-to-date
Recreating loud_admin ... done
Recreating loud_enduser ... done
</code></pre>

<p>On the host of each server I have docker installed then as you can see in above yaml for container loud-devops  I mount as volumes various docker dirs to enable the execution of docker in docker from inside container loud-devops </p>

<p>A critical piece of evidence :  on a server which has finished its rebuild cycle where it successfully rebuilt both images loudweb-enduser and loudweb-admin  then incorrectly shows</p>

<pre><code>docker-compose -f /docker-compose.yml up -d 
loud_mail is up-to-date
loud_mongo is up-to-date
loud_devops is up-to-date
loud_admin is up-to-date
loud_enduser is up-to-date
</code></pre>

<p>where the <code>docker-compose up</code> command gets executed by the system while inside container  loud-devops ... IF I then manually login to that same server then manually issue the same command it works correctly :</p>

<pre><code>docker-compose -f /docker-compose.yml up -d 
loud_devops is up-to-date
loud_mail is up-to-date
loud_mongo is up-to-date
Recreating loud_admin ... done
Recreating loud_enduser ... done
</code></pre>

CC BY-SA 3.0
2018-04-04T12:45:17.273Z
Do you tag the built images? this might happen if you build & tag an image with the tag which already exists = rewrite the previous version, then when you do docker-compose up it won't recreate because the version is the same. I would like to know if --force-rebuild and/or --always-recreate-deps solves the issue and I would also like to see the docker-compose file, I might be able to advice further afterwards
Tomas Tibensky
CC BY-SA 3.0
2018-04-04T17:11:16.043Z
@TomášTibenský  to respond to your questions - the rebuild logic is outside the docker-compose.yaml so I have never used nor believe I can use --force-rebuild or --always-recreate-deps
Scott Stensland
CC BY-SA 3.0
2018-04-06T11:38:45.447Z
what about running docker-compose -f /docker-compose.yml pull and then docker-compose -f /docker-compose.yml up -d ?
Tomas Tibensky
<docker><docker-compose><docker-in-docker>
docker-compose fails to recognize when images are rebuilt sometimes ... often same code is OK
Scott Stensland
# docker-compose fails to recognize when images are rebuilt sometimes ... often same code is OK

(Asked by: username_0 on 2018-03-29)

I have a devops container inside of which I execute docker build and docker-compose up commands which run OK then (on same server or different server with same code) sometimes incorrectly fail to Recreate sibling containers after their images are rebuilt.
My process successfully rebuilds a pair of two underlying docker images however 
docker-compose up 

sometimes fails to pickup that change and incorrectly says up-to-date on freshly rebuilt images when it should be saying Recreating
I am executing identical code across several servers each with varying OS ( ubuntu 16.04 or debian 8 or debian 9) and varying docker versions ( either  18.04.0-ce-rc1  or  18.03.0-ce ) ... am currently working to track down root cause however see good and bad behavior across each of above OS and docker versions
This issue does not seem related to https://github.com/docker/compose/issues/3574
Yes I am tagging the images and doing a push to local docker repository
docker build --tag localhost:5000/hygge/loudweb-admin .

docker push to my local docker repository is 
docker push localhost:5000/hygge/loudweb-admin

it also does similar to above image loudweb-enduser
here my my docker-compose.yaml
version: '3'
services:

nodejs-admin:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-admin
  container_name: loud_admin
  restart: always
  depends_on:
    - loudmongo
    - loudmail
  volumes:
    - /cryptdata4/var/log/loudlog-admin:/loudlog-admin
    - /cryptdata5/var/log/blobs:/blobs
    - /cryptdata5/var:/cryptdata5/var
    - /cryptdata5/var/tools:/tools
    - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
    - $SOURCE_REPO_DIR/tests:/tmp/tests
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle:/tmp
  environment:
    - MONGO_SERVICE_HOST=loudmongo
    - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
    - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
    - METEOR_SETTINGS=${METEOR_SETTINGS}
    - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:blah@loudmail:587/
    - GKE_NOTIF_TASK_OVERDUE=$GKE_NOTIF_TASK_OVERDUE
    - GKE_NOTIF_TASK_PUSH=$GKE_NOTIF_TASK_PUSH
    - GKE_NOTIF_PLANS_RECUR=$GKE_NOTIF_PLANS_RECUR
  links:
    - loudmongo
    - loudmail
  ports:
   - 127.0.0.1:3001:3001
  working_dir: /tmp
  command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

nodejs-enduser:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-enduser
  container_name: loud_enduser
  restart: always
  depends_on:
    - nodejs-admin
    - loudmongo
    - loudmail
  volumes:
    - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
    - /cryptdata5/var/log/blobs:/blobs
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-cert.pem:/private/config/apn-cert.pem
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-key.pem:/private/config/apn-key.pem
  environment:
    - MONGO_SERVICE_HOST=loudmongo
    - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
    - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
    - METEOR_SETTINGS=${METEOR_SETTINGS}
    - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:puMDcxNmEwMDU5MmJjZjNiNzQ3M2ZlNTJjNDYyOGI1NTggIC0Ksy@loudmail:587/
  links:
    - loudmongo
    - loudmail
  ports:
   - 127.0.0.1:3000:3000
  working_dir: /tmp
  command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

loudmongo:
  image: mongo
  container_name: loud_mongo
  restart: always
  ports:
   - 127.0.0.1:$GKE_MONGO_PORT:$GKE_MONGO_PORT
  volumes:
   - /cryptdata7/var/data/db:/data/db

loudmail:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudmail
  hostname: mail
  domainname: ${GKE_DOMAIN_NAME}
  container_name: loud_mail
  restart: always
  environment:
    - DMS_DEBUG=1
    - ENABLE_MANAGESIEVE=1
    - ENABLE_FAIL2BAN=1
    - SA_TAG=2.0
    - SA_TAG2=6.31
    - SA_KILL=6.31
    - SSL_TYPE=letsencrypt
  ports:
    - 127.0.0.1:25:25
    - 127.0.0.1:110:110
    - 127.0.0.1:143:143
    - 127.0.0.1:587:587
    - 127.0.0.1:993:993
    - 127.0.0.1:995:995
    - 127.0.0.1:4190:4190


loud-devops:
  image: dind
  container_name: loud_devops
  restart: always
  ports:
    - 127.0.0.1:9000:9000
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock   
    - /usr/bin/docker:/usr/bin/docker   
    - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7  
    - /usr/lib/x86_64-linux-gnu/libgpm.so.2:/usr/lib/x86_64-linux-gnu/libgpm.so.2
    - /home/asya/src/github.com/blahblah:/inner_home/asya/src/github.com/blahblah 
    - /home/asya/.docker:/inner_home/asya/.docker
    - /cryptdata5/var/tools/inner_home/asya:/inner_home/asya
    - $GKE_DIND_SUPERVISOR_LOG_DIR:/var/log/supervisor
    - /cryptdata5/var/tools/usr/local/go:/usr/local/go
    - /cryptdata6/var/log/tmp/asya01:/cryptdata6/var/log/tmp/asya01
    - /cryptdata6/var/log/tmp/shared:/cryptdata6/var/log/tmp/shared
    - /cryptdata5/var/tools/usr/local/bin:/usr/local/bin 
    - /cryptdata:/cryptdata
    - /cryptdata4:/cryptdata4
    - /cryptdata5:/cryptdata5
    - /cryptdata6:/cryptdata6
    - /cryptdata7:/cryptdata7
    - /etc/letsencrypt/live:/etc/letsencrypt/live
    - /usr/local/ssl:/usr/local/ssl
    - /var/lib/docker:/var/lib/docker
  command: /usr/bin/supervisord -c /etc/supervisor/supervisord.conf

The bottom most container in above ( loud-devops ) contains a webhook daemon which listens for a git push to trigger a rebuild cycle. While inside that container the system issues all my docker command including docker build create fresh images for two containers ( nodejs-admin and nodejs-enduser ) then while inside the container I issue the 
docker-compose -f /docker-compose.yaml up -d 

which then sometimes incorrectly displays
loud_devops is up-to-date
loud_mongo is up-to-date
loud_mail is up-to-date
loud_admin is up-to-date
loud_enduser is up-to-date

other times it incorrectly displays
docker-compose -f /docker-compose.yml up -d
Recreating loud_devops ...
Recreating loud_mongo ...
Recreating loud_mail ...

which should NEVER happen since those 3 images are never rebuilt ... to me that scenario is a blatant docker logic bug ... 
whereas other times works OK and correctly displays
loud_mongo is up-to-date
loud_mail is up-to-date
loud_devops is up-to-date
Recreating loud_admin ... done
Recreating loud_enduser ... done

On the host of each server I have docker installed then as you can see in above yaml for container loud-devops  I mount as volumes various docker dirs to enable the execution of docker in docker from inside container loud-devops 
A critical piece of evidence :  on a server which has finished its rebuild cycle where it successfully rebuilt both images loudweb-enduser and loudweb-admin  then incorrectly shows
docker-compose -f /docker-compose.yml up -d 
loud_mail is up-to-date
loud_mongo is up-to-date
loud_devops is up-to-date
loud_admin is up-to-date
loud_enduser is up-to-date

where the docker-compose up command gets executed by the system while inside container  loud-devops ... IF I then manually login to that same server then manually issue the same command it works correctly :
docker-compose -f /docker-compose.yml up -d 
loud_devops is up-to-date
loud_mail is up-to-date
loud_mongo is up-to-date
Recreating loud_admin ... done
Recreating loud_enduser ... done

**Tags:** docker, docker-compose, docker-in-docker

> **Comments:**
> **username_1**: Do you tag the built images? this might happen if you build & tag an image with the tag which already exists = rewrite the previous version, then when you do docker-compose up it won't recreate because the version is the same. I would like to know if --force-rebuild and/or --always-recreate-deps solves the issue and I would also like to see the docker-compose file, I might be able to advice further afterwards  
> **username_0**: @TomášTibenský  to respond to your questions - the rebuild logic is outside the docker-compose.yaml so I have never used nor believe I can use --force-rebuild or --always-recreate-deps  
> **username_1**: what about running docker-compose -f /docker-compose.yml pull and then docker-compose -f /docker-compose.yml up -d ?  

---

## 1 answers

### Answer by username_0 on 2018-06-05. Score: 0 | Accepted answer

Solution was to put a docker pull into my launch script ... as in
docker-compose -f ${GKE_COMPOSE_YAML} pull

just before issuing 
docker-compose -f ${GKE_COMPOSE_YAML} up -d

which seems strange since most of the time it works OK without issuing pull ... I am using a local docker registry


---

