<pre><code>| Product    | Version |
|:-----------|--------:|
| EC2 plugin |       X |
| Jenkins    | 2.190.1 |
</code></pre>

<p>Looking for help troubleshooting an intermittent issue;</p>

<p>We run an Ubuntu version of Jenkins on AWS infrastructure using on-demand spot ECS instances. I've noticed that while most agents spin down and correctly terminate their ECS counterparts, occasionally an agent will quit without killing its instance, leaving orphaned instances that it could take us weeks to notice. </p>

<p>This problem has been happening on and off for a year, through many different versions of Jenkins. I can't find a common thread between the orphaned instances, and would appreciate any insight on what I can troubleshoot to find the cause. </p>

<p><strong>Can you clarify how you're terminating the instances?</strong></p>

<p>Nodes are automatically terminated x minutes after the job they're running finishes, presumably taking the instance down with them. I'm not clear on the actual particulars; it's just a setting in Jenkins for "Idle termination time". </p>

CC BY-SA 3.0
2018-03-27T16:11:01.737Z
Can you check the jenkins log? Is it showing any errors when trying to execute the terminate/shutdown command? I'm assuming the plugin outputs to a log file somewhere.
Preston Martin
CC BY-SA 3.0
2018-03-28T13:46:13.920Z
@PrestonM I can't find a specific error right now, but that's a good idea of what to check next time I spot one getting stuck, thanks!
Alex
CC BY-SA 4.0
2019-12-23T14:11:24.993Z
@030 never found a solution ¯\_(ツ)_/¯ it's still plaguing us maybe 1/500 jobs
Alex
CC BY-SA 4.0
2019-12-23T14:26:05.213Z
Could you check the jenkins master, slave, cloudtrail and cloudwatch logs? Did you check whether there is a related issue on [Jenkins' Jira](https://issues.jenkins-ci.org/browse/JENKINS-60482?jql=project%20%3D%20JENKINS%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20%3D%20ec2-plugin)?
030
CC BY-SA 4.0
2019-12-23T14:43:57.643Z
Could you indicate whether the jenkins master is often restarted if there are still VMs running that were launched by the jenkins master?
030
<jenkins><amazon-web-services><amazon-ecs><jenkins-amazon-ec2-plugin>
Why does Jenkins' Amazon EC2 Plugin not terminate some VMs when they are idle for X minutes on AWS US-east-1?
Alex
# Why does Jenkins' Amazon EC2 Plugin not terminate some VMs when they are idle for X minutes on AWS US-east-1?

(Asked by: username_0 on 2018-03-27)

| Product    | Version |
|:-----------|--------:|
| EC2 plugin |       X |
| Jenkins    | 2.190.1 |

Looking for help troubleshooting an intermittent issue;
We run an Ubuntu version of Jenkins on AWS infrastructure using on-demand spot ECS instances. I've noticed that while most agents spin down and correctly terminate their ECS counterparts, occasionally an agent will quit without killing its instance, leaving orphaned instances that it could take us weeks to notice. 
This problem has been happening on and off for a year, through many different versions of Jenkins. I can't find a common thread between the orphaned instances, and would appreciate any insight on what I can troubleshoot to find the cause. 
Can you clarify how you're terminating the instances?
Nodes are automatically terminated x minutes after the job they're running finishes, presumably taking the instance down with them. I'm not clear on the actual particulars; it's just a setting in Jenkins for "Idle termination time".

**Tags:** jenkins, amazon-web-services, amazon-ecs, jenkins-amazon-ec2-plugin

> **Comments:**
> **username_1**: Can you check the jenkins log? Is it showing any errors when trying to execute the terminate/shutdown command? I'm assuming the plugin outputs to a log file somewhere.  
> **username_0**: @PrestonM I can't find a specific error right now, but that's a good idea of what to check next time I spot one getting stuck, thanks!  
> **username_0**: @username_2 never found a solution ¯\_(ツ)_/¯ it's still plaguing us maybe 1/500 jobs  
> **username_2**: Could you check the jenkins master, slave, cloudtrail and cloudwatch logs? Did you check whether there is a related issue on [Jenkins' Jira](https://issues.jenkins-ci.org/browse/JENKINS-60482?jql=project%20%3D%20JENKINS%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20%3D%20ec2-plugin)?  
> **username_2**: Could you indicate whether the jenkins master is often restarted if there are still VMs running that were launched by the jenkins master?  

---

## 0 answers

