<p>I've created a couple crude methods of grabbing the latest major versions of releases, as well as absolute latest.</p>

<pre><code># Grab Latest Tomcat 7
curl -i https://www-us.apache.org/dist/tomcat/tomcat-7/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'

# Grab Latest Tomcat 8
curl -i https://www-us.apache.org/dist/tomcat/tomcat-8/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'

# Grab Latest Tomcat 9
curl -i https://www-us.apache.org/dist/tomcat/tomcat-9/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'

# Grab Latest Tomcat
curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"
</code></pre>

<p>It's a bit of a hack where I'm looking for text in-between two values on a web page to grab the version number. It's crude and I'm sure there's a more elegant way to grab this, but this what I was able to figure out with my skillset! Maybe it can help someone in the future, but if anyone has better ideas I'm all ears.</p>

CC BY-SA 4.0
2019-03-21T21:13:24.143Z
2019-03-21T21:13:24.143Z
Keegan Jacobson
<p>In general, I would not recommend scraping version numbers and download URLs from websites.  You will end up having to modify your scripts when they inevitably end up breaking — I have learned this the hard way time and time again.</p>
<p>For example, take the following script I wrote in 2016 to scrape the latest Tomcat version:</p>
<pre class="lang-sh prettyprint-override"><code>curl https://tomcat.apache.org/whichversion.html | awk -n '
/class=&quot;detail-table&quot;/ {
    phase = 1
}
phase == 1 &amp;&amp; /&lt;th&gt;/ {
    target_column++
    if (/Actual release revision/) phase = 2
}
phase == 2 &amp;&amp; /&lt;tr&gt;/ {
    current_column = 0
}
phase == 2 &amp;&amp; /&lt;td&gt;/ {
    current_column++
}
phase == 2 &amp;&amp; current_column == target_column &amp;&amp; !/alpha|beta|superceded/ {
    print substr($0, match($0, /[0-9]+\.[0-9]+\.[0-9]+/), RLENGTH)
    exit
}'
</code></pre>
<p>Unfortunately, that script <strong>no longer works</strong>, and I do not know when it broke.  No amount of fancy scripting will save you from unexpected structural changes to a website you are scraping.</p>
<p>Sure, maybe a little bit of finagling will fix this one particular script, but if you try to download all your dependencies this way, you'll be playing catch-up till the end of time.  Aim for a solution that you only need to code once.</p>
<p><strong>EDIT:</strong> There's a good chance that I've been successfully nerd-sniped by @Bill_Stewart, but a working solution today (October 18th, 2024) is to simply replace <code>/Actual release revision/</code> with <code>/Latest Released Version/</code> in the above <code>awk</code> script. My original sentiments still stand and I would never use this in production unless I literally had no other choice.</p>

CC BY-SA 4.0
2024-10-17T13:52:16.750Z
This is sound advice, but these comments don't constitute an actual answer to the question.
Bill_Stewart
CC BY-SA 4.0
2019-04-01T18:09:24.193Z
2024-10-18T23:48:51.777Z
2024-10-18T23:48:51.777Z
eeowaa
<p>Looking for some assistance or ideas on how to grab the latest version of a Tomcat release from their website or other website.</p>

<p>I came across these Stack Overflow/Stack Exchange links that were relevant below:</p>

<ul>
<li><a href="https://stackoverflow.com/questions/22510705/get-the-latest-download-link-programmatically">https://stackoverflow.com/questions/22510705/get-the-latest-download-link-programmatically</a> (Nginx, but same idea)</li>
<li><a href="https://devops.stackexchange.com/questions/3262/how-to-get-the-latest-tomcat-version">How to get the latest Tomcat version?</a></li>
</ul>

<p>..But unfortunately both led to some dead-ends. It did give me some ideas though.</p>

<p>I have some installation scripts where the download URL is predictable based on the version number of the software, so if there's a consistent way of scraping for the latest version of the software I simply add a variable into the script for it to curl / grep for that version on the text and use that variable for download links, untaring, moving, etc.</p>

<p>Hoping to do something with Tomcat in the same vein. A installation script I won't need to keep updating (unless they change their site).</p>

<p>Any sort of thoughts, ideas, are appreciated.</p>

<p>I've also looked at <a href="http://tomcat.apache.org/whichversion.html" rel="nofollow noreferrer">http://tomcat.apache.org/whichversion.html</a> which lists the "Latest Released Version" and this seems like maybe the best, more reliable location to grab the version, but I had trouble getting that with Curl / Grep because of the table structure. I am only just scratching the surface with learning those commands, so maybe someone more well versed could get that working, otherwise I'm definitely open to other thoughts!</p>

<pre><code>curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"
</code></pre>

<p>It seems to work but I am not sure if I am entirely happy with it. An additional use-case that I thought of, is it would be nice to grab the latest version of each major version instead of just the absolute latest.</p>

<p>Do you have suggestions on how you would parse the highest number with the link you provided, and/or how to start in one of those directories and parse the highest there? That way for example I could create a latest version install script for Tomcat 7, 8, 9, etc.</p>

<p>Thank you!! (Like I said I am just getting started, so please my apologies with the lack of knowledge but I do appreciate digging in and understanding how the commands work or reached their conclusion)</p>

CC BY-SA 4.0
2019-03-21T15:01:05.667Z
You should not upgrade tomcat without prior reading of the changelog, even minor security fix may break your application.
Tensibai
CC BY-SA 4.0
2019-03-21T15:06:24.047Z
Hi @Tensibai thanks for the advise. The goal here is to create a version agnostic installation script that will always grab the latest version. So that when we do approve the latest version, I can simply run the script that will already be present on the machines or systems.

Assume here that we already taking the necessary precautions surrounding the version, and just care to have a script to grab the latest for a version (Tomcat 7, Tomcat 8, Tomcat 9, etc).

Appreciate your concern and insight!
Keegan Jacobson
CC BY-SA 4.0
2019-03-21T15:44:16.040Z
Best course of action would be to pass as a parameter the version to install, specially as there's no "latest" tomcat, but 3 latest depending on major version, so taking the latest by date may be 7, 8 or 9 and you'll have to at least pass this major as parameter, better pass the whole desired version directly.
Tensibai
CC BY-SA 4.0
2019-03-21T20:19:18.737Z
What I'm actually going to do is have three different installation scripts, one for 7, 8, and 9 (and then in the future one for each of the future versions) but I want to be able to grab the latest release for a particular major version.
Keegan Jacobson
CC BY-SA 4.0
2019-03-22T08:45:32.303Z
And you'll have 3 scripts to maintain after that, that's just a bad idea.
Tensibai
CC BY-SA 4.0
2019-03-22T16:27:52.273Z
@Tensibai I'm not sure how to pass a parameter so this is probably the best I'm going to get otherwise. I'm open to suggestions if you know how to do that, it seems more like a preference than a bad idea to me though.
Keegan Jacobson
CC BY-SA 4.0
2019-03-22T21:40:45.313Z
OK, you need a sheel tutorial then, exemple: `./command parameter1` in the script named command, parameter1 will be available by using the variable $1, just pass the version. And no, it's not a preference, but hopefully your future self will remember this exchange when a bunch of nearly identical scripts will all fail at the same time and you'll understand why Dry (Don't repeat yourself) is something to do while coding.
Tensibai
CC BY-SA 4.0
2019-03-22T22:12:52.220Z
Thank you for the reply! Ultimately I'm trying to solve the solution at hand, and also understand if there's a better or more efficient way to solve problems. I do see the benefit of that for a lot of scenarios not just this one. I'll need to do some research and testing at a shell when I get an opportunity.
Keegan Jacobson
<shell-script><tomcat>
Programmatically scrape the latest version of Tomcat for an installation script
Keegan Jacobson
# Programmatically scrape the latest version of Tomcat for an installation script

(Asked by: username_0 on 2019-03-21)

Looking for some assistance or ideas on how to grab the latest version of a Tomcat release from their website or other website.
I came across these Stack Overflow/Stack Exchange links that were relevant below:

https://stackoverflow.com/questions/22510705/get-the-latest-download-link-programmatically (Nginx, but same idea)
How to get the latest Tomcat version?

..But unfortunately both led to some dead-ends. It did give me some ideas though.
I have some installation scripts where the download URL is predictable based on the version number of the software, so if there's a consistent way of scraping for the latest version of the software I simply add a variable into the script for it to curl / grep for that version on the text and use that variable for download links, untaring, moving, etc.
Hoping to do something with Tomcat in the same vein. A installation script I won't need to keep updating (unless they change their site).
Any sort of thoughts, ideas, are appreciated.
I've also looked at http://tomcat.apache.org/whichversion.html which lists the "Latest Released Version" and this seems like maybe the best, more reliable location to grab the version, but I had trouble getting that with Curl / Grep because of the table structure. I am only just scratching the surface with learning those commands, so maybe someone more well versed could get that working, otherwise I'm definitely open to other thoughts!
curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"

It seems to work but I am not sure if I am entirely happy with it. An additional use-case that I thought of, is it would be nice to grab the latest version of each major version instead of just the absolute latest.
Do you have suggestions on how you would parse the highest number with the link you provided, and/or how to start in one of those directories and parse the highest there? That way for example I could create a latest version install script for Tomcat 7, 8, 9, etc.
Thank you!! (Like I said I am just getting started, so please my apologies with the lack of knowledge but I do appreciate digging in and understanding how the commands work or reached their conclusion)

**Tags:** shell-script, tomcat

> **Comments:**
> **username_1**: You should not upgrade tomcat without prior reading of the changelog, even minor security fix may break your application.  
> **username_0**: Hi @username_1 thanks for the advise. The goal here is to create a version agnostic installation script that will always grab the latest version. So that when we do approve the latest version, I can simply run the script that will already be present on the machines or systems.  Assume here that we already taking the necessary precautions surrounding the version, and just care to have a script to grab the latest for a version (Tomcat 7, Tomcat 8, Tomcat 9, etc).  Appreciate your concern and insight!  
> **username_1**: Best course of action would be to pass as a parameter the version to install, specially as there's no "latest" tomcat, but 3 latest depending on major version, so taking the latest by date may be 7, 8 or 9 and you'll have to at least pass this major as parameter, better pass the whole desired version directly.  
> **username_0**: What I'm actually going to do is have three different installation scripts, one for 7, 8, and 9 (and then in the future one for each of the future versions) but I want to be able to grab the latest release for a particular major version.  
> **username_1**: And you'll have 3 scripts to maintain after that, that's just a bad idea.  
> **username_0**: @username_1 I'm not sure how to pass a parameter so this is probably the best I'm going to get otherwise. I'm open to suggestions if you know how to do that, it seems more like a preference than a bad idea to me though.  
> **username_1**: OK, you need a sheel tutorial then, exemple: `./command parameter1` in the script named command, parameter1 will be available by using the variable $1, just pass the version. And no, it's not a preference, but hopefully your future self will remember this exchange when a bunch of nearly identical scripts will all fail at the same time and you'll understand why Dry (Don't repeat yourself) is something to do while coding.  
> **username_0**: Thank you for the reply! Ultimately I'm trying to solve the solution at hand, and also understand if there's a better or more efficient way to solve problems. I do see the benefit of that for a lot of scenarios not just this one. I'll need to do some research and testing at a shell when I get an opportunity.  

---

## 2 answers

### Answer by username_0 on 2019-03-21. Score: 0 | Accepted answer

I've created a couple crude methods of grabbing the latest major versions of releases, as well as absolute latest.
# Grab Latest Tomcat 7
curl -i https://www-us.apache.org/dist/tomcat/tomcat-7/ | grep -Po '(?<=(v)'

# Grab Latest Tomcat 8
curl -i https://www-us.apache.org/dist/tomcat/tomcat-8/ | grep -Po '(?<=(v)'

# Grab Latest Tomcat 9
curl -i https://www-us.apache.org/dist/tomcat/tomcat-9/ | grep -Po '(?<=(v)'

# Grab Latest Tomcat
curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"

It's a bit of a hack where I'm looking for text in-between two values on a web page to grab the version number. It's crude and I'm sure there's a more elegant way to grab this, but this what I was able to figure out with my skillset! Maybe it can help someone in the future, but if anyone has better ideas I'm all ears.


---

### Answer by username_2 on 2019-04-01. Score: 0

In general, I would not recommend scraping version numbers and download URLs from websites.  You will end up having to modify your scripts when they inevitably end up breaking — I have learned this the hard way time and time again.
For example, take the following script I wrote in 2016 to scrape the latest Tomcat version:
curl https://tomcat.apache.org/whichversion.html | awk -n '
/class="detail-table"/ {
    phase = 1
}
phase == 1 && // {
    target_column++
    if (/Actual release revision/) phase = 2
}
phase == 2 && // {
    current_column = 0
}
phase == 2 && // {
    current_column++
}
phase == 2 && current_column == target_column && !/alpha|beta|superceded/ {
    print substr($0, match($0, /[0-9]+\.[0-9]+\.[0-9]+/), RLENGTH)
    exit
}'

Unfortunately, that script no longer works, and I do not know when it broke.  No amount of fancy scripting will save you from unexpected structural changes to a website you are scraping.
Sure, maybe a little bit of finagling will fix this one particular script, but if you try to download all your dependencies this way, you'll be playing catch-up till the end of time.  Aim for a solution that you only need to code once.
EDIT: There's a good chance that I've been successfully nerd-sniped by @username_3, but a working solution today (October 18th, 2024) is to simply replace /Actual release revision/ with /Latest Released Version/ in the above awk script. My original sentiments still stand and I would never use this in production unless I literally had no other choice.

> **Comments:**
> **username_3**: This is sound advice, but these comments don't constitute an actual answer to the question.  

---

