<p>This question is a little vague, with multiple aspects, but I shall do my best.</p>
<h2>Computing resources</h2>
<p>Some universities have their own clusters or supercomputers. But many countries also have regional or national facilities. For example, <a href="https://www.archer2.ac.uk/" rel="nofollow noreferrer">ARCHER2</a> is the latest iteration of the UK's national supercomputer service. Small amounts of computing time are available for free to UK academics, but for large projects they must bid for it.</p>
<p>It's rare for simulations to require something of the scale of ARCHER or ARCHER2; depending on the model being run, often a departmental or institutional system of just a few nodes may be adequate, especially if one isn't in a hurry.</p>
<h2>Building code for different systems</h2>
<p>So long as the code has been well-written, this is not as hard as it sounds. FORTRAN is reasonably well standardised (though some codes will require the use of a particular compiler), and MPI - despite its varied implementations - is too. And most modern supercomputers are massively parallel machines whose actual compute nodes are standard x64 servers, using CPUs such as Intel Xeons. So you can do a lot of development on a desktop PC before you try to build it somewhere big and expensive. The biggest differences between different systems tend to lie in the interconnectors (the high-speed, low-latency networking between nodes), but usually the system admins will have optimised an MPI installation for that system and made it available to users, so all the details of inter-process and inter-node communications are abstracted.</p>
<h2>Making changes or improvements</h2>
<p>As you've identified, for most people the key is to improve one small aspect at a time, such as a particular parametrisation. Once somebody has the standard code working, then if it is well written in a maintainable manner (not a given in the scientific world) then it is relatively straightforward for them to modify one aspect and test it. The hardest thing often is obtaining measurements or other data against which to validate the new version.</p>

CC BY-SA 4.0
2021-09-17T16:16:36.207Z
Yes, this makes a lot of sense. As a non-specialist in climate models, your answer clarifies a lot. It seem like there is a lot of compartmentalization of these different modelling levels. So a person can innovate on a parameterization model or develop a new model on their computer. Once that works, they can use their local cluster for larger scale simulation (perhaps at finer resolution). If the model is useful, then someone can write the code for ARCHER2, etc.
krishnab
CC BY-SA 4.0
2021-09-17T16:35:54.930Z
@krishnab sort of, but also it is often possible to run the SAME CODE on all those computers. Just faster or slower. Remember that these days, clusters and supercomputers are basically the same as your desktop computer, repeated many times.
Semidiurnal Simon
CC BY-SA 4.0
2021-09-18T11:14:56.503Z
Good answer & I know this isn't a discussion site, ... but, as a supplementary question, is distributed computing used for serious climate modeling?
Fred
CC BY-SA 4.0
2021-09-18T14:28:36.890Z
@fred by distributed computing, you mean computers in different locations working together? Not so far as I know, but I do not claim complete knowledge of the climate modelling space. In the types of models I'm familiar with - which, again, is not exhaustive - latency between nodes is important to performance.
Semidiurnal Simon
CC BY-SA 4.0
2021-09-18T15:10:36.807Z
@Fred I think distributed computing usually refers to a cluster right, meaning a bunch of servers that are connected across a local network, and which breakup a large compute job into pieces. So it seems the answer is yes. Architecture for this can vary, but seems like someone was using the Amazon cloud for running climate models. I am not sure about distributing server over longer geographic distances--the latency would be a killer as Simon indicated.
krishnab
CC BY-SA 4.0
2021-09-18T15:45:01.470Z
@krishnab: [Distributed computing](https://www.ibm.com/docs/en/txseries/8.2?topic=overview-what-is-distributed-computing), computers are that "can be physically close together and connected by a local network, or they can be geographically distant and connected by a wide area network". I was thinking of something similar to [Seti@home or Albert@home](https://en.wikipedia.org/wiki/List_of_distributed_computing_projects). I know one of the newer radio astronomy installations in Western Australia, within the last few years, used this type of distributed computing to get its data analyzed. ...
Fred
CC BY-SA 4.0
2021-09-18T15:47:54.177Z
... one of the disadvantages of using such a system is the results from each packet of data needs to be verified a number of times, which tends to delay the final result. For time critical analyses it wouldn't be used. I don't know how time critical some climate modeling would be & whether distributed computing would be appropriate.
Fred
CC BY-SA 4.0
2021-09-18T15:51:48.833Z
Having stated what I just did, I have just noticed, from the listed of distributed computing projects in the link in my previous comment, Oxford University has a [climate predicting](https://en.wikipedia.org/wiki/Climateprediction.net) application via BOINC distributed computing platform.
Fred
CC BY-SA 4.0
2021-09-18T17:26:43.313Z
@Fred oh right, I get what you mean. It reminds me of those protein folding applications that I used to run as my screen saver. I think it really depends upon the scale of the computing. For a climate model, the scale of the grids/meshes are adjusted to be computed in a certain amount of time and within certain memory requirements. But if your model is really just much larger than the available resources you can access, then something distributed makes sense. But you are correct, it is not clear how long it would take to run.
krishnab
CC BY-SA 4.0
2021-09-18T17:33:00.997Z
But I agree that running these models in wider distributed fashion might become necessary--unless quantum computing solves the challenge. Climate models are fine, but if we want to solve a large number of climate models as an input to other models of interest, then compute is the big issue. I was interested in looking at flooding prediction based on hurricanes. So I would need to simulate hundreds of hurricanes and trajectories (big compute task), and then model the resulting storm surge and flooding hundreds of times. So 2 big compute tasks.
krishnab
CC BY-SA 4.0
2021-09-17T06:11:24.480Z
2021-09-18T11:11:59.397Z
2021-09-18T11:11:59.397Z
Semidiurnal Simon
<p>Another UK perspective here to supplement Semidiurnal Simon’s answer (which reflects my experience too).</p>
<p>The UK research community is dominated by a single family of models known collectively as the Unified Model.  The code for these is owned by the UK's national weather and climate modelling center, the Met Office, and used gratis under license by academics.  That license gives academics access to repositories of model source code and experiment setups shared across the MO and academic community.  Simon mentioned ARCHER2, which is shared by the academics across all disciplines, but the Met Office also provides <a href="https://www.metoffice.gov.uk/research/approach/collaboration/jwcrp/monsoon-hpc" rel="nofollow noreferrer">Monsoon</a> specifically for atmospheric modelling collaborations with the academic community.</p>
<p>Because the MO is an operational forecast center, they put a <em>lot</em> of effort into making sure the code and support software run efficiently and reliably, so what academics have access to is pretty battle hardened.  But the academic side also has <a href="https://cms-helpdesk.ncas.ac.uk" rel="nofollow noreferrer">NCAS-CMS</a>, whose job it is to make sure the model works on machines like ARCHER2 for the whole community.  All in all there’s a good level of national support for this model on these machines, and when I send a student on the training course for the model they can be running climate simulations on that hardware within 30 minutes of arriving.</p>
<blockquote>
<p>something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough</p>
</blockquote>
<p>Well, the UM has about 1.3 million lines of code and I’d estimate that I know about 15% of that code very very well (mainly a particular area of science) and the rest of it barely at all.  When I encounter bugs they’re almost always because of the thing that I’ve just changed or something closely related in areas of the code I know well.  When bugs lead into other parts of the model, it’s usually best to go and ask someone who knows about those areas rather than digging too hard yourself.</p>
<blockquote>
<p>Writing and running an individual component model or &quot;parameterization&quot; model for one of these larger models seems manageable... But then how can that same person test his/her model inside of one of the larger climate models?</p>
</blockquote>
<p>Yes, new parameterizations are often developed separately from the full climate model before being added to it.  But the longer a parameterization is developed in isolation the more likely it is that it will be conceptually or technically incompatible with the full model.  The trick is knowing early on that you want to couple it into the larger model later and to design your parameterization accordingly.  In my experience, however modular we aspire to make these models, it can still be quite a pain to couple in parameterization code that’s had a well-established, independent life outside of the climate model.  In general though, this is where the shared code repositories and an active community are really useful.</p>
<blockquote>
<p>Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to &quot;abstract&quot; away some of this configuration in objects.</p>
</blockquote>
<p>I remember years ago, as a student, a computer scientist friend of mine avoided doing an industry placement at a climate modelling center because he had such a low opinion of their software.  &quot;It's so basic and boring&quot;, he said, &quot;They just use Fortran!&quot;  But those same things that are off-putting to a computing student are beneficial to the largely self-taught programmers (i.e., physical scientists) who are working with these models.  Fortran is a fairly straightforward and safe language to learn and use, with relatively few concepts and gotchas.  Compare that with OOP paradigms, which are hard to use well without significant training.</p>
<p>But those are more comments on the portability of the programmers than of the programs.  As Simon mentions, the difficult bits of getting a climate model running on new hardware (big or small) tend to be handled by the support staff of that hardware rather than the academic researchers themselves.</p>

CC BY-SA 4.0
2021-09-19T18:05:08.193Z
yes this is very helpful. I can see how the ARCHER2, Monsoon, or other systems allow researchers to plug-in their models into the larger frameworks. So that makes a lot more sense. I understand the workflow better now. 

I can see how highly standardized the code has to be to work with the UM models/hardware. 

Sorry if it sounded like I was criticizing Fortran, as that was not my intent. I was really just talking about the portability of the code to other hardware or environments.
krishnab
CC BY-SA 4.0
2021-09-19T18:08:39.857Z
just an additional question. So given that the UM models are so highly standardized, is it possible to experiment with different meshing strategies for a model. Like if I wanted to experiment with different discretizations or use spectral elements versus discontinuous galerkin methods? I just wanted to understand what the range of flexibility was when using these kinds of models. Or is the meshing strategy set by the system, and everyone has to use that defined method--to ensure the rest of the model pieces work?
krishnab
CC BY-SA 4.0
2021-09-19T17:21:42.463Z
2021-09-19T17:21:42.463Z
Deditos
<p>I have been brushing up on numerical methods for PDEs lately, and climate models provide such a rich set of models and ideas to practice with. <strong>However, I was a little confused about how academics can do research on large climate models--like the CESM, WRF, or other models produced by GDFL or NCAR--given the computational challenges of running these models?</strong> These are global circulation models, or even limited regional atmospheric models with physics happening at multiple scales in the simulation. I was not sure if most institutions have setups to run these models locally, or is there some way to send a new model to NCAR or similar institute to test the accuracy, etc.</p>
<p>To run large climate simulations requires supercomputers or large cluster resources, as well as delicate MPI and mapreduce operations. Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to &quot;abstract&quot; away some of this configuration in objects.</p>
<p>Writing and running an individual component model or &quot;parameterization&quot; model for one of these larger models seems manageable. That task just requires the scholar to write something doable over one or a few machines. But then how can that same person test his/her model inside of one of the larger climate models?</p>
<p>I have been doing some research on this, but have not found much. I found some descriptions of how these larger scale models are built from their development documentation. The code is available as open source, but something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough--though I don't have any person experience doing that with a climate model. I have also spoken with some climate folks at Caltech who are working on developing their own large scale climate models, but they could only explain the challenges that they faced with building a flexible meshing scheme, etc., The Caltech folks did not tell me about their experiences using the established large scale climate models.</p>
<p>Hence, I figured I would ask the SE community.</p>
<p>UPDATE:
As per some feedback below, I just wanted to explain why this question is posted on ES instead of the Academics.SE. The thought was that Academics.SE is a much more general site across all of academia and many people there don't know about the nuances of numerical computing and the computational setup that goes along with it. Hence, I posted the question on ES where the audience is more familiar with running these types of simulations. I recognize that this question is &quot;soft&quot; however it seemed relevant to others conducting ES and Atmospheric research--especially those who want to do research on this topic and come from other disciplines.</p>

CC BY-SA 4.0
2021-09-15T15:18:02.547Z
@gansub this is helpful. So there is a workflow for running sims on some established implementations of the models. Yeah, I thought about posting this on academics, but that is a much more general site. So I thought that the audience there might not have detailed knowledge of the computational and numerical challenges of large PDE simulations. I understand that this is a "soft" question, but it is also something that is not very clear from reading academic papers--since those might explain a numerical setup but not how the corresponding computational setup was created.
krishnab
CC BY-SA 4.0
2021-09-15T15:19:01.850Z
But perhaps I can add a comment to explain why I did not post on Academics.SE, as your point is valid.
krishnab
CC BY-SA 4.0
2021-09-17T06:26:03.027Z
I agree with krishnab that it seems a very reasonable Earth Science question to me, as I'd think the answer may in fair part be about the scale of the models rather than getting super computing resources itself.  I know some of the highest resolution may use super computers, but I would think there's a chance quite reasonable climate models can be run at a pretty decent resolution.
JeopardyTempest
CC BY-SA 4.0
2021-09-18T13:15:19.643Z
Relevant: https://theconversation.com/how-machine-learning-is-helping-us-fine-tune-climate-models-to-reach-unprecedented-detail-165818
Gimelist
<climate-models><numerical-modelling><atmospheric-circulation>
How do academics/institutions run large climate simulations given the computing challenges?
krishnab
# How do academics/institutions run large climate simulations given the computing challenges?

(Asked by: username_0 on 2021-09-15)

I have been brushing up on numerical methods for PDEs lately, and climate models provide such a rich set of models and ideas to practice with. **However, I was a little confused about how academics can do research on large climate models--like the CESM, WRF, or other models produced by GDFL or NCAR--given the computational challenges of running these models?** These are global circulation models, or even limited regional atmospheric models with physics happening at multiple scales in the simulation. I was not sure if most institutions have setups to run these models locally, or is there some way to send a new model to NCAR or similar institute to test the accuracy, etc.

To run large climate simulations requires supercomputers or large cluster resources, as well as delicate MPI and mapreduce operations. Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to "abstract" away some of this configuration in objects.

Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable. That task just requires the scholar to write something doable over one or a few machines. But then how can that same person test his/her model inside of one of the larger climate models?

I have been doing some research on this, but have not found much. I found some descriptions of how these larger scale models are built from their development documentation. The code is available as open source, but something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough--though I don't have any person experience doing that with a climate model. I have also spoken with some climate folks at Caltech who are working on developing their own large scale climate models, but they could only explain the challenges that they faced with building a flexible meshing scheme, etc., The Caltech folks did not tell me about their experiences using the established large scale climate models.

Hence, I figured I would ask the SE community.

UPDATE:
As per some feedback below, I just wanted to explain why this question is posted on ES instead of the Academics.SE. The thought was that Academics.SE is a much more general site across all of academia and many people there don't know about the nuances of numerical computing and the computational setup that goes along with it. Hence, I posted the question on ES where the audience is more familiar with running these types of simulations. I recognize that this question is "soft" however it seemed relevant to others conducting ES and Atmospheric research--especially those who want to do research on this topic and come from other disciplines.

**Tags:** climate-models, numerical-modelling, atmospheric-circulation

> **Comments:**
> **username_0**: @gansub this is helpful. So there is a workflow for running sims on some established implementations of the models. Yeah, I thought about posting this on academics, but that is a much more general site. So I thought that the audience there might not have detailed knowledge of the computational and numerical challenges of large PDE simulations. I understand that this is a "soft" question, but it is also something that is not very clear from reading academic papers--since those might explain a numerical setup but not how the corresponding computational setup was created.  
> **username_0**: But perhaps I can add a comment to explain why I did not post on Academics.SE, as your point is valid.  
> **username_1**: I agree with username_0 that it seems a very reasonable Earth Science question to me, as I'd think the answer may in fair part be about the scale of the models rather than getting super computing resources itself.  I know some of the highest resolution may use super computers, but I would think there's a chance quite reasonable climate models can be run at a pretty decent resolution.  
> **username_2**: Relevant: https://theconversation.com/how-machine-learning-is-helping-us-fine-tune-climate-models-to-reach-unprecedented-detail-165818  

---

## 2 answers

### Answer by username_3 on 2021-09-17. Score: 5 | Accepted answer

This question is a little vague, with multiple aspects, but I shall do my best.

## Computing resources

Some universities have their own clusters or supercomputers. But many countries also have regional or national facilities. For example, [ARCHER2](https://www.archer2.ac.uk/) is the latest iteration of the UK's national supercomputer service. Small amounts of computing time are available for free to UK academics, but for large projects they must bid for it.

It's rare for simulations to require something of the scale of ARCHER or ARCHER2; depending on the model being run, often a departmental or institutional system of just a few nodes may be adequate, especially if one isn't in a hurry.

## Building code for different systems

So long as the code has been well-written, this is not as hard as it sounds. FORTRAN is reasonably well standardised (though some codes will require the use of a particular compiler), and MPI - despite its varied implementations - is too. And most modern supercomputers are massively parallel machines whose actual compute nodes are standard x64 servers, using CPUs such as Intel Xeons. So you can do a lot of development on a desktop PC before you try to build it somewhere big and expensive. The biggest differences between different systems tend to lie in the interconnectors (the high-speed, low-latency networking between nodes), but usually the system admins will have optimised an MPI installation for that system and made it available to users, so all the details of inter-process and inter-node communications are abstracted.

## Making changes or improvements

As you've identified, for most people the key is to improve one small aspect at a time, such as a particular parametrisation. Once somebody has the standard code working, then if it is well written in a maintainable manner (not a given in the scientific world) then it is relatively straightforward for them to modify one aspect and test it. The hardest thing often is obtaining measurements or other data against which to validate the new version.

> **Comments:**
> **username_0**: Yes, this makes a lot of sense. As a non-specialist in climate models, your answer clarifies a lot. It seem like there is a lot of compartmentalization of these different modelling levels. So a person can innovate on a parameterization model or develop a new model on their computer. Once that works, they can use their local cluster for larger scale simulation (perhaps at finer resolution). If the model is useful, then someone can write the code for ARCHER2, etc.  
> **username_3**: @username_0 sort of, but also it is often possible to run the SAME CODE on all those computers. Just faster or slower. Remember that these days, clusters and supercomputers are basically the same as your desktop computer, repeated many times.  
> **username_4**: Good answer & I know this isn't a discussion site, ... but, as a supplementary question, is distributed computing used for serious climate modeling?  
> **username_3**: @username_4 by distributed computing, you mean computers in different locations working together? Not so far as I know, but I do not claim complete knowledge of the climate modelling space. In the types of models I'm familiar with - which, again, is not exhaustive - latency between nodes is important to performance.  
> **username_0**: @username_4 I think distributed computing usually refers to a cluster right, meaning a bunch of servers that are connected across a local network, and which breakup a large compute job into pieces. So it seems the answer is yes. Architecture for this can vary, but seems like someone was using the Amazon cloud for running climate models. I am not sure about distributing server over longer geographic distances--the latency would be a killer as username_3 indicated.  
> **username_4**: @username_0: [Distributed computing](https://www.ibm.com/docs/en/txseries/8.2?topic=overview-what-is-distributed-computing), computers are that "can be physically close together and connected by a local network, or they can be geographically distant and connected by a wide area network". I was thinking of something similar to [Seti@home or Albert@home](https://en.wikipedia.org/wiki/List_of_distributed_computing_projects). I know one of the newer radio astronomy installations in Western Australia, within the last few years, used this type of distributed computing to get its data analyzed. ...  
> **username_4**: ... one of the disadvantages of using such a system is the results from each packet of data needs to be verified a number of times, which tends to delay the final result. For time critical analyses it wouldn't be used. I don't know how time critical some climate modeling would be & whether distributed computing would be appropriate.  
> **username_4**: Having stated what I just did, I have just noticed, from the listed of distributed computing projects in the link in my previous comment, Oxford University has a [climate predicting](https://en.wikipedia.org/wiki/Climateprediction.net) application via BOINC distributed computing platform.  
> **username_0**: @username_4 oh right, I get what you mean. It reminds me of those protein folding applications that I used to run as my screen saver. I think it really depends upon the scale of the computing. For a climate model, the scale of the grids/meshes are adjusted to be computed in a certain amount of time and within certain memory requirements. But if your model is really just much larger than the available resources you can access, then something distributed makes sense. But you are correct, it is not clear how long it would take to run.  
> **username_0**: But I agree that running these models in wider distributed fashion might become necessary--unless quantum computing solves the challenge. Climate models are fine, but if we want to solve a large number of climate models as an input to other models of interest, then compute is the big issue. I was interested in looking at flooding prediction based on hurricanes. So I would need to simulate hundreds of hurricanes and trajectories (big compute task), and then model the resulting storm surge and flooding hundreds of times. So 2 big compute tasks.  

---

### Answer by username_5 on 2021-09-19. Score: 3

Another UK perspective here to supplement username_3’s answer (which reflects my experience too).

The UK research community is dominated by a single family of models known collectively as the Unified Model. The code for these is owned by the UK's national weather and climate modelling center, the Met Office, and used gratis under license by academics. That license gives academics access to repositories of model source code and experiment setups shared across the MO and academic community. username_3 mentioned ARCHER2, which is shared by the academics across all disciplines, but the Met Office also provides [Monsoon](https://www.metoffice.gov.uk/research/approach/collaboration/jwcrp/monsoon-hpc) specifically for atmospheric modelling collaborations with the academic community.

Because the MO is an operational forecast center, they put a *lot* of effort into making sure the code and support software run efficiently and reliably, so what academics have access to is pretty battle hardened. But the academic side also has [NCAS-CMS](https://cms-helpdesk.ncas.ac.uk), whose job it is to make sure the model works on machines like ARCHER2 for the whole community. All in all there’s a good level of national support for this model on these machines, and when I send a student on the training course for the model they can be running climate simulations on that hardware within 30 minutes of arriving.

> something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough

Well, the UM has about 1.3 million lines of code and I’d estimate that I know about 15% of that code very very well (mainly a particular area of science) and the rest of it barely at all. When I encounter bugs they’re almost always because of the thing that I’ve just changed or something closely related in areas of the code I know well. When bugs lead into other parts of the model, it’s usually best to go and ask someone who knows about those areas rather than digging too hard yourself.

> Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable... But then how can that same person test his/her model inside of one of the larger climate models?

Yes, new parameterizations are often developed separately from the full climate model before being added to it. But the longer a parameterization is developed in isolation the more likely it is that it will be conceptually or technically incompatible with the full model. The trick is knowing early on that you want to couple it into the larger model later and to design your parameterization accordingly. In my experience, however modular we aspire to make these models, it can still be quite a pain to couple in parameterization code that’s had a well-established, independent life outside of the climate model. In general though, this is where the shared code repositories and an active community are really useful.

> Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to "abstract" away some of this configuration in objects.

I remember years ago, as a student, a computer scientist friend of mine avoided doing an industry placement at a climate modelling center because he had such a low opinion of their software. "It's so basic and boring", he said, "They just use Fortran!" But those same things that are off-putting to a computing student are beneficial to the largely self-taught programmers (i.e., physical scientists) who are working with these models. Fortran is a fairly straightforward and safe language to learn and use, with relatively few concepts and gotchas. Compare that with OOP paradigms, which are hard to use well without significant training.

But those are more comments on the portability of the programmers than of the programs. As username_3 mentions, the difficult bits of getting a climate model running on new hardware (big or small) tend to be handled by the support staff of that hardware rather than the academic researchers themselves.

> **Comments:**
> **username_0**: yes this is very helpful. I can see how the ARCHER2, Monsoon, or other systems allow researchers to plug-in their models into the larger frameworks. So that makes a lot more sense. I understand the workflow better now.   I can see how highly standardized the code has to be to work with the UM models/hardware.   Sorry if it sounded like I was criticizing Fortran, as that was not my intent. I was really just talking about the portability of the code to other hardware or environments.  
> **username_0**: just an additional question. So given that the UM models are so highly standardized, is it possible to experiment with different meshing strategies for a model. Like if I wanted to experiment with different discretizations or use spectral elements versus discontinuous galerkin methods? I just wanted to understand what the range of flexibility was when using these kinds of models. Or is the meshing strategy set by the system, and everyone has to use that defined method--to ensure the rest of the model pieces work?  

---

