<p>Unfortunately, there are many possible levels of description for a problem; choosing one among them is strictly dependent on your aims and on the tools available to you. In general, there are three possible approaches: analytical, numerical and observational. Each one is characterized by pro and cons. When you need to build a model, there are many possible decisions that must be made. For instance, you may decide among:</p>

<ul>
<li>specific - general </li>
<li>model estimation - first-principles models</li>
<li>numerical - analytical </li>
<li>stochastic -deterministic </li>
<li>microscopic - macroscopic </li>
<li>discrete - continuous </li>
<li>qualitative - quantitative</li>
</ul>

<p>and so on. You must understand that, in general, there are no rigorous rules of thumb when making these choices. As pointed out by @Suresh, this a matter of experience, requires a careful understanding of the domain, and some inspired guessing as well. The good news is that once you have made your choices, then there are rigorous ways to proceed. Probably, the best model overall for a given problem is the one given by the Minimum Description Length Principle (essentially the Occam's razor): the best model is the smallest one. But determining it is beyond hope. So, the best model is almost always, the one that works best for you, given your explicit and even implicit assumptions.</p>

<p>That said, analytical and numerical approaches allows you to explore the behavior of a model, but the assumption here is that the underlying model is known. It's up to you to derive this model. Observational approaches allows you to derive a model from the experimentally observed data. </p>

<p>Analytical models may lead to a closed-form solution; they are of great importance, given their power: if and when an analytical model may be successfully applied, then you will be able to deduce almost everything there is to know about a system. Their major drawback is limited applicability, since much of the world is simply too complicated to be described this way. Examples include ODE, PDE, difference equations, variational calculus and stochastic processes.</p>

<p>Numerical models are required when the equations in an analytical model are too complex to be solved exactly in closed-form. This usually happens as soon as we move too far from linearity to nonlinearity. It's important to remember that numerical and analytical solutions are complementary approaches rather than exclusive alternatives. Examples include Runge-Kutta methods, finite element methods, cellular automata and lattice gases.</p>

<p>Observational models can come from introspection or observation (or both).In some cases you may want to infer a model from experimentally measured data.This model may then be used to characterize and classify the data at hand, or to generalize from the measurements to predict the outcome of new observations etc. Examples include function fitting, SVD, Fourier and Wavelets transforms, neural networks, genetic algorithms, Expectation-Maximization algorithms, Wiener and Kalman filters, HMM, and time series. </p>

<p>A very good introductory book you may want to read is <a href="http://rads.stackoverflow.com/amzn/click/0521570956" rel="nofollow">The nature of mathematical modeling</a>. Of course, given the huge amount of topics covered, do not expect a detailed treatment. </p>

CC BY-SA 3.0
2012-05-11T19:41:16.973Z
I can hardly believe that my question has prompted a response from such knowledgeable people. Thank you.
user612921
CC BY-SA 3.0
2012-05-11T18:46:48.467Z
2012-05-11T18:46:48.467Z
Massimo Cafaro
<p>In case you know the generation model of your data, you may use the  maximum likelihood estimation (MLE)</p>

CC BY-SA 3.0
2012-05-22T04:10:47.593Z
2012-05-22T04:10:47.593Z
Dan Feldman
<p>This is a vague question. I will do my best, I think it has definite answers. I am hoping for answers of the form "Read book x, learn this specific topic, read this paper/s".</p>

<p>What is bothering me is that every AI system I have read about seems to have an internal model which is set up to operate in a specific way. The task of the learning algorithm is to then optimise the parameters of this model to solve the problem.</p>

<p>I am thinking that this is akin to setting up a custom description language which preserves or translates some structure of the thing to be modelled in such a way that it becomes amenable to solution through some understood algorithmic technique; iterative gradient descent, analytic, whatever.</p>

<p>The Question: Is there any framework in which the optimum model/description language can be discovered? For instance in a GA the mapping from the genome to the phenotype is critical to the effective operation of the algorithm. This mapping is manually defined and, I would posit, is the most critical component of the GA. Specifically how would one search through the space of mappings? Or going meta how would one search through the space of algorithms for finding that best mapping? It feels like you could disappear up that ladder forever. In order to solve the problem you must first solve the problem.</p>

<p>Neural networks seem to represent an effective model for solving real world problems, why? How would one generate another model for a different set of problems? Cellular Automata seem good for modelling traffic flows or disease transmission, why? How would I automatically generate alternative systems which also model the set of problems effectively? etc.</p>

<p>Is this something that is thought about and discussed? Where would I find it? This is my first question, please be gentle! :)</p>

CC BY-SA 3.0
2012-05-10T22:25:17.833Z
note that neural networks are just as sensitive as GA to how you represent the input (i.e. how you encode the features of your problem into the activations of the input neurons). I think automated _feature detection_ and _model selection_ are the keywords you are looking for. You might get better results about this over at [stats.SE](http://stats.stackexchange.com/).
Artem Kaznatcheev
CC BY-SA 3.0
2012-05-11T02:10:11.997Z
This is a tricky business in general. The real answer is that no single model works for all problems, and choosing the right way to model a problem is a matter of experience, a careful understanding of the domain, and some inspired guessing
Suresh Venkat
CC BY-SA 3.0
2012-05-11T20:13:22.267Z
@Suresh, I appreciate your response and I'm trying not to be facetious: is the substrate which hosts the careful understanding and the inspired guessing not a good general system for designing models? Is there any formal notion around for something like different basis(s) of computation and comparisons between them for different kinds of problem? Perhaps this should be a different question...
user612921
CC BY-SA 3.0
2012-05-11T21:04:57.210Z
@SureshVenkat +1 to inspired guessing. There's some element of that in many fields, not just AI.
Joe
<machine-learning><big-picture>
Discerning the best model for a problem
user612921
# Discerning the best model for a problem

(Asked by: username_0 on 2012-05-10)

This is a vague question. I will do my best, I think it has definite answers. I am hoping for answers of the form "Read book x, learn this specific topic, read this paper/s".

What is bothering me is that every AI system I have read about seems to have an internal model which is set up to operate in a specific way. The task of the learning algorithm is to then optimise the parameters of this model to solve the problem.

I am thinking that this is akin to setting up a custom description language which preserves or translates some structure of the thing to be modelled in such a way that it becomes amenable to solution through some understood algorithmic technique; iterative gradient descent, analytic, whatever.

The Question: Is there any framework in which the optimum model/description language can be discovered? For instance in a GA the mapping from the genome to the phenotype is critical to the effective operation of the algorithm. This mapping is manually defined and, I would posit, is the most critical component of the GA. Specifically how would one search through the space of mappings? Or going meta how would one search through the space of algorithms for finding that best mapping? It feels like you could disappear up that ladder forever. In order to solve the problem you must first solve the problem.

Neural networks seem to represent an effective model for solving real world problems, why? How would one generate another model for a different set of problems? Cellular Automata seem good for modelling traffic flows or disease transmission, why? How would I automatically generate alternative systems which also model the set of problems effectively? etc.

Is this something that is thought about and discussed? Where would I find it? This is my first question, please be gentle! :)

**Tags:** machine-learning, big-picture
> **Comments:**
> **username_1**: note that neural networks are just as sensitive as GA to how you represent the input (i.e. how you encode the features of your problem into the activations of the input neurons). I think automated _feature detection_ and _model selection_ are the keywords you are looking for. You might get better results about this over at [stats.SE](http://stats.stackexchange.com/).  
> **username_2**: This is a tricky business in general. The real answer is that no single model works for all problems, and choosing the right way to model a problem is a matter of experience, a careful understanding of the domain, and some inspired guessing  
> **username_0**: @Suresh, I appreciate your response and I'm trying not to be facetious: is the substrate which hosts the careful understanding and the inspired guessing not a good general system for designing models? Is there any formal notion around for something like different basis(s) of computation and comparisons between them for different kinds of problem? Perhaps this should be a different question...  
> **username_3**: @SureshVenkat +1 to inspired guessing. There's some element of that in many fields, not just AI.  

---

## 2 answers

### Answer by username_4 on 2012-05-11. Score: 4 | Accepted answer

Unfortunately, there are many possible levels of description for a problem; choosing one among them is strictly dependent on your aims and on the tools available to you. In general, there are three possible approaches: analytical, numerical and observational. Each one is characterized by pro and cons. When you need to build a model, there are many possible decisions that must be made. For instance, you may decide among:

* specific - general
* model estimation - first-principles models
* numerical - analytical
* stochastic -deterministic
* microscopic - macroscopic
* discrete - continuous
* qualitative - quantitative

and so on. You must understand that, in general, there are no rigorous rules of thumb when making these choices. As pointed out by @Suresh, this a matter of experience, requires a careful understanding of the domain, and some inspired guessing as well. The good news is that once you have made your choices, then there are rigorous ways to proceed. Probably, the best model overall for a given problem is the one given by the Minimum Description Length Principle (essentially the Occam's razor): the best model is the smallest one. But determining it is beyond hope. So, the best model is almost always, the one that works best for you, given your explicit and even implicit assumptions.

That said, analytical and numerical approaches allows you to explore the behavior of a model, but the assumption here is that the underlying model is known. It's up to you to derive this model. Observational approaches allows you to derive a model from the experimentally observed data.

Analytical models may lead to a closed-form solution; they are of great importance, given their power: if and when an analytical model may be successfully applied, then you will be able to deduce almost everything there is to know about a system. Their major drawback is limited applicability, since much of the world is simply too complicated to be described this way. Examples include ODE, PDE, difference equations, variational calculus and stochastic processes.

Numerical models are required when the equations in an analytical model are too complex to be solved exactly in closed-form. This usually happens as soon as we move too far from linearity to nonlinearity. It's important to remember that numerical and analytical solutions are complementary approaches rather than exclusive alternatives. Examples include Runge-Kutta methods, finite element methods, cellular automata and lattice gases.

Observational models can come from introspection or observation (or both).In some cases you may want to infer a model from experimentally measured data.This model may then be used to characterize and classify the data at hand, or to generalize from the measurements to predict the outcome of new observations etc. Examples include function fitting, SVD, Fourier and Wavelets transforms, neural networks, genetic algorithms, Expectation-Maximization algorithms, Wiener and Kalman filters, HMM, and time series.

A very good introductory book you may want to read is [The nature of mathematical modeling](http://rads.stackoverflow.com/amzn/click/0521570956). Of course, given the huge amount of topics covered, do not expect a detailed treatment.

> **Comments:**
> **username_0**: I can hardly believe that my question has prompted a response from such knowledgeable people. Thank you.  

---

### Answer by username_5 on 2012-05-22. Score: 1

In case you know the generation model of your data, you may use the maximum likelihood estimation (MLE)


---

