I have come across the term "closed-form solution" quite often. What does a closed-form solution mean? How does one determine if a close-form solution exists for a given problem? Searching online, I found some information, but nothing in the context of developing a statistical or probabilistic model / solution. 

I understand regression very well, so if any one can explain the concept with reference to regression or model-fitting, it will be easy to consume. :)


  "An equation is said to be a closed-form solution if it solves a given
  problem in terms of functions and mathematical operations from a given
  generally accepted set. For example, an infinite sum would generally
  not be considered closed-form. However, the choice of what to call
  closed-form and what not is rather arbitrary since a new "closed-form"
  function could simply be defined in terms of the infinite sum."
  --Wolfram Alpha


and 


  "In mathematics, an expression is said to be a closed-form expression
  if it can be expressed analytically in terms of a finite number of
  certain "well-known" functions. Typically, these well-known functions
  are defined to be elementary functions—constants, one variable x,
  elementary operations of arithmetic (+ − × ÷), nth roots, exponent and
  logarithm (which thus also include trigonometric functions and inverse
  trigonometric functions). Often problems are said to be tractable if
  they can be solved in terms of a closed-form expression." -- Wikipedia


An example of a closed form solution in linear regression would be the least square equation

$$\hat\beta=(X^TX)^{-1}X^Ty$$
