<p>The generalization is to add a regularization term of the form <span class="math-container">$-\log p(\theta)$</span> where <span class="math-container">$p(\theta)$</span> is the probability of coefficients <span class="math-container">$\theta$</span> according to your prior.  Note how Ridge and LASSO can be viewed as special cases of this (try calculating what <span class="math-container">$\log p(\theta)$</span> is for the Gaussian and Laplacian distributions).</p>

CC BY-SA 4.0
2021-12-23T00:39:42.163Z
Thank you @D.W.  Thus, applied to my original questions: (1) given a prior, there is a corresponding regularized regression; the regularization term may not be a metric. (2) remains an open question for me, but a symmetric prior seems a necessary condition?
Colin Rowat
CC BY-SA 4.0
2021-12-23T01:00:23.050Z
@ColinRowat, my answer works whether the prior is symmetric or not, so I believe it answers both questions.  If that doesn't sound right to you, I suggest asking a new questions and providing more context to explain what you are asking.  (Incidentally, I don't know why you want a distance metric or what it would even mean here; $\theta$ is a single variable, and distance is only meaningful if you have two points, not just one.  There is no requirement that regularization terms be associated with a distance metric.)  You might be interested in https://stats.stackexchange.com/
D.W.
CC BY-SA 4.0
2021-12-28T23:22:35.147Z
Thanks @D.W.  I've clarified my original question in response to your comments, and provided what I think is an example demonstrating that a prior need not be associated with a regularization term interpretable as a norm.
Colin Rowat
CC BY-SA 4.0
2021-12-28T23:25:16.497Z
@ColinRowat, that's fine.  To tell whether it matches a regularization term that can be expressed as a norm, compute the regularization term as explained in my answer, then check whether that term is a norm.  In my experience it doesn't matter whether the regularization term is a norm or not.
D.W.
CC BY-SA 4.0
2021-12-29T13:14:02.270Z
thanks @D.W. - my interest in whether the regularization term is a norm arises purely from curiosity: I don't have any practical application in mind at this point.
Colin Rowat
CC BY-SA 4.0
2021-12-12T21:12:19.837Z
2021-12-12T21:12:19.837Z
D.W.
<p>Ridge and LASSO can be interpreted as OLS with priors over the coefficients (respectively, Gaussian and Laplacian).  How much does this generalize?  Given a prior, does it imply a regularization term corresponding to a norm?  Given a symmetric prior, is there such a regularization term?</p>
<p>I believe that the answer to the first question is `no'.  To see why, consider a linear model with Gaussian <em>likelihood function</em>
<span class="math-container">$
  p(y | X, \beta) = \frac{1}{\sigma \sqrt{2 \pi}} \exp \left( - \frac{1}{2 \sigma^2} (y - \beta X)^2 \right)
$</span>
and lognormal <em>prior</em>
<span class="math-container">$
  p (\beta) = \frac{1}{\beta \tau \sqrt{2 \pi}} \exp \left( - \frac{1}{2 \tau^2} (\ln \beta)^2 \right).
$</span>
Its <em>posterior</em> is then
<span class="math-container">$
  p(\beta | y, X) \propto \exp \left( - \frac{1}{2 \sigma^2} (y - \beta X)^2 - \frac{1}{2 \tau^2} (\ln \beta)^2 - \ln \beta \right),
$</span>
with <em>regularization term</em>
<span class="math-container">$\frac{\sigma^2}{\tau^2} (\ln \beta)^2 + 2 \sigma^2 \ln \beta$</span>.</p>
<p>This does not correspond to a <em>norm</em> as it does not satisfy non-negativity for <span class="math-container">$\beta \in \left( \exp(-2 \tau^2) , 1 \right)$</span>, a non-degenerate interval for all <span class="math-container">$\tau &gt; 0$</span>.</p>

<machine-learning>
equivalence between Bayesian prior distribution and regularization metric?
Colin Rowat
# equivalence between Bayesian prior distribution and regularization metric?

(Asked by: username_0 on 2021-12-11)

Ridge and LASSO can be interpreted as OLS with priors over the coefficients (respectively, Gaussian and Laplacian). How much does this generalize? Given a prior, does it imply a regularization term corresponding to a norm? Given a symmetric prior, is there such a regularization term?

I believe that the answer to the first question is `no'. To see why, consider a linear model with Gaussian *likelihood function*
$
p(y | X, \beta) = \frac{1}{\sigma \sqrt{2 \pi}} \exp \left( - \frac{1}{2 \sigma^2} (y - \beta X)^2 \right)
$
and lognormal *prior*
$
p (\beta) = \frac{1}{\beta \tau \sqrt{2 \pi}} \exp \left( - \frac{1}{2 \tau^2} (\ln \beta)^2 \right).
$
Its *posterior* is then
$
p(\beta | y, X) \propto \exp \left( - \frac{1}{2 \sigma^2} (y - \beta X)^2 - \frac{1}{2 \tau^2} (\ln \beta)^2 - \ln \beta \right),
$
with *regularization term*
$\frac{\sigma^2}{\tau^2} (\ln \beta)^2 + 2 \sigma^2 \ln \beta$.

This does not correspond to a *norm* as it does not satisfy non-negativity for $\beta \in \left( \exp(-2 \tau^2) , 1 \right)$, a non-degenerate interval for all $\tau > 0$.

**Tags:** machine-learning


---

## 1 answers

### Answer by username_1 on 2021-12-12. Score: 1

The generalization is to add a regularization term of the form $-\log p(\theta)$ where $p(\theta)$ is the probability of coefficients $\theta$ according to your prior. Note how Ridge and LASSO can be viewed as special cases of this (try calculating what $\log p(\theta)$ is for the Gaussian and Laplacian distributions).

> **Comments:**
> **username_0**: Thank you @username_1  Thus, applied to my original questions: (1) given a prior, there is a corresponding regularized regression; the regularization term may not be a metric. (2) remains an open question for me, but a symmetric prior seems a necessary condition?  
> **username_1**: @ColinRowat, my answer works whether the prior is symmetric or not, so I believe it answers both questions.  If that doesn't sound right to you, I suggest asking a new questions and providing more context to explain what you are asking.  (Incidentally, I don't know why you want a distance metric or what it would even mean here; $\theta$ is a single variable, and distance is only meaningful if you have two points, not just one.  There is no requirement that regularization terms be associated with a distance metric.)  You might be interested in https://stats.stackexchange.com/  
> **username_0**: Thanks @username_1  I've clarified my original question in response to your comments, and provided what I think is an example demonstrating that a prior need not be associated with a regularization term interpretable as a norm.  
> **username_1**: @ColinRowat, that's fine.  To tell whether it matches a regularization term that can be expressed as a norm, compute the regularization term as explained in my answer, then check whether that term is a norm.  In my experience it doesn't matter whether the regularization term is a norm or not.  
> **username_0**: thanks @username_1 - my interest in whether the regularization term is a norm arises purely from curiosity: I don't have any practical application in mind at this point.  

---

