  We study frequentist properties of Bayesian and $L_0$ model selection, with a
focus on (potentially non-linear) high-dimensional regression. We propose a
construction to study how posterior probabilities and normalized $L_0$ criteria
concentrate on the (Kullback-Leibler) optimal model and other subsets of the
model space. When such concentration occurs, one also bounds the frequentist
probabilities of selecting the correct model, type I and type II errors. These
results hold generally, and help validate the use of posterior probabilities
and $L_0$ criteria to control frequentist error probabilities associated to
model selection and hypothesis tests. Regarding regression, we help understand
the effect of the sparsity imposed by the prior or the $L_0$ penalty, and of
problem characteristics such as the sample size, signal-to-noise, dimension and
true sparsity. A particular finding is that one may use less sparse
formulations than would be asymptotically optimal, but still attain consistency
and often also significantly better finite-sample performance. We also prove
new results related to misspecifying the mean or covariance structures, and
give tighter rates for certain non-local priors than currently available.
