  Since model selection is ubiquitous in data analysis, reproducibility of
statistical results demands a serious evaluation of reliability of the employed
model selection method, no matter what label it may have in terms of good
properties. Instability measures have been proposed for evaluating model
selection uncertainty. However, low instability does not necessarily indicate
that the selected model is trustworthy, since low instability can also arise
when a certain method tends to select an overly parsimonious model. F- and
G-measures have become increasingly popular for assessing variable selection
performance in theoretical studies and simulation results. However, they are
not computable in practice. In this work, we propose an estimation method for
F- and G-measures and prove their desirable properties of uniform consistency.
This gives the data analyst a valuable tool to compare different variable
selection methods based on the data at hand. Extensive simulations are
conducted to show the very good finite sample performance of our approach. We
further demonstrate the application of our methods using several micro-array
gene expression data sets, with intriguing findings.
