  We consider regression under the "extremely small $n$ large $p$" condition,
where the number of samples $n$ is so small compared to the dimensionality $p$
that predictors cannot be estimated without prior knowledge. This setup occurs
in personalized medicine, for instance, when predicting treatment outcomes for
an individual patient based on noisy high-dimensional genomics data. A
remaining source of information is expert knowledge, which has received
relatively little attention in recent years. We formulate the inference problem
of asking expert feedback on features on a budget, propose an elicitation
strategy for a simple "small $n$" setting, and derive conditions under which
the elicitation strategy is optimal. Experiments on simulated experts, both on
synthetic and genomics data, demonstrate that the proposed strategy can
drastically improve prediction accuracy.
