  We consider the problem of estimating the structural function in
nonparametric instrumental regression, where in the presence of an instrument W
a response Y is modeled in dependence of an endogenous explanatory variable Z.
The proposed estimator is based on dimension reduction and additional
thresholding. The minimax optimal rate of convergence of the estimator is
derived assuming that the structural function belongs to some ellipsoids which
are in a certain sense linked to the conditional expectation operator of Z
given W. We illustrate these results by considering classical smoothness
assumptions. However, the proposed estimator requires an optimal choice of a
dimension parameter depending on certain characteristics of the unknown
structural function and the conditional expectation operator of Z given W,
which are not known in practice. The main issue addressed in our work is an
adaptive choice of this dimension parameter using a model selection approach
under the restriction that the conditional expectation operator of Z given W is
smoothing in a certain sense. In this situation we develop a penalized minimum
contrast estimator with randomized penalty and collection of models. We show
that this data-driven estimator can attain the lower risk bound up to a
constant over a wide range of smoothness classes for the structural function.
