  Due to the curse of dimensionality, estimation in a multidimensional
nonparametric regression model is in general not feasible. Hence, additional
restrictions are introduced, and the additive model takes a prominent place.
The restrictions imposed can lead to serious bias. Here, a new estimator is
proposed which allows penalizing the nonadditive part of a regression function.
This offers a smooth choice between the full and the additive model. As a
byproduct, this penalty leads to a regularization in sparse regions. If the
additive model does not hold, a small penalty introduces an additional bias
compared to the full model which is compensated by the reduced bias due to
using smaller bandwidths. For increasing penalties, this estimator converges to
the additive smooth backfitting estimator of Mammen, Linton and Nielsen [Ann.
Statist. 27 (1999) 1443-1490]. The structure of the estimator is investigated
and two algorithms are provided. A proposal for selection of tuning parameters
is made and the respective properties are studied. Finally, a finite sample
evaluation is performed for simulated and ozone data.
