  In this paper, we consider a reconfigurable intelligent surface
(RIS)-assisted 3D direction-of-arrival (DoA) estimation system, in which a
uniform planar array (UPA) RIS is deployed to provide virtual line-of-sight
(LOS) links and reflect the uplink pilot signal to sensors. To overcome the
mutually coupled problem between the beamforming design at the RIS and DoA
estimation, we explore the separable sparse representation structure and
propose an alternating optimization algorithm. The grid-based DoA estimation is
modeled as a joint-sparse recovery problem considering the grid bias, and the
Joint-2D-OMP method is used to estimate both on-grid and off-grid parts. The
corresponding Cram\'er-Rao lower bound (CRLB) is derived to evaluate the
estimation. Then, the beampattern at the RIS is optimized to maximize the
signal-to-noise (SNR) at sensors according to the estimated angles. Numerical
results show that the proposed alternating optimization algorithm can achieve
lower estimation error compared to benchmarks of random beamforming design.
