  We investigate possible quantum ground states as well as the classical limit
of a frustrated J1-J2 Heisenberg model on the three-dimensional (3D)
hyperhoneycomb lattice. Our study is inspired by the recent discovery of
beta-Li2IrO3, where Ir^{4+} ions form a 3D network with each lattice site being
connected to three nearest neighbors. We focus on the influence of magnetic
frustration caused by the second-nearest neighbor spin interactions. Such
interactions are likely to be significant due to large extent of 5d orbitals in
iridates or other 5d transition metal oxides. In the classical limit, the
ground state manifold is given by line degeneracies of the spiral
magnetic-order wavevectors when J2/J1\gtrsim 0.17 while the collinear stripy
order is included in the degenerate manifold when J2/J1 = 0.5. Quantum
order-by-disorder effects are studied using both the semi-classical $1/S$
expansion in the spin wave theory and Schwinger boson approach. In general,
certain coplanar spiral orders are chosen from the classical degenerate
manifold for a large fraction of the phase diagram. Nonetheless quantum
fluctuations favor the collinear stripy order over the spiral orders in an
extended parameter region around J2/J1 = 0.5, despite the spin-rotation
invariance of the underlying Hamiltonian. This is in contrast to the emergence
of stripy order in the Heisenberg-Kitaev model studied earlier on the same
lattice, where the Kitaev-type Ising interactions are important for stabilizing
the stripy order. As quantum fluctuations become stronger, U(1) and Z2 quantum
spin liquid phases are shown to arise via quantum disordering of the Neel,
stripy and spiral magnetically-ordered phases. The effects of magnetic
anisotropy and their relevance to future experiments are also discussed.
