  Neural network quantum states provide a novel representation of the many-body
states of interacting quantum systems and open up a promising route to solve
frustrated quantum spin models that evade other numerical approaches. Yet its
capacity to describe complex magnetic orders with large unit cells has not been
demonstrated, and its performance in a rugged energy landscape has been
questioned. Here we apply restricted Boltzmann machines and stochastic gradient
descent to seek the ground states of a compass spin model on the honeycomb
lattice, which unifies the Kitaev model, Ising model and the quantum
120$^\circ$ model with a single tuning parameter. We report calculation results
on the variational energy, order parameters and correlation functions. The
phase diagram obtained is in good agreement with the predictions of tensor
network ansatz, demonstrating the capacity of restricted Boltzmann machines in
learning the ground states of frustrated quantum spin Hamiltonians. The
limitations of the calculation are discussed. A few strategies are outlined to
address some of the challenges in machine learning frustrated quantum magnets.
