Competition between spin liquids and valence-bond order in the frustrated spin-1/2 Heisenberg model on the honeycomb lattice

Using variational wave functions and Monte Carlo techniques, we study the antiferromagnetic Heisenberg model with first-neighbor J(1) and second-neighbor J(2) antiferromagnetic couplings on the honeycomb lattice. We perform a systematic comparison of magnetically ordered and nonmagnetic states (spin liquids and valence-bond solids) to obtain the ground-state phase diagram. Neel order is stabilized for small values of the frustrating second-neighbor coupling. Increasing the ratio J(2)/J(1), we find strong evidence for a continuous transition to a nonmagnetic phase at J(2)/J(1) approximate to 0.23. Close to the transition point, the Gutzwiller-projected uniform resonating valence-bond state gives an excellent approximation to the exact ground-state energy. For 0.23 less than or similar to J(2)/J(1) less than or similar to 0.36, a gapless Z(2) spin liquid with Dirac nodes competes with a plaquette valence-bond solid. In contrast, the gapped spin liquid considered in previous works has significantly higher variational energy. Although the plaquette valence-bond order is expected to be present as soon as the Neel order melts, this ordered state becomes clearly favored only for J(2)/J(1) greater than or similar to 0.3. Finally, for 0.36 less than or similar to J(2)/J(1) <= 0.5, a valence-bond solid with columnar order takes over as the ground state, being also lower in energy than the magnetic state with collinear order. We perform a detailed finite-size scaling and standard data collapse analysis, and we discuss the possibility of a deconfined quantum critical point separating the Neel antiferromagnet from the plaquette valence-bond solid.