  Co-based honeycomb magnets have been actively studied recently for the
potential realization of emergent quantum magnetism therein such as the Kitaev
spin liquid. Here we employ density functional and dynamical mean-field theory
methods to examine a family of the Kitaev magnet candidates
BaCo$_2$($X$O$_4$)$_2$ ($X$ = P, As, Sb), where the compound with $X$ = Sb
being not synthesized yet. Our study confirms the formation of Mott insulating
phase and the $J_{\rm eff}$ = 1/2 spin moments at Co$^{2+}$ sites despite the
presence of a sizable amount of trigonal crystal field in all three compounds.
The pnictogen substitution from phosphorus to antimony significantly changes
the in-plane lattice parameters and direct overlap integral between the
neighboring Co ions, leading to the suppression of the Heisenberg interaction.
More interestingly, the marginal antiferromagnetic nearest-neighbor Kitaev term
changes sign into a ferromagnetic one and becomes sizable at the $X$ = Sb
limit. Our study suggests that the pnictogen substitution can be a viable route
to continuously tune magnetic exchange interactions and to promote magnetic
frustration for the realization of potential spin liquid phases in
BaCo$_2$($X$O$_4$)$_2$.
