Crystal structure, spin flop transition, and magnetoelectric effect in the honeycomb-lattice frustrated Fe-doped <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msub><mml:mi>Ni</mml:mi><mml:mn>2</mml:mn></mml:msub><mml:msub><mml:mi>Mo</mml:mi><mml:mn>3</mml:mn></mml:msub><mml:msub><mml:mi mathvariant="normal">O</mml:mi><mml:mn>8</mml:mn></mml:msub></mml:mrow></mml:math> antiferromagnets

The magnetoelectric (ME) effect, with the cross coupling between magnetism and electric polarization, provides a new controlling dimension for creating novel information storage devices. Here, we report crystal structure, magnetic field-induced spin flop transition, and ME behaviors of Fe-doped ${\mathrm{Ni}}_{2}{\mathrm{Mo}}_{3}{\mathrm{O}}_{8}$ by using comprehensive characterization methods of x-ray diffraction, high-angle annular dark-field images, magnetization, specific-heat capacity, dielectric and electric polarization measurements. The Fe/Ni honeycomb lattice is endowed with strong magnetic frustration. Through Fe doping, a metamagnetic spin flop (SF) transition and the related linear magnetoelectric effect are achieved under a lower magnetic field, which is due to the delicate modification of magnetic interactions. Consequently, the maximum direct ME coefficient ${\ensuremath{\alpha}}_{\mathrm{H}}$ ($=238\phantom{\rule{0.16em}{0ex}}\mathrm{ps}\phantom{\rule{0.16em}{0ex}}{\mathrm{m}}^{\ensuremath{-}1}$) and converse ME coefficient ${\ensuremath{\alpha}}_{\mathrm{E}}$ ($=110\phantom{\rule{0.16em}{0ex}}\mathrm{ps}\phantom{\rule{0.16em}{0ex}}{\mathrm{m}}^{\ensuremath{-}1}$) were achieved in ${\mathrm{Ni}}_{0.5}{\mathrm{Fe}}_{1.5}{\mathrm{Mo}}_{3}{\mathrm{O}}_{8}$ at the phase boundary of antiferromagnetic-SF and SF phase, respectively.