Quantum spin liquid and cluster Mott insulator phases in the Mo3O8 magnets

<jats:title>Abstract</jats:title><jats:p>We unveil the microscopic origin of largely debated magnetism in the Mo<jats:sub>3</jats:sub>O<jats:sub>8</jats:sub> quantum systems. Upon considering an extended Hubbard model at 1/6 filling on the anisotropic kagomé lattice formed by the Mo atoms, we argue that its ground state is determined by the competition between kinetic energy and intersite Coulomb interactions, which is controlled by the trimerisation of the kagomé lattice into the Mo<jats:sub>3</jats:sub>O<jats:sub>13</jats:sub> clusters, and the sign of hopping parameters, specifying the electron localisation at such clusters. Based on first-principles calculations, we show that the strong interaction limit reveals a plaquette charge order with unpaired spins at the resonating hexagons that can be realised in LiZn<jats:sub>2</jats:sub>Mo<jats:sub>3</jats:sub>O<jats:sub>8</jats:sub>, and whose origin is solely related to the opposite signs of intracluster and intercluster hoppings, in contrast to all previous scenarios. On the other hand, both Li<jats:sub>2</jats:sub>InMo<jats:sub>3</jats:sub>O<jats:sub>8</jats:sub> and Li<jats:sub>2</jats:sub>ScMo<jats:sub>3</jats:sub>O<jats:sub>8</jats:sub> are demonstrated to fall into the weak interaction limit where the electrons are well localised at the Mo<jats:sub>3</jats:sub>O<jats:sub>13</jats:sub> clusters. While the former is found to exhibit long-range antiferromagnetic order, the latter is more likely to reveal short-range order with quantum spin liquid-like excitations. Our results not only reproduce most of the experimentally observed features of the Mo<jats:sub>3</jats:sub>O<jats:sub>8</jats:sub> systems, but will also help to describe various properties in other quantum cluster magnets.</jats:p>