Suppressing electron kinetic energy in solids through the creation of less dispersive flat bands (FBs) is a promising route to enhance many-body correlation effects1. In addition to mundane cases attributed to atomic orbital localizations [i.e., flat atomic bands (FABs)], wavefunction overlapping and electron hopping processes are preserved in flat topological bands (FTBs), where the suppression of kinetic energy arises from destructive interference effects2,3. Examples of FTBs are obtained in artificially engineered van der Waals systems by creating moiré supercells that fold and flatten the band structure4-8. In twisted bilayer graphene (TBG), the flat bands are near the EF (Fermi energy), leading to correlated insulating states8-10, magnetism10,11, and strong-coupling superconductivity12,13 tuned by twist angle and electron doping. However, due to the large size of the moiré supercell in real space, the carrier densities in those moiré samples are usually low2. In addition, the existence of flat bands (FBs) and the corresponding quantum phases in TBG have strict stacking-angle requirements5, posing challenges to attain practical samples at the meso/macroscale, which is essential for technological applications.
Although some of limitations might be overcome in natural crystalline materials, the FTB has yet to be realized in stoichiometric materials with high quality. Recent theoretical efforts have addressed this issue by predicting a large library of candidates (e.g., KAg (CN)2, and Pb2Sb2O7) via high-throughput calculations2. Experimentally, extensive research has been performed on transition metal kagome magnets14–19, such as Fe3Sn2, Co3Sn2S2, and YMn6Sn6, where the transition metal atoms form a corner-shared triangle network. Due to the nonnegligible hopping along the z-direction, the flat bands in these 3D compounds usually occupy a limited momentum space and/or are not isolated in the energy space18–20. In this sense, single atomically thin crystals should be a preferable platform to manifest 2D kagome physics in a neat way. A colouring-triangular lattice has recently been visualized in a Mo5Te8 monolayer, in which two sets of kagome bands are found locating away from the Fermi level by tens of meV21. Thus, the correlated phenomena driven by the FBs right at the EF have yet to be realized in the single-atomic-layer limit.
In this work, we show that by well-controlled postgrowth annealing, telluride atoms in monolayer H-MoTe2 desorb, transforming the monolayer into a new phase (MoTe56/33) exhibiting a lattice constant of 19.1 Å. The MoTe56/33 monolayer contains an ultradense, uniformly sized mirror twin boundary (MTB) superlattice, which is thermally favourable under a certain range of Te chemical potentials. We emphasize that the MTBs form a periodic lattice rather than the previously identified local line defects with metallic defect states22–24 or harbouring 1D correlation phenomena, such as 1D charge density waves (CDWs) or Luttinger liquids23–26. By combining scanning tunnelling microscopy (STM), noncontact atomic force microscopy (nc-AFM), and theoretical calculations, we reveal that the formation of an MTB supercell results in FTBs residing inside the pristine band gap of H-MoTe2. Two sets of kagome bands are mixed across the EF, which results in partially filled bands sitting at the Fermi level, where a hard gap of 15 meV was experimentally measured at 0.35 K. This gap was elucidated to be a correlated one by temperature-dependent spectroscopic measurements. In addition, magnetization-polarized Zeeman shifts are observed for the occupied and unoccupied bands, suggesting that the correlated insulating states are spin polarized. This work demonstrates a new route for materializing exotic FB-driven quantum phases in a vdW monolayer via easy and scalable structural engineering.
H-phase monolayer MoTe2 was first synthesized on bilayer graphene (BLG)/SiC (0001) by utilizing a delicate MBE method [see Methods]. We obtained a phase-temperature diagram of the Mo-Te components (Fig. S1), in which various phases or structures were identified after postgrowth annealing from the H phase. MTBs formed under annealing temperatures from 280°C to 500°C, and Mo5Te821 and Mo6Te629 phases start to appear when Tannealing > 500°C. A new ordered superstructure phase can be observed dominating the monolayer sample prepared at Tannealing = 450 ℃. The zoomed-in images of this superstructure (Fig. 1b) show the apparent wagon-wheel morphology, which is composed of six lines (spokes of the wagon-wheel) encircled in a skewed hexagonal pattern. The periodicity of this superlattice is 1.91 nm. The line boundaries appear as bright stripes at high sample bias (2.0 V in Fig. 1b), and they manifest as dark trenches at low sample bias (0.8 V in Fig. 1c). These morphologic characteristics resemble the STM results of the MTBs formed in other H-phase TMDs27,28. An argument was raised in recent work that this 1.91 nm superperiodicity in MoTe2 could be a moiré pattern29. In Fig. S2, we show that its appearance is independent of the twist angle relative to the BLG substrate, and it could also be prepared on a WSe2 substrate, thus ruling out the possibility of a moiré pattern.
Frequency-shift nc-AFM images (Figs. 1d and 1e) offer further information on the atomic structures. According to previous nc-AFM studies of the MTBs26,30 in TMDs, only the top-layer Te atoms are imaged here. There are discernible and reversible (in the repulsive and attractive force regimes) contrasts in the Te atoms between the boundary and the domain regions, which can be attributed to the discrepancy of their relative heights and/or electron densities26,30. Each triangular domain consists of six top-layer Te atoms (i.e., three Mo-Te units) with an interatomic distance of 0.365 nm, nearly the same as that in pristine H-MoTe2. In both images, there is one Te atom at the rotational centre. Based on these observations, we propose an atomic structure for the observed superlattice, as schematically shown in Fig. 1f31. The validity of this model was further verified by its lowest formation energy among various candidate models in a certain range of the Te chemical potential (Fig. S4) and the agreements between experiment-theory comparisons of bias-dependent dI/dV images (Fig. S5). Te atoms on the MTB form closed triangular loops, which were previously termed MTB loops32. We denoted the MoTe2 domain enclosed by individual MTB loops as domain A (TDA), while domain B (TDB) is located among three MTB loops. The corner joint region of three MTB loops is denoted as the centre site. Given this detailed atomic model, we revealed the Mo:Te ratio of this new phase to be 33:56; therefore, the rigorous stoichiometry is MoTe2 − x with x = 10/33.
Although the notion of an MTB network has been previously reported, we emphasize that the present structure is distinctive in two aspects: the smallness and uniformity of the 1.91-nm periodicity over the whole monolayer sample. Therefore, a periodic potential should modify the electronic behaviour globally rather than the previously used local defect picture23,27. Figure 2a plots the differential conductance spectra (dI/dV) of both MoTe56/33 and H-MoTe2 monolayers for a clearer comparison. Inside the approximately 2 eV band gap of H-MoTe229, at least five pronounced peaks were observed in the MoTe56/33 spectrum, labelled P− 1, P0, P1, P2, and P3 in ascending order of energy. Among these peaks, P0 resides across the Fermi level along with a zero-bias dip feature. Moreover, the spatially dependent dI/dV spectra of four representative sites (Fig. 2b) indicate that the zero-bias dip feature of P0 presents, with varying weights, over major regions of the supercell. This result confirms that the observed peaks should be viewed as a property of the system as a whole instead of localized defect states. In Fig. S6, we thoroughly inspect the transition of local boundary states into delocalized states as the interboundary separation shrinks.
Figures 2c and 2e plot the density-functional theory (DFT) band structures in which the spin degree of freedom was non-polarized and polarized (with consideration of spin-orbit coupling, SOC), respectively. An on-site Coulomb potential U = 0.5 eV was used in all calculations, and the U dependence on electronic structures is discussed in Fig. S7. Notably, although the band/spin degeneracies are lifted in the spin-polarized band structure in tens of meV, both the spin-nonpolarized and spin-polarized band structures show comparable overall line shapes of their associated density of states (DOS), as shown in Figs. 2c and 2e. In particular, they both primarily resemble the spectroscopic peaks P− 1 to P3, with one peak located right at the EF. The zero-bias dip on P0 is absent in both DFT DOSs, indicating a likely strong correlation nature, which we discuss later.
These DOS maxima correspond to a cascade of nearly dispersion-free bands (namely, FBs), as presented in the band structure. For simplification, we use the spin-nonpolarized band structure (Fig. 2c) to discuss the origin of these nearly flat bands. Near the Fermi level (− 0.3 eV to + 0.1 eV), there are six bands with a relatively simple structure. In a close-up panel on this energy window (Fig. 2d), we label them B1–B6 in descending order of energy. B1–B3 overlap in energy around the Fermi level, which corresponds to the P0 states. B4–B6 are located at slightly lower energy levels and correspond to the P− 1 states. Our subsequent discussions are mainly focused on these six bands near the Fermi level, while a brief discussion for P1–P3 is provided in Fig. S5.
We next unravel the origin of the dispersionless B1–B6 bands as kogame flat bands based on the symmetry analysis. The MTB lattice has the space group P\(\stackrel{-}{6}\) (No. 174) and the C3h point group, which includes a threefold rotation \({\widehat{C}}_{3z}\) around the out-of-plane \(\widehat{z}\)axis and a mirror symmetry \(\widehat{M}\)by the \(xy\)plane. The six bands of B1–B6 can be classified into two groups that are odd (B3, B4, B5, as FBs1) and even (B1, B2, B6, as FBs2) under the \(\widehat{M}\) operation (details in Supplementary Table S1), as marked by the green and brown lines in Fig. 2d, respectively. In the mirror odd sector, bands B3, B4, and B5 have the prototypical energy dispersion of a kagome lattice without inversion symmetry. The B5 band is nearly flat and touches the B4 band at the \({\Gamma }\) point (Brillouin zone centre), and the B3 and B4 bands are separated by an energy gap at the \(K\) point (Brillouin zone corner). The formation of an effective kagome lattice by these three bands can be established theoretically using band representation theory33. The irreducible representations (irreps) of the three bands are calculated to be \({{\Gamma }}_{2}+{{\Gamma }}_{4}+{{\Gamma }}_{6}\) and \({\text{K}}_{2}+{\text{K}}_{4}+{\text{K}}_{6}\) at \({\Gamma }\) and \(K\) points, respectively (Fig. 2d). These symmetry data are the same as those for bands formed by Wannier orbitals placed at the \(3j\) Wyckoff positions, i.e., three inequivalent sites per unit cell that are related by threefold rotation34. From geometry, the \(3j\) Wyckoff positions indeed form an effective kagome lattice.
Moreover, the effective kagome lattice can also be revealed by the real-space distribution. Notably, the three bands within one set of FBs have similar wavefunctions (Fig. S8); here, we take the \({{\Gamma }}_{4}+{{\Gamma }}_{6}\) states from the B4 and B5 bands as an example (Fig. 2h), where the local density of states (LDOS) peaks along the MTBs. The calculated result closely matches the experimental conductance mapping for the P− 1 state (Fig. 2f). The centres of the boundaries form a breathing kagome lattice (illustrated by the grid in Figs. 2f and 2h) that is akin to the line graph of a honeycomb lattice35. Based on similar analysis, FBs2 can also be attributed to kagome bands and have good consistency with experimental mapping for the P0 state, as shown in Figs. 2g and 2i.
Next, we discuss the puzzling dip at the Fermi level. In Fig. 3a, we present high-resolution tunnelling spectroscopy within a zoomed-in energy range (± 100 meV) taken at 0.35 K. A hard gap can be determined unambiguously with Δ0 ~ 15 meV for the zero-conductance region. The magnitude of Δ0 persists among different locations, although minor differences in the line shapes occur. To determine the origin of this hard gap, we measured its evolution with temperature. As shown in Fig. 3b, the hard gap continuously narrows with increasing temperature and closes above 12 K. However, the observed gap closing cannot be explained only by thermal broadening. In Fig. 3c, we take the lowest-temperature spectrum (i.e., 0.35 K, black curve) to simulate the thermal broadening with T = 20 K (red curve, see Supplementary Note 1), which deviates significantly from the measured curve at 20 K (grey curve). To quantify the gap size over a wide temperature range, we define ΔΝ by the midpoints of the edge slopes. The inset shows the simulated thermal broadening of ΔΝ and the experimentally measured values as a function of temperature, where the experimental results exhibit a rapid decay. This spectroscopic behaviour demonstrates the correlated nature of the observed gap36,37 rather than the single-particle gap observed in the cousin system of monolayer Mo5Te821.
Regarding the mechanism of gap formation, the CDW gap and the quantum confinement effect can first be ruled out by supplementary experimental results, in which no charge modulations was seen in the bias-dependent STM imaging (Fig. S9), and the gap size is independent of the monolayer flake size (Fig. S10). The possibility of the superconducting gap is also excluded as the gap cannot be suppressed by applying magnetic fields. However, other responses of the insulating states to the magnetic field appear surprisingly. Figure 4a shows the dI/dV spectra of the correlation gap taken under magnetic fields applied perpendicularly to the sample basal plane (i.e., z direction). The unoccupied and occupied edges both shift progressively towards lower energies with increasing field magnitude independent of the + z or -z field direction. For quantitative analysis, we take the peak positions in the second derivative spectra (Fig. 4b) to represent the energy locations of gap edges. The energy shifts ΔE for both unoccupied and occupied edges exhibit a linear relationship with the magnitude of B, conforming to the Zeeman-type energy shift. The sign of ΔE regardless of the field direction can be understood as a magnetization-polarized Zeeman effect16,17, which indicates the following facts: net magnetic moments exist (at least locally) in the MoTe56/33 monolayer and the moments m are always parallelly aligned to the external field direction (in our measurements, a nonzero minimum |B| is 2 T). The values of the effective magnetic moment (meff) can be extracted from the linear fitting of the Zeeman shift ΔE = − meff·B (as labelled in Figs. 4c and 4d)16,17. With statistical analyses of multiple measurements over the superlattice (Figs. 4e and 4f), we determined meff to be 0.4 ± 0.3 µB and 3.0 ± 0.3 µB for the unoccupied and occupied states, respectively.
Such net magnetic moments cannot be solely attributed to the magnetizations induced by an external field (e.g., Pauli paramagnetism)38 by considering the following facts: the B-independent moments derived from the linear energy shifts and no signature of splitting in the spectral line shapes under the magnetic field. In addition, our data show no saturation of the energy shifts to the maximum field (12 T) applied in our experiments (Figs. 4c and 4d), which is not favourable for the possibility of spin-orientation-dependent quantum phases, such as those seen in Fe3Sn239,40. Therefore, we conclude that with the formation of the MTB superlattice, monolayer MoTe56/33 possesses spontaneous magnetizations (at least locally), and the correlated insulating state might be closely interrelated with the magnetic structure. Moreover, the moment values are greatly different between the occupied and unoccupied sides, and the moment extracted for occupied edge states (~ 3 µB) is significantly larger than a pure spin Zeeman effect (1 µB). This finding suggests that the orbital movements are nonnegligible in the total magnetism of our system.
Although DFT, by coincidence, captures the spin polarized state in a local moment picture, it is insufficient to reproduce the observed insulating gap formed at the Fermi level well (see discussions in Fig. S7). In the spin-nonpolarized case (Fig. 2c), three bands (B1–B3) contribute to the P0 states. The SOC lifts the kagome band degeneracy at the Γ point (ΔSOC ~ 20 meV), and the spin degeneracy at the Γ point is lifted when the spin-exchange interaction is considered (Δex ~ 100 meV). With such splittings, a pronounced energetic separation is observed between the two spin components of B1–B3 (Fig. 2e), but EF crosses one branch of B2 and B3. A comprehensive understanding of the magnetic properties (e.g., ground states, long-range order, and gap origin) demands combinations of multiple analytical techniques, especially tuning the filling factor, and theoretical simulations beyond the local moment picture41,42. This investigation will be a follow-up work because it is beyond the scope of the present work, which aims mainly to introduce a new 2D crystalline structure with FB-driven correlated physics.
In conclusion, we have achieved a highly controllable and scalable conversion of a monolayer nonmagnetic semiconductor into an FB-driven correlated system by a carefully designed thermal process. Theoretical calculations confirm the formation of multiple kagome flat bands in the MTB superlattice. In the partially filled bands, the electron correlations lead to a gap opening as large as 15 meV. The magnetization-polarized Zeeman shifts provide strong evidence of spontaneous spin polarizations. Further studies are expected to investigate the doping dependence of the correlated insulator by tuning the carrier density. Our work presents a new strategy for designing monolayer vdW materials that manifest strong correlation effects, and it opens versatile opportunities for tailoring nontrivial emergent quantum phenomena.