All the experiments were carried out in ultrahigh vacuum (UHV) chambers with a base pressure of ~ 1×10− 10 mbar (see Methods). First, a Bi(111) thin film was grown on a clean Si(111) surface to 10 BLs by molecular beam epitaxy (MBE) 39,40. For some of the ARPES experiments, a 250 BL-thick Bi(111) film was grown on a clean Ge(111) surface 41,42. Here, we adopt the rhombohedral crystallographic notation to describe the plane index of the film 20,21. The crystallinity of the prepared samples was confirmed with scanning tunneling microscopy (STM) and low-energy electron diffraction (LEED). The surface of the Bi(111) film was then covered by 1–2 BL Sb (for LEED patterns, see Supplementary Information A). Since Bi(111) and Sb(111) bilayers share the same buckled honeycomb structure with similar lattice constants of 0.454 nm and 0.431 nm, respectively, a Sb(111) film could grow epitaxially on a Bi(111) film 43,44. However, the lattice constant of the free-standing Sb bilayer (1BL antimonene), which is predicted to be 0.408–0.412 nm 45,46, is significantly smaller than that of bulk Bi(111). This allows antimonene to grow nonepitaxially on Bi(111) and to form a moiré superlattice, which we indeed observe as follows.
The main panel of Fig. 1a shows a representative STM image of a Bi(111) surface covered with more than 1 BL of Sb. The lower terrace in the image (Region I) features a triangular lattice structure, which is a moiré superlattice made of 1BL antimonene (1BL Sb) on a Bi(111) surface. Although this superlattice includes defects and local deformations, the presence of a well-defined periodicity is clear from its fast Fourier transform (FFT) image (inset of Fig. 1a). From repeated experiments with different surface regions and samples, we determined the moiré lattice constant to be 4.70 ± 0.30 nm (Supplementary Information B). On the upper terrace, there exists another region of the moiré superlattice with a longer periodicity (Region II). Since the height difference of ~ 0.4 nm between Regions I and II (Fig. 1b) is approximately equal to the height of the Sb bilayer (0.374 nm for bulk) 20, Region II is identified as 2BL antimonene (2BL Sb) on a Bi(111) surface. Its moiré lattice constant was determined to be 6.59 ± 0.89 nm. The relatively large uncertainty is due to variations throughout different surface regions, presumably reflecting very small differences in energy. This 2BL Sb layer is bordered by another 1BL Sb layer (Region III), which is located in the upper-right corner of the image. Since they have almost the same topographic heights 20, the boundary (indicated by the dashed line) is identified as the location of a buried atomic step of the Bi(111) surface. Our repeated experiments indicate that antimonene layers grow from the step edges of Bi(111) surfaces (Supplementary Information C).
Magnified STM images of 1BL and 2BL Sb are displayed in Fig. 1c and Fig. 1d, respectively, where the Sb atomic lattices are clearly resolved. The moiré unit cells are indicated by the dashed parallelograms. We determined the lattice constant of 1BL Sb to be 0.415 ± 0.004 nm (Supplementary Information B). The fact that this value is greater than that of free antimonene (0.408–0.412 nm) 45,46 is attributed to the tensile strain exerted from the Bi(111) surface. Likewise, the lattice constant of 2BL Sb was determined to be 0.423 ± 0.005 nm. This value is closer to that of bulk Sb(111) (0.431 nm) than that of 1BL Sb, suggesting lattice relaxation toward the bulk crystal. Combined with the moiré lattice constant determined above, the ratio of the numbers of Bi and Sb atoms can be calculated. We find (NBi:NSb) = (10:11) for 1BL Sb and (NBi:NSb) = (13:14) − (17:18) for 2BL Sb, where NBi and NSb are the number of Bi and Sb atoms within the unit cells along the principal axis, respectively. Our FFT analysis of STM images over an extended area reveals that there is no twisting between the moiré and Sb lattices on average (Supplementary Information B), although there are some local deviations due to deformations. Furthermore, Fig. 1c, d shows that the surface is divided into three characteristic regions in terms of topographic height. By comparing these observations to previous reports on related moiré superstructures 47–51, we can safely assign them to the regions of the AA, AB, and AC stacking sequences (Fig. 1e). In the AA stacking, all atoms in the two layers are vertically overlapped, while only half of them are in the AB and AC stacking layers. Because of the significant buckling of the honeycomb lattice, the vertical distance between the overlapping atoms in the AB stacking is greater than that in the AA stacking. This leads to the lowering of the top layer by an attractive force. Conversely, in the AC stacking, the vertical distance between the overlapped atoms is smaller than that in the AA stacking. This leads to the raising of the top layer by a repulsive force. As a result of structural relaxation, the areas corresponding to the AB and AC stacking regions expand and shrink, respectively 47,48. These features are clearly observed in Fig. 1c, d.
The electronic states of the moiré superlattices and their spatial modulations were investigated by scanning tunnelling spectroscopy (STS). First, for 1BL Sb on Bi(111), dI/dV spectra were taken at the center of the AA stacking region at five locations, and this process was repeated for the AB and AC stackings. The selected spectral sites are shown in the topographic STM image in Fig. 2a with red (AA), blue (AB), and green (AC) squares. Figure 2b shows the results of the STS measurements. The broken lines show individual dI/dV spectra, and the solid lines show the average values for the same stacking sequences, with their colors corresponding to those in Fig. 2a. The average of all the measured spectra are also shown by the solid black line. For all of these spectra, clear peak structures are noticeable near the zero bias voltage, with the full width at half maximum of 80 − 100 mV. The spectral peaks at the AB sites are particularly conspicuous, while those at the AA and AC sites are relatively suppressed. More detailed information was obtained through line spectroscopy; dI/dV spectra were taken along a straight line connecting the centers of the AC, AB, AA, and AC sites in this sequence. The right panel of Fig. 2c shows a 2D plot of color-coded dI/dV spectra as a function of bias voltage and lateral distance from the starting point. In the left panel of Fig. 2c, the topographic profile along the line is also shown. We find that the spectral peak is fixed at approximately 0 − 30 mV, while its intensity varies. This result strongly suggests the presence of delocalized states around the Fermi level that are weakly modulated by the moiré superlattice.
Figure 2d-f displays site-dependent dI/dV spectra for 2BL Sb on Bi(111) obtained in the same manner. The symbols and colors used in the figure follow the conventions of Fig. 2a-c. Figure 2e shows that spectral peaks are shifted from zero bias by 48 mV (AA site), -48 mV (AB site), and 144 mV (AC site) on average. The right panel of Fig. 2f shows that clear peak structures at approximately 40 mV and − 100 mV are confined within the AA and AB regions, respectively. This result indicates the presence of multiple states near the Fermi level that are localized due to the moiré superlattice 52.
To clarify the origin of the spectral peaks observed by STS, we performed laser-based high-resolution ARPES/SARPES measurements 53. For simplicity, the data were analyzed based on the Brillouin zone of Bi(111) (Fig. 3a). First, we focused on the results for 1BL of Sb on a Bi(111) surface. Figure 3b shows a 2D plot of the ARPES intensity along the Γ-M direction and as a binding energy EB (dark: high, bright: low). We can recognize two bands, denoted S1 and S2, starting from an EB ≅ 0.2 eV and dispersing upward. These bands disappear at approximately kx = 0.05–0.1 Å−1 by crossing the Fermi level but seem to disperse downward and reappear at approximately kx = 0.4–0.5 Å−1. The band dispersions determined from the plot are highlighted with red dashed curves. The signals are better visualized by the SARPES signal plotted for the range of -0.22 Å−1 < kx < + 0.22 Å−1, where the intensity and the spin polarization in the y direction are indicated by brightness (dark: high, bright: low) and color (red: positive, blue: negative), respectively. The maximum spin polarization of the photoelectron is ~ 0.6. The spin polarizations of the S1 and S2 bands are opposite to each other and are antisymmetric with respect to kx = 0 Å−1. This is characteristic of Rashba-type spin-polarization, which will be discussed later. Figure 3c shows a similar 2D plot of the ARPES intensity along the \(\stackrel{-}{{\Gamma }}-\stackrel{-}{\text{K}}\) (ky) direction. The S1 and S2 bands are also noticeable (highlighted with red dashed lines), but the S2 band reaches a local maximum near the Fermi level at approximately ky = 0.1 Å−1 and then disperses downward.
Figure 3e shows the 2D plot of the ARPES intensity measured near the Femi level (EB = 0.02 eV) in the kx - ky space, which gives the Fermi surface contour. The central ring and a surrounding star-like structure are clearly noticeable and can be identified as the S1 and S2 bands, respectively. Notably, some parts of the S2 band appear very weak due to the anisotropic transfer of matrix elements during the photoemission process. By referring to the band dispersions in Fig. 3b-d, we can identify the areas indicated by the red ellipsoids as saddle points, where the S2 band takes a local maximum in the \(\stackrel{-}{{\Gamma }}-\stackrel{-}{\text{K}}\) direction and a local minimum in the orthogonal direction. This means that the van Hove singularity exists at the Fermi level 54 and explains the origin of the zero bias peak for 1BL Sb/Bi(111) described above. To confirm this result, we also performed Fermi surface mapping with an imaging-type ARPES instrument, which allows us to access a larger momentum space at a faster speed (Fig. 3f) 42. The acquired Femi surface well reproduces the features observed in Fig. 3e while better reflecting the sixfold symmetry expected from the C3 and time-reversal symmetries of the present system. We note that the band structure and the Fermi surface resemble those of Bi(111) and Sb(111) surfaces 21,40,41,55,56, while saddle points are absent near the Fermi level in the latter cases.
The distribution of spin polarization in momentum space was investigated with the same imaging-type instrument for 1BL Sb/Bi(111). Figure 3g shows a 2D plot of the SARPES intensity and spin polarization in the y direction (Py) measured near the Fermi level (EB = 0.03 eV). The observed signal is mostly attributed to the S2 band, the location of which is reproduced from Fig. 3f (red solid lines). Along the kx axis (white dashed line), the distribution of the Py signal is antisymmetric with respect to kx = 0. This is consistent with a Rashba-type spin polarization, as mentioned above. We should note that the actual distribution of spin polarization deviates from the ideal vortical form, as indicated by the reversal of Py with respect to the lines at ±60° to the kx axis (black dashed lines). An analogous behavior was also predicted and observed for a clean Bi(111) surface with a giant Rashba splitting 42,57,58.
We also performed ARPES/SARPES measurements of 2BL Sb on a Bi(111) surface (Supplementary Information D). These results are nearly identical to those for 1BL Sb/Bi(111) (Fig. 3), but the observed ARPES signals are clearer than those for 1BL Sb/Bi(111). This difference may be attributed to the better moiré periodicity observed with STM (Fig. 1a).
Ab initio calculations of the electronic structure of Sb/Bi(111) moiré superlattices are difficult because of the large number of heavy atoms involved within a moiré unit cell. To circumvent this problem, we carried out DFT calculations based on an epitaxial model consisting of 1BL Sb(111) on 5BL Bi(111) (Methods). Although this model does not include the effect of moiré periodicity, it can account for the overall band structures within the Bi(111) Brillouin zone. The structural relaxation within each stacking region in the actual moiré structure (Fig. 1c,d) rationalizes this treatment. Figure 4a shows the band diagrams calculated for the AB stacking sequence. The orange and blue rectangles correspond to the same marked areas in Fig. 3b,c. The sizes of the purple (light blue) circles represent the contributions of the top Sb (Bi) BL. Overall, the two bands starting from the \(\stackrel{-}{{\Gamma }}\) point below the Fermi level (designated by the red dashed lines) are mostly derived from the top Sb BL, indicating that they can be preferentially detected in surface-sensitive STM and ARPES measurements. Judging from their dispersions, they can be assigned to the S1 and S2 bands identified above (Fig. 3b-d). The same calculations for the AA and AC stackings give very similar band structures near the \(\stackrel{-}{{\Gamma }}\) point and around the Fermi level (Supplementary Information E, Fig. E1). Therefore, the presence of saddle points near the Fermi level is theoretically confirmed. These features are reflected in the projected density of states (PDOS) on the top Sb BL (Fig. 4c). The three sharp peaks indicated by the arrows at E – EF = 0.02–0.05 eV, corresponding to the AA, AB and AC stackings, are due to the van Hove singularity of the saddle point. The peak energies are very close to one another, reproducing the STS results shown in Fig. 2b,c. Assuming that the Fermi level is aligned near these peaks in real samples, we plot the Fermi surface contour of 1BL Sb(111)/5BL Bi(111) with AB stacking at E – EF = -0.02 eV (Fig. 4b). The central rings and a surrounding star-like structure are consistent with the ARPES results (Fig. 3e, f).
The same calculations were also conducted based on an epitaxial model of 2BL Sb(111) on 5BL Bi(111). The band structures and the Fermi surface (E – EF = -0.02 eV) obtained for the AB stack (Fig. 4d, e) resemble those obtained for the 1BL Sb(111) model (Fig. 4a, c) as well as the ARPES results (Supplementary Information D). This is also the case for the AA and AC stackings (Supplementary Information E, Fig. E2). However, the PDOSs around the Fermi level calculated for the AA, AB, and AC stacks exhibit more separated energies (Fig. 4f). Qualitatively, these results are in line with the STS data (Fig. 2e, f), but there are some clear discrepancies; e.g., the sharp peak at E – EF = 0 eV for the AC stacking has no corresponding structure in the STS data (Fig. 2e). This difference may be attributed to incomplete structural optimization of the 2BL Sb model, which results from our simplified models of fixing the locations of the Bi atom to those of the bulk Bi crystal (see Methods).