As one of the key technologies for autonomous driving, traffic sign detection and recognition (TSDR) remains a challenging task due to the difficulty and complexity of traffic signs. Moreover, real-world traffic signs exhibit a long-tailed distribution (i.e., data for most categories are scarce while others are abundant), which has negative impact on the detection model. In this paper, we propose a novel model to tackle these challenging problems. Two adaptive copy-paste methods are proposed for the long-tailed data problem to expand the tail categories efficiently. To detect difficult traffic signs, we propose roulette-based re-sampling to emphasis on data of difficult categories. Furthermore, an adaptive ensemble YOLOX model with double-level head is proposed which adaptively assembles hierarchical head that emphasizes features at different levels by learning the feature representation of level-aware traffic signs. An adaptive class box fusion module is utilized to further improve the ensemble efficiency. Extensive experiments on TT100K and Comprehensive TT100K datasets show that the method proposed in this paper achieves significant improvements in both difficult traffic signs and alleviates long-tailed data problems. It can effectively improve the miss-detection problem of difficult traffic signs and meet the TSDR requirements of real-time and high accuracy.