Small object detection has always been one of the most challenging tasks in the computer vision. Up to now, a prior bounding box is often applied to Unmanned Aerial Vehicle (UAV) image object detection. However, anchors need to be pre-set and not optimal for training data in many object detection algorithms. In 2022, the Diffusion Model was introduced in object detection method, in which the random boxes are employed. Inspired by this method, we propose a Switchable Atrous Convolution Network based on the Diffusion Model. First of all, the normalized Wasserstein distance (NWD) is led into measuring the similarity between the prediction box and the ground truth box. The purpose is to eliminate the influence of positional sensitivity on the matching between the predicted box and the ground truth box. Secondly, a Switchable Atrous Convolution (SAC) with different atrous rates and shared weights is introduced, and similar features at different scales are extracted. This can solve the challenge of significant scale variations of the same target in the image. Finally, Pixel shuffle and Upsampling operations are employed in the feature extraction backbone network to enhance the model's ability to detect small targets. The optimal results of 27.89% mAP on the VisDrone dataset and 8.42% mAP on the Tinyperson dataset demonstrate the effectiveness of the proposed model.