Boron-doped diamond thin films exhibit extensive applications in chemical sensing, in which functionalized nanostructures on the surfaces enhances further the performance of these films. However, targets detecting within such nanostructures faces great challenges such as noise, unclear object boundaries, and mutual occlusion, leading to inaccuracies in existing detection models. To tackle these challenges, we optimized the YOLOv8 model and introduced DWS-YOLOv8 for target detection of diamond nanostructures. The integration of the Deformable Convolutional C2f (DCN_C2f) module into the backbone network allowed adaptive adjustment of the network's receptive field. Moreover, incorporating the Shuffle Attention (SA) mechanism effectively addressed detail loss during convolutional iterations and reduced noise's impact on prediction results. Finally, leveraging Wise-IoU (WIoU) v3 as the bounding box regression loss enhanced the model's focus on diamond nanostructure samples, thereby improving localization capability. Experimental results showcase that compared to YOLOv8, our model achieves a 9.4% higher detection accuracy with reduced computational complexity. Furthermore, the recall rate (R) saw an increase of 0.6%, [email protected] improved by 2.6%, and [email protected]:0.95 increased by 0.6%. Additionally, DWS-YOLOv8 demonstrated enhancements in precision (P), recall (R), [email protected], and [email protected]:0.95, validating the effectiveness of our approach in enhancing target detection performance.