In this paper, a novel system is proposed to detect coconut trees from the images taken on an unmanned aerial vehicle (UAV). We propose a model based on the YOLOv4 Detector (CSPDarknet53) to achieve accurate prediction and fast speeds that enable real-time detection and object localization for a single object class-coconut tree. The pipeline is trained on two separate datasets of images from camera embedded UAV and images from multiple sources are fed into the multi-scale detector to predict bounding boxes and labels. A simple procedure is also proposed to enable detection on a larger scale, whereby the bigger image can be cropped into multiple single images and then fed into the detector. This model is compared statistically with the other state-of-the-art deep-learning models for object detection. After field studies and experiments on the images shot via a UAV (drone), it is proved that this system can efficiently and accurately detect coconut trees on a field at a speed of 15 frames per second when trained on 500 aerial images of the coconut trees.