A study has been conducted on construction site object detection utilizing YOLOv8 model, employing a comprehensive methodology, when comparing YOLOv5 and YOLOv8, both renowned for image and video processing, our analysis centers on crucial factors: architecture, performance, and practicality, guiding us in discerning their respective strengths and limitations. Initially, a dataset is being collected, consisting of images featuring construction site workers and five target classes: Helmet, Vest, Gloves, Boots, and Humans. Around 2092 images were gathered from internet databases and actual construction sites, ensuring diversity in angles, lighting conditions, and contexts. Challenges encountered during data collection included capturing a comprehensive range of shots, managing occlusion and clutter, and addressing issues with data clarity from high vantage points. Following data collection, data cleaning procedures were rigorously implemented to ensure the integrity and quality of the dataset. This encompassed comprehensive measures to reduce noise, handle occlusions, and account for variability in lighting conditions and scene complexities. To reduce noise, advanced image processing techniques such as filtering and segmentation were applied, effectively eliminating irrelevant elements such as debris and vehicles. Additionally, sophisticated annotation methods were employed to accurately annotate obscured objects and incorporate diverse occlusion scenarios, thereby enhancing the model's robustness. Furthermore, augmentation techniques were utilized to address variability in lighting conditions and scene complexities, ensuring the dataset's adaptability. Consistent annotations were maintained through strict adherence to labeling conventions and regular quality checks. Moreover, efforts were made to balance class distribution by collecting additional samples for underrepresented classes and employing resampling techniques where necessary. Overall, these rigorous data cleaning procedures were instrumental in preparing a high-quality dataset for subsequent analysis and model training, ensuring accurate and reliable results without compromising on integrity or quality. Accurate annotation of objects using rectangular bounding boxes and class labels is crucial, facilitated by manual annotation using the LabelImg tool. The YOLOv8 model underwent training and optimization, with evaluation of various model variants to strike a balance between precision and computational efficiency. Hyperparameter tuning techniques were applied, including adjustments to learning rate, batch size, and optimizer configurations. The efficacy of the trained YOLOv8 model has been assessed using standard metrics such as precision, recall, and mean average precision. Overall, the thorough approach ensured that the YOLOv8 model has been trained and optimized to effectively detect objects of interest on construction sites, thereby enhancing safety measures and monitoring capabilities in real-world scenarios.
Table 1
The identified TP, TN, FP, FN values from YOLOv8 detection model
Parameters | TP (True Positive) | TN (True Negative) | FP (False Positive) | FN (False Negative) |
---|
Boots | 560 | 1265 | 33 | 30 |
Gloves | 30 | 1837 | 11 | 30 |
Helmets | 307 | 1553 | 13 | 15 |
Human | 360 | 1483 | 16 | 20 |
Vest | 450 | 1399 | 107 | 22 |
Table 2
Performance metrics calculated for YOLOv8 model
Parameters | Accuracy | Recall | Precision |
---|
Boots | 96.6% | 94.9% | 94.4% |
Gloves | 98.9% | 75% | 73.1% |
Helmets | 98.5% | 95.3% | 95.9% |
Human | 97.6% | 94.7% | 95.7% |
Vest | 97.9% | 95.3% | 96.3% |
The Table 1 presents the metrics for evaluating the performance of the model, including True Positives (TP), False Positives (FP), False Negatives (FN), and True Negatives (TN) that has been obtained from the trained dataset of 2092 images for the epoch of 20. Using the values obtained from Table 1, the performance metrics such as Accuracy, Recall and Precision have been calculated and illustrated in Table 2. The formula for calculating the Precision, Accuracy and Recall are given in equations (1), (2), and (3) respectively. Such metrics offer significant insights into the model's efficacy in accurately identifying and categorizing objects, thereby contributing to informed decision-making across diverse applications.
$$Precision=\frac{True Positives\left(Tp\right)}{True Posivites\left(Tp\right)+False Posivitves\left(Fp\right) }$$
1
$$Accuracy=\frac{Tp+Tn}{Tp+Tn+Fp+Fn}$$
2
$$Recall=\frac{True Positives\left(Tp\right)}{True Positives\left(Tp\right)+ False Negatives \left(Fn\right)}$$
3
From Fig. 7 we can depict a model's performance measures throughout several training epochs. Here epochs indicate both the individual epoch number and the total epoch count. For instance, "17/20" indicates that this era is the 17th of 20th total epochs. With 20 epochs, the YOLOv8 object detection training procedure started. Several indicators were tracked during the training phase in order to evaluate the model's development and performance. The particulars of each epoch, which includes processing speed, instance counts, GPU memory utilization, and loss values, were noted and analyzed.
During the initial epoch, GPU memory utilization peaked at 7.16 GB, accompanied by box, cls, and dfl losses of 1.017, 0.079, and 1.239, respectively, with the model exhibiting a size of 640 and detecting 107 instances at a processing speed of 1.88 iterations per second. Promising results across evaluation metrics were observed, indicating potential performance enhancements with continued training. Subsequent epochs displayed consistent trends, albeit with variations in processing speeds, instance counts, and loss values. Notably, metrics such as precision, recall scores, and loss values demonstrated positive trends, culminating in a significant decrease in box, cls and dfl losses to 0.5322, 0.2684, and 0.9351 respectively, by the 20th epoch, with the GPU memory usage stabilizing at 7.42GB. Despite identifying fewer instances, the model showcased notable performance improvements achieving a processing speed of 1.97 iterations per second and a remarkable increase in mean average precision at 50% IoU (mAP50) from 71.9–91.9% over the course of training. The efficiency of the training process is being further underscored by completing all 20 epochs within a mere 0.850 hours or 51 minutes, highlighting both the effectiveness of the training setup and the model's rapid learning capabilities. This thorough analysis highlights how model training progresses iteratively, stressing the importance of steady performance improvements and increased confidence in the model's abilities, which are essential for practical applications across different domains.
In Fig. 8, a confusion matrix is presented, a fundamental tool frequently utilized to assess the performance of classification models on a designated test dataset where true values are known. This particular matrix originates from a model designed to classify images into five distinct categories: boots, gloves, helmets, humans, and vests. The matrix is structured with the true labels on the y-axis (vertical axis) and the predicted labels on the x-axis (horizontal axis). Each cell within the matrix denotes the count of predictions made by the model, with the rows representing the true labels and the columns representing the predicted labels. For instance, the cell located in the 'boots' row and 'boots' column indicates 560 true positive predictions for boots.
The primary diagonal of the matrix, spanning from the top left to the bottom right, showcases the number of accurate predictions for each category, where the predicted label aligns with the true label. Ideally, in a perfectly performing classifier, all values would lie exclusively on this diagonal, with off-diagonal cells registering zero, signifying no misclassifications. However, upon inspection of this matrix, it becomes evident that the model has excelled in certain classes (such as 'human' and 'vest') but has also exhibited misclassifications, including instances of confusing 'gloves' with 'background' or 'boots' with 'vest'. The colour scale provided on the right side of the matrix denotes intensity corresponding to the number of observations, with darker shades typically indicating higher counts. This visual representation facilitates rapid identification of accurately classified categories and those frequently confused, thereby offering valuable insights into the model's performance nuances and areas for potential improvement.
In Fig. 9 Frame 103/322 shows a construction site, and it has been annotated that an object detection algorithm has been used to identify various elements. The object detection algorithm's study shows that there is a good level of confidence in detecting important components in the image, especially when it comes to safety and the presence of people at a work site. The algorithm shows a high degree of assurance, with most assurance scores being higher than 80%. It assigns assurance levels ranging from 0.71 to 0.94 and recognizes individuals, helmets, vests, and boots with proficiency. On the other hand, there are certain cases of label overlap and redundancy, particularly when people are identified repeatedly in particular image regions. The algorithm's segmented picture processing, which produces independent human feature detection, could be the source of this redundancy. Although this kind of duplication is typical in object detection, it emphasizes the need for sophisticated algorithms to maximize label assignments and improve overall efficiency. Despite these minor challenges, the algorithm's robust performance underscores its potential applicability in safety-critical environments, showcasing promising implications for real-world deployment and utilization.
After testing the trained model on both day and night lighting conditions, we concluded that the trained YOLOv8 model shows maximum accuracy possible even in dark lighting conditions as shown in Fig. 10, though it is observed that there is a slight drop in performance metrics of the night settings when compared to day settings. It is important to mention that though both images show different lighting conditions but they both are 2 different images as it wasn’t possible to capture the same scenario in different lighting conditions. Below is a comparison of the performance metrics of both day and night lighting condition.
Table 3
Performance metrics calculated for day and night lighting conditions
Lighting conditions | Accuracy | Recall | Precision |
---|
Day light setting | 98.01% | 94.36% | 94.9% |
Night light setting | 94.12% | 93.04% | 91.1% |
While the model's performance remains impressive in nighttime conditions, with an accuracy rate of 94.12% as shown in Table 3, there is a slight decrease compared to daytime conditions. However, this marginal drop is indicative of the model's consistent and commendable performance across varying lighting environments. The precision rate of 91.1% further validates the model's capability to effectively identify night lighting instances, showcasing its reliability in low-light scenarios. Additionally, the recall rate of 93.04% highlights the model's adeptness in accurately capturing the presence of night lighting conditions. Overall, the YOLOv8 model demonstrates exemplary performance across both daylight and nighttime settings, with its slight superiority in daylight conditions emphasizing its robustness and efficacy. These findings underscore the model's suitability and effectiveness in tasks requiring precise lighting condition detection, thereby affirming its value and applicability in practical scenarios.
Figure 11 presents a comprehensive depiction of diverse metrics tracked across 20 epochs during the training of a machine learning model, indicative of its deep learning nature, as inferred from the contextual relevance of the metric terminology. The horizontal axis spans from 0 to 20 epochs, representing the progression of training iterations, while the vertical axis denotes metric values. Each graph encapsulates performance metrics and loss values discerned during both the training (train) and validation (val) phases, providing an insightful visualization of the model's learning trajectory and performance evolution throughout the training process.
In the provided graphs (a, b, c), representing "train/box_loss", "train/cls_loss", and "train/dfl_loss" respectively, the consistent decrease in training box loss, classification loss, and directional feature loss with increasing epochs suggests an improvement in accuracy across box prediction, classification, and directional feature prediction. Similarly, graphs (d, e), depicting metrics for Precision and Recall, exhibit an increasing trend with the number of epochs, indicating enhanced precision and accuracy over time. This is attributed to the iterative learning process of the model, where each epoch iteration refines the model's understanding of the data. On the contrary, graphs (f, g, h), displaying validation box loss, classification loss, and directional feature loss, demonstrate plateauing trends, signaling potential model convergence or overfitting during the validation phase. The validation loss plateau may be attributed to the model's tendency to memorize the training data rather than generalize to new data, highlighting the need for regularization techniques to prevent overfitting. Moving to graphs (i, j), representing mean Average Precision (mAP) at different Intersection over Union (IoU) thresholds, the increasing trend over 20 epochs indicates improved precision in object detection as training progresses. This is due to the model's ability to better discern objects from the background and accurately localize them within the image. Overall, the increase in epochs allows the model to iteratively adjust its parameters based on the training data, leading to improved performance metrics such as precision and recall. However, the plateauing of validation loss suggests a need for caution to avoid overfitting and ensure the model's generalization to unseen data.