Market saturation coupled with intense competition in emerging markets have driven manufacturers to streamline their production processes and strategies for high production yield in order to have competitive edges for market leadership. Defective products escalate costs and disrupt production [1, 2]. Timely defect detection enhances quality control, stabilizes production and maintain competitive edge over competition [3].
In the meantime, with technology advancement coupled with process breakthrough, electronic components continue to miniaturize along with the increase in component density. These have resulted in soldering defects occur in microscopic and complex form, demanding advanced and effective computer vision capability. These challenges have been actively instigated by many researchers to develop accurate inspection algorithms [4, 5]
Using AOI to detect solder joint defect, of which not visually discernible, is the most common technology in the manufacturing revolutionizing industrial automation environment [6, 7]
Recent years, extensive research to enhance vision inspection algorithms with deep learning and reinforced learning techniques were reported, which promise significantly improve the detection and classification accuracy [8, 9, 10, 11, 12].
In the meantime, numerous studies were conducted to develop industrial applicable real-time vision inspection system [13, 14, 15, 16].
Nevertheless, the efforts to enhance the accuracy particularly for inferring minuscule, curved and specular reflective solder defects often lead to slower processing speed, which in turn hinders the practical application in the realm of manufacturing industry.
This paper aims to comprehensively analyse the processing and analysing time of the entire image processing workflow including imageacquisition, enhancement, ROI localization and segmentation, feature extraction, defect detection and classification. (See Fig. 1)
Potential process synergizing that could accelerate the overall processing speed while maintaining the accuracy of defect detection and classification is identified.
A novel vision inspection technique is proposed and the performance of the recommended methodology is assessed based on industrial actual PCB to validate for applicability under real-time settings.
(a) Image Acquisition
Recent advancements in machine vision technology for image acquisition to enhance the accuracy and efficiency of solder joint inspections, particularly within the industrial manufacturing settings, are cantered around developing cutting-edge, high-performance hardware infrastructure solutions:
1. Advanced Optical Lighting System: Innovations in lighting techniques, such as adaptive lighting schemes and stroboscopic light source are being employed to mitigate shadow and reflection effects that can obscure defect during optical inspections [17, 18, 19, 20]. These systems optimize the lighting conditions according to the surface characteristics of the solder joints.
2. High-Resolution Camera Sensor: The introduction of high-resolution compact sensor has led to substantial improvements in the clarity and detail of images captured during solder joint inspection [21, 22, 23, 24]. This advancement enhances the ability to detect even the smallest defects that could critically affect the reliability of the joints.
3. Accelerated Computing Power: State-of-art high performance processor particularly GPU from leading technology providers have dramatically boosted the computation power by leaps and bounces [25, 26].
All the abovetechnological breakthroughshave greatly reduced the research efforts
required to ensure obtaining high quality images, a prerequisite for efficient and accurate inspection results [27, 28].
(b) Image Enhancement
Many image enhancing techniques can be used to improve the image quality.
Yan et al. demonstrated that wavelet thresholding method is a powerful approach for noise reduction [29]. Fukushima et al. proposed an extension of guided image filtering for smoothing and edge enhancement [30]. Guo and Wan suggested improving images by adjusting the background data [31]. Ma et al. introduced a reflection image enhancement algorithm to make the quality of the images captured better [32]. Gao et al. employed an improved Retinex algorithm to augment the colour images [33]. Al-Hashim et al. used Retinex-Based Multiphase Algorithm for Low-Light Image Enhancement [34]. Roman et al. proposed a contrast enhancement algorithm to boost the colour images [35] while Shi et al. utilized an improved adaptive scene transformation algorithm to intensify the images [36].
However, with cutting-edge commercial machine vision hardware (e.g. high resolution Cognex In-sight 3800 Vision System embedded with robust rule-based algorithm and innovative learning technology), high-quality images can be acquired with reduced noises and enhanced features (see Fig 2)
Hence, these excellent images captured can do away with the requirement for noise removal, contrast adjustment, filtering, normalization and image enhancement processes.
(c) ROI Localization and Segmentation
ROI localization needs to accurately detect the position where each and every solder joint is formed in a given FOV image. With FOV of size 40x40 mm, it can have approximately 30 to 40 solder joints depending on the PCB design.
In general, ROI localization is carried out through a series of object detection methods. The ROI localization algorithms can basically group into 3 approaches namely template-matching; image histogram and clustering of pixels [37].
Researches done including Rapid Object Detection investigated by Viola and Jones [38], Histogram of Orientated Gradient plus Support Vector Machine proposed by Shumin et al. [39], Deformable Part Model used by Felzenszwalb et al. [40], Selective Search studied by Uijlings et al. [41], Edge Boxes suggested by Zitnick and Dollar [42] as well as Fast Template Matching researched by Dou et al. [43].
Meanwhile, Alarcon-Herrera et al. proposed to utilize View Point Selection [44], Hao et al. put forward an innovative Adaptive Template [45] and Sibiryakov employed a new template matching methodology [46].
Through leveraging the PCB soldering pad information embedded in Gerber file used for PCB fabrication, ROI localization step can be avoided. During image acquisition, the prior known coordinates and solder pad shapes extracted will be used to superimpose on the FOV to locate the exact solder joint position. (see Fig 3)
(Note: a specific software is required to interpret, view and export the required data from the Gerber file)
(d) Feature Extraction, Defect Detection and Classification
Most of the machine vision inspection studies conducted either using learning algorithm or pattern matching approach for image feature extraction, defect detection and classification.
Wu et.al. put forward the ResNet on Improved Yolo v3 to extract the required features for defect classification [47] whereas Zhang et al. classified defects based on improved ResNet mode [48]. Zhang et al. proposed the Convolutional Neural Network-based multi-label classification method [49]. Wu et al. suggested an algorithm with logical shape features [50]. Goodfellow et al. applied Deep Learning methodology for classification [51]. Seul et al. conducted research using template matching operation to extract features from ROI [52]. Wu et al. based on Bayes and Support Vector Machine to classify solder defects [53]. Cai et al. proposed to useVisual Background Extraction Algorithm for automatically inspect the solder joint [54]. Abdelhameed et al. utilized the enhanced threshold-based segmentation method with Discrete Cosine to improve the defect detection capabilities [55]. Wu and Xu applied the transformed “eigensolder” feature to classify the solder joint [56].
Different features and learning algorithms are effective for detecting different types of soldering defects [3, 4, 5, 8, 9, 11, 37, 57], This paper does not conduct a comparative analysis of the accuracy and efficiency performance of these features and learning algorithms. The rationale is that each algorithm possesses distinct analytical strengths tailored to specific defect categories, and any modifications intended to enhance the processing speed could lead to trading-off the accuracy.
Processing Time Estimation
For comparison study purposes, following assumptions are made
(1) An industrial PCB having 1000 electronics components with 3000 solder joints
(2) Image size of 140x70 pixels per solder joint
(3) NVIDIA GeForce RTX 2060 series GPU
(4) Multiscale Morphology Algorithm for image enhancement
(5) Template Matching Algorithm for ROI localization
Under these conditions, the estimated image enhancement time would be approximately 3 secondsper PCB.
Meanwhile, the estimated time required for ROI localization is around 2 seconds per PCB.
In summary, proposal to leverage the state-of-art vision hardware to eliminate image enhancement coupled with directly performing ROI localization using extracted PCB solder pads coordination could save the inspection cycle up to 5 seconds per PCB.