A Road Defect Detection System Using Smartphones

Xin H., Ye Y., Na X., Hu H., Wang G., Wu C., Hu S.

Real-time road quality monitoring, involves using technologies to collect data on the conditions of the road, including information on potholes, cracks, and other defects. This information can help to improve safety for drivers and reduce costs associated with road damage. Traditional methods are time-consuming and expensive, leading to limited spatial coverage and delayed responses to road conditions. With the widespread use of smartphones and ubiquitous computing technologies, data can be collected from built-in sensors of mobile phones and in-vehicle video, on a large scale. This has raised the question of how these data can be used for road pothole detection and has significant practical relevance. Current methods either use acceleration sequence classification techniques, or image recognition techniques based on deep learning. However, accelerometer-based detection has limited coverage and is sensitive to the driving speed, while image recognition-based detection is highly affected by ambient light. To address these issues, this study proposes a method that utilizes the fusion of accelerometer data and in-vehicle video data, which is uploaded by the participating users. The preprocessed accelerometer data and intercepted video frames, were then encoded into real-valued vectors, and projected into the public space. A deep learning-based training approach was used to learn from the public space and identify road anomalies. Spatial density-based clustering was implemented in a multi-vehicle scenario, to improve reliability and optimize detection results. The performance of the model is evaluated with confusion matrix-based classification metrics. Real-world vehicle experiments are carried out, and the results demonstrate that the proposed method can improve accuracy by 6% compared to the traditional method. Consequently, the proposed method provides a novel approach for large-scale pavement anomaly detection.

Alhussan A.A., Khafaga D.S., El-Kenawy E.M., Ibrahim A., Eid M.M., Abdelhamid A.A.

Self-driving car plays a crucial role in implementing traffic intelligence. Road smoothness in front of self-driving cars has a significant impact on the car’s driving safety and comfort. Having potholes on the road may lead to several problems, including car damage and the occurrence of collisions. Therefore, self-driving cars should be able to change their driving behavior based on the real-time detection of road potholes. Various methods are followed to address this problem, including reporting to authorities, employing vibration-based sensors, and 3D laser imaging. However, limitations, such as expensive setup costs and the danger of discovery, affected these methods. Therefore, it is necessary to automate the process of potholes identification with sufficient precision and speed. A novel method based on adaptive mutation and dipper throated optimization (AMDTO) for feature selection and optimization of the random forest (RF) classifier is presented in this paper. In addition, we propose a new adaptive method for dataset balancing, referred to as optimized hashing SMOTE, to boost the performance of the optimized model. Data on potholes in different weather conditions and circumstances were collected and augmented before training the proposed model. The effectiveness of the proposed method is shown in experiments in classifying road potholes accurately. Eleven feature selection methods, including WOA, GWO, and PSO, and three machine learning classifiers were included in the conducted experiments to measure the superiority of the proposed method. The proposed method, AMDTO+RF, achieved a pothole classification accuracy of (99.795%), which outperforms the accuracy achieved by the other approaches, WOA+RF of 97.5%, GWO+RF of 98.6%, PSO+RF of 98.1%, and transfer learning approaches, AlexNet of 86.8%, VGG-19 of 87.3%, GoogLeNet of 90.4%, and ResNet-50 of 93.8%. In addition, an in-depth statistical analysis is performed on the recorded results to study the significance and stability of the proposed method.

Zhou B., Zhao W., Guo W., Li L., Zhang D., Mao Q., Li Q.

Road surface condition detection is an important application for many intelligent transportation systems (ITSs). A manhole cover depression is one of the common factors affecting road conditions. Smartphones are equipped with different sensors, which can be used to collect image data and inertial data. A new large-scale manhole cover detection dataset is developed by using smartphones to collect road image data, and a hierarchical classification method based on the convolutional neural network is proposed in this paper. The proposed method first coarsely classifies the images into nonrainy and rainy types and then performs manhole cover detections based on the coarse classification results. As a result, the proposed method achieves an accuracy of approximately 86.3% for road manhole cover detection. Based on the observation that different degrees of manhole cover subsidence produce different degrees of inertial sensor data, this paper used a machine learning method, which can automatically classify the detected manhole covers into different degrees of subsidence, namely good, average, and poor. The average recalls, average precisions, and average F1-measures achieve approximately 87.3%, 86.9%, and 87.2% accuracy, respectively. The results show that the proposed approach can effectively detect manhole covers in different weather and road conditions, which can effectively reduce the cost of road manhole cover data collection and detection, providing a new method for road manhole cover detection. • Road manhole cover detection and classification using smartphones. • The hierarchical model improves the image detection accuracy of manhole covers. • A method to classify the subsidence of road manhole covers based on inertial data. • The method can detect manhole cover subsidence in different weather conditions.

Julio-Rodríguez J.D., Rojas-Ruiz C.A., Santana-Díaz A., Bustamante-Bello M.R., Ramirez-Mendoza R.A.

This work presents the development of a classification method that can contribute to precise and increased awareness of the situational context of vehicles, for it to be used in autonomous driving applications. This work aims to obtain a method for machine-learning-based driving environment classification that does not involve computer vision but instead makes use of dynamics variables from Inertial-Measurement-Unit (IMU) sensors and instantaneous energy consumption measurements. This article includes details about the data acquisition, the electric vehicle used for the experiments, and the pre-processing methods employed. This explores the viability of a method for classifying a vehicle’s driving environment. The results of such a system can potentially be used to provide precise information for path planning, energy optimization, or safety purposes. Information about the driving context could be also used to decide if the conditions are safe for autonomous driving or if human intervention is recommended or required. In this work, the feature selection process and statistical data pre-processing methods are evaluated. The pre-processed data are used to compare 13 different classification algorithms and then the best three are selected for further testing and data dimensionality reduction. Two approaches for feature selection based on feature importance and final classification scores are tested, achieving a classification mean accuracy of 93 percent with a real testing dataset that included three driving scenarios and eight different drivers. The obtained results and high classification accuracy represent a first approach for the further development of such classification systems and the potential for direct implementation into autonomous driving technology.

Agebure M.A., Oyetunji E.O., Baagyere E.Y.

Road surface anomaly detection and classification based on crowd-sourced smart phone sensor data has become an important area of research over the last decade due to its potential benefits to road maintenance. Previous studies focused on paved roads in which anomaly classification were modelled as single-staged events mostly using machine learning and threshold-based methods. Little or no attention has been paid to road type classification and anomaly detection and classification on unpaved roads, which constitute a larger percentage of roads in the developing world. In this paper, road condition classification is approached as a multi-tier activity, comprising of models for road type classification, anomaly classification models for paved roads as well as unpaved roads using a novel Spiking Neural Network (SNN) learning model. To demonstrate the viability of the proposed system, road condition data for the various tasks were collected via an Android Application developed by the authors from which statistical features were extracted and used to train and evaluate the models. Experimental results showed that the proposed SNN model yielded significantly higher classification performance when compared to a Support Vector Machine (SVM) and Multilayer Perceptron (MLP) trained and tested using the collected datasets and classification models reported in existing studies.

Ferjani I., Ali Alsaif S.

Road condition monitoring is essential for improving traffic safety and reducing accidents. Machine learning methods have recently gained prominence in the practically important task of controlling road surface quality. Several systems have been proposed using sensors, especially accelerometers present in smartphones due to their availability and low cost. However, these methods require practitioners to specify an exact set of features from all the sensors to provide more accurate results, including the time, frequency, and wavelet-domain signal features. It is important to know the effect of these features change on machine learning model performance in handling road anomalies classification tasks. Thus, we address such a problem by conducting a sensitivity analysis of three machine learning models which are Support Vector Machine, Decision Tree, and Multi-Layer Perceptron to test the effectiveness of the model by selecting features. We built a feature vector from all three axes of the sensors that boosts classification performance. Our proposed approach achieved an overall accuracy of 94% on four types of road anomalies. To allow an objective analysis of different features, we used available accelerometer datasets. Our objective is to achieve a good classification performance of road anomalies by distinguishing between significant and relatively insignificant features. Our chosen baseline machine learning models are based on their comparative simplicity and powerful empirical performance. The extensive analysis results of our study provide practical advice for practitioners wishing to select features effectively in real-world settings for road anomalies detection.

Aparna, Bhatia Y., Rai R., Gupta V., Aggarwal N., Akula A.

The presence of potholes on the roads is one of the major causes of road accidents as well as wear and tear of vehicles. In order to solve this problem, various techniques have been implemented ranging from manual reporting to authorities to the use of vibration-based sensors to 3D reconstruction using laser imaging. But all these techniques have some drawbacks such as the high setup cost, risk while detection or no provision for night vision. Therefore, the objective of this work is to analyze the feasibility and accuracy of thermal imaging in the field of pothole detection. After collecting a suitable amount of data containing the images of potholes under various conditions and weather, and implementing augmentation techniques on the data, convolutional neural networks approach of deep learning has been adopted, that is a new approach in this problem domain using thermal imaging. Also, a comparison between the self-built convolutional neural model and some of the pre-rained models has been done. The results show that images were correctly identified with the best accuracy of 97.08% using one of the pre-trained convolutional neural networks based residual network models. The results of this work will be helpful in guiding the future researches in this novel application of thermal imaging in pothole detection field.

Zheng Z., Zhou M., Chen Y., Huo M., Sun L., Zhao S., Chen D.

To discover the condition of roads, a large number of detection algorithms have been proposed, most of which apply machine learning methods by time and frequency processing in acceleration and velocity data. However, few of them pay attention to the similarity of the data itself when the vehicle passes over the road anomalies. In this article, we propose a method to detect road anomalies by comparing the data windows with various length using Dynamic Time Warping(DTW) method. We propose a model to prove that the maximum acceleration of a vehicle passing through a road anomaly is linear with the height of the road barrier, and it’s verified by an experiment. This finding suggests that it is reasonable to divide the window by threshold detection. We also apply a brief random forest filter to roughly distinguish normal windows from anomaly windows using the aforementioned theory, in order to reduce the time consumption. From our study, a system is proposed that utilizes a series of acceleration data to discover where might be anomalies on the road, named as Quick Filter Based Dynamic Time Warping (QFB-DTW). We show that our method performs clearly beyond some existing methods. To support this conclusion, experiments are conducted based on three data sets and the results are statistically analyzed. We expect to lay the first step to some new thoughts to the field of road anomalies detection in subsequent work.

Bustamante-Bello R., García-Barba A., Arce-Saenz L.A., Curiel-Ramirez L.A., Izquierdo-Reyes J., Ramirez-Mendoza R.A.

Analyzing data related to the conditions of city streets and avenues could help to make better decisions about public spending on mobility. Generally, streets and avenues are fixed as soon as they have a citizen report or when a major incident occurs. However, it is uncommon for cities to have real-time reactive systems that detect the different problems they have to fix on the pavement. This work proposes a solution to detect anomalies in streets through state analysis using sensors within the vehicles that travel daily and connecting them to a fog-computing architecture on a V2I network. The system detects and classifies the main road problems or abnormal conditions in streets and avenues using Machine Learning Algorithms (MLA), comparing roughness against a flat reference. An instrumented vehicle obtained the reference through accelerometry sensors and then sent the data through a mid-range communication system. With these data, the system compared an Artificial Neural Network (supervised MLA) and a K-Nearest Neighbor (Supervised MLA) to select the best option to handle the acquired data. This system makes it desirable to visualize the streets’ quality and map the areas with the most significant anomalies.

Sattar S., Li S., Chapman M.

• Monitoring road surface roughness and detecting anomalies using smartphone sensors. • A vibration-based approach to detect road surface anomalies. • Crowdsourcing mobile app to collect road surface condition data. • Machin learning approach to classify discomfort level of road surface anomalies. Road surface hazards affect the driving safety and comfort of road users. Recently, smartphones and mobile devices equipped with motion sensors such as accelerometers and gyroscope sensors have attracted researchers’ attention for the development of low-cost approaches for road surface monitoring. However, processing smartphone sensors to monitor road surface conditions is technically challenging due to dissimilar sensor properties, different smartphone placement, and also different vehicle mechanical properties. This study aimed to develop a hybrid method using threshold based and Machine Learning approaches for near real-time detection and classification of road surface anomalies using smartphone sensor data with higher-level accuracy. The proposed algorithm has self-adapting and self-updating capabilities to adapt itself to any type of smartphone and the dynamic behaviors of various vehicles and road surface conditions. A prototype is developed using MATLAB and ArcGIS to perform sensor data analysis, geocoding, geo-visualizing, and data querying for performance evaluation.

Guan J., Yang X., Ding L., Cheng X., Lee V.C., Jin C.

Automated pavement distress detection based on 2D images is facing various challenges. To efficiently complete the crack and pothole segmentation in a practical environment, an automated pixel-level pavement distress detection framework integrating stereo vision and deep learning is developed in this study. Based on the multi-view stereo imaging system, multi-feature pavement image datasets containing color images, depth images and color-depth overlapped images are established, providing a new perspective for deep learning. To alleviate computational burden, a modified U-net deep learning architecture introducing depthwise separable convolution is proposed for crack and pothole segmentation. These methods are tested in asphalt roads with different circumstances. The results show that the 3D pavement image achieves millimeter-level accuracy. The enhanced 3D crack segmentation model outperforms other models in terms of segmentation accuracy and inference speed. After obtaining the high-resolution pothole segmentation map, the automated pothole volume measurement is realized with high accuracy. • Stereo vision and deep learning were integrated for automated pavement crack and pothole segmentation. • Multi-feature image datasets containing 2D, 3D and enhanced-3D images were established by stereo vision. • A modified U-net embedding depthwise separable convolution was proposed for faster segmentation. • The deep learning efficiency using different types of images was investigated. • Automated pothole volume measurement was achieved based on 3D image segmentation.

Masihullah S., Garg R., Mukherjee P., Ray A.

In this paper, we propose a novel attention based coupled framework for road and pothole segmentation. In many developing countries as well as in rural areas, the drivable areas are neither well-defined, nor well-maintained. Under such circumstances, an Advance Driver Assistant System (ADAS) is needed to assess the drivable area and alert about the potholes ahead to ensure vehicle safety. Moreover, this information can also be used in structured environments for assessment and maintenance of road health. We demonstrate few-shot learning approach for pothole detection to leverage accuracy even with fewer training samples. We report the exhaustive experimental results for road segmentation on KITTI and IDD datasets. We also present pothole segmentation on IDD.

Wu C., Wang Z., Hu S., Lepine J., Na X., Ainalis D., Stettler M.

Road surface monitoring and maintenance are essential for driving comfort, transport safety and preserving infrastructure integrity. Traditional road condition monitoring is regularly conducted by specially designed instrumented vehicles, which requires time and money and is only able to cover a limited proportion of the road network. In light of the ubiquitous use of smartphones, this paper proposes an automatic pothole detection system utilizing the built-in vibration sensors and global positioning system receivers in smartphones. We collected road condition data in a city using dedicated vehicles and smartphones with a purpose-built mobile application designed for this study. A series of processing methods were applied to the collected data, and features from different frequency domains were extracted, along with various machine-learning classifiers. The results indicated that features from the time and frequency domains outperformed other features for identifying potholes. Among the classifiers tested, the Random Forest method exhibited the best classification performance for potholes, with a precision of 88.5% and recall of 75%. Finally, we validated the proposed method using datasets generated from different road types and examined its universality and robustness.

Baek J., Chung K.

Since the image related to road damage includes objects such as potholes, cracks, shadows, and lanes, there is a problem that it is difficult to detect a specific object. In this paper, we propose a pothole classification model using edge detection in road image. The proposed method converts RGB (red green and blue) image data, including potholes and other objects, to gray-scale to reduce the amount of computation. It detects all objects except potholes using an object detection algorithm. The detected object is removed, and a pixel value of 255 is assigned to process it as a background. In addition, to extract the characteristics of a pothole, the contour of the pothole is extracted through edge detection. Finally, potholes are detected and classified based by the (you only look once) YOLO algorithm. The performance evaluation evaluates the distortion rate and restoration rate of the image, and the validity of the model and accuracy of the classification. The result of the evaluation shows that the mean square error (MSE) of the distortion rate and restoration rate of the proposed method has errors of 0.2–0.44. The peak signal to noise ratio (PSNR) is evaluated as 50 db or higher. The structural similarity index map (SSIM) is evaluated as 0.71–0.82. In addition, the result of the pothole classification shows that the area under curve (AUC) is evaluated as 0.9.

Gupta S., Sharma P., Sharma D., Gupta V., Sambyal N.

A pothole is a depression caused on roads due to seepage of water into soil structure or weight of continuously moving traffic. This not only damages the suspension of the vehicles but is also a prime reason for road accidents worldwide. This necessitates the need to develop an efficient automatic pothole detection system which can assist concerned authorities for timely repair and maintenance of the roads. This paper proposes a novel approach of bounding box based pothole localization from thermal images using deep neural networks. The modified ResNet34-single shot multibox detector gives an average precision of 74.53% whereas modified ResNet50-RetinaNet model provides 91.15% precision. The results obtained by the proposed modified ResNet50-RetinaNet model are the state-of-the-art results for localization of potholes using thermal images. In real-world scenarios such a system can assist relevant authorities to judge the severity of road damage and take appropriate effective measures accordingly.

Aguilar-González A., Medina Santiago A.

Road event detection is critical for tasks such as monitoring, anomaly detection, and optimization. Traditional approaches often require complex feature engineering or the use of machine learning models, which can be computationally intensive, especially when dealing with real-time data from high-frequency vibration and acceleration sensors. In this work, we propose a Random Forest-based event classification algorithm designed to handle the unique patterns of vibration and acceleration data in road event detection for an urban traffic scenario. Our method utilizes vibration and acceleration data in three axes (x, y, z) to classify events in a robust and scalable manner. The Random Forest model is trained to identify patterns in the sensor data and assign them to predefined event categories, providing an efficient and accurate classification mechanism. Experimental results prove the effectiveness of our approach: it reaches an accuracy of 91.99%, with a precision of 80% and a recall of 75%, demonstrating reliable event classification. Additionally, the Area Under the Curve (AUC) score of 0.9468 confirms the model’s strong discriminative capability. Further, compared to a rule-based approach, our method offers greater generalization and adaptability, reducing the need for manual parameter tuning. While the rule-based approach attains a higher precision of 92%, it requires frequent adjustments for each dataset and lacks robustness across different road conditions.

Paramarthalingam A., Sivaraman J., Theerthagiri P., Vijayakumar B., Baskaran V.

Visually impaired individuals encounter numerous impediments when traveling, such as navigating unfamiliar routes, accessing information, and transportation, which can limit their mobility and restrict their access to opportunities. However, assistive technologies and infrastructure solutions such as tactile paving, audio cues, voice announcements, and smartphone applications have been developed to mitigate these challenges. Visually impaired individuals also face difficulties when encountering potholes while traveling. Potholes can pose a significant safety hazard, as they can cause individuals to trip and fall, potentially leading to injury. For visually impaired individuals, identifying and avoiding potholes can be particularly challenging. The solutions ensure that all individuals can travel safely and independently, regardless of their visual abilities. An innovative approach that leverages the You Only Look Once (YOLO) algorithm to detect potholes and provide auditory or haptic feedback to visually impaired individuals has been proposed in this paper. The dataset of pothole images was trained and integrated into an application for detecting potholes in real-time image data using a camera. The app provides feedback to the user, allowing them to navigate potholes and increasing their mobility and safety. This approach highlights the potential of YOLO for pothole detection and provides a valuable tool for visually impaired individuals. According to the testing, the model achieved 82.7% image accuracy and 30 Frames Per Second (FPS) accuracy in live video. The model is trained to detect potholes close to the user, but it may be hard to detect potholes far away from the user. The current model is only trained to detect potholes, but visually impaired people face other challenges. The proposed technology is a portable option for visually impaired people.

Zhang Y., Pu C., Zhang Y., Niu M., Hao L., Wang J.

Bonding distance is defined by the projected distance on a substrate plane between two solder points of a bonding wire, which can directly affect the morphology of the bonding wire and the performance between internal components of the chip. For the inspection of the bonding distance, it is necessary to accurately recognize gold wires and solder points within the complex imagery of the chip. However, bonding wires at arbitrary angles and small-sized solder points are densely distributed across the complex background of bonding images. These characteristics pose challenges for conventional image detection and deep learning methods to effectively recognize and measure the bonding distances. In this paper, we present a novel method to measure bonding distance using a hierarchical measurement structure. First, we employ an image acquisition device to capture surface images of integrated circuits and use multi-layer convolution to coarsely locate the bonding region and remove redundant background. Second, we apply a multi-branch wire bonding inspection network for detecting bonding spots and segmenting gold wire. This network includes a fine location branch that utilizes low-level features to enhance detection accuracy for small bonding spots and a gold wire segmentation branch that incorporates an edge branch to effectively extract edge information. Finally, we use the bonding distance measurement module to develop four types of gold wire distribution models for bonding spot matching. Together, these modules create a fully automated method for measuring bonding distances in integrated circuits. The effectiveness of the proposed modules and overall framework has been validated through comprehensive experiments.