YOLOv5-Pothole: An Improved Pothole Perception Method Based on YOLOv5-Seg

Wang C., Bochkovskiy A., Liao H.M.

Ma N., Fan J., Wang W., Wu J., Jiang Y., Xie L., Fan R.

Abstract Computer vision algorithms have been utilized for 3-D road imaging and pothole detection for over two decades. Nonetheless, there is a lack of systematic survey articles on state-of-the-art (SoTA) computer vision techniques, especially deep learning models, developed to tackle these problems. This article first introduces the sensing systems employed for 2-D and 3-D road data acquisition, including camera(s), laser scanners and Microsoft Kinect. It then comprehensively reviews the SoTA computer vision algorithms, including (1) classical 2-D image processing, (2) 3-D point cloud modelling and segmentation and (3) machine/deep learning, developed for road pothole detection. The article also discusses the existing challenges and future development trends of computer vision-based road pothole detection approaches: classical 2-D image processing-based and 3-D point cloud modelling and segmentation-based approaches have already become history; and convolutional neural networks (CNNs) have demonstrated compelling road pothole detection results and are promising to break the bottleneck with future advances in self/un-supervised learning for multi-modal semantic segmentation. We believe that this survey can serve as practical guidance for developing the next-generation road condition assessment systems.

Kortmann F., Fassmeyer P., Funk B., Drews P.

While autonomous driving technology made significant progress in the last decade, road damage detection as a relevant challenge for ensuring safety and comfort is still under development. This paper addresses the lack of algorithms for detecting road damages that meet autonomous driving systems’ requirements. We investigate the environmental perception systems’ architecture and current algorithm designs for road damage detection. Based on the autonomous driving architecture, we develop an end-to-end concept that leverages data from low-cost pre-installed sensors for real-time road damage and damage severity detection as well as cloud- and crowd-based HD Feature Maps to share information across vehicles. In a design science research approach, we develop three artifacts in three iterations of expert workshops and design cycles: the end-to-end concept featuring road damages in the system architecture and two lightweight deep neural networks, one for detecting road damages and another for detecting their severity as the central components of the system. The research design draws on new self-labeled automotive-grade images from front-facing cameras in the vehicle and interdisciplinary literature regarding autonomous driving architecture and the design of deep neural networks. The road damage detection algorithm delivers cutting-edge performance while being lightweight compared to the winners of the IEEE Global Road Damage Detection Challenge 2020, which makes it applicable in autonomous vehicles. The road damage severity algorithm is a promising approach, delivering superior results compared to a baseline model. The end-to-end concept is developed and evaluated with experts of the autonomous driving application domain.

Alshammari S., Song S.

Chen D., Chen N., Zhang X., Guan Y.

Vehicle-road collaboration is inseparable for smart cities, and the potholes detection is an essential direction in vehicle-road collaboration. With the development of surveying and mapping technologies, road potholes detection accuracy has improved in recent years. However, the legacy detection methods do not have the ease of service and real-time observation capability, and thus, the potholes in the road cannot be mapped in time. To solve this key issue, we proposed a reflectometry method to realize real-time potholes observation with vibration signals analysis and Spatio-temporal trajectory fusion. We further developed several prototype devices for validation. These prototype devices measure the acceleration signal mounted on the wheel steering lever and implement edge signal processing and Spatio-temporal information fusion on the prototype. Observation results and Spatio-temporal information are been rapidly transmitted to the sensing server via narrow band Internet of Things. Results and analyses demonstrated that this method successfully enabled the potholes observation in real-time through light and a rapidly deployable platform that relies on repeated trajectory data from the vehicle. Results also demonstrated that this method was not limited by vehicle type, speed, or engine operating condition. In the road experiment, the proposed method provided stable, efficient, and real-time potholes observation results. Compared with traditional methods, the method reduces costs and improves sensing efficiency based on Spatio-temporal trajectory fusion, and potholes information can be prompted in real-time. This innovation provides a strategic exploration and thinking to deal with road potholes in real-time sensing.

Kim Y., Kim Y., Son S., Lim S., Choi B., Choi D.

Potholes, a kind of road defect, can damage vehicles and negatively affect drivers’ safe driving, and in severe cases can lead to traffic accidents. Efficient and preventive management of potholes in a complex road environment plays an important role in securing driver safety. It is also expected to contribute to the prevention of traffic accidents and the smooth flow of traffic. In the past, pothole detection was mainly performed via visual inspection by human experts. Recently, automated pothole-detection methods apply various technologies that converge basic technologies such as sensors and signal processing. The automated pothole-detection methods can be classified into three types according to the technology used in the pothole-recognition process: a vision-based method, a vibration-based method, and a 3D reconstruction-based method. In this paper, three methods are compared, and the strengths and weaknesses of each method are summarized. The detection process and technology proposed in the latest research related to automated pothole detection are described for each method. The development plans of future technology that is connected with those studies are also presented in this paper.

Ahmed A., Ashfaque M., Ulhaq M.U., Mathavan S., Kamal K., Rahman M.

Machine vision based evaluation systems are receiving increased attention, day by day, for automated quality inspection of roads. Industrial pavement scanners consist of laser scanners and are very expensive, hence inaccessible for everyone. The proposed work presents a simple and novel approach for 3D reconstruction of potholes for an automated inspection and road surface evaluation. The technique utilizes a Structure from Motion based 3D reconstruction algorithm, along with laser triangulation, to generate 3D point clouds of potholes. Alongside, a novel low-cost system, consisting of a single camera and a laser pointer, is also proposed. Keypoint matching techniques are employed, with the 5-point algorithm, on successive image frames to generate a point cloud. However, this point cloud is not metric yet, without scale information. The scale ambiguity is solved by making use of the laser pointer, and using the principle of triangulation. The laser spot is also detected in the same image sequence that is used for point-cloud building, cutting down the image capturing and processing overhead. The system has been benchmarked on artificial indentations with known dimensions, proving the robustness of the measurement scheme and hardware. Static and dynamic tests have been performed. The mean depth errors for measurement made by the imager statically and at dynamic speeds of 10 km/hr, 15 km/hr, and 20 km/hr are 5.3%, 7.9%, 14.4%, and 26.6%, whereas for perimeter the errors are 5.2%, 6.83 %, 11.8%, and 27.8%. The proposed, low-cost technique shows promising results in generating 3D point clouds for potholes.

Li Y., Ma R., Zhang B., Liu H.

Das S., Kale A.

While several approaches to pothole detection take place in processed live video, there is a huge need for an End-to-End learning system which can help us understand the depth of roads dynamically. The original video needs to be optimized for the purposes of cameras and other sensors. In P3De, calibration of live video feed on a frequency graph will yield better results for many reasons. During driving, it's intuitive to know that it's important to know the depth of certain sections of roads to enable better vision and perception understanding. Much emphasis is put on figuring out and estimating depth of the road through 3D simulation, which we consider the very heart of P3De.

Patel U., Joshi B., Pandey S.V., Darji S.

As one type of asphalt troubles, potholes are indispensable signs showing basic imperfections of the streets and identification of these potholes is one among the basic errands for determinative right techniques on road surfaced asphalt support and recovery. Be that as it may, physical location and assessing ways are cherished and tedious. The main cons of potholes are the escalating death rate and damage/bending in vehicle tires. Consequently, numerous endeavors are made for building up an innovation that may precisely watch and perceive potholes, which can add to the improvement in asphalt quality through past examination and quick activity. This paper compares different case studies such as image processing, clustering analysis, black box camera, laser technology, and Zigbee protocol for predicting the potholes accurately. This paper presents state of the art on well-known pothole identification techniques that discusses various methodologies and identifies optimum solutions for real-time implementation under worst environment and operating conditions thus ensuring human safety.

Ma N., Li D., He W., Deng Y., Li J., Gao Y., Bao H., Zhang H., Xu X., Liu Y., Wu Z., Chen L.

Yebes J.J., Montero D., Arriola I.

Among several road hazards that are present in any paved way in the world, potholes are one of the most annoying and involving higher maintenance costs. There is an increasing interest on the automated detection of these hazards enabled by technological and research progress. Our work tackled the challenge of pothole detection from images of real world road scenes. The main novelty resides on the application of latest progress in Artificial Intelligence to learn the visual appearance of potholes. We built a large dataset of images with pothole annotations. They contained road scenes from different cities in the world, taken with different cameras, vehicles and viewpoints under varied environmental conditions. Then, we fine-tuned four different object detection models based on Deep Neural Networks. We achieved mean average precision above 75% and we used the pothole detector on the Nvidia DrivePX2 platform running at 5–6 frames per second. Moreover, it was deployed on a real vehicle driving at speeds below 60 km/h to notify the detected potholes to a given Internet of Things platform as part of AUTOPILOT H2020 project.

Fan R., Liu M.

This article presents a novel road damage detection algorithm based on unsupervised disparity map segmentation. Firstly, a disparity map is transformed by minimizing an energy function with respect to stereo rig roll angle and road disparity projection model. Instead of solving this energy minimization problem using non-linear optimization techniques, we directly find its numerical solution. The transformed disparity map is then segmented using Otus's thresholding method, and the damaged road areas can be extracted. The proposed algorithm requires no parameters when detecting road damage. The experimental results illustrate that our proposed algorithm performs both accurately and efficiently. The pixel-level road damage detection accuracy is approximately 97.56%. The source code is publicly available at: https://github.com/ruirangerfan/unsupervised_disparity_map_segmentation.git.

Fan R., Ozgunalp U., Hosking B., Liu M., Pitas I.

Pothole detection is one of the most important tasks for road maintenance. Computer vision approaches are generally based on either 2D road image analysis or 3D road surface modeling. However, these two categories are always used independently. Furthermore, the pothole detection accuracy is still far from satisfactory. Therefore, in this paper, we present a robust pothole detection algorithm that is both accurate and computationally efficient. A dense disparity map is first transformed to better distinguish between damaged and undamaged road areas. To achieve greater disparity transformation efficiency, golden section search and dynamic programming are utilized to estimate the transformation parameters. Otsu's thresholding method is then used to extract potential undamaged road areas from the transformed disparity map. The disparities in the extracted areas are modeled by a quadratic surface using least squares fitting. To improve disparity map modeling robustness, the surface normal is also integrated into the surface modeling process. Furthermore, random sample consensus is utilized to reduce the effects caused by outliers. By comparing the difference between the actual and modeled disparity maps, the potholes can be detected accurately. Finally, the point clouds of the detected potholes are extracted from the reconstructed 3D road surface. The experimental results show that the successful detection accuracy of the proposed system is around 98.7% and the overall pixel-level accuracy is approximately 99.6%.

He K., Gkioxari G., Dollar P., Girshick R.

We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without tricks, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition. Code will be made available.

Gunathilaka W.M., Kahandawa G., Ibrahim M.Y., Hewawasam H.S., Nguyen L.

Paramarthalingam A., Sivaraman J., Theerthagiri P., Vijayakumar B., Baskaran V.

Visually impaired individuals encounter numerous impediments when traveling, such as navigating unfamiliar routes, accessing information, and transportation, which can limit their mobility and restrict their access to opportunities. However, assistive technologies and infrastructure solutions such as tactile paving, audio cues, voice announcements, and smartphone applications have been developed to mitigate these challenges. Visually impaired individuals also face difficulties when encountering potholes while traveling. Potholes can pose a significant safety hazard, as they can cause individuals to trip and fall, potentially leading to injury. For visually impaired individuals, identifying and avoiding potholes can be particularly challenging. The solutions ensure that all individuals can travel safely and independently, regardless of their visual abilities. An innovative approach that leverages the You Only Look Once (YOLO) algorithm to detect potholes and provide auditory or haptic feedback to visually impaired individuals has been proposed in this paper. The dataset of pothole images was trained and integrated into an application for detecting potholes in real-time image data using a camera. The app provides feedback to the user, allowing them to navigate potholes and increasing their mobility and safety. This approach highlights the potential of YOLO for pothole detection and provides a valuable tool for visually impaired individuals. According to the testing, the model achieved 82.7% image accuracy and 30 Frames Per Second (FPS) accuracy in live video. The model is trained to detect potholes close to the user, but it may be hard to detect potholes far away from the user. The current model is only trained to detect potholes, but visually impaired people face other challenges. The proposed technology is a portable option for visually impaired people.