Visual Inspection of Surface Defects Based on Lightweight Reconstruction Network
-
摘要: 基于深度学习的方法在某些工业产品的表面缺陷识别和分类方面表现出优异的性能, 然而大多数工业产品缺陷样本稀缺, 而且特征差异大, 导致这类需要大量缺陷样本训练的检测方法难以适用. 本文提出一种基于重构网络的无监督缺陷检测算法(Reconstruction network for defects detection, ReNet-D), 仅使用容易大量获得的无缺陷样本数据实现对异常缺陷的检测. 本文提出的算法包括两个阶段: 图像重构网络训练阶段和表面缺陷区域检测阶段. 训练阶段通过一种轻量化结构的全卷积自编码器设计重构网络, 仅使用少量正常样本进行训练, 使得重构网络能够生成无缺陷重构图像, 进一步提出一种结合结构性损失和L1损失的函数作为重构网络的损失函数, 解决自编码器检测算法对不规则纹理表面缺陷检测效果较差的问题; 缺陷检测阶段以重构图像与待测图像的残差作为缺陷的可能区域, 通过常规图像操作即可实现缺陷的定位. 本文对所提出的ReNet-D方法的网络结构、训练像素块(patch)大小、损失函数系数等影响因素进行了详细的实验分析, 并在多个缺陷图像样本集上与其他同类算法做了对比, 结果表明ReNet-D有较强的鲁棒性和准确性. 由于ReNet-D的轻量化结构, 检测1024x1024像素大小的图像仅仅耗时2.82 ms, 适合工业在线检测.Abstract: Deep learning-based methods show excellent performance in identifying and classifying surface defects of certain industrial products. However, most industrial product defect samples are scarce and feature differences are large, making it difficult to apply this type of detection method that requires a large number of defect samples. This paper proposes an image reconstruction-based unsupervised defect detection algorithm (Reconstruction network for defects detection, ReNet-D), which uses only non-defective sample data that is easily available in large quantities to detect abnormal defects. The algorithm proposed in this paper includes two stages: image reconstruction network training stage and surface defect area detection stage. In the training phase, the reconstruction network is designed by a fully convolutional self-encoder with a lightweight structure, and only a small number of normal samples are used for training, so that the reconstruction network can generate defect-free reconstruction images, and a combination of structural loss and L1 is further proposed. The loss function is used as the loss function of the reconstructed network to solve the problem of poor detection of irregular texture surface defects by the self-encoder detection algorithm; the residual area of the reconstructed image and the image to be tested is used as a possible defect area in the defect detection stage. The final inspection result can be obtained through conventional image operations. In this paper, the network structure, training patch size, loss function coefficient and other influencing factors of the proposed ReNet-D method are analyzed in detail, and compared with other similar algorithms on multiple defect image sample sets. The results show that ReNet -D has strong robustness and accuracy. Due to the lightweight structure of ReNet-D, it takes only 2.82 ms to detect 1024x1024 pixel images, which is suitable for industrial online detection.
-
Key words:
- Defect detection /
- deep learning /
- small samples /
- fully convolutional auto encoder /
- loss function
-
图 1 各种表面缺陷. (a) 暗缺陷. (b) 明缺陷. (c) 覆盖图像的大尺度缺陷. (d) 微小缺陷. (e) 色差小的缺陷. (f)~(g) 与纹理相似的缺陷. (h) 模糊缺陷
Fig. 1 Various surface defects. (a) Dark defects. (b) Bright defects. (c) Large-scale defects covering the image. (d) Minor defects. (e) Defects with small color difference. (f)~(g) Defects similar to texture. (h) Fuzzy defects.
图 4 残差图处理流程. (a) 输入模型的原图. (b) ReNet-D重构图. (c) 由公式(9)得到的残差图. (d) 残差图滤波. (e) 缺陷定位(红色区域)
Fig. 4 The residual graph processing flow. (a) The original input image of the model. (b) Reconstruction map by ReNet-D. (c) The residual image obtained by formula (9). (d) Filtered residual map. (e) Defect location (red area).
图 6 不同损失函数下ReNet-D的检测结果. (a) 不规则纹理样本. (b) 规则纹理样本. (c) 样本(a)在不同损失函数下的收敛测试. (d) 样本(b)在不同损失函数下的收敛测试
Fig. 6 ReNet-D detection results under different loss functions. (a) Irregular texture samples. (b) Regular texture samples. (c) Convergence test under loss function of sample(a). (d) Convergence test under loss function of sample(b).
图 9 不同Patch size下ReNet-D的残差图和检测结果对比. (a) 不规则纹理样本. (b) 规则纹理样本. (c)不规则纹理收敛趋势比较. (d)规则纹理收敛趋势比较
Fig. 9 Comparison of the residual image and detection results of ReNet-D under different patch sizes. (a) Irregular texture samples. (b) Regular texture samples. (c) Comparison of irregular texture convergence trends. (d) Comparison of regular texture convergence trends
表 1 计算机系统配置
Table 1 Computer system configuration
系统 Ubantu16.04 内存 128G GPU NVIDIA GTX-1080Ti CPU Intel E5-2650v4@2.2GHz 深度学习框架 Pytorch, CUDA 9.0, CUDNN 5.1 表 2 默认网络参数
Table 2 Default network parameters
Patch size 32x32 Batch size 256 迭代步数 1000 损失函数权重α 0.15 表 3 不同损失函数下检测结果的比较. (A-不规则纹理样本, B-规则纹理样本)
Table 3 Comparison of test results under different loss functions. (A-irregular texture sample, B-regular texture sample)
损失函数
指标, 样本L1 MSE MSE+SSIM L1+SSIM SSIM Recall A 0.51 0.38 0.5 0.75 0.59 B 0.76 0.70 0.67 0.71 0.59 Precision A 0.93 0.35 0.52 0.89 0.93 B 0.84 0.65 0.70 0.87 0.96 F1-Measure A 0.66 0.36 0.51 0.82 0.72 B 0.80 0.67 0.69 0.78 0.73 表 4 不同权重系数下的检测结果比较
Table 4 Comparison of test results under different weight coefficients
权重系数
指标0 0.15 0.25 0.35 0.45 0.55 0.65 0.75 0.85 1 Recall 0.72 0.79 0.62 0.73 0.65 0.67 0.52 0.55 0.72 0.45 Precision 0.71 0.69 0.58 0.28 0.46 0.53 0.23 0.89 0.54 0.62 F1-Measure 0.71 0.73 0.60 0.41 0.54 0.60 0.32 0.68 0.62 0.52 表 5 不同Patch size下的检测结果比较. (A-不规则纹理样本, B-规则纹理样本)
Table 5 Comparison of test results under different patch sizes. (A-irregular texture sample, B-regular texture sample)
Patch size
指标,样本16×16 32×32 64×64 Recall A 0.82 0.67 0.40 B 0.83 0.64 0.53 Precision A 0.64 0.86 0.77 B 0.53 0.89 0.74 F1-Measure A 0.72 0.76 0.52 B 0.64 0.75 0.62 表 6 无监督样本的测试结果
Table 6 Test results of unsupervised samples
指标
样本Recall Precision F1-Measure 油污 0.71 0.94 0.80 破损 0.66 0.48 0.55 磨痕 0.63 0.89 0.70 涂抹 0.27 0.47 0.32 胶带 0.16 0.35 0.20 表 7 不同算法的检测效果比较
Table 7 Comparison of detection effects of different algorithms
算法
指标, 样本LCA PHOT MSCDAE ReNet-D Recall A 0.663 0.155 0.562 0.772 B 0.117 0.341 0.966 0.707 C 0.478 0.133 0.203 0.799 D 0.561 0.610 0.358 0.659 E 0.612 0.318 0.359 0.946 F 0.641 0.414 0.881 0.948 Precision A 0.436 0.324 0.463 0.884 B 0.002 0.478 0.444 0.793 C 0.024 0.112 0.143 0.855 D 0.639 0.299 0.642 0.940 E 0.412 0.367 0.696 0.824 F 0.899 0.006 0.920 0.935 F1-Measure A 0.526 0.210 0.508 0.824 B 0.004 0.398 0.608 0.732 C 0.045 0.122 0.168 0.822 D 0.597 0.401 0.460 0.771 E 0.492 0.341 0.662 0.881 F 0.748 0.012 0.900 0.941 表 8 处理耗时的比较
Table 8 Comparison of processing time
检测方法 Phot LCA MSCDAE ReNet-D 耗时(ms) 450 430 9746.59 2.82 -
[1] Deng S, Cai W, Xu Q, Liang B. Defect detection of bearing surfaces based on machine vision technique. In: 2010 International Conference on Computer Application and System Modeling (ICCASM 2010), USA: IEEE Press, Vol. 4, pp. V4−548 [2] Jian C, Gao J, Ao Y. Automatic surface defect detection for mobile phone screen glass based on machine vision. Applied Soft Computing, 2017, 52: 348−358 doi: 10.1016/j.asoc.2016.10.030 [3] Bulnes F G, Usamentiaga R, Garcia D F, Molleda J. An efficient method for defect detection during the manufacturing of web materials. Journal of Intelligent Manufacturing, 2016, 27(2): 431−445 doi: 10.1007/s10845-014-0876-9 [4] 金侠挺, 王耀南, 张辉, 刘理, 钟杭, 贺振东. 基于贝叶斯 CNN 和注意力网络的钢轨表面缺陷检测系统. 自动化学报, 2019, 45(12): 2312−2327Jin XiaTing, Wang YaoNan, Zhang Hui, Liu Li, Zhong Hang, He ZhenDong. DeepRail: automatic visual detection system for railway surface defect using Bayesian CNN and attention network. Acta Automatica Sinica, 2019, 45(12): 2312−2327 [5] 李良福, 马卫飞, 李丽, 陆铖. 基于深度学习的桥梁裂缝检测算法研究. 自动化学报, 2019, 45(9): 1727−1742Li Liang-Fu, Ma Wei-Fei, Li Li, Lu Cheng. Research on detection algorithm for bridge cracks based on deep learning. Acta Automatica Sinica, 2019, 45(9): 1727−1742 [6] Chen S, Hu T, Liu G, Pu Z, Li M, Du L. (2008, May). Defect classification algorithm for IC photomask based on PCA and SVM. In: 2008 Congress on Image and Signal Processing, USA: IEEE Press, Vol. 1, pp. 491−496. [7] Jiexian H, Di L, Feng, Y. Detection of surface of solder on flexible circuit. Optics and Precision Engineering, 2010, 18(11): 2443−2453 [8] Krizhevsky A, Sutskever I, Hinton, G. E. Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems 2012, pp. 1097−1105. [9] Napoletano P, Piccoli F, Schettini, R. Anomaly detection in nanofibrous materials by CNN-based self-similarity. Sensors, 2018, 18(1): 209 doi: 10.1109/JSEN.2017.2771313 [10] Cha Y J, Choi W, Suh G, Mahmoudkhani S, Büyüköztürk O. Autonomous structural visual inspection using region‐based deep learning for detecting multiple damage types. Computer‐Aided Civil and Infrastructure Engineering, 2018, 33(9): 731−747 doi: 10.1111/mice.12334 [11] Gao Y, Gao L, Li X, Yan X. A semi-supervised convolutional neural network-based method for steel surface defect recognition. Robotics and Computer-Integrated Manufacturing, 2020, 61: 101825 doi: 10.1016/j.rcim.2019.101825 [12] Zhao Z, Xu G, Qi Y, Liu N, Zhang T. Multi-patch deep features for power line insulator status classification from aerial images. In: International Joint Conference on Neural Networks (IJCNN) IEEE Press, 2016.pp. 3187−3194 [13] Wang T, Chen Y, Qiao M, Snoussi H. A fast and robust convolutional neural network-based defect detection model in product quality control. The International Journal of Advanced Manufacturing Technology, 2018, 94(9-12): 3465−3471 doi: 10.1007/s00170-017-0882-0 [14] Xu X, Zheng H, Guo Z, Wu X, Zheng Z. SDD-CNN: Small Data-Driven Convolution Neural Networks for Subtle Roller Defect Inspection. Applied Sciences, 2019, 9(7): 1364 doi: 10.3390/app9071364 [15] Paolo N, Flavio P, Raimondo S. Anomaly Detection in Nanofibrous Materials by CNN-Based Self-Similarity. Sensors, 2018, 18(2): 209−216 doi: 10.3390/s18010209 [16] Weimer D, Scholz-Reiter B, Shpitalni M. Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection. CIRP Annals. Manufacturing Technology, 2016, 65(1): 417−420 doi: 10.1016/j.cirp.2016.04.072 [17] Girshick R, Donahue J, Darrell T, et al. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[C]. v, 2014: 580−587. [18] Liu W, Anguelov D, Erhan D, et al. SSD: Single Shot MultiBox Detector[C].In: European Conference on Computer Vision. Springer International Publishing, 2016: pp 21−37. [19] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection, ” in Proc. In: IEEE Conf. Comput. Vis. Pattern Recognit. Jun. 2016, pp. 779−788 [20] Chen J, Liu Z, Wang H, Núñez A, Han, Z. Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network. IEEE Transactions on Instrumentation and Measurement, 2017, 67(2): 257−269 [21] Cha Y J, Choi W, Suh G, et al. Autonomous Structural Visual Inspection Using Region-Based Deep Learning for Detecting Multiple Damage Types. Computer Aided Civil & Infrastructure Engineering, 2017, 00(4): 1−17 [22] Huang Y, Qiu C, Guo Y, Wang X, Yuan, K. (2018, August). Surface defect saliency of magnetic tile. In: 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE).IEEE Press: pp. 612−617. [23] Lingteng, Qiu, Xiaojun, et al. A High-Efficiency Fully Convolutional Networks for Pixel-Wise Surface Defect Detection. IEEE Access, 2019: 15884−15893 [24] Yu W, Zhang Y, Shi H. Surface Defect Inspection Under a Small Training Set Condition. In: International Conference on Intelligent Robotics and Applications. Springer, Cham. 2019, August. pp. 517−528. [25] Masci J, Meier U, Cireşan D, Schmidhuber J. Stacked convolutional auto-encoders for hierarchical feature extraction. In International Conference on Artificial Neural Networks. Berlin, Heidelberg. Springer, 2011, June. pp. 52−59 [26] Li Y, Zhao W, Pan J. Deformable patterned fabric defect detection with fisher criterion-based deep learning. IEEE Transactions on Automation Science and Engineering, 2016, 14(2): 1256−1264 [27] Chalapathy R, Menon A K, Chawla S. Robust, Deep and Inductive Anomaly Detection.In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases: Springer, 2017.36−51. [28] 袁静, 章毓晋. 融合梯度差信息的稀疏去噪自编码网络在异常行为检测中的应用. 自动化学报, 2017, 43(4): 604−610Yuan Jing, Zhang YuJin. Application of sparse denoising autoencoder network with gradient difference information for abnormal action detection. Acta Automatica Sinica, 2017, 43(4): 604−610 [29] Mei S, Yang H, Yin Z. An Unsupervised-Learning-Based Approach for Automated Defect Inspection on Textured Surfaces. IEEE Transactions on Instrumentation and Measurement, 2018: 1−12 [30] D. Aiger and H. Talbot, “The phase only transform for unsupervised surface defect detection, ”.In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., IEEE press, Jun. 2010, pp. 295−302. [31] H.-D. Lin. Tiny surface defect inspection of electronic passive components using discrete cosine transform decomposition and cumulative sum techniques. Image Vis. Comput, 2008, 26(5): 603−621 doi: 10.1016/j.imavis.2007.07.009 [32] Yang H, Chen Y, Song K, Yin Z. Multiscale Feature-Clustering-Based Fully Convolutional Autoencoder for Fast Accurate Visual Inspection of Texture Surface Defects. IEEE Transactions on Automation Science and Engineering. 2019, 1−18. [33] Makhzani A, Shlens J, Jaitly N, Goodfellow I, Frey B. (2015). Adversarial autoencoders. arXiv preprint arXiv: 1511.05644. [34] Zhao Z, Li B, Dong R, et al. A Surface Defect Detection Method Based on Positive Samples[C]. International Conference on Artificial Intelligence, Pacific Rim, Springer, Cham, 2018: 473−481. [35] Di He, et al. "Surface defect classification of steels with a new semi-supervised learning method." Optics and Lasers in Engineering, 2019, 117: 40−48. [36] Schlegl T, Seeböck P, Waldstein, S M, Schmidt-Erfurth U, Langs, G. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International Conference on Information Processing in Medical Imaging, Springer, Cham. 2017, June.pp. 146−157. [37] Paul Bergmann, Michael Fauser, David Sattlegger, Carsten Steger; A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), USA: IEEE Press: 2019.pp. 9592−9600 [38] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167. 2015. [39] Zhao H, Gallo O, Frosio I, Kautz J. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 2016, 3(1): 47−57 [40] Bergmann P, Löwe S, Fauser M, et al. Improving unsupervised defect segmentation by applying structural similarity to autoencoders[J]. arXiv preprint arXiv: 1807.02011 [41] Lv C, Zhang Z, Shen F, et al. A Fast Surface Defect Detection Method Based on Background Reconstruction[J].In: International Journal of Precision Engineering and Manufacturing, 2019: 1−13. [42] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 2004, 13(4): 600−612 doi: 10.1109/TIP.2003.819861 [43] Silvestre-Blanes J, Albero-Albero T, Miralles I, Pérez-Llorens R, Moreno J. A Public Fabric Database for Defect Detection Methods and Results. Autex Research Journal, 2019, 19(4): 363−374 doi: 10.2478/aut-2019-0035 [44] Jager M, Knoll C, Hamprecht F A. Weakly supervised learning of a classifier for unusual event detection. IEEE Transactions on Image Processing, 2019, 17(9): 1700−1708 [45] G. Kylberg and I. Sintorn. (2013). Kylberg Sintorn RotationDataset.[Online]. Available: http://www.cb.uu.se/~gustaf/KylbergSintornRotation/ [46] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. pp. 3431−3440. [47] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention Springer, Cham. 2015, October. pp. 234−241. [48] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. USA: IEEE Press, 2016.pp. 770−778. [49] Shi W, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, ... Wang, Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition USA: IEEE Press, 2016. pp. 1874−1883. [50] Huang G, Liu Z, Van Der Maaten L, Weinberger, K. Q.Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. USA: IEEE Press, 2017. pp. 4700−4708. [51] Tsai D M, Huang T Y. Automated surface inspection for statistical textures. Image and Vision computing, 2003, 21(4): 307−323 doi: 10.1016/S0262-8856(03)00007-6 -

计量
- 文章访问数: 244
- HTML全文浏览量: 52
- 被引次数: 0