基于轻量化重构网络的表面缺陷视觉检测

余文勇; 张阳; 姚海明; 石绘

doi:10.16383/j.aas.c200535

基于轻量化重构网络的表面缺陷视觉检测

doi: 10.16383/j.aas.c200535

余文勇^1,,
张阳^1,,
姚海明^1,,
石绘^2,

1.
华中科技大学机械科学与工程学院武汉 430074
2.
武汉理工大学机电学院武汉 430070

基金项目: 国家自然科学基金 (51775214)资助

详细信息

作者简介:
余文勇：华中科技大学机械科学与工程学院副教授. 2004 年获得华中科技大学博士学位. 主要研究方向为机器视觉, 智能机器人和缺陷检测. E-mail: ywy@hust.edu.cn

张阳：2020年获华中科技大学硕士学位. 主要研究方向为机器视觉和缺陷检测. E-mail: ywy20052006@126.com

姚海明：主要研究方向为机器视觉和缺陷检测. E-mail: weyoung@hust.edu.cn

石绘：武汉理工大学博士研究生, 2001年获华中科技大学硕士学位. 主要研究方向为机器视觉和缺陷检测. 本文通信作者. E-mail: clove_shi@163.com

计量
- 文章访问数: 244
- HTML全文浏览量: 52
- 被引次数: 0
出版历程

Visual Inspection of Surface Defects Based on Lightweight Reconstruction Network

1.
School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074.
2.
School of Mechanical and Electronic Engineering, Wuhan University of Technology, Wuhan 430070.

Funds: Supported by National Natural Science Foundation of China (51775214)

摘要

摘要: 基于深度学习的方法在某些工业产品的表面缺陷识别和分类方面表现出优异的性能, 然而大多数工业产品缺陷样本稀缺, 而且特征差异大, 导致这类需要大量缺陷样本训练的检测方法难以适用. 本文提出一种基于重构网络的无监督缺陷检测算法(Reconstruction network for defects detection, ReNet-D), 仅使用容易大量获得的无缺陷样本数据实现对异常缺陷的检测. 本文提出的算法包括两个阶段: 图像重构网络训练阶段和表面缺陷区域检测阶段. 训练阶段通过一种轻量化结构的全卷积自编码器设计重构网络, 仅使用少量正常样本进行训练, 使得重构网络能够生成无缺陷重构图像, 进一步提出一种结合结构性损失和L1损失的函数作为重构网络的损失函数, 解决自编码器检测算法对不规则纹理表面缺陷检测效果较差的问题; 缺陷检测阶段以重构图像与待测图像的残差作为缺陷的可能区域, 通过常规图像操作即可实现缺陷的定位. 本文对所提出的ReNet-D方法的网络结构、训练像素块(patch)大小、损失函数系数等影响因素进行了详细的实验分析, 并在多个缺陷图像样本集上与其他同类算法做了对比, 结果表明ReNet-D有较强的鲁棒性和准确性. 由于ReNet-D的轻量化结构, 检测1024x1024像素大小的图像仅仅耗时2.82 ms, 适合工业在线检测.
- 缺陷检测 /
- 深度学习 /
- 小样本 /
- 全卷积自编码器 /
- 损失函数
Abstract: Deep learning-based methods show excellent performance in identifying and classifying surface defects of certain industrial products. However, most industrial product defect samples are scarce and feature differences are large, making it difficult to apply this type of detection method that requires a large number of defect samples. This paper proposes an image reconstruction-based unsupervised defect detection algorithm (Reconstruction network for defects detection, ReNet-D), which uses only non-defective sample data that is easily available in large quantities to detect abnormal defects. The algorithm proposed in this paper includes two stages: image reconstruction network training stage and surface defect area detection stage. In the training phase, the reconstruction network is designed by a fully convolutional self-encoder with a lightweight structure, and only a small number of normal samples are used for training, so that the reconstruction network can generate defect-free reconstruction images, and a combination of structural loss and L1 is further proposed. The loss function is used as the loss function of the reconstructed network to solve the problem of poor detection of irregular texture surface defects by the self-encoder detection algorithm; the residual area of the reconstructed image and the image to be tested is used as a possible defect area in the defect detection stage. The final inspection result can be obtained through conventional image operations. In this paper, the network structure, training patch size, loss function coefficient and other influencing factors of the proposed ReNet-D method are analyzed in detail, and compared with other similar algorithms on multiple defect image sample sets. The results show that ReNet -D has strong robustness and accuracy. Due to the lightweight structure of ReNet-D, it takes only 2.82 ms to detect 1024x1024 pixel images, which is suitable for industrial online detection.
- Defect detection /
- deep learning /
- small samples /
- fully convolutional auto encoder /
- loss function

HTML全文

图 1 各种表面缺陷. (a) 暗缺陷. (b) 明缺陷. (c) 覆盖图像的大尺度缺陷. (d) 微小缺陷. (e) 色差小的缺陷. (f)~(g) 与纹理相似的缺陷. (h) 模糊缺陷

Fig. 1 Various surface defects. (a) Dark defects. (b) Bright defects. (c) Large-scale defects covering the image. (d) Minor defects. (e) Defects with small color difference. (f)~(g) Defects similar to texture. (h) Fuzzy defects.

下载: 全尺寸图片幻灯片

图 2 ReNet-D算法模型

Fig. 2 ReNet-D algorithm model

下载: 全尺寸图片幻灯片

图 3 ReNet-D网络结构

Fig. 3 ReNet-D network structure

下载: 全尺寸图片幻灯片

图 4 残差图处理流程. (a) 输入模型的原图. (b) ReNet-D重构图. (c) 由公式(9)得到的残差图. (d) 残差图滤波. (e) 缺陷定位(红色区域)

Fig. 4 The residual graph processing flow. (a) The original input image of the model. (b) Reconstruction map by ReNet-D. (c) The residual image obtained by formula (9). (d) Filtered residual map. (e) Defect location (red area).

下载: 全尺寸图片幻灯片

图 5 实验采用的表面缺陷数据集. (a) AITEX数据集. (b)~(e) DAGM2007数据集. (f) Kylberg Sintorn数据集

Fig. 5 Surface defect data set used in the experiment. (a) AITEX data set. (b)~(e) DAGM2007 data set. (f) Kylberg Sintorn data set.

下载: 全尺寸图片幻灯片

图 6 不同损失函数下ReNet-D的检测结果. (a) 不规则纹理样本. (b) 规则纹理样本. (c) 样本(a)在不同损失函数下的收敛测试. (d) 样本(b)在不同损失函数下的收敛测试

Fig. 6 ReNet-D detection results under different loss functions. (a) Irregular texture samples. (b) Regular texture samples. (c) Convergence test under loss function of sample(a). (d) Convergence test under loss function of sample(b).

下载: 全尺寸图片幻灯片

图 7 不同权重系数下ReNet-D性能比较. (a) 残差热力图对比. (b)训练LOSS曲线比较

Fig. 7 Comparison of ReNet-D performance under different weight coefficients. (a) Comparison of residual heat maps. (b) Comparison of training LOSS curves

下载: 全尺寸图片幻灯片

图 8 不同特征提取网络下ReNet-D的残差图对比

Fig. 8 Comparison of residual maps of ReNet-D under different feature extraction networks

下载: 全尺寸图片幻灯片

图 9 不同Patch size下ReNet-D的残差图和检测结果对比. (a) 不规则纹理样本. (b) 规则纹理样本. (c)不规则纹理收敛趋势比较. (d)规则纹理收敛趋势比较

Fig. 9 Comparison of the residual image and detection results of ReNet-D under different patch sizes. (a) Irregular texture samples. (b) Regular texture samples. (c) Comparison of irregular texture convergence trends. (d) Comparison of regular texture convergence trends

下载: 全尺寸图片幻灯片

图 10 无监督样本的测试结果

Fig. 10 Test results of unsupervised samples

下载: 全尺寸图片幻灯片

图 11 多种算法测试效果对比

Fig. 11 Comparison of test results of multiple algorithms.

下载: 全尺寸图片幻灯片

表 1 计算机系统配置

Table 1 Computer system configuration

系统	Ubantu16.04
内存	128G
GPU	NVIDIA GTX-1080Ti
CPU	Intel E5-2650v4@2.2GHz
深度学习框架	Pytorch, CUDA 9.0, CUDNN 5.1

下载: 导出CSV

表 2 默认网络参数

Table 2 Default network parameters

Patch size	32x32
Batch size	256
迭代步数	1000
损失函数权重α	0.15

下载: 导出CSV

表 3 不同损失函数下检测结果的比较. (A-不规则纹理样本, B-规则纹理样本)

Table 3 Comparison of test results under different loss functions. (A-irregular texture sample, B-regular texture sample)

损失函数指标, 样本		L1	MSE	MSE+SSIM	L1+SSIM	SSIM
Recall	A	0.51	0.38	0.5	0.75	0.59
Recall	B	0.76	0.70	0.67	0.71	0.59
Precision	A	0.93	0.35	0.52	0.89	0.93
Precision	B	0.84	0.65	0.70	0.87	0.96
F1-Measure	A	0.66	0.36	0.51	0.82	0.72
F1-Measure	B	0.80	0.67	0.69	0.78	0.73

下载: 导出CSV

表 4 不同权重系数下的检测结果比较

Table 4 Comparison of test results under different weight coefficients

权重系数指标	0	0.15	0.25	0.35	0.45	0.55	0.65	0.75	0.85	1
Recall	0.72	0.79	0.62	0.73	0.65	0.67	0.52	0.55	0.72	0.45
Precision	0.71	0.69	0.58	0.28	0.46	0.53	0.23	0.89	0.54	0.62
F1-Measure	0.71	0.73	0.60	0.41	0.54	0.60	0.32	0.68	0.62	0.52

下载: 导出CSV

表 5 不同Patch size下的检测结果比较. (A-不规则纹理样本, B-规则纹理样本)

Table 5 Comparison of test results under different patch sizes. (A-irregular texture sample, B-regular texture sample)

Patch size 指标,样本		16×16	32×32	64×64
Recall	A	0.82	0.67	0.40
Recall	B	0.83	0.64	0.53
Precision	A	0.64	0.86	0.77
Precision	B	0.53	0.89	0.74
F1-Measure	A	0.72	0.76	0.52
F1-Measure	B	0.64	0.75	0.62

下载: 导出CSV

表 6 无监督样本的测试结果

Table 6 Test results of unsupervised samples

指标样本	Recall	Precision	F1-Measure
油污	0.71	0.94	0.80
破损	0.66	0.48	0.55
磨痕	0.63	0.89	0.70
涂抹	0.27	0.47	0.32
胶带	0.16	0.35	0.20

下载: 导出CSV

表 7 不同算法的检测效果比较

Table 7 Comparison of detection effects of different algorithms

算法指标, 样本		LCA	PHOT	MSCDAE	ReNet-D
Recall	A	0.663	0.155	0.562	0.772
	B	0.117	0.341	0.966	0.707
	C	0.478	0.133	0.203	0.799
	D	0.561	0.610	0.358	0.659
	E	0.612	0.318	0.359	0.946
	F	0.641	0.414	0.881	0.948
Precision	A	0.436	0.324	0.463	0.884
	B	0.002	0.478	0.444	0.793
	C	0.024	0.112	0.143	0.855
	D	0.639	0.299	0.642	0.940
	E	0.412	0.367	0.696	0.824
	F	0.899	0.006	0.920	0.935
F1-Measure	A	0.526	0.210	0.508	0.824
	B	0.004	0.398	0.608	0.732
	C	0.045	0.122	0.168	0.822
	D	0.597	0.401	0.460	0.771
	E	0.492	0.341	0.662	0.881
	F	0.748	0.012	0.900	0.941

下载: 导出CSV

表 8 处理耗时的比较

Table 8 Comparison of processing time

检测方法	Phot	LCA	MSCDAE	ReNet-D
耗时(ms)	450	430	9746.59	2.82

下载: 导出CSV

[1]	Deng S, Cai W, Xu Q, Liang B. Defect detection of bearing surfaces based on machine vision technique. In: 2010 International Conference on Computer Application and System Modeling (ICCASM 2010), USA: IEEE Press, Vol. 4, pp. V4−548
[2]	Jian C, Gao J, Ao Y. Automatic surface defect detection for mobile phone screen glass based on machine vision. Applied Soft Computing, 2017, 52: 348−358 doi: 10.1016/j.asoc.2016.10.030
[3]	Bulnes F G, Usamentiaga R, Garcia D F, Molleda J. An efficient method for defect detection during the manufacturing of web materials. Journal of Intelligent Manufacturing, 2016, 27(2): 431−445 doi: 10.1007/s10845-014-0876-9
[4]	金侠挺, 王耀南, 张辉, 刘理, 钟杭, 贺振东. 基于贝叶斯 CNN 和注意力网络的钢轨表面缺陷检测系统. 自动化学报, 2019, 45(12): 2312−2327 Jin XiaTing, Wang YaoNan, Zhang Hui, Liu Li, Zhong Hang, He ZhenDong. DeepRail: automatic visual detection system for railway surface defect using Bayesian CNN and attention network. Acta Automatica Sinica, 2019, 45(12): 2312−2327
[5]	李良福, 马卫飞, 李丽, 陆铖. 基于深度学习的桥梁裂缝检测算法研究. 自动化学报, 2019, 45(9): 1727−1742 Li Liang-Fu, Ma Wei-Fei, Li Li, Lu Cheng. Research on detection algorithm for bridge cracks based on deep learning. Acta Automatica Sinica, 2019, 45(9): 1727−1742
[6]	Chen S, Hu T, Liu G, Pu Z, Li M, Du L. (2008, May). Defect classification algorithm for IC photomask based on PCA and SVM. In: 2008 Congress on Image and Signal Processing, USA: IEEE Press, Vol. 1, pp. 491−496.
[7]	Jiexian H, Di L, Feng, Y. Detection of surface of solder on flexible circuit. Optics and Precision Engineering, 2010, 18(11): 2443−2453
[8]	Krizhevsky A, Sutskever I, Hinton, G. E. Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems 2012, pp. 1097−1105.
[9]	Napoletano P, Piccoli F, Schettini, R. Anomaly detection in nanofibrous materials by CNN-based self-similarity. Sensors, 2018, 18(1): 209 doi: 10.1109/JSEN.2017.2771313
[10]	Cha Y J, Choi W, Suh G, Mahmoudkhani S, Büyüköztürk O. Autonomous structural visual inspection using region‐based deep learning for detecting multiple damage types. Computer‐Aided Civil and Infrastructure Engineering, 2018, 33(9): 731−747 doi: 10.1111/mice.12334
[11]	Gao Y, Gao L, Li X, Yan X. A semi-supervised convolutional neural network-based method for steel surface defect recognition. Robotics and Computer-Integrated Manufacturing, 2020, 61: 101825 doi: 10.1016/j.rcim.2019.101825
[12]	Zhao Z, Xu G, Qi Y, Liu N, Zhang T. Multi-patch deep features for power line insulator status classification from aerial images. In: International Joint Conference on Neural Networks (IJCNN) IEEE Press, 2016.pp. 3187−3194
[13]	Wang T, Chen Y, Qiao M, Snoussi H. A fast and robust convolutional neural network-based defect detection model in product quality control. The International Journal of Advanced Manufacturing Technology, 2018, 94(9-12): 3465−3471 doi: 10.1007/s00170-017-0882-0
[14]	Xu X, Zheng H, Guo Z, Wu X, Zheng Z. SDD-CNN: Small Data-Driven Convolution Neural Networks for Subtle Roller Defect Inspection. Applied Sciences, 2019, 9(7): 1364 doi: 10.3390/app9071364
[15]	Paolo N, Flavio P, Raimondo S. Anomaly Detection in Nanofibrous Materials by CNN-Based Self-Similarity. Sensors, 2018, 18(2): 209−216 doi: 10.3390/s18010209
[16]	Weimer D, Scholz-Reiter B, Shpitalni M. Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection. CIRP Annals. Manufacturing Technology, 2016, 65(1): 417−420 doi: 10.1016/j.cirp.2016.04.072
[17]	Girshick R, Donahue J, Darrell T, et al. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[C]. v, 2014: 580−587.
[18]	Liu W, Anguelov D, Erhan D, et al. SSD: Single Shot MultiBox Detector[C].In: European Conference on Computer Vision. Springer International Publishing, 2016: pp 21−37.
[19]	J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection, ” in Proc. In: IEEE Conf. Comput. Vis. Pattern Recognit. Jun. 2016, pp. 779−788
[20]	Chen J, Liu Z, Wang H, Núñez A, Han, Z. Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network. IEEE Transactions on Instrumentation and Measurement, 2017, 67(2): 257−269
[21]	Cha Y J, Choi W, Suh G, et al. Autonomous Structural Visual Inspection Using Region-Based Deep Learning for Detecting Multiple Damage Types. Computer Aided Civil & Infrastructure Engineering, 2017, 00(4): 1−17
[22]	Huang Y, Qiu C, Guo Y, Wang X, Yuan, K. (2018, August). Surface defect saliency of magnetic tile. In: 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE).IEEE Press: pp. 612−617.
[23]	Lingteng, Qiu, Xiaojun, et al. A High-Efficiency Fully Convolutional Networks for Pixel-Wise Surface Defect Detection. IEEE Access, 2019: 15884−15893
[24]	Yu W, Zhang Y, Shi H. Surface Defect Inspection Under a Small Training Set Condition. In: International Conference on Intelligent Robotics and Applications. Springer, Cham. 2019, August. pp. 517−528.
[25]	Masci J, Meier U, Cireşan D, Schmidhuber J. Stacked convolutional auto-encoders for hierarchical feature extraction. In International Conference on Artificial Neural Networks. Berlin, Heidelberg. Springer, 2011, June. pp. 52−59
[26]	Li Y, Zhao W, Pan J. Deformable patterned fabric defect detection with fisher criterion-based deep learning. IEEE Transactions on Automation Science and Engineering, 2016, 14(2): 1256−1264
[27]	Chalapathy R, Menon A K, Chawla S. Robust, Deep and Inductive Anomaly Detection.In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases: Springer, 2017.36−51.
[28]	袁静, 章毓晋. 融合梯度差信息的稀疏去噪自编码网络在异常行为检测中的应用. 自动化学报, 2017, 43(4): 604−610 Yuan Jing, Zhang YuJin. Application of sparse denoising autoencoder network with gradient difference information for abnormal action detection. Acta Automatica Sinica, 2017, 43(4): 604−610
[29]	Mei S, Yang H, Yin Z. An Unsupervised-Learning-Based Approach for Automated Defect Inspection on Textured Surfaces. IEEE Transactions on Instrumentation and Measurement, 2018: 1−12
[30]	D. Aiger and H. Talbot, “The phase only transform for unsupervised surface defect detection, ”.In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., IEEE press, Jun. 2010, pp. 295−302.
[31]	H.-D. Lin. Tiny surface defect inspection of electronic passive components using discrete cosine transform decomposition and cumulative sum techniques. Image Vis. Comput, 2008, 26(5): 603−621 doi: 10.1016/j.imavis.2007.07.009
[32]	Yang H, Chen Y, Song K, Yin Z. Multiscale Feature-Clustering-Based Fully Convolutional Autoencoder for Fast Accurate Visual Inspection of Texture Surface Defects. IEEE Transactions on Automation Science and Engineering. 2019, 1−18.
[33]	Makhzani A, Shlens J, Jaitly N, Goodfellow I, Frey B. (2015). Adversarial autoencoders. arXiv preprint arXiv: 1511.05644.
[34]	Zhao Z, Li B, Dong R, et al. A Surface Defect Detection Method Based on Positive Samples[C]. International Conference on Artificial Intelligence, Pacific Rim, Springer, Cham, 2018: 473−481.
[35]	Di He, et al. "Surface defect classification of steels with a new semi-supervised learning method." Optics and Lasers in Engineering, 2019, 117: 40−48.
[36]	Schlegl T, Seeböck P, Waldstein, S M, Schmidt-Erfurth U, Langs, G. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International Conference on Information Processing in Medical Imaging, Springer, Cham. 2017, June.pp. 146−157.
[37]	Paul Bergmann, Michael Fauser, David Sattlegger, Carsten Steger; A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), USA: IEEE Press: 2019.pp. 9592−9600
[38]	Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167. 2015.
[39]	Zhao H, Gallo O, Frosio I, Kautz J. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 2016, 3(1): 47−57
[40]	Bergmann P, Löwe S, Fauser M, et al. Improving unsupervised defect segmentation by applying structural similarity to autoencoders[J]. arXiv preprint arXiv: 1807.02011
[41]	Lv C, Zhang Z, Shen F, et al. A Fast Surface Defect Detection Method Based on Background Reconstruction[J].In: International Journal of Precision Engineering and Manufacturing, 2019: 1−13.
[42]	Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 2004, 13(4): 600−612 doi: 10.1109/TIP.2003.819861
[43]	Silvestre-Blanes J, Albero-Albero T, Miralles I, Pérez-Llorens R, Moreno J. A Public Fabric Database for Defect Detection Methods and Results. Autex Research Journal, 2019, 19(4): 363−374 doi: 10.2478/aut-2019-0035
[44]	Jager M, Knoll C, Hamprecht F A. Weakly supervised learning of a classifier for unusual event detection. IEEE Transactions on Image Processing, 2019, 17(9): 1700−1708
[45]	G. Kylberg and I. Sintorn. (2013). Kylberg Sintorn RotationDataset.[Online]. Available: http://www.cb.uu.se/~gustaf/KylbergSintornRotation/
[46]	Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. pp. 3431−3440.
[47]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention Springer, Cham. 2015, October. pp. 234−241.
[48]	He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. USA: IEEE Press, 2016.pp. 770−778.
[49]	Shi W, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, ... Wang, Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition USA: IEEE Press, 2016. pp. 1874−1883.
[50]	Huang G, Liu Z, Van Der Maaten L, Weinberger, K. Q.Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. USA: IEEE Press, 2017. pp. 4700−4708.
[51]	Tsai D M, Huang T Y. Automated surface inspection for statistical textures. Image and Vision computing, 2003, 21(4): 307−323 doi: 10.1016/S0262-8856(03)00007-6