2.765

2022影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

De-DDPM: 可控、可迁移的缺陷图像生成方法

岳忠牧 张喆 赵瑞祥 吕武 马杰

岳忠牧, 张喆, 赵瑞祥, 吕武, 马杰. De-DDPM: 可控、可迁移的缺陷图像生成方法. 自动化学报, xxxx, xx(x): x−xx doi: 10.16383/j.aas.c230688
引用本文: 岳忠牧, 张喆, 赵瑞祥, 吕武, 马杰. De-DDPM: 可控、可迁移的缺陷图像生成方法. 自动化学报, xxxx, xx(x): x−xx doi: 10.16383/j.aas.c230688
Yue Zhong-Mu, Zhang Zhe, Zhao Rui-Xiang, Lv Wu, Ma Jie. De-DDPM: A controllable and transferable defect image generation method. Acta Automatica Sinica, xxxx, xx(x): x−xx doi: 10.16383/j.aas.c230688
Citation: Yue Zhong-Mu, Zhang Zhe, Zhao Rui-Xiang, Lv Wu, Ma Jie. De-DDPM: A controllable and transferable defect image generation method. Acta Automatica Sinica, xxxx, xx(x): x−xx doi: 10.16383/j.aas.c230688

De-DDPM: 可控、可迁移的缺陷图像生成方法

doi: 10.16383/j.aas.c230688
基金项目: 国家自然科学基金(U1913602, 61991412), 装备预先研究基金(50911020603)资助.
详细信息
    作者简介:

    岳忠牧:华中科技大学人工智能与自动化学院硕士研究生. 主要研究方向为缺陷数据生成,表面缺陷检测. E-mail: HUST_Y2021@163.com

    华中科技大学人工智能与自动化学院博士研究生. 主要研究方向为缺陷检测,缺陷数据生成,深度学习. 张喆与岳忠牧在本工作中同等贡献. E-mail: zhangzhe1997@hust.edu.cn

    赵瑞祥:中船航海科技有限责任公司工程师. 从事舰船导航系统设计及智能装备研发工作,主要研究方向为环境态势感知,船体缺陷检测. E-mail: zhaoruixiang12@126.com

    吕武:中船航海科技有限责任公司高级工程师. 从事船舶综合导航系统集成技术研究及智能装备研发工作,主要研究方向为综合导航,装备智能维护. E-mail: 18911990785@163.com

    马杰:华中科技大学人工智能与自动化学院教授. 主要研究方向为图像信息处理,目标检测与识别,无人艇环境感知. 本文通信作者. E-mail: majie@hust.edu.cn

De-DDPM: A Controllable and Transferable Defect Image Generation Method

Funds: Supported by National Natural Science Foundation of China (U1913602, 61991412) and the Foundation of Equipment Pre-research Area (50911020603).
More Information
    Author Bio:

    YUE Zhong-Mu Master student at the School of Artificial Intelligence and Automation, Huazhong University of Science and Technology. His research interest covers defect data generation and surface defect detection

    ZHANG Zhe Ph.D. candidate at the School of Artifical Intelligence and Automation, Huazhong University of Science and Technology. His research interest covers defect detection, defect data generation and deep learning. Zhang Zhe and Yue Zhong-Mu contributed equally to this work

    ZHAO Rui-Xiang Engineer at CSSC Marine Technology CO., LTD. Engaged in the design of ship navigation systems and the research and development of intelligent equipment. His research interest covers environmental situational awareness and hull defect detection

    LV Wu Senior engineer at CSSC Marine Technology CO., LTD. Engaged in the research of integrated technology of ship integrated navigation system and the development of intelligent equipment. His research interest covers integrated navigation and intelligent maintenance of equipment

    MA Jie Professor at the School of Artifical Intelligence and Automation, Huazhong University of Science and Technology. His research interest covers image information processing, target detection and identification, unmanned vessel environmental sensing. Corresponding author of this paper

  • 摘要: 基于深度学习的表面缺陷检测技术是工业上的一项重要应用, 而缺陷图像数据集质量对缺陷检测性能有重要影响. 为解决实际工业生产过程中缺陷样本获取成本高、缺陷数据量少的痛点, 提出了一种基于去噪扩散概率模型(Denoising Diffusion Probabilistic Model, DDPM)的缺陷生成方法. 该方法在训练过程中加强了模型对缺陷部位和无缺陷背景的差异化学习. 在生成过程中通过缺陷控制模块对生成缺陷的类别、形态、显著性等特征进行精准控制, 通过背景融合模块, 能将缺陷在不同的无缺陷背景上进行迁移, 大大降低新背景上缺陷样本的获取难度. 实验验证了该模型的缺陷控制和缺陷迁移能力, 其生成结果能有效扩充训练数据集, 提升下游缺陷检测任务的准确率.
  • 图  1  De-DDPM简要流程

    Fig.  1  Brief process of De-DDPM

    图  2  DDPM简要流程

    Fig.  2  Brief process of DDPM

    图  3  特征Unet网络结构

    Fig.  3  The structure of feature Unet network

    图  4  De-DDPM单步生成过程结构

    Fig.  4  The single-step generation process of De-DDPM

    图  5  缺陷控制模块结构

    Fig.  5  The structure of defect control module

    图  10  缺陷控制效果

    Fig.  10  Defect control effect

    图  6  背景融合模块结构

    Fig.  6  The structure of background fusion module

    图  11  不同背景下缺陷迁移效果

    Fig.  11  Defect migration effect in diverse backgrounds

    图  7  模型生成结果灰度分布统计

    Fig.  7  Statistics of gray scale distribution

    图  8  数据集扩充实验流程

    Fig.  8  The process of dataset augmentation experiment

    图  9  各模型生成结果

    Fig.  9  The results of each model

    表  1  评价指标统计

    Table  1  Statistics of evaluation metrics

    评价指标 Pix2pix StyleGAN2 DFMGAN RePaint模型 所提方法
    IS 1.388 1.368 1.301 1.474 1.541
    FID 59.056 124.748 96.783 72.750 57.650
    KID 0.024 0.098 0.053 0.044 0.020
    MS-SSIM 0.189 0.161 0.174 0.187 0.159
    PSNR 28.308 28.284 28.357 28.273 28.223
    * RePaint模型使用裂纹掩码生成缺陷效果差,改为区块掩码
    * 箭头标明评价指标得分更好的方向
    下载: 导出CSV

    表  2  分类结果统计

    Table  2  Statistics of classification results

    测试集Pix2pixStyleGAN2DFMGANRePaint模型所提方法
    缺陷检出率总正确率缺陷检出率总正确率缺陷检出率总正确率缺陷检出率总正确率缺陷检出率总正确率
    D130.7765.3826.9263.467.6953.8513.4654.8188.4694.23
    D254.4677.0749.9474.9732.2666.1345.7670.9988.3293.76
    *RePaint模型使用裂纹掩码生成缺陷效果差, 改为区块掩码
    下载: 导出CSV
  • [1] 张辉, 张邹铨, 陈煜嵘, 吴天月, 钟杭, 王耀南. 工业铸件缺陷无损检测技术的应用进展与展望. 自动化学报, 2022, 48(4): 935−956

    Zhang Hui, Zhang Zou-Quan, Chen Yu-Rong, Wu Tian-Yue, Zhong Hang, Wang Yao-Nan. Application advance and prospect of nondestructive testing technology for industrial casting defects. Acta Automatica Sinica, 2022, 48(4): 935−956
    [2] 罗东亮, 蔡雨萱, 杨子豪, 章哲彦, 周瑜, 白翔. 工业缺陷检测深度学习方法综述. 中国科学:信息科学, 2022, 52(6): 1002−1039 doi: 10.1360/SSI-2021-0336

    Luo Dong-Liang, Cai Yu-Xuan, Yang Zi-Hao, Zhang Zhe-Yan, Zhou Yu, Bai Xiang. Survey on industrial defect detection with deep learning. Scientia Sinica Informationis, 2022, 52(6): 1002−1039 doi: 10.1360/SSI-2021-0336
    [3] 陶晓天, 何博侠, 张鹏辉, 田德旭. 基于深度学习的航天密封圈表面缺陷检测. 仪器仪表学报, 2021, 42(1): 199−206

    Tao Xiao-Tian, He Bo-Xia, Zhang Peng-Hui, Tian De-Xu. Surface defect detection of aerospace sealing rings based on deep learning. Chinese Journal of Scientific Instrument, 2021, 42(1): 199−206
    [4] 田娟秀, 刘国才, 谷珊珊, 鞠忠建, 刘劲光, 顾冬冬. 医学图像分析深度学习方法研究与挑战. 自动化学报, 2018, 44(3): 401−424

    Tian Juan-Xiu, Liu Guo-Cai, Gu Shan-Shan, Ju Zhong-Jian, Liu Jin-Guang, Gu Dong-Dong. Deep learning in medical image analysis and its challenges. Acta Automatica Sinica, 2018, 44(3): 401−424
    [5] 王国力, 孙宇, 魏本征. 医学图像图深度学习分割算法综述. 计算机工程与应用, 2022, 58(12): 37−50 doi: 10.3778/j.issn.1002-8331.2112-0225

    Wang Guo-Li, Sun Yu, Wei Ben-Zheng. Systematic review on graph deep learning in medical image segmentation. Computer Engineering and Applications, 2022, 58(12): 37−50 doi: 10.3778/j.issn.1002-8331.2112-0225
    [6] 李书林, 冯朝路, 于鲲, 刘鑫, 江鑫, 赵大哲. 基于深度学习的心脏磁共振影像超分辨率前沿进展. 中国图象图形学报, 2022, 27(3): 704−721 doi: 10.11834/j.issn.1006-8961.2022.3.zgtxtxxb-a202203005

    Li Shu-Lin, Feng Chao-Lu, Yu Kun, Liu Xin, Jiang Xin, Zhao Da-Zhe. Critical review of human cardiac magnetic resonance image super resolution reconstruction based on deep learning method. Journal of Image and Graphics, 2022, 27(3): 704−721 doi: 10.11834/j.issn.1006-8961.2022.3.zgtxtxxb-a202203005
    [7] Redmon J, Divvala S, Girshick R, et al. You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016. 779−788
    [8] Liu W, Anguelov D, Erhan D, et al. Ssd: single shot multibox detector. In: Computer Vision–ECCV 2016: 14th European Conference. Amsterdam, The Netherlands: Springer International Publishing, 2016. 21−37
    [9] Ren S, He K, Girshick R, et al. Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39(6): 1137−1149
    [10] Redmon J, Farhadi A. Yolov3: an incremental improvement. arXiv: 1804.02767, 2018.
    [11] Santos C F G D, Papa J P. Avoiding overfitting: a survey on regularization methods for convolutional neural networks. ACM Computing Surveys (CSUR), 2022, 54(10s): 1−25
    [12] 陶显, 侯伟, 徐德. 基于深度学习的表面缺陷检测方法综述. 自动化学报, 2021, 47(05): 1017−1034

    Tao Xian, Hou Wei, Xu De. A survey of surface defect detection methods based on deep learning. Acta Automatica Sinica, 2021, 47(05): 1017−1034
    [13] Niu S, Li B, Wang X, et al. Defect image sample generation with GAN for improving defect recognition. IEEE Transactions on Automation Science and Engineering, 2020, 17(3): 1611−1622
    [14] 伍麟, 郝鸿宇, 宋友. 基于计算机视觉的工业金属表面缺陷检测综述. 自动化学报, DOI: 10.16383/j.aas.c230039

    Wu Lin, Hao Hong-Yu, Song You. A review of metal surface defect detection based on computer vision. Acta Automatica Sinica, DOI: 10.16383/j.aas.c230039
    [15] Rippel O, Müller M, Merhof D. GAN-based defect synthesis for anomaly detection in fabrics. In: 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). Vienna, Austria: IEEE, 2020. 1: 534−540
    [16] Zhang G, Cui K, Hung T Y, et al. Defect-GAN: high-fidelity defect synthesis for automated defect inspection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa, HI, USA: IEEE, 2021. 2524−2534
    [17] Zhang H, Pan D, Liu J, et al. A novel MAS-GAN-based data synthesis method for object surface defect detection. Neurocomputing, 2022, 499: 106−114 doi: 10.1016/j.neucom.2022.05.021
    [18] Wang R, Hoppe S, Monari E, et al. Defect transfer GAN: diverse defect synthesis for data augmentation. arXiv: 2302.08366, 2023.
    [19] 丁鹏, 卢文壮, 刘杰, 袁志响. 基于生成对抗网络的叶片表面缺陷图像数据增强. 组合机床与自动化加工技术, 2022(7): 18−21

    Ding Peng, Lu Wen-Jie, Liu Jie, Yuan Zhi-Xiang. image data augmentation of blade surface defects based on generative adversarial network. Modular Machine Tool & Automatic Manufacturing Technique, 2022(7): 18−21
    [20] Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 2021, 34: 8780−8794
    [21] Carlini N, Hayes J, Nasr M, et al. Extracting training data from diffusion models. In: 32nd USENIX Security Symposium (USENIX Security 23). ANAHEIM, CA, USA, 2023. 5253−5270
    [22] Jain S, Seth G, Paruthi A, et al. Synthetic data augmentation for surface defect detection and classification using deep learning. Journal of Intelligent Manufacturing, 20221−14
    [23] Zhang H, Cisse M, Dauphin Y N, et al. Mixup: beyond empirical risk minimization. arXiv: 1710.09412, 2017.
    [24] Zhang L, Deng Z, Kawaguchi K, et al. How does mixup help with robustness and generalization?. arXiv: 2010.04819, 2020.
    [25] Chou H P, Chang S C, Pan J Y, et al. Remix: rebalanced mixup. In: Proceedings of Computer Vision–ECCV 2020 Workshops. Glasgow, UK: Springer International Publishing, 2020. 95−110
    [26] Ren X, Lin W, Yang X, et al. Data augmentation in defect detection of sanitary ceramics in small and non-iid datasets. IEEE Transactions on Neural Networks and Learning Systems, 20228669−8678
    [27] Isola P, Zhu J Y, Zhou T, et al. Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, 2017. 1125−1134
    [28] Duan Y, Hong Y, Niu L, et al. Few-shot defect image generation via defect-aware feature manipulation. In: Proceedings of the AAAI Conference on Artificial Intelligence. Washington DC, USA: AAAI, 2023. 37(1): 571−578
    [29] Sohl-Dickstein J, Weiss E, Maheswaranathan N, et al. Deep unsupervised learning using nonequilibrium thermodynamics. In: 32nd International Conference on Machine Learning. PMLR, 2015. 2256−2265
    [30] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020, 33: 6840−6851
    [31] Ho J, Salimans T. Classifier-free diffusion guidance. arXiv: 2207.12598, 2022.
    [32] Saharia C, Ho J, Chan W, et al. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 4713−4726
    [33] Lugmayr A, Danelljan M, Romero A, et al. RePaint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA: IEEE, 2022. 11461−11471
    [34] Wang W, Bao J, Zhou W, et al. Semantic image synthesis via diffusion models. arXiv: 2207.00050, 2022.
    [35] Nichol A, Dhariwal P, Ramesh A, et al. Glide: towards photorealistic image generation and editing with text-guided diffusion models. arXiv: 2112.10741, 2021.
    [36] Ramesh A, Dhariwal P, Nichol A, et al. Hierarchical text-conditional image generation with clip latents. arXiv: 2204.06125, 2022, 1(2): 3.
    [37] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. New Orleans, LA, USA: IEEE, 2022. 10684−10695
    [38] Song Y, Sohl-Dickstein J, Kingma D P, et al. Score-based generative modeling through stochastic differential equations. arXiv: 2011.13456, 2020.
    [39] Tabernik D, Šela S, Skvarč J, et al. Segmentation-based deep-learning approach for surface-defect detection. Journal of Intelligent Manufacturing, 2020, 31(3): 759−776 doi: 10.1007/s10845-019-01476-x
    [40] Matthias Wieler, Tobias Hahn, Fred. A. Hamprecht. (2007) Weakly supervised learning for industrial optical inspection. [Dataset].
    [41] Song K, Yan Y. A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects. Applied Surface Science, 2013, 285(21): 858−864
    [42] Karras T, Laine S, Aittala M, et al. Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Seattle, WA, USA: IEEE, 2020. 8110−8119
    [43] Barratt S, Sharma R. A note on the Inception Score. arXiv: 1801.01973, 2018.
    [44] Heusel M, Ramsauer H, Unterthiner T, et al. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 2017, 30:
    [45] Bińkowski M, Sutherland D J, Arbel M, et al. Demystifying mmd gans. arXiv: 1801.01401, 2018.
    [46] Wang Z, Simoncelli E P, Bovik A C. Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers. IEEE, 2003. 2: 1398−1402
    [47] Huynh-Thu Q, Ghanbari M. Scope of validity of PSNR in image/video quality assessment. Electronics letters, 2008, 44(13): 800−801 doi: 10.1049/el:20080522
  • 加载中
计量
  • 文章访问数:  33
  • HTML全文浏览量:  22
  • 被引次数: 0
出版历程
  • 收稿日期:  2023-11-07
  • 录用日期:  2024-02-20
  • 网络出版日期:  2024-06-30

目录

    /

    返回文章
    返回