2.765

2022影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

用于超分辨率重建的深度网络递进学习方法

张毅锋 刘袁 蒋程 程旭

张毅锋, 刘袁, 蒋程, 程旭. 用于超分辨率重建的深度网络递进学习方法. 自动化学报, 2020, 46(2): 274-282. doi: 10.16383/j.aas.2018.c180158
引用本文: 张毅锋, 刘袁, 蒋程, 程旭. 用于超分辨率重建的深度网络递进学习方法. 自动化学报, 2020, 46(2): 274-282. doi: 10.16383/j.aas.2018.c180158
ZHANG Yi-Feng, LIU Yuan, JIANG Cheng, CHENG Xu. A Curriculum Learning Approach for Single Image Super Resolution. ACTA AUTOMATICA SINICA, 2020, 46(2): 274-282. doi: 10.16383/j.aas.2018.c180158
Citation: ZHANG Yi-Feng, LIU Yuan, JIANG Cheng, CHENG Xu. A Curriculum Learning Approach for Single Image Super Resolution. ACTA AUTOMATICA SINICA, 2020, 46(2): 274-282. doi: 10.16383/j.aas.2018.c180158

用于超分辨率重建的深度网络递进学习方法

doi: 10.16383/j.aas.2018.c180158
基金项目: 

国家自然科学基金 61673108

国家自然科学基金 61802058

江苏省自然科学基金 BK20151102

北京大学机器感知与智能教育部重点实验室开放课题 K-2016-03

东南大学水声信号处理教育部重点实验室开放项目 UASP1502

详细信息
    作者简介:

    张毅锋   博士, 东南大学信息科学与工程学院副教授, IEEE高级会员.主要研究方向为计算机视觉, 机器学习, 数字水印与信息隐藏, 混沌神经信息处理和无线通信. E-mail: yfz@seu.edu.cn

    蒋程  东南大学信息科学与工程学院硕士研究生.主要研究方向为机器学习, 人脸检测. E-mail: 220150747@seu.edu.cn

    程旭   南京信息工程大学计算机与软件学院副教授, 2015年在东南大学获得博士学位.主要研究方向为计算机视觉和模式识别. E-mail: xcheng@nuist.edu.cn

    通讯作者:

    刘袁  东南大学信息科学与工程学院硕士研究生.主要研究方向为超分辨率, 视频理解, 语义分割.本文通信作者. E-mail: liuyuan@seu.edu.cn

A Curriculum Learning Approach for Single Image Super Resolution

Funds: 

National Natural Science Foundation of China 61673108

National Natural Science Foundation of China 61802058

National Natural Science Foundation of Jiangsu Province BK20151102

Opening Project of Key Laboratory of Machine Perception, Peking University K-2016-03

Opening Project of Key Laboratory of underwater acoustic signal processing, Southeast University UASP1502

More Information
    Author Bio:

    ZHANG Yi-Feng    Ph.D., associate professor at the School of Information Science and Engineering, Southeast University. Senior member of the IEEE. His research interest covers computer vision, machine learning, digital watermarking and information hiding, chaotic neural information processing, and wireless communication

    JIANG Cheng  Master student at the School of Information Science and Engineering, Southeast University. His research interest covers machine learning and face detection

    CHENG Xu   Associate professor at the School of Computer and Software, Nanjing University of Information Science and Technology. He received his Ph.D. degree in Information and Communication Engineering from Southeast University (Nanjing), in 2015. His research interest covers computer vision and pattern recognition

    Corresponding author: LIU Yuan   Master student at the School of Information Science and Engineering, Southeast University. His research interest covers super resolution, video understanding and semantic segmentation. Corresponding author of this paper
  • 摘要: 本文针对深度学习在单幅图像超分辨率方面难以恢复高频纹理细节的问题, 提出了一种基于递进学习的超分辨率算法.该算法首先采用灰度共生矩阵提取图像纹理特征, 然后利用基于密度峰值的聚类方法实现对整个训练集的分类, 其中每个训练子集具有相似的纹理复杂度.针对传统的递进学习方法会出现对已掌握知识"遗忘"的问题, 本文根据网络模型在各个训练子集上的拟合情况, 实时调整当前训练样本在各个子集上的概率分布, 从而实现快速收敛, 并获得更好的纹理细节复原效果.将本文提出的递进学习用于DRCN、VDSR、SRCNN等超分辨率网络的训练, 实验结果表明超分辨率网络收敛速度得到提升, 同时网络对复杂纹理等细节较多的图像也获得了较好的视觉恢复效果, 峰值信噪比则平均获得0.158 dB、0.18 dB、0.092 dB的提升.
    Recommended by Associate Editor WANG Liang
    1)  本文责任编委 王亮
  • 图  1  灰度共生矩阵示意图

    Fig.  1  The schematic diagram of gray-level co-occurrence matrix

    图  2  DRCN在不同学习方法下的训练收敛情况(测试集是Set14, 放大倍数为3倍)

    Fig.  2  The training performance of DRCN under different learning strategies on × 3 SR (The test set is Set14)

    图  3  VDSR在不同学习方法下的训练收敛情况(测试集是Set5, 放大倍数为4倍)

    Fig.  3  The training performance of VDSR under different learning strategies on × 4 SR (The test set is Set5)

    图  4  不同超分辨率算法视觉效果对比图(放大倍数为3倍)

    Fig.  4  Comparison of different algorithms in visual effects with upscaling factor 3

    表  1  不同聚类算法在CP、SP、DBI、DVI上的性能指标

    Table  1  The performance of different clustering algorithms in CP, SP, DBI and DVI

    FCM BIRCH MCLUST STING DP (ours)
    CP 3.04 2.19 2.86 2.32 1.78
    SP 2.67 2.96 3.08 3.12 3.89
    DBI 7.23 6.91 8.23 6.58 6.01
    DVI 0.52 0.57 0.49 0.55 0.63
    下载: 导出CSV

    表  2  基于不同聚类算法的VDSR在数据集Set5、Set14、BSD 100、Urban100上的性能指标

    Table  2  The performance of VDSR based on different clustering algorithms in Set5, Set14, BSD 100, and Urban100

    数据集 放大比例 VDSR
    (PSNR/SSIM)
    VDSR + FCM
    (PSNR/SSIM)
    VDSR + BI
    (PSNR/SSIM)
    VDSR + MC
    (PSNR/SSIM)
    VDSR + ST
    (PSNR/SSIM)
    VDSR + DP
    (PSNR/SSIM)
    Set5 × 2 37.53/0.9587 37.56/0.9462 37.68/0.9538 37.56/0.9589 37.65/0.9581 37.74/0.9592
    × 3 33.66/0.9213 33.67/0.9241 33.70/0.9258 33.71/0.9232 33.63/0.9222 33.79/0.9264
    × 4 31.35/0.8838 31.38/0.8799 31.53/0.8812 31.41/0.8861 31.47/0.8846 31.49/0.8897
    Set14 × 2 33.03/0.9124 33.06/0.9126 33.01/0.9112 33.07/0.9129 33.09/0.9125 33.11/0.9122
    × 3 29.77/0.8314 29.80/0.8352 29.86/0.8356 29.79/0.8329 29.81/0.8329 29.91/0.8402
    × 4 28.01/0.7674 28.12/0.7650 28.29/0.7710 28.13/0.7703 28.26/0.7717 28.32/0.7738
    BSD 100 × 2 31.90/0.8960 31.99/0.8978 31.93/0.9010 32.09/0.8992 32.05/0.8993 32.13/0.9071
    × 3 28.82/0.7976 28.84/0.7977 28.92/0.7954 28.89/0.7988 29.04/0.8004 29.11/0.8011
    × 4 27.29/0.7251 27.41/0.7196 27.32/0.7260 27.35/0.7273 27.32/0.7278 27.28/0.7310
    Urban100 × 2 30.76/0.9140 30.77/0.9139 30.74/0.9123 30.91/0.9169 30.84/0.9156 30.81/0.9193
    × 3 27.14/0.8279 27.22/0.8264 27.22/0.8282 27.29/0.8277 27.16/0.8288 27.35/0.8291
    × 4 25.18/0.7524 25.33/0.7569 25.21/0.7554 25.29/0.7551 25.32/0.7542 25.41/0.7567
    下载: 导出CSV

    表  3  不同算法在数据集Set5、Set14、BSD 100、Urban100上的性能指标

    Table  3  The performance of different algorithms in Set5, Set14, BSD 100, and Urban100

    数据集 放大比例 SRCNN
    (PSNR/SSIM)
    SRCNN + CL
    (PSNR/SSIM)
    VDSR
    (PSNR/SSIM)
    VDSR + CL
    (PSNR/SSIM)
    DRCN
    (PSNR/SSIM)
    DRCN + CL
    (PSNR/SSIM)
    Set5 × 2 36.66/0.9542 36.92/0.9623 37.53/0.9587 37.74/0.9592 37.63/0.9588 37.71/0.9591
    × 3 32.75/0.9090 32.81/0.9136 33.66/0.9213 33.79/0.9264 33.82/0.9226 33.91/0.9239
    × 4 30.48/0.8628 30.56/0.8623 31.35/0.8838 31.49/0.8897 31.53/0.8854 31.61/0.8896
    Set14 × 2 32.42/0.9063 32.63/0.9136 33.03/0.9124 33.11/0.9122 33.04/0.9118 33.11/0.9145
    × 3 29.28/0.8209 29.41/0.8261 29.77/0.8314 29.91/0.8402 29.76/0.8311 29.81/0.8423
    × 4 27.49/0.7503 27.62/0.7501 28.01/0.7674 28.32/0.7738 28.02/0.7670 28.13/0.7722
    BSD 100 × 2 31.36/0.8879 31.52/0.8935 31.90/0.8960 32.13/0.9071 31.85/0.8942 31.91/0.9062
    × 3 28.41/0.7863 28.63/0.7912 28.82/0.7976 29.11/0.8011 28.80/0.7963 28.92/0.8037
    × 4 26.90/0.7101 26.99/0.7234 27.29/0.7251 27.28/0.7310 27.23/0.7233 27.35/0.7274
    Urban100 × 2 29.50/0.8946 29.72/0.9064 30.76/0.9140 30.81/0.9193 30.75/0.9133 30.86/0.9201
    × 3 26.24/0.7989 26.41/0.8035 27.14/0.8279 27.35/0.8291 27.15/0.8276 27.23/0.8294
    × 4 24.52/0.7221 24.69/0.7316 25.18/0.7524 25.41/0.7567 25.14/0.7510 25.21/0.7572
    下载: 导出CSV
  • [1] Dong C, Chen C L, He K M, Tang X O. Image super-resolution using using deep convolutional networks. IEEE Transactions on Pattern & Machine Intelligence, 2016, 38(2): 295-307 http://d.old.wanfangdata.com.cn/Periodical/jsjfzsjytxxxb201709007
    [2] 胡长胜, 詹曙, 吴从中.基于深度特征学习的图像超分辨率重建.自动化学报, 2017, 43(5): 814-821 doi: 10.16383/j.aas.2017.c150634

    Hu Chang-Sheng, Zhan Shu, Wu Cong-Zhong. Image super-resolution reconstruction based on deep feature learning. Acta Automatica Sinica, 2017, 43(5): 814-821 doi: 10.16383/j.aas.2017.c150634
    [3] Sun X, Li X G, Li J F, Zhuo L. Review on deep learning based image super-resolution restoration algorithms. Acta Automatica Sinica, 2017, 43(5): 697-709
    [4] 李滔, 何小海, 卿粼波, 滕奇志.基于自适应块组割先验的噪声图像超分辨率重建.自动化学报, 2017, 43(5): 765-777 doi: 10.16383/j.aas.2017.c160268

    Li Tao, He Xiao-Hai, Qing Lin-Bo, Teng Qi-Zhi. Noisy image super-resolution reconstruction with adaptive patch-group-cuts prior. Acta Automatica Sinica, 2017, 43(5): 765-777 doi: 10.16383/j.aas.2017.c160268
    [5] Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA: IEEE, 2016, 1646-1654
    [6] Ledig C, Theis L, Huszár F, Caballero J, Cunn A, Acosta A, et al. Photo-realistic single image superresolution using a generative adversarial network, arXiv preprint, arXiv: 1609.04802, 2016
    [7] Lim B, Son S, Kim H, Nah S, MuLee K. Enhanced deep residual networks for single image super-resolution. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA: IEEE, July 2017, 1132-1140 http://www.researchgate.net/publication/318337451_Enhanced_Deep_Residual_Networks_for_Single_Image_Super-Resolution
    [8] Sajjadi M S, Scholkopf B, Hirsch M. Enhancenet: single image super-resolution through automated texture synthesis, arXiv preprint, arXiv: 1612.07919, 2016
    [9] Bengio Y, Louradour J, Collobert R, Weston J. Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. Montreal Quebec, Canada: ACM, 2009
    [10] Graves A, Bellemare G, Menick J, Munos R, Kavukcuoglu K. Automated curriculum learning for neural networks. arXiv preprint arXiv: 1704.03003, 2017
    [11] Hochreiter, Sepp, Jürgen Schmidhuber. LSTM can solve hard long time lag problems. Advances in Neural Information Processing Systems. 1997
    [12] Haralick R M, Shanmugam K. Textural features for image classification. IEEE Transactions on Systems, Man, and Cybernetics, 1973, (6): 610-621 http://d.old.wanfangdata.com.cn/OAPaper/oai_doaj-articles_16d3aac51c12c1b20a2512bd82d7cd5e
    [13] Rodriguez A, Laio A. Clustering by fast search and find of density peaks. Science, 2014, 344(6191): 1492 doi: 10.1126/science.1242072
    [14] Kim J, Lee J K, Lee K M. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA: IEEE, 2016, 1637-1645
    [15] Timofte R, Agustsson E, Gool L V, Yang Ming-Hsuan, Zhang L, Lim B, et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA: IEEE, 2017
    [16] Bevilacqua M, Roumy A, Guillemot C. Low-complexity singleimage super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 2012 British Machine Vision Conference, Surrey, UK: 2012, 135.1-135.10
    [17] Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations. In: Proceedings of the 2010 International conference on curves and surfaces. Avignon, France: Springer, 2010, 711-730 http://www.springerlink.com/content/56276x8370377023/
    [18] Martin D, Fowlkes C, Tal D. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the 8th IEEE International Conference on Computer Vision, 2001, 2(11): 416-423 http://www.researchgate.net/publication/3906161_A_database_of_human_segmented_natural_images_and_its_application_toevaluating_segmentation_algorithms_and_measuring_ecological_statistics
    [19] Huang J B, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars. In: Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, Massachusetts, USA: 2015, 5197-5206 http://www.researchgate.net/publication/275523282_Single_Image_Super-resolution_from_Transformed_Self-Exemplars
    [20] Abadi M, Agarwal A, Barham P, Brevdo E, Chen Zhi-Feng, Citro C, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint, arXiv: 1603.04467, 2016
    [21] Bezdek J C, Ehrlich R, Full W. FCM: the fuzzy c-means clustering algorithm. Computers & Geosciences, 1984, 10(2-3): 191-203 http://d.old.wanfangdata.com.cn/Periodical/gpxygpfx201012036
    [22] Zhang T, Raghu R, Miron L. BIRCH: an efficient data clustering method for very large databases. ACM Sigmod Record. 25(2): ACM, 1996
    [23] Fraley C, Raftery A E. MCLUST: Software for model-based cluster analysis. Journal of Classification, 1999, 16(2): 297-306 doi: 10.1007/s003579900058
    [24] Wang W, Jiong Y, Richard M. STING: A statistical information grid approach to spatial data mining. VLDB. Athens, Greece, Vol. 97, 1997: 186-195
    [25] Timofte R, Smet V D, Gool L V. A +: Adjusted anchored neighborhood regression for fast super-resolution. In: Proceedings of the 2014 Asian Conference on Computer Vision (ACCV). Singapore: Springer, 2014, 111-126
  • 加载中
图(4) / 表(3)
计量
  • 文章访问数:  3285
  • HTML全文浏览量:  561
  • PDF下载量:  319
  • 被引次数: 0
出版历程
  • 收稿日期:  2018-03-20
  • 录用日期:  2018-09-10
  • 刊出日期:  2020-03-06

目录

    /

    返回文章
    返回