Computational Photography: Keys, Methods and Applications
-
摘要: 针对现有计算机视觉、图形学、信号处理、数字图像处理、应用光学等领域无法通过现有成像模型与装置及计算方法获取足够目标场景信息的难题,计算摄像学研究提出新的成像机制与对应的计算重构方法,在光信号观测领域另辟蹊径,创新性地将视觉信息处理与计算前移至成像过程,从而极大地提高了信息优化计算的自由度,能够在维度、尺度与分辨率上实现质的突破,从而观测到传统成像系统看不清与看不见的场景信息.本文沿着计算摄像学思路、方法与目标三条主线,对国内外研究现状进行分析与综述,期望能够帮助读者更快地了解及进入相关研究.Abstract: Current imaging mechanisms and systems cannot capture sufficient visual information of target objects/scenes in many fields, such as computer vision, graphics, signal processing, digital image processing, applied optics, etc. To address these challenges, computational photography has proposed new imaging mechanisms and corresponding reconstruction methods that bring the visual information processing forward to the acquisition process and largely raise the degree of freedom on information optimization. The computational acquisition approaches are able to breakthrough the bottlenecks in dimension, scale, and resolution, and thus can observe the scenes that cannot be captured clearly by traditional imaging systems. This review focuses on three main aspects of computational photographystrategy, approach, and targetand attempts to familiarize the readers with the studies in this field.
-
[1] Adelson E H, Bergen J R. The plenoptic function and the elements of early vision. Computational Models of Visual Processing. Cambridge, MA:MIT Press, 1991. 3-20 [2] [2] Liu Y B, Dai Q H, Xu W L. A point-cloud-based multiview stereo algorithm for free-viewpoint video. IEEE Transactions on Visualization and Computer Graphics, 2010, 16(3):407-418 [3] [3] Adams A, Talvala E V, Park S H, Jacobs D E, Ajdin B, Gelfand M, Dolson J, Vaquero D, Baek J, Tico M, Lensch H P A, Matusik W, Pulli K, Horowitz M, Levoy M. The Frankencamera:an experimental platform for computational photography. ACM Transactions on Graphics, 2010, 29(4):Article No.29 [4] [4] Ng R, Levoy M, Brdif M, Duval G, Horowitz M, Hanrahan P. Light Field Photography with a Hand-Held Plenoptic Camera, Computer Science Technical Report CTSR 2005-02, Stanford, 2005. [5] [5] Ng R. Fourier slice photography. ACM Transactions on Graphics, 2005, 24(3):735-744 [6] [6] Johnson M K, Cole F, Raj A, Adelson E H. Microgeometry capture using an elastomeric sensor. ACM Transactions on Graphics, 2011, 30(4):Article No.46 [7] [7] Debevec P, Wenger A, Tchou C, Gardner A, Waese J, Hawkins T. A lighting reproduction approach to live-action compositing. ACM Transactions on Graphics, 2002, 21(3):547-556 [8] [8] Wenger A, Gardner A, Tchou C, Unger J, Hawkins T, Debevec P. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics, 2005, 24(3):756-764 [9] [9] Einarsson P, Chabert C F, Jones A, Ma W C, Lamond B, Hawkins T, Bolas M, Sylwan S, Debevec P. Relighting human locomotion with flowed reflectance fields. In:Proceedings of the 17th Eurographics Conference on Rendering Techniques. Aire-la-Ville, Switzerland, Switzerland:Eurographics Association, 2006. 183-194 [10] Wu D, Liu Y B, Ihrke I, Dai Q H, Theobalt C. Performance capture of high-speed motion using staggered multi-view recording. Computer Graphics Forum, 2012, 31(7):2019-2028 [11] Vlasic D, Peers P, Baran I, Debevec P, Popović J, Rusinkiewicz S, Matusik W. Dynamic shape capture using multi-view photometric stereo. ACM Transactions on Graphics, 2009, 28(5):Article No.174 [12] Ghosh A, Fyffe G, Tunwattanapong B, Busch J, Yu X, Debevec P. Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics, 2011, 30(6):Article No.129 [13] Tunwattanapong B, Fyffe G, Graham P, Busch J, Yu X M, Ghosh A, Debevec P. Acquiring reflectance and shape from continuous spherical harmonic illumination. ACM Transactions on Graphics, 2013, 32(4):Article No.109 [14] Chen T B, Lensch H P, Fuchs C, Seidel H P. Polarization and phase-shifting for 3D scanning of translucent objects. In:Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, USA:IEEE, 2007. 1-8 [15] Gupta M, Tian Y D, Narasimhan S G, Zhang L. Defocusing on global light transport for active scene recovery. In:Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL:IEEE, 2009. 2969-2976 [16] Zhang L, Nayar S. Projection defocus analysis for scene capture and image display. ACM Transactions on Graphics, 2006, 25(3):907-915 [17] Gupta M, Agrawal A, Veeraraghavan A, Narasimhan S G. Structured light 3D scanning in the presence of global illumination. In:Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI:IEEE, 2011. 713-720 [18] Gupta M, Nayar S K. Micro phase shifting. In:Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI:IEEE, 2012. 813-820 [19] Gu J W, Nayar S, Grinspun E, Belhumeur P, Ramamoorthi R. Compressive structured light for recovering inhomogeneous participating media. In:Proceedings of the 10th European Conference on Computer Vision. Berlin, Heidelberg:Springer, 2008. 845-858 [20] Cossairt O S, Miau D, Nayar S K. Scaling law for computational imaging using spherical optics. Journal of the Optical Society of America A, 2011, 28(12):2540-2553 [21] Brady D J, Gehm M E, Stack R A, Marks D L, Kittle D S, Golish D R, Vera E M, Feller S D. Multiscale gigapixel photography. Nature, 2012, 486(7403):386-389 [22] Wu C L, Liu Y B, Dai Q H, Wilburn B. Fusing multiview and photometric stereo for 3D reconstruction under uncalibrated illumination. IEEE Transactions on Visualization and Computer Graphics, 2011, 17(8):1082-1095 [23] Ben-Ezra M, Zomet A, Nayar S K. Video super-resolution using controlled subpixel detector shifts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(6):977-987 [24] Deng Y, Liu Y B, Dai Q H, Zhang Z K, Wang Y. Noisy depth maps fusion for multiview stereo via matrix completion. IEEE Journal of Selected Topics in Signal Processing, 2012, 6(5):566-582 [25] Heide F, Rouf M, Hullin M B, Labitzke B, Heidrich W, Kolb A. High-quality computational imaging through simple lenses. ACM Transactions on Graphics, 2013, 32(5):Article No.149 [26] Wetzstein G, Raskar R, Heidrich W. Hand-held schlieren photography with light field probes. In:Proceedings of the 2011 IEEE International Conference on Computational Photography. Pittsburgh, USA:IEEE, 2011. 1-8 [27] Wetzstein G, Roodnick D, Heidrich W, Raskar R. Refractive shape from light field distortion. In:Proceedings of the 2011 IEEE International Conference on Computer Vision. Barcelona, Spain:IEEE, 2011. 1180-1186 [28] Levoy M, Ng R, Adams A, Footer M, Horowitz M. Light field microscopy. ACM Transactions on Graphics, 2006, 25(3):924-934 [29] Liang C K, Lin T H, Wong B Y, Liu C, Chen H H. Programmable aperture photography:multiplexed light field acquisition. ACM Transactions on Graphics, 2008, 27(3):Article No.55 [30] Veeraraghavan A, Raskar R, Agrawal A, Mohan A, Tumblin J. Dappled photography:mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Transactions on Graphics, 2007, 26(3):Article No.69 [31] Raskar R, Agrawal A, Wilson C A, Veeraraghavan A. Glare aware photography:4d ray sampling for reducing glare effects of camera lenses. ACM Transactions on Graphics, 2008, 27(3):Article No.56 [32] Wilburn B, Joshi N, Vaish V, Talvala E V, Antunez E, Barth A, Adams A, Horowitz M, Levoy M. High performance imaging using large camera arrays. ACM Transactions on Graphics, 2005, 24(3):765-776 [33] Vaish V, Garg G, Talvala E V, Antunez E, Wilburn B, Horowitz M, Levoy M. Synthetic aperture focusing using a shear-warp factorization of the viewing transform. In:Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, D.C., USA:IEEE, 2005. 129 [34] Green P, Sun W Y, Matusik W, Durand F. Multi-aperture photography. ACM Transactions on Graphics, 2007, 26(3):Article No.68 [35] Levin A, Fergus R, Durand F, Freeman W T. Image and depth from a conventional camera with a coded aperture. ACM Transactions on Graphics, 2007, 26(3):Article No.70 [36] Dowski Jr E R, Johnson G E. Wavefront coding:a modern method of achieving high-performance and/or low-cost imaging systems. In:Proceedings of the 1999 SPIE Conference on Current Developments in Optical Design and Optical Engineering VⅢ. SPIE, 1999, 3779:137-145 [37] Levin A, Hasinoff S W, Green P, Durand F, Freeman W T. 4D frequency analysis of computational cameras for depth of field extension. ACM Transactions on Graphics, 2009, 28(3):Article No.97 [38] Lin X, Suo J L, Wetzstein G, Dai Q H, Raskar R. Coded focal stack photography. In:Proceedings of the 2013 IEEE International Conference on Computational Photography. Cambridge, USA:IEEE, 2013. 1-9 [39] Bando Y, Holtzman H, Raskar R. Near-invariant blur for depth and 2D motion via time-varying light field analysis. ACM Transactions on Graphics, 2013, 32(2):Article No.13 [40] Wetzstein G, Lanman D, Heidrich W, Raskar R. Layered 3D:tomographic image synthesis for attenuation-based light field and high dynamic range displays. ACM Transactions on Graphics, 2011, 30(4):Article No.95 [41] Lanman D, Wetzstein G, Hirsch M, Heidrich W, Raskar R. Polarization fields:dynamic light field display using multi-layer LCDs. ACM Transactions on Graphics, 2011, 30(6):Article No.186 [42] Wetzstein G, Lanman D, Hirsch M, Raskar R. Tensor displays:compressive light field synthesis using multilayer displays with directional backlighting. ACM Transactions on Graphics, 2012, 31(4):Article No.80 [43] Grosse M, Wetzstein G, Grundhoefer A, Bimber O. Coded aperture projection. ACM Transactions on Graphics, 2010, 29(3):Article No.22 [44] Ma C G, Suo J L, Dai Q H, Raskar R, Wetzstein G. High-rank coded aperture projection for extended depth of field. In:Proceedings of the 2013 IEEE International Conference on Computational Photography. Cambridge, USA:IEEE, 2013. 1-9 [45] Levin A, Freeman W T, Durand F. Understanding camera trade-offs through a Bayesian analysis of light field projections. In:Proceedings of the 10th European Conference on Computer Vision. Berlin, Heidelberg:Springer-Verlag, 2008. 88-101 [46] Bishop T E, Zanetti S, Favaro P. Light field superresolution. In:Proceedings of the 2009 IEEE International Conference on Computational Photography. San Francisco, USA:IEEE, 2009. 1-9 [47] Marwah K, Wetzstein G, Bando Y, Raskar R. Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Transactions on Graphics, 2013, 32(4):Article No.46 [48] Levin A, Durand F. Linear view synthesis using a dimensionality gap light field prior. In:Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, CA:IEEE, 2010. 1831-1838 [49] Lin X, Suo J L, Cao X, Dai Q H. Iterative feedback estimation of depth and radiance from defocused images. In:Proceedings of the 11th Asian Conference on Computer Vision. Berlin, Heidelberg:Springer, 2013. 95-109 [50] Karsch K, Hedau V, Forsyth D, Hoiem D. Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics, 2011, 30(6):Article No.157 [51] Mohan A, Raskar R, Tumblin J. Agile spectrum imaging:Programmable wavelength modulation for cameras and projectors. Computer Graphics Forum, 2008, 27(2):709-717 [52] Cao X, Du H, Tong X, Dai Q H, Lin S. A prism-mask system for multispectral video acquisition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(12):2423-2435 [53] Cao X, Tong X, Dai Q H, Lin S. High resolution multispectral video capture with a hybrid camera system. In:Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI:IEEE, 2011. 297-304 [54] Luo X Y, Zhang J, Dai Q H. A regional image fusion based on similarity characteristics. Signal Processing, 2012, 92(5):1268-1280 [55] Edgerton H E. Electronic flash, strobe (3rd edition). Cambridge:The MIT Press, 1987. [56] Theobalt C, Albrecht I, Haber J, Magnor M, Seidel H P. Pitching a baseball:tracking high-speed motion with multi-exposure images. ACM Transactions on Graphics, 2004, 23(3):540-547 [57] Narasimhan S G, Koppal S J, Yamazaki S. Temporal dithering of illumination for fast active vision. In:Proceedings of the 10th European Conference on Computer Vision. Berlin, Heidelberg:Springer, 2008. 830-844 [58] Veeraraghavan A, Reddy D, Raskar R. Coded strobing photography:compressive sensing of high speed periodic videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(4):671-686 [59] Bub G, Tecza M, Helmes M, Lee P, Kohl P. Temporal pixel multiplexing for simultaneous high-speed, high-resolution imaging. Nature Methods, 2010, 7(3):209-211 [60] Shechtman E, Caspi Y, Irani M. Increasing space-time resolution in video. In:Proceedings of the 7th European Conference on Computer Vision. Berlin, Heidelberg:Springer, 2002. 753-768 [61] Shechtman E, Caspi Y, Irani M. Space-time super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(4):531-545 [62] Wilburn B, Joshi N, Vaish V, Levoy M, Horowitz M. High-speed videography using a dense camera array. In:Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, D.C.:IEEE, 2004. Ⅱ-294-Ⅱ-301 [63] Agrawal A, Gupta M, Veeraraghavan A, Narasimhan S G. Optimal coded sampling for temporal super-resolution. In:Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, USA:IEEE, 2010. 599-606 [64] Liu Y B, Cao X, Dai Q H, Xu W L. Continuous depth estimation for multi-view stereo. In:Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL:IEEE, 2009. 2121-2128 [65] Li K, Dai Q H, Xu W L. Markless shape and motion capture from video sequences. IEEE Transactions on Circuits and System for Video Technology, 2011 21(3):320-334 [66] Li H, Luo L J, Vlasic D, Peers P, Popović J, Pauly M, Rusinkiewicz S. Temporally coherent completion of dynamic shapes. ACM Transactions on Graphics, 2012, 31(1):Article No.2 [67] Joshi N, Kang S B, Zitnick C L, Szeliski R. Image deblurring using inertial measurement sensors. ACM Transactions on Graphics, 2010, 29(4):Article No.30 [68] Nayar S, Ben-Ezra M. Motion-based motion deblurring. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(6):689-698 [69] Tai Y W, Du H, Brown M S, Lin S. Correction of spatially varying image and video motion blur using a hybrid camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(6):1012-1028 [70] Agrawal A, Xu Y, Raskar R. Invertible motion blur in video. ACM Transactions on Graphics, 2009, 28(3):Article No.95 [71] Raskar R, Agrawal A, Tumblin J. Coded exposure photography:motion deblurring using fluttered shutter. ACM Transactions on Graphics, 2006, 25(3):795-804 [72] Levin A, Sand P, Cho T S, Durand F, Freeman W T. Motion-invariant photography. ACM Transactions on Graphics, 2008, 27(3):Article No.71 [73] Cho T S, Levin A, Durand F, Freeman W T. Motion blur removal with orthogonal parabolic exposures. In:Proceedings of the 2010 IEEE International Conference on Computational Photography. Cambridge, USA:IEEE, 2010. 1-8 [74] Li W, Zhang J, Dai Q H. Exploring aligned complementary image pair for blind motion deblurring. In:Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI:IEEE, 2011. 273-280 [75] Fergus R, Singh B, Hertzmann A, Roweis S T, Freeman W T. Removing camera shake from a single photograph. ACM Transactions on Graphics, 2006, 25(3):787-794 [76] Shan Q, Jia J Y, Agarwala A. High-quality motion deblurring from a single image. ACM Transactions on Graphics, 2008, 27(3):Article No.73 [77] Cho S, Matsushita Y, Lee S. Removing non-uniform motion blur from images. In:Proceedings of the 11th IEEE International Conference on Computer Vision. Rio de Janeiro, Brazil:IEEE, 2007. 1-8 [78] Tai Y W, Tan P, Brown M S. Richardson-lucy deblurring for scenes under a projective motion path. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(8):1603-1618 [79] Whyte O, Sivic J, Zisserman A, Ponce J. Non-uniform deblurring for shaken images. International Journal of Computer Vision, 2012, 98(2):168-186 [80] Mahajan D, Huang F C, Matusik W, Ramamoorthi R, Belhumeur P. Moving gradients:a path-based method for plausible image interpolation. ACM Transactions on Graphics, 2009, 28(3):Article No.42 [81] Velten A, Willwacher T, Gupta O, Veeraraghavan A, Bawendi M G, Raskar R. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications, 2012, 3:Article No.745 [82] Velten A, Wu D, Jarabo A, Masia B, Barsi C, Joshi C, Lawson E, Bawendi M, Gutierrez D, Raskar R. Femto-photography:capturing and visualizing the propagation of light. 2013, 32(4):Article No.44 [83] Gupta O, Willwacher T, Velten A, Veeraraghavan A, Raskar R. Reconstruction of hidden 3d shapes using diffuse reflections. Optics Express, 2012, 20(17):19096-19108 [84] Kirmani A, Hutchison T, Davis J, Raskar R. Looking around the corner using transient imaging. In Proceedings of the 2009 IEEE International Conference on Computer Vision, Kyoto, Japan, 2009. 159-166 [85] Naik N, Zhao S, Velten A, Raskar R, Bala K. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Transactions on Graphics, 2011, 30(6):Article No.171 [86] Pandharkar R, Velten A, Bardagjy A, Lawson E, Bawendi M, Raskar R. Estimating motion and size of moving non-line-of-sight objects in cluttered environments. In:Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI:IEEE, 2011. 265-272 [87] Wu D, O'Toole M, Velten A, Agrawal A, Raskar R. Decomposing global light transport using time of flight imaging. In:Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, USA:IEEE, 2012. 366-373 [88] Wu D, Wetzstein G, Barsi C, Willwacher T, O'Toole M, Naik N, Dai Q, Kutulakos K, Raskar R. Frequency analysis of transient light transport with applications in bare sensor imaging. In:Proceedings of the 12th European Conference on Computer Vision. Berlin, Heidelberg:Springer, 2012. 542-555 [89] Talvala E V, Adams A, Horowitz M, Levoy M. Veiling glare in high dynamic range imaging. ACM Transactions on Graphics, 2007, 26(3):Article No.37 [90] Narasimhan S G, Nayar S K, Sun B, Koppal S J. Structured light in scattering media. In:Proceedings of the 10th IEEE International Conference on Computer Vision. Beijing, China:IEEE, 2005. 420-427 [91] Gupta M, Narasimhan S G, Schechner Y Y. On controlling light transport in poor visibility environments. In:Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, USA:IEEE, 2008. 1-8 [92] Debevec P E, Malik J. Recovering high dynamic range radiance maps from photographs. In:Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. New York:ACM, 2008. 369-378 [93] Kang S B, Uyttendaele M, Winder S, Szeliski R. High dynamic range video. ACM Transactions on Graphics, 2003, 22(3):319-325 [94] Narasimhan S G, Nayar S K. Vision and the atmosphere. International Journal of Computer Vision, 2002, 48(3):233-254 [95] Narasimhan S G, Nayar S K. Contrast restoration of weather degraded images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(6):713-724 [96] Narasimhan S G, Nayar S K. Interactive (de)weathering of an image using physical models. In:Proceedings of the 2003 IEEE Workshop on Color and Photometric Methods in Computer Vision. Nice, France:IEEE, 2003. 1-8 [97] Rempel A G, Trentacoste M, Seetzen H, Young H D, Heidrich W, Whitehead L, Ward G. Ldr2hdr:on-the-fly reverse tone mapping of legacy video and photographs. ACM Transactions on Graphics, 2007, 26(3):Article No.39 [98] Tan R T. Visibility in bad weather from a single image. In:Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, USA:IEEE, 2008. 1-8 [99] Fattal R. Single image dehazing. ACM Transactions on Graphics, 2008, 27(3):Article No.72 [100] He K M, Sun J, Tang X O. Single image haze removal using dark channel prior. In:Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL:IEEE, 2009. 1956-1963 [101] Tarel J P, Hautiere N. Fast visibility restoration from a single color or gray level image. In:Proceedings of the 2009 IEEE International Conference on Computer Vision. Kyoto, Japan:IEEE, 2009. 2201-2208 [102] Sen P, Chen B, Garg G, Marschner S R, Horowitz M, Levoy M, Lensch H P A. Dual photography. ACM Transactions on Graphics, 2005, 24(3):745-755 [103] Peers P, Mahajan D K, Lamond B, Ghosh A, Matusik W, Ramamoorthi R, Debevec P. Compressive light transport sensing. ACM Transactions on Graphics, 2009, 28(1):Article No.3 [104] Katz O, Small E, Silberberg Y. Looking around corners and through thin turbid layers in real time with scattered incoherent light. Nature Photonics, 2012, 6(8):549-553 [105] Greenbaum A, Luo W, Su T W, Grcs Z, Xue L, Isikman S O, Coskun A F, Mudanyali O, Ozcan A. Imaging without lenses:achievements and remaining challenges of wide-field on-chip microscopy. Nature Methods, 2012, 9(9):889-895
点击查看大图
计量
- 文章访问数: 3439
- HTML全文浏览量: 156
- PDF下载量: 2048
- 被引次数: 0