keywords: CNN(Convolutional Neural Network), SICE，MEF

論文名稱：

提高影象對比度的方法分單影象提高對比度演算法(SICE)和同一場景多個曝光度合成的演算法(MEF)。MEF之後的圖片可以展現更多的圖片細節，但是由於要拍攝多個曝光度的原因，如果拍攝中有動態的物體的話，合成後會出現虛影(ghosting)。另一方面，SICE可以只用一張圖強化圖片對比度，所以不會出現ghosting。但是合成的效果不如MEF，圖片細節也不如MEF出來的圖片效果好。有沒有一種方法，既能沒有ghosting，又能合成高對比度的圖片呢？機器學習(ML)正好能做到這一點。如果輸入是一張圖片，label是同一場景下MEF合成的圖片，通過適當的學習就能達到MEF的效果。這篇文章有靈性的地方就在於巧妙的運用ML架起了橋，繞過了這個坎。

目的：用新的提取影象特徵的方法，結合不同的loss function用做實時的影象風格轉化和超解析度

重點和主要貢獻：

1. 提出了一個多曝光度的hdr圖片的資料庫，叫SICE。
2. 用learning-based演算法實現SICE，而且能達到MEF的效果。

==================================================================================

下面是部分重點的細節介紹，上面看懂的不用再往下看了。

==================================================================================

Review：

SICE方法：Histogram-based algorithms [4]–[6], which increase the contrast of an image by redistributing the luminous intensity on histogram. Retinex based algorithms [7]–[9], which enhance the reflectance and illumination components of the image separately.

以上SICE的缺點：

1. 複雜的自然場景和有限的影象資訊很難恢復：These methods, however, are difficult to reproduce a high-quality image due to the complex natural scenes and the limited information in a single low contrast image.  
2. 重新分配發光強度會使圖片強烈失真：Histogram-based methods [4], [5] attempt to redistribute the luminous intensity on histogram in a global or local manner.  However, such simple redistribution operations may produce serious unrealistic effects in the enhanced images since they ignore image structural information [26]
3. 多數SICE演算法是基於圖片是高質量圖片的假設，而沒有充分利用(exploit)影象資訊(這是作者說的)：Most of the previous SICE methods are based on some assumptions on high-quality images, while they may not fully exploit the information in the input image.
4. [9]說SICE增強能力有限，因為受限於低對比度影象的有限的資訊：the enhancement capability of existing SICE methods is rather limited due to the limited information in a single low-contrast image .

MEF方法: [10], [11], development of imaging devices are able to capture a sequence of multi-exposure images in a short time to fulfil the dynamic range of a scene. Multi-exposure image fusion (MEF) [2], [12], [13] and stack-based high dynamic range (HDR) imaging methods [14], [15] can be applied to blend the multiple images with different exposures into a perceptually more appealing image.

MEF和stack-based HDR方法的區別：stack-based HDR merges bracketed multiple exposure images into an HDR irradiance map, then employ a tone mapping operator to compress the dynamic range of HDR irradiance map so that the high-contrast image can be displayed on regular monitors. MEF methods attempt to fuse the images directly in the non-liner brightness domain to reproduce a high-visibility image.

電子影象因為動態範圍而失去細節：[30] Because of the limited dynamic range, traditional digital imaging systems may lose structural details when shooting a natural scene.

CNN方法：Methods [28], [31] have been proposed to train a CNN network to map the low dynamic range (LDR) images to HDR images. In [29], a CNN is trained to set the parameters of bilateral filters, which are then used to enhance an input image to a desired image edited by professional photographers.

Ghosting問題: [16], [17] the acquisition of multi-exposure images will complicate the imaging process, and camera shake or moving objects will lead to unpleasant fusion artifacts such as the ghosting artifacts .

Deghosting方法: [2], [14], [32],其中[2] is the state-of-the-art deghosting MEF algorithm.  Learning-based methods like [33] proposed to map the multi-exposure image sequences to an HDR image.

SICE是為了提高DR下拍攝的低對比度影象的對比度：[25] Single image contrast enhancement (SICE) aims to improve the visibility of the scene in a given single low-contrast image. It provides a way to enhance the low contrast photographs captured from a high dynamic range scene。

CNN的用途：

1. 去噪 denoising [34], [35]
2. 超解析度 super-resolution [21]
3. 去模糊 deblurring [36]

以上應用很容易弄出訓練資料和高質量的label：In those applications, pairs of degraded images and their high-quality coun- terparts can be easily generated. With those paired training data, CNN can be used to learn a mapping function between the degraded observations and their corresponding high-quality reference images.

但是SICE這種基於規則的資料對於真實世界的低對比度影象太理想，CNN不好學習：However, for the application of SICE, computer-generated training datasets are too ideal to be true for real-world low-contrast images, where the distribution of luminance is much more complex and varies with different scenes, cameras and camera settings.

[37], [38]有多曝光度的資料庫，但是數量少，不狗多樣化：Some multi-exposure image sequences are available in literature [37], [38]. However, the total amount of such publicly available sequences is very limited, and many of them were taken under indoor environ- ment. Neither the number of sequences nor the diversity of sequence exposure levels meets the requirement of real-world applications.

收集資料的裝置：Seven types of consumer grade cameras are used to collect the image sequences, including Sony α7RII, Sony NEX-5N, Canon EOS-5D Mark II, Canon EOS-750D, Nikon D810, Nikon D7100 and iPhone 6s.

跟13種MEF對比：13 state-of-the-art MEF and HDR algorithms are employed in this process, including 8 MEF methods: Mertens09 [13], Raman09 [39], Shen11 [40], Zhang12 [41], Li13 [12], Shen14 [42], Ma17 [2], Kou17 [43], and 5 stack- based HDR methods: Sen12 [14], Hu13 [32], Bruce14 [44], Oh15 [15], Photomatix [45].

用MEF-SSIM [16]對比評分，選出最高的做label。

作者嘗試直接用DNN學習，像下面的網路結構，但是效果不佳。

[49]說低頻資訊代表整體自然度，高頻資訊代表區域性細節：Retinex theory [49], the low-frequency information of an image represents the global naturalness, and the high-frequency information represents the local details.[50]–[53]拆解影象到高低頻率：have been proposed to decompose an image into high and low frequency components to preserve image details and colors in the brightest/darkest regions.

然後作者有了想法，把影象拆成兩個頻率，低頻用cnn加一次residual，高頻用unet。分別算不同的loss。最後加在一起用第三個網路微調。然後作者覺得棒棒噠了~

網路細節如下：

作者用PSNR和FSIM [60]來衡量圖片質量。用實驗作對比。結構好像是依賴主觀的想象，並沒有做出深入的分析為什麼用某個結構，某個loss，為什麼用residual。

最後有意思的是作者列出了failure case(很嚴肅，絕對不是幸災樂禍啊). 

個人感覺，深入研究這些failure cases，才是發NB論文的正道。

作者對問題的描述：However, it is also found that our method may fail to recover the details for large and severely over- exposed regions. Figure 18 shows an example. One can see that the missing color and structures in the color chart are not recovered, while the details can be seen in the reference image generated by MEF/HDR methods.

作者認為是過曝光太嚴重了，區域又大，資訊太少卷積不出來：The reason for the failure may be that the over-exposure is too severe (in terms of both level and area) so that there is little information the CNN can use to synthesize the missing details in the neighborhood.

Ref:

專案主頁 - http://vllab.ucmerced.edu/wlai24/LapSRN/