Fast Bayesian Uncertainty Estimation and reduction of Batch Normalized Single Image Super-Resolution Network

Aupendu Kar, Prabir Kumar Biswas

Department of Electronics and Electrical Communication Engineering
Indian Institute of Technology Kharagpur, India

the left image is an LR image,
and the model’s uncertainty during upscaling is in the right.

In the first row images, the left image is an LR image, and the model’s uncertainty during upscaling is in the right. Two cropped regions have different textures. We can observe from two patches in the second and third-row that whenever the model fails to reconstruct the texture correctly (second-row image), it leads to higher uncertainty.

Abstract

Convolutional neural network (CNN) has achieved unprecedented success in image super-resolution tasks in recent years. However, the network's performance depends on the distribution of the training sets and degrades on out-of-distribution samples. This paper adopts a Bayesian approach for estimating uncertainty associated with output and applies it in a deep image super-resolution model to address the concern mentioned above. We use the uncertainty estimation technique using the batch-normalization layer, where stochasticity of the batch mean and variance generate Monte-Carlo (MC) samples. The MC samples, which are nothing but different super-resolved images using different stochastic parameters, reconstruct the image, and provide a confidence or uncertainty map of the reconstruction. We propose a faster approach for MC sample generation, and it allows the variable image size during testing. Therefore, it will be useful for image reconstruction domain. Our experimental findings show that this uncertainty map strongly relates to the quality of reconstruction generated by the deep CNN model and explains its limitation. Furthermore, this paper proposes an approach to reduce the model's uncertainty for an input image, and it helps to defend the adversarial attacks on the image super-resolution model. The proposed uncertainty reduction technique also improves the performance of the model for out-of-distribution test images. To the best of our knowledge, we are the first to propose an adversarial defense mechanism in any image reconstruction domain.

Highlights

We propose a faster implementation of Monte Carlo batch-normalization uncertainty to generate MC samples and overcome the hurdle of variable image size.
We address the uncertainty of deep SISR models using Bayesian approach to measure it. To the best of our knowledge, we are the first to estimate uncertainty in deep learning models for image reconstruction. We also discuss the advantages and implications of uncertainty in SISR and its usefulness in understanding the model's limitations and outcome.
We also propose an approach to reduce the model uncertainty for a test image. The uncertainty reduction method acts as a defense against the adversarial attack on the SISR model and proves to be beneficial for out-of-distribution noisy low-resolution images.

Modified Method for Uncertainty Estimation

Proposed Method for Uncertainty Reduction

Image of Proposed Algorithm. — Algorithm 3

A plotting — Effect of perturbation level for adversarial defense

Understanding Uncertainty

PSNR vs MC Samples Image. Present effect of PSNR and uncertainty with the increase of MC samples — PSNR vs MC Samples

Uncertainty vs MC Samples. Present effect of PSNR and uncertainty with the increase of MC samples — Uncertainty vs MC Samples

PSNR vs Uncertainty. Present impact of PSNR and adv. noise with uncertainty — PSNR vs Uncertainty

Uncertainty vs Adv. Noise Level. Present impact of PSNR and adv. noise with uncertainty — Uncertainty vs Adv. Noise Level

Uncertainty vs Scale. Present the influence of Bayesian uncertainty with different Scale factor models and random noise on LR images — Uncertainty vs Scale

Uncertainty vs Noise Level. Present the influence of Bayesian uncertainty with different Scale factor models and random noise on LR images — Uncertainty vs Noise Level

(a)-(b) present effect of PSNR and uncertainty with the increase of MC samples and (c)-(d) present impact of PSNR and adv. noise with uncertainty. (e)-(f) present the influence of Bayesian uncertainty with different Scale factor models and random noise on LR images. (click on the image for the best view)

Performance Analysis of Adversarial Defense

A table showing Quantitative evaluation of our proposed Bayesian
uncertainty reduction technique — Table 2. Quantitative evaluation of our proposed Bayesian uncertainty reduction technique based adversarial defense mechanism.

Visual Results of Adversarial Defense

Image of a man's face. Adversarial LR — Adversarial LR

Image of a man's face. Input Image. SR (No Def.) — SR (No Def.)

Image of a man's face. Input Image. SR (After Def.) — SR (After Def.)

Image of a man's face. Input Image. UN (No Def.) — UN (No Def.)

Image of a man's face. Input Image. UN (After Def.) — UN (After Def.)

Performance of our proposed adversarial defense mechanism using the Bayesian uncertainty reduction technique. The cropped region shows undesired artifacts in the SR image without any defense, and our defense mechanism successfully suppresses those artifacts.

Downloads

Poster

Code

References

Kim, J., Lee, J.K. and Lee, K.M., 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1646-1654).
Choi, J.H., Zhang, H., Kim, J.H., Hsieh, C.J. and Lee, J.S., 2019. Evaluating robustness of deep image super-resolution against adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 303-311).
Gal, Y. and Ghahramani, Z., 2016, June. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). PMLR.
Teye, M., Azizpour, H. and Smith, K., 2018, July. Bayesian uncertainty estimation for batch normalized deep networks. In International Conference on Machine Learning (pp. 4907-4916). PMLR.
Kendall, A., Badrinarayanan, V. and Cipolla, R., 2017, July. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. In British Machine Vision Conference 2017, BMVC 2017.

Citation (BibTeX)

@InProceedings{Kar_2021_CVPR, author = {Kar, Aupendu and Biswas, Prabir Kumar},
title = {Fast Bayesian Uncertainty Estimation and reduction of Batch Normalized Single Image Super-Resolution Network},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2021},
pages = {4957-4966}
}