  Deep image prior (DIP) serves as a good inductive bias for diverse inverse
problems. Among them, denoising is known to be particularly challenging for the
DIP due to noise fitting with the requirement of an early stopping. To address
the issue, we first analyze the DIP by the notion of effective degrees of
freedom (DF) to monitor the optimization progress and propose a principled
stopping criterion before fitting to noise without access of a paired ground
truth image for Gaussian noise. We also propose the `stochastic temporal
ensemble (STE)' method for incorporating techniques to further improve DIP's
performance for denoising. We additionally extend our method to Poisson noise.
Our empirical validations show that given a single noisy image, our method
denoises the image while preserving rich textual details. Further, our approach
outperforms prior arts in LPIPS by large margins with comparable PSNR and SSIM
on seven different datasets.
