Denoising Diffusion Probabilistic Models

Denoising Diffusion Implicit Models

Diffusion Model Clearly Explained! by Steins Medium-min.pdf

DDPM:

adding gaussian noise and trying to restore from noisy to original

q forward pass: adding random noise on a fixed schedule

A closed form formula allows us to directly sample a noisy image at specific time t in one step

$$ z \sim N(\mu, \sigma^2) \\ z = \mu + \sigma \epsilon \ \text{where} \ \epsilon \sim N(0,1) $$

Therefore, we can express $x_t$ with $x_{t-1}$

$$ x_t = \sqrt{1-\beta_t} \ x_{t-1}\ + \ \sqrt{\beta_t} \ \epsilon_{t-1}\\ = \sqrt{\alpha_t}x_{t-1} + \sqrt{1-\alpha_t}\epsilon_{t-1} $$

$$ \begin{aligned}\mathbf{x}t &= \sqrt{\alpha_t}\mathbf{x}{t-1} + \sqrt{1 - \alpha_t}\boldsymbol{\epsilon}{t-1} & \text{ ;where } \boldsymbol{\epsilon}{t-1}, \boldsymbol{\epsilon}{t-2}, \dots \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) \\&= \sqrt{\alpha_t \alpha{t-1}} \mathbf{x}{t-2} + \sqrt{1 - \alpha_t \alpha{t-1}} \bar{\boldsymbol{\epsilon}}{t-2} & \text{ ;where } \bar{\boldsymbol{\epsilon}}{t-2} \text{ merges two Gaussians (*).} \\&= \dots \\&= \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\boldsymbol{\epsilon} \\q(\mathbf{x}_t \vert \mathbf{x}_0) &= \mathcal{N}(\mathbf{x}_t; \sqrt{\bar{\alpha}_t} \mathbf{x}_0, (1 - \bar{\alpha}_t)\mathbf{I})\end{aligned} $$

Now, we can directly sample $x_t$ at any timestep

p reverse pass: learning gaussian transitions to reconstruct original image

$$ p_\theta(\mathbf{x}{0:T}) = p(\mathbf{x}T) \prod^T{t=1} p\theta(\mathbf{x}{t-1} \vert \mathbf{x}t) \quad p\theta(\mathbf{x}{t-1} \vert \mathbf{x}t) = \mathcal{N}(\mathbf{x}{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}t, t), \boldsymbol{\Sigma}\theta(\mathbf{x}_t, t)) $$

train to estimate the right mean and covariance matrix

negative log likelihood: loss = $-log(p_\theta(x_\theta))$