Resource: Training Stable Diffusion with Dreambooth

Blog Post

This article describes experiments with different Learning Rates in training models using Dreambooth.

FTA: Summary of Initial Results

To get good results training Stable Diffusion with Dreambooth, it’s important to tune the learning rate and training steps for your dataset.

  • High learning rates and too many training steps will lead to overfitting. The model will mostly generate images from your training data, no matter what prompt is used.
  • Low learning rates and too few steps will lead to underfitting: the model will not be able to generate the concept we were trying to incorporate.

Faces are harder to train. In our experiments, a learning rate of 2e-6 with 400 training steps works well for objects but faces required 1e-6 (or 2e-6) with ~1200 steps.

Image quality degrades a lot if the model overfits, and this happens if:

  • The learning rate is too high.
  • We run too many training steps.
  • In the case of faces, when no prior preservation is used, as shown in the next section.