This article describes experiments with different Learning Rates in training models using Dreambooth.
FTA: Summary of Initial Results
To get good results training Stable Diffusion with Dreambooth, it’s important to tune the learning rate and training steps for your dataset.
- High learning rates and too many training steps will lead to overfitting. The model will mostly generate images from your training data, no matter what prompt is used.
- Low learning rates and too few steps will lead to underfitting: the model will not be able to generate the concept we were trying to incorporate.
Faces are harder to train. In our experiments, a learning rate of 2e-6 with 400 training steps works well for objects but faces required 1e-6 (or 2e-6) with ~1200 steps.
Image quality degrades a lot if the model overfits, and this happens if:
- The learning rate is too high.
- We run too many training steps.
- In the case of faces, when no prior preservation is used, as shown in the next section.