Prodigy is not working well with Stable Diffusion 3.5 Medium training

[turbo-boo]

I have been trying to train with the stock settings of this optimizer and have not been successful yet. Specifically, it seems that it is not learning nearly as well as it should.

adamW8bit seems to be working with 4e-4 lr

[LoganBooker]

Hey Bocchi-Chan2023, are you able to take note of the value of d during training? If it rises very slowly or not at all, you might need to bump up d0 (to say, 1e-5 or 1e-4). I've found sometimes Prodigy just needs to get a larger read of the gradients to start with, otherwise it can take quite a few steps before it finds a good LR, by which point you're already a good portion through training.

For example, here's the results of an SDXL LoRA training, batch 8, with a modified Prodigy that treats each parameter group independently (#20). From left to right, the graphs show the d value for TE1, TE2 and the Unet.

As you can see, both TEs hit a good LR quickly, but the Unet took until steps 200-300 to find a decent LR, and even then it continued to search. I've been experimenting with ways to combat this but haven't been successful so far.

Also double check you're setting the regular LRs to 1 (as the LR is multiplied by d), and I'd also suggest using betas of (0.9,0.99) if you're not already (as suggested here: #8 (comment)). If beta3 is not set explicitly, then beta2 ** 0.5 is used in its place, so beta2 affects more than just the second moment.

Not sure if any of this will help, but sharing my experiences while playing around with the internals.

[turbo-boo]

Okay, I will try to record the value of d while adjusting the value of d0

[turbo-boo]

I set d0 from 1e-6 to 1e-5 and prodigy seems to be working well!

[konstmish]

Thanks for sharing your experience and a especially thanks to @LoganBooker for giving a solution to the problem. Since the problem seems to be solved, I'm closing the issue, but feel free to reopen it if you have more questions. I'll also add a comment on changing d0 in the readme with a link to this discussion.

[AdAstra-93]

Hey Bocchi-Chan2023, are you able to take note of the value of d during training? If it rises very slowly or not at all, you might need to bump up d0 (to say, 1e-5 or 1e-4). I've found sometimes Prodigy just needs to get a larger read of the gradients to start with, otherwise it can take quite a few steps before it finds a good LR, by which point you're already a good portion through training.

For example, here's the results of an SDXL LoRA training, batch 8, with a modified Prodigy that treats each parameter group independently (#20). From left to right, the graphs show the d value for TE1, TE2 and the Unet.

As you can see, both TEs hit a good LR quickly, but the Unet took until steps 200-300 to find a decent LR, and even then it continued to search. I've been experimenting with ways to combat this but haven't been successful so far.

Also double check you're setting the regular LRs to 1 (as the LR is multiplied by d), and I'd also suggest using betas of (0.9,0.99) if you're not already (as suggested here: #8 (comment)). If beta3 is not set explicitly, then beta2 ** 0.5 is used in its place, so beta2 affects more than just the second moment.

Not sure if any of this will help, but sharing my experiences while playing around with the internals.

hi, @LoganBooker, just a quick question, how did you get those graphs? how can I keep an eye on the LRs during training (I'm using LoRA_Easy_Training_scripts, which uses kohya sd_scripts under the hood)?

[arledesma]

@LoRAMilkshake those are from tensorboard. activate your venv and run tensorboard.
tensorboard serve --logdir .

The specific timeseries would be under the lr section (but only if you are using the changes from #34)