Skip to content

Update options.md #1979

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bmaltais merged 3 commits into from
Feb 18, 2024
Merged

Update options.md #1979

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6d0a9ba
Merge pull request #1967 from bmaltais/dev
bmaltais Feb 15, 2024
f8d2673
Merge pull request #1975 from bmaltais/dev
bmaltais Feb 17, 2024
a63c49c
Update options.md
mikeboensel Feb 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions docs/LoRA/options.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,15 +177,15 @@ The default value is 0.0001.

### LR Scheduler:

You can change the learning rate in the middle of learning. A scheduler is a setting for how to change the learning rate.

adafactor: Select this to set the optimizer (described later) to Adafactor . Learn while automatically adjusting the learning rate according to the situation to save VRAM
constant: the learning rate does not change from beginning to end
constant_with_warmup: Start with a learning rate of 0 and gradually increase it toward the set value of Learning rate during warm-up, and use the set value of Learning rate during main learning.
cosine : Gradually decrease the learning rate toward 0 while drawing a wave (cosine curve)
cosine _with_restarts: repeat cosine many times (see also description of LR number of cycles)
linear: Start at the Learning rate setting and decrease linearly towards 0
polynomial: Same behavior as linear, but a bit more complicated to reduce (see also LR power description)
You can change the learning rate in the middle of learning. A scheduler is a setting for how to change the learning rate. Possible values include:

- `adafactor`: Select this to set the optimizer (described later) to Adafactor . Learn while automatically adjusting the learning rate according to the situation to save VRAM
- `constant`: the learning rate does not change from beginning to end
- `constant_with_warmup`: Start with a learning rate of 0 and gradually increase it toward the set value of Learning rate during warm-up, and use the set value of Learning rate during main learning.
- `cosine` : Gradually decrease the learning rate toward 0 while drawing a wave (cosine curve)
- `cosine _with_restarts`: repeat cosine many times (see also description of LR number of cycles)
- `linear`: Start at the Learning rate setting and decrease linearly towards 0
- `polynomial`: Same behavior as linear, but a bit more complicated to reduce (see also LR power description)
Set to constant if you want the learning rate to be fixed at the Learning rate setting.

Default is cosine
Expand Down Expand Up @@ -785,4 +785,4 @@ Here are some commonly used settings:

Default is blank. When the field is blank, the description example is displayed in faint color, so please refer to it.