Loss is nan #19
Description
#----------Creating logger----------#
#----------GPU init----------#
#----------Preparing dataset----------#
#----------Prepareing Models----------#
[H_SS2D] 2 order with dims= [8, 16] scale=0.3333
[H_SS2D] 2 order with dims= [8, 16] scale=0.3333
[H_SS2D] 3 order with dims= [8, 16, 32] scale=0.3333
[H_SS2D] 3 order with dims= [8, 16, 32] scale=0.3333
[H_SS2D] 4 order with dims= [8, 16, 32, 64] scale=0.3333
[H_SS2D] 4 order with dims= [8, 16, 32, 64] scale=0.3333
[H_SS2D] 5 order with dims= [8, 16, 32, 64, 128] scale=0.3333
[H_SS2D] 5 order with dims= [8, 16, 32, 64, 128] scale=0.3333
SC_Att_Bridge was used
[H_SS2D] 5 order with dims= [16, 32, 64, 128, 256] scale=0.3333
[H_SS2D] 5 order with dims= [16, 32, 64, 128, 256] scale=0.3333
[H_SS2D] 4 order with dims= [16, 32, 64, 128] scale=0.3333
[H_SS2D] 4 order with dims= [16, 32, 64, 128] scale=0.3333
[H_SS2D] 3 order with dims= [16, 32, 64] scale=0.3333
[H_SS2D] 3 order with dims= [16, 32, 64] scale=0.3333
[H_SS2D] 2 order with dims= [16, 32] scale=0.3333
[H_SS2D] 2 order with dims= [16, 32] scale=0.3333
#----------Prepareing loss, opt, sch and amp----------#
#----------Set other params----------#
#----------Training----------#
train: epoch 1, iter:0, loss: nan, lr: 0.001
train: epoch 1, iter:20, loss: nan, lr: 0.001
train: epoch 1, iter:40, loss: nan, lr: 0.001
train: epoch 1, iter:60, loss: nan, lr: 0.001
train: epoch 1, iter:80, loss: nan, lr: 0.001
train: epoch 1, iter:100, loss: nan, lr: 0.001
train: epoch 1, iter:120, loss: nan, lr: 0.001
train: epoch 1, iter:140, loss: nan, lr: 0.001
100%|██████████| 150/150 [00:05<00:00, 27.16it/s]
val epoch: 1, loss: nan
train: epoch 2, iter:0, loss: nan, lr: 0.0009990232305719944
train: epoch 2, iter:20, loss: nan, lr: 0.0009990232305719944
train: epoch 2, iter:40, loss: nan, lr: 0.0009990232305719944
train: epoch 2, iter:60, loss: nan, lr: 0.0009990232305719944
train: epoch 2, iter:80, loss: nan, lr: 0.0009990232305719944
train: epoch 2, iter:100, loss: nan, lr: 0.0009990232305719944
train: epoch 2, iter:120, loss: nan, lr: 0.0009990232305719944
train: epoch 2, iter:140, loss: nan, lr: 0.0009990232305719944
100%|██████████| 150/150 [00:06<00:00, 24.69it/s]
val epoch: 2, loss: nan
train: epoch 3, iter:0, loss: nan, lr: 0.0009960967771506664
train: epoch 3, iter:20, loss: nan, lr: 0.0009960967771506664
train: epoch 3, iter:40, loss: nan, lr: 0.0009960967771506664
train: epoch 3, iter:60, loss: nan, lr: 0.0009960967771506664
train: epoch 3, iter:80, loss: nan, lr: 0.0009960967771506664
train: epoch 3, iter:100, loss: nan, lr: 0.0009960967771506664
train: epoch 3, iter:120, loss: nan, lr: 0.0009960967771506664
train: epoch 3, iter:140, loss: nan, lr: 0.0009960967771506664
100%|██████████| 150/150 [00:05<00:00, 26.92it/s]
val epoch: 3, loss: nan
train: epoch 4, iter:0, loss: nan, lr: 0.0009912321891107007
train: epoch 4, iter:20, loss: nan, lr: 0.0009912321891107007
train: epoch 4, iter:40, loss: nan, lr: 0.0009912321891107007
train: epoch 4, iter:60, loss: nan, lr: 0.0009912321891107007
train: epoch 4, iter:80, loss: nan, lr: 0.0009912321891107007
train: epoch 4, iter:100, loss: nan, lr: 0.0009912321891107007
train: epoch 4, iter:120, loss: nan, lr: 0.0009912321891107007
train: epoch 4, iter:140, loss: nan, lr: 0.0009912321891107007
100%|██████████| 150/150 [00:05<00:00, 26.66it/s]
val epoch: 4, loss: nan
train: epoch 5, iter:0, loss: nan, lr: 0.0009844486647586721
train: epoch 5, iter:20, loss: nan, lr: 0.0009844486647586721
train: epoch 5, iter:40, loss: nan, lr: 0.0009844486647586721
train: epoch 5, iter:60, loss: nan, lr: 0.0009844486647586721
train: epoch 5, iter:80, loss: nan, lr: 0.0009844486647586721
train: epoch 5, iter:100, loss: nan, lr: 0.0009844486647586721
train: epoch 5, iter:120, loss: nan, lr: 0.0009844486647586721
train: epoch 5, iter:140, loss: nan, lr: 0.0009844486647586721
100%|██████████| 150/150 [00:05<00:00, 27.23it/s]
val epoch: 5, loss: nan
train: epoch 6, iter:0, loss: nan, lr: 0.0009757729755661009
train: epoch 6, iter:20, loss: nan, lr: 0.0009757729755661009
train: epoch 6, iter:40, loss: nan, lr: 0.0009757729755661009
train: epoch 6, iter:60, loss: nan, lr: 0.0009757729755661009
train: epoch 6, iter:80, loss: nan, lr: 0.0009757729755661009
train: epoch 6, iter:100, loss: nan, lr: 0.0009757729755661009
train: epoch 6, iter:120, loss: nan, lr: 0.0009757729755661009
train: epoch 6, iter:140, loss: nan, lr: 0.0009757729755661009
100%|██████████| 150/150 [00:05<00:00, 25.76it/s]
val epoch: 6, loss: nan
train: epoch 7, iter:0, loss: nan, lr: 0.0009652393605146842
train: epoch 7, iter:20, loss: nan, lr: 0.0009652393605146842
train: epoch 7, iter:40, loss: nan, lr: 0.0009652393605146842
train: epoch 7, iter:60, loss: nan, lr: 0.0009652393605146842
train: epoch 7, iter:80, loss: nan, lr: 0.0009652393605146842
train: epoch 7, iter:100, loss: nan, lr: 0.0009652393605146842
train: epoch 7, iter:120, loss: nan, lr: 0.0009652393605146842
train: epoch 7, iter:140, loss: nan, lr: 0.0009652393605146842
100%|██████████| 150/150 [00:05<00:00, 26.47it/s]
val epoch: 7, loss: nan
train: epoch 8, iter:0, loss: nan, lr: 0.0009528893909706795
train: epoch 8, iter:20, loss: nan, lr: 0.0009528893909706795
train: epoch 8, iter:40, loss: nan, lr: 0.0009528893909706795
train: epoch 8, iter:60, loss: nan, lr: 0.0009528893909706795