Class Error Problem

[Hatins]

Hi, @orrzohar,
I'm facing the same issue as described in input #24. I followed the installation instructions and used four TITAN (12G) GPUs to run the code. I set the batch size to 2 and reduced the learning rate from 2e-4 to 8e-5, as well as the lr_backbone from 2e-5 to 8e-6.
However, even after 2 epochs, the class_error metrics remain quite high, ranging from 80 to 100. Is this normal? If not, what could be causing this problem?

This value has been fluctuating, unstable

[orrzohar]

Hi @Hatins,
have you tried using the same hyper-parameters as in OW-DETR?
They used a 4-GPU system with a batch size 2, and our class predictions follow the exact same inference scheme'
Best,
Orr

[Hatins]

Hi, @orrzohar
thank you for your reply, I will try to experiment with the parameters of OW-DETR as soon as possible.

[orrzohar]

Hi @Hatins,
As you closed this issue I assume that the class error issue has resolved.

If that is the case, could you please write the hyper parameters here for future reference?

Best,
Orr

[Hatins]

Hi @orrzohar
I haven't actually tried it yet as I've been busy with other things recently. I'm closing this issue now because I don't want to disrupt the record that you've resolved all the issues on GitHub (^_^). However, once I find some time and successfully solve the problem, I will provide an explanation there.

[Sidd1609]

@Hatins @orrzohar I am also having the same issue but I am only using 1 GPU (I do not have a slurm setup but do have a single system with 2GPUs)

I configured the hyperparameters as lr: 8e-6 and lr-backbone: 8e-8
And I get results like
person has 0 predictions.
bicycle has 0 predictions.
car has 1165400 predictions

How to resolve this issue ?

[orrzohar]

Hi @Hatins,
That is much appreciated, haha! Yeah I really hope PROB can be as reproducible as possible and hopefully an easy baseline for future methods due to its simplicity & it having very few hyper-parameters :). When you get back to this, please feel free to re-open the issue so I can help troubleshoot running PROB with a batch-size of 2.

@Sidd1609, could you try the OW-DETR hyper parameters if you use the same batch-size as them? namely:

    parser.add_argument('--lr', default=2e-4, type=float)
    parser.add_argument('--lr_backbone', default=2e-5, type=float)
    parser.add_argument('--weight_decay', default=1e-4, type=float)
    parser.add_argument('--lr_drop', default=40, type=int)
    parser.add_argument('--clip_max_norm', default=0.1, type=float,
                        help='gradient clipping max norm')
    parser.add_argument('--mask_loss_coef', default=1, type=float)
    parser.add_argument('--dice_loss_coef', default=1, type=float)
    parser.add_argument('--cls_loss_coef', default=2, type=float)
    
    parser.add_argument('--bbox_loss_coef', default=5, type=float)
    parser.add_argument('--giou_loss_coef', default=2, type=float)
    parser.add_argument('--focal_alpha', default=0.25, type=float)

OW-DETR and PROB are both exactly the DDETR model for the known classes during training (notice that the revised inference scheme is used in test-time/evaluation only, while the class_error is calculated during training and objectness plays no role here). The only other option is that you have an issue with your dataset configuration, but that seems rather unlikely.

Best,
Orr

[Hatins]

Hi @orrzohar
Thanks for your explanation and conscientiousness! I found the hyper-parameters of OW-DETR are set as below:

    parser = argparse.ArgumentParser('Deformable DETR Detector', add_help=False)
    parser.add_argument('--lr', default=2e-4, type=float)
    parser.add_argument('--lr_backbone_names', default=["backbone.0"], type=str, nargs='+')
    parser.add_argument('--lr_backbone', default=2e-5, type=float)
    parser.add_argument('--lr_linear_proj_names', default=['reference_points', 'sampling_offsets'], type=str, nargs='+')
    parser.add_argument('--lr_linear_proj_mult', default=0.1, type=float)
    parser.add_argument('--batch_size', default=2, type=int)
    parser.add_argument('--weight_decay', default=1e-4, type=float)
    parser.add_argument('--epochs', default=51, type=int)
    parser.add_argument('--lr_drop', default=40, type=int)
    parser.add_argument('--lr_drop_epochs', default=None, type=int, nargs='+')
    parser.add_argument('--clip_max_norm', default=0.1, type=float,
                        help='gradient clipping max norm')

So I would like to use those parameters and test it tonight and return the results as soon as possible!
By the way, it will be nice if there is a formula to compute all the parameters according to the number of GPUs and batch size (^_^)

[Sidd1609]

Hi @orrzohar thank you so much for your explanation, will definitely test it out with these hyper parameters of OW-DETR. I did notice the similarities with OW-DETR, and am little worried I might not be able to run with a batch size more than 2 (due to constraint of single GPU).

[Hatins]

I'm sorry, I can't do the experiment this time due to the machine being occupied by the others. But from the results (4 epochs) I have got, I found the class error still fluctuates from 50 to 95. Is that normal? When you train, is the value decrease smoothly and don't fluctuate?

[orrzohar]

Hi @Hatins,
Sorry for taking a bit to reply (I was OOO).

I am not sure as I never trained PROB with a system like yours. I can share the evaluation curves if that helps, and then you could tell if the mAP/U-Recall have similar trends to mine. They may be different due to the different batch_size, but hopefully should have a similar trend.

LMK how this does / if the class error issue is resolved - I'll update the codebase accordingly to support systems such as yours as this seems to be a common machine.

Best,
Orr

[orrzohar]

Hi @Sidd1609,

That is fine - OW-DETR was trained with a batch size of 2 so this should hopefully work.

Best,
Orr

[Hatins]

Hi @orrzohar
No problem, This isn't a very question. I found a machine with four Nvidia GeForce RTX 3090 GPUs and ran it for a while. It seems that I obtained relatively normal results after 13 epochs, as shown in the image below. Previously, when I used a GPU with 12GB of VRAM, the trend seemed to be similar to the graph below. However, due to some strange reasons, the server encountered an error (it kept failing at the fourth epoch), but it shouldn't be an issue with your code.
the result after 13 epochs (come from wandb):

the error of the machine with 12G VRAM:

I will update my results after I get the epochs to finish.
In the end, I wonder if the batch size is relevant to the number of GPUs ? For example, if I set the lr = 0.0002 when I use 4 GPUs with 2 batch size, what number should I set when using 8 GPUs with 2 batch size?