[Reimplementation] The FCOS detector cannot be reproduced. The mAP is much lower than the model zoo.

[dmhz]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

sys.platform: linux
Python: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 3090
CUDA_HOME: /opt/cuda
NVCC: Cuda compilation tools, release 11.8, V11.8.89
GCC: gcc (GCC) 12.2.0
PyTorch: 1.8.2+cu111
PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.9.2+cu111
OpenCV: 4.6.0
MMCV: 1.6.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMRotate: 0.3.3+4bbef14

Reproduces the problem - code sample

python tools/train.py configs/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90.py
python tools/train.py configs/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90.py

Reproduces the problem - command or script

python tools/test.py configs/rotated_fcos/rotated_fcos_r50_fpn_1x_dota_le90.py work_dirs/rotated_fcos_r50_fpn_1x_dota_le90/latest.pth --format-only --eval-options submission_dir=task1

python tools/test.py configs/rotated_faster_rcnn/rotated_faster_rcnn_r50_fpn_1x_dota_le90.py work_dirs/rotated_faster_rcnn_r50_fpn_1x_dota_le90/latest.pth --format-only --eval-options submission_dir=task1

Reproduces the problem - error message

The detection accuracy of FCOS on the DOTA dataset is much lower than the model provided by model zoo.

Additional information

Thank the authors for providing such an excellent rotating target detection framework.

I downloaded the latest code of mmrotate and replicated the FCOS algorithm on the DOTA dataset with RTX3090.
I used the same data split method, consistent configuration files, and failed to get a good mAP after running it many times.

The following are the results submitted on the dota test server twice:
AP50: 0.7032243822038402
AP75: 0.38472094328866135
mAP: 0.3986036947128665
AP50: 0.7047541260814543
AP75: 0.3857469785305509
mAP: 0.39624439296748515

I also verified the algorithm in v0.3.1, and also got a low map:
AP50: 0.7016260775289945
AP75: 0.3813169279633645
mAP: 0.39712732623330294

In addition, I run the configuration file rotated_ faster_ rcnn_ r50_ fpn_ 1x_ dota_ Le90.py can obtain the same detection accuracy as the officially published model:
AP50: 0.7362327956605763
AP75: 0.40734393918484924
mAP: 0.41351331849739337

[yangxue0827]

For DOTA dataset, an accuracy gap of about 0.5% seems normal. It is recommended that you follow the results you have obtained. Our results are only for your reference.

[dmhz]

Thanks for the reply of the author. In my experiments, the average accuracy of FCOS is 0.8-1 (70.2-70.4) lower than Model Zoo (71.28). Is there any other additional tricks in the model provided by Model Zoo?

I want to know whether the precision of mmrotate is related to the version of mmcv or mmdet?

[yangxue0827]

I will give you feedback after I train again.

[dmhz]

okay, thank you.

[yangxue0827]

Any suggestions? @liuyanyi

[liuyanyi]

The result of FCOS model is unstable, i still no idea why it behaive like this, maybe norm_on_bbox and center_sampling affect the result, but that need more experiment, looking for your feedback too.

In our 1.x refactor(#543), i got a lower result too. Reported ap50 is result in 0.x. The result is not stable and may have 1% ap difference.

model	Reported ap50	map	ap50	ap75
Rotated FCOS	71.28	41.10	70.87	41.21
Rotated FCOS kld	71.89	40.37	71.55	38.92
Rotated FCOS hbb csl	71.76	40.22	70.79	39.26

[dmhz]

Thanks for your reply. I'll test dev1.x and give you feedback.

If the model provided in the initial version (v0.3.1) is the better one after many trainings?
Can you tell me the FCOS (v0.3.x) approximate range of mAP during many experiments for reference.

@liuyanyi

[liuyanyi]

The logic in the fcos of the initial version and newer version should be the same and no major changes have been made. So I think version wont affect training.

While develop fcos in 0.3, i remember 71.28 is the best result i got.

Here is my exps during refactor 1.x, there was a 1% difference in the ap of the three experiments.

rotated_fcos

run	ap50	ap75	map
1	71.63	42.34	41.85
2	70.72	41.06	40.95
3	71.32	41.26	41.25

rotated_fcos_csl

run	ap50	ap75	map	w
1	70.84	39.51	40.15	w0.2
2	71.15	39.64	40.38	w0.2
3	71.26	40.03	40.39	w0.2
4	70.94	40.51	40.78	w0.6

[yangxue0827]

My reimplementation:

This is your evaluation result for task 1 (VOC metrics):

mAP: 0.7121946689619715
ap of each class: plane:0.8906049957433385, baseball-diamond:0.7352599576125792, bridge:0.4853550551625228, ground-track-field:0.6272555673055425, small-vehicle:0.7847658521031989, large-vehicle:0.7335348612561778, ship:0.8618551298188438, tennis-court:0.9090909090909093, basketball-court:0.8132608032925924, storage-tank:0.8407974408264521, soccer-ball-field:0.5744272576514078, roundabout:0.614308353447493, harbor:0.6327183850671182, swimming-pool:0.6676415731071267, helicopter:0.5120438929442703
COCO style result:

AP50: 0.7121946689619715
AP75: 0.38454486600750865
mAP: 0.3960396939468972
The submitted information is :

Description: rotated_fcos_r50_fpn_1x_dota_le90

Environment:

2022-11-01 18:43:07,213 - mmrotate - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.3, V11.3.109
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.9.0+cu111
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.3-Product Build 20210617 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.10.0+cu111
OpenCV: 4.5.5
MMCV: 1.6.0
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: 11.3
MMRotate: 0.2.0+
------------------------------------------------------------

Log:
20221101_184300.log

[yangxue0827]

My codebase: https://github.com/yangxue0827/h2rbox-mmrotate

[dmhz]

The logic in the fcos of the initial version and newer version should be the same and no major changes have been made. So I think version wont affect training.

While develop fcos in 0.3, i remember 71.28 is the best result i got.

Here is my exps during refactor 1.x, there was a 1% difference in the ap of the three experiments.

rotated_fcos

run ap50 ap75 map
1 71.63 42.34 41.85
2 70.72 41.06 40.95
Here is my exps during refactor 1.x, there was a 1% difference in the ap of the three experiments.

rotated_fcos

run ap50 ap75 map
1 71.63 42.34 41.85
2 70.72 41.06 40.95
3 71.32 41.26 41.25
rotated_fcos_cs
3 71.32 41.26 41.25
rotated_fcos_csl

run ap50 ap75 map  w
1 70.84 39.51 40.15 w0.2
2 71.15 39.64 40.38 w0.2
3 71.26 40.03 40.39 w0.2
4 70.94 40.51 40.78 w0.6

I tested the mmrotate development version, which can achieve higher results. The following are the test results I ran twice:
AP50: 0.7150217002058156
AP75: 0.42193368883257576
mAP: 0.41591037131929853

AP50: 0.7120718620827703
AP75: 0.4098099929434117
mAP: 0.4078548301075865

The software version I tested is as follows:
mmcv==2.0.0rc2
mmdet==3.0.0rc2
mmengine==0.2.0
mmrotate==1.0.0rc0

However, the accuracy of mmrotate (v0.3.x) is not high, which may be due to a specific software version or other reasons. I have no ability to find the differences.

[dmhz]

https://github.com/yangxue0827/h2rbox-mmrotate

rotated_fcos_r50_fpn_1x_dota_le90.py

mmcv-full==1.6.0
mmdet==2.25.3
mmrotate==0.2.0

This is your evaluation result for task 1 (VOC metrics):

mAP: 0.7080817910895376
ap of each class: plane:0.891049340837604, baseball-diamond:0.7265259086435176, bridge:0.48298530261650086, ground-track-field:0.5933720824738311, small-vehicle:0.790293514900079, large-vehicle:0.7293038272089639, ship:0.8593364303461166, tennis-court:0.9089545113551116, basketball-court:0.8169677011977725, storage-tank:0.842900847286389, soccer-ball-field:0.5647016098941234, roundabout:0.6549169070790201, harbor:0.6268073214237, swimming-pool:0.6857934950110997, helicopter:0.4473180660692365
COCO style result:

AP50: 0.7080817910895376
AP75: 0.3762085109574612
mAP: 0.39150821002761227

Seems to be a reasonable result.

20221103_003210.log

[hu-wei-carrzy]

Excuse me, may I ask, i am novice.I want to reproduce oriented RCNN，i used ss_trainval.json,ss_test.json cut picture，and used oriented_rcnn_r50_fpn_1x_dota_le90-6d2b2ce0.pth，but i can get the good result，if you have time to see my question ,i will tank you very much，i also ask at https://github.com/open-mmlab/mmrotate/issues/594，my E-mail is 2780222819@qq.com,