Imagery Component ValueError breaks the execution

[Schwartz-Zha]

Following the instruction of setting up this environment, and execute the script in README.md, I got the following error that breaks the execution.

(venv_dgmeval) zhajiajun@consistency-models-2-l786n-2659292-worker-0:/data/dgm-eval$ python -m dgm_eval CIFAR10:train CIFAR10:t
est \                                                                                                                          
                                        --model dinov2 \                                                                       
                                        --metrics fd kd prdc \                                                                 
                                        --device cuda \                                                                        
                                        --batch_size 256 \                                                                     
                                        --nsample 512                                                                          
                                                                                                                               
Loading Model                                                                                                                  
Using cache found in /home/zhajiajun/.cache/torch/hub/facebookresearch_dinov2_main                                             
/home/zhajiajun/.cache/torch/hub/facebookresearch_dinov2_main/dinov2/layers/swiglu_ffn.py:43: UserWarning: xFormers is availabl
e (SwiGLU)                                                                                                                     
  warnings.warn("xFormers is available (SwiGLU)")                                                                              
/home/zhajiajun/.cache/torch/hub/facebookresearch_dinov2_main/dinov2/layers/attention.py:27: UserWarning: xFormers is available
 (Attention)                                                                                                                   
  warnings.warn("xFormers is available (Attention)")                                                                           
/home/zhajiajun/.cache/torch/hub/facebookresearch_dinov2_main/dinov2/layers/block.py:33: UserWarning: xFormers is available (Bl
ock)            
  warnings.warn("xFormers is available (Block)")                                                                               
Getting DataLoader for path: CIFAR10:train                                                                                     
                                                                                                                               
Files already downloaded and verified                                                                                          
Loading saved representations from: experiments/                                                                               
                                                                                                                               
Loading from: experiments/reps_CIFAR10_dinov2_nimage-512_train
No saved representations found: experiments/

Calculating Representations

100%|████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:22<00:00, 11.22s/it]
Getting DataLoader for path: CIFAR10:test

Files already downloaded and verified
Loading saved representations from: experiments/
Loading from: experiments/reps_CIFAR10_dinov2_nimage-512_test
No saved representations found: experiments/

Calculating Representations

100%|████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.99s/it]
Computing scores between reference dataset and CIFAR10:test

Computing FD 

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/dgm-eval/dgm_eval/__main__.py", line 367, in <module>
    main()
  File "/data/dgm-eval/dgm_eval/__main__.py", line 343, in main 
    scores_i, vendi_scores_i = compute_scores(args, reps, repsi_test, dataloaderi.labels)

  File "/data/dgm-eval/dgm_eval/__main__.py", line 148, in compute_scores
    scores['fd'] = compute_FD_with_reps(*reps)
  File "/data/dgm-eval/dgm_eval/metrics/fd.py", line 76, in compute_FD_with_reps
    return compute_FD_with_stats(mu1, mu2, sigma1, sigma2, eps=eps)
  File "/data/dgm-eval/dgm_eval/metrics/fd.py", line 53, in compute_FD_with_stats
    raise ValueError('Imaginary component {}'.format(m))
ValueError: Imaginary component 0.6804371534058078

And the error comes from this line.

  # Numerical error might give slight imaginary component
  if np.iscomplexobj(covmean):
      if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):
          m = np.max(np.abs(covmean.imag))
          raise ValueError('Imaginary component {}'.format(m))
      covmean = covmean.real

If the imaginary component is not avoidable, why throw an error here and stops execution instead of just a small warning?

[Schwartz-Zha]

And if I surpress this ValueError, I found my result deviates from the provided one by a lot.

Num real: 512 Num fake: 512
fd: -5298.96106

kd_value: 1.61775

kd_variance: 0.00388

precision: 0.83594

recall: 0.86719

density: 0.81016

coverage: 0.95703

Saving scores

[WonwoongCho]

I have two servers with different envs (one is from EC2 and the other one is kinda local server), and the same issue happened from the local server. I could use the extracted representations obtained from the local server in EC2 though. What I can confirm is that Dino v2 works well in both servers. Some parts after getting the representations could be the cause of the problem.