Compute error for `PSNR` and `SSIM`

[leftthomas]

Convert RGB image to YCbCr and take luminance should be Y = 0.256789 R + 0.504129 G +0.097906 B, not

piq/piq/psnr.py

Line 39 in e8f6513

rgb_to_grey = torch.tensor([0.299, 0.587, 0.114]).view(1, -1, 1, 1).to(x)

As for SSIM, to make sure the results can be same with Matlab Code, it must use DoubleTensor, and these three lines should change:

piq/piq/ssim.py

Line 55 in e8f6513

x = x.type(torch.float32)

piq/piq/ssim.py

Line 56 in e8f6513

y = y.type(torch.float32)

piq/piq/functional/filters.py

Line 35 in e8f6513

coords = torch.arange(kernel_size).to(dtype=torch.float32)

Please refer to my implement of SSIM and PSNR.

[jzakirov]

Hi @leftthomas thanks for raising the issue.
I can't yet agree with your point on PSNR conversion formula. Can you please provide a reference that mentions Y = 0.256789 R + 0.504129 G +0.097906 B values? Wikipedia pages on both YUV and YCbCr mention values we currently use with reference to ITU-R BT.601 standard.

With regard to DoubleTensor — how much is the absolute difference between 2 methods using single and double precisions?
BTW link to your implementation shows 404 error.

[leftthomas]

@zakajd I have make the repo of my implementation public, and you could refer to.

With regard to DoubleTensor — I have tested with Rain1400 in terms of SSIM, which is a Derain dataset, the absolute difference between 2 methods using single and double precisions is 0.0003, this value can not be ignored for Derain task, and I think you could refactor the API as like this:
coords = torch.arange(kernel_size, dtype=input.dtype, device=input.device)
this could compatible with both FloatTensor and DoubleTensor, according to input data.

About Y channel compute for YCbCr, I recommend you to refer the part of ITU-R BT.601 and Matlab Doc or this link

[leftthomas]

@zakajd Can you tell me the bug is fixed or not? And about the compute of YCbCr.

[jzakirov]

@leftthomas PR #292 adds dtype propagation though computation. You can fork it and use in your project or wait until it's merged into master (this will take up to 1 week).

Computation of Y channel in PSNR won't be changed due to insufficient proofs. Neither PIL (docs), neither OpenCV (docs) use the formula you provided and I assume those guys know what they are doing.

[leftthomas]

@zakajd about the Y channel compute, Python code is not same with Matlab, and some tasks use the Matlab code to obtain the metrics, and I think maybe you could add an args to let user to choose which implement to use, such as convert_to_y='python' or convert_to_y='matlab' , I suggest you to refer the implement of Matlab:
https://github.com/victorca25/traiNNer/blob/12d006fd44ed304e4178839c53b1f3d95ca25dcb/metrics/calculate_PSNR_SSIM.py#L122
and
https://github.com/victorca25/traiNNer/blob/master/metrics/calculate_PSNR_SSIM.m

[leftthomas]

@zakajd , This is another implement about the Y channel compute
https://github.com/chaofengc/IQA-PyTorch/blob/1f1ba21e73f6039bacdcd6e72147193ac9cae4ef/pyiqa/utils/color_util.py#L49

[denproc]

@leftthomas thank you for your feedback regarding PSNR and SSIM performance.
PSNR compares grayscale images. In order to get a grayscale image, we use conversion proposed in PIL (docs) and OpenCV (docs). This conversion is identical to Y channel of RGB2YIQ transformation. Sadly, previous comment in code was misleading. It will be fixed to avoid misunderstanding in future.

Following your quote, PSNR implementation proposed in IQA-PyTorch uses RBG2YIQ as a default colour conversion, which matches our approach.
Any additional colour conversions other than standard grayscale could be added as preprocessing step to your pipeline.

UPD: In addition, MATLAB version rgb2gray conversion function matches our approach as well.

[leftthomas]

@denproc, the value used in PSNR implementation proposed in IQA-PyTorch is same as Matlab: Y = 0.256789 R + 0.504129 G +0.097906 B (after divided by 255), but in this repo is rgb_to_grey = torch.tensor([0.299, 0.587, 0.114]).view(1, -1, 1, 1).to(x), PSNR should be computed on Y channel of YCbCr, not YIQ

[jzakirov]

PSNR should be computed on Y channel of YCbCr, not YIQ

Any source for such claim?
This metric is ~30 years old and there is no single reference implementation that should be matched. For RGB images it can be computed on all channels independently and then averaged, or use Y channel of YIQ, or Y channel of YCbCr, or Hue channel of HSL. All this is just a design choice that authors make.
We made ours to compute on Y channel of YIQ.

You can fork and modify the script to suit your specific needs.