MS-SSIM minimum crop size: 161px

[trougnouf]

I found that the MS-SSIM loss generates an error with crop sizes of less than 161 pixels.

Is that a limitation of MS-SSIM, a possibly sub-optimal crop size, a missing parameter, and/or an implementation specificity?

>>> import torch
>>> import piqa
>>> dim=160
>>> piqa.MS_SSIM()(torch.rand(1,3,dim,dim),torch.rand(1,3,dim,dim))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/lib/python3.9/site-packages/piqa/ssim.py", line 330, in forward
    l = ms_ssim(
  File "/usr/lib/python3.9/site-packages/piqa/ssim.py", line 172, in ms_ssim
    ss, cs = ssim(
  File "/usr/lib/python3.9/site-packages/piqa/ssim.py", line 95, in ssim
    mu_x = channel_convs(x, window)
  File "/usr/lib/python3.9/site-packages/piqa/utils/functional.py", line 74, in channel_convs
    x = channel_conv(x, k)
  File "/usr/lib/python3.9/site-packages/piqa/utils/functional.py", line 38, in channel_conv
    return F.conv1d(x, kernel, padding=padding, groups=x.size(1))
RuntimeError: Calculated padded input size per channel: (10 x 10). Kernel size: (11 x 1). Kernel size can't be greater than actual input size
>>> dim=161
>>> piqa.MS_SSIM()(torch.rand(1,3,dim,dim),torch.rand(1,3,dim,dim))
tensor(0.0547)

[francois-rozet]

Hello @trougnouf 👋

This is a limitation of the metric itself. During the processing, the image is recursively sub-sampled 4 times, meaning its size is reduced by a factor 2^4 = 16. But the convolution kernel size is 11×11 by default in SSIM, and as the error says: kernel size can't be greater than actual input size. In the original paper the metric is used on 768×512 images or similar.

A solution could be to add (virtual) padding during the convolutions (which I actually did in the past), but the metric would not be consistent with other implementations anymore. An alternative is to add a (boolean) parameter that would trigger that behavior.

[trougnouf]

Thank you for your response! I guess that means an ideal minimum size for the loss would be 11*16=176 ?

A virtual parameter-enabled padding could be useful :) I guess the window size is the 11x11 convolution, do you have any insight as to what may work better or more faithful to the original definition between padding or reduced window size?

[francois-rozet]

Thank you for your response! I guess that means an ideal minimum size for the loss would be 11*16=176 ?

Well, in fact, anything between 161 and 176 pixels will result in 11 pixels in the last scale. The reason is that if the number of pixels is odd, i.e. 2n + 1, the downscaled image will have n + 1 pixels, not n.

what may work better [...] between padding or reduced window size?

Reducing the window size is perfectly sensible, but it could change the properties of the metric, especially if the metric is used as an optimization criterion. On the other hand, adding padding overestimates the MS-SSIM because the "black borders" are necessarily "similar", although they do not impact very much the gradient direction because they are constant. Therefore, both could work, depending on the situation.

However, the MS-SSIM metric is designed to compare both the fine details locally (at full scale) AND the rough features (at low scale). But, if you have images close to 161x161, there likely is very few details and using MS-SSIM doesn't make much sense. For such sizes, I would not use a "multi-scale" metric.