avformat: validate dovi config in muxers

[gnattu]

FFmpeg tends to copy any available side packets from the input streams into the muxed output without considering their validity. This behavior is problematic for Dolby Vision configuration records, as invalid records might crash the player. I haven't found a better place to perform such validations with the current Dolby Vision handling process, so I had to place the validation code into each muxers.

This PR checks whether the Dolby Vision record is valid, meaning it is spec-compliant, so the player will likely handle it without issues in movenc, mpegtsenc, and matroskaenc. It first verifies if the codec matches the profile, then checks if the base layer (BL) is encoded as required by the compatibility ID. If the compatibility ID is 0, it also checks if the codec tag is explicitly set to the dovi fourcc, as these videos will not have backward compatibility.

The check in mpegtsenc is slightly stripped down as we are only passing HEVC with BL compat in that muxer.

Changes

Issues

Fixes #483

[nyanmisaka]

Is it really necessary to check color_range? Many Profile 5 videos are not correctly labeled. Doing so will prevent them from being recognized by ffprobe -show_streams and change the existing behavior.

[gnattu]

The intention for this is to not creating tv range profile 5 videos and the muxer will not affect ffprobe. The only problem is that this would prevent those buggy profile 5 videos being remuxed. Maybe we need to remove this check for Profile 5 videos as DoVi players tends to ignore what the container says.

[nyanmisaka]

The intention for this is to not creating tv range profile 5 videos and the muxer will not affect ffprobe. The only problem is that this would prevent those buggy profile 5 videos being remuxed. Maybe we need to remove this check for Profile 5 videos as DoVi players tends to ignore what the container says.

That's my concern. I haven't tested it yet, but the existing Profile 5 videos that are incorrectly tagged as TV range may not be streamed correctly over fMP4.

[nyanmisaka]

I suggest putting these validations in the public area like libavformat/dovi_isom.{c,h} to reuse them in multiple muxers. Then we can merge these two patches into one debian/patches/0031-pass-dovi-sidedata-to-muxers.patch.

my draft dovi-validation.patch

[gnattu]

movenc is slightly different as the code tag is in its own track instead of the par, so the interface needs to be modified slightly.

[nyanmisaka]

movenc is slightly different as the code tag is in its own track instead of the par, so the interface needs to be modified slightly.

+int ff_isom_validate_dovi_config(const AVDOVIDecoderConfigurationRecord *dovi,
+                                 AVCodecParameters *codec_par, int codec_tag);

I saw that. We can use a separate parameter to handle codec_tag. The upstream will have to deal with this one day.