Full screen bitmap smpte-tt subtitle not rendered correctly

[szaboa]

[REQUIRED] Issue description

Support for image based SMPTE-TT subtitles was implemented in #1583, however it's not working quite well for "full screen" subtitles as it renders these out of the screen and with wrong height. By full screen I mean when the subtitle should take size of the surface. This is the case when tt tts:extent = region tts:extent in case of pixel based regions.

[REQUIRED] Reproduction steps

It can be reproduced with the ExoPlayer demo app, just needs a stream with image based SMPTE-TT subtitle track where the Base64 encoded bitmap is "full screen".

[REQUIRED] Link to test content

Email

[REQUIRED] A full bug report captured from the device

No logs required.

[REQUIRED] Version of ExoPlayer being used

ExoPlayer 2.9.3 - 2.9.6

[REQUIRED] Device(s) and version(s) of Android being used

It can be reproduced every time and not device dependent.
In my tests, I've used Pixel 1.

[szaboa]

Example:
The manifest contains in tt tag let's say tts:extent="720px 576px" and there are regions like <region xml:id="region_0" tts:extent="720px 576px" tts:origin="0px 0px"/>.

Two problems causing this issue:

1. Incorrect height calculation
In the example above the bitmap will take the width of the parent correctly, but the height is scaled up to keep the bitmap's aspect ration.

Height calculation in com.google.android.exoplayer2.ui.SubtitlePainter#setupBitmapLayout:

int height = cueBitmapHeight != Cue.DIMEN_UNSET ? Math.round(parentHeight * cueBitmapHeight)
        : Math.round(width * ((float) cueBitmap.getHeight() / cueBitmap.getWidth()));

The cueBitmapHeight will be Cue.DIMEN_UNSET because that's what we pass in com.google.android.exoplayer2.text.ttml.TtmlNode#getCues (line 238).

This logic is OK for non-full screen subtitles, but for a full screen one it doesn't force the bitmap to fit the parent size. This results incorrect positioning.

Inside com.google.android.exoplayer2.text.ttml.TtmlDecoder#parseRegionAttributes the width and height calculations are OK and both of them are 1 (which means 100% width and 100% height). But the height is not passed to the TtmlRegion.

Proposed solution:

Pass the height from parseRegionAttributes to TtmlRegion and inside com.google.android.exoplayer2.text.ttml.TtmlNode#getCues pass this height to Cue (line 238) or pass it only if both of the region's width and height is 1, this would only affect the full screen bitmaps. In this way the bitmap's height would be set to the parent height in com.google.android.exoplayer2.ui.SubtitlePainter#setupBitmapLayout.

2. Incorrect position anchor

Inside com.google.android.exoplayer2.text.ttml.TtmlNode#getCues we pass Cue.ANCHOR_TYPE_MIDDLE as a default position anchor for image based subtitles. This will render a full screen image based subtitle offscreen (top half of the bitmap is outside of the view).

Again, because the y calculation in com.google.android.exoplayer2.ui.SubtitlePainter#setupBitmapLayout:

int y = Math.round(cuePositionAnchor == Cue.ANCHOR_TYPE_END ? (anchorY - height)
        : cuePositionAnchor == Cue.ANCHOR_TYPE_MIDDLE ? (anchorY - (height / 2)) : anchorY);

Proposed solution:

Change the default position anchor to Cue.ANCHOR_TYPE_START or do not set anything, this way it will fall back to anchor start.

@ojw28
What do you think? Once we agree on a solution, I'll fix this.

[szaboa]

Can someone have a look on this please?

[AquilesCanta]

@szaboa, have you shared a stream with us? We will need sample media before we look into this. If you think the solution is straightforward, you might want to send a pull request directly: But most likely, we will still need a sample stream.

[szaboa]

I will try to get a sample stream as soon as I can.

Actually, I was who implemented the support for image based subtitles (smpte-tt) and I didn't pay enough attention to this use case. That's why I proposed the fix.

[szaboa]

Sorry, accidentally closed the issue, miss click :)

[szaboa]

@AquilesCanta, sorry for the late response :)

Updated ticket's description with sample streams. During testing make sure you pick the second Norwegian subtitle track and note that you might need to wait a little for the subs to show up (probably not synced correctly in this test content).

I've also created a pull request with the proposed solution (#5711).

---

And to make sure nothing changed for regular (not full screen) image based subtitles, I've also tested with this sample stream (which I used as a reference when I implemented support for image based subs at first place): https://livesim.dashif.org/dash/vod/testpic_2s/img_subs.mpd

---

@ojw28, you might want to have a look too on the proposed solution, as we have discussed this feature quite a lot at the beginning :)

[szaboa]

Any update on this? :)