Skip to content

Support pre-generated web thumbnails#147

Draft
po5 wants to merge 28 commits intomasterfrom
wip-storyboards
Draft

Support pre-generated web thumbnails#147
po5 wants to merge 28 commits intomasterfrom
wip-storyboards

Conversation

@po5
Copy link
Copy Markdown
Owner

@po5 po5 commented Apr 8, 2025

Supports YouTube videos and Twitch VODs. cf. #3, #111
No actual logic is specific to those sites, it's just that yt-dlp doesn't currently support anything else.
I may make PRs to yt-dlp later implementing "storyboards" for more websites.

It's good enough that I'd like others to try it and report issues.
Commits are a mess because I'm still experimenting a lot, they'll be squashed eventually.

I wonder if it may be possible to avoid spinning up a whole mpv process for each atlas, and instead use EDL or lavf:// to open multiple images and save them separately with an output template.

TODO:

  • Reorganize the code, remove useless stuff
  • Test with private videos (forward cookies setting)
  • Better feedback if yt-dlp isn't found
  • Mirror video filters to storyboard thumbnails (e.g. rotation, flipping, cropping)
  • Even quicker thumbnail prioritization (by reserving n slots to the cursor?)
  • Make it work nicely with the raw thumbfast-render API (even though there are currently no users of it)
  • Make sure we never crash when switching between files
  • Make sure we don't try to render thumbnails before they're written
  • Make regular network thumbnailing placeholder consistent with this? (only broadcast info once we have the first thumbnail's dimensions, or the other way around, broadcast estimated dimensions before we get back the storyboard data)
  • Make sure that absolutely everything is still fine with regular thumbnailing
  • Fix audio crackling when spawning fragment downloads (happens even with --ao=null --ao-null-untimed) likely an mpv bug

My primary test videos:

@po5 po5 marked this pull request as draft April 8, 2025 07:35
@Sneakpeakcss
Copy link
Copy Markdown

On Windows beside reverting 9deb073 i also had to remove --vo=null to get it working.

Using --vo=null ends with an error: thumbfast: storyboard download failed atlas: 2 status: 2. The subprocess terminates without generating any .bgra files.

[   0.248][v][demux] Detected file format: webp_pipe (libavformat)
[   0.248][v][cplayer] Opening done: https://i.ytimg.com/sb/HOpbPQGEH9k/storyboard3_L3/M6.jpg?sqp=-oaymwENSDfyq4qpAwVwAcABBqLzl_8DBgjQ5MWpBg==&sigh=rs$AOn4CLAmRjZeNEAFVmD_SSdYuAV5WVIAiQ
[   0.248][v][cplayer] Running hook: auto_profiles/on_preloaded
[   0.248][v][cplayer] Failed sending hook command auto_profiles/on_preloaded. Removing hook.
[   0.248][v][lavf] select track 0
[   0.248][i][cplayer] ● Image  --vid=1  (webp 972x540)
[   0.248][f][cplayer] Error opening/initializing the selected video_out (--vo) device.
[   0.248][v][lavf] deselect track 0
[   0.248][i][cplayer] Video: no video
[   0.248][f][cplayer] No video or audio streams selected.
[   0.248][d][cplayer] Terminating demuxers...
[   0.248][d][ffmpeg] AVIOContext: Statistics: 41056 bytes read, 0 seeks
[   0.248][d][cplayer] Done terminating demuxers.
[   0.248][v][cplayer] finished playback, no audio or video data played (reason 4)
[   0.248][f][encode] no data written to target file
[   0.248][i][cplayer] Exiting... (Interrupted by error)
[   0.248][d][positioning] Destroying client handle...
[   0.248][d][cplayer] Run command: del, flags=64, args=[name="user-data/mpv/console"]
[   0.248][d][select] Destroying client handle...
[   0.248][d][commands] Destroying client handle...
[   0.249][d][stats] Destroying client handle...
[   0.249][d][console] Destroying client handle...
[   0.251][d][SystemMediaTransportControls] Destroying client handle...

Also about Twitch vods, i wonder if this won't be a potential issue here.

@po5
Copy link
Copy Markdown
Owner Author

po5 commented Apr 9, 2025

Can reproduce, turns out in newer mpv versions (0.39+?) vo=null no longer works with the encoding mode? I removed it.
I'll look into the issue with 9deb073 next time I boot into Windows.

Also about Twitch vods, i wonder if mpv-player/mpv#14840 won't be a potential issue here.

This is also a newly introduced issue, and it's completely irrelevant to us. They'll fix it eventually, probably.

Love running into new issues after updating mpv, wonder what else it broke (:
Edit: Looks like you can no longer set an empty value for input-commands in an auto-profile, mpv will straight up refuse to run. Fun.

Comment on lines +1149 to +1152
{input = path_ytdl, patterns = youtube_patterns_free, condition = function(input) return true end},
{input = path, patterns = youtube_patterns, condition = function(input) return path_has_prefix end},
{input = path, patterns = twitch_patterns, condition = function(input) return path_has_prefix and string.match(input, twitch_base) end},
{input = referer_ytdl, patterns = youtube_patterns_free, condition = function(input) return input ~= "" end},
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These *_ytdl variables just seem to be longer versions of the stripped ones, so why check against them?

Suggested change
{input = path_ytdl, patterns = youtube_patterns_free, condition = function(input) return true end},
{input = path, patterns = youtube_patterns, condition = function(input) return path_has_prefix end},
{input = path, patterns = twitch_patterns, condition = function(input) return path_has_prefix and string.match(input, twitch_base) end},
{input = referer_ytdl, patterns = youtube_patterns_free, condition = function(input) return input ~= "" end},
{input = path, patterns = youtube_patterns_free, condition = function(input) return true end},
{input = path, patterns = youtube_patterns, condition = function(input) return path_has_prefix end},
{input = path, patterns = twitch_patterns, condition = function(input) return path_has_prefix and string.match(input, twitch_base) end},
{input = referer, patterns = youtube_patterns_free, condition = function(input) return input ~= "" end},

Copy link
Copy Markdown
Owner Author

@po5 po5 Apr 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The freeform patterns in youtube_patterns_free match strings with anything at the start. Because the user may have prefixed the path with ytdl:// to force ytdl on their end, which is an mpv invention that the yt-dlp command line doesn't understand, we have to make sure we're matching on the version on the path that we've stripped the ytdl:// prefix from.

For clarification, the ytdl_prefix pattern replace (^ytdl://(.+) -> %1) is used to get the path without this prefix while falling back to the full string if it didn't match.

Edit: Actually thinking about it some more I'm not sure I understand my own code anymore, why do I need to perform these matches on the version of the path that wasn't stripped from the http prefix? I'll look into it.

Comment on lines +1298 to +1312
-- Storyboard upscaling factor
local scale = properties["display-hidpi-scale"] or 1
if sb.width / sb.height > options.max_width / options.max_height then
real_w = options.max_width * scale
real_h = math.floor(sb.height / sb.width * real_w)
real_w = math.floor(real_w)
else
real_h = options.max_height * scale
real_w = math.floor(sb.width / sb.height * real_h)
real_h = math.floor(real_h)
end
local storyboard_scale = {w=real_w/sb.width, h=real_h/sb.height}
local thumbnail_size = {w=real_w, h=real_h}
effective_w, effective_h = real_w, real_h
info(real_w, real_h)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scaling the storyboards to not fit the max should probably be an option, since it may not look good at some scaling ratios.

thumbfast.lua Outdated
Comment on lines +1032 to +1038
for line = 0, thumbnail_size.h - 1 do
atlas:seek("set", 4 * x_start + (y_start + line) * stride)
local data = atlas:read(thumbnail_size.w * 4)
if data ~= nil then
thumb_file:write(data)
end
end
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems like an awful lot of writes. Can't you just read the lines one by one, and keep appending them to a buffer, then write the complete buffer all at once?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

io.open very likely defaults to buffered io. Like most standard libraries in most languages.

"^"..anycase("player%.twitch%.tv/.*[?&]video=v?")..twitch_id,
}

local function storyboard_supported_url(path, referer)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also wonder if there even is any point having checks here as to whether the site supports yt-dlp, instead of just always attempting to fetch storyboards, and only continuing if they are returned.

Copy link
Copy Markdown
Owner Author

@po5 po5 Apr 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would slow down the initial load of any non-storyboard network thumbnails, and cause extra requests because yt-dlp still has to fetch the list of all formats. It's not smart enough to lazily fetch format info based on --format and what's supported, which was a surprise to me. (and somewhat a blessing because it means that the existing json returned by ytdl_hook in newer mpv versions will always contain storyboard info, removing the need to spawn yt-dlp for those mpv versions)
I agree that this code is overengineered, but it's a port of the patterns from yt-dlp itself and should cover all cases.

messy, and doesn't auto-update in all cases yet
crop handling is still missing
it's progress (:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants