trying to only use next endpoint for the major data by unixfox · Pull Request #5003 · iv-org/invidious

unixfox · 2024-10-13T17:47:01Z

Description

This PR aims to avoid throwing a raw error and instead display all the data that we can extract from the /next endpoint but without displaying the "player" (video.js) element.

This is one big step towards #4985. Invidious will have to be able to display the watch page even when YouTube is blocking the IP.

This will give the same result as on www.youtube.com:

This way the user can still view the title of the video, accessing the channel page, viewing the comments and much more which is not yet blocked.

What changed?

The watch page is displayed properly but without the player if an error is found.
All the important data like title, viewcount is fetched from /next endpoint is unable from the /player endpoint.
The error message now throws the reason AND the subreason. Makes it clearer to the user what's really happening.

TODO

Integrate the embed page with the changes
Display the subreason on the watch page when there is an error
Fix description
Add mocks for a video without videoDetails

Try these videos ID for testing - on both a blocked IP and a not blocked one

98TyXNPdMRo (private video)

7KSfDyainYU (georestricted video)

KIuDBxqz__k (familySafe no)

yqaS07znJWY (unlisted video)

aaaaaaaaaaaa (notfound video)

Fixes

ic-scm · 2024-10-17T08:27:57Z

Thank you very much for this. Looking at instance yewtu.be which uses invidious-custom, it seems to be working just as expected, exactly as described in my linked issue, and with a 100% success rate.

Thank you for this great work - this is a huge step towards making Invidious convenient to use again. For me personally, it solves the entire problem.

The only thing I can potentially suggest is for the video thumbnail to be loaded and displayed on the page, separately from the broken video player. But that would be just a nice-to-have feature, and I'm guessing it may not be trivial to implement, when the official Youtube website just shows a black error screen, with the thumbnail nowhere to be seen.

syeopite · 2024-10-18T19:11:07Z

-  if reason = info["reason"]?
+  if info["reason"]? && info["subreason"]?
+    reason = info["reason"].as_s
+    puts info


Please remove the debug puts here

syeopite · 2024-10-18T19:33:39Z

-  next_steps = error_redirect_helper(env)
+  error_message = <<-END_HTML
+    <div class="error_message">
+      <h2>#{translate(locale, "error_processing_data_youtube")}</h2>
+      <p>#{translate(locale, message)}</p>
+      #{error_redirect_helper(env)}
+    </div>
+  END_HTML


Your changes here will show the Error while processing the data sent by YouTube message on any InfoException while also removing the next steps message on all InfoExceptions

Example:

And InfoExceptions are raised in places unrelated to YouTube as well such as when a user inserts the wrong password.

Related: #4651

SamantazFox · 2024-10-19T14:41:32Z


 <div class="h-box">
    <%= error_message %>
-    <%= next_steps %>


The error page is used for everything and not just the watch page, so it must contain at least the "refresh" and "switch instance" buttons. Alternatively, we could split the error page logic for the watch page from the rest.

SamantazFox · 2024-10-19T14:45:02Z

+            <%= translate(locale, "error_from_youtube_unplayable") %> <%= video.reason %>
+        </h3>
+        <h3>
+            <%= translate(locale, "next_steps_error_message") %>


You probably want to use error_redirect_helper(). This error message alone (without any actions listed underneath) will be very confusing.

SamantazFox · 2024-10-19T14:47:26Z

+            <span id="refresh-page">
+                <a href="<% env.request.resource %>"><%= translate(locale, "refresh_page") %></a>
+            </span>


Why did you add this link here? All browsers allow to refresh the page natively, and the close procimity with the "watch on youtube" link is prone to miss-clicks.

SamantazFox · 2024-10-19T14:53:58Z

+  if !(video_details = player_response.dig?("videoDetails"))
+    video_details = {} of String => JSON::Any
+  end


Here you can use the more compact form:

Suggested change

if !(video_details = player_response.dig?("videoDetails"))

video_details = {} of String => JSON::Any

end

video_details = player_response.dig?("videoDetails")

video_details ||= {} of String => JSON::Any

SamantazFox · 2024-10-19T15:06:28Z

+    published_txt = video_primary_renderer
+      .try &.dig?("dateText", "simpleText")
+
+    if published_txt.try &.as_s.includes?("ago") && !published_txt.nil?
+      published = decode_date(published_txt.as_s.lchop("Started streaming "))
+    elsif published_txt && published_txt.try &.as_s.matches?(/(\w{3} \d{1,2}, \d{4})$/)
+      published = Time.parse(published_txt.as_s.match!(/(\w{3} \d{1,2}, \d{4})$/)[0], "%b %-d, %Y", Time::Location::UTC)
+    else
+      published = Time.utc
+    end


Here I'd recommend using "scheduledStartTime" (see below) in the player response, as it's more accurate and doesn't need complex parsing logic.

{ "playabilityStatus": { "status": "LIVE_STREAM_OFFLINE", "reason": "This live event will begin in 32 hours.", "playableInEmbed": true, "liveStreamability": { "liveStreamabilityRenderer": { "videoId": "hUoukmX_BRQ", "offlineSlate": { "liveStreamOfflineSlateRenderer": { "scheduledStartTime": "1729468800", "mainText": { "runs": [ {"text": "Live in "}, {"text": "32 hours"} ] }, "subtitleText": { "simpleText": "October 21 at 12:00 AM" } } } } } } }

Note: if you plan to fallback to the "subtitleText" value, there is a non-breaking space before AM/PM:

SamantazFox · 2024-10-19T15:29:59Z

    <% if video.reason %>
        <h3>
-            <%= video.reason %>
+            <%= translate(locale, "error_from_youtube_unplayable") %> <%= video.reason %>


Ditto: keeping the error message inside a player placeholder.

SamantazFox · 2024-10-19T15:37:34Z

-      # dont error when it's a premiere.
-      # we already parsed most of the data and display the premiere date
-      raise InfoException.new(reason.as_s || "")
+      raise NotFoundException.new(reason + ": Video not found" || "")


That will expand to "Video unavailable: Video not found", which is not very explicit, imo.

SamantazFox · 2024-10-19T15:41:06Z

+    if info.dig?("subreason").nil?
+      subreason = info["subreason"].as_s
+    else
+      subreason = "No additional reason"
+    end


This bloc of code can be summarized to the following, as you're already checking info["subreason"]? falsiness in the parent if

Suggested change

if info.dig?("subreason").nil?

subreason = info["subreason"].as_s

else

subreason = "No additional reason"

end

subreason = info["subreason"].as_s

SamantazFox · 2024-10-19T15:42:49Z

  else
    video = fetch_video(id, region)
-    Invidious::Database::Videos.insert(video) if !region
+    Invidious::Database::Videos.insert(video) if !region && !video.info.dig?("reason")


You should make a getter dunction inside the Video class for reason/subreason.

SamantazFox · 2024-10-19T15:43:51Z

-        "reason"  => JSON::Any.new(reason),
+        "version"   => JSON::Any.new(Video::SCHEMA_VERSION.to_i64),
+        "reason"    => JSON::Any.new(reason),
+        "subreason" => JSON::Any.new(subreason),


Don't forget to bump SCHEMA_VERSION in the Video class!

NintendoManiac64 · 2025-02-19T03:00:54Z

Integrate the embed page with the changes

To clarify, does this mean using the YouTube embedded player within the usual invidious video page? I know it wouldn't be "privacy respecting" in the same way as traditional invidious, so maybe as a sort of optional "compatibility mode" in the settings?

It sounds like some people have suggested something similar but instead mention just linking to the embed URL (youtube.com/embed/fo0bArBAZ) - I couldn't find anyone suggesting to directly embed the actual embedded YouTube player right on invidious.

(heck, I found out that you can even use various custom adblock filters to hide the "fluff" of the embed player such as suggestions & recommendations, and even end-screen video links/thumbnails, which makes me wonder if even invidious could similarly do that on the server side, or if that's only possible on the client side.)

...and now as I'm typing this, I'm remembering that YouTube supposedly has a newer type of embedded player that's "brandless" and lacks the fluff? Maybe that'd be better to use for embedding into the invidious video page...except, now that I review where I read about it, sounds like it's not something the user can choose and it's determined by the video channel itself?

Honestly my main use of invidious is just avoiding the "modern" bloated web page design of "javascript all of the things!" and the like, hence why I tend to still use invidious for navigating youtube even if I subsequently watch a video via the youtube.com/embed/fo0bArBAZ page

syeopite · 2025-02-19T05:58:14Z

To clarify, does this mean using the YouTube embedded player within the usual invidious video page?

No, it would just mean displaying video info even if the player is unable to be rendered for whatever reason. Everything will still be in Invidious.

ChunkyProgrammer · 2025-03-18T18:28:18Z

This should also close #4073

mk-pmb · 2025-03-19T05:35:41Z

Found this via #4744 . Is there any feedback still needed? Can I do something to help this move forward?

unixfox · 2025-03-19T07:48:08Z

@mk-pmb There is still all the reviews that needs to be taken care off.

If you know about crystal, I would be happy to receive some contribution. If that's what you can help us, feel free to do a PR on my branch: https://github.com/unixfox/invidious/tree/use-only-next-for-extra-info or create a new PR in the invidious project :).

mk-pmb · 2025-03-19T16:08:54Z

I don't know crystal but I can try and replicate your changes based on current master, with the suggestions from this thread applied. 👍

unixfox · 2025-03-21T23:41:14Z

hello @mk-pmb, yes sure sorry for the delay. I have created the branch: https://github.com/unixfox/invidious/tree/use-only-next-for-extra-info-base

I have opened the issues on my fork too.

I have a better idea, I invited you as a collaborator on my forked invidious repo, this way it will be easier to work on the branch use-only-next-for-extra-info directly. Don't screw up my others branches please 😄

Thanks

unixfox requested review from a team and SamantazFox as code owners October 13, 2024 17:47

unixfox marked this pull request as draft October 13, 2024 17:47

unixfox force-pushed the use-only-next-for-extra-info branch 2 times, most recently from de76e24 to 250cffb Compare October 13, 2024 18:35

unixfox mentioned this pull request Oct 13, 2024

update the mocks to today iv-org/mocks#3

Merged

unixfox force-pushed the use-only-next-for-extra-info branch 2 times, most recently from da00958 to a340868 Compare October 13, 2024 19:53

unixfox requested a review from syeopite October 13, 2024 21:40

unixfox mentioned this pull request Oct 13, 2024

[Feature request / Enhancement] Allow viewing video info when Youtube blocks video access for signed-out users. #4744

Open

unixfox added 2 commits October 14, 2024 00:07

trying to only use next endpoint for the major data

2a40f3b

extract islisted info + throw error only on not found video

9438a6d

unixfox force-pushed the use-only-next-for-extra-info branch from 60a5c1c to 9438a6d Compare October 13, 2024 22:08

fix detecting live_now

126f8fd

syeopite reviewed Oct 18, 2024

View reviewed changes

SamantazFox requested changes Oct 19, 2024

View reviewed changes

unixfox mentioned this pull request Dec 18, 2024

[Enhancement] Add YouTube embed link to video error page #5119

Closed

unixfox mentioned this pull request Mar 7, 2025

[Enhancement] Support for sensitive content videos. iv-org/invidious-companion#51

Closed

This comment was marked as outdated.

Sign in to view

mk-pmb mentioned this pull request Mar 22, 2025

Rebase the draft for "trying to only use next endpoint for the major data" onto master unixfox/invidious#1

Closed

mk-pmb mentioned this pull request Apr 11, 2025

Towards using only next endpoint for the major data. (Partial solution to #5003) #5237

Open

unixfox mentioned this pull request Aug 9, 2025

[Enhancement] The bug page should have a link to YouTube #5411

Closed

unixfox mentioned this pull request Aug 27, 2025

Be more informative when youtube says that a video is private #5435

Closed

Fijxu mentioned this pull request Nov 30, 2025

Remove signature helper completely from Invidious #5550

Merged

unixfox mentioned this pull request Jan 11, 2026

Add support for SABR #5588

Draft

6 tasks

Uh oh!

Conversation

unixfox commented Oct 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What changed?

TODO

Try these videos ID for testing - on both a blocked IP and a not blocked one

Fixes

Uh oh!

ic-scm commented Oct 17, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

syeopite Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NintendoManiac64 commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

syeopite commented Feb 19, 2025

Uh oh!

ChunkyProgrammer commented Mar 18, 2025

Uh oh!

mk-pmb commented Mar 19, 2025

Uh oh!

unixfox commented Mar 19, 2025

Uh oh!

mk-pmb commented Mar 19, 2025

Uh oh!

This comment was marked as outdated.

unixfox commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

unixfox commented Oct 13, 2024 •

edited

Loading

syeopite Oct 18, 2024 •

edited

Loading

NintendoManiac64 commented Feb 19, 2025 •

edited

Loading

unixfox commented Mar 21, 2025 •

edited

Loading