Dash update Manifest Location tag not followed for relative urls

[kcook98765]

Running latest Omega versions. Had an issue where Live TV via Hulu was only lasting 5 minutes, then stalling.

Added to issue at matthuisman/slyguy.addons#584

I have turned off Matt's proxy to test having just IA in the loop.

The first Manifest in such a test is attached. It contains a Location tag that is a relative url :
<Location>/v1/live/dash/continue/531124fc-2566-4206-b155-1b4f216e9ec8/manifest.mpd?cdns=sf&amp;cluster=blue&amp;origin=us-west-2&amp;duration=10800s&amp;manifest_cdn=ak&amp;repeat=true&amp;patch=true&amp;hpbsid_enabled=false&amp;uas_enabled=false&amp;segment_type_enabled=false&amp;query_params_v2=false&amp;unencrypted=false&amp;cver=0&amp;acquired_source_name=ABC_KOMO_SEA_MMA_CLT2-USW2&amp;stream_filter=fhd_high_stereo&amp;prev_start=1698533614322&amp;prev_boundary_start=1697130402514&amp;prev_pts_values=AAC_1_CMAF_CENC_CTR%3D68325770240%7CH264_1_CMAF_CENC_CTR%3D85407211890&amp;authToken=1698534232_cf0ec596424db8ccae76227b5f0f506210b14e605f7c02bb59aec3cbfa912859</Location>

But monitoring the Manifest copies stored by IA when run never shows the Location being followed, instead every Manifest request is to the original Manifest, which only has 5 minutes of streaming.

Shouldn't IA follow the Location for the next update Manifest (and in turn if it has a Location to then follow it)?

I have tested from an Rpi 3b and a firestick, same issue.

manifest_1698533632.txt

When I did test with Matt's proxy, I had modified to make the Location absolute which in turn at least had IA follow it, thought something else is going on as it did not get past the first 5 minutes, like processing of the partial update did not work (or maybe something else going on), but I want to first start here.

Also, if following these update manifests, and one call should have an issue, should IA fall back to "lasl" followed Location Manifest, instead of falling back to the very first one which it does now should any of the Location url's fail?

Thank You for your time and effort on this project.

[CastagnaIT]

I have turned off Matt's proxy to test having just IA in the loop.
The first Manifest in such a test is attached. It contains a Location tag that is a relative url :
But monitoring the Manifest copies stored by IA when run never shows the Location being followed, instead every Manifest request is to the original Manifest, which only has 5 minutes of streaming.

hmm have sense,
a relative url on Location tag is not supported yet on ISAdaptive, then the value is ignored, but could be an easy fix

reading the dash spec i havent found a clear explanation of Location tag with relative url
but imo could have sense merge the Location relative url with the base domain url from the downloaded manifest url
from your attached manifest an example become as follow:

first manifest url (your downloaded mpd):
https://livem-akc-na.media.hulustream.com/v1/live/dash/initial/531124fc-2566-4206-b155-1b4f216e9ec8/manifest.mpd?acquired_source_name=ABC_KOMO_SEA_MMA_CLT2-USW2&cdns=sf&cluster=blue&cver=0&duration=10800s&manifest_cdn=ak&origin=us-west-2&patch=true&query_params_v2=false&repeat=true&segment_type_enabled=false&stream_filter=fhd_high_stereo&uas_enabled=false&authToken=1698533927_86b00f91d453bf82a5677d3a437f87f55602c402c47e371595e17a151340dd57

nexy update manifest url (base domain taken from above + location tag):
https://livem-akc-na.media.hulustream.com/v1/live/dash/continue/531124fc-2566-4206-b155-1b4f216e9ec8/manifest.mpd?cdns=sf&cluster=blue&origin=us-west-2&duration=10800s&manifest_cdn=ak&repeat=true&patch=true&hpbsid_enabled=false&uas_enabled=false&segment_type_enabled=false&query_params_v2=false&unencrypted=false&cver=0&acquired_source_name=ABC_KOMO_SEA_MMA_CLT2-USW2&stream_filter=fhd_high_stereo&prev_start=1698533614322&prev_boundary_start=1697130402514&prev_pts_values=AAC_1_CMAF_CENC_CTR%3D68325770240%7CH264_1_CMAF_CENC_CTR%3D85407211890&authToken=1698534232_cf0ec596424db8ccae76227b5f0f506210b14e605f7c02bb59aec3cbfa912859

[CastagnaIT]

hoping its not the only problem for your stream i made a fix
you can find some test builds here:
https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.adaptive/detail/PR-1409/1/artifacts

[kcook98765]

That did work, in that it followed the relative Location URL's !

There is something else going on though with partial update Manifests.

The initial Manifest loads and playback begins.

But, something is wrong with the update manifests.

After the first Update manifest is downloaded, I get:

2023-10-30 11:52:32.504 T:3580 debug <general>: AddOnLog: inputstream.adaptive: Download finished: https://livem-akc-na.media.hulustream.com/v1/live/dash/continue/531124fc-2566-4206-b155-1b4f216e9ec8/manifest.mpd?cdns=sf&cluster=blue&origin=us-west-2&duration=10800s&manifest_cdn=ak&repeat=true&patch=true&hpbsid_enabled=false&uas_enabled=false&segment_type_enabled=false&query_params_v2=false&unencrypted=false&cver=0&acquired_source_name=ABC_KOMO_SEA_MMA_CLT2-USW2&stream_filter=fhd_high_surround&prev_start=1698691927479&prev_boundary_start=1697130402514&prev_pts_values=AAC_1_CMAF_CENC_CTR%3D75924801536%7CEC3_1_CMAF_CENC_CTR%3D75924801575%7CH264_1_CMAF_CENC_CTR%3D94906001190&authToken=1698692545_851ff4108edb841b7c164a9f5335a64970bf810c13b9196511c2afd40f16bd25 (downloaded 2460 byte, speed 3322.00 byte/s) 2023-10-30 11:52:32.505 T:3580 debug <general>: AddOnLog: inputstream.adaptive: Manifest saved to: C:\Users\kcook\AppData\Roaming\Kodi\userdata\addon_data\inputstream.adaptive\manifests\manifest_1698691952.txt 2023-10-30 11:52:32.505 T:3580 warning <general>: AddOnLog: inputstream.adaptive: adaptive::CDashTree::ParseTagAdaptationSet: Skipped AdaptationSet with id: "4289974598", has no representations. 2023-10-30 11:52:32.505 T:3580 warning <general>: AddOnLog: inputstream.adaptive: adaptive::CDashTree::ParseTagAdaptationSet: Skipped AdaptationSet with id: "2008774996", has no representations. 2023-10-30 11:52:32.505 T:3580 warning <general>: AddOnLog: inputstream.adaptive: adaptive::CDashTree::ParseTagAdaptationSet: Skipped AdaptationSet with id: "901119627", has no representations. 2023-10-30 11:52:32.505 T:3580 warning <general>: AddOnLog: inputstream.adaptive: No periods in the manifest

I am including the Initial Manifest as well as the first Update Manifest here.

manifest_1698691946.txt
manifest_1698691952.txt

The update Manifest indeed contains no representations , but the Period, AdaptationSet, SegmentTemplate and SegmentTimeline sections do appear.

Should this be enough information for the update Manifest to function?
kodi.log

[CastagnaIT]

thanks for new provided info helped a lot,
by looking at your attached manifest i think this is a new use case, that isnt currently supported

IIRC the current implementation assume to have a full live update manifest data and not a partial one with only segments, therefore as we can see the update fails

this will require some changes in to
CDashTree::RefreshLiveSegments to handle manifest with only segments
and some little adjustments in how we parse the manifest data to prevent that the parsing stop the manifest update due to missing data

so a fix will be possible

PS. Note that this type of improvement will be able to be made for Kodi 21 and not on Kodi 20 due to many non-backportable changes

[kcook98765]

Thank You!

I will be available to test changes for this when you have some ready.

[kcook98765]

Any possibility of getting example initial manifest and an update manifest that "work" for IA currently. They don't need to be able to be played, I just want to review the XML and experiment with a proxy for Hulu Partial updates to see if I can work out a way to convert the Hulu partial updates into a form IA currently understands to patch this issue until IA can handle partial updates?

I have been testing many variations to try and do this, but an example set would help me greatly , and then I might be able to make this work.

Thank you!

[CastagnaIT]

not all live manifests are the same, so comparing with others services may cause confusion
you already have your manifests above

its possible that just coping the relative Representation tags from the first manifest to future update manifests, ofc by respecting the parent AdaptationSet id's, could temporary solve the problem until a real fix is implemented

but i think you will need also find a solution for the relative urls, since with a proxy more likely they will be broken

[kcook98765]

Thank you for the feedback. I had tried adding just the missing Representation elements (in the correct Adaption ID areas) from a copy of the original manifest when it gets to the proxy where I check matching period and adaptation id to determine which representation elements to insert , but then the stream halts within 10 seconds (even while additional update manifests continue to be downloaded via the "Location" tag which has the proper proxy adjustment.

That is why I wanted examples, as I was going to examine which elements/attributes were used in both Initial and update manifests to see if there were other pieces of information I need to modify in the partial update manifests.

Some other things I had tried were to force MPD to have attribute type="dynamic" (as it is in intial, but not partial manifests). I also tried many other adjustments like this, but can't seem to find the magic combination of changes to get it to work.

The proxy is solid and doing it's job, requests are coming back with 200 status, saved debug manifests are looking as expected with each change, so I think it is some other key piece(s) of data that I need to account for.

Also wondering if maybe partial update in IA is not triggered without a "start_number" as mentioned in IA code:

// YouTube needs segment start number as parameter
bool urlHaveStartNumber = manifestParams.find("$START_NUMBER$") != std::string::npos;

and

if (urlHaveStartNumber) // Partitial update

I have not tried this yet, was thinking of adding in this to the url (and stripping it when proxy contacts service).

[kcook98765]

Been trying all kinds of adjustments based on initial manifest. MPD "type" attribute added. Representation elements added. BaseURL added. SegmentTemplate attributes added (plus calculating a statrNumber to insert) .

Still not quite working as it cuts out quickly, but logs at least show :

2023-11-09 17:01:10.388 T:943 debug : AddOnLog: inputstream.adaptive: RefreshLiveSegments: Full update with start number (repr. id: AAC_1_CMAF_CENC_CTR_128K current start:1685)
2023-11-09 17:01:10.388 T:943 debug : AddOnLog: inputstream.adaptive: RefreshLiveSegments: Full update with start number (repr. id: AAC_1_CMAF_CENC_CTR_64K current start:1685)
2023-11-09 17:01:10.388 T:943 debug : AddOnLog: inputstream.adaptive: RefreshLiveSegments: Full update with start number (repr. id: H264_1_CMAF_CENC_CTR_192K current start:2)

still plugging away, but running out of things to try.

Please do give me an example initial and partial update manifest I can study, even if it is another service as it might shed a light on some of the attributes, etc that may help me.

Thanks!

[matthuisman]

have you tried just removing Location tag?
Or having your partial update just return the latest full manifest?

[kcook98765]

It is a live feed, so fairly sure it must use Location, as initial manifest never updates after first call.

Also, I forked and am testing changes in your proxy code to do this (bit hacky as I test things), you can see current changes at:

matthuisman/slyguy.addons@master...kcook98765:slyguy.addons:master

I collect various data from initial manifest, then re-route update manifests (currently triggered on "continue" in the url to a new section of code where I make adjustments to the update manifest before returning it to IA

I'll be out until tomorrow now, but any hints would be helpful if you spot something, thanks!