INF-314 AI prometheus metrics to set correct "pipeline" and "model_name" labels

[pwilczynskiclearcode]

What does this pull request do? Explain your changes. (required)
Normalising AI metrics to have correct pipeline, model_name and orchestratorUri tags.

Specific updates (required)

• ai_container_in_use to set correct mode_name label (orchestratorUri got removed)
• ai_container_idle to set correct mode_name label
• ai_gpus_idle to set mode_name label (orchestratorUri got removed)
• ai_current_live_pipelines to set model_name label (orchestratorUri got removed)
• ai_live_attempt to introduce model_name label

How did you test each of these updates (required)
...

Does this pull request close any open issues?
no

Checklist:

• Read the contribution guide
• make runs successfully
• All tests in ./test.sh pass
• README and other documentation updated
• Pending changelog updated

[linear]

INF-314 implement missing pipeline label in go-livepeer metrics

[codecov]

Codecov Report

❌ Patch coverage is 28.94737% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 31.90880%. Comparing base (531a043) to head (c3a5874).

Files with missing lines	Patch %	Lines
monitor/census.go	20.83333%	19 Missing ⚠️
server/ai_mediaserver.go	0.00000%	5 Missing ⚠️
ai/worker/docker.go	66.66667%	3 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##              master       #3699         +/-   ##
===================================================
- Coverage   31.92117%   31.90880%   -0.01237%     
===================================================
  Files            156         156                 
  Lines          47445       47454          +9     
===================================================
- Hits           15145       15142          -3     
- Misses         31405       31417         +12     
  Partials         895         895                 

Files with missing lines	Coverage Δ
ai/worker/docker.go	72.79412% <66.66667%> (-0.18382%)	⬇️
server/ai_mediaserver.go	4.66853% <0.00000%> (ø)
monitor/census.go	61.04053% <20.83333%> (-0.45582%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 531a043...c3a5874. Read the comment docs.

Files with missing lines	Coverage Δ
ai/worker/docker.go	72.79412% <66.66667%> (-0.18382%)	⬇️
server/ai_mediaserver.go	4.66853% <0.00000%> (ø)
monitor/census.go	61.04053% <20.83333%> (-0.45582%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[leszko]

Looks good. My only concern is that we all the time assuming that the O is running a single pipeline. It doesn't need to be in this PR, but we can think how to make it working for multiple pipelines. On other words, how to make GetCapacity() working with the pipeline param.

[leszko]

Note that this will only work if a given Orchestrator is running one single pipeline. It won't work if the O is running 2 separate pipelines. I think it may be ok for now, because in our infra we're using only a single pipeline for an orchestrator. But at least add a comment and some // TODO. We should work on supporting multiple pipelines in the metrics, because I believe that soon we'll have more pipelines.

[pwilczynskiclearcode]

I think that each pipeline+model would have its own DockerManager reporting metrics separately, right?

[leszko]

No, unfortunately, DockerManager is shared. So you can have 1 O running different pipelines.

[pwilczynskiclearcode]

This pipeline is actually our modelID 🤦🏻

[pwilczynskiclearcode]

so here (worker) ai_container_idle is produced with pipeline and model_name and here (orch.) with just model and orchestratorUri. Should we have separate metrics from O and W?

[leszko]

Now, I think it would be better to have a separate metric from Gateway and a separate from Orchestrator. Because it's confusing right now.

Anyway, I'm ok if it's done later, as a separate PR.