Use common observability stack for CI and experimentation #1708
Description
CI runs (track-performance.yml) and ad-hoc EC2 runs (aws-launch.sh) both execute the same reexecution binary but collect metrics separately. CI pushes to a self-hosted observability stack. EC2 runs install a local Grafana + Prometheus stack that dies with the instance.
Right now metrics are lost every time an EC2 instance gets terminated, and there's no way to compare a CI run against an experimental run because the data lives in different places. Putting both in one stack gives full history across all runs and allows A/B comparison. The maintenance of keeping two dashboards in sync is also eliminated since there's one central source.
Once the AvalancheGo team finishes cutting over to cloud, we should point both workflows at it and drop the local stack from aws-launch.sh.
To do that we need to:
- Wire Grafana Cloud credentials into aws-launch.sh (PROMETHEUS_PUSH_URL, PROMETHEUS_USERNAME, PROMETHEUS_PASSWORD)
- Remove the install-grafana.sh call from cloud-init
- Agree on a label scheme with the platform team (Stylianos flagged cardinality)
Blocked on platform's cloud cut-over