Skip to content

update: add probe tiers#1151

Merged
leondz merged 4 commits intoNVIDIA:mainfrom
leondz:update/add_probe_tiers
Apr 11, 2025
Merged

update: add probe tiers#1151
leondz merged 4 commits intoNVIDIA:mainfrom
leondz:update/add_probe_tiers

Conversation

@leondz
Copy link
Collaborator

@leondz leondz commented Apr 7, 2025

Add tier values to probes, indicating the priority of each probe.

Probe Tier Description Notes
1 Of concern Low pass rate or low z-score may be problematic
2 Compete with state of the art Low z-score may be problematic
3 Informational Context sensitive; low scores may affect suitable use cases
"u" Unlisted Duplicated, deprecated, fluctuating, or non-adversarial probe.

Verification

  • python -m pytest tests/probes/test_probes.py::test_probe_metadata
  • Check tier values in probe instances

@leondz leondz added architecture Architectural upgrades probes Content & activity of LLM probes labels Apr 7, 2025
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tier list should be a constant enum in probes/base.Probe and each entry should be a value from that enum. This will allow testing to validate and in the future allow stronger typing validation of metadata.

@leondz
Copy link
Collaborator Author

leondz commented Apr 7, 2025 via email

@erickgalinkin
Copy link
Collaborator

What's your favourite implementation of constant enums in py?

https://docs.python.org/3/library/enum.html

@leondz
Copy link
Collaborator Author

leondz commented Apr 10, 2025

What's your favourite implementation of constant enums in py?

https://docs.python.org/3/library/enum.html

:vibrating eyes:

@leondz leondz self-assigned this Apr 11, 2025
@leondz leondz merged commit 0e26d89 into NVIDIA:main Apr 11, 2025
9 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Apr 11, 2025
@jmartin-tech
Copy link
Collaborator

After this landed I notice this does not set tier as a standard attribute for base.Probe meaning the tier for the specific probe entry is not in the cache. This also stuffed the CONSTANT entries in every probe. I will open a PR shortly to refactor this slightly.

@leondz
Copy link
Collaborator Author

leondz commented Apr 15, 2025

Former is due to some multiply inheriting classes having poor inheritance order. Latentinjection refactor addresses some of these cases.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

architecture Architectural upgrades probes Content & activity of LLM probes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants