Skip to content

update detectors to reflect immutable attempt prompt#1415

Merged
leondz merged 3 commits intoNVIDIA:mainfrom
jmartin-tech:fix/detectors-output-none-support
Oct 28, 2025
Merged

update detectors to reflect immutable attempt prompt#1415
leondz merged 3 commits intoNVIDIA:mainfrom
jmartin-tech:fix/detectors-output-none-support

Conversation

@jmartin-tech
Copy link
Collaborator

Attempt.prompt is considered immutable once set as of #1254. This revision aligns expectation that outputs is only set once on an attempt after the generator provides inference results.

  • remove usage of Attempt.all_outputs
    • update detectors to process only latest Attempt.outputs
    • update detectors to support None value outputs

Additional optimization are envisioned and should be considered in a future iteration:

  • removal of Attempt.all_outputs
  • raise error when a call to set outputs performed more than once

Verification

List the steps needed to make sure this thing works

  • Execute probes to validate results are still consistent for a few end to end runs
  • Verify all unit test pass
  • Verify the thing does not do what it should not

@leondz
Copy link
Collaborator

leondz commented Oct 20, 2025

atkgen.Tox assumes evaluation of all_outputs, and this is a probe/detector eval scenario included in the conversation/turn spec (maybe that should be moved into the docs?) - suggest this behaviour is switchable and controlled by probe rather than globally locking us into one mode

@jmartin-tech
Copy link
Collaborator Author

atkgen.Tox assumes evaluation of all_outputs

This is no longer the case as of #1254, happy to discuss further.

* remove usage of `Attempt.all_outputs`
  * update detectors to process only latest `Attempt.outputs`
  * update detectors to support `None` value outputs

Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
@jmartin-tech jmartin-tech force-pushed the fix/detectors-output-none-support branch from 8468014 to e390e4b Compare October 21, 2025 20:54
@leondz
Copy link
Collaborator

leondz commented Oct 22, 2025

outcome of discussion was that we should support different scopes of detection, controllable by probe and expressed in attempt

will this PR limit atkgen's ability to have all outputs it generates scanned until that support is in place?

@jmartin-tech
Copy link
Collaborator Author

will this PR limit atkgen's ability to have all outputs it generates scanned until that support is in place?

atkgen was refactored in #1254 to emit attempts for each target inference request and is not impacted by this PR

@leondz
Copy link
Collaborator

leondz commented Oct 22, 2025

Great, lgtm

Copy link
Collaborator

@leondz leondz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few clarifications, no major blockers

* expand test for detector instance to ensure result order if preserved
* denote testing detectors are not expected to return `None`
* account for `0.0` treated as `False`
* account for `None` in message text of productkey detector
* return `None` for `None` in Json detector

Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
@jmartin-tech jmartin-tech requested a review from leondz October 27, 2025 17:23
* output `None` -> detector result `None`
* output `Messsage(text=None)` -> detector result `None`

Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
@leondz leondz merged commit 58afcd5 into NVIDIA:main Oct 28, 2025
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Oct 28, 2025
@jmartin-tech jmartin-tech deleted the fix/detectors-output-none-support branch October 28, 2025 20:41
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants