Overhauling perturbations to support custom perturbations#26
Conversation
| return Paraphrase._process_similar_sentence_reponse(response) | ||
|
|
||
| @staticmethod | ||
| def _process_similar_sentence_reponse(response): |
There was a problem hiding this comment.
@mshergad This should address the bug you reported for paraphrasing single sentences. At a later point we should add a class that can paraphrase multi-line prompts.
|
@tspthomas @lior-fiddler This PR allows users to define their own perturbations. Let me know if you have any thoughts |
tests/test_perturbations.py
Outdated
|
|
||
| def test_paraphrase(self): | ||
| for prompt in TRUTHFUL_DATASET: | ||
| sim_prompt = self.perturber.perturb(prompt) |
There was a problem hiding this comment.
under the hood the test will hit openai APIs, no ? should we mock out the openai call and return a pre-canned sample response ? i.e. avoid any side effects and slowness of calling external APIs in a test (assuming what we really want to test here is the parsing logic and the handling of the response)
There was a problem hiding this comment.
Yeah, we need to de-couple unit-tests from end-to-end testing. I created a separate ticket to resolve this #27
…3. deprecated perturbations_per_sample
2 participants
This PR introduces Perturbations base class that can be inherited to create custom perturbations this addressing #15
As part of this I also re-wrote the Paraphrase perturbations and fixed the error reported in #23 by @mshergad