Skip to content

Fix pairwise_judge to use A/B instead of col_A/col_B. col_A/col_B use 2 tokens, so cascade fails#248

Merged
liana313 merged 12 commits intomainfrom
fix/cascade-token
Apr 13, 2026
Merged

Fix pairwise_judge to use A/B instead of col_A/col_B. col_A/col_B use 2 tokens, so cascade fails#248
liana313 merged 12 commits intomainfrom
fix/cascade-token

Conversation

@harshitgupta412
Copy link
Copy Markdown
Collaborator

Essential Elements of an Effective PR Description Checklist

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating README.md and examples for new features.

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS ABOVE HAVE BEEN CONSIDERED.

Purpose

Test Plan

Test Results

(Optional) Documentation Update

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Performance improvement
  • Refactoring (no functional changes)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, updating docstrings
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

BEFORE SUBMITTING, PLEASE READ https://github.com/lotus-data/lotus/blob/main/CONTRIBUTING.md
anything written below this line will be removed by GitHub Actions

Copy link
Copy Markdown
Collaborator

@liana313 liana313 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also update CI tests to include the pairwise judge and test for edge cases on this

Comment thread .github/tests/lm_tests.py
assert list(df["_judge_1"].values) == ["A", "B"]


@pytest.mark.parametrize("model", get_enabled("gpt-4o-mini"))
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a start, but we need more than one CI test for cascade testing. We should cover edge cases (e.g., when there's an existing col A and col B)

@liana313 liana313 merged commit ce544e1 into main Apr 13, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants