fix(sem_search): add semantic search quality evaluation with llm judge and improve descriptions

[amitksingh1490]

No description provided.

[amitksingh1490]

Fixed in commit 65b896c ✅

The review comment was correct - the script was conditionally adding parameters that llm_judge.ts requires unconditionally.

Changes made:

• Always pass both --expected-file-types and --should-avoid to llm_judge.ts
• Set empty string defaults when EXPECTED_TYPES or SHOULD_AVOID are not provided
• This ensures the command always has the required arguments, preventing "Missing required arguments" errors

Co-Authored-By: ForgeCode noreply@forgecode.dev

[graphite-app]

Hardcoded absolute path /Users/amit/code-forge/benchmarks/evals/semantic_search_quality will fail on any machine other than the original developer's. This breaks portability and will cause test failures in CI/CD or on other developer machines.

Fix: Use relative paths or dynamically determine the script directory:

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

if cd "$SCRIPT_DIR" && \
   ./run_eval.sh /tmp/test_semantic_eval/full_context.json > /dev/null 2>&1; then

Suggested change

	if cd /Users/amit/code-forge/benchmarks/evals/semantic_search_quality && \
	./run_eval.sh /tmp/test_semantic_eval/full_context.json > /dev/null 2>&1; then
	echo " ✓ PASSED (score >= 70)"
	PASSED=$((PASSED + 1))
	else
	echo " ✗ FAILED (expected pass)"
	FAILED=$((FAILED + 1))
	fi
	# Test 2: Documentation queries - should PASS (may be marginal)
	echo "Test 2: Documentation queries..."
	if cd /Users/amit/code-forge/benchmarks/evals/semantic_search_quality && \
	./run_eval.sh /tmp/test_semantic_eval/doc_context.json > /dev/null 2>&1; then
	echo " ✓ PASSED (score >= 70)"
	PASSED=$((PASSED + 1))
	else
	echo " ✗ FAILED (expected pass)"
	FAILED=$((FAILED + 1))
	fi
	# Test 3: Bad queries - should FAIL
	echo "Test 3: Bad queries (generic keywords)..."
	if cd /Users/amit/code-forge/benchmarks/evals/semantic_search_quality && \
	./run_eval.sh /tmp/test_semantic_eval/bad_context.json > /dev/null 2>&1; then
	echo " ✗ FAILED (expected failure, got pass)"
	FAILED=$((FAILED + 1))
	else
	echo " ✓ PASSED (correctly failed - score < 70)"
	PASSED=$((PASSED + 1))
	fi
	# Test 4: Missing sem_search - should FAIL early
	echo "Test 4: Missing sem_search tool..."
	if cd /Users/amit/code-forge/benchmarks/evals/semantic_search_quality && \
	./run_eval.sh /tmp/test_semantic_eval/no_sem_search_context.json > /dev/null 2>&1; then
	echo " ✗ FAILED (expected early exit failure)"
	FAILED=$((FAILED + 1))
	else
	echo " ✓ PASSED (correctly failed early)"
	PASSED=$((PASSED + 1))
	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
	if cd "$SCRIPT_DIR" && \
	./run_eval.sh /tmp/test_semantic_eval/full_context.json > /dev/null 2>&1; then
	echo " ✓ PASSED (score >= 70)"
	PASSED=$((PASSED + 1))
	else
	echo " ✗ FAILED (expected pass)"
	FAILED=$((FAILED + 1))
	fi
	# Test 2: Documentation queries - should PASS (may be marginal)
	echo "Test 2: Documentation queries..."
	if cd "$SCRIPT_DIR" && \
	./run_eval.sh /tmp/test_semantic_eval/doc_context.json > /dev/null 2>&1; then
	echo " ✓ PASSED (score >= 70)"
	PASSED=$((PASSED + 1))
	else
	echo " ✗ FAILED (expected pass)"
	FAILED=$((FAILED + 1))
	fi
	# Test 3: Bad queries - should FAIL
	echo "Test 3: Bad queries (generic keywords)..."
	if cd "$SCRIPT_DIR" && \
	./run_eval.sh /tmp/test_semantic_eval/bad_context.json > /dev/null 2>&1; then
	echo " ✗ FAILED (expected failure, got pass)"
	FAILED=$((FAILED + 1))
	else
	echo " ✓ PASSED (correctly failed - score < 70)"
	PASSED=$((PASSED + 1))
	fi
	# Test 4: Missing sem_search - should FAIL early
	echo "Test 4: Missing sem_search tool..."
	if cd "$SCRIPT_DIR" && \
	./run_eval.sh /tmp/test_semantic_eval/no_sem_search_context.json > /dev/null 2>&1; then
	echo " ✗ FAILED (expected early exit failure)"
	FAILED=$((FAILED + 1))
	else
	echo " ✓ PASSED (correctly failed early)"
	PASSED=$((PASSED + 1))
	fi

Spotted by Graphite Agent



Is this helpful? React 👍 or 👎 to let us know.

[graphite-app]

Using eval with string interpolation is a shell injection vulnerability. If $CONTEXT_FILE, $INTENT, $EXPECTED_TYPES, or $SHOULD_AVOID contain shell metacharacters, arbitrary commands could be executed.

# Fix: Execute command directly without eval
npx tsx llm_judge.ts --context "$CONTEXT_FILE" --intent "$INTENT" --expected-file-types "$EXPECTED_TYPES" --should-avoid "$SHOULD_AVOID"

Suggested change

	CMD="npx tsx llm_judge.ts --context \"$CONTEXT_FILE\" --intent \"$INTENT\" --expected-file-types \"$EXPECTED_TYPES\" --should-avoid \"$SHOULD_AVOID\""
	eval $CMD
	npx tsx llm_judge.ts --context "$CONTEXT_FILE" --intent "$INTENT" --expected-file-types "$EXPECTED_TYPES" --should-avoid "$SHOULD_AVOID"

Spotted by Graphite Agent



Is this helpful? React 👍 or 👎 to let us know.