BrowseComp reproduction with Seed-OSS-36B-Instruct via vLLM yields 0 score

[hb-studying]

Congratulations on releasing this work.

I’m trying to reproduce the reported BrowseComp evaluation results using the exact code provided in this repository. My setup is:

• Backend: vLLM

• Model: Seed-OSS-36B-Instruct

• I only changed two runtime flags:

  • --max-model-len from 131072 to 65536
  • --num_workers from 32 to 2

However, my evaluation result is always 0, which seems abnormal:

Overall - Avg Score: 0.0000, Success: 150/150

By Data Source:
  bc_test_easy: 0.0000 (50 items)
  bc_test_hard: 0.0000 (50 items)
  bc_test_meduim: 0.0000 (50 items)

From the logs, it looks like the tasks “finish” successfully, but the judged score is always 0, and the model’s answers seem consistently incorrect. Here are a few examples:

False <class 'envs.local_search.LocalSearch'>
[BRANCH] Identify Person 5991
[BRANCH] Find Hospital 6413
[TASK] Task Finish, Start Reward
[Judged] score=0
Label: Sanatorio de la Trinidad
Model: Paris, France
('', 0)

False <class 'envs.local_search.LocalSearch'>
[BRANCH] Identify Developer 15663
[BRANCH] Verify Commodore 64 Claim 16529
[CallAPI] Attempt 4 failed: Request timed out.. Retrying in 8s...
[TASK] Task Finish, Start Reward
[Judged] score=0
Label: Atari 130XE
Model: Commodore 64
('', 0)

False <class 'envs.local_search.LocalSearch'>
[TASK] Task Finish, Start Reward
[Judged] score=0
Label: Owlman
Model: Raccoon
('', 0)

False <class 'envs.local_search.LocalSearch'>
[TASK] Task Finish, Start Reward
[Judged] score=0
Label: Pierre Montale
Model: Charlotte Emma Tilbury
('', 0)

False <class 'envs.local_search.LocalSearch'>
[BRANCH] Author 2 Clues 7734
[BRANCH] Author 1 Institution 8594
[BRANCH] Verify Variety Term 9448
[TASK] Task Finish, Start Reward
[Judged] score=0
Label: Nicotiana tabacum variety Wisconsin 38
Model: amino acid variety
('', 0)

Could you please let me know:

1. Have you encountered a similar “all-zero score” issue during evaluation?
2. If possible, could you share a reference evaluation output/log snippet (or a minimal example run) so I can compare against my run?

Thank you very much for your time and for maintaining this project.

[xzk1234]

I think this might be related to the issue I described in #10.
The assigned research task doesn't appear to be reaching the branch agent prompts. As a potential workaround, try explicitly wrapping the message like so:
<ASSIGNED_TASK> {message} </ASSIGNED_TASK>
Hope this helps!

[hb-studying]

Thanks for the quick response. I’ll try it.

In the meantime, I noticed another potential issue around LOCAL_SEARCH_URL that might affect reproduction:

• The README suggests setting:
  export LOCAL_SEARCH_URL="http://127.0.0.1:8010"
• But in eval_bc, it uses:
  os.environ["LOCAL_SEARCH_URL"] = args.local_search_url
• And the argument is defined as:
  parser.add_argument('--local_search_url', default='http://localhost:8010')

So if one follows the README strictly (as I initially did), the port/host may not be applied as expected unless --local_search_url is explicitly passed. After I corrected this on my side, the score for Seed-OSS-36B-Instruct is no longer 0.

However, I’m still seeing surprisingly low performance with Qwen-3 8B: on the 150 BrowseComp questions it only solves 1, and the single success is not consistent across runs.

Have you been able to successfully reproduce the training pipeline for the open-sourced Qwen3-8B training code? I’m curious what level of performance the trained Qwen3-8B model can reach on BrowseComp after training.

I’m planning to reproduce the Qwen3-8B training as well — would be happy to keep exchanging notes and discuss results as I make progress.

Thanks again for your help.

I think this might be related to the issue I described in #10. The assigned research task doesn't appear to be reaching the branch agent prompts. As a potential workaround, try explicitly wrapping the message like so: Hope this helps!<ASSIGNED_TASK> {message} </ASSIGNED_TASK>