GAIA Benchmark

[technocreep]

Hello guys! I really impressed by you work and would like to run MaAS on GAIA benchmark. I saw the results in the paper but I'd like to reproduce them by myself. Unfortunately it is unclear to me how to run MaAS although there are templates in maas/ext/maas/scripts/optimized/ for HumanEval, MATH, and GSM8K. Could you please share some runnable script? Thanks in advance

[SimaDidari]

Hi, I'm using the default settings from the code for HumanEval training, but I'm not getting the same results as reported (87% instead of 92% for gpt4-mini), were you able to run for HumanEval ?

[technocreep]

Hi, I'm using the default settings from the code for HumanEval training, but I'm not getting the same results as reported (87% instead of 92% for gpt4-mini), were you able to run for HumanEval ?

Hi

I'm trying to run as they say in the readme, starting from this part:

python -m examples.maas.optimize --dataset HumanEval --round 1 --sample 4 --exec_model_name "gpt-4o-mini"

It starts logging some gibberish in terminal:

Not sure if this is right thing, but I'm going to wait until the end. As far as I understood the outcome of it should be Controller model, then I will proceed with this step:

python -m examples.maas.optimize --dataset HumanEval --round 1 --sample 4 --exec_model_name "gpt-4o-mini" --is_test True

Was it the same for you?

[technocreep]

It is not clear to me btw how to operate Human Eval dataset. They say data folder should be as follows:

data
└── gsm8k_train.jsonl
└── gsm8k_test.jsonl

But there is only one split for HumanEval and it is TEST

[SimaDidari]

Hi,
I assumed that they downloaded both splits from AFlow (here is the link) . You can check
https://github.com/FoundationAgents/AFlow/blob/main/data/download_data.py
the link for humaneval is
https://drive.google.com/uc?id=1DNoegtZiUhWtvkd2xoIuElmIi4ah7k8e
the appendix of the supernet paper mentioned that 164 total samples are split as 131 and 33 , and you can find the split in aflow google drive link.

[technocreep]

Hi, I assumed that they downloaded both splits from AFlow (here is the link) . You can check https://github.com/FoundationAgents/AFlow/blob/main/data/download_data.py the link for humaneval is https://drive.google.com/uc?id=1DNoegtZiUhWtvkd2xoIuElmIi4ah7k8e the appendix of the supernet paper mentioned that 164 total samples are split as 131 and 33 , and you can find the split in aflow google drive link.

Then I changed this line in the code : file_path = "maas/ext/maas/data/humaneval_public_test.jsonl" --> file_path = "maas/ext/maas/data/humaneval_test.jsonl" in extract_test_cases_from_jsonl function in /MaAS/maas/ext/maas/scripts/utils.py

Thanks for your response, I managed to do so in the mean time.

If you don't mind I'd like to ask you. How long does it take to obtain controller for humaneval? Approximately

[technocreep]

It took at least 5 hours, where you able to reproduce the result ?

Hi, I reproduced:

2025-10-06 06:58:24.323 | INFO | maas.ext.maas.benchmark.benchmark:run_evaluation:205 - Average score on HumanEval dataset: 0.93130

[SimaDidari]

Thank you, I ran twice and once I got 87% and the second time I got 91%. Then I turned on the text-grad because I though the default is with no text grad and I got 91% again. Did you run it with text-grad as well and what was your llm model?

[technocreep]

Thank you, I ran twice and once I got 87% and the second time I got 91%. Then I turned on the text-grad because I though the default is with no text grad and I got 91% again. Did you run it with text-grad as well and what was your llm model?

What I did is:

1. Loaded data that you mentioned
2. Placed it into maas/ext/maas/data
3. In maas/ext/maas/scripts/evaluator.py I changed a single line in _get_data_path method to make downloaded data be accessible:

return f"{base_path}_test.jsonl" if test else f"{base_path}_validation.jsonl"

4. In maas/ext/maas/scripts/optimizer_utils/data_utils.py I changed method create_result_data signature to:

    def create_result_data(self, round: int, score: float, avg_cost = 0, total_cost = 0, token = 'nan') -> dict:

in order to make execution not fail in error of cost calculation as I'm really not interested in it.

5. I ran

python -m examples.maas.optimize --dataset HumanEval --round 1 --sample 4 --exec_model_name "gpt-4o-mini"

after 40 minutes (depends on hardware I suppose) I ran

python -m examples.maas.optimize --dataset HumanEval --round 1 --sample 4 --exec_model_name "gpt-4o-mini" --is_test True

6. I got mentioned results

So, answering your questions, I would say that I never changed any settings, only made minor code tweaks

[SimaDidari]

Thank you! I was able to reproduce the results. Thanks again!

[SimaDidari]

Also now it takes about the same time as yours to train, about 50 mins.

[technocreep]

Also now it takes about the same time as yours to train, about 50 mins.

That's great, congras :)

Now I would be awesome to figure out how to deal with GAIA, as it was my main goal

[oximi123]

Also now it takes about the same time as yours to train, about 50 mins.

That's great, congras :)

Now I would be awesome to figure out how to deal with GAIA, as it was my main goal

Hi, I hope everything’s going well.
Have you had any success figuring out how to deal with GAIA?