Installation/replicating point mass experiments

[ajdroid]

Hi, I've been trying to replicate some of your results and compare against Reptile, for the reinforcement learning case. Specifically, I'm trying to run the point mass experiments.

After installing everything and sourcing the conda env rllab3 and running python maml_examples/maml_trpo_point.py, it complains about Theano and Lasagne not being installed.

From the README, it seems like this was built on the Tensorflow version of rllab. Does it still need Theano or am I doing something wrong?

PS: Thanks for open-sourcing your work!

[cbfinn]

Yes, unfortunately, the Tensorflow version of rllab does require theano.

FYI, with regard to reptile, the easiest thing to try would be first-order MAML by adding a stop gradient to the existing code. I believe that I tried this awhile back, and found that it didn't work. But, I can't remember exactly which setting I was looking at.

[ajdroid]

Alright, so I was able to get the train script to run with theano 0.7 (nothing newer works). (Worth adding in the README, maybe? I'm trying to add more instructions to everything so if I understand everything properly, I can send a PR when done if you'd like)

Also, when I run python maml_examples/trpo_point.py which I assume is the baseline, it throws:

AttributeError: 'GaussianMLPPolicy' object has no attribute 'all_param_vals'

To test (after training with maml_examples/maml_trpo_point.py) I'm trying to use the script maml_examples/test_maml_point.py. It however imports the Mujoco Walker2D env, which I don't have access to currently. It doesn't seem to be used inside that file. Is it required?

At this point, my question also is, what is correct order of scripts to run to replicate your 2D pointmass experiments from the paper? I was trying:

1. run python maml_examples/maml_trpo_point.py
2. run python maml_examples/trpo_point.py
3. run python maml_examples/test_maml_point.py
   Maybe I'm doing this wrong?

[cbfinn]

The particular version of theano that works is in environment.yml.

It however imports the Mujoco Walker2D env

Sorry about that. It is not needed.

what is correct order of scripts to run to replicate your 2D pointmass experiments from the paper? I was trying...

Yes, that's correct. Step 1 will train MAML and step 2 will train the baseline.

I don't have time to look into the error right now, but I can look into it later this week. I suspect that it is a very simple fix.

[ajdroid]

Okay, thanks! :)

Meanwhile, I ran the testing code for just the MAML training output.
This assertion always returns false. The last column of the csv is, in fact "AveragePolicyStd" rather than "AverageReturn".

I've changed it to find "Average Return" now, but does this unexpected behaviour mean something else is wrong in the pipeline?

[ajdroid]

@cbfinn Hi, sorry did you have any time to look at the bug?

[shubhamag]

Also, when I run python maml_examples/trpo_point.py which I assume is the baseline, it throws:

AttributeError: 'GaussianMLPPolicy' object has no attribute 'all_param_vals'

Hi, I'm encountering the same issue as well. @ajdroid @cbfinn Could either of you figure out a fix for this ?

[ajdroid]

Hi, just asking again if you had any time to look at this @cbfinn !

[zackenton]

I ran into this too, I think this can be fixed by changing the -1 to 1 in lines 102 and 104 https://github.com/cbfinn/maml_rl/blob/master/maml_examples/test_maml_point.py#L102.
You may also need to change 'w' to 'wb' in https://github.com/cbfinn/maml_rl/blob/master/maml_examples/test_maml_point.py#L120

[hl00]

Hi, have any of you fixed this issue? @cbfinn @ajdroid @shubhamag

[yahya-sattar]

@cbfinn any update on the following problem?
AttributeError: 'GaussianMLPPolicy' object has no attribute 'all_param_vals'

[yahya-sattar]

Alright, so I was able to get the train script to run with theano 0.7 (nothing newer works). (Worth adding in the README, maybe? I'm trying to add more instructions to everything so if I understand everything properly, I can send a PR when done if you'd like)

Also, when I run python maml_examples/trpo_point.py which I assume is the baseline, it throws:

AttributeError: 'GaussianMLPPolicy' object has no attribute 'all_param_vals'

To test (after training with maml_examples/maml_trpo_point.py) I'm trying to use the script maml_examples/test_maml_point.py. It however imports the Mujoco Walker2D env, which I don't have access to currently. It doesn't seem to be used inside that file. Is it required?

At this point, my question also is, what is correct order of scripts to run to replicate your 2D pointmass experiments from the paper? I was trying:

1. run `python maml_examples/maml_trpo_point.py`

2. run `python maml_examples/trpo_point.py`

3. run `python maml_examples/test_maml_point.py`
   Maybe I'm doing this wrong?

@ajdroid Hi,
Were you able to run step 2?

[GbengaOdesanmi]

@ALL I run python maml_trpo_point.py but am having this error

python3 /home/gbenga/Documents/2019 vrep code/maml_rl-master/scripts/run_experiment_lite.py --n_parallel '1' --snapshot_mode 'last' --seed '1' --plot 'False' --exp_name 'trpomaml1_fbs20_mbs40_flr_0.5metalr_0.01_step11' --log_dir '/home/gbenga/Documents/2019 vrep code/maml_rl-master/data/local/vpg-maml-point100/trpomaml1_fbs20_mbs40_flr_0.5metalr_0.01_step11' --use_cloudpickle 'False' --args_data 'gANjcmxsYWIubWlzYy5pbnN0cnVtZW50ClN0dWJNZXRob2RDYWxsCnEAKYFxAX1xAihYBgAAAF9fYXJnc3EDKGNybGxhYi5taXNjLmluc3RydW1lbnQKU3R1Yk9iamVjdApxBCmBcQV9cQYoWAQAAABhcmdzcQcpWAYAAABrd2FyZ3NxCH1xCShYAwAAAGVudnEKaAQpgXELfXEMKGgHKWgIfXENWAsAAAB3cmFwcGVkX2VudnEOaAQpgXEPfXEQKGgHKWgIfXERaApoBCmBcRJ9cRMoaAcpaAh9cRRYCwAAAHByb3h5X2NsYXNzcRVjbWFtbF9leGFtcGxlcy5wb2ludF9lbnZfcmFuZGdvYWwKUG9pbnRFbnZSYW5kR29hbApxFnVic2gVY3JsbGFiLmVudnMubm9ybWFsaXplZF9lbnYKTm9ybWFsaXplZEVudgpxF3Vic2gVY3NhbmRib3gucm9ja3kudGYuZW52cy5iYXNlClRmRW52CnEYdWJYBgAAAHBvbGljeXEZaAQpgXEafXEbKGgHKWgIfXEcKFgEAAAAbmFtZXEdaBlYCAAAAGVudl9zcGVjcR5jcmxsYWIubWlzYy5pbnN0cnVtZW50ClN0dWJBdHRyCnEfKYFxIH1xIShYBAAAAF9vYmpxImgLWAoAAABfYXR0cl9uYW1lcSNYBAAAAHNwZWNxJHViWA4AAABncmFkX3N0ZXBfc2l6ZXElRz/gAAAAAAAAWBMAAABoaWRkZW5fbm9ubGluZWFyaXR5cSZjdGVuc29yZmxvdy5weXRob24ub3BzLmdlbl9ubl9vcHMKcmVsdQpxJ1gMAAAAaGlkZGVuX3NpemVzcShLZEtkhnEpdWgVY3NhbmRib3gucm9ja3kudGYucG9saWNpZXMubWFtbF9taW5pbWFsX2dhdXNzX21scF9wb2xpY3kKTUFNTEdhdXNzaWFuTUxQUG9saWN5CnEqdWJYCAAAAGJhc2VsaW5lcStoBCmBcSx9cS0oaAcpaAh9cS5oHmgfKYFxL31xMChoImgLaCNoJHVic2gVY3JsbGFiLmJhc2VsaW5lcy5saW5lYXJfZmVhdHVyZV9iYXNlbGluZQpMaW5lYXJGZWF0dXJlQmFzZWxpbmUKcTF1YlgKAAAAYmF0Y2hfc2l6ZXEySxRYDwAAAG1heF9wYXRoX2xlbmd0aHEzS2RYDwAAAG1ldGFfYmF0Y2hfc2l6ZXE0SyhYEAAAAG51bV9ncmFkX3VwZGF0ZXNxNUsBWAUAAABuX2l0cnE2S2RYCAAAAHVzZV9tYW1scTeIWAkAAABzdGVwX3NpemVxOEc/hHrhR64Ue1gEAAAAcGxvdHE5iHVoFWNzYW5kYm94LnJvY2t5LnRmLmFsZ29zLm1hbWxfdHJwbwpNQU1MVFJQTwpxOnViWAUAAAB0cmFpbnE7KX1xPHRxPVgIAAAAX19rd2FyZ3NxPn1xP3ViLg=='
python3: can't open file '/home/gbenga/Documents/2019': [Errno 2] No such file or directory

[GUO99131]

same problem. Have u fixed? @GbengaOdesanmi