Skip to content

Conversation

@jpn--
Copy link
Member

@jpn-- jpn-- commented Mar 27, 2024

We recently fixed a problem where sharrow was holding on to data references, sometimes preventing memory from being freed.

This introduced a bug in the sharrow-test error handler of the interaction simulate code, where sharrow re-runs data if it does not match utility values with the original non-sharrow code. This PR fixes the bug.

@jpn--
Copy link
Member Author

jpn-- commented Mar 27, 2024

@i-am-sijia this is the fix for the intermediate error we discussed yesterday.

@i-am-sijia
Copy link
Contributor

Thank you. This fixes the NoneType error, but Sharrow is still crashing at the non_mandatory_tour_scheduling.vectorize_tour_scheduling.tour_6.interaction_sample_simulate.eval_interaction_utils, due to the difference in utilities. See attached logs.

activitysim_nonetype_error.log
activitysim.log

You mentioned Sharrow can find and print out the utility expression(s) that have the biggest value difference. It will be helpful in this case. What setting should I use?

I did not get this error when running the full-scale benchmarking run with Sharrow. So this might just be happening with the small test example. Should I open a separate issue?

@i-am-sijia
Copy link
Contributor

Would this PR have any memory implication?

@jpn--
Copy link
Member Author

jpn-- commented Apr 2, 2024

You mentioned Sharrow can find and print out the utility expression(s) that have the biggest value difference. It will be helpful in this case. What setting should I use?

You just run in debug mode setting a breakpoint here:

raise # enter debugger now to see what's up

and the variable look_for_problems_here should point the way.

@jpn--
Copy link
Member Author

jpn-- commented Apr 2, 2024

Would this PR have any memory implication?

There are no implications for memory usage unless sharrow and non-sharrow disagree on what the utility values are. In that case, sharrow does gobble up a bunch of memory to write out all the expression values into memory so you can peruse them in the debugger, but this is obviously desirable in this situation.

@i-am-sijia
Copy link
Contributor

Would this PR have any memory implication?

There are no implications for memory usage unless sharrow and non-sharrow disagree on what the utility values are. In that case, sharrow does gobble up a bunch of memory to write out all the expression values into memory so you can peruse them in the debugger, but this is obviously desirable in this situation.

That makes sense. I believe the utility comparison only happens in the sharrow compile mode. This change should not impact if the user is running sharrow production mode, which is when we are usually concerned about memory.

@jpn-- jpn-- merged commit bb437b3 into ActivitySim:main Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants