Improvements to the circularities feature

[tsmbland]

The current approach for dealing with circular commodity flows involves the following additions to the regular investment algorithm:

• Optimise the order of investments within the set of commodities making up the circularity to minimise the number of reverse dependencies (i.e. scenarios where commodity A is solved before commodity B yet commodity B can consume commodity A). This way, the normal hierarchical approach to investments will work for most of the system, but unavoidably there will be at least commodity where it doesn't.
• Perform post-investment dispatch after each commodity with a (small) degree of freedom on the capacities of newly installed assets (the capacity_margin model parameter). This way we get most of the benefits of the agent-based approach (i.e. decision makers for each commodity are self-serving), but we allow some degree of whole-system optimisation to balance out potential supply/demand mismatches (the capacity_margin sets the trade off between agent optimisation and whole-system optimisation).

This seems to work at least some of the time with the default capacity_margin (0.2/20%). If not, you can often increase the capacity_margin to get it to work. However, it may be problematic to increase this too much because you might reaonably expect a large change in demands to qualitatively impact investment decisions, whereas the current "flexible capacities" approach only allows for quantitative changes in capacities as demands change. Furthermore, there are some scenarios where the "flexible capacities" approach won't work:

• if there are no newly installed assets (i.e. no assets to which capacity_margin can apply)
• or, if newly installed assets make up a very small portion of the total, then capacity_margin will have to be very high to have any impact
• if all newly installed assets derive their inputs from within the circularity. At some point the energy mismatch has to be resolved by new energy coming in from outside, it can't just keep going round the loop forever

Very interested to hear any ideas about how to address these, or other general suggestions.

Some ideas:

• iterate. If the capacity_margin feature cannot balance-out supply/demand mismatches, find the commodities that need more capacity and re-do the investments for these commodities. One downside of an iterative approach is that it could just keep going round and round with capacities spiraling out of control. If we add investment limits (Implement process investment constraints #1069) then this may not be be such a problem.
• give flexible capacities to pre-existing assets. Especially now with partial decommissioning as a feature, we should allow partial/complete decommissioning of pre-existing (retained) assets at the dispatch stage. Not sure this will help as this can only ever decrease capacities, but probably still worth doing.
• add any assets not selected as part of the appraisal as a "dummy" asset with zero capacity. To address the third point above, guarantees that there will always be a potential source of external energy. Could apply a penalty to this so it only ever installs new assets as a last resort. Not sure how this will work with capacity_margin as a margin on zero is zero, but we could:
• apply the capacity_limit as a fraction of the total capacity, not each individual asset capacity. E.g. if total capacity for the commodity is 50, then a capacity_margin of 0.2 means that all assets can change their capacities by +- 10 units. Would also address the second point above.
• the algorithm to order the commodities treats each edge equally, but we could weight each edge (i.e. a conversion from commodity A to commodity B) based on activity in the previous year, so that highly active edges are less likely to be solved out of order. This would require us to compute the ordering each year rather than pre-computing for all years at the start.

[alexdewar]

Reviving this discussion so we have a place to discuss circularities-related ideas and issues.

The main problem with models with circularities at present is that they don't work with the marginal and full pricing strategies (#1168), which is particularly acute as we are due to change the default pricing strategy to one of these before the next release (#1173; or similar: #1200).