-
Notifications
You must be signed in to change notification settings - Fork 84
Description
Context
While adapting openfisca-paris-rh to the feat/transition-formula branch (PR #1368), two bugs were found when using as_of variables with a transition_formula that references period.last_month.
Both bugs surface in the same scenario: a transition_formula on variable X at period T calls individu('X', period.last_month) or individu('Y', period.last_month), where T-1 is before the base instant established by set_input.
Bug 1 — _reconstruct_at returns None before the base, causing a scalar comparison in _set_as_of
Root cause
_reconstruct_at returns None when target_instant < _as_of_base_instant (line 144 of holder.py).
When a formula result for a pre-base period is put in cache via put_in_cache → _set_as_of, the code does:
prev = self._reconstruct_at(instant) # returns None
changed = value != prev # EnumArray != None → scalar bool (0-d array)
idx = numpy.where(changed)[0] # ValueError: Calling nonzero on 0d arraysThis always crashes for Enum variables because EnumArray.__ne__(None) returns a scalar True rather than an element-wise boolean array.
Proposed fix — holder.py::_reconstruct_at
def _reconstruct_at(self, target_instant):
if self._as_of_base is None:
return None
if target_instant < self._as_of_base_instant:
# The initial state is the best estimate for any period before
# the base was established.
return self._as_of_base
# ... existing logicThis makes semantic sense: set_input establishes the earliest known state, which applies to all time before that point too.
Bug 2 — _calculate_transition runs the transition_formula at/before the base instant, causing infinite recursion
Root cause
_calculate_transition (simulation.py) checks _as_of_base is None to skip the formula when no input has been set, but does not check whether the requested period is at or before the base instant.
When a transition_formula for period T references period.last_month (T-1), and T-1 ≤ base_instant, the engine:
- Calls
calculate(var, T-1)→_calculate_transition - Base exists, period <
start_computation_periodguard not triggered - Runs the
transition_formulaforT-1→ which asks forT-2→ … → infinite recursion
SpiralError is not raised because each recursive call is for a different period.
Proposed fix — simulation.py::_calculate_transition
Insert a guard after the _as_of_base is None check:
# Period at or before the base instant: the set_input value is authoritative.
# Running the formula here would recurse infinitely into the past.
if (
holder._as_of_base is not None
and holder._as_of_base_instant is not None
and instant <= holder._as_of_base_instant
):
holder._as_of_transition_computed.add(instant)
result = holder.get_array(period)
return result if result is not None else holder.default_array()Semantics: set_input for period T0 means "the state at T0 is known; compute transitions only for T > T0."
Minimal reproducer
from openfisca_core.model_api import *
from openfisca_core.simulations import SimulationBuilder
from openfisca_core.taxbenefitsystems import TaxBenefitSystem
from openfisca_core.entities import Entity
import numpy as np
person = Entity('person', 'persons', '', '')
class MyEnum(Enum):
a = "A"
b = "B"
class my_enum_var(Variable):
value_type = Enum
possible_values = MyEnum
default_value = MyEnum.a
entity = person
definition_period = MONTH
as_of = True
# No transition_formula: value persists
class my_int_var(Variable):
value_type = int
entity = person
definition_period = MONTH
as_of = True
def transition_formula(p, period, params):
prev = p('my_int_var', period.last_month) # triggers Bug 2
prev_enum = p('my_enum_var', period.last_month) # triggers Bug 1
return np.ones(p.count, dtype=bool), prev + 1
tbs = TaxBenefitSystem([person])
tbs.add_variables(my_enum_var, my_int_var)
sb = SimulationBuilder()
sb.create_entities(tbs)
sim = sb.build_default_simulation(tbs, 2)
sim.set_input('my_enum_var', '2024-01', np.array([MyEnum.a, MyEnum.b]))
sim.set_input('my_int_var', '2024-01', np.array([0, 10]))
# Bug 2: infinite recursion when computing '2024-02'
# (transition_formula for '2024-02' asks for '2024-01' which asks for '2023-12' …)
result = sim.calculate('my_int_var', '2024-02')
print(result) # expected: [1, 11]Workaround (applied in openfisca-paris-rh)
Both bugs are currently patched via monkey-patches in the consumer project's __init__.py while waiting for upstream fixes:
from openfisca_core.holders.holder import Holder
from openfisca_core.simulations.simulation import Simulation
_orig_reconstruct_at = Holder._reconstruct_at
def _patched_reconstruct_at(self, target_instant):
if self._as_of_base is not None and target_instant < self._as_of_base_instant:
return self._as_of_base
return _orig_reconstruct_at(self, target_instant)
Holder._reconstruct_at = _patched_reconstruct_at
_orig_calculate_transition = Simulation._calculate_transition
def _patched_calculate_transition(self, variable, population, holder, period):
instant = period.start if variable.as_of == "start" else period.stop
if (
holder._as_of_base is not None
and holder._as_of_base_instant is not None
and instant <= holder._as_of_base_instant
):
holder._as_of_transition_computed.add(instant)
result = holder.get_array(period)
return result if result is not None else holder.default_array()
return _orig_calculate_transition(self, variable, population, holder, period)
Simulation._calculate_transition = _patched_calculate_transitionBoth fixes are confirmed to pass the full test suite of openfisca-paris-rh (16 tests).