Skip to content

feat/transition-formula: two bugs with as_of variables before the base instant #1369

@benjello

Description

@benjello

Context

While adapting openfisca-paris-rh to the feat/transition-formula branch (PR #1368), two bugs were found when using as_of variables with a transition_formula that references period.last_month.

Both bugs surface in the same scenario: a transition_formula on variable X at period T calls individu('X', period.last_month) or individu('Y', period.last_month), where T-1 is before the base instant established by set_input.


Bug 1 — _reconstruct_at returns None before the base, causing a scalar comparison in _set_as_of

Root cause

_reconstruct_at returns None when target_instant < _as_of_base_instant (line 144 of holder.py).

When a formula result for a pre-base period is put in cache via put_in_cache → _set_as_of, the code does:

prev = self._reconstruct_at(instant)   # returns None
changed = value != prev                 # EnumArray != None → scalar bool (0-d array)
idx = numpy.where(changed)[0]          # ValueError: Calling nonzero on 0d arrays

This always crashes for Enum variables because EnumArray.__ne__(None) returns a scalar True rather than an element-wise boolean array.

Proposed fix — holder.py::_reconstruct_at

def _reconstruct_at(self, target_instant):
    if self._as_of_base is None:
        return None
    if target_instant < self._as_of_base_instant:
        # The initial state is the best estimate for any period before
        # the base was established.
        return self._as_of_base
    # ... existing logic

This makes semantic sense: set_input establishes the earliest known state, which applies to all time before that point too.


Bug 2 — _calculate_transition runs the transition_formula at/before the base instant, causing infinite recursion

Root cause

_calculate_transition (simulation.py) checks _as_of_base is None to skip the formula when no input has been set, but does not check whether the requested period is at or before the base instant.

When a transition_formula for period T references period.last_month (T-1), and T-1 ≤ base_instant, the engine:

  1. Calls calculate(var, T-1)_calculate_transition
  2. Base exists, period < start_computation_period guard not triggered
  3. Runs the transition_formula for T-1 → which asks for T-2 → … → infinite recursion

SpiralError is not raised because each recursive call is for a different period.

Proposed fix — simulation.py::_calculate_transition

Insert a guard after the _as_of_base is None check:

# Period at or before the base instant: the set_input value is authoritative.
# Running the formula here would recurse infinitely into the past.
if (
    holder._as_of_base is not None
    and holder._as_of_base_instant is not None
    and instant <= holder._as_of_base_instant
):
    holder._as_of_transition_computed.add(instant)
    result = holder.get_array(period)
    return result if result is not None else holder.default_array()

Semantics: set_input for period T0 means "the state at T0 is known; compute transitions only for T > T0."


Minimal reproducer

from openfisca_core.model_api import *
from openfisca_core.simulations import SimulationBuilder
from openfisca_core.taxbenefitsystems import TaxBenefitSystem
from openfisca_core.entities import Entity
import numpy as np

person = Entity('person', 'persons', '', '')

class MyEnum(Enum):
    a = "A"
    b = "B"

class my_enum_var(Variable):
    value_type = Enum
    possible_values = MyEnum
    default_value = MyEnum.a
    entity = person
    definition_period = MONTH
    as_of = True
    # No transition_formula: value persists

class my_int_var(Variable):
    value_type = int
    entity = person
    definition_period = MONTH
    as_of = True

    def transition_formula(p, period, params):
        prev = p('my_int_var', period.last_month)   # triggers Bug 2
        prev_enum = p('my_enum_var', period.last_month)  # triggers Bug 1
        return np.ones(p.count, dtype=bool), prev + 1

tbs = TaxBenefitSystem([person])
tbs.add_variables(my_enum_var, my_int_var)

sb = SimulationBuilder()
sb.create_entities(tbs)
sim = sb.build_default_simulation(tbs, 2)
sim.set_input('my_enum_var', '2024-01', np.array([MyEnum.a, MyEnum.b]))
sim.set_input('my_int_var',  '2024-01', np.array([0, 10]))

# Bug 2: infinite recursion when computing '2024-02'
# (transition_formula for '2024-02' asks for '2024-01' which asks for '2023-12' …)
result = sim.calculate('my_int_var', '2024-02')
print(result)  # expected: [1, 11]

Workaround (applied in openfisca-paris-rh)

Both bugs are currently patched via monkey-patches in the consumer project's __init__.py while waiting for upstream fixes:

from openfisca_core.holders.holder import Holder
from openfisca_core.simulations.simulation import Simulation

_orig_reconstruct_at = Holder._reconstruct_at
def _patched_reconstruct_at(self, target_instant):
    if self._as_of_base is not None and target_instant < self._as_of_base_instant:
        return self._as_of_base
    return _orig_reconstruct_at(self, target_instant)
Holder._reconstruct_at = _patched_reconstruct_at

_orig_calculate_transition = Simulation._calculate_transition
def _patched_calculate_transition(self, variable, population, holder, period):
    instant = period.start if variable.as_of == "start" else period.stop
    if (
        holder._as_of_base is not None
        and holder._as_of_base_instant is not None
        and instant <= holder._as_of_base_instant
    ):
        holder._as_of_transition_computed.add(instant)
        result = holder.get_array(period)
        return result if result is not None else holder.default_array()
    return _orig_calculate_transition(self, variable, population, holder, period)
Simulation._calculate_transition = _patched_calculate_transition

Both fixes are confirmed to pass the full test suite of openfisca-paris-rh (16 tests).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions