Setting timing manager event ratio to 1 increases Run Time significantly in some cases

[Known4225]

Description

At control task rates above 50kHz and PWM frequency rates close to that of the control task (such that a timing manager event ratio is set to 1), the run-time of the control task is significantly higher than it should be.

Details

There's some sort of issue with the timing manager (or perhaps statistics reporting) that causes the runtime of a task to increase significantly with certain timing manager + pwm frequency configurations.

This was tested on the controller task from the VSI tutorial.

The runtime for this task should be around 3.5us, but in some cases it is much higher (as demonstrated in this PR thread)

I did additional testing with other PWM frequencies to see where this bug occurs (with an event ratio of 1, and TASK_UPDATES_PER_SEC set to 10kHz).

Tested PWM frequencies:
250kHz - Run Mean is 22.8us
110kHz - Run Mean is 16.6us
105kHz - Run Mean is 10.2us
100kHz - Run Mean is 10.1us
95kHz - Run Mean is 10.1us
94kHz - Run Mean is 10.1us
93kHz - Run Mean is 8.4us
92kHz - Run Mean is 3.5us
91kHz - Run Mean is 3.5us
50kHz - Run Mean is 3.5us

Additional testing has confirmed that if PWM Frequency / Timing manager event ratio > 93kHz, then this bug occurs, and the severity of it appears to be somewhat linear with the PWM Frequency. (That is, having 250kHz PWM and 2 as timing manager event ratio yields a Run Mean of 22.8us still, because 250 / 2 > 93, and the timing manager event ratio has no effect on the linear scaling of this bug).

Actually, it is more like 22.4us with an event ratio of 2 (and 250kHz). Not sure what to make of any of this information.

The bug does not appear to be caused by calling analog_getf(ANALOG_IN5, &LOG_current_a); inside the control loop code. Removing the calls to analog_getf did not have impact on the severity of the bug (though it did change the runtime, but not in a way that fixed the bug at least at high PWM frequencies, did not do full testing at lower PWM frequencies).

[codecubepi]

Hmm some of these operating points with higher PWM frequencies scream scheduler overrun to me. One thing to note is that dividing the PWM frequency by the Timing Manager ratio will give you the "control frequency" or "control rate". I.e. how often the Timing Manager generates the interrupt which tells the scheduler to start running tasks.

In "default operation" the AMDC uses 100 kHz for the PWM frequency, and a ratio of 10, meaning control tasks are scheduled to be run at a rate of 10 kHz, or a control period of 100 usec. Therefore the sum runtime of all control tasks must be less than 100 usec, or they will not all complete before the next scheduler interrupt is generated.

In your case, you've set the event ratio to 1, meaning PWM freq = control rate frequency. This is a very atypical operating condition, but still worth digging into 🙂.

I really don't expect the 250 kHz operating condition to work very well, as that is a control period of 4 usec for a task that looks like it takes 3.5 usec. But the 100 kHz case I feel should probably still work... you are running your 3.5 usec task in a 10 usec control period. I think it makes sense to start here (or at your 93 kHz frequency where you begin to observe the issue, but it's unlikely that anyone would pick 93 kHz as their PWM freq).

One thing that may help me advise your further is if you could also provide the loop time you are seeing for your task. Also, it is always good to be aware what loop time your are requesting the system run your app at to make sure it matches. This is usually defined in the app's task header file (task_cntrl.h or similar). Because not all tasks are necessarily run every control period.

[elsevers]

Thanks @codecubepi, @Known4225.

We would like to be able to operate this with control rates in the 100 kHz range which would imply event ratios of 1.

Nathan actually did a fun demo with a control rate of 200 kHz (PWM of 1 MHz) in this paper. Would that combination of frequencies (event ratio of 5) have this problem?

We also should support Si IGBTs, which typically have switching frequencies between 5 kHz and 20 kHz, and so these would be apt to result in event ratios of 1. I see from your notes above, that this is likely okay, but can you confirm?

@Known4225, could you do some more debugging on this?

@codecubepi, do you have suggestions on what he can check to identify the root cause of this issue? (@Known4225 says he'll check the loop time)

[Known4225]

I've compiled about 100 data points changing the PWM frequency and the TASK_UPDATES_PER_SEC. All of these experiements used a timing manager event ratio of 1. The results are strange (formatting is run-time/loop-time).

https://docs.google.com/spreadsheets/d/13YE25-4grKAyvpcY-a8ns06K56rwdKYc6dkgXZXi2XQ/edit?usp=sharing

Also I did a separate test with 1000kHz PWM, the control task at 200kHz, and a timing manager ratio of 5. The result is a run-time of 10.08us and a loop-time of 22.00us (it should be loop-time of 5us, so it isn't working).

I found that the bug happens at PWM frequencies of 70 and 80kHz (with a TASK_UPDATES_PER_SEC of 10kHz), but not at 90kHz. So I really don't know what's going on or what's causing this.