Performance Issue with SUB_INTERPRETER Mode Under High Concurrency

[Arrowa70]

I am using pemja as the Python script execution engine in a Conductor-based orchestration system. However, when the QPS is very high, meaning under high concurrency scenarios, the execution time for a Python script calculating the sum of squares (N=1000000) becomes extremely high. The time taken increases linearly over time, eventually reaching tens of seconds. I have tried both Python 3.8 and 3.11 versions, and also attempted switching ExecType to SUB_INTERPRETER, but the results did not vary much.
Additionally, through JVM monitoring, I observed that most threads in the thread pool are in a blocked state.
Additionally, in some of our actual business scripts (which include some batch parameter conversion logic without any I/O operations), the execution time can even reach up to hundreds of seconds. We have tried using Groovy, and the execution time for Groovy scripts is less than 1 second.
In theory, SUB_INTERPRETER should offer better performance, but in our use case, we did not observe any significant improvement. In some cases, there was even a performance degradation. Could you please clarify if there are any prerequisites for achieving performance improvements with SUB_INTERPRETER? Additionally, could you provide some best practice examples on how to configure and use pemja effectively in high concurrency scenarios?

[HuangXingBo]

I think the problem comes from GIL problemn before CPython3.12. You can try to use python3.12