Setting incorrect CPU affinity in network.py can cause CPU load to spike

[rodwsmith]

Bug Description

On some systems, the network.py script can set an incorrect CPU affinity (via the -A option to iperf3) -- typically to core 0 on devices that don't support NUMA. This can cause CPU load to spike -- for instance, this has been observed to spike to over 90% on a Socionext Developerbox with an ARM Cortex-A53 CPU using a 1 Gbps network test, whereas omitting the -A option to iperf3 in the script produces CPU loads of around 30-35%. On the Socionext system, the throughput dropped from 92% (without the -A option) to about 82% (with the incorrect CPU affinity set).

To Reproduce

1. Run network.py manually (for instance, "sudo /usr/lib/checkbox-provider-base/bin/network.py test -i {interface} -t iperf --iperf3 --scan-timeout 36 --fail-threshold 80 --cpu-load-fail-threshold 90 --runtime 90 --num_runs 4 --target {ipaddr}"
2. Monitor CPU load with top; or wait for the test to finish and look at the reported CPU load.

Environment

My own tests were done on a Socionext Developerbox running Ubuntu 23.04; however, Rick Wu (rickwu4444 on MM) reported similar issues on an imx8-based platform.

Relevant log output

No response

Additional context

No response

[rickwu666666]

This is my log for running network.py on imx8MP platform, and this is the command for iperf3 that network.py running iperf3 -V -b 2000M -c 10.102.88.220 -t 90 -i 1 -f m -P 1 -A 0

root@ubuntu:/snap/checkbox20/current/providers/checkbox-provider-base/bin# ./network.py test -i etheqos -t iperf --iperf3 --scan-timeout 3600 --fail-threshold 80 --cpu-load-fail-threshold 90 --runtime 90 --num_runs 1 --target 10.102.88.220
INFO:root:Testing etheqos against 10.102.88.220
INFO:root:Have successfully pinged 10.102.88.220 on etheqos
INFO:root:-------------------- Test Run Number 1 --------------------
INFO:root:Using 1 thread.
WARNING:root:WARNING: Could not find the NUMA node associated with etheqos!
WARNING:root:Setting the association to NUMA node 0, which may not be optimal!
INFO:root:NUMA node of etheqos is 0....
INFO:root:Connecting to port 5201 on server....
INFO:root:Avg Transfer speed: 938.2555555555556 Mb/s
INFO:root:93.83% of theoretical max 1000 Mb/s
INFO:root:Average CPU utilization: 97.4%
WARNING:root:The network test against 10.102.88.220 failed because:
ERROR:root:  CPU load: 97.4%
ERROR:root:  CPU load is above 90% maximum

output of numa node for ethernet

root@ubuntu:/home/iotuc# ls /sys/class/net/eth*/device/
/sys/class/net/etheqos/device/:
driver    driver_override  mdio_bus  modalias  net  of_node  power  ptp  subsystem  uevent

/sys/class/net/ethfec/device/:
driver    driver_override  mdio_bus  modalias  net  of_node  power  ptp  subsystem  uevent

output of lscpu

root@ubuntu:/home/iotuc# lscpu 
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           4
Model name:                      Cortex-A53
Stepping:                        r0p4
CPU max MHz:                     1200.0000
CPU min MHz:                     800.0000
BogoMIPS:                        16.00
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        512 KiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

[Hook25]

We have decided to clean up our issue tracker to systematize our approach to backlog grooming and planning. If this issue is still relevant to you, please reopen it and tag one of the maintainers or reach out on ~Checkbox on Mattermost. Going forward all issues that have received no activity for more than a year will be marked as 'stale' and closed after one week