DFReader Speedup

[robertlong13]

I frequently work with ~2GB logs, and it's nice to be able to use mavutil to knock together quick analysis scripts, but it can take on the order of 30 seconds just to load the log before I can even start calling recv_msg or recv_match.

This PR is mainly a rewrite of init_arrays in Cython for about a 10X speedup of the initial load.

[peterbarker]

Ping @stephendade - any chance you could test this on Windows?

[peterbarker]

pbarker@crun:~/rc/pymavlink((HEAD detached at robertlong13/pr/dfreader-speedup))$ mavlogdump.py $DFLOG
Traceback (most recent call last):
  File "/home/pbarker/venv-ardupilot/lib/python3.12/site-packages/pymavlink/DFReader.py", line 28, in <module>
    from . import dfindexer
ImportError: cannot import name 'dfindexer' from 'pymavlink' (/home/pbarker/venv-ardupilot/lib/python3.12/site-packages/pymavlink/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pbarker/venv-ardupilot/bin/mavlogdump.py", line 82, in <module>
    mlog = mavutil.mavlink_connection(filename, planner_format=args.planner,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pbarker/venv-ardupilot/lib/python3.12/site-packages/pymavlink/mavutil.py", line 1894, in mavlink_connection
    from pymavlink import DFReader
  File "/home/pbarker/venv-ardupilot/lib/python3.12/site-packages/pymavlink/DFReader.py", line 32, in <module>
    from pymavlink import dfindexer
ImportError: cannot import name 'dfindexer' from 'pymavlink' (/home/pbarker/venv-ardupilot/lib/python3.12/site-packages/pymavlink/__init__.py)
pbarker@crun:~/rc/pymavlink((HEAD detached at robertlong13/pr/dfreader-speedup))$ 

[robertlong13]

Need to set PYTHONPATH. I think it's trying to grab dfindexer from your pip installed pymavlink.

[stephendade]

Got the same error message as Peter on Windows. Used pip install -U . to install. Using Win11/Python 3.11

I think it's not building the c extensions? Hence not finding the dfindexer module

[robertlong13]

Got the same error message as Peter on Windows. Used pip install -U . to install. Using Win11/Python 3.11

I think it's not building the c extensions? Hence not finding the dfindexer module

I had only tested with setup.py. I have fixed it now so that it works correctly through pip install. However, on Windows, I've made it not install the Cython module by default (to avoid having users need to go install Visual C build tools). Set an environment variable PYMAVLINK_FAST_INDEX=1 before running the pip install. I'll need to open a PR in MAVProxy to set that flag when it builds MAVExplorer.exe in CI.

[robertlong13]

Unfortunately, due to me not fully understanding the messages dictionary, I broke it. MAVExplorer relies on that for tab complete. I can very easily fix 90% of it, but I just realized that it tracks messages and their indices, which is a harder problem to solve (solvable, still). Hopefully I'll have something for that in the next day or so.

Edit: eh, I got it faster than I expected.

[tridge]

@robertlong13 really nice work!
an environment variable PYMAVLINK_DISABLE_FAST_INDEX would be nice for testing

[robertlong13]

@robertlong13 really nice work! an environment variable PYMAVLINK_DISABLE_FAST_INDEX would be nice for testing

Done, though I made it PYMAVLINK_FAST_INDEX to 0 to disable (instead of PYMAVLINK_DISABLE_FAST_INDEX to 1).

[robertlong13]

I have tested MAVExplorer on Windows too, using the github CI, so I'm ready with a MAVProxy PR as soon as this PR makes its way into PyPI. Only the environment variable portion of the following commit will be needed for the MAVProxy build, the rest is temporary checking out and building pymavlink so I can test. robertlong13/MAVProxy@ffe2cf2

[tridge]

@robertlong13 great work Bob!