DFReader: Create DFMetaData class to handle logger metadata

[shancock884]

This PR adds a DFMetaData class to the DFReader.m file, which is used to handle LogMessage XML meta-data for use by MAVExplorer and potentially other tools. It links to PR ArduPilot/MAVProxy#1337.

The class is set as the metadata attribute of DFReader class, and at present includes methods:

• reset - clear cached data, in case of new download, or change of vehicle type
• dot_mavproxy - copy of function in mp_util, to avoid dependency on MavProxy
• metadata_tree - return metadata map for use by other function, including cached version once loaded.
• print_help - print the help for specified log message
• get_description - return the description of the specified log message

Tested using the "logmessage help" command in MAVExplorer (with PR ArduPilot/MAVProxy#1337). For example:

MAV> logmessage help ORGN
Log Message: ORGN
Vehicle navigation origin or other notable position

TimeUS [μs] : Time since system startup
Type [instance] : Position type
Lat [deglatitude] : Position latitude
Lng [deglongitude] : Position longitude
Alt [m] : Position altitude

[peterbarker]

Have you checked how much extra RAM this consumes when you load an ArduPlane logfile? Is it simply roughly equal to the size of the metadata file, or does it blow out?

Also, what's the parse-time like on the file? "not noticable" is probably the answer, but a quick check with a pair of time.time() calls would be nice.

[peterbarker]

On Wed, 20 Mar 2024, Simon Hancock wrote: @shancock884 commented on this pull request. _______________________________________________________________________________________________________________________________________________________________________________________________________________ In DFReader.py: > + def dot_pymavlink(self, *args): + '''return a path to store mavproxy data''' + if 'HOME' not in os.environ: + dir = os.path.join(os.environ['LOCALAPPDATA'], '.pymavlink') + else: + dir = os.path.join(os.environ['HOME'], '.pymavlink') + if len(args) == 0: + return dir + return os.path.join(dir, *args) @peterbarker - It feels like this should be in mavutil or something for consistent use by all pymavlink functions. (for example one place that accesses .mavproxy specifies "~" - which looks very Linux specific!) Would you agree, and if so, shall I do so on this PR, or on a pre-PR, or a later PR?

I'd suggest keeping this self-contained initially and we can look at factorying out on a future PR.

[shancock884]

Have you checked how much extra RAM this consumes when you load an ArduPlane logfile? Is it simply roughly equal to the size of the metadata file, or does it blow out?

Also, what's the parse-time like on the file? "not noticable" is probably the answer, but a quick check with a pair of time.time() calls would be nice.

I still need to do the RAM/CPU time check, but I think all other comments are addressed.

[peterbarker]

LGTM

[peterbarker]

Probably don't need this compatability code. IIRC the "exists_ok" used below makes this stuff Py3 only anyway...

[peterbarker]

I've become rather partial to pathlib myself, but I do understand this is just swiped from MAVProxy.

[peterbarker]

I wonder if we really should be doing this rather than making the caller catch the exceptions...

[peterbarker]

We actually also produce .xz for these... they're about 75% the size of the .gz

[peterbarker]

Tested it. Merged,thanks!