Skip to content

Comments

Update #104 to new release#105

Open
vee1e wants to merge 1 commit intomandiant:mainfrom
vee1e:fix-parsing
Open

Update #104 to new release#105
vee1e wants to merge 1 commit intomandiant:mainfrom
vee1e:fix-parsing

Conversation

@vee1e
Copy link
Contributor

@vee1e vee1e commented Feb 15, 2026

No description provided.

@puffyCid
Copy link
Collaborator

@vee1e thanks for following up. The issue associated with a bookmark is created/exists and filtering will always be done even if --resume flag is not provided has been fixed. Thanks.

However, if i run the following for the first time and let the example binary run until complete:

unifiedlog_iterator.exe --resume -m log-archive -i ..\..\..\tests\test_data\system_logs_big_sur_private_enabled.logarchive -o out.jsonl

I get a out.jsonl file with a size of 526746796 bytes

if I run

unifiedlog_iterator.exe -m log-archive -i ..\..\..\tests\test_data\system_logs_big_sur_private_enabled.logarchive -o out2.jsonl

I get a out2.jsonl file with a size of 851943689bytes

unifiedlog_iterator.exe --resume -m log-archive -i ..\..\..\tests\test_data\system_logs_big_sur_private_enabled.logarchive -o out.jsonl
....
unifiedlog_iterator.exe -m log-archive -i ..\..\..\tests\test_data\system_logs_big_sur_private_enabled.logarchive -o out2.jsonl
..

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           2/20/2026  9:09 PM      526746796 out.jsonl
-a---           2/20/2026  9:09 PM      851943689 out2.jsonl

I would expect both to be same size?
Any idea why the sizes are different?

@vee1e
Copy link
Contributor Author

vee1e commented Feb 21, 2026

Wait now that look at the flags you're parsing it's the opposite issue: when your don't run with the resume flag it gives the log after the bookmark and when you don't you get the full json. I'm not sure what's the issue here then?

@vee1e
Copy link
Contributor Author

vee1e commented Feb 21, 2026

You said you ran it till complete, can I get the output of your bookmark file and the logarchive so I can debug? Thanks

// Update bookmark with max timestamp seen (not just filtered) to avoid re-scanning
if let Some(max_time) = max_seen_timestamp && let Ok(mut book) = bookmark.lock() {
book.update_timestamp(max_time);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be the cause of the issue in regards to the smaller output size?

The Unified Log daemon writes to multiple files at the same time. So you could have multiple files that could technically start before the end of another file.

Ex: The Unified Log daemon starts logging to the files 1.tracev3, 2.traceve3, 3.tracev3 starting on 2026-02-20.
It finishes logging to all 3 at 2026-02-21.

Since the library parses one file at a time. When it completes 1.tracev3 the most recent max_time is 2026-02-21.
When it gets to 2.tracev3 which starts at 2026-02-20, it ends up skipping the earlier entries.

@puffyCid
Copy link
Collaborator

You said you ran it till complete, can I get the output of your bookmark file and the logarchive so I can debug? Thanks

The logarchive is from the tests.zip for the library.

bookmark file

{
  "last_timestamp": 1.6502539691645522e+18,
  "processed_files": {},
  "boot_uuid": null,
  "last_updated": "2026-02-21T02:27:30.582194800+00:00",
  "source_id": "..\\..\\..\\tests\\test_data\\system_logs_big_sur_private_enabled.logarchive"
}

I would expect running with the option:

unifiedlog_iterator.exe --resume -m log-archive -i ..\..\..\tests\test_data\system_logs_big_sur_private_enabled.logarchive -o out.jsonl

And i do not interrupt with ctrl-c. I would expect the same output size as if i ran with:

unifiedlog_iterator.exe -m log-archive -i ..\..\..\tests\test_data\system_logs_big_sur_private_enabled.logarchive -o out2.jsonl

@vee1e
Copy link
Contributor Author

vee1e commented Feb 21, 2026

Got it, I'll work on a patch for this, thanks for catching this early. I assumed incrementing .tracev3 files should have sequential timestamps but I never actually tested the patch on anything but live mode. I'll get back to you with a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants