Pushing merged.mg to Windows agents fails due to EOL conversion

[SvenKiljan]

Most likely related to:
https://groups.google.com/forum/#!topic/ossec-list/OsUrHR4ykuo

This bug has been observed in OSSEC 2.8.3 and 2.9.0. 2.9.1 is not available for Windows yet, but looking at the changelog it is most likely this bug still exists.

Situation:
An OSSEC server combines all files in /var/ossec/etc/shared into a single file known as merged.mg. Since the server uses Linux, all lines in merged.mg end with a line feed (LF, hex code 0A).

merged.mg is pushed to agents when they connect. A Windows agent receives and reconstructs the file (send over UDP 1514) and an MD5 hash, and verifies the MD5 hash of the file to see whether the transfer was successful.

Expected outcome:
Assuming the transfer went without problems, the Windows agent reconstructs the file as it was on the server to allow proper verification using the MD5 hash that was also received. Therefore, line feed will be used as an end-of-line character. The file is reconstructed exactly as it was on the server, and the MD5 sum verifies.

Actual outcome:
Assuming the transfer went without problems, the Windows agent reconstructs the file using carriage return (CR, hex code 0D) and a line feed (LF, hex code 0A) as newline characters. The result is that the MD5 verification will always fail, since the MD5 hash generated by the Linux server (using LF newlines) will never match that of the MD5 generated by the Windows agent (using CRLF newlines).

Suggestion:
Change the verification process of the agent.

1. Receive the data and the MD5 sum.
2. Reconstruct the file as-is. Do not do any conversion. If this is not possible (due to the way the server sends files), reconstruct the file using all possible EOL combinations and verify all variations in the next step.
3. Verify the MD5 hash.
4. When the MD5 hash matches, do an EOL conversion to the target platform of the agent. For Windows, this would be a conversion to CRLF.

[nbuuck]

Confirmed that MD5 is failing consistently for Windows agents. Set windows.debug to 1 in local_internal_options.conf of a Windows agent, restarted the agent, and observed:

2017/07/22 19:29:59 ossec-agentd: Failed md5 for: shared/merged.mg -- deleting.

[nbuuck]

I tried to verify this by reviewing the code but between the project structure (receiver.c versus receiver-win.c), build automation, and minimal code comments, I'll likely have to review in a debugger to verify.

[SvenKiljan]

Steps to verify the bug without using a debugger on the agent (assuming a default installation on a x86-64 system):

1. Stop the OSSEC HIDS service.
2. Reconfigure the OSSEC HIDS service to run as a user with administrator rights.
3. Remove any files in C:\Program Files (x86)\ossec-agent\shared.
4. Reconfigure the NTFS permissions of C:\Program Files (x86)\ossec-agent\shared. Deny the user account used by the OSSEC HIDS service the advanced permissions to 'Delete subfolders and files' and 'Delete'.
5. Start the OSSEC HIDS service.

C:\Program Files (x86)\ossec-agent\shared\merged.md will be written and verified (which will fail due to the bug). It will not be deleted by OSSEC HIDS since the NTFS permissions prevent this.

Open merged.md from the agent in a hex editor. Open merged.md from the server in a hex editor (or let xxd parse it). See the different line ending characters.

[SvenKiljan]

I confirm that the bug is still present in 2.9.1.

[SvenKiljan]

Made a pull request to fix this issue.

A 'problem' with C functions that work with files is that they do so intelligently by default. fopen() opens files in text mode by default. OSSEC uses this function to save received configuration files, which is problematic since under Windows lines will be written with CRLF as line ends, something that is undesirable when the file has to be verified against an MD5 hash that is based on a file with LF as line ends. This is avoided in the pull request by adding the binary mode parameter to fopen(). Files will be saved with the line ends the server sends, and hash verification will succeed.

[nbuuck]

I tested #1207 on a Windows Server 2012 R2 agent, rebuilt with the patch using https://ossec.github.io/docs/manual/installation/compile-ossec-mingw.html. The agent successfully retrieved the shared config and a subsequent test file later added to /etc/shared on the OSSEC server. Thanks for the fix, @Quantumheart. Can this issue be closed?

[SvenKiljan]

Excellent!

@ddpbsd Thank you for the rapid merge.

@nbuuck Thank you for the verification.

I will close this issue.

[billy-shears]

Into which release will this change be included?
It has been merged in in July 2017, but neither 2.9.2 nor 2.9.3 releases have it (according to reference source code supplied - e.g. see https://github.com/ossec/ossec-hids/archive/2.9.3.zip, released in Dec 2017)

[ddpbsd]

@billy-shears It must have been missed when 2.9.3 was being released. We'll try to correct that soon. Hopefully in the future these types of things can be caught before the release is finalized.

[billy-shears]

@ddpbsd, thanks a lot for the instant response!