New Feature - Accumulator (Multiline logs with consistent IDs) by reyjrar · Pull Request #198 · ossec/ossec-hids

reyjrar · 2014-04-30T13:58:58Z

As detailed in this blog post OSSEC-HIDS Accumulator

Running in production for >12 months, maybe closer to two years.

Accumulator allows the collection of data across multiple log entries by decoding and grouping on an id field. The cache for the accumulator lasts: Default expire to 2 minutes (on lookup of exact key) Default purge count to 200 lookups (on lookup of any key) Default purge interval to 5 minutes (on lookup of any key) == hash_op.(c|h) * Change to design of the API, OSHash owns the key's memory, caller owns the data elements memory. * OSHash_Add() strdup's the key * OSHash_Free() now free()'s the key element * OSHash_Delete() added so hashes can be dynamic == accumulator.(c|h) * This allows log lines to accumulate details based on the previous log entries which share the same id element. Lines must contain, therefor the decoder must extract, the id element, or the accumulator simply returns without performing any action. It is important to note, this is not a multi-line reader, each log line is treated as it's own event. As more data is learned about each ID, the events are able to use that data. * Accumulate_Init() sets up the accumulator constants and initilizations variables * Accumulate() performs the accumulation of event data. The default expiry for an event ID is 5 minutes of inactivity. * Accumulate_CleanUp() is called once every 100 lookups or once every 10 minutes, whichever happens first. == etc/decoder.xml * Enabled accumulator plugin for the OpenLDAP Decoder. This allows for rules to be created with <same_source_ip/>, like with SSHD multiple login failures. Bug fixes included: * Accumulator working with in-memory structs * Testrule.c now is very flagrant about the accumulator usage * Fixed bug in Accumulator_CleanUp() which caused crashes

syscheck. The Eventinfo struct only included file data for builds with Prelude integration. This prevented the AR from handing filename off anyways. * Eventinfo now contains file data always * Added *expect* option for 'filename' * Added FILENAME to ar.h * Filename validation is tricky, so shell escape instead * Added os_shell_escape() to string_op.h, allocates memory for an escaped string and passes the pointer back to caller. Caller must cleanup that memory. * Call os_shell_escape() before passing to execd. * Added string_test.c to test the os_shell_escape() function.

This quick fix will add setenv function using the avaiable putenv call. This should be in a win32 shared section of the code, but I will take care of that as I clean more of the win32 build process up.

This should allow the user to specify a debug level for the remoted daemon using the remoted.debug option in the internal_options.conf. The debug level specified on the command line takes precedence.

…aming conflict to struct config in zlib

… global remoted struct logr

Conflicts: .gitignore src/Makeall src/win32/make.sh

Accumulator allows the collection of data across multiple log entries by decoding and grouping on an id field. The cache for the accumulator lasts: Default expire to 2 minutes (on lookup of exact key) Default purge count to 200 lookups (on lookup of any key) Default purge interval to 5 minutes (on lookup of any key) == hash_op.(c|h) * Change to design of the API, OSHash owns the key's memory, caller owns the data elements memory. * OSHash_Add() strdup's the key * OSHash_Free() now free()'s the key element * OSHash_Delete() added so hashes can be dynamic == accumulator.(c|h) * This allows log lines to accumulate details based on the previous log entries which share the same id element. Lines must contain, therefor the decoder must extract, the id element, or the accumulator simply returns without performing any action. It is important to note, this is not a multi-line reader, each log line is treated as it's own event. As more data is learned about each ID, the events are able to use that data. * Accumulate_Init() sets up the accumulator constants and initilizations variables * Accumulate() performs the accumulation of event data. The default expiry for an event ID is 5 minutes of inactivity. * Accumulate_CleanUp() is called once every 100 lookups or once every 10 minutes, whichever happens first. == etc/decoder.xml * Enabled accumulator plugin for the OpenLDAP Decoder. This allows for rules to be created with <same_source_ip/>, like with SSHD multiple login failures. Bug fixes included: * Accumulator working with in-memory structs * Testrule.c now is very flagrant about the accumulator usage * Fixed bug in Accumulator_CleanUp() which caused crashes

…nto feature/accumulator Conflicts: etc/decoder.xml src/analysisd/Makefile

jrossi · 2014-05-02T12:59:23Z

This is a huge patch and some I want to pull in and start get others test and playing with. I was wondering if you could get a document ion pull request started?

Also how do we test this I was wondering if you integrated it into logtest. I don't see it in my first go around of the code but I could easy have missed that.

Thank you again this is something ossec has needed for a while now and will solve a lot of problems for people.

ddpbsd · 2014-05-02T13:41:38Z

Is this something that can wait until after 2.8?

On Fri, May 2, 2014 at 8:59 AM, Jeremy Rossi notifications@github.comwrote:

This is a huge patch and some I want to pull in and start get others test
and playing with. I was wondering if you could get a document ion pull
request started?

Also how do we test this I was wondering if you integrated it into
logtest. I don't see it in my first go around of the code but I could easy
have missed that.

Thank you again this is something ossec has needed for a while now and
will solve a lot of problems for people.

Reply to this email directly or view it on GitHubhttps://github.com//pull/198#issuecomment-42028587
.

jrossi · 2014-05-02T17:46:36Z

We would not pull this into the 2.8 code base. This would pull into master for the next release 2.9

On May 2, 2014, at 9:41 AM, "ddpbsd" notifications@github.com wrote:

Is this something that can wait until after 2.8?

On Fri, May 2, 2014 at 8:59 AM, Jeremy Rossi notifications@github.comwrote:

This is a huge patch and some I want to pull in and start get others test
and playing with. I was wondering if you could get a document ion pull
request started?

Also how do we test this I was wondering if you integrated it into
logtest. I don't see it in my first go around of the code but I could easy
have missed that.

Thank you again this is something ossec has needed for a while now and
will solve a lot of problems for people.

Reply to this email directly or view it on GitHubhttps://github.com//pull/198#issuecomment-42028587
.

—
Reply to this email directly or view it on GitHub.

ddpbsd · 2014-05-02T17:48:57Z

Yes, but can that all wait until after 2.8 is released? Spending time on
HEAD probably means not spending time on testing 2.8.

On Fri, May 2, 2014 at 1:46 PM, Jeremy Rossi notifications@github.comwrote:

We would not pull this into the 2.8 code base. This would pull into master
for the next release 2.9

On May 2, 2014, at 9:41 AM, "ddpbsd" notifications@github.com wrote:

Is this something that can wait until after 2.8?

On Fri, May 2, 2014 at 8:59 AM, Jeremy Rossi notifications@github.comwrote:

This is a huge patch and some I want to pull in and start get others
test
and playing with. I was wondering if you could get a document ion pull
request started?

Also how do we test this I was wondering if you integrated it into
logtest. I don't see it in my first go around of the code but I could
easy
have missed that.

Thank you again this is something ossec has needed for a while now and
will solve a lot of problems for people.

Reply to this email directly or view it on GitHub<
https://github.com/ossec/ossec-hids/pull/198#issuecomment-42028587>
.

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHubhttps://github.com//pull/198#issuecomment-42058475
.

reyjrar · 2014-05-05T19:36:57Z

I'll start on the docs.. and yes, it's incorporated into logtest, because I would have lost my mind without an easy way to test. It also fixes issues with some of the core library's hash functions and includes tests for those changes as well.

2.9 is fine, meant to get this out the door sooner for 2.8, but failed to do so.

jrossi · 2014-05-26T14:20:51Z

Been running this for a while.....have not written and real rules for it yet, but thing it make sense to get into master soon. Anyone else running this and testing it yet? @ossec

jrossi · 2014-05-26T14:26:24Z

@reyjrar how are you coming with the ossec/ossec-docs pull request? I will pull this in once we docs ready to go.

ddpbsd · 2014-05-28T12:34:32Z

I can't upload them now, but I have added the documentation for this. I'll try to make a pull request tonight.

New Feature - Accumulator (Multiline logs with consistent IDs)

@jesusfidalgo

From Wazuh pull request ossec#198 by @jesusfidalgo

reyjrar and others added 30 commits February 13, 2014 23:13

Git ignore for files which we build

bb28c78

Ignore cJSON.h

5b72b81

more stuff with the travis keys and deploy

291a959

adding windows agent to the top level make file

724a562

Oops make agent requires root permissions

8d9bb3c

more clean up for make clean'

4e37090

Fixes to exec.c by Micha Nasriachi

c71f790

apply inode problem fix patch

47d9fdf

Convert csyslog JSON output to use the cJSON library.

a235f2b

Linking and libraries sorted out.

7a351d7

Something isn't building, guessing that it's the eventchannel stuff.

a79b659

Fully remove the event channel stuff.

704ed03

I guess this is stupid too.

23c34de

setenv is not avaiable on Win32

e55fc2e

This quick fix will add setenv function using the avaiable putenv call. This should be in a win32 shared section of the code, but I will take care of that as I clean more of the win32 build process up.

Merging in changes from @cgzones

909653b

merging in @cgzones geoip clean

aad6d9c

removing hg files

fd97eb0

Make remoted.debug in internal_options.conf work

54ba8aa

This should allow the user to specify a debug level for the remoted daemon using the remoted.debug option in the internal_options.conf. The debug level specified on the command line takes precedence.

adding more details to the README page

2c8a39a

clean up section and add build status

a22e28c

correct deploy to s3 so that we can test win32 agents.

f80594f

removing cond test for deploy

8a590b2

adding irc notifications

54ce153

Fixed merge conflicts with upstream master

74510d4

ignore eclipse project files

ac687d9

fix spelling

0faa070

remove static cJSON library on make clean

bc836a9

fix buffer overflow

c249e4a

cgzones and others added 16 commits February 24, 2014 22:18

fix missing returns reported by eclipse

2d9a27a

surround binary expression with parenthesis

7f8db51

fix missing breaks

03e1b25

remove unused declarations

04b5eed

rename syscheck config struct from config to syscheck_config due to n…

55cb955

…aming conflict to struct config in zlib

rename global agent struct from logr to agt due to naming conflict to…

6a0009c

… global remoted struct logr

fix cyclic header relationship mem_op.h <-> shared.h

4c172bd

exit on error during making zlib or cJSON

0aa172b

fix spelling preventing building geoip support

41a1b1e

fixing gcc -Wall warnings

42c132f

Merge branch 'master' of github.com:ossec/ossec-hids

0830624

Conflicts: .gitignore src/Makeall src/win32/make.sh

Merge branch 'feature/accumulator' of github.com:reyjrar/ossec-hids i…

223fd98

…nto feature/accumulator Conflicts: etc/decoder.xml src/analysisd/Makefile

Merge branch 'master' of github.com:ossec/ossec-hids

c36e880

Merge branch 'master' of github.com:ossec/ossec-hids

cf189f5

Merge branch 'master' into feature/accumulator

d5cc669

jrossi added the rn:master label May 30, 2014

jrossi self-assigned this May 30, 2014

jrossi added a commit that referenced this pull request Jun 5, 2014

Merge pull request #198 from reyjrar/feature/accumulator

d612d57

New Feature - Accumulator (Multiline logs with consistent IDs)

jrossi merged commit d612d57 into ossec:master Jun 5, 2014

ddpbsd added a commit to ddpbsd/ossec-hids that referenced this pull request Sep 27, 2017

Remove the P's entirely to support upper and lower case.

4f52525

From Wazuh pull request ossec#198 by @jesusfidalgo

ddpbsd mentioned this pull request Sep 27, 2017

Remove the P's entirely to support upper and lower case. #1262

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Feature - Accumulator (Multiline logs with consistent IDs)#198

New Feature - Accumulator (Multiline logs with consistent IDs)#198
jrossi merged 47 commits intoossec:masterfrom
reyjrar:feature/accumulator

reyjrar commented Apr 30, 2014

Uh oh!

jrossi commented May 2, 2014

Uh oh!

ddpbsd commented May 2, 2014

Uh oh!

jrossi commented May 2, 2014

Uh oh!

ddpbsd commented May 2, 2014

Uh oh!

reyjrar commented May 5, 2014

Uh oh!

jrossi commented May 26, 2014

Uh oh!

jrossi commented May 26, 2014

Uh oh!

ddpbsd commented May 28, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

reyjrar commented Apr 30, 2014

Uh oh!

jrossi commented May 2, 2014

Uh oh!

ddpbsd commented May 2, 2014

Uh oh!

jrossi commented May 2, 2014

Uh oh!

ddpbsd commented May 2, 2014

Uh oh!

reyjrar commented May 5, 2014

Uh oh!

jrossi commented May 26, 2014

Uh oh!

jrossi commented May 26, 2014

Uh oh!

ddpbsd commented May 28, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants