[Bug Report] SIEM Logger sometimes failing to start

[soda-pop-ice-cream]

Describe the bug:

First of all, I want to thank you for creating opensnitch, currently I using it on all desktops running linux, and the only problem i have so far is that SIEM logger is quite unreliable, It seems that if the daemon is unable to connect to the log collector's syslog port and never tries again. I'm using it with Splunk and any other services logging their stuff perfectly fine and never miss a thing.

Also

My logger configuration:

sudo cat /etc/opensnitchd/default-config.json 
{
    "Server": {
        "Loggers": [
            {
                "Name": "remote_syslog",
                "Server": "192.168.1.142:51452",
                "Protocol": "udp",
                "Format": "rfc3164",
                "Tag": "opensnitchd"
            }
        ],

Include the following information:

• OpenSnitch version: v1.7.0-rc.2
• OS: Linux Mint
• OS version: 22.1 (e.g. ubuntu noble)
• Window Manager: Cinnamon
• Kernel version: 6.11.0-25

To Reproduce:

Steps to reproduce the behavior:

1. Reboot PC, opensnitch daemon not sending logs to SIEM
2. Wait a few minutes, then restart the daemon. Sometimes it will send the logs, and sometimes it won't. If it doesn't, restart it again.

(Usually, reboots cause this issue, but sometimes the daemon simply stops sending events to the logger that is always perfectly accessible over the network)

Post error logs:

 INF  NewRemoteSyslog logger: {remote_syslog rfc3164 udp 192.168.1.142:51452   opensnitchd 0}
 INF  NewRemote logger: {remote_syslog rfc3164 udp 192.168.1.142:51452   opensnitchd 0}
 ERR  Error loading logger [remote]: dial udp 192.168.1.142:51452: connect: network is unreachable

Additional context:

I've seen that there is some retries mention in this commit, but it's not documented on wiki how to configure daemon to make retries work(or they enabled by default?).

[gustavo-iniguez-goya]

Hi @soda-pop-ice-cream !

Right now there're no reconnection attempts if the remote server is not listening on a port. But if we connect on the first attempt, and there're write errors while sending the events, the daemon will try to restablish the connection up to 300 times until it gives up.

There're 2 (not documented) configuration items to control the write and connection timeouts:

 "ConnectTimeout": "5s",
 "WriteTimeout": "3s",

We should try to reconnect even if the remote server is not ready. Many times, we'll run before all these services.

We could add "ConnectRetries". 0 indefinitely, with the intervals controlled by the above config items. Or > 0 , to limit the amount connection attempts.

[soda-pop-ice-cream]

We could add "ConnectRetries". 0 indefinitely, with the intervals controlled by the above config items. Or > 0 , to limit the amount connection attempts.

That sounds great, would be glad to test build with these changes 👍️

[gustavo-iniguez-goya]

hey @soda-pop-ice-cream , I think this issue is solved. It'd be fantastic if you could test it.
I've tested it with rsyslog and seems to work as expected:

• When the daemon starts it'll try to connect up to MaxConnectAttempts to the remote server. If MaxConnectAttempts is not specified or if it's 0, it'll try it indefinitely.
• If there're too much errors writing to the remote server, the demon will close the connection and try to reestablish it again.
• If the user changes the configuration of the logger(s), they'll be reloaded without having to restart the daemon.