Modification to Correct IP Connectivity Issues on BSD Servers#1412
Modification to Correct IP Connectivity Issues on BSD Servers#1412ddpbsd merged 5 commits intoossec:masterfrom davestoddard:ip-multiplex-mods
Conversation
I have encountered a significant issue in OSSEC 2.9.3 and the current OSSEC beta (GitHub repository) release that affects the ability of OSSEC clients to be able to connect to FreeBSD servers. This issue also probably affects all current BSD derivatives and possibly some Linux systems as well. I have identified and resolved the issue in OSSEC by rewriting some of the networking code, however, before I describe the solution, I wanted to provide some background information on the problem first. Problem Symptoms The server is running FreeBSD 11.1 with all of the current updates. This is a standard OSSEC server installation with all standard options enabled and no bells and whistles (no MySQL and no MQ). After correcting for some BSD-specific issues for header files and libraries, OSSEC installs normally on the server with the install.sh script and the compilation completes without any fatal errors (GCC does generate a few warnings on analysisd/decoders/syscheck.c and analysisd/decoders/syscheck-test.c, but these are not show stoppers). The server has an assigned IP address of 192.168.1.80 (IPv4 private IP space) and all of the clients it associates with are on the local 192.168.1.0/24 network. The OSSEC 2.9.3 client software is installed on each Windows workstation and valid client keys are obtained for each client system. Everything appears fine, except that there are no alerts logged on the server for any of the configured clients. In short, the problem manifests itself as an inability for the client systems to connect to the OSSEC server. In the ossec.log file on each client there are messages like this that repeat over and over: 2018/03/08 12:11:06 ossec-agentd: INFO: Trying to connect to server 192.168.1.80, port 1514. 2018/03/08 12:11:06 INFO: Connected to 192.168.1.80 at address 192.168.1.80:1514, port 1514 2018/03/08 12:11:27 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: '192.168.1.80'. 2018/03/08 12:12:41 ossec-agentd: INFO: Trying to connect to server 192.168.1.80, port 1514. 2018/03/08 12:12:41 INFO: Connected to 192.168.1.80 at address 192.168.1.80:1514, port 1514 2018/03/08 12:13:02 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: '192.168.1.80'. Because the clients use UDP, and UDP is a connectionless protocol, there is no protocol handshake to ensure a connection has actually been established like there would be with TCP. As long as the sendto() function returns a positive integer (the number of bytes sent), the call is deemed successful. This is true even when packets are not received on the destination host. To compensate for this, the server is supposed to return a control message to indicate the packet was received. Because the client never receives this confirmation, a warning is generated in the client log "Waiting for server reply (not started)". On the server, remoted starts fine. It displays the maximum number of agents allowed (2048) and logs the fact that it successfully read the authentication keys file. However, the server logs hint that there is an issue when remoted starts because each client in the client.keys file results in a message stating "Assigning counter for agent Workstation01: '0:0'", or "Assigning sender counter: 0:0". These messages are generated when a client is defined in the etc/client.keys file, but no initial connection has ever occurred. This is normal for a first time start of OSSEC, but if this happens every time you restart OSSEC for every client defined in the system, your clients are not communicating. To identify whether remoted was listening on UDP port 1514, I used the "netstat -an" command and the FreeBSD "sockstat" command (sockstat is like "lsof" for sockets). The only address bound to port 1514 was an IPv6 address, not the IPv4 address. The server uses only a single IPv4 address for network communication, and no IPv6 addresses. However, it turns out that FreeBSD (and many other systems) create a link-local IPv6 address based on the MAC address of each interface when IPv6 is not specifically configured for the interface. This is specified in RFC 2462. The standard solution for this would be to specify an IP address for the "secure" connection in the remote configuration using the <local_ip> option. However, when that is used, OSSEC ignores it and the system continues to bind to the IPv6 link-local address. While I could have experimented with disabling IPv6 in the operating system, this could have an adverse impact on other applications running on the server. Normally, the system should simply query all available addresses on the system using getaddrinfo(), create a socket for each address, and bind the address to the socket. If the protocol is TCP, then you also need to use listen() to establish a queue. Then select() can be used to handle the dispatch of the individual sockets as messages arrive. I have a lot of experience with writing network interfaces in C, so I decided I would take a look at what was going on inside OSSEC that was creating this issue for me. Initial Analysis The OSSEC remoted process is responsible for handling the connections between the clients and the server. All of this code, except for the actual networking code, is found in the src/remoted directory. The initial connections for the UDP secure server and the syslog server are initiated in the HandleRemote() function inside remoted.c through calls to OS_Bindportudp() or OS_Bindporttcp() which are found in os_net/os_net.c. What is immediately obvious is that only one socket is returned as an integer value, instead of a group of sockets (one for each interface and protocol family). The actual processing of incoming messages is divided between secure.c for secure UDP connections, syslog.c for syslog UDP connections, and syslogtcp.c for syslog TCP connections. Most of the outgoing connections are handled by send_msg.c (with some exceptions). Because there is only one socket used per communication strategy (secure UDP, syslog UDP, and syslog TCP), all of the routines use a while(1) loop with blocking reads to process incoming messages. To handle multiple sockets without dragging the CPU into the mud, these would have to be modified to use select(). The next thing I looked at was the code that was doing the network calls, which is found in the file os_net/os_net.c. Both the OS_Bindportudp() and OS_Bindporttcp() calls referred to a function called OS_Bindport() to handle all of the network initialization code including querying the available addresses, creating the socket, binding the address, invoking listen() for the queue (TCP only), and returning the active socket to the caller. The code in os_net/os_net.c clearly showed my dilemma; the call to getaddrinfo() was setup to use AF_INET6 for the address family and AI_V4MAPPED, AI_PASSIVE, and AI_ADDRCONFIG for flags. Under this specification, IPv6 is preferred over IPv4, and if no IPv6 addresses exist and IPv4 addresses do exist, the IPv4 address is to be returned as a mapped IPv4 address using IPv6. A better solution is to use PF_UNSPEC for the address family with AI_PASSIVE and AI_ADDRCONFIG to pick up all addressed on configured interfaces. The original author of this module recognized this as a problem early on, but s/he assumed that if they had AI_V4MAPPED defined in netdb.h, then the system would handle it properly. However, if IPv4 is preferred and there are any IPv6 addresses configured on the system, AI_V4MAPPED will not yield any IPv4 addresses. There is a good article by IBM on this that you can read here: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.hale001/ipv6d0131000748.htm This is further compounded because OSSEC is returning only a single socket bound to a single address making the process of supporting both IPv4 and IPv6 at the same time is impossible. The only correct solution is to modify OSSEC to handle multiple IP addresses under both IPv4 and IPv6 simultaneously. The Solution The overall solution to this problem was to return all of the addresses that meet the selection criteria and use the select() function to return sockets that are ready for reading. While this change affected a number of modules in the OSSEC system, the change was relatively straight forward to implement. A list of modules affected is provided at the end of this document. Under the current version of OSSEC, a function named OS_Bindport() is used to locate the first address that is returned from getaddrinfo(). Then a socket for the appropriate connection type is created (SOCK_DGRAM for UDP or SOCK_STREAM for TCP), and the address is bound to the socket. If the connection is a TCP connection, a queue is established for the bound socket using listen(). Then the integer value socket is returned. If the socket value is -1, an error is assumed and various error actions are taken by the caller. Under the modified version, OS_Bindport() was modified in a number of ways. First, instead of returning an integer value representing a bound socket to the caller, the routine now returns a pointer to an OSNetInfo structure. The OSNetInfo structure is defined in os_net/os_net.h as follows: typedef struct _OSNetInfo { fd_set fdset; /* set of sockets used by select ()*/ int fdmax; /* fd max socket for select() */ int fds[FD_SETSIZE]; /* array of bound sockets for send() */ int fdcnt; /* number of sockets in array */ int status; /* return status (-1 is error) */ int retval; /* return value (additional info) */ } OSNetInfo; In order to multiplex multiple sockets efficiently, we need to use the select() function. The select() function operates on a struct called fd_set that is built on the successful acquisition of a socket with a bound address. Each successful acquisition is added to the set using the FD_SET macro. We also track the highest value socket number returned in fdmax. When all processing is completed, fdmax is incremented by 1 so it can be used directly with the select() function in other modules. We also track the individual bound sockets in an integer table called fds, and a count of the number of sockets in the table using fdcnt. Because the original implementation relied upon a return value of -1 to indicate an error, we include an element in the struct called status that performs the same function. If status is -1, then an error has occurred. Upon a successful return, the status element will be 0. If status is -1, the return value element retval will contain the error number associated with the error status. The OSNetInfo struct made the process of modifying the OSSEC networking code relatively simple because it preserved all of the functionality that was inherent with the original code. In addition to the OSNetInfo structure as a return value, the second major change to os_net/os_net.c was the definition of hints to the getaddrinfo() function. The getaddrinfo() function was created by the POSIX standards committee to provide support both IPv4 and IPv6 simultaneously. The getaddrinfo() function modifies a struct that returns a pointer to a linked list of addresses that match the selection criteria passed to getaddrinfo() through the hints parameter struct. Under the original OSSEC code, if the operating system platform had AI_V4MAPPED defined in <netdb.h> (the vast majority of modern OS's support this), then the hints structure passed to getaddrinfo() used AF_INET6 for the protocol family. This essentially guaranteed that only IPv6 addresses would be returned if they were defined on the system. Even if there are no IPv6 addresses configured on the system, the AF_INET6 protocol family with the AI_V4MAPPED flag will return a mapped IPv4 address in IPv6 format. For current *BSD systems, if an IPv6 address is not defined on an interface, the system will create a link-local IPv6 address based on the MAC address of each interface. IPv6 link-local addresses are essentially useless for external communication, so the net result is that *BSD systems (and others) are not able to communicate using IPv4. By modifying this to use AF_UNSPEC instead of AF_INET6, eliminating the AI_V4MAPPED flag, and utilizing all of the addresses returned by the call to getaddrinfo(), we have the ability to communicate through all of the interfaces defined on the system using both IPv4 nd IPv6. Nonetheless, there are a number of Linux systems that are working fine for OSSEC with the AF_INET6 family and the AI_V4MAPPED flag. As such, an "#if defined" definition was placed in the code to support the original behavior on Linux systems, with the PF_UNSPEC behavior defined for non-Linux systems. In the event you do want to use the PF_UNSPEC code with a Linux system (I suspect this will become the norm as more versions of Linux adopt RFC 2462 and the link-local address), all you need to do is to uncomment the line in the Makefile that defines NOV4MAP and the system will use AF_UNSPEC instead of AF_INET6. Once the call is made to getaddrinfo(), we utilize a loop to process each of the addresses returned in the linked list of addresses. Each socket that is bound successfully is added to the fd_set for subsequent select() processing, and also added to an array of integers containing bound sockets. The only other substantive change that was made to this module is that we set a socket option for the SO_REUSEADDR flag. In the event OSSEC is stopped and restarted again, the use of the SO_REUSEADDR socket option allows OSSEC to restart immediately without having to wait for a lingering socket to close. It also improves performance for short duration transactions. The other changes in the os_net/os_net.c module include changing the return parameter from integer to a pointer to an OSNetInfo struct for OS_Bindportudp() and OS_Bindporttcp(), and a number of helper functions were added to the bottom of the module to provide additional information as the addresses are retrieved, sockets created, and addresses are bound to sockets. Once the changes were made to os_net/os_net.c, the remoted struct in config/remote-config.h was modified to add a pointer called netinfo to the remoted struct using a typedef of OSNetInfo (directly below where the sock element is defined), and remoted/remoted.c was modified to accept the new OSNetInfo pointer and store it in the netinfo element in the logr configuration struct. Next, I modified the routines that use the sockets to use the logr.netinfo element instead of the logr.sock element that was originally used for reads. Blocking reads were replaced with select() to multiplex the sockets that became available for reading. This change affected secure.c, syslog.c, and syslogtcp.c in the remoted subdirectory, and main-server.c in the os_auth subdirectory. The remoted/sendmsg.c module was modified to use multiple sockets for sending. In the event a message was received on one of the sockets, that socket was used for sending the packet out. In the event no packet had been received, the routine was modified to loop through the list of available sockets to ensure the packet is sent. The Makefile was also modified to added a define for NOV4MAP, which is commented out by default. When this is defined, Linux systems can build remoted with AF_UNSPEC instead of AF_INET6 and the AI_V4MAPPED attribute that was described earlier. As more operating systems adopt the link-local specification to establish an IPv6 address based on the MAC address, this could help Linux users with being able to support IPv4 on their systems. There are two other changes I made that are not related to the IP address issue but relate to successfully compiling and installing OSSEC on FreeBSD. Because I had to modify the Makefile for the NOV4MAP define, it made sense to include these changes as part of this pull request. Both of these changes are to make files that are provided with the distribution. The first change is to the Makefile in the top src directory. BSD systems use the /usr/local directory for managing files and programs that are not part of the standard operating system distribution. MySQL, Postgres, MQ, readline, and other applications are extensions that get installed under /usr/local. As such, it is necessary to add /usr/local/include directory for C header files and /usr/local/lib for object libraries. This already existed in Makefile for OpenBSD, but not for FreeBSD or NetBSD. The second Makefile change I made was to the LUA Makefile for BSD distributions. Like the Makefile in the src directory described above, the LUA build needs to have the same definitions for header files and libraries. The file I modified under the src directory is external/lua/src/Makefile. The modification added parameters for �-I/usr/local/include� and �-L/usr/local/lib�. With these changes, FreeBSD and other BSD systems will compile without problems using gmake and gcc. Header Files Modified os_net/os_net.h - Defined a new typedef struct called OSNetInfo that is used to return a list of bound sockets back to the caller, and changed the return values defined for OS_Bindporttcp() and OS_Bindportudp() to use a pointer to the OSNetInfo struct instead of the integer value they used to return. The pointer to the OSNetInfo struct is stored in the remoted struct that is defined as "logr" in the code. The pointer to the OSNetInfo pointer is named netinfo. config/remote-config.h - Added a #include for "os_net/os_net.h" to ensure the appropriate network function headers were defined for select() and added a pointer to an OSNetInfo struct named netinfo to the remoted struct. The netinfo pointer serves the same purpose as the old sock element, except that it handles multiple sockets instead of just a single socket. The remoted struct, which is usually referred to as �logr� in the remoted code, is used to pass a number of configuration elements between the remoted modules. Source Modules Modified os_net/os_net.c - Modified the module to return a struct called OSNetInfo that contains the bound sockets for each of the addresses in a format ready for use by select(), and an array of sockets that can be referenced for individual socket usage. The OSNetInfo struct is described earlier in this document. Also added code to use PF_UNSPEC for all interfaces except those that are for Linux and also have AI_V4MAPPED defined in <netdb.h>. Also added a number of helper functions to display addresses and other status information in the ossec.log file using verbose(). remoted/remoted.c - Modified the module to use the new OSNetInfo struct for return values from calls to OS_Bindporttcp() and OS_Bindportudp(). remoted/secure.c - Modified secure.c to utilize OSNetInfo data with multiple bound sockets via logr.netinfo instead of the original single bound socket with logr.sock. Also changed the message receipt loop to use select() to multiplex the sockets, instead of blocking on a read for a single socket. remoted/syslog.c - Modified syslog.c to support UDP messages the same way we modified secure.c, including using the OSNetInfo struct for multiple sockets and the select() call to support multiplexing. remoted/syslogtcp.c - Modified syslogtcp.c to support TCP messages the same way we modified secure.c and syslog.c, including using the OSNetInfo struct for multiple sockets and the select() call to support multiplexing. remoted/sendmsg.c - Modified sendmsg.c to use the bound socket array in the netinfo structure. If we have previously identified a valid socket, we use that socket to send on. Otherwise, we cycle through the array of valid bound sockets until we get a successful completion on the sendto() call. This approach appears to work well. os_auth/main-server.c - This module was an outlier, but because it called OS_Bindporttcp() it had to be modified to support the new multi-address approach. Modified main-server.c to use new OSNetInfo struct for data returned by OS_Bindporttcp and implemented the select() call to multiplex the group of sockets. Makefile Modifications Makefile - Found in the src directory at the top of the directory tree, I added /usr/local/include to CFLAGS and /usr/local/lib OSSEC_LDFLAGS to support successful compilation on FreeBSD and NetBSD. Also added the define for NOV4MAP that is commented out. When NOV4MAP is defined, Linux distributions will use AF_UNSPEC to bind sockets instead of AF_INET6 with the AI_V4MAPPED flag. external/lua/src/Makefile - LUA compilation fails with the current Makefile because the /usr/local/include directory is not specified for headers, and the /usr/local/lib is not specified for object libraries. I added parameters to the definitions for bsd and freebsd in the Makefile to make sure those directories were included. This allows the LUA compile process to complete without errors. Modification Summary and Pull Request I had originally performed this modification for the OSSEC 2.9.3 release and tested it thoroughly on FreeBSD 11.1 for several weeks. I then ported these changes into the current code branch of OSSEC that I retrieved from GitHub on May 1, 2018. This required my changes to be re-ported into newer versions of os_auth/main-server.c, os_net/os_net.c, and os_net/os_net.h. I also had to re-port changes into the two Makefiles I identified earlier. I only have a day of use on this, but it seemd to be running ithout issues. If anyone is interested in the changes that work for the 2.9.3 release, you can download the modified OSSEC 2.9.3 release from https://networkalarmcorp.com/dnld/ossec-hids-2.9.3.1.tgz. I should have the 2.9.3 code up on my server by Thursday. I have submitted a pull request for this modification to be incorporated into the OSSEC code base. If anyone has any questions or suggestions related to this modification, I am happy to discuss them. I hope this modification helps the OSSEC community at large. Dave Stoddard Network Alarm Corporation https://networkalarmcorp.com https://redgravity.net Office: 301-850-0668 x101 Email: dgs at networkalarmcorp dot com
| char ipport[NI_MAXSERV]; /* printed port */ | ||
| static char buf[256]; /* message buffer */ | ||
|
|
||
| rc = getnameinfo ((struct sockaddr *) sa, sa->sa_len, ipaddr, |
There was a problem hiding this comment.
As you might have seen from Travis: os_net/os_net.c:708:49: error: ‘struct sockaddr’ has no member named ‘sa_len’
Changing the sa->sa_len to sizeof(sa) allows it to compile on Linux.
There was a problem hiding this comment.
On Solaris 11.4 this causes the issue below, i think;
/var/ossec/bin/ossec-control start
Starting OSSEC HIDS v3.1.0 (by Trend Micro Inc.)...
ossec-execd already running...
2018/12/12 23:24:59 ossec-agentd: INFO: Using notify time: 600 and max time to reconnect: 1800
Started ossec-agentd...
Started ossec-logcollector...
2018/12/12 23:25:02 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:02 rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:10 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:10 rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:23 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:23 rootcheck(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
ossec-syscheckd did not start
There was a problem hiding this comment.
on Solaris, it compiles but cannot start
/var/ossec/bin/ossec-control start
Starting OSSEC HIDS v3.1.0 (by Trend Micro Inc.)...
ossec-execd already running...
2018/12/12 23:24:59 ossec-agentd: INFO: Using notify time: 600 and max time to reconnect: 1800
Started ossec-agentd...
Started ossec-logcollector...
2018/12/12 23:25:02 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:02 rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:10 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:10 rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:23 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Destination address required'.
2018/12/12 23:25:23 rootcheck(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
ossec-syscheckd did not start
There was a problem hiding this comment.
The issue with Solaris is not related to this change. The code you referred to in src/os_net/os_net.c in function OS_DecodeSockaddr() is simply a function to list the contents of a sockaddr struct (passed as a pointer) for diagnostic purposes. It does not change the contents of the struct or modify it in any way.
Prior to the change to using getaddrinfo() with AF_UNSPEC, OSSEC would only operate on IPv6 addresses (IPv4 addresses were mapped onto IPv6 addresses). The modification, when combined with multiple address retrieval and select() calls, allowed all of the addresses assigned to all network interfaces on the machine to participate with OSSEC.
If you are experiencing issues with creating/accessing sockets in the queue directory, it is likely you are having permission problems with the directories. The /var/ossec/queue directory should be chmod 550 and owned by owner:group root:ossec. The ossec subdirectory under it should be chmod 750 and owned by ossec:ossec. The socket named queue that is created inside that subdirectory is chmod 660 and owned by ossec:ossec - this socket should be created automatically by OSSEC.
If you are encountering problems with permissions, try changing the directory permissions to chmod 777 on /var/ossec/queue and /var/ossec/queue/ossec . If it works, reset permissions on /var/ossec/queue and try again. If that works, try adjusting permission on /var/ossec/queue/ossec. You will be able to isolate it this way. If you have to run with relaxed permissions, you can also try using the sticky bit on the /var/ossec/queue/ossec directory - that will limit access to the creating owner and group. You can set the sticky bit on a directory using:
chmod 1777 /var/ossec/queue/ossec
Good luck!
Dave Stoddard
Network Alarm Corporation
|
Epic post! It'll take me a while to get through the whole diff, but it sounds really neat. I had hoped to not make any major changes to |
|
Dan,
Thank you – it amazes me that there are still differences between BSD and Linux after so many years. I made the fix and I will push the commit back to the OSSEC master in a couple of hours. Thanks again. Best,
Dave
From: Dan Parriott <notifications@github.com>
Sent: Thursday, May 03, 2018 7:27 AM
To: ossec/ossec-hids <ossec-hids@noreply.github.com>
Cc: Dave Stoddard <dgs@accelix.net>; Author <author@noreply.github.com>
Subject: Re: [ossec/ossec-hids] Modification to Correct IP Connectivity Issues on BSD Servers (#1412)
@ddpbsd commented on this pull request.
_____
In src/os_net/os_net.c <#1412 (comment)> :
+ return 0;
+}
+
+
+/*
+ * OS_DecodeSockaddr() will decode a socket address and return a string with
+ * the IP version, address, and port number.
+ */
+
+char *OS_DecodeSockaddr (struct sockaddr *sa) {
+ int rc; /* return code */
+ char ipaddr[INET6_ADDRSTRLEN]; /* printed address */
+ char ipport[NI_MAXSERV]; /* printed port */
+ static char buf[256]; /* message buffer */
+
+ rc = getnameinfo ((struct sockaddr *) sa, sa->sa_len, ipaddr,
As you might have seen from Travis: os_net/os_net.c:708:49: error: ‘struct sockaddr’ has no member named ‘sa_len’
Changing the sa->sa_len to sizeof(sa) allows it to compile on Linux.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#1412 (review)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AToSI33Hsv4n9bSDLgojEIZGev0mZVJAks5tuulrgaJpZM4TwVEr> . <https://github.com/notifications/beacon/AToSI17xZ9TERUsDODr-q5WcHKuKc-sbks5tuulrgaJpZM4TwVEr.gif>
|
Fixed incompatibility between Linux and *BSD regarding the sa_len element in the sockaddr structure for os_net.c (replaced with sizeof(sa) - thanks Dan!). Corrected small issue with multiplexing sockets and the availability of a socket for send_msg(). This affected secure.c and sendmsg.c. Also fixed two variable scope complaints issued by Codacy on the original PR.
…_net.c for the ip multiplexing modification. It turns out that getnameinfo() is much pickier on BSD than it is on Linux, and it returns EAI_FAIL if the exact length returned by getaddrinfo() in the sa->sa_len parameter is not passed in the call to getnameinfo(). Most versions of Linux do not have sa_len as part of the addrinfo struct so you can just pass sizeof (sa), but if you do this on BSD it fails with EAI_FAIL (error 4). Also changed the counter being evaluated for socket creation. Under some circumstances, socket 0 is a valid socket. Instead of looking at the socket vlue, we count the number of sockets and use that for comparicon instead.
… sa->sa_len parameter in the addrinfo struct. This fixes that issue.
|
|
||
| /* Bind a specific port */ | ||
| int OS_Bindport(char *_port, unsigned int _proto, const char *_ip) | ||
| /* Bind all relevant ports */ |
There was a problem hiding this comment.
Does this change affect the local_ip option?
http://www.ossec.net/docs/syntax/head_ossec_config.remote.html#element-local_ip
There was a problem hiding this comment.
No, it does not affect the local_ip option - it continues to work the same way is always did. The OSNetInfo() function in os_net/os_net.c receives the port and IP address as character pointers (text strings) and the connection prototype (TCP or UDP) as an integer. Those values are passed to getaddrinfo() in the same way they were passed in the original code. However, in the absence of the local_ip being specified, this change will allow it to bind to all active IP addresses instead of the first address it finds in the system interface list.
There are several advantages to this approach. First, under the old code, you got the first address getaddrinfo() found. On FreeBSD, if that first address was not your primary interface address (i.e., receiving an IPv6 address when you are configured for an IPv4 network), you had to disable IPv6 to get OSSEC to work (which broke other things). Second, the old code did not support multiple interfaces (like multiple ethernet ports on pfSense routers) - this new code does. Third, the old code did not let you support both IPv6 and IPv4 on the same interface (although, with the old code you could do "mapped" IPv4 over IPv6). Under the new code, this is no longer an issue - both IPv4 and IPv6 are supported at the same time.
Note that I intentionally left the Linux code as it was because it was working without issues (but for just one address). However, it might be beneficial to use AF_UNSPEC in place of AF_INET6 for the Linux code too. I added the ability for Linux to support AF_UNSPEC with a #define NOV4MAP. Allowing all systems running OSSEC to support multiple IP addresses using both IPv4 and IPv6 and/or multiple interfaces is really useful.
I have this code running on seven FreeBSD 11.1 and 11.2 systems and one pfSense system and it is working like a charm. I also have a CentOS Linux system in my office that has been running the modified code without issues. I have seen no performance issues with the use of select() in the new code for handling the incoming messages.
There was a problem hiding this comment.
Thank you for the detailed reply. Since 3.0 has finally dropped, I've finally gotten a chance to play with this. I haven't seen any problems so far. I'll try to give it a shot with Linux/NOV4MAP.
|
Excellent!! Thank you Dan!
Dave Stoddard
From: Dan Parriott <notifications@github.com>
Sent: Wednesday, July 25, 2018 8:32 AM
To: ossec/ossec-hids <ossec-hids@noreply.github.com>
Cc: Dave Stoddard <dgs@accelix.net>; Author <author@noreply.github.com>
Subject: Re: [ossec/ossec-hids] Modification to Correct IP Connectivity Issues on BSD Servers (#1412)
Merged #1412 <#1412> .
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#1412 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AToSI6iIpC4ZmPgALakwM_wiBlaFuk-nks5uKGVNgaJpZM4TwVEr> . <https://github.com/notifications/beacon/AToSI8n-3bWRCz7q2LtgFyjo-cxPvYGPks5uKGVNgaJpZM4TwVEr.gif>
|
|
@ddpbsd @davestoddard It seems that you forgot to call select() before checking for FD_ISSET in src/main-server.c which leads to empty fdwork. It makes the main-server.c to run infinity checking for active socket and the whole authentication between server and agent to hang forever. I have created a new PR to update this missing. Please help to review and merge this |
|
I also fix the typo for definition of NOV4MAP in Makefile within the above PR also. |
I have encountered a significant issue in OSSEC 2.9.3 and the current OSSEC beta (GitHub repository) release that affects the ability of OSSEC clients to be able to connect to FreeBSD servers. This issue also probably affects all current BSD derivatives and possibly some Linux systems as well. I have identified and resolved the issue in OSSEC by rewriting some of the networking code, however, before I describe the solution, I wanted to provide some background information on the problem first.
Problem Symptoms
The server is running FreeBSD 11.1 with all of the current updates. This is a standard OSSEC server installation with all standard options enabled and no bells and whistles (no MySQL and no MQ). After correcting for some BSD-specific issues for header files and libraries, OSSEC installs normally on the server with the install.sh script and the compilation completes without any fatal errors (GCC does generate a few warnings on analysisd/decoders/syscheck.c and analysisd/decoders/syscheck-test.c, but these are not show stoppers). The server has an assigned IP address of 192.168.1.80 (IPv4 private IP space) and all of the clients it associates with are on the local 192.168.1.0/24 network.
The OSSEC 2.9.3 client software is installed on each Windows workstation and valid client keys are obtained for each client system. Everything appears fine, except that there are no alerts logged on the server for any of the configured clients. In short, the problem manifests itself as an inability for the client systems to connect to the OSSEC server. In the ossec.log file on each client there are messages like this that repeat over and over:
Because the clients use UDP, and UDP is a connectionless protocol, there is no protocol handshake to ensure a connection has actually been established like there would be with TCP. As long as the sendto() function returns a positive integer (the number of bytes sent), the call is deemed successful. This is true even when packets are not received on the destination host. To compensate for this, the server is supposed to return a control message to indicate the packet was received. Because the client never receives this confirmation, a warning is generated in the client log "Waiting for server reply (not started)".
On the server, remoted starts fine. It displays the maximum number of agents allowed (2048) and logs the fact that it successfully read the authentication keys file. However, the server logs hint that there is an issue when remoted starts because each client in the client.keys file results in a message stating "Assigning counter for agent Workstation01: '0:0'", or "Assigning sender counter: 0:0". These messages are generated when a client is defined in the etc/client.keys file, but no initial connection has ever occurred. This is normal for a first time start of OSSEC, but if this happens every time you restart OSSEC for every client defined in the system, your clients are not communicating.
To identify whether remoted was listening on UDP port 1514, I used the "netstat -an" command and the FreeBSD "sockstat" command (sockstat is like "lsof" for sockets). The only address bound to port 1514 was an IPv6 address, not the IPv4 address. The server uses only a single IPv4 address for network communication and no IPv6 addresses. However, it turns out that FreeBSD (and many other systems) create a link-local IPv6 address based on the MAC address of each interface when IPv6 is not specifically configured for the interface. This is specified in RFC 2462.
The standard solution for this would be to specify an IP address for the "secure" connection in the remote configuration using the <local_ip> option. However, when that is used, OSSEC ignores it and the system continues to bind to the IPv6 link-local address. While I could have experimented with disabling IPv6 in the operating system, this could have an adverse impact on other applications running on the server.
Normally, the system should simply query all available addresses on the system using getaddrinfo(), create a socket for each address, and bind the address to the socket. If the protocol is TCP, then you also need to use listen() to establish a queue. Then select() can be used to handle the dispatch of the individual sockets as messages arrive. I have a lot of experience with writing network interfaces in C, so I decided I would take a look at what was going on inside OSSEC that was creating this issue for me.
Initial Analysis
The OSSEC remoted process is responsible for handling the connections between the clients and the server. All of this code, except for the actual networking code, is found in the src/remoted directory. The initial connections for the UDP secure server and the syslog server are initiated in the HandleRemote() function inside remoted.c through calls to OS_Bindportudp() or OS_Bindporttcp() which are found in os_net/os_net.c. What is immediately obvious is that only one socket is returned as an integer value, instead of a group of sockets (one for each interface and protocol family).
The actual processing of incoming messages is divided between secure.c for secure UDP connections, syslog.c for syslog UDP connections, and syslogtcp.c for syslog TCP connections. Most of the outgoing connections are handled by send_msg.c (with some exceptions). Because there is only one socket used per communication strategy (secure UDP, syslog UDP, and syslog TCP), all of the routines use a while(1) loop with blocking reads to process incoming messages. To handle multiple sockets without dragging the CPU into the mud, these would have to be modified to use select().
The next thing I looked at was the code that was doing the network calls, which is found in the file os_net/os_net.c. Both the OS_Bindportudp() and OS_Bindporttcp() calls referred to a function called OS_Bindport() to handle all of the network initialization code including querying the available addresses, creating the socket, binding the address, invoking listen() for the queue (TCP only), and returning the active socket to the caller.
The code in os_net/os_net.c clearly showed my dilemma; the call to getaddrinfo() was setup to use AF_INET6 for the address family and AI_V4MAPPED, AI_PASSIVE, and AI_ADDRCONFIG for flags. Under this specification, IPv6 is preferred over IPv4, and if no IPv6 addresses exist and IPv4 addresses do exist, the IPv4 address is to be returned as a mapped IPv4 address using IPv6. A better solution is to use PF_UNSPEC for the address family with AI_PASSIVE and AI_ADDRCONFIG to pick up all addressed on configured interfaces.
The original author of this module recognized this as a problem early on, but s/he assumed that if they had AI_V4MAPPED defined in netdb.h, then the system would handle it properly. However, if IPv4 is preferred and there are any IPv6 addresses configured on the system, AI_V4MAPPED will not yield any IPv4 addresses. There is a good article by IBM on this that you can read here:
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.hale001/ipv6d0131000748.htm
This is further compounded because OSSEC is returning only a single socket bound to a single address making the process of supporting both IPv4 and IPv6 at the same time is impossible. The only correct solution is to modify OSSEC to handle multiple IP addresses under both IPv4 and IPv6 simultaneously.
The Solution
The overall solution to this problem was to return all of the addresses that meet the selection criteria and use the select() function to return sockets that are ready for reading. While this change affected a number of modules in the OSSEC system, the change was relatively straight forward to implement. A list of modules affected is provided at the end of this document.
Under the current version of OSSEC, a function named OS_Bindport() is used to locate the first address that is returned from getaddrinfo(). Then a socket for the appropriate connection type is created (SOCK_DGRAM for UDP or SOCK_STREAM for TCP), and the address is bound to the socket. If the connection is a TCP connection, a queue is established for the bound socket using listen(). Then the integer value socket is returned. If the socket value is -1, an error is assumed and various error actions are taken by the caller.
Under the modified version, OS_Bindport() was modified in a number of ways. First, instead of returning an integer value representing a bound socket to the caller, the routine now returns a pointer to an OSNetInfo structure. The OSNetInfo structure is defined in os_net/os_net.h as follows:
In order to multiplex multiple sockets efficiently, we need to use the select() function. The select() function operates on a struct called fd_set that is built on the successful acquisition of a socket with a bound address. Each successful acquisition is added to the set using the FD_SET macro. We also track the highest value socket number returned in fdmax. When all processing is completed, fdmax is incremented by 1 so it can be used directly with the select() function in other modules. We also track the individual bound sockets in an integer table called fds, and a count of the number of sockets in the table using fdcnt.
Because the original implementation relied upon a return value of -1 to indicate an error, we include an element in the struct called status that performs the same function. If status is -1, then an error has occurred. Upon a successful return, the status element will be 0. If status is -1, the return value element retval will contain the error number associated with the error status. The OSNetInfo struct made the process of modifying the OSSEC networking code relatively simple because it preserved all of the functionality that was inherent with the original code.
In addition to the OSNetInfo structure as a return value, the second major change to os_net/os_net.c was the definition of hints to the getaddrinfo() function. The getaddrinfo() function was created by the POSIX standards committee to provide support both IPv4 and IPv6 simultaneously. The getaddrinfo() function modifies a struct that returns a pointer to a linked list of addresses that match the selection criteria passed to getaddrinfo() through the hints parameter struct.
Under the original OSSEC code, if the operating system platform had AI_V4MAPPED defined in <netdb.h> (the vast majority of modern OS's support this), then the hints structure passed to getaddrinfo() used AF_INET6 for the protocol family. This essentially guaranteed that only IPv6 addresses would be returned if they were defined on the system. Even if there are no IPv6 addresses configured on the system, the AF_INET6 protocol family with the AI_V4MAPPED flag will return a mapped IPv4 address in IPv6 format.
For current *BSD systems, if an IPv6 address is not defined on an interface, the system will create a link-local IPv6 address based on the MAC address of each interface. IPv6 link-local addresses are essentially useless for external communication, so the net result is that *BSD systems (and others) are not able to communicate using IPv4.
By modifying this to use AF_UNSPEC instead of AF_INET6, eliminating the AI_V4MAPPED flag, and utilizing all of the addresses returned by the call to getaddrinfo(), we have the ability to communicate through all of the interfaces defined on the system using both IPv4 nd IPv6. Nonetheless, there are a number of Linux systems that are working fine for OSSEC with the AF_INET6 family and the AI_V4MAPPED flag. As such, an "#if defined" definition was placed in the code to support the original behavior on Linux systems, with the PF_UNSPEC behavior defined for non-Linux systems. In the event you do want to use the PF_UNSPEC code with a Linux system (I suspect this will become the norm as more versions of Linux adopt RFC 2462 and the link-local address), all you need to do is to uncomment the line in the Makefile that defines NOV4MAP and the system will use AF_UNSPEC instead of AF_INET6.
Once the call is made to getaddrinfo(), we utilize a loop to process each of the addresses returned in the linked list of addresses. Each socket that is bound successfully is added to the fd_set for subsequent select() processing, and also added to an array of integers containing bound sockets. The only other substantive change that was made to this module is that we set a socket option for the SO_REUSEADDR flag. In the event OSSEC is stopped and restarted again, the use of the SO_REUSEADDR socket option allows OSSEC to restart immediately without having to wait for a lingering socket to close. It also improves performance for short duration transactions.
The other changes in the os_net/os_net.c module include changing the return parameter from integer to a pointer to an OSNetInfo struct for OS_Bindportudp() and OS_Bindporttcp(), and a number of helper functions were added to the bottom of the module to provide additional information as the addresses are retrieved, sockets created, and addresses are bound to sockets.
Once the changes were made to os_net/os_net.c, the remoted struct in config/remote-config.h was modified to add a pointer called netinfo to the remoted struct using a typedef of OSNetInfo (directly below where the sock element is defined), and remoted/remoted.c was modified to accept the new OSNetInfo pointer and store it in the netinfo element in the logr configuration struct.
Next, I modified the routines that use the sockets to use the logr.netinfo element instead of the logr.sock element that was originally used for reads. Blocking reads were replaced with select() to multiplex the sockets that became available for reading. This change affected secure.c, syslog.c, and syslogtcp.c in the remoted subdirectory, and main-server.c in the os_auth subdirectory.
The remoted/sendmsg.c module was modified to use multiple sockets for sending. In the event a message was received on one of the sockets, that socket was used for sending the packet out. In the event no packet had been received, the routine was modified to loop through the list of available sockets to ensure the packet is sent.
The Makefile was also modified to added a define for NOV4MAP, which is commented out by default. When this is defined, Linux systems can build remoted with AF_UNSPEC instead of AF_INET6 and the AI_V4MAPPED attribute that was described earlier. As more operating systems adopt the link-local specification to establish an IPv6 address based on the MAC address, this could help Linux users with being able to support IPv4 on their systems.
There are two other changes I made that are not related to the IP address issue but relate to successfully compiling and installing OSSEC on FreeBSD. Because I had to modify the Makefile for the NOV4MAP define, it made sense to include these changes as part of this pull request. Both of these changes are to make files that are provided with the distribution.
The first change is to the Makefile in the top src directory. BSD systems use the /usr/local directory for managing files and programs that are not part of the standard operating system distribution. MySQL, Postgres, MQ, readline, and other applications are extensions that get installed under /usr/local. As such, it is necessary to add /usr/local/include directory for C header files and /usr/local/lib for object libraries. This already existed in Makefile for OpenBSD, but not for FreeBSD or NetBSD.
The second Makefile change I made was to the LUA Makefile for BSD distributions. Like the Makefile in the src directory described above, the LUA build needs to have the same definitions for header files and libraries. The file I modified under the src directory is external/lua/src/Makefile. The modification added parameters for "-I/usr/local/include" and "-L/usr/local/lib". With these changes, FreeBSD and other BSD systems will compile without problems using gmake and gcc.
Header Files Modified
os_net/os_net.h - Defined a new typedef struct called OSNetInfo that is used to return a list of bound sockets back to the caller, and changed the return values defined for OS_Bindporttcp() and OS_Bindportudp() to use a pointer to the OSNetInfo struct instead of the integer value they used to return. The pointer to the OSNetInfo struct is stored in the remoted struct that is defined as "logr" in the code. The pointer to the OSNetInfo pointer is named netinfo.
config/remote-config.h - Added a #include for "os_net/os_net.h" to ensure the appropriate network function headers were defined for select() and added a pointer to an OSNetInfo struct named netinfo to the remoted struct. The netinfo pointer serves the same purpose as the old sock element, except that it handles multiple sockets instead of just a single socket. The remoted struct, which is usually referred to as "logr" in the remoted code, is used to pass a number of configuration elements between the remoted modules.
Source Modules Modified
os_net/os_net.c - Modified the module to return a struct called OSNetInfo that contains the bound sockets for each of the addresses in a format ready for use by select(), and an array of sockets that can be referenced for individual socket usage. The OSNetInfo struct is described earlier in this document. Also added code to use PF_UNSPEC for all interfaces except those that are for Linux and also have AI_V4MAPPED defined in <netdb.h>. Also added a number of helper functions to display addresses and other status information in the ossec.log file using verbose().
remoted/remoted.c - Modified the module to use the new OSNetInfo struct for return values from calls to OS_Bindporttcp() and OS_Bindportudp().
remoted/secure.c - Modified secure.c to utilize OSNetInfo data with multiple bound sockets via logr.netinfo instead of the original single bound socket with logr.sock. Also changed the message receipt loop to use select() to multiplex the sockets, instead of blocking on a read for a single socket.
remoted/syslog.c - Modified syslog.c to support UDP messages the same way we modified secure.c, including using the OSNetInfo struct for multiple sockets and the select() call to support multiplexing.
remoted/syslogtcp.c - Modified syslogtcp.c to support TCP messages the same way we modified secure.c and syslog.c, including using the OSNetInfo struct for multiple sockets and the select() call to support multiplexing.
remoted/sendmsg.c - Modified sendmsg.c to use the bound socket array in the netinfo structure. If we have previously identified a valid socket, we use that socket to send on. Otherwise, we cycle through the array of valid bound sockets until we get a successful completion on the sendto() call. This approach appears to work well.
os_auth/main-server.c - This module was an outlier, but because it called OS_Bindporttcp() it had to be modified to support the new multi-address approach. Modified main-server.c to use new OSNetInfo struct for data returned by OS_Bindporttcp and implemented the select() call to multiplex the group of sockets.
Makefile Modifications
Makefile - Found in the src directory at the top of the directory tree, I added /usr/local/include to CFLAGS and /usr/local/lib OSSEC_LDFLAGS to support successful compilation on FreeBSD and NetBSD. Also added the define for NOV4MAP that is commented out. When NOV4MAP is defined, Linux distributions will use AF_UNSPEC to bind sockets instead of AF_INET6 with the AI_V4MAPPED flag.
external/lua/src/Makefile - LUA compilation fails with the current Makefile because the /usr/local/include directory is not specified for headers, and the /usr/local/lib is not specified for object libraries. I added parameters to the definitions for bsd and freebsd in the Makefile to make sure those directories were included. This allows the LUA compile process to complete without errors.
Modification Summary and Pull Request
I had originally performed this modification for the OSSEC 2.9.3 release and tested it thoroughly on FreeBSD 11.1 for several weeks. I then ported these changes into the current code branch of OSSEC that I retrieved from GitHub on May 1, 2018. This required my changes to be re-ported into newer versions of os_auth/main-server.c, os_net/os_net.c, and os_net/os_net.h. I also had to re-port changes into the two Makefiles I identified earlier. I only have a day of use on this, but it seemd to be running ithout issues.
If anyone is interested in the changes that work for the 2.9.3 release, you can download the modified OSSEC 2.9.3 release from https://networkalarmcorp.com/dnld/ossec-hids-2.9.3.1.tgz. I should have the 2.9.3 code up on my server by Thursday.
I have submitted a pull request for this modification to be incorporated into the OSSEC code base. If anyone has any questions or suggestions related to this modification, I am happy to discuss them. I hope this modification helps the OSSEC community at large.
Dave Stoddard
Network Alarm Corporation
https://networkalarmcorp.com
https://redgravity.net
Office: 301-850-0668 x101
Email: dgs at networkalarmcorp dot com