[Security Manager Replacement] Strengthen OS core security via systemd configuration

[kumargu]

Please describe the end goal of this project

The OS core's security in absence of security manager can be strengthened by using a stronger systemd unit configuration. We can imagine this as sandboxing via systemd protecting itself from vulnerability in core or untrusted code (plugins). However, this is not a complete replacement of security manager, a part of it like controlling access to egress network, controlling access to specific file locations can be sought as a replacement. Some of the newly introduced configs will add more security than what is being offered by security manager.

Supporting References

#1687

Issues

#16634

Related component

Other

[kumargu]

A list of systemd unit configs which could be useful to restrict access to sys resources, lock down network access, restrict system call etc. This is a fairly exhaustive list but not a complete list -- such as access to socket ip addresses are not exhaustive. Similarly, some of them could be also removed depending on the issues seen during actual integration.

configs to be amended to exiting config

---

New Inclusions:

Filesystem restrictions: Only specific directories are writable or accessible; the rest of the filesystem can be made read-only or completely inaccessible
E.g. OpenSearch might be allowed to write to its data directory and /tmp, and nothing else).

Syscall filtering: A Seccomp filter can block dangerous system calls
– e.g. preventing plugins from forking new processes, rebooting the host, or calling low-level kernel interfaces. This drastically reduces the kernel attack surface.

Capability bounding: Linux capabilities (fine-grained privileges normally held by root) can be dropped from the OpenSearch process
. For instance, removing CAP_SYS_ADMIN and others ensures OpenSearch (and its plugins) cannot perform system admin operations.

Network access limits: The service can be restricted to bind only to expected ports and even to allow or deny outgoing network connections to certain hosts

And more: Systemd can also restrict access to kernel keyrings, IPC, and apply many other protections (many of which can exceed what JSM provided

# Prevent modifications to the control group filesystem
ProtectControlGroups=true

# Prevent loading or reading kernel modules
ProtectKernelModules=true

# Prevent altering kernel tunables (sysctl parameters)
ProtectKernelTunables=true

# Restrict access to the filesystem:
# 'strict' makes /usr and /boot read-only, and hides other directories
ProtectSystem=strict

# Set device access policy to 'closed', allowing access only to specific devices
DevicePolicy=closed

# Make /proc invisible to the service, enhancing isolation
ProtectProc=invisible

# Make /usr, /boot, and /etc read-only (less restrictive than 'strict')
ProtectSystem=full

# Prevent changes to control groups (redundant with earlier setting, can be removed)
ProtectControlGroups=yes

# Prevent changing the execution domain
LockPersonality=yes


# System call filtering
# System call filterings which restricts which system calls a process can make
# @ means allowed 
# ~ means not allowed
SystemCallFilter=madvise mincore mlock mlock2 munlock get_mempolicy sched_getaffinity sched_setaffinity fcntl
SystemCallFilter=@system-service
SystemCallFilter=~@reboot
SystemCallFilter=~@swap

SystemCallErrorNumber=EPERM

# Capability restrictions
# Remove the ability to block system suspends
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND

# Remove the ability to establish leases on files
CapabilityBoundingSet=~CAP_LEASE

# Remove the ability to use system resource accounting
CapabilityBoundingSet=~CAP_SYS_PACCT

# Remove the ability to configure TTY devices
CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG

# Remove below capabilities:
# - CAP_SYS_ADMIN: Various system administration operations
# - CAP_SYS_PTRACE: Ability to trace processes
# - CAP_NET_ADMIN: Various network-related operations
CapabilityBoundingSet=~CAP_SYS_ADMIN ~CAP_SYS_PTRACE ~CAP_NET_ADMIN


# Address family restrictions
RestrictAddressFamilies=~AF_INET ~AF_INET6 ~AF_NETLINK ~AF_PACKET ~AF_UNIX

 
# Filesystem Access
 
ReadWritePaths=/var/log/opensearch /var/lib/opensearch 

RestrictNamespaces=true

NoNewPrivileges=true

# Memory and execution protection
MemoryDenyWriteExecute=true           # Prevent creating writable executable memory mappings
SystemCallArchitectures=native        # Allow only native system calls
KeyringMode=private                   # Service does not share key material with other services
LockPersonality=true                  # Prevent changing ABI personality
RestrictSUIDSGID=true                 # Prevent creating SUID/SGID files
RestrictRealtime=true                 # Prevent acquiring realtime scheduling
ProtectHostname=true                  # Prevent changes to system hostname
ProtectKernelLogs=true                # Prevent reading/writing kernel logs
ProtectClock=true                     # Prevent tampering with the system clock

Cannot include

# Socket restrictions
SocketBindAllow=tcp:9200
SocketBindAllow=tcp:9300                         
SocketBindDeny=any                   # Deny all other socket bindings

[kumargu]

// todo

explore below configs

# Optional directives
IPAddressDeny=                       # Deny access to specific IP addresses
PrivateNetwork=true                  # Use a private network namespace

[rmuir]

nice to see this work here, especially sandboxing things such as filesystem with systemd fills a big gap that security manager was doing, and IMO does it in a much better way.

I'd be curious to see change in reported exposure level from systemd-analyze security opensearch.service with your improvements.

[kumargu]

thanks @rmuir. I will post the results from systemd-analyze security opensearch.service.

[kumargu]

cc @andrross

[andrross]

Thanks @kumargu, I think this approach is super promising.

It does raise some questions around testing and maintaining a properly secured systemd config. We'll need to evaluate our integration testing and release pipeline to ensure we have proper coverage of this, as the evolving code base may sometimes (rarely I hope) require tweaks to this config.

[rmuir]

You don't need to have all the SystemCallFilter= entries. Most of what you have listed is already excluded via @system-service.

i'd nuke all the CapabilityBoundingSet entries, just replace with CapabilityBoundingSet=. I'd set NoNewPrivileges=true along the same lines of not allowing escalation.

Same goes with your RestrictNamespaces= entries, they are not needed. it is enough to just set RestrictNamespaces=true.

[kumargu]

Ack @rmuir, I'll make changes as suggested by you, all your comments makes sense to me. I am going to try out the actual integration tomorrow and post results.

---

(edit) 3/12/24

Updated the initially proposed configs based on rmuir@ comments. Haven't yet changed the CapabilityBoundingSet since I need to read a little bit of it, rest all suggestions are incorporated.

[kumargu]

@andrross -- thanks for bringing up the testing part of it. I will think more about it. At the moment, I could think of having a test.service which gets spawned in test suite and verifies that it doesn't have access to restricted resources, e.g; no read/ write access to var/log/elasticsearch.

I don't think we will be able to get a full coverage, but we can cover for the most critical ones. And yes, it would be rare we'd be changing the configs.

[kumargu]

Adding the output of systemd-analyze security opensearch.service Before vs After

Before

(represents the systemd config currently shipped with 2.18)

→ Overall exposure level for opensearch.service: 8.8 EXPOSED 🙁

---

After

(systemd configs edited as per comment to add restrictive policies)

→ Overall exposure level for opensearch.service: 4.6 OK 🙂

systemd status

[ec2-user@ip-172-31-0-63 multi-user.target.wants]$ sudo systemctl status opensearch.service
● opensearch.service - OpenSearch
     Loaded: loaded (/usr/lib/systemd/system/opensearch.service; enabled; preset: disabled)
     Active: active (running) since Mon 2024-12-02 07:51:55 UTC; 6min ago
       Docs: https://opensearch.org/
   Main PID: 17776 (java)
      Tasks: 179 (limit: 601012)
     Memory: 1.5G
        CPU: 45.962s
     CGroup: /system.slice/opensearch.service
             └─17776 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow >

Full report

✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities
✗ PrivateDevices=                                             Service potentially has access to hardware devices                                                       0.2
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                                                  0.2
✗ CapabilityBoundingSet=~CAP_KILL                             Service may send UNIX signals to arbitrary processes                                                     0.1
✗ ProtectKernelLogs=                                          Service may read from or write to the kernel log ring buffer                                             0.2
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service may program timers that wake up the system                                                       0.1
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                                                     0.2
✗ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service may mark files immutable                                                                         0.1
✗ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service may lock memory into RAM                                                                         0.1
✗ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service may issue reboot()                                                                               0.1
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service may issue chroot()                                                                               0.1
✓ SystemCallArchitectures=                                    Service may execute system calls only with native ABI
✗ MemoryDenyWriteExecute=                                     Service may create writable executable memory mappings                                                   0.1
✗ CapabilityBoundingSet=~CAP_MKNOD                            Service may create device nodes                                                                          0.1
✗ RestrictSUIDSGID=                                           Service may create SUID/SGID files                                                                       0.2
✗ ProtectHostname=                                            Service may change system host/domainname                                                                0.1
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unrestricted                                  0.2
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                                                       0.3
✗ RestrictAddressFamilies=~AF_PACKET                          Service may allocate packet sockets                                                                      0.2
✗ RestrictAddressFamilies=~AF_NETLINK                         Service may allocate netlink sockets                                                                     0.1
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                                                       0.1
✗ RestrictAddressFamilies=~…                                  Service may allocate exotic sockets                                                                      0.3
✗ CapabilityBoundingSet=~CAP_MAC_*                            Service may adjust SMACK MAC                                                                             0.1
✗ RestrictRealtime=                                           Service may acquire realtime scheduling                                                                  0.1
✗ ProtectSystem=                                              Service has very limited write access to the OS file hierarchy                                           0.1
✓ ProtectProc=                                                Service has restricted access to process tree (/proc hidepid=)
✗ CapabilityBoundingSet=~CAP_SYS_RAWIO                        Service has raw I/O access                                                                               0.2
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                                                                 0.3
✗ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has privileges to change resource use parameters                                                 0.1
✓ SupplementaryGroups=                                        Service has no supplementary groups
✓ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has no administrator privileges
✓ PrivateTmp=                                                 Service has no access to other software's temporary files
✗ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has network configuration privileges                                                             0.2
✗ ProcSubset=                                                 Service has full access to non-process /proc files (/proc subset=)                                       0.1
✗ ProtectHome=                                                Service has full access to home directories                                                              0.2
✗ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has elevated networking privileges                                                               0.1
✗ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has audit subsystem access                                                                       0.1
✗ PrivateNetwork=                                             Service has access to the host's network                                                                 0.5
✗ PrivateUsers=                                               Service has access to other users                                                                        0.2
✗ CapabilityBoundingSet=~CAP_SYSLOG                           Service has access to kernel logging                                                                     0.1
✓ DeviceAllow=                                                Service has a minimal device ACL
✓ KeyringMode=                                                Service doesn't share key material with other services
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree
✗ IPAddressDeny=                                              Service does not define an IP address allow list                                                         0.2
✓ NotifyAccess=                                               Service child processes cannot alter service state
✓ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service cannot use acct()
✓ ProtectControlGroups=                                       Service cannot modify the control group file system
✓ ProtectKernelModules=                                       Service cannot load or read kernel modules
✓ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service cannot load kernel modules
✓ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service cannot issue vhangup()
✓ PrivateMounts=                                              Service cannot install system mounts
✓ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service cannot establish wake locks
✓ RestrictNamespaces=~user                                    Service cannot create user namespaces
✓ RestrictNamespaces=~pid                                     Service cannot create process namespaces
✓ RestrictNamespaces=~net                                     Service cannot create network namespaces
✓ RestrictNamespaces=~uts                                     Service cannot create hostname namespaces
✓ RestrictNamespaces=~mnt                                     Service cannot create file system namespaces
✓ CapabilityBoundingSet=~CAP_LEASE                            Service cannot create file leases
✓ RestrictNamespaces=~cgroup                                  Service cannot create cgroup namespaces
✓ RestrictNamespaces=~ipc                                     Service cannot create IPC namespaces
✓ LockPersonality=                                            Service cannot change ABI personality
✓ ProtectKernelTunables=                                      Service cannot alter kernel tunables (/proc/sys, …)
✓ RestrictAddressFamilies=~AF_(INET|INET6)                    Service cannot allocate Internet sockets
✗ UMask=                                                      Files created by service are world-readable by default                                                   0.1

→ Overall exposure level for opensearch.service: 4.6 OK 🙂