Hardware CPU/RAM/DISK Run Omni On-Prem Requirements and Recommendations

[chrislawso]

Hi,
Unable to find searching most documentation pages https://docs.siderolabs.com/omni/self-hosted/run-omni-on-prem and also searched github and elsewhere.

For a docker compose deployment what system requirements and/or recommendations for CPU/RAM/DISK? If possible, please identify how hardware requirements change from 4-10 nodes versus 10-100 + . I want to learn on homelab with very few nodes and want to allocate minimum required resources.

Cheers

[rothgar]

I'll answer this here and we can add a section to https://docs.siderolabs.com/omni/self-hosted/omni-deployment-options with clarification but in reality Omni and etcd are very low utilization even with hundreds of nodes.

If you're fully hosting everything you need for Omni (image factory + container registry) then here's some rough estimates on what you should expect.

Full stack

If you want to run Omni with all supporting services on a single VM for up to 200 nodes you should have

4 CPU cores
8-16 GB memory
500 GB storage (image storage for 10 releases of Talos + cache + etcd)
2.5 GB network (mostly for wireguard throughput for machines)

A <50 node Omni would be roughly 1/2 those requirements

Container registry

3-4gb per talos+k8s release (including all extensions and system containers)

Image factory

Depends on how many builds you perform and if you have caching. A typical configuration I've seen has less than 10 installation images. Those images can be cached so they don't have to be rebuilt each time a node tries to use them (via initial install or upgrade)

This is probably 1-2gb total storage for a single Omni

Omni

Omni itself is very light weight and scaling from 10s of nodes to 100s of nodes doesn't impact Omni very much. Sharing the same host with etcd is usually fine (assuming your etcd isn't HA).

etcd

we defer to etcd for recommendations but in general 2 CPU cores, 8gb RAM, and a decently fast disk is enough.