Skip to content

Latest commit

 

History

History
653 lines (534 loc) · 17.7 KB

STACK.md

File metadata and controls

653 lines (534 loc) · 17.7 KB

🏗️ BitBuilder Hypervisor Stack Architecture

Stack Architecture Templates Compliance

🧩 Comprehensive template system for creating fully compliant, multi-tenant rootfs filesystems with layered architecture and systemd integration.

📋 Table of Contents

Section Description
🎯 Template System Overview Layered template architecture
🏠 Tenant Templates Infrastructure, VM, container templates
🔧 Extension Image Templates sysext and confext templates
🌐 Network Templates Networking configuration patterns
⚙️ Systemd Generators Dynamic unit generation
🔄 Tenant Instantiation Flow Provisioning workflow
✅ Directory Structure Compliance UAPI compliance validation

🎯 Template System Overview

BitBuilder Hypervisor uses a layered template system that creates fully compliant rootfs filesystems for each tenant component. Each template is a Git repository that contains a complete filesystem hierarchy adhering to the Linux Userspace API specifications.

🧩 Template Categories

graph TB
    T[📦 Templates] --> I[🏗️ Infrastructure Templates]
    T --> M[🖥️ Machine Templates]
    T --> C[📦 Container Templates]
    T --> H[🏠 Homed Templates]
    T --> N[🌐 Network Templates]

    I --> IT[Management layer for each tenant]
    M --> MT[VM templates - systemd-vmspawn]
    C --> CT[Container templates - systemd-nspawn]
    H --> HT[User home directory templates]
    N --> NT[Network configuration templates]

    style T fill:#e3f2fd
    style I fill:#f3e5f5
    style M fill:#e8f5e8
    style C fill:#fff3e0
    style H fill:#fce4ec
    style N fill:#f1f8e9
Loading

🏠 Tenant Templates

1️⃣ Tenant Infrastructure Manager Template (tenant-infra-template)

🎯 Purpose: The infrastructure manager runs as a privileged systemd-vmspawn VM within each tenant's namespace and manages all tenant resources.

🗂️ Repository Structure

tenant-infra-template/
├── metadata.json                      # Template metadata
├── rootfs/                            # Compliant root filesystem
│   ├── etc/
│   │   ├── os-release                # Required OS identification
│   │   ├── systemd/
│   │   │   ├── system/
│   │   │   │   ├── multi-user.target.wants/
│   │   │   │   ├── infra-manager.service
│   │   │   │   ├── git-sync.service
│   │   │   │   ├── git-sync.timer
│   │   │   │   └── varlink-api.service
│   │   │   └── network/
│   │   │       ├── 10-management.network
│   │   │       └── 10-management.netdev
│   │   └── extension-release.d/       # Extension compatibility
│   ├── usr/
│   │   ├── lib/
│   │   │   ├── os-release -> ../../etc/os-release
│   │   │   ├── systemd/system/        # Vendor units
│   │   │   └── bitbuilder/
│   │   │       ├── infra-manager      # Management binary
│   │   │       └── lib/               # Shared libraries
│   │   └── share/
│   │       └── bitbuilder/
│   │           └── schemas/           # JSON schemas
│   ├── var/
│   │   └── lib/
│   │       └── infra/                 # Persistent infra data
│   └── run/                           # Runtime directory (empty)
├── config/                            # Default configuration
│   ├── infra-config.yaml
│   └── resource-limits.yaml
└── scripts/
    ├── provision.sh                   # Provisioning script
    └── health-check.sh               # Health monitoring

📋 metadata.json Configuration

{
  "template": {
    "type": "infrastructure",
    "version": "1.0.0",
    "name": "tenant-infra-template"
  },
  "requirements": {
    "systemd_version": ">=258",
    "kernel": ">=6.0",
    "features": ["vmspawn", "varlink", "networkd"]
  },
  "defaults": {
    "resources": {
      "cpu": 2,
      "memory": "2G",
      "storage": "10G"
    },
    "network": {
      "mode": "private",
      "management_interface": true
    }
  }
}

2️⃣ Tenant Machine Template (tenant-machine-template)

Template for creating tenant VMs using systemd-vmspawn.

Repository Structure:

tenant-machine-template/
├── metadata.json
├── disk-image/                        # Base disk image
│   ├── efi/                          # EFI partition
│   │   └── EFI/
│   │       └── systemd/
│   │           └── systemd-bootx64.efi
│   ├── boot/                         # Boot partition
│   │   ├── vmlinuz                   # Kernel
│   │   └── initrd.img               # Initial ramdisk
│   └── rootfs/                      # Root filesystem
│       ├── etc/
│       │   ├── os-release
│       │   ├── machine-id           # Empty (generated)
│       │   └── systemd/
│       │       ├── system/
│       │       └── network/
│       ├── usr/                     # Minimal userspace
│       └── var/
├── config/
│   ├── vm-config.yaml
│   └── cloud-init/                  # Cloud-init templates
│       ├── user-data
│       └── meta-data
└── extensions/                      # VM-specific extensions
    ├── development.sysext.raw
    └── monitoring.confext.raw

3️⃣ Tenant Container Template (tenant-container-template)

Template for creating tenant containers using systemd-nspawn.

Repository Structure:

tenant-container-template/
├── metadata.json
├── rootfs/                          # Container root filesystem
│   ├── etc/
│   │   ├── os-release
│   │   └── systemd/
│   │       ├── system/
│   │       │   └── container-init.service
│   │       └── network/
│   │           └── 80-container.network
│   ├── usr/
│   │   ├── bin/                     # Essential binaries
│   │   └── lib/
│   └── var/
│       └── lib/
├── config/
│   └── nspawn-settings.conf         # systemd-nspawn settings
└── scripts/
    └── container-init.sh

⚙️ nspawn-settings.conf

[Exec]
Boot=yes
ProcessTwo=yes
NotifyReady=yes

[Network]
Private=yes
VirtualEthernet=yes
Bridge=br-tenant

[Files]
PrivateUsersChown=yes
BindReadOnly=/usr/lib/extensions

4️⃣ Tenant Homed Template (tenant-homed-template)

Template for user home directories managed by systemd-homed.

Repository Structure:

tenant-homed-template/
├── metadata.json
├── home-skeleton/                   # Default home structure
│   ├── .config/
│   │   └── systemd/
│   │       └── user/                # User services
│   ├── .local/
│   │   ├── bin/                     # User binaries
│   │   └── share/
│   └── .ssh/                        # SSH configuration
│       └── authorized_keys.template
├── config/
│   ├── homed-config.json            # systemd-homed config
│   └── user-record.json             # User record template
└── scripts/
    └── home-provision.sh

🔧 Extension Image Templates

🔧 System Extension (sysext) Template

base-tools.sysext/
├── usr/                             # Only /usr and /opt allowed
│   ├── bin/
│   │   ├── htop
│   │   ├── tmux
│   │   └── vim
│   └── lib/
│       └── extension-release.d/
│           └── extension-release.base-tools
└── opt/
    └── bitbuilder/
        └── tools/

📋 extension-release.base-tools

ID=bitbuilder
VERSION_ID=1.0
SYSEXT_LEVEL=1.0
ARCHITECTURE=x86-64

⚙️ Configuration Extension (confext) Template

network-policies.confext/
└── etc/                             # Only /etc allowed
    ├── systemd/
    │   └── network/
    │       ├── 20-wireguard.netdev
    │       └── 20-wireguard.network
    └── extension-release.d/
        └── extension-release.network-policies

🌐 Network Templates

1️⃣ Bridge Network Template

📄 /usr/lib/systemd/system/10-tenant-bridge.netdev:

[NetDev]
Name=br-tenant-%i
Kind=bridge

[Bridge]
STP=yes
Priority=32768
ForwardDelaySec=15
HelloTimeSec=2
MaxAgeSec=20

📄 /usr/lib/systemd/network/10-tenant-bridge.network:

[Match]
Name=br-tenant-%i

[Network]
DHCP=no
IPv6AcceptRA=no
IPForward=yes
IPMasquerade=yes
Address=10.%i.0.1/24

[DHCPServer]
PoolOffset=100
PoolSize=100
EmitDNS=yes
DNS=10.%i.0.1
EmitRouter=yes

2️⃣ WireGuard VPN Template

📄 /usr/lib/systemd/network/20-wg-tenant.netdev:

[NetDev]
Name=wg-tenant-%i
Kind=wireguard

[WireGuard]
PrivateKeyFile=/var/lib/tenants/%i/network/wireguard.key
ListenPort=51820

[WireGuardPeer]
PublicKey=${PEER_PUBLIC_KEY}
AllowedIPs=10.%i.100.0/24
Endpoint=${PEER_ENDPOINT}:51820
PersistentKeepalive=25

📄 /usr/lib/systemd/network/20-wg-tenant.network:

[Match]
Name=wg-tenant-%i

[Network]
Address=10.%i.100.1/24

[Route]
Destination=10.%i.100.0/24
Scope=link

3️⃣ VXLAN Overlay Template

📄 /usr/lib/systemd/network/30-vxlan-tenant.netdev:

[NetDev]
Name=vxlan-tenant-%i
Kind=vxlan

[VXLAN]
VNI=%i
Group=239.0.0.1
DestinationPort=4789
MacLearning=yes

📄 /usr/lib/systemd/network/30-vxlan-tenant.network:

[Match]
Name=vxlan-tenant-%i

[Network]
Bridge=br-tenant-%i

[BridgeVLAN]
PVID=1
EgressUntagged=1

4️⃣ VLAN Segmentation Template

📄 /usr/lib/systemd/network/40-vlan-tenant.netdev:

[NetDev]
Name=vlan-tenant-%i
Kind=vlan

[VLAN]
Id=%i

📄 /usr/lib/systemd/network/40-vlan-tenant.network:

[Match]
Name=vlan-tenant-%i

[Network]
DHCP=yes
IPv6AcceptRA=yes

[DHCP]
RouteMetric=200
UseDomains=route

5️⃣ Network Namespace Template

/usr/lib/systemd/system/netns-tenant@.service:

[Unit]
Description=Network namespace for tenant %i
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/ip netns add tenant-%i
ExecStart=/usr/bin/ip netns exec tenant-%i ip link set lo up
ExecStop=/usr/bin/ip netns delete tenant-%i

[Install]
WantedBy=multi-user.target

⚙️ Systemd Generators

1️⃣ Tenant Discovery Generator

/usr/lib/systemd/system-generators/tenant-generator:

#!/bin/bash
# Discovers tenants from Git repositories and generates systemd units

GENERATOR_DIR="$1"
TENANT_REGISTRY="/var/lib/bitbuilder/tenants.json"

generate_tenant_units() {
    local tenant_id="$1"
    local tenant_config="$2"

    # Generate main tenant service
    cat > "${GENERATOR_DIR}/tenant@${tenant_id}.service" <<EOF
[Unit]
Description=Tenant ${tenant_id}
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStartPre=/usr/lib/bitbuilder/tenant-provision ${tenant_id}
ExecStart=/usr/lib/bitbuilder/tenant-manager ${tenant_id}
ExecStop=/usr/lib/bitbuilder/tenant-cleanup ${tenant_id}
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

    # Generate infrastructure manager service
    cat > "${GENERATOR_DIR}/tenant-infra@${tenant_id}.service" <<EOF
[Unit]
Description=Infrastructure manager for tenant ${tenant_id}
After=tenant@${tenant_id}.service
BindsTo=tenant@${tenant_id}.service

[Service]
Type=notify
ExecStart=/usr/bin/systemd-vmspawn \
    --machine=infra-${tenant_id} \
    --directory=/var/lib/tenants/${tenant_id}/infra/rootfs \
    --network-bridge=br-tenant-${tenant_id} \
    --bind-ro=/usr/lib/extensions \
    --setenv=TENANT_ID=${tenant_id}
Restart=always

[Install]
WantedBy=tenant@${tenant_id}.service
EOF

    # Generate network setup service
    cat > "${GENERATOR_DIR}/tenant-network@${tenant_id}.service" <<EOF
[Unit]
Description=Network setup for tenant ${tenant_id}
Before=tenant@${tenant_id}.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/lib/bitbuilder/setup-tenant-network ${tenant_id}
ExecStop=/usr/lib/bitbuilder/teardown-tenant-network ${tenant_id}

[Install]
RequiredBy=tenant@${tenant_id}.service
EOF
}

# Read tenant registry and generate units
if [[ -f "$TENANT_REGISTRY" ]]; then
    jq -r '.tenants[] | @base64' "$TENANT_REGISTRY" | while read -r tenant_data; do
        tenant_json=$(echo "$tenant_data" | base64 -d)
        tenant_id=$(echo "$tenant_json" | jq -r '.id')

        generate_tenant_units "$tenant_id" "$tenant_json"
    done
fi

2️⃣ Mount Generator

/usr/lib/systemd/system-generators/mount-generator:

#!/bin/bash
# Generates mount units for tenant directories and extensions

GENERATOR_DIR="$1"
TENANT_BASE="/var/lib/tenants"

generate_extension_mounts() {
    local tenant_id="$1"

    # Generate sysext mount
    cat > "${GENERATOR_DIR}/var-lib-tenants-${tenant_id}-sysext.mount" <<EOF
[Unit]
Description=System extensions for tenant ${tenant_id}
Before=tenant@${tenant_id}.service

[Mount]
What=overlay
Where=/var/lib/tenants/${tenant_id}/overlay/usr
Type=overlay
Options=lowerdir=/usr:/var/lib/tenants/${tenant_id}/extensions/sysext,upperdir=/var/lib/tenants/${tenant_id}/overlay/usr,workdir=/var/lib/tenants/${tenant_id}/overlay/.usr-work

[Install]
RequiredBy=tenant@${tenant_id}.service
EOF

    # Generate confext mount
    cat > "${GENERATOR_DIR}/var-lib-tenants-${tenant_id}-confext.mount" <<EOF
[Unit]
Description=Configuration extensions for tenant ${tenant_id}
Before=tenant@${tenant_id}.service

[Mount]
What=overlay
Where=/var/lib/tenants/${tenant_id}/overlay/etc
Type=overlay
Options=lowerdir=/etc:/var/lib/tenants/${tenant_id}/extensions/confext,upperdir=/var/lib/tenants/${tenant_id}/overlay/etc,workdir=/var/lib/tenants/${tenant_id}/overlay/.etc-work

[Install]
RequiredBy=tenant@${tenant_id}.service
EOF
}

# Generate mounts for each existing tenant
if [[ -d "$TENANT_BASE" ]]; then
    for tenant_dir in "$TENANT_BASE"/*; do
        if [[ -d "$tenant_dir" ]]; then
            tenant_id=$(basename "$tenant_dir")
            generate_extension_mounts "$tenant_id"
        fi
    done
fi

🔄 Tenant Instantiation Flow

1️⃣ Enable Tenant

# Enable and start a new tenant
systemctl enable --now tenant@tenant123.service

2️⃣ Instantiation Process

graph TD
    A[systemctl enable tenant@ID] --> B[Tenant Generator]
    B --> C[Read tenant registry]
    C --> D[Clone Git repositories]
    D --> E[Generate systemd units]
    E --> F[Setup network namespace]
    F --> G[Mount extensions]
    G --> H[Start infra manager VM]
    H --> I[Provision tenant resources]
    I --> J[Start tenant workloads]
Loading

3️⃣ Generated Unit Hierarchy

tenant@tenant123.service
 Requires: tenant-network@tenant123.service
 Wants: tenant-infra@tenant123.service
 After: network-online.target
 Triggers:
     var-lib-tenants-tenant123-sysext.mount
     var-lib-tenants-tenant123-confext.mount
     netns-tenant@tenant123.service
     br-tenant-123.netdev

✅ Directory Structure Compliance

📁 Linux File System Hierarchy Compliance

Each template ensures compliance with the Linux File System Hierarchy specification:

  1. Root Filesystem Requirements:

    • /etc/os-release or /usr/lib/os-release present
    • Proper symlink from /usr/lib/os-release to /etc/os-release
    • No files in / root directory itself

  2. Extension Image Requirements:

    • sysext: Only /usr/ and /opt/ directories
    • confext: Only /etc/ directory
    • Proper extension-release files in correct locations

  3. Verification Structure:

    • VOA hierarchy at /etc/voa/ and /usr/share/voa/
    • Proper certificate fingerprint naming
    • ASCII-armored OpenPGP files

✅ Template Validation

Each template includes a validation script:

#!/bin/bash
# validate-template.sh

validate_rootfs() {
    local rootfs="$1"

    # Check for required files
    [[ -f "$rootfs/etc/os-release" ]] || error "Missing /etc/os-release"
    [[ -L "$rootfs/usr/lib/os-release" ]] || error "Missing /usr/lib/os-release symlink"

    # Validate directory structure
    for dir in etc usr var proc sys dev run tmp; do
        [[ -d "$rootfs/$dir" ]] || error "Missing /$dir directory"
    done

    # Check permissions
    [[ $(stat -c %a "$rootfs/tmp") == "1777" ]] || error "Invalid /tmp permissions"

    echo "Template validation successful"
}

🎯 Conclusion

This template system provides the following capabilities:

  1. Standardization: All tenants use consistent, validated templates
  2. Compliance: Full adherence to Linux Userspace API specifications
  3. Flexibility: Multiple template types for different use cases
  4. Security: Layered isolation with proper namespace separation
  5. Automation: Systemd generators handle all instantiation logic
  6. Networking: Comprehensive network isolation and connectivity options

The combination of Git-based configuration, systemd integration, and compliant filesystem templates creates a robust, secure, and manageable multi-tenant hypervisor platform.