Groups V2 (Swap Matrix) Design

[mostlygeek]

There have been a steady stream of asks for improved group functionality. In August I wrote about my position that the groups design already felt like close to the right balance of complexity and what they were meant to do.

Since then there have been requests for more complex swapping logic to fit all sorts of combinations of models people are looking to run over various numbers of GPUs. This revealed the flaws and limitations of the existing groups design. llama-swap has also grown to support generative image and audio endpoints. It has been on my mind for a while now on how to enable more complex swap strategies while also keeping things easy to understand. Additionally, coding agents are good enough now that I can be more ambitious on the features I'm comfortable maintaining and supporting for this this project.

For the new design llama-swap will introduce a basic DSL for expressing sets of models that can run together. When a request comes in the solver will decide what needs to be unloaded to make space for the new model. llama-swap will still expect the admin to provide configurations that work.

(edit) Final configuration design:

# =============================================================================
# matrix: run concurrent models with a solver-based swap DSL
# =============================================================================
#
# Note:
# A config must use either groups or matrix not both. A configuration error
# will happen if both are defined.
#
# The matrix declares valid combinations of models that can run concurrently.
# When a model is requested, the solver finds the cheapest way to make it
# available by evicting as few (and least costly) running models as possible.
#
# Solver behavior:
#   1. Request arrives for model X
#   2. If X is already running, forward immediately. Done.
#   3. Find all sets containing X
#   4. For each candidate set, compute cost: sum of evict_costs for
#      every running model NOT in that set
#   5. Pick lowest cost candidate. Ties broken by definition order.
#   6. Evict what needs to stop. Start X. Forward request.
#
# Subset semantics: a set [a, b, c] means any subset is valid.
# Only the requested model is started — others are not preloaded.
#
# A model not appearing in any set can only run alone.
#
matrix:
  # vars: short names for models (alphanumeric, 1-8 chars)
  # - required for sets and evict_costs settings
  # - each entry is a short name to a real model ID. Do not use an alias
  # - used to keep set DSL logic short and easier to read
  # - sets and evict_costs only use identifiers defined in vars
  vars:
    g: gemma-model
    q: qwen-model
    m: mistral-model
    v: voxtral-model
    e: reranker-model
    L: llama-70B
    sd: stable-diffusion

  # evict_costs: relative cost of losing a running model (default: 1)
  evict_costs:
    v: 50 # vllm backend, slow cold start
    L: 30 # 70B weights, slow to load

  # sets: named sets of concurrent model combinations
  # Values are DSL strings with operators:
  #   &     AND (models run together)
  #   |     OR  (alternatives)
  #   ()    grouping
  #   +ref  inline another set's expression
  #
  # Expansion examples:
  #   "L"                  → [L]
  #   "a & b"              → [a, b]
  #   "a | b"              → [a], [b]
  #   "(a | b) & c"        → [a, c], [b, c]
  #   "(a | b) & (c | d)"  → [a,c], [a,d], [b,c], [b,d]
  #   "+llms & v"          → expands llms inline, then applies & v
  sets:
    # LLM + TTS: switching between g/q/m won't evict v
    # expands to: [g,v], [q,v], [m,v]
    standard: "(g | q | m) & v"

    # LLM + TTS + reranker
    # expands to: [g,v,e], [q,v,e]
    with_rerank: "(g | q) & v & e"

    # LLM + image generation, no TTS
    # expands to: [g,sd], [q,sd]
    creative: "(g | q) & sd"

    # 70B model uses all GPUs, can only run alone
    # expands to: [L]
    full: "L"

Refs:

• Group exclusivity is only one-way #215
• proxy: make exclusive groups bidirectional #631
• Use group as models #633
• More flexible groups exclusivity #634

[sousekd]

Thank you @mostlygeek for taking this forward — and also for stepping back and rethinking the problem instead of just patching groups further. The design here looks great.

Putting this together with the earlier discussion around exclusivity, I think my concern is slightly different from just "which exact model combinations are valid."

In my case, I have independent resource boundaries:

• one GPU where I swap among one set of models
• another GPU where I may run one of several valid VRAM layouts, like 96, 16+80, or 16+40+40

So the problem is partly about valid model combinations, but also partly about which resources are allowed to affect each other.

Because of that, I wonder whether it would still be useful to have some notion of resource partitioning / pools first, and then apply the new advanced swap logic within those partitions?

In other words: this new design looks great, but I am not sure whether a single global matrix fully captures cases where the machine is really split into semi-independent regions (separate GPUs, or even different VRAM layouts on one GPU).

Do you think it would still make sense for llama-swap to support something like “virtual partitions,” and then use the scheduling logic inside each partition? Or, does this design actually covers that? :)

[mostlygeek]

Hi,

Thanks for bringing that up. I’m still experimenting with the design so now is a good time to question everything. :)

In the new DSL syntax pooling can be accomplished with +pool1 & +pool2. pool1 can be all the models that run on a specific set of GPUs. pool2 can be defined like (a | b | c | etc), or (a & b & c) and can run on another group of gpus. The model config will have be done manually to make sure sets fit correctly.

The expanded sets from that would allow the solver to pick what to swap out to make room for
the new model.

[sousekd]

I see. In other words - "actually, it seems to cover that" - which is great :).
Sorry, my brain is not functioning particularly well today.

[sousekd]

@mostlygeek One feature for you to consider when working on this:

Sometimes, you might want to lock one or more loaded model(s) from unloading. In a single-user scenario, it is rarely needed, but it can still be handy. The reasons I had a wish feature like this existed in the past:

• When running a long running job using one of the models, to prevent other users from unloading it (accidentaly).
• To avoid annoying quirks of some UIs (for example, Silly Tavern can "touch" a model when you try to connect another one).

Some of my models take 10+ minutes to load (for example, SGLang + KTransformers need ages), and as currently implemented, the accident can easily cause the model to load-unload-load-unload-load :), so it is truly painful.

I was thinking that a "lock" button in the UI, either on the loaded model, or at least globally, would be nice and pair well with this new groups redesign...

[mostlygeek]

@sousekd to lock a model so it can never be unloaded?

I think it it's possible to configure that with this config:

matrix: 
    vars: 
        # pinned and never allowed to be unloaded by anything
        p1=never-unload1
        p2=never-unload2

        # swappable
        m1=vllm-model
        m2=lcpp-1
        m3=lcpp-2
    sets: 
        s: "(p1 & p2) & ( m1 | m2 | m3 )"

In the above set s, p1 and p2 would never be swapped out once they are loaded. This would effectively lock them. However m1, m2, and m3 would form a swap-able pool where only one of them can be running at a time.

[sousekd]

Yeah I mean temporarily - without modification of the configuration...

[lee-b]

Hmm, interesting set expression/DSL approach here. Seems elegant and expressive.

FYI (perhaps only for use case ideas) I hacked a similar solution (told an AI to do it, really) in personal repo, that implements a DAG of groups within groups as well as the normal models within groups. Then it implements swapping out the the smallest branch that fits the model, essentially walking up the tree searching for child nodes that are the model desired, and swapping at the level above that (if exclusive).

I also tasked the AI with breaking down my 4x24GB + 8GB GPU setup into groups for each of the configurations: all 104GB as a combined model space, or split into 96GB + 8GB, the 96GB split into 1x96GB model space or 1x72GB+24GB, 2x48GB, and 4x24GB, the 8GB split into 1x1GB and 2x4GB.

Together, these give you the ability to load a 4GB model on a 4GB "slot" of an 8GB card whilst your 96GB model stays put, or loading in an 8GB model without affecting your 96GB model, etc.

I can't quite recall if I implemented (or had the AI implement) aliases -- I think not, but using aliases with the above resolution process, it should allow loading whichever vision model least disturbs the current memory layout / loaded model set.

I think this all works, but it's a complex config tree. E.g.:

  four-x-24gb:
    swap: false
    exclusive: false
    members: [gpu-0, gpu-1, gpu-2, gpu-3]

  gpu-0:
    swap: true
    exclusive: false
    members: []

  gpu-1:
    swap: true
    exclusive: false
    members: []

  gpu-2:
    swap: true
    exclusive: false
    members: []

  gpu-3:
    swap: true
    exclusive: false
    members: []

  # Two 48GB slots (GPU 0+1 and GPU 2+3)
  two-x-48gb:
    swap: false
    exclusive: false
    members:
      - left-48gb
      - right-48gb

[...]

  gpu-3-only:
    swap: true
    exclusive: false
    members:
      - local/kokoro-82M
      - local/Qwen3-Embedding-0.6B
      - llama.cpp/nomic-embed-text-v1.5

  # Full 96GB: all 4 cards together (main models)
  full-96gb:
    swap: true
    exclusive: false
    members:
      - local/MiniMax-M2.5
      - local/Leanstral
      - local/Qwen3.5-397B-A17B
      - local/Qwen3.5-122B-A10B--vllm

Not unmanageable with AI, and probably something that a tool could auto-generate, but not "fun" :)

Sounds like this set-based approach would be pretty similar mathematically, but simpler in expression/config.

One thing I'd encourage is that never-unload1 and never-unload2 would ideally be tags or aliases or subsets themselves here: you never want to unload some (TTS & ASR & LLM & embedding & reranking) model set, for example, and then you want some alias sharing & resolution mechanism that allows you to use whichever one is loaded -- allowing shared aliases, and aliases in the set expressions would provide that. That's probably the intent of having sets be a dict (so defining more than just s, and allowing s to be used as a subset in other sets), but I thought it worth pointing out the use case.

[Stebalien]

This PR solves all my issues. Thanks!