fix: prevent duplicate model downloads with atomic check-and-set

[ianliuy]

What's broken?

When a new ClusterBaseModel is created with --num-download-worker=2 or more, two download workers both start downloading the same model simultaneously. This duplicates all model shards and can produce a corrupted config.json from concatenated writes.

Who is affected?

Any OME deployment with multiple download workers (--num-download-worker>=2). Single-worker deployments are unaffected.

When does it trigger?

When a Download and DownloadOverride task for the same model are processed within milliseconds of each other — both workers enter processTask() before either registers in activeDownloads.

Where is the bug?

pkg/modelagent/gopher.go, lines 278-284 (before this fix). The context creation and activeDownloads registration were separate from any existence check:

// Both workers reach here before either registers
ctx, cancel = context.WithCancel(context.Background())
s.activeDownloadsMutex.Lock()
s.activeDownloads[modelUID] = cancel  // No check before registration
s.activeDownloadsMutex.Unlock()

Why does it happen?

The check-then-act on activeDownloads was not atomic. Worker 1 could create its context and be about to lock the mutex, while Worker 2 also creates its context — neither has registered yet, so neither sees the other.

How did we fix it?

Atomic check-and-set inside the mutex: lock → check if model is already downloading → if not, create context and register → unlock. If another worker is already downloading the model, the duplicate task returns early with an info log.

s.activeDownloadsMutex.Lock()
if _, isDownloading := s.activeDownloads[modelUID]; isDownloading {
    s.activeDownloadsMutex.Unlock()
    s.logger.Infof("Model %s is already being downloaded, skipping duplicate download", modelInfo)
    return nil
}
ctx, cancel = context.WithCancel(context.Background())
s.activeDownloads[modelUID] = cancel
s.activeDownloadsMutex.Unlock()

This is the minimal fix — 1 file, 12 insertions / 8 deletions. The existing defer cleanup and Delete task cancellation logic are unchanged.

How do we know it works?

The fix ensures the TOCTOU window is eliminated: no two workers can both pass the "not downloading" check for the same model, because the check and registration happen inside the same lock acquisition.

Fixes #308

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!