feat: pre-warm preview cache for indexed photos#3494
Open
karlitschek wants to merge 1 commit into
Open
Conversation
The first scroll over a fresh library is the slowest user-facing
moment in photos: every visible tile fires `/api/v1/preview/{fileId}`
cold, NC re-encodes the thumbnail on demand at 64×64 and 1024×1024,
and a few dozen parallel ~200-500ms preview generations stutter the
load. After that the NC preview cache kicks in and everything is
fast.
Drains that cost into a background job so the user never sees it.
Schema
- New `oc_photos_preview_warmup` table — file_id PK, state
(pending → warming → warmed / failed), attempts, updated_at.
PK is per-file (not per-user) because NC's preview cache is
global; warming once covers every user that mounts the file.
Service
- `PhotoPreviewWarmupMapper` mirrors `PhotoTranscodeMapper` but
hands back batches of fileIds rather than one at a time —
warming is sub-second per file so per-file claim round trips
would dominate the wall budget.
- `PhotoPreviewWarmupService::warmFile` calls
`IPreview::getPreview` at exactly the two sizes FileComponent
asks for (64 and 1024). Idempotent: re-warming an already-warm
file is a stat-and-return.
- `PREVIEW_SIZES` is the contract — if photos changes the layer
sizes in FileComponent, this constant has to follow or the
warmup misses.
- Disabled via `enable_preview_warmup` AppValue (default `true` —
the perf win is the whole point). Admins on tight quotas
(warming a 100k library adds ~3-5 GB to NC's appdata preview
cache) leave it off.
Worker
- `PhotoPreviewWarmupJob` (TimedJob, 5-min interval, 5-min wall
budget). Per tick: reap stale `warming` rows from a dead worker
→ backfill new pending rows from `oc_photos_index` (one-time
bootstrap; the listener takes over for new uploads) → drain
the queue in batches of 50.
Listener
- `PhotoIndexNodeListener.onWritten` now also marks the file
pending warmup (idempotent — `markPending` is a no-op for
already-pending / warmed files).
- `onDeleted` drops the warmup row alongside the index + transcode
cleanup.
Bumped app version to 7.0.0-dev.2 so `occ upgrade` re-runs the
migration loader and picks the new table + job up.
Signed-off-by: Frank Karlitschek <frank@nextcloud.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first scroll over a fresh library is the slowest user-facing moment in photos: every visible tile fires
/api/v1/preview/{fileId}cold, NC re-encodes the thumbnail on demand at 64×64 and 1024×1024, and a few dozen parallel ~200-500ms preview generations stutter the load. After that the NC preview cache kicks in and everything is fast.Drains that cost into a background job so the user never sees it.
Schema
oc_photos_preview_warmuptable — file_id PK, state (pending → warming → warmed / failed), attempts, updated_at. PK is per-file (not per-user) because NC's preview cache is global; warming once covers every user that mounts the file.Service
PhotoPreviewWarmupMappermirrorsPhotoTranscodeMapperbut hands back batches of fileIds rather than one at a time — warming is sub-second per file so per-file claim round trips would dominate the wall budget.PhotoPreviewWarmupService::warmFilecallsIPreview::getPreviewat exactly the two sizes FileComponent asks for (64 and 1024). Idempotent: re-warming an already-warm file is a stat-and-return.PREVIEW_SIZESis the contract — if photos changes the layer sizes in FileComponent, this constant has to follow or the warmup misses.enable_preview_warmupAppValue (defaulttrue— the perf win is the whole point). Admins on tight quotas (warming a 100k library adds ~3-5 GB to NC's appdata preview cache) leave it off.Worker
PhotoPreviewWarmupJob(TimedJob, 5-min interval, 5-min wall budget). Per tick: reap stalewarmingrows from a dead worker → backfill new pending rows fromoc_photos_index(one-time bootstrap; the listener takes over for new uploads) → drain the queue in batches of 50.Listener
PhotoIndexNodeListener.onWrittennow also marks the file pending warmup (idempotent —markPendingis a no-op for already-pending / warmed files).onDeleteddrops the warmup row alongside the index + transcode cleanup.Bumped app version to 7.0.0-dev.2 so
occ upgradere-runs the migration loader and picks the new table + job up.