Skip to content

Add separate PVC for tiered storage cache#239

Open
eduardagarici wants to merge 11 commits intomasterfrom
CORE-152486
Open

Add separate PVC for tiered storage cache#239
eduardagarici wants to merge 11 commits intomasterfrom
CORE-152486

Conversation

@eduardagarici
Copy link
Copy Markdown

Add tiered storage cache as a separate PVC

Summary

  • Introduces a tieredStorageCache: true flag on StorageConfig that marks a volume as a Kafka tiered storage cache (e.g. rsm.config.fetch.chunk.cache.path). Cache volumes are provisioned as independent PVCs, separate from
    Kafka log volumes.
  • Cache volumes are excluded from log.dirs so Kafka does not treat them as data directories.
  • Cache volumes are excluded from Cruise Control capacity calculations and disk rebalance operations — CC must not account for ephemeral storage.
  • Controller-only nodes skip cache PVCs entirely since they do not serve broker traffic.
  • Shrink resize: because cache data is ephemeral, koperator supports shrinking a cache PVC via a safe delete-and-recreate flow coordinated with the rolling upgrade machinery. The old PVC is annotated pending-deletion, a
    replacement is created immediately (provisioning overlaps with gate evaluation), and the old PVC is deleted once the broker pod stops. A grow continues to use the standard Kubernetes in-place expansion path. Full details in
    docs/tiered-storage-pvc-resize.md.

Test plan

  • testTieredStorageCachePvcResize passes against a local kind cluster — verifies the full shrink lifecycle (staging, pod cycle, annotation cleanup, correct final size)
  • Cache volume does not appear in log.dirs on the broker
  • CC capacity config does not include the cache mount path
  • Normal log volume grow/shrink behaviour is unaffected

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants