Skip to content

feat: Adds logging when there are no shards during backup and restore#27035

Open
devanbenz wants to merge 5 commits intomain-2.xfrom
db/logging-backup-shards
Open

feat: Adds logging when there are no shards during backup and restore#27035
devanbenz wants to merge 5 commits intomain-2.xfrom
db/logging-backup-shards

Conversation

@devanbenz
Copy link

@devanbenz devanbenz commented Dec 22, 2025

We are experiencing a bug where we have null shards for a given ShardGroup in the boltdb metadata. This seems to show up during backup -> restore. Currently we have not been able to reproduce organically. This PR adds logging to backup and restore handlers for when a given ShardGroup metadata is like the following example:

            {
              "ID": 789,
              "StartTime": "2025-12-01T00:00:00Z",
              "EndTime": "2025-12-08T00:00:00Z",
              "DeletedAt": "0001-01-01T00:00:00Z",
              "Shards": null,
              "TruncatedAt": "0001-01-01T00:00:00Z"
            },

Notice there is no DeletedAt and the Shards are null. This is what we want to avoid.

Example for backup:

ts=2025-12-22T23:09:24.489579Z lvl=warn msg="Backup: ShardGroup has not been deleted and has no shards" log_id=0~z4z0lG000 service=bucket-manifest-writer shard-group-id=2 start-time=2025-12-22T17:23:00.000000Z end-time=2025-12-22T17:24:00.000000Z
ts=2025-12-22T23:09:24.489655Z lvl=warn msg="Backup: ShardGroup has not been deleted and has no shards" log_id=0~z4z0lG000 service=bucket-manifest-writer shard-group-id=3 start-time=2025-12-22T18:11:00.000000Z end-time=2025-12-22T18:12:00.000000Z
ts=2025-12-22T23:09:24.489671Z lvl=warn msg="Backup: ShardGroup has not been deleted and has no shards" log_id=0~z4z0lG000 service=bucket-manifest-writer shard-group-id=4 start-time=2025-12-22T18:12:00.000000Z end-time=2025-12-22T18:13:00.000000Z
ts=2025-12-22T23:09:24.489678Z lvl=warn msg="Backup: ShardGroup has not been deleted and has no shards" log_id=0~z4z0lG000 service=bucket-manifest-writer shard-group-id=5 start-time=2025-12-22T22:58:00.000000Z end-time=2025-12-22T22:59:00.000000Z

Example for restoration:

ts=2025-12-22T23:20:24.971567Z lvl=warn msg="Restore: ShardGroup has not been deleted and has no shards" log_id=0~z5Zsi0000 handler=restore shard-group-id=2 start-time=2025-12-22T17:23:00.000000Z end-time=2025-12-22T17:24:00.000000Z
ts=2025-12-22T23:20:24.971590Z lvl=warn msg="Restore: ShardGroup has not been deleted and has no shards" log_id=0~z5Zsi0000 handler=restore shard-group-id=3 start-time=2025-12-22T18:11:00.000000Z end-time=2025-12-22T18:12:00.000000Z
ts=2025-12-22T23:20:24.971594Z lvl=warn msg="Restore: ShardGroup has not been deleted and has no shards" log_id=0~z5Zsi0000 handler=restore shard-group-id=4 start-time=2025-12-22T18:12:00.000000Z end-time=2025-12-22T18:13:00.000000Z
ts=2025-12-22T23:20:24.971602Z lvl=warn msg="Restore: ShardGroup has not been deleted and has no shards" log_id=0~z5Zsi0000 handler=restore shard-group-id=5 start-time=2025-12-22T22:58:00.000000Z end-time=2025-12-22T22:59:00.000000Z

@devanbenz devanbenz marked this pull request as ready for review December 22, 2025 23:43
@philjb philjb added the area/2.x OSS 2.0 related issues and PRs label Jan 3, 2026
Copy link
Contributor

@davidby-influx davidby-influx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestions for even more logging information.

for _, rpManifest := range rpManifests {
for _, sg := range rpManifest.ShardGroups {
if len(sg.Shards) <= 0 && sg.DeletedAt == nil {
b.logger.Warn("Backup: ShardGroup has not been deleted and has no shards",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we print the database and retention policy (or bucket, or whatever) here?

for _, rps := range b.RetentionPolicies {
for _, sg := range rps.ShardGroups {
if len(sg.Shards) <= 0 && sg.DeletedAt == nil {
h.Logger.Warn("Restore: ShardGroup has not been deleted and has no shards",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As comment above, can we print the database and retention policy, or bucket, or whatever?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/2.x OSS 2.0 related issues and PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants