Change logs support to track all changes being made to managed resources

[jbw976]

Description of your changes

This PR contains the crossplane-runtime changes to support cloud change logs.

• Issue: Proposal: Cloud Change Log crossplane#5477
• Design: design: cloud change logs crossplane#5822

Further PRs for changes to the providers themselves (to consume these runtime changes) and for the change logs sidecar container are coming soon, but should not block this PR from being merged. Previews of those changes can be found in:

• crossplane-contrib/provider-kubernetes@main...jbw976:provider-kubernetes:change-logs
• https://github.com/jbw976/change-log-sidecar

Testing

I have prepared some testing instructions at https://github.com/jbw976/change-log-sidecar/blob/main/README.md that reviewers can try out fairly easily on their local laptops.

Here is some example output that demonstrates what a subset of fields of a change log entry looks like, for example when an object is created, updated, then deleted:

❯ kubectl -n crossplane-system logs -l pkg.crossplane.io/provider=provider-kubernetes -c change-log-sidecar | jq '.timestamp + " " + .provider + " " + .name + " " + .operation'
"2024-08-09T04:48:43Z provider-kubernetes:v0.0.6 object-0 OPERATION_TYPE_CREATE"
"2024-08-09T04:49:22Z provider-kubernetes:v0.0.6 object-0 OPERATION_TYPE_UPDATE"
"2024-08-09T04:50:32Z provider-kubernetes:v0.0.6 object-0 OPERATION_TYPE_DELETE"

A full change log entry looks like the below:

{"apiVersion":"kubernetes.crossplane.io/v1alpha2","externalName":"object-0","kind":"Object","name":"object-0","operation":"OPERATION_TYPE_CREATE","provider":"provider-kubernetes:v0.0.6","snapshot":{"apiVersion":"kubernetes.crossplane.io/v1alpha2","kind":"Object","metadata":{"annotations":{"crossplane.io/composition-resource-name":"resource-object-0","crossplane.io/external-name":"object-0"},"creationTimestamp":"2024-08-09T04:48:43Z","generateName":"traceperf-tester-wnw9g-","generation":2,"labels":{"crossplane.io/claim-name":"traceperf-tester","crossplane.io/claim-namespace":"default","crossplane.io/composite":"traceperf-tester-wnw9g"},"managedFields":[{"apiVersion":"kubernetes.crossplane.io/v1alpha2","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:crossplane.io/composition-resource-name":{}},"f:generateName":{},"f:labels":{"f:crossplane.io/claim-name":{},"f:crossplane.io/claim-namespace":{},"f:crossplane.io/composite":{}},"f:ownerReferences":{"k:{\"uid\":\"c21ef3e7-adb8-4b1f-9b10-b7fbf33bf623\"}":{}}},"f:spec":{"f:forProvider":{"f:manifest":{"f:apiVersion":{},"f:data":{".":{},"f:key":{}},"f:kind":{},"f:metadata":{"f:namespace":{}}}}}},"manager":"apiextensions.crossplane.io/composed/010d7ea543f4d932601bc4472c6b11a93716eecdfa070bfdcea16def1d7efd74","operation":"Apply","time":"2024-08-09T04:48:43Z"},{"apiVersion":"kubernetes.crossplane.io/v1alpha2","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:crossplane.io/external-name":{}}},"f:spec":{"f:readiness":{".":{},"f:policy":{}}}},"manager":"crossplane-kubernetes-provider","operation":"Update","time":"2024-08-09T04:48:43Z"}],"name":"object-0","ownerReferences":[{"apiVersion":"trace-perf.crossplane.io/v1alpha1","blockOwnerDeletion":true,"controller":true,"kind":"XTracePerf","name":"traceperf-tester-wnw9g","uid":"c21ef3e7-adb8-4b1f-9b10-b7fbf33bf623"}],"resourceVersion":"998","uid":"905af6bb-eb03-4569-9fe1-b257fa8019e9"},"spec":{"deletionPolicy":"Delete","forProvider":{"manifest":{"apiVersion":"v1","data":{"key":"value-0-tester-foo"},"kind":"ConfigMap","metadata":{"namespace":"default"}}},"managementPolicies":["*"],"providerConfigRef":{"name":"default"},"readiness":{"policy":"SuccessfulCreate"}},"status":{"atProvider":{"manifest":null}}},"timestamp":"2024-08-09T04:48:43Z"}

I have:

• Read and followed Crossplane's contribution process.
• Run earthly +reviewable to ensure this PR is ready for review.
• Added or updated unit tests.
• Linked a PR or a docs tracking issue to document this change.
• Added backport release-x.y labels to auto-backport this PR.

Need help with this checklist? See the cheat sheet.

[jbw976]

Pushed 2dd9458 to address all the protobuf linter errors found in https://github.com/crossplane/crossplane-runtime/actions/runs/10229176882/job/28302502501?pr=754.

Looks like all those are green now in https://github.com/crossplane/crossplane-runtime/actions/runs/10257477240/job/28378586893?pr=754 🟢

[negz]

A few last small nits, but this looks great to me. Thanks @jbw976!

[jbw976]

@negz I've got all of your feedback integrated now, thanks for the awesome reviews dude! 😎

Pay special attention to #754 (comment) (using WaitForReady(), since that's not something you asked for, but I do think it's an improvement personally.

I've pushed all changes to my provider-kubernetes fork and the https://github.com/jbw976/change-log-sidecar prototype server implementation:

• crossplane-contrib/provider-kubernetes@main...jbw976:provider-kubernetes:change-logs
• https://github.com/jbw976/change-log-sidecar

Testing instructions and published images are also up to date in https://github.com/jbw976/change-log-sidecar/blob/main/README.md. Let me know if there's anything else I can improve with this PR! 🙇‍♂️

[negz]

Thanks @jbw976! One final thing I forgot, but otherwise this looks good. Feel free to squash and merge once you've added the timestamp.

[jbw976]

Awesome, 0 unresolved conversations! I will likely squash/merge this today after a bit more testing 😎