Improvement: Add Backoff for Rate Limits

[s4ke]

Currently it can happen that for whatever reason the requests to the Hetzner Cloud API are going a bit crazy, causing 429 errors.

Currently, once a rate limit is reached, we keep hammering the rate API and we never get out of the rate limit issue.

[s4ke]

To me it is still a bit unclear whether we want to just communicate back the error to the docker engine directly or whether we want to wrap the calling code in https://github.com/cenkalti/backoff to automatically do the backoff.

[s4ke]

Prototype here: master...neuroforgede:docker-volume-hetzner:master

Currently evaluating.

[s4ke]

@costela What are your thoughts on this?

[s4ke]

@costela ping

[costela]

sorry, I'm bit swamped, as always 😓

I understand the motivation, but I'm not sure about the particular solution. Why not something a bit more general like WithBackoffFunc?

[s4ke]

No worries.

If I read the docs for WithBackoffFunc correctly this is used for automatic retrying of requests. We don't want automatic retries though, right? I'd rather have the volume creation fail?

[costela]

ah, you mean the fact that the retries are endless? Yes, that's a bit of a problem. But I'd suggest making that improvement at the hcloud-go side? Adding a way to stop retrying?

That feels like the better way to put this?

[s4ke]

Hmm, this looks new: hetznercloud/hcloud-go#470 but depending on the call patterns this could not be enough.

[s4ke]

Theoretically, this could also be improved on the moby side:

https://github.com/moby/moby/blob/d8b7ae78c0bcaf296609d6b45bc5df9a1d48eea8/pkg/plugins/client.go#L20C2-L20C17

[s4ke]

@costela Maybe it would be better to maintain on each node how many volumes are attached and if there is no slot left, the driver should just fail to create the volume? Right now we do this by using generic resources, but this should be easy enough to do in this driver, no?

Because honestly, this is the main reason this happens to beging with.

[costela]

Yes, if we can reliably detect the situation, then it would make sense to short-circuit the error. I'm a bit wary of hard-coding any hetzner constants (e.g. max number of slots) in the driver though, since those could change.