[core][autoscaler] Retry GCP project metadata updates on HTTP 412 errors (ray-project#60429)

When the autoscaler tries to launch a Ray cluster on GCP, it puts a new
SSH key into the project metadata if necessary. The update may results
into an HTTP 412 precondition failure if there are concurrent tries to
update the metadata. The error will look like this:

```python
googleapiclient.errors.HttpError: <HttpError 412 when requesting https://compute.googleapis.com/compute/v1/projects/my_gcp_project/setCommonInstanceMetadata?alt=json returned "Supplied fingerprint does not match current metadata fingerprint.". Details: "[{'message': 'Supplied fingerprint does not match current metadata fingerprint.', 'domain': 'global', 'reason': 'conditionNotMet', 'location': 'If-Match', 'locationType': 'header'}]">
```

The error can only be resolved by retrying. Therefore, to provide a
better user experience, this PR does the retry for the users
automatically:
1. Catch the error.
2. Reload the metadata and update it again.

---------

Signed-off-by: Rueian Huang <rueiancsie@gmail.com>
Signed-off-by: peterxcli <peterxcli@gmail.com>

Author: rueian

python/ray/autoscaler/_private/gcp/config.py

@@ -549,7 +549,24 @@ def _configure_key_pair(config, compute):
             )
             public_key, private_key = generate_rsa_key_pair()
 
-            _create_project_ssh_key_pair(project, public_key, ssh_user, compute)
+            for attempt in range(MAX_POLLS):
+                try:
+                    _create_project_ssh_key_pair(project, public_key, ssh_user, compute)
+                    break
+                except errors.HttpError as e:
+                    if e.resp.status != 412 or attempt == MAX_POLLS - 1:
+                        raise
+                    logger.warning(
+                        "GCP project metadata update conflict for %s (%s); retrying",
+                        config["provider"]["project_id"],
+                        e,
+                    )
+                    time.sleep(POLL_INTERVAL)
+                    project = (
+                        compute.projects()
+                        .get(project=config["provider"]["project_id"])
+                        .execute()
+                    )
 
             # Create the directory if it doesn't exists
             private_key_dir = os.path.dirname(private_key_path)