reworking toolbox dumping logic

[upodroid]

This PR tweaks the instance dumping logic:

1. Prioritises instances that fail to join the cluster
2. Use the control plane instances as an SSH bastion if a bastion doesn't exist and the control plane instances have a public IP

We need this for troubleshooting infra flakes with kops

[hakman]

Stores ssh logs by instance names instead of IPs

Not sure this makes things better or worse. IPs make it obvious that the node has not registered.

[upodroid]

Stores ssh logs by instance names instead of IPs

Not sure this makes things better or worse. IPs make it obvious that the node has not registered.

kops validate logs failures like this:

VALIDATION ERRORS
KIND	NAME																MESSAGE
Machine	https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-027/zones/us-west1-b/instances/nodes-us-west1-b-dksv	machine "https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-027/zones/us-west1-b/instances/nodes-us-west1-b-dksv" has not yet joined cluster

looking by nodes should make it easier

[hakman]

Stores ssh logs by instance names instead of IPs

Not sure this makes things better or worse. IPs make it obvious that the node has not registered.

kops validate logs failures like this:

VALIDATION ERRORS
KIND	NAME																MESSAGE
Machine	https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-027/zones/us-west1-b/instances/nodes-us-west1-b-dksv	machine "https://www.googleapis.com/compute/v1/projects/k8s-infra-e2e-boskos-027/zones/us-west1-b/instances/nodes-us-west1-b-dksv" has not yet joined cluster

looking by nodes should make it easier

The usual workflow is to just look at nodes why they failed. It is not really useful to know the ID of the failed one if you just know it will be one with IP address.

[upodroid]

I'll change it back to ip

[upodroid]

/test presubmit-kops-gce-small-scale-ipalias-using-cl2

[upodroid]

ready to be merged, it's working correctly

[upodroid]

/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment