NRG: Fix cluster size drop to 1 on replaying EntryAddPeer after restart

[sciascid]

On restart, replaying EntryAddPeer could incorrectly leave a raft node at cluster size 1 instead of restoring the expected size and quorum from persisted state.
This bug could lead to the following scenario: a node in a 3 node cluster could restart, reset set cluster size to 1. If the node did not receive any message from other nodes, it could campaign to become leader. Being in a single node cluster, it would win the election. Resulting in the original cluster splitting into two clusters (or two leaders at the same time).
Specifically, if an EntryAddPeer was replayed on from the log, it would overwrite the cluster size and quorum to 1. The peer set is now restored before the log is replayed, and it is taken from the snapshot (if no snapshot is present then we fallback to peer.idx).
If a log entry that changes membership is replayed, it will now update the cluster and quorum size correctly.

Signed-off-by: Daniele Sciascia daniele@nats.io

[sciascid]

Notice that before this PR initSingleMemRaftNode would start in a weird configuration. The node would have cluster size = 3, but only 1 node its peer list.

With the changes in this PR we go back to a sane initial state:

  func TestNRGInitSingleMemRaftNodeDefaults(t *testing.T) {
          n, cleanup := initSingleMemRaftNode(t)
          defer cleanup()
          require_Equal(t, n.ID(), "esFhDys3")
          require_Equal(t, len(n.Peers()), 1)
          require_Equal(t, n.Peers()[0].ID, "esFhDys3")
          require_Equal(t, n.ClusterSize(), 1)
          require_True(t, n.Quorum())
  }

Some tests were relying on that initial configuration. Which explains why some tests need tweaking.

[MauriceVanVeen]

LGTM!

[neilalexander]

LGTM

[neilalexander]

@sciascid Looks like there's now a merge conflict in the tests, mind rebasing & resolving please?

[sciascid]

@neilalexander Done