Add metrics for monitoring PeerForwarder

[asifsmohammed]

Metrics for peer forwarder include:

Gauge:

• peerEndpoints

Counter: (records)

• recordsActuallyProcessedLocally
• recordsToBeProcessedLocally
• recordsToBeForwarded
• recordsFailedForwarding
• recordsSuccessfullyForwarded
• recordsReceivedFromPeers

Counter: (requests)

• requests
• requestsFailed (any request with other than 200 OK response)
• requestsSuccessful
• requests
• requestsTooLarge (buffer)
• requestTimeouts (buffer)
• requestsUnprocessable (buffer not found, NPE)
• badRequests

Timer:

• requestForwardingLatency
• requestProcessingLatency

[dlvenable]

Thanks @asifsmohammed !

I have a few naming suggestions, but I'm open to either.

recordsFailedToBeForwardedCounter -> recordsFailedForwardingCounter

recordsProcesssedLocallyCounter -> recordsActuallyProcessedLocallyCounter

forwardRequestTimer/forwardRequestErrorCounter -> Start with forwarded to stay consistent with forwardedRequestCounter.

internalServerErrorCounter -> We should clarify if this is on the server-side, or when a client receives a 5xx error. Perhaps forwardingServerErrorCounter and receivingServerErrorCounter.

Also, which of these do you consider the most important metrics? It might be useful to get a few of the key ones in for 2.0 and then see if others can be added later.

[asifsmohammed]

I updated the metrics. Naming for HTTP requests start with requests and for records it'll start with records.

[asifsmohammed]

Reopened to add requestsSuccessful metric.