[TESTING][FUNCTIONALITY]: Observability manual test plan (metrics, logging, tracing, health)

[crivetimihai]

📊 [TESTING][FUNCTIONALITY]: Observability Manual Test Plan

Goal

Produce a comprehensive manual test plan for observability including metrics, logging, tracing, and health endpoints.

Why Now?

Observability enables operational insight:

1. Metrics: Measure system performance
2. Logging: Debug issues and audit
3. Tracing: Track requests across services
4. Health: Monitor system status

---

📖 User Stories

US-1: SRE - System Monitoring

As an SRE
I want comprehensive observability
So that I can monitor and troubleshoot

Acceptance Criteria:

Feature: System Monitoring

  Scenario: Access metrics
    Given Prometheus is configured
    When I scrape /metrics
    Then I should see gateway metrics

  Scenario: Access logs
    Given structured logging is enabled
    When I query logs
    Then I should find request details

---

🏗 Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                      OBSERVABILITY ARCHITECTURE                              │
└─────────────────────────────────────────────────────────────────────────────┘

    GATEWAY                EXPORTERS                 BACKENDS
    ───────                ─────────                 ────────

  ┌──────────────┐       ┌──────────────┐       ┌──────────────┐
  │  Gateway     │       │  /metrics    │──────▶│  Prometheus  │
  │              │       └──────────────┘       └──────────────┘
  │              │       ┌──────────────┐       ┌──────────────┐
  │              │──────▶│  Logs        │──────▶│  Loki/ELK    │
  │              │       └──────────────┘       └──────────────┘
  │              │       ┌──────────────┐       ┌──────────────┐
  │              │       │  Traces      │──────▶│  Jaeger      │
  └──────────────┘       └──────────────┘       └──────────────┘

---

📋 Test Environment Setup

export GATEWAY_URL="http://localhost:8000"
export LOG_LEVEL=DEBUG
export STRUCTURED_LOGGING_ENABLED=true
export TOKEN=$(python -m mcpgateway.utils.create_jwt_token \
  --username tester@example.com --secret "$JWT_SECRET")

---

🧪 Manual Test Cases

Section 1: Metrics

Case	Scenario	Endpoint	Expected	Validation
MT-01	Prometheus metrics	/metrics	Metrics	Prometheus format
MT-02	Request counters	After requests	Incremented	Count increases
MT-03	Latency histograms	After requests	Buckets	Distribution

MT-01: Prometheus Metrics Endpoint

Steps:

# Step 1: Fetch metrics
curl -s "$GATEWAY_URL/metrics" | head -50

# Step 2: Check for gateway-specific metrics
curl -s "$GATEWAY_URL/metrics" | grep "mcpgateway_"

# Step 3: Verify Prometheus format
curl -s "$GATEWAY_URL/metrics" | grep "# HELP"
curl -s "$GATEWAY_URL/metrics" | grep "# TYPE"

Expected Result:

• Prometheus exposition format
• Gateway-specific metrics present
• HELP and TYPE annotations

Section 2: Logging

Case	Scenario	Level	Expected	Validation
LG-01	Structured logs	JSON	Valid JSON	Parseable
LG-02	Request logging	Request	Request ID	Traceable
LG-03	Error logging	Error	Stack trace	Debuggable

LG-01: Structured JSON Logging

Steps:

# Step 1: Make request
curl -s "$GATEWAY_URL/gateways" \
  -H "Authorization: Bearer $TOKEN" > /dev/null

# Step 2: Check logs (assuming stdout or file)
# Logs should be JSON formatted
docker logs mcpgateway 2>&1 | tail -5 | jq .

# Step 3: Verify log fields
# Should have: timestamp, level, message, request_id

Expected Result:

• Logs in JSON format
• Contains expected fields
• Request ID for correlation

Section 3: Health Endpoints

Case	Scenario	Endpoint	Expected	Validation
HE-01	Liveness	/health/live	200 OK	Alive
HE-02	Readiness	/health/ready	200/503	Ready state
HE-03	Dependencies	/health	Details	Component status

HE-03: Health with Dependencies

Steps:

# Step 1: Get full health
curl -s "$GATEWAY_URL/health" | jq .

# Step 2: Check dependency status
curl -s "$GATEWAY_URL/health" | jq '.dependencies'

# Expected: database, redis, etc. with status

Expected Result:

• Overall health status
• Individual dependency status
• Degraded if any dependency down

Section 4: Tracing

Case	Scenario	Header	Expected	Validation
TR-01	Trace propagation	X-Request-ID	Propagated	Same ID in logs
TR-02	Span creation	Request	Spans	In Jaeger

TR-01: Trace ID Propagation

Steps:

# Step 1: Make request with trace ID
curl -s "$GATEWAY_URL/gateways" \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Request-ID: test-trace-12345" > /dev/null

# Step 2: Check logs for trace ID
docker logs mcpgateway 2>&1 | grep "test-trace-12345"

Expected Result:

• Trace ID appears in logs
• All related log entries have same ID

---

📊 Test Matrix

Test Case	Metrics	Logging	Health	Tracing
MT-01	✓
MT-02	✓
MT-03	✓
LG-01		✓
LG-02		✓		✓
LG-03		✓
HE-01			✓
HE-02			✓
HE-03			✓
TR-01				✓
TR-02				✓

---

✅ Success Criteria

• All 11 test cases pass
• Metrics endpoint works
• Structured logging enabled
• Health endpoints respond correctly
• Trace IDs propagate

---

🔗 Related Files

• mcpgateway/middleware/logging.py
• mcpgateway/services/health_service.py

---

🔗 Related Issues

• [TESTING][OPERATIONS]: Health Monitoring Manual Test Plan (Liveness, Readiness, Dependencies) #2462 - Health Monitoring
• [TESTING][OBSERVABILITY]: Metrics Accuracy, Tracing Completeness, and Dashboard Validation #2476 - Observability accuracy