AI Test Architect - RAG + MCP System

A complete AI-driven Test Architect system that generates API and UI test cases automatically using Retrieval-Augmented Generation (RAG) and Model Context Protocol (MCP) tools.

🎯 Overview

This project demonstrates how to build an intelligent test generation system that:

• Ingests various documentation sources (OpenAPI specs, wikis, business docs, UI docs)
• Uses RAG to find relevant context for test generation
• Generates canonical API test definitions
• Exports tests to multiple formats (Postman, RestAssured)
• Provides MCP tools for integration with AI assistants like Claude

🏗️ Architecture

Components

1. RAG Service (Python + FastAPI)

   • Document ingestion and embedding
   • Vector similarity search
   • API test generation
   • Structured logging with correlation IDs
   • Data persistence via Docker volumes

2. MCP Server (TypeScript + Node.js)

   • MCP tool layer for AI assistants
   • OpenAPI spec parsing
   • Format converters (Postman, RestAssured)
   • RAG service client

3. Sample Data

   • OpenAPI specifications (auth, customers)
   • Wiki documentation
   • Business requirements
   • UI specifications
   • Sequence diagrams

Data Flow

Documentation → RAG Service → Vector Embeddings → Test Generation
                      ↓
                MCP Tools → Export Formats (Postman/RestAssured)

🚀 Quick Start

Prerequisites

• Docker and Docker Compose
• 4GB+ RAM available
• Port 8000 and 3000 available

Start the System

# Build and start all services
docker-compose up --build

# Wait for health check to pass (check logs)
docker-compose logs -f rag-service

Verify Services

# Check RAG service health
curl http://localhost:8000/health

# Expected response:
# {"status":"healthy","timestamp":"...","document_count":0}

# View API documentation
open http://localhost:8000/docs

Ingest Sample Data

# Run ingestion script
docker-compose exec rag-service python ingest_docs.py

# Check ingestion results in logs
docker-compose logs rag-service | grep "Total documents ingested"

Expected output: ~30-40 documents ingested from various sources.

📚 Usage Examples

Using FastAPI Directly

1. Search for Relevant Documentation

curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How does login work?",
    "top_k": 3
  }'

2. Generate API Tests

curl -X POST http://localhost:8000/api_generate_tests \
  -H "Content-Type: application/json" \
  -d '{
    "query": "POST /login",
    "profile": "smoke"
  }'

Response includes canonical test definitions:

{
  "query": "POST /login",
  "profile": "smoke",
  "tests": [
    {
      "id": "API_LOGIN_SMOKE_01",
      "name": "Valid login - happy path",
      "priority": "P0",
      "endpoint": {"method": "POST", "path": "/login"},
      "request": {
        "headers": {"Content-Type": "application/json"},
        "body": {"email": "[email protected]", "password": "Correct123!"}
      },
      "expected": {
        "status": 200,
        "bodyChecks": ["has property token", "has property refreshToken"]
      }
    }
  ]
}

Using MCP Tools (via Claude or other AI assistants)

Once the MCP server is configured with your AI assistant:

> Generate smoke tests for POST /login endpoint

> Export those tests as a Postman collection

> Generate functional tests for the customers API

> Export as RestAssured Java tests

🧪 Test Profiles

The system supports different test coverage profiles:

Profile	Coverage	Use Case
smoke	Happy path only	Quick sanity checks
functional	Positive + negative cases	Standard coverage
regression	Deep coverage + edge cases	Comprehensive testing
security	Auth, injection, XSS tests	Security validation
bugfix	Targeted tests	Bug verification

🔧 API Endpoints

RAG Service (Port 8000)

Endpoint	Method	Description
/health	GET	Health check
/ingest	POST	Ingest documents
/search	POST	Search for similar docs
/api_generate_tests	POST	Generate API tests
/stats	GET	Vector store statistics
/docs	GET	Swagger UI

MCP Tools

Tool	Description
scan_api_specs	Find OpenAPI files
index_api_spec	Ingest OpenAPI spec
ingest_docs_from_folders	Ingest all documentation
api_generate_tests	Generate tests with RAG
api_export_postman	Export to Postman
api_export_restassured	Export to RestAssured
ui_generate_playwright_instructions	Generate UI tests

🧪 Running Tests

RAG Service Tests (Python)

# Run all tests with coverage
docker-compose exec rag-service pytest --cov=. --cov-report=term

# Run specific test file
docker-compose exec rag-service pytest tests/test_embeddings.py -v

# Generate HTML coverage report
docker-compose exec rag-service pytest --cov=. --cov-report=html

MCP Server Tests (TypeScript)

# Run all tests with coverage
docker-compose exec mcp-test-architect npm test -- --coverage

# Run in watch mode
docker-compose exec mcp-test-architect npm run test:watch

# Run specific test file
docker-compose exec mcp-test-architect npm test -- converters.test.ts

📊 Monitoring & Debugging

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f rag-service
docker-compose logs -f mcp-test-architect

# Filter by level
docker-compose logs rag-service | grep ERROR

Structured Logging

All logs are in JSON format with:

• timestamp: ISO 8601 timestamp
• level: DEBUG, INFO, WARNING, ERROR
• correlation_id: Request tracing ID
• service_name: Service identifier
• message: Log message
• Additional context fields

Example:

{
  "timestamp": "2025-01-08T10:30:00.000Z",
  "level": "INFO",
  "service_name": "rag-service",
  "correlation_id": "abc123",
  "message": "Documents ingested successfully",
  "count": 25
}

Health Checks

# RAG service health
curl http://localhost:8000/health

# Check vector store stats
curl http://localhost:8000/stats

📁 Project Structure

.
├── rag-service/              # Python FastAPI backend
│   ├── Dockerfile
│   ├── main.py              # FastAPI app + endpoints
│   ├── models.py            # Pydantic models
│   ├── vector_store.py      # Vector search + persistence
│   ├── fake_embedding.py    # Deterministic embeddings
│   ├── test_generator.py    # Test generation logic
│   ├── logger.py            # Structured logging
│   ├── ingest_docs.py       # Document ingestion script
│   ├── requirements.txt
│   └── tests/               # Unit + integration tests
│
├── mcp-test-architect/       # TypeScript MCP server
│   ├── Dockerfile
│   ├── src/
│   │   ├── server.ts        # MCP server + tools
│   │   ├── rag_client.ts    # HTTP client for RAG service
│   │   ├── converters.ts    # Postman/RestAssured exporters
│   │   ├── openapi_parser.ts
│   │   ├── logger.ts
│   │   └── types.ts
│   ├── tests/               # Unit tests
│   └── package.json
│
├── specs/                    # OpenAPI specifications
│   ├── auth-openapi.yaml
│   └── customers-openapi.yaml
│
├── wiki/                     # Wiki markdown exports
├── gdocs/                    # Google Docs exports
├── word/                     # Word document exports
├── ui_docs/                  # UI documentation
├── diagrams/                 # Sequence diagrams
├── data/                     # Persisted vector store (volume)
│
├── docker-compose.yml        # Orchestration
└── README.md                # This file

🛠️ Development

Adding New Documentation

1. Place files in appropriate folder (wiki/, gdocs/, etc.)
2. Re-run ingestion:

   docker-compose exec rag-service python ingest_docs.py

Modifying Test Generation

Edit rag-service/test_generator.py:

• _generate_smoke_tests() - Happy path tests
• _generate_functional_tests() - Functional tests
• _generate_security_tests() - Security tests
• etc.

Adding Export Formats

Edit mcp-test-architect/src/converters.ts:

• Add new conversion method
• Update MCP tools in server.ts

Environment Variables

RAG Service

• LOG_LEVEL: Logging level (DEBUG, INFO, WARNING, ERROR)
• DATA_DIR: Data persistence directory
• CORS_ORIGINS: Allowed CORS origins

MCP Server

• RAG_BASE_URL: RAG service URL
• LOG_LEVEL: Logging level
• MCP_PORT: Server port

🎓 Learning Outcomes

This project demonstrates:

1. Vector Embeddings: Deterministic fake embeddings using character frequency and hashing
2. Cosine Similarity: Finding relevant documents through vector similarity
3. RAG Pattern: Retrieving context to enhance generation quality
4. MCP Protocol: Building tool layers for AI assistant integration
5. Docker-First Development: Containerized development and deployment
6. Structured Logging: JSON logs with correlation IDs for tracing
7. Data Persistence: Docker volumes for stateful services
8. Test-Driven Development: 70%+ code coverage with pytest and vitest

🔍 Troubleshooting

Services Won't Start

# Check logs for errors
docker-compose logs

# Rebuild from scratch
docker-compose down -v
docker-compose build --no-cache
docker-compose up

No Documents Found

# Verify ingestion
docker-compose exec rag-service python ingest_docs.py

# Check stats
curl http://localhost:8000/stats

Tests Failing

# Rebuild and run tests
docker-compose build rag-service
docker-compose exec rag-service pytest -v

# Check test dependencies
docker-compose exec rag-service pip list

Port Already in Use

# Change ports in docker-compose.yml
ports:
  - "8001:8000"  # Change host port

📖 API Documentation

Full API documentation available at: http://localhost:8000/docs (Swagger UI)

🤝 Contributing

This is a learning/demonstration project. Feel free to extend it with:

• Actual LLM integration (OpenAI, Anthropic)
• More export formats (k6, Newman, Cypress)
• Additional test profiles
• Real embedding models
• Production deployment configs

📝 License

MIT License - Feel free to use for learning and experimentation.

🎉 Acknowledgments

• Built using FastAPI, Model Context Protocol, and TypeScript
• Inspired by modern RAG and test automation patterns
• Designed for learning vector embeddings and MCP integration

---

Happy Testing! 🚀

For questions or issues, check the logs with docker-compose logs -f