Skip to content

pranaypaine/AI-Voice-Agent

BranchesTags

Repository files navigation

🎤 AI Voice Agent - Multilingual Voice Assistant

A complete open-source AI voice assistant with automatic language detection supporting Hindi, Punjabi, Tamil, Bengali, and English. Built with FastAPI, React, and modern AI models.

✨ Features

  • 🗣️ Real-time Voice Interaction - Speak naturally and get instant responses
  • 🌐 Multilingual Support - Hindi, Punjabi, Tamil, Bengali, English
  • 🤖 Auto Language Detection - Automatically detects the language you're speaking
  • 🎧 Advanced Audio Processing - VAD, noise reduction, and audio enhancement
  • 💬 Text & Voice Input - Type or speak your messages
  • 🔊 Natural TTS - High-quality text-to-speech in multiple languages
  • 📱 Modern Web UI - Responsive React interface with real-time updates
  • 🐳 Docker Ready - One-command deployment with Docker Compose
  • 🔒 Privacy First - Runs completely offline after setup

🚀 Quick Start

One-Command Deployment

# Clone the repository
git clone https://github.com/your-username/AI-Voice-Agent.git
cd AI-Voice-Agent

# Deploy with Docker (recommended)
./deploy.sh

That's it! Open http://localhost in your browser and start talking! 🎉

Manual Setup

If you prefer manual setup or development:

# 1. Setup environment
cp .env.example .env

# 2. Start services
docker-compose up -d

# 3. Setup models
docker-compose exec ollama ollama pull llama3.1

# 4. Access the app
open http://localhost

📋 Requirements

System Requirements

  • RAM: 8GB minimum, 16GB recommended
  • CPU: 4 cores minimum, 8 cores recommended
  • Storage: 20GB free space
  • GPU: Optional NVIDIA GPU for better performance

Software Requirements

  • Docker & Docker Compose
  • Modern web browser with microphone support
  • Internet connection (for initial model downloads)

🏗️ Architecture

graph TD
    A[Web Browser] --> B[React Frontend]
    B --> C[FastAPI Backend]
    C --> D[Audio Processor]
    C --> E[Language Detector]
    C --> F[STT Service]
    C --> G[LLM Service]
    C --> H[TTS Service]
    F --> I[Whisper Models]
    G --> J[Ollama/LLaMA]
    H --> K[Piper/IndicTTS]
Loading

🌍 Language Support

Language Native Code STT LLM TTS Status
English English en ✅ ✅ ✅ Full Support
Hindi हिन्दी hi ✅ ✅ ✅ Full Support
Tamil தமிழ் ta ✅ ✅ ✅ Full Support
Bengali বাংলা bn ✅ ✅ ✅ Full Support
Punjabi ਪੰਜਾਬੀ pa ✅ ✅ ✅ Full Support

🎯 Usage

Voice Interaction

  1. Open http://localhost in your browser
  2. Allow microphone permissions when prompted
  3. Select language or use "Auto Detect"
  4. Click the microphone button
  5. Speak naturally in any supported language
  6. Listen to the AI response

Text Interaction

  1. Click "Or type a message" below the microphone
  2. Type your message in any supported language
  3. Press Enter or click send
  4. Receive text and audio responses

Language Detection

  • Set language to "Auto" for automatic detection
  • System detects language from speech and text
  • Responses generated in the same detected language
  • Real-time language switching supported

⚙️ Configuration

Environment Variables

Edit .env file to customize:

# API Configuration
API_HOST=0.0.0.0
API_PORT=8000
CORS_ORIGINS=http://localhost:3000,http://localhost

# Model Configuration  
OLLAMA_HOST=http://ollama:11434
LLM_MODEL=llama3.1
INDIC_LLM_MODEL=indic-llama

# Audio Settings
SAMPLE_RATE=16000
VAD_AGGRESSIVENESS=2
MAX_FILE_SIZE=10485760

Model Configuration

Customize models in config/config.json:

{
  "models": {
    "whisper": {
      "model_size": "base"
    },
    "ollama": {
      "default_model": "llama3.1",
      "indic_model": "indic-llama"
    }
  }
}

🔧 Development

Local Development Setup

# Backend
cd backend
python -m pip install -r requirements.txt
python main.py

# Frontend  
cd frontend
npm install
npm start

# Ollama (separate terminal)
ollama serve
ollama pull llama3.1

Project Structure

AI-Voice-Agent/
├── backend/                 # FastAPI application
│   ├── app/
│   │   ├── services/       # AI services (STT, LLM, TTS)
│   │   ├── models/         # Data models
│   │   └── utils/          # Utilities
│   └── main.py             # Main application
├── frontend/               # React application
│   ├── src/
│   │   ├── components/     # React components
│   │   └── utils/          # Frontend utilities
│   └── package.json
├── docker/                 # Docker configurations
├── config/                 # Configuration files
├── scripts/               # Setup and utility scripts
├── models/                # AI model storage
└── docker-compose.yml     # Docker orchestration

API Endpoints

  • GET /health - Service health check
  • GET /languages - Supported languages list
  • POST /transcribe - Audio to text transcription
  • POST /chat - Text-based chat completion
  • WS /ws/{client_id} - WebSocket for real-time voice

🐛 Troubleshooting

Common Issues

Microphone Not Working

# Ensure HTTPS in production
# Check browser permissions
# Verify microphone access in browser settings

Models Not Loading

# Check disk space
df -h

# Re-download models
docker-compose exec ollama ollama pull llama3.1

# Check model directory
ls -la models/

Connection Issues

# Check service status
docker-compose ps

# View logs
docker-compose logs backend
docker-compose logs frontend
docker-compose logs ollama

Performance Issues

# Check resource usage
docker stats

# Restart services
docker-compose restart

# Use smaller models for lower-end hardware
# Edit .env: LLM_MODEL=llama3.1:8b

Health Checks

# Backend health
curl http://localhost:8000/health

# Frontend accessibility
curl http://localhost

# Ollama status
curl http://localhost:11434/api/version

# Full system test
./scripts/test_system.sh

📊 Performance

Benchmark Results

  • Response Time: < 2 seconds average
  • Language Detection: < 500ms
  • Audio Processing: Real-time streaming
  • Memory Usage: 4-8GB depending on models
  • CPU Usage: 2-4 cores active processing

Optimization Tips

  • Use GPU for faster model inference
  • Adjust VAD_AGGRESSIVENESS for your environment
  • Use smaller models on resource-constrained systems
  • Enable Redis caching for better performance

🤝 Contributing

We welcome contributions! Here's how:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Development Guidelines

  • Follow Python PEP 8 style guidelines
  • Use TypeScript for new React components
  • Add tests for new features
  • Update documentation for API changes

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • OpenAI Whisper - Speech recognition
  • Ollama - LLM inference
  • Piper TTS - Text-to-speech synthesis
  • AI4Bharat - Indic language models
  • FastAPI - Backend framework
  • React - Frontend framework

📞 Support

🔮 Roadmap

Coming Soon

  • More Languages: Marathi, Gujarati, Telugu
  • Voice Cloning: Custom voice synthesis
  • Mobile App: React Native application
  • API Keys: External model integration
  • Cloud Deploy: One-click cloud deployment

Future Features

  • Multi-turn Conversations: Context awareness
  • Function Calling: Tool integration
  • Speech Translation: Real-time translation
  • Offline Mode: Complete offline operation
  • Plugin System: Extensible functionality

Built with ❤️ for multilingual AI interaction

⭐ Star on GitHub • 🐛 Report Bug • 💡 Request Feature

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages