README.md

# Unified AI Services

A comprehensive AI platform that integrates Named Entity Recognition (NER), Optical Character Recognition (OCR), and Retrieval-Augmented Generation (RAG) services into a unified application.

## 🌟 Features

### Core Services
- **NER Service** (Port 8500): Advanced named entity recognition with relationship extraction
- **OCR Service** (Port 8400): Document processing with Azure Document Intelligence
- **RAG Service** (Port 8401): Vector search and document retrieval
- **Unified App** (Port 8000): Coordinated workflows and service management

### Key Capabilities
- ✅ Multi-language support (Thai + English)
- ✅ Complex relationship extraction 
- ✅ Entity deduplication
- ✅ Graph database exports (Neo4j, GraphML, GEXF)
- ✅ Vector search with semantic similarity
- ✅ Document processing (PDF, images, text)
- ✅ Real-time service health monitoring
- ✅ Unified workflows combining all services
- ✅ Comprehensive API documentation

## 🚀 Quick Start

### Prerequisites
- Python 3.8 or higher
- PostgreSQL with vector extension support
- Azure OpenAI account
- Azure Document Intelligence account
- DeepSeek API account (for advanced NER)

### Automated Setup

1. **Clone and navigate to the project directory**
   ```bash

   cd unified-ai-services

   ```

2. **Run the automated setup**
   ```bash

   python setup.py

   ```
   
   This will:
   - Check your Python environment
   - Create necessary directories
   - Help you configure .env file
   - Install dependencies
   - Validate configuration
   - Create startup scripts

3. **Start the unified application**
   ```bash

   python app.py

   ```
   
   Or use the generated scripts:
   - Windows: `start_services.bat`
   - Unix/Linux/Mac: `./start_services.sh`

4. **Run comprehensive tests**
   ```bash

   python test_unified.py

   ```
   
   Or use the generated scripts:
   - Windows: `run_tests.bat`
   - Unix/Linux/Mac: `./run_tests.sh`

### Manual Setup

If you prefer manual setup:

1. **Install dependencies**
   ```bash

   pip install -r requirements.txt

   ```

2. **Create .env file** (copy from .env.example)
   ```bash

   cp .env.example .env

   # Edit .env with your configuration

   ```

3. **Set up directories**
   ```bash

   mkdir -p services exports logs temp tests data

   ```

4. **Place service files in the services directory**
   ```

   services/

   ├── ner_service.py

   ├── ocr_service.py

   └── rag_service.py

   ```

## 📁 Project Structure

```

unified-ai-services/

├── app.py                    # Main unified application

├── configs.py               # Centralized configuration

├── setup.py                 # Automated setup script

├── requirements.txt         # Python dependencies

├── test_unified.py          # Comprehensive test suite

├── .env                     # Environment configuration

├── services/                # Individual service files

│   ├── ner_service.py      # NER service implementation

│   ├── ocr_service.py      # OCR service implementation

│   └── rag_service.py      # RAG service implementation

├── exports/                 # Generated export files

├── logs/                    # Application logs

├── temp/                    # Temporary files

├── tests/                   # Additional test files

└── data/                    # Data files

```

## ⚙️ Configuration

### Environment Variables

The system uses a `.env` file for configuration. Key variables include:

#### Server Configuration
```bash

HOST=0.0.0.0

DEBUG=True

MAIN_PORT=8000

NER_PORT=8500

OCR_PORT=8400

RAG_PORT=8401

```

#### Database Configuration
```bash

POSTGRES_HOST=your-postgres-server.com

POSTGRES_PORT=5432

POSTGRES_USER=your-username

POSTGRES_PASSWORD=your-password

POSTGRES_DATABASE=postgres

```

#### Azure OpenAI Configuration
```bash

AZURE_OPENAI_ENDPOINT=https://your-openai.openai.azure.com/

AZURE_OPENAI_API_KEY=your-api-key

EMBEDDING_MODEL=text-embedding-3-large

```

#### DeepSeek Configuration
```bash

DEEPSEEK_ENDPOINT=https://your-deepseek-endpoint/

DEEPSEEK_API_KEY=your-deepseek-key

DEEPSEEK_MODEL=DeepSeek-R1-0528

```

#### Azure Document Intelligence Configuration
```bash

AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT=https://your-di.cognitiveservices.azure.com/

AZURE_DOCUMENT_INTELLIGENCE_KEY=your-di-key

```

#### Azure Storage Configuration
```bash

AZURE_STORAGE_ACCOUNT_URL=https://yourstorage.blob.core.windows.net/

AZURE_BLOB_SAS_TOKEN=your-sas-token

BLOB_CONTAINER=historylog

```

## 🔧 API Documentation

Once running, access the interactive API documentation:
- **Unified API**: http://localhost:8000/docs
- **NER Service**: http://localhost:8500/docs
- **OCR Service**: http://localhost:8400/docs
- **RAG Service**: http://localhost:8401/docs

## 🎯 API Usage Examples

### 1. Unified Analysis (Text + RAG Indexing)

```python

import httpx



async def unified_analysis():

    data = {

        "text": "Your text content here...",

        "extract_relationships": True,

        "include_embeddings": False,

        "generate_graph_files": True,

        "export_formats": ["neo4j", "json"],

        "enable_rag_indexing": True,

        "rag_title": "My Document",

        "rag_keywords": ["keyword1", "keyword2"]

    }

    

    async with httpx.AsyncClient() as client:

        response = await client.post("http://localhost:8000/analyze/unified", json=data)

        return response.json()

```

### 2. Combined Search with NER Analysis

```python

async def combined_search():

    data = {

        "query": "search query here",

        "limit": 10,

        "similarity_threshold": 0.2,

        "include_ner_analysis": True

    }

    

    async with httpx.AsyncClient() as client:

        response = await client.post("http://localhost:8000/search/combined", json=data)

        return response.json()

```

### 3. File Upload Analysis

```python

async def analyze_file():

    files = {"file": ("document.pdf", open("document.pdf", "rb"), "application/pdf")}

    data = {

        "extract_relationships": "true",

        "generate_graph_files": "true",

        "export_formats": "neo4j,json"

    }

    

    async with httpx.AsyncClient() as client:

        response = await client.post("http://localhost:8000/ner/analyze/file", files=files, data=data)

        return response.json()

```

## 🧪 Testing

### Comprehensive Test Suite

The project includes comprehensive tests covering:
- ✅ Service health checks
- ✅ Individual service functionality
- ✅ Unified workflow testing
- ✅ Service proxy functionality
- ✅ Error handling and resilience
- ✅ Performance testing
- ✅ File upload/download testing

Run tests with:
```bash

python test_unified.py

```

### Individual Service Tests

Test individual services:
```bash

# Test NER service

python test_ner.py



# Test RAG service  

python test_rag.py

```

### Quick Health Check

```bash

curl http://localhost:8000/health

```

## 🔍 Monitoring and Health Checks

### Health Endpoints
- **Unified System**: `GET /health`
- **Individual Services**: `GET /ner/health`, `GET /ocr/health`, `GET /rag/health`
- **Detailed Status**: `GET /status`
- **Service Discovery**: `GET /services`

### Monitoring Features
- Real-time service health monitoring
- Response time tracking
- Service uptime monitoring
- Error rate tracking
- Resource usage monitoring

## 📊 Service Architecture

```mermaid

graph TB

    Client[Client Applications]

    

    subgraph "Unified AI Services (Port 8000)"

        UA[Unified App]

        Proxy[Service Proxies]

        Health[Health Monitor]

    end

    

    subgraph "Core Services"

        NER[NER Service<br/>Port 8500]

        OCR[OCR Service<br/>Port 8400]

        RAG[RAG Service<br/>Port 8401]

    end

    

    subgraph "External Services"

        Azure[Azure Services]

        DeepSeek[DeepSeek API]

        DB[(PostgreSQL)]

    end

    

    Client --> UA

    UA --> Proxy

    Proxy --> NER

    Proxy --> OCR

    Proxy --> RAG

    

    NER --> Azure

    NER --> DeepSeek

    NER --> DB

    

    OCR --> Azure

    

    RAG --> Azure

    RAG --> DB

    RAG --> OCR

```

## 🛠️ Development

### Adding New Features

1. **Service Modifications**: Update individual service files in `services/`
2. **Unified Workflows**: Modify `app.py` for new combined workflows
3. **Configuration**: Update `configs.py` for new settings
4. **Tests**: Add tests to `test_unified.py`

### Debugging

1. **Check Service Logs**: Services log to console
2. **Health Checks**: Use `/health` endpoints
3. **Configuration**: Run `python configs.py` to validate
4. **Database**: Check PostgreSQL connectivity
5. **Azure Services**: Verify API keys and endpoints

### Service Management

Start individual services for development:
```bash

# Start NER service only

cd services && python ner_service.py



# Start OCR service only  

cd services && python ocr_service.py



# Start RAG service only

cd services && python rag_service.py

```

## 🚨 Troubleshooting

### Common Issues

#### 1. Services Won't Start
- Check port availability: `netstat -an | grep :8000`
- Verify Python dependencies: `pip list`
- Check .env configuration: `python configs.py`

#### 2. Database Connection Issues
- Verify PostgreSQL is running
- Check connection string in .env
- Test connectivity: `python -c "import asyncpg; asyncio.run(asyncpg.connect('your-connection-string'))"`

#### 3. Azure Service Issues
- Verify API keys and endpoints
- Check Azure service status
- Review rate limits and quotas

#### 4. Performance Issues
- Monitor resource usage: `top` or Task Manager
- Check database performance
- Review log files for errors

### Error Codes

- **500**: Internal service error
- **503**: Service unavailable
- **400**: Bad request (check input data)
- **422**: Validation error
- **404**: Endpoint not found

## 📈 Performance Optimization

### Recommended Settings

#### Production Configuration
```bash

DEBUG=False

MAX_FILE_SIZE=50

REQUEST_TIMEOUT=300

CHUNK_SIZE=1000

CHUNK_OVERLAP=200

```

#### Database Optimization
- Use connection pooling
- Configure appropriate indexes
- Monitor query performance
- Regular maintenance

#### Service Optimization
- Enable caching where appropriate
- Use async operations
- Optimize batch processing
- Monitor memory usage

## 🔐 Security Considerations

### API Security
- Implement authentication/authorization as needed
- Use HTTPS in production
- Validate all input data
- Rate limiting

### Data Security
- Secure database connections (SSL)
- Encrypt sensitive data
- Regular security updates
- Monitor access logs

### Azure Security
- Rotate API keys regularly
- Use managed identities where possible
- Monitor usage and costs
- Follow Azure security best practices

## 📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Run the test suite
6. Submit a pull request

## 📞 Support

For support and questions:
1. Check this README for common issues
2. Review the test suite for usage examples
3. Check service logs for error details
4. Verify configuration with `python configs.py`

## 🎯 Roadmap

### Current Version (1.0.0)
- ✅ Unified service integration
- ✅ Comprehensive testing
- ✅ Multi-language support
- ✅ Graph database exports

### Future Enhancements
- 🔄 Advanced caching mechanisms
- 🔄 Enhanced monitoring and analytics
- 🔄 Additional export formats
- 🔄 Improved error recovery
- 🔄 Performance optimizations
- 🔄 Additional language support