PROJECT_STRUCTURE.md

VedaMD Project Structure

Clean, organized codebase for production deployment

Last updated: October 23, 2025

---

Directory Structure

SL Clinical Assistant/
├── app.py                          # Gradio interface (HF Spaces entry point)
├── requirements.txt                # Python dependencies
├── .env.example                    # Environment variable template
├── .gitignore                      # Git ignore rules
│
├── src/                            # Core application code
│   ├── __init__.py
│   ├── enhanced_groq_medical_rag.py       # Main RAG system (Cerebras-powered)
│   ├── enhanced_backend_api.py            # FastAPI backend for frontend
│   ├── simple_vector_store.py             # Vector store loader
│   ├── vector_store_compatibility.py      # Compatibility wrapper (temporary)
│   ├── enhanced_medical_context.py        # Medical context enhancement
│   └── medical_response_verifier.py       # Response verification & safety
│
├── scripts/                        # Automation scripts
│   ├── build_vector_store.py      # Build complete vector store from PDFs
│   └── add_document.py             # Add single document incrementally
│
├── frontend/                       # Next.js frontend (separate deployment)
│   ├── src/
│   │   ├── app/
│   │   ├── components/
│   │   └── lib/
│   │       └── api.ts              # API client (FastAPI + Gradio support)
│   ├── public/
│   ├── package.json
│   └── .env.local.example
│
├── data/                           # Data files (local only, not in git)
│   ├── guidelines/                 # Source PDF files (moved from Obs/)
│   ├── vector_store/               # Built vector store (FAISS + metadata)
│   │   ├── faiss_index.bin
│   │   ├── documents.json
│   │   ├── metadata.json
│   │   ├── config.json
│   │   └── backups/                # Automatic backups
│   └── processed/                  # Processed documents (optional)
│
├── docs/                           # Documentation index
│   └── README.md                   # Documentation directory index
│
├── archive/                        # Old/deprecated files (not in git)
│   ├── old_scripts/                # batch_ocr_pipeline.py, convert_pdf.py
│   └── old_docs/                   # output.md, cleanup_plan.md, etc.
│
├── test_pdfs/                      # Test files (not in git)
├── test_vector_store/              # Test vector store (not in git)
│
└── Documentation Files             # Root-level docs
    ├── README.md                   # Main project README
    ├── PIPELINE_GUIDE.md           # Document pipeline usage guide
    ├── LOCAL_TESTING_GUIDE.md      # Local development guide
    ├── IMPROVEMENT_PLAN.md         # Project roadmap
    ├── DEPLOYMENT.md               # Deployment instructions
    ├── SECURITY_SETUP.md           # Security configuration
    ├── CEREBRAS_MIGRATION_GUIDE.md # Cerebras migration details
    ├── QUICK_START_CEREBRAS.md     # Cerebras quickstart
    ├── PRODUCTION_READINESS_REPORT.md  # Production assessment
    ├── CHANGES_SUMMARY.md          # Summary of changes
    └── CEREBRAS_SUMMARY.md         # Cerebras integration summary

---

Core Files

Application Entry Points

File	Purpose	Deployment
app.py	Gradio interface	Hugging Face Spaces
src/enhanced_backend_api.py	FastAPI REST API	Hugging Face Spaces (port 7862)
frontend/	Next.js frontend	Netlify / Vercel

RAG System

File	Purpose	Key Features
src/enhanced_groq_medical_rag.py	Main RAG orchestrator	Cerebras integration, multi-stage retrieval, medical safety
src/simple_vector_store.py	Vector store loader	HF Hub download, FAISS search
src/enhanced_medical_context.py	Medical context enhancement	Entity extraction, relevance scoring
src/medical_response_verifier.py	Response verification	Claim validation, source traceability

Automation Scripts

Script	Purpose	Usage
scripts/build_vector_store.py	Build complete vector store	python scripts/build_vector_store.py --input-dir ./data/guidelines --output-dir ./data/vector_store --upload
scripts/add_document.py	Add single document	python scripts/add_document.py --file new.pdf --vector-store-dir ./data/vector_store --upload

Startup Scripts

Script	Purpose
run_backend.sh	Start FastAPI backend (port 7862)
run_frontend.sh	Start Next.js frontend (port 3000)
kill_backend.sh	Stop backend processes

---

Data Files

Vector Store Files (data/vector_store/)

Generated by build_vector_store.py:

File	Purpose	Format
faiss_index.bin	FAISS vector index	Binary
documents.json	Document chunks	JSON array of strings
metadata.json	Document metadata	JSON array of objects
config.json	Build configuration	JSON object
build_log.json	Build information	JSON object

Metadata Structure:

{
  "source": "guideline.pdf",
  "section": "Management",
  "chunk_id": 0,
  "chunk_size": 1000,
  "file_hash": "a3f2c9d8...",
  "extraction_method": "pymupdf",
  "total_pages": 15,
  "citation": "SLCOG Guidelines 2025",
  "category": "Obstetrics",
  "processed_at": "2025-10-23T15:08:30.273544"
}

---

Configuration Files

Environment Variables

.env (local development):

CEREBRAS_API_KEY=csk_your_key_here
HF_TOKEN=hf_your_token_here  # For uploading vector store

Hugging Face Spaces Secrets:

CEREBRAS_API_KEY  # Required
HF_TOKEN          # Optional (for vector store upload)
ALLOWED_ORIGINS   # Optional (CORS, comma-separated)

Requirements

requirements.txt - Python dependencies:

• cerebras-cloud-sdk - Cerebras API client
• gradio - Web interface
• fastapi - REST API
• sentence-transformers - Embeddings
• faiss-cpu - Vector search
• huggingface-hub - Model/data hosting
• PyMuPDF, pdfplumber - PDF extraction

---

Git Ignore Strategy

Ignored (Local Only)

• data/guidelines/ - Source PDFs
• data/vector_store/ - Built vector store
• archive/ - Old files
• test_pdfs/, test_vector_store/ - Test files
• frontend/ - Separate deployment
• .env - Local environment variables
• *.log - Log files

Committed (Version Control)

• src/ - Application code
• scripts/ - Automation scripts
• app.py - Gradio entry point
• requirements.txt - Dependencies
• .env.example - Environment template
• *.md - Documentation

---

Workflow

Development Workflow

1. Add new guideline:

   cp ~/Downloads/new_guideline.pdf data/guidelines/
   

2. Update vector store:

   python scripts/add_document.py \
     --file data/guidelines/new_guideline.pdf \
     --citation "SLCOG Guidelines 2025" \
     --vector-store-dir ./data/vector_store
   

3. Test locally:

   # Terminal 1: Start backend
   ./run_backend.sh
   
   # Terminal 2: Start frontend
   ./run_frontend.sh
   
   # Or just test Gradio
   python app.py
   

4. Deploy to production:

   # Upload vector store to HF Hub
   python scripts/build_vector_store.py \
     --input-dir ./data/guidelines \
     --output-dir ./data/vector_store \
     --upload --repo-id sniro23/VedaMD-Vector-Store
   
   # Push code to HF Spaces
   git add src/ app.py requirements.txt
   git commit -m "Update: Add new guidelines"
   git push origin main
   

Production Deployment

Backend (Hugging Face Spaces):

• Gradio interface: Automatic from app.py
• FastAPI API: Runs on port 7862
• Vector store: Downloaded from HF Hub on startup
• Secrets: Set in HF Spaces settings

Frontend (Netlify):

• Build: cd frontend && npm run build
• Deploy: Automatic from GitHub
• Environment: NEXT_PUBLIC_API_URL=https://sniro23-vedamd-enhanced.hf.space

---

Migration Notes

From Old Structure

Moved:

• Obs/*.pdf → data/guidelines/*.pdf
• Vector store logic remains in src/

Archived:

• batch_ocr_pipeline.py → archive/old_scripts/
• convert_pdf.py → archive/old_scripts/
• output*.md → archive/old_docs/
• cleanup_plan.md → archive/old_docs/

Created New:

• scripts/ - Automation scripts
• data/ - Data directory structure
• docs/ - Documentation index
• archive/ - Old files

---

Key Improvements

Before Cleanup

SL Clinical Assistant/
├── app.py
├── src/
├── Obs/                    # Unclear name
├── batch_ocr_pipeline.py   # Old script at root
├── convert_pdf.py          # Old script at root
├── output.md               # Temporary file
├── output_new.md           # Temporary file
└── 15+ .md files at root   # Disorganized docs

After Cleanup

SL Clinical Assistant/
├── app.py                  # Clear entry point
├── src/                    # Core code
├── scripts/                # Automation scripts
├── data/                   # Data files
│   ├── guidelines/         # Clear purpose
│   └── vector_store/       # Clear purpose
├── docs/                   # Documentation index
├── archive/                # Old files preserved
└── Documentation files     # Organized at root

---

Best Practices

Code Organization

1. Core Logic: Keep in src/
2. Automation: Keep in scripts/
3. Data: Keep in data/ (gitignored)
4. Tests: Keep in tests/ (if created)

Documentation

1. User Guides: Root level (PIPELINE_GUIDE.md, etc.)
2. Technical Docs: Root level (DEPLOYMENT.md, etc.)
3. Code Docs: Inline docstrings in Python files
4. Index: docs/README.md for navigation

Data Management

1. Source Data: data/guidelines/
2. Processed Data: data/vector_store/
3. Backups: Automatic in data/vector_store/backups/
4. Test Data: test_pdfs/, test_vector_store/

Version Control

1. Commit Code: src/, scripts/, app.py
2. Ignore Data: data/, archive/, test_*/
3. Commit Docs: All .md files
4. Templates: .env.example, not .env

---

Quick Reference

Common Commands

# Build vector store from scratch
python scripts/build_vector_store.py --input-dir ./data/guidelines --output-dir ./data/vector_store

# Add single document
python scripts/add_document.py --file new.pdf --vector-store-dir ./data/vector_store

# Start backend
./run_backend.sh

# Start frontend
./run_frontend.sh

# Test Gradio interface
python app.py

# Upload to HF Hub
python scripts/build_vector_store.py ... --upload --repo-id sniro23/VedaMD-Vector-Store

Important Paths

• PDFs: data/guidelines/
• Vector Store: data/vector_store/
• RAG System: src/enhanced_groq_medical_rag.py
• API: src/enhanced_backend_api.py
• Scripts: scripts/
• Docs: Root level + docs/README.md

---

Clean codebase = Maintainable codebase = Production-ready codebase