---
language:
- multilingual
- bg
- en
- fr
- de
- ru
- es
- sw
- tr
- vi
tags:
- deberta
- deberta-v3
- mdeberta
license: mit
---

# mdeberta-v3-base-lite

This model was created through vocabulary pruning of the original [microsoft/mdeberta-v3-base](https://huggingface.co/microsoft/mdeberta-v3-base) model while maintaining full quality for Latin and Cyrillic-based languages.

## Supported Languages
- Bulgarian
- English
- French
- German
- Russian
- Spanish
- Swahili
- Turkish
- Vietnamese

## Usage

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("rustemgareev/mdeberta-v3-base-lite")
model = AutoModel.from_pretrained("rustemgareev/mdeberta-v3-base-lite")

# Example usage
text = "This is an example text in English."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
```

## Performance Evaluation

### Size Comparison
| Metric | Original Model | Lite Model | Reduction |
|--------|----------------|------------|-----------|
| Vocabulary Size | 250,102 tokens | 163,211 tokens | 34.74% |
| Disk Size | 1.06 GB | 817 MB | 23.23% |

### VRAM Usage Comparison
*Estimated using [Hugging Face Accelerate Model Estimator](https://huggingface.co/docs/accelerate/main/en/usage_guides/model_size_estimator).*

| Metric | Original Model | Lite Model | Reduction |
|--------|----------------|------------|-----------|
| Largest Layer (float32) | 735.35 MB | 478.16 MB | 34.99% |
| Total Size (float32) | 1.04 GB | 804.13 MB | 22.68% |
| Training using Adam (Peak vRAM) | 4.15 GB | 3.14 GB | 24.34% |

### Semantic Similarity Comparison

**Evaluation Method**: Cosine similarity between embeddings of parallel sentences in different languages, using English as reference.

**Test Phrases Used**:
- English: "Artificial intelligence learns to understand human languages and helps people communicate."
- Bulgarian: "Изкуственият интелект се учи да разбира човешките езици и помага на хората да общуват."
- French: "L'intelligence artificielle apprend à comprendre les langages humains et aide les gens à communiquer."
- German: "Künstliche Intelligenz lernt, menschliche Sprachen zu verstehen und hilft Menschen bei der Kommunikation."
- Russian: "Искусственный интеллект учится понимать человеческие языки и помогает людям общаться."
- Spanish: "La inteligencia artificial aprende a entender los idiomas humanos y ayuda a las personas a comunicarse."
- Swahili: "Akili ya kisasa inajifunza kuelewa lugha za wanadamu na kusaidia watu kuwasiliana."
- Turkish: "Yapay zeka, insan dillerini anlamayı öğrenir ve insanların iletişim kurmasına yardımcı olur."
- Vietnamese: "Trí tuệ nhân tạo học cách hiểu ngôn ngữ con người và giúp mọi người giao tiếp."

**Similarity Results**:

| Language Pair | Original Similarity | Lite Similarity | Difference |
|---------------|-----------------|-----------------|------------|
| English-Bulgarian | 0.9276 | 0.9276 | 0.0000 |
| English-French | 0.9322 | 0.9322 | 0.0000 |
| English-German | 0.9178 | 0.9178 | 0.0000 |
| English-Russian | 0.9335 | 0.9335 | 0.0000 |
| English-Spanish | 0.9228 | 0.9228 | 0.0000 |
| English-Swahili | 0.9591 | 0.9591 | 0.0000 |
| English-Turkish | 0.9450 | 0.9450 | 0.0000 |
| English-Vietnamese | 0.7955 | 0.7955 | 0.0000 |

## License

This model is distributed under the [MIT License](https://opensource.org/licenses/MIT).