Improve model card for mmBERT training checkpoints with metadata and usage

This PR significantly enhances the model card for the mmBERT raw training checkpoints by:

* Updating the `language` tag to `mul` to accurately reflect its multilingual nature (over 1800 languages).
* Adding `pipeline_tag: feature-extraction` to improve discoverability on the Hub and enable the automated widget. While this repository contains raw checkpoints, the associated models are commonly used for feature extraction.
* Specifying `library_name: transformers` as the linked GitHub repository provides extensive usage examples with the `transformers` library.
* Adding descriptive `tags` like `bert`, `multilingual`, and `encoder-only` for better searchability.
* Updating the main title to reflect the paper's title and linking to the Hugging Face paper page.
* Incorporating the paper's abstract.
* Including comprehensive usage examples from the GitHub README's "Quick Start" and "Getting Started" sections, showcasing how to use the associated `mmBERT` models (e.g., `mmbert-small`, `mmbert-base`) for various tasks like feature extraction, masked language modeling, classification, and retrieval.
* Clarifying the purpose of this repository as containing raw training checkpoints and directing users to the model collection for runnable models.
* Integrating other valuable sections from the GitHub README such as "Model Family", "Training Details", "Evaluation", "FAQ", and "Limitations" to provide a complete overview of the project.

These changes make the model card more informative, discoverable, and user-friendly on the Hugging Face Hub.

README.md

@@ -1,32 +1,341 @@
 ---
-license: mit
 language:
-- en
+- mul
+license: mit
+pipeline_tag: feature-extraction
+library_name: transformers
+tags:
+- bert
+- multilingual
+- encoder-only
 ---
 
-# Ettin Checkpoints
+# mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Paper](https://img.shields.io/badge/Paper-Arxiv-red)](https://arxiv.org/abs/2509.06888)
-[![Models](https://img.shields.io/badge/🤗%20Hugging%20Face-12%20Models-blue)](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4)
+[![Paper](https://img.shields.io/badge/Paper-%F0%9F%A4%97Hugging_Face-red)](https://huggingface.co/papers/2509.06888)
+[![Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-12%20Models-blue)](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4)
 [![GitHub](https://img.shields.io/badge/GitHub-Code-black)](https://github.com/jhu-clsp/mmBERT)
 
-This repository contains the raw training checkpoints for the mmBERT models.  Each model contains three subfolders for `decay`, `ext`, and `pretrain`.
+> 🌍 **TL;DR**: State-of-the-art multilingual encoder models trained on 3T tokens across 1833 languages with novel annealed language learning. Outperforms XLM-R and can even beat OpenAI's o3 and Google's Gemini 2.5 Pro.
+
+This repository contains the raw training checkpoints for the mmBERT models, presented in the paper [mmBERT: A Modern Multilingual Encoder with Annealed Language Learning](https://huggingface.co/papers/2509.06888). Each model contains three subfolders for `decay`, `ext`, and `pretrain`.
+
+These raw checkpoints are intended for resuming pre-training using Composer and contain all state needed for this purpose. For direct inference with the runnable `mmbert-small` and `mmbert-base` models, please refer to the [mmBERT Model Collection](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4).
+
+## Abstract
+Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent research for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements, including an inverse mask ratio schedule and an inverse temperature sampling ratio. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data. Despite only including these low-resource languages in the short decay phase we achieve similar classification performance to models like OpenAI's o3 and Google's Gemini 2.5 Pro. Overall, we show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks -- on both high and low-resource languages.
+
+## 🚀 Quick Start
+
+### Installation
+```bash
+pip install torch>=1.9.0
+pip install transformers>=4.48.0
+```
+
+### 30-Second Examples
+The following examples demonstrate how to use the pre-trained `mmbert-small` and `mmbert-base` models (available in the [mmBERT Model Collection](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4)) for various tasks.
+
+**Small Model for Fast Inference:**
+```python
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-small")
+model = AutoModel.from_pretrained("jhu-clsp/mmbert-small")
+
+# Example: Get multilingual embeddings
+inputs = tokenizer("Hello world! 你好世界! Bonjour le monde!", return_tensors="pt")
+outputs = model(**inputs)
+embeddings = outputs.last_hidden_state.mean(dim=1)
+```
+
+**Base Model for Masked Language Modeling:**
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+import torch
+
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/mmbert-base")
+
+# Example: Multilingual masked language modeling
+text = "The capital of [MASK] is Paris."
+inputs = tokenizer(text, return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# Get predictions for [MASK] tokens
+mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)
+predictions = outputs.logits[mask_indices]
+top_tokens = torch.topk(predictions, 5, dim=-1)
+predicted_words = [tokenizer.decode(token) for token in top_tokens.indices[0]]
+print(f"Predictions: {predicted_words}")
+```
+
+## 🌍 Model Family
+
+### Main Models
+
+| Size | Model | Parameters | Languages | Context | Best For | Download |
+|:-----|:------|:-----------|:----------|:--------|:---------|:---------|
+| Small | [mmbert-small](https://huggingface.co/jhu-clsp/mmbert-small) | 140M | 1833 | 8192 | Fast inference, edge deployment | [![Download](https://img.shields.io/badge/%F0%9F%A4%97-Download-blue)](https://huggingface.co/jhu-clsp/mmbert-small) |
+| Base | [mmbert-base](https://huggingface.co/jhu-clsp/mmbert-base) | 307M | 1833 | 8192 | Best performance, production use | [![Download](https://img.shields.io/badge/%F0%9F%A4%97-Download-blue)](https://huggingface.co/jhu-clsp/mmbert-base) |
+
+### Key Features
+
+-   **1833 Languages**: Covers more languages than any previous multilingual encoder
+-   **Extended Context**: Up to 8192 tokens (vs 512 for XLM-R)
+-   **Efficiency**: 2-4x faster inference than previous multilingual models
+-   **Modern Architecture**: Based on ModernBERT with RoPE, GLU activations, and Flash Attention 2
+-   **Open Training**: Complete training data, recipes, and checkpoints available
+
+## 🔬 Getting Started
+
+### Training Data
+
+The complete multilingual training dataset spans 3T tokens:
+
+-   **Pre-training Data**: 2.0T tokens across 60 languages
+-   **Mid-training Data**: 600B tokens across 110 languages
+-   **Decay Phase Data**: 100B tokens across 1833 languages
+-   **Data Sources**: FineWeb2, DCLM, Dolmino, Wikipedia, ArXiv, and curated multilingual corpora
+
+### Usage Examples
+
+<details>
+<summary><strong>Classification Task</strong></summary>
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch.nn as nn
+
+# Load model for classification
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+encoder = AutoModel.from_pretrained("jhu-clsp/mmbert-base")
+
+# Add classification head
+class MultilingualClassifier(nn.Module):
+    def __init__(self, encoder, num_classes):
+        super().__init__()
+        self.encoder = encoder
+        self.classifier = nn.Linear(encoder.config.hidden_size, num_classes)
+        self.dropout = nn.Dropout(0.1)
+    
+    def forward(self, input_ids, attention_mask=None):
+        outputs = self.encoder(input_ids, attention_mask=attention_mask)
+        pooled_output = outputs.last_hidden_state[:, 0]  # Use [CLS] token
+        pooled_output = self.dropout(pooled_output)
+        return self.classifier(pooled_output)
+
+# Initialize classifier
+model = MultilingualClassifier(encoder, num_classes=3)
+
+# Example multilingual inputs
+texts = [
+    "This is a positive review.",
+    "Ceci est un avis négatif.",
+    "这是一个中性评价。"
+]
+inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
+predictions = model(**inputs)
+```
+
+</details>
+
+<details>
+<summary><strong>Multilingual Retrieval</strong></summary>
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+import numpy as np
+
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+model = AutoModel.from_pretrained("jhu-clsp/mmbert-base")
+
+def get_embeddings(texts):
+    inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
+    with torch.no_grad():
+        outputs = model(**inputs)
+    # Mean pooling
+    embeddings = outputs.last_hidden_state.mean(dim=1)
+    return embeddings.numpy()
+
+# Multilingual document retrieval
+documents = [
+    "Artificial intelligence is transforming healthcare.",
+    "L'intelligence artificielle transforme les soins de santé.",
+    "人工智能正在改变医疗保健。",
+    "Climate change requires immediate action.",
+    "El cambio climático requiere acción inmediata."
+]
 
-These files work with Composer and contain all state needed to resume pre-training. Please see the [ModernBERT repository](https://github.com/AnswerDotAI/ModernBERT) for usage details. 
+query = "AI in medicine"
 
+# Get embeddings
+doc_embeddings = get_embeddings(documents)
+query_embedding = get_embeddings([query])
 
-## 🔗 Related Resources
+# Compute similarities
+similarities = np.dot(doc_embeddings, query_embedding.T).flatten()
+ranked_docs = np.argsort(similarities)[::-1]
 
-- **Models**: [mmBERT Model Suite](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4)
-- **Phase 1**: [Pre-training Data](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) (2.3T tokens)
-- **Phase 2**: [Mid-training Data](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining) (600B tokens)
-- **Phase 3**: [Decay Phase Data](https://huggingface.co/datasets/jhu-clsp/mmbert-decay) (100B tokens)
-- **Paper**: [Arxiv link](https://arxiv.org/abs/2509.06888)
-- **Code**: [GitHub Repository](https://github.com/jhu-clsp/mmBERT)
+print("Most similar documents:")
+for i, doc_idx in enumerate(ranked_docs[:3]):
+    print(f"{i+1}. {documents[doc_idx]} (score: {similarities[doc_idx]:.3f})")
+```
+
+</details>
+
+<details>
+<summary><strong>Long Context Processing</strong></summary>
+
+```python
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+model = AutoModel.from_pretrained("jhu-clsp/mmbert-base")
+
+# Process long multilingual document (up to 8192 tokens)
+long_text = """
+This is a very long multilingual document...
+Ceci est un très long document multilingue...
+这是一个非常长的多语言文档...
+""" * 100  # Simulate long text
+
+# Tokenize with extended context
+inputs = tokenizer(
+    long_text, 
+    return_tensors="pt", 
+    max_length=8192,
+    truncation=True
+)
+
+# Process efficiently with Flash Attention
+with torch.no_grad():
+    outputs = model(**inputs)
+    
+print(f"Processed {inputs['input_ids'].shape[1]} tokens")
+print(f"Output shape: {outputs.last_hidden_state.shape}")
+```
+
+</details>
+
+## 📋 Training
+
+Using 8xH100s, training took approximately 10 days for mmBERT-small and 40 days for mmBERT-base. 
+
+### Training Recipe: Cascading Annealed Language Learning
+
+mmBERT introduces novel training techniques:
+
+1.  **Inverse Masking Schedule**: Start with 30% masking, gradually reduce to 5%
+2.  **Language Progression**: 60 → 110 → 1833 languages across training phases
+3.  **Temperature Annealing**: 0.7 → 0.5 → 0.3 for increasingly uniform language sampling
+4.  **High-Quality Data**: Progressive upgrade from web crawl to filtered premium sources
+
+## Training Details
+
+### Architecture
+
+| Component | Small | Base |
+|:----------|:------|:-----|
+| Layers | 22 | 22 |
+| Hidden Size | 384 | 768 |
+| Intermediate Size | 1152 | 1152 |
+| Attention Heads | 6 | 12 |
+| Parameters (Total) | 140M | 307M |
+| Parameters (Non-Embed) | 42M | 110M |
+| Max Sequence Length | 8192 | 8192 |
+| Vocabulary Size | 256,000 | 256,000 |
+
+### Training Configuration
+
+**Data Mixture:**
+-   Pre-training (2.0T tokens): Web crawl, code, scientific papers, reference materials
+-   Mid-training (600B tokens): Higher quality filtered data with context extension
+-   Decay phase (100B tokens): Premium sources including textbooks and curated content
+
+**Architecture Features:**
+-   ModernBERT-based transformer with RoPE positional embeddings
+-   GLU activations and prenorm layer normalization
+-   Flash Attention 2 for efficient long-context processing
+-   Gemma 2 tokenizer for multilingual coverage
+
+**Training Phases:**
+1.  **Base Pre-training**: 60 languages, 30% masking, learning rate warmup
+2.  **Context Extension**: 110 languages, 15% masking, extended context to 8K
+3.  **Decay Phase**: 1833 languages, 5% masking, high-quality data focus
+
+## Evaluation
+Evaluation code for retrieval tasks is the same as [Ettin](https://github.com/JHU-CLSP/ettin-encoder-vs-decoder/tree/main/retrieval_eval).
+
+Evaluation code for efficiency is taken from the [ModernBERT](https://github.com/AnswerDotAI/ModernBERT/tree/main/efficiency) repo. 
+
+Evaluation code for NLU tasks is based on the [mGTE codebase](https://github.com/izhx/nlu-evals) and our fork will be uploaded soon. Please raise an issue or message us if this would be helpful for you.
+
+## ❓ FAQ
+
+**Q: How does mmBERT compare to XLM-R?**
+**A:** mmBERT significantly outperforms XLM-R across all benchmarks:
+-   +2.4 points average on XTREME
+-   +3.0 points on GLUE
+-   16x more languages (1833 vs 100)
+-   16x longer context (8K vs 512 tokens)
+-   2-4x faster inference
+
+**Q: Which languages does mmBERT support?**
+**A:** mmBERT supports 1833 languages and scripts from FineWeb2, including:
+-   All major world languages (English, Chinese, Spanish, etc.)
+-   European languages (including low-resource ones like Faroese)
+-   African languages (Swahili, Amharic, etc.)
+-   Asian languages (Hindi, Bengali, Thai, etc.)
+-   Many low-resource and indigenous languages
+
+**Q: How does the annealed language learning work?**
+**A:** We progressively add languages in three phases:
+1.  Start with 60 high-resource languages (pre-training)
+2.  Add 50 mid-resource languages (mid-training) 
+3.  Add 1723 low-resource languages (decay phase)
+
+This allows efficient learning without overfitting on low-resource data.
+
+**Q: Can I fine-tune mmBERT for my specific task?**
+**A:** Yes! mmBERT works as a drop-in replacement for XLM-R:
+```python
+from transformers import AutoModel, AutoTokenizer
+
+# Load for fine-tuning
+model = AutoModel.from_pretrained("jhu-clsp/mmbert-base")
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmbert-base")
+
+# Add task-specific head and fine-tune normally
+```
+
+**Q: What about efficiency and memory requirements?**
+**A:** mmBERT is significantly more efficient:
+-   2-4x faster inference than XLM-R
+-   Flash Attention 2 reduces memory usage for long sequences
+-   Support for variable-length batching
+-   Optimized for both CPU and GPU deployment
+
+**Q: How do I access the training data and checkpoints?**
+**A:** All data and checkpoints are publicly available:
+-   Training data: [jhu-clsp/mmbert-pretraining-data](https://huggingface.co/datasets/jhu-clsp/mmbert-pretraining-data)
+-   Checkpoints: [jhu-clsp/mmbert-checkpoints](https://huggingface.co/models/jhu-clsp/mmbert-checkpoints)
+-   Github code: [GitHub repository](https://github.com/jhu-clsp/mmBERT)
+-   Data processing code: [Same as Ettin models](https://github.com/jhu-clsp/ettin-encoder-vs-decoder)
+
+## Limitations
+
+-   Structured prediction tasks (NER, POS) show slightly lower scores due to tokenizer prefix space handling
+-   Very low-resource languages still have limited training data
+-   High-quality educational content filtering could benefit from more languages
 
 ## Citation
 
+If you use mmBERT models in your research, please cite our work:
+
 ```bibtex
 @misc{marone2025mmbertmodernmultilingualencoder,
       title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning},