FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Paper
•
2506.20920
•
Published
•
75
AI & ML interests
Contributors who are invited to beta-test our next big feature! Contact us if you want to join this team :-)
Recent Activity
craffel
authored
a
paper
5 months ago
craffel
authored
a
paper
6 months ago
craffel
authored
a
paper
10 months ago
Post
1690
New state-of-the-art BERT-size retrieval model: *cde-small-v2* 🥳🍾
Hi everyone! We at Cornell are releasing a new retrieval model this week. It uses the contextual embeddings framework, is based on ModernBERT backbone, and gets state-of-the-art results on the MTEB benchmark for its model size (140M parameters). cde-small-v2 gets an average score of 65.6 across the 56 datasets and sees improvements from our previous model in *every* task domain (retrieval, classification, etc.).
We made a lot of changes to make this model work. First of all, ModernBERT has a better tokenizer, which probably helped this work out-of-the-box. We also followed the principles from the CDE paper and used harder clusters and better hard-negative filtering, which showed a small performance improvement. And we made a few small changes that have been shown to work on the larger models: we disabled weight decay, masked out the prefix tokens during pooling, and added a residual connection from the first-stage to the second-stage for better gradient flow.
We're still looking for a computer sponsor to help us scale CDE to larger models. Since it's now state-of-the-art at the 100M parameter scale, it seems to be a reasonable bet that we could train a state-of-the-art large model if we had the GPUs. If you're interested in helping with this, please reach out!
Here's a link to the model: jxm/cde-small-v2
And here's a link to the paper: Contextual Document Embeddings (2410.02525)
Hi everyone! We at Cornell are releasing a new retrieval model this week. It uses the contextual embeddings framework, is based on ModernBERT backbone, and gets state-of-the-art results on the MTEB benchmark for its model size (140M parameters). cde-small-v2 gets an average score of 65.6 across the 56 datasets and sees improvements from our previous model in *every* task domain (retrieval, classification, etc.).
We made a lot of changes to make this model work. First of all, ModernBERT has a better tokenizer, which probably helped this work out-of-the-box. We also followed the principles from the CDE paper and used harder clusters and better hard-negative filtering, which showed a small performance improvement. And we made a few small changes that have been shown to work on the larger models: we disabled weight decay, masked out the prefix tokens during pooling, and added a residual connection from the first-stage to the second-stage for better gradient flow.
We're still looking for a computer sponsor to help us scale CDE to larger models. Since it's now state-of-the-art at the 100M parameter scale, it seems to be a reasonable bet that we could train a state-of-the-art large model if we had the GPUs. If you're interested in helping with this, please reach out!
Here's a link to the model: jxm/cde-small-v2
And here's a link to the paper: Contextual Document Embeddings (2410.02525)
Post
902
Open-Devin has been rebranded as OpenHands on GitHub.
Docs: https://docs.all-hands.dev/modules/usage/llms/local-llms
GitHub: https://github.com/All-Hands-AI/OpenHands
Paper: https://arxiv.org/abs/2407.16741
Post on X: https://x.com/gneubig/status/1828097484599759349?s=61&t=KYwlyP7fHVw9cFnwZFzIag
Docs: https://docs.all-hands.dev/modules/usage/llms/local-llms
GitHub: https://github.com/All-Hands-AI/OpenHands
Paper: https://arxiv.org/abs/2407.16741
Post on X: https://x.com/gneubig/status/1828097484599759349?s=61&t=KYwlyP7fHVw9cFnwZFzIag
- 1 reply
craffel
authored
a
paper
over 1 year ago
thawani
authored
a
paper
over 1 year ago
craffel
authored
2
papers
almost 2 years ago
hbredin
authored
a
paper
about 2 years ago
craffel
authored
4
papers
over 2 years ago
Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models
Paper
•
2306.04529
•
Published
•
1
Scaling Data-Constrained Language Models
Paper
•
2305.16264
•
Published
•
16
Crosslingual Generalization through Multitask Finetuning
Paper
•
2211.01786
•
Published
•
2
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Paper
•
2211.05100
•
Published
•
34