|
|
--- |
|
|
title: SWE-Bench Verified Discriminative Subsets Leaderboard |
|
|
emoji: 🏆 |
|
|
colorFrom: blue |
|
|
colorTo: red |
|
|
sdk: gradio |
|
|
sdk_version: 5.34.2 |
|
|
app_file: app.py |
|
|
pinned: false |
|
|
license: mit |
|
|
tags: |
|
|
- leaderboard |
|
|
- software-engineering |
|
|
- swe-bench |
|
|
- evaluation |
|
|
- benchmark |
|
|
--- |
|
|
|
|
|
# 🏆 SWE-Bench Verified Discriminative Subsets Leaderboard |
|
|
|
|
|
This interactive leaderboard displays SWE-agents performance across SWE-Bench_Verified and four discriminative subsets designed to provide enhanced evaluation sensitivity for state-of-the-art systems. |
|
|
|
|
|
## 🎯 Why Discriminative Subsets? |
|
|
|
|
|
As SWE-agents improve, achieving 70%+ success rates on the full SWE-Bench Verified benchmark, traditional evaluation loses discriminative power. These targeted subsets focus on the most challenging problems to better distinguish between top-tier systems. |
|
|
|
|
|
## 📊 The Four Discriminative Subsets |
|
|
|
|
|
1. **🔥 Frontier Subset** (95 instances): Problems solved by ≤5 agents - maximum evaluative sensitivity |
|
|
- Combines unsolved, ultra-rare, and very-rare problems |
|
|
- Top agent: 11.6% vs 73.2% on full benchmark (6x better discrimination) |
|
|
|
|
|
2. **⚡ Challenging Subset** (155 instances): Problems solved by ≤20 agents - strong evaluative power |
|
|
- Balances discrimination with statistical significance |
|
|
- Includes frontier + rare and uncommon problems |
|
|
|
|
|
3. **💪 Hard Subset** (45 instances): All Hard difficulty problems regardless of solve rate |
|
|
- Traditional difficulty-based evaluation |
|
|
- Focuses on problems originally classified as most difficult |
|
|
|
|
|
4. **📁 MultiFile Subset** (40 instances): Multi-file problems solved by ≤10 agents |
|
|
- Targets real-world complexity requiring coordinated edits |
|
|
- Even leading agents achieve only 10% success rate |
|
|
|
|
|
## 🔬 Methodology |
|
|
|
|
|
Subsets were created through systematic analysis of solve distribution across 83 evaluated SWE-agents: |
|
|
- Problems solved by fewer agents provide better discrimination |
|
|
- Analysis covers submissions from October 2023 to May 2025 |
|
|
- "Solved" means the agent's fix passed the verification test suite |
|
|
|
|
|
## 📈 Key Insights |
|
|
|
|
|
- **Enhanced Resolution**: Frontier subset provides 6x better discrimination between top systems |
|
|
- **Multi-file Complexity**: Represents genuine software engineering challenges |
|
|
- **Statistical Significance**: Challenging subset offers robust evaluation with strong discrimination |
|
|
- **Real Progress**: Performance on these subsets indicates genuine capability advances |
|
|
|
|
|
## 🔗 Resources |
|
|
|
|
|
- **Blog Post**: [From 73% to 11%: Revealing True SWE-Agent Capabilities](https://jatinganhotra.dev/blog/swe-agents/2025/06/05/swe-bench-verified-discriminative-subsets.html) |
|
|
- **Dataset**: [SWE-bench_Verified-discriminative](https://huggingface.co/datasets/jatinganhotra/SWE-bench_Verified-discriminative) |
|
|
- **Original SWE-Bench**: [SWE-bench.com](https://www.swebench.com/) |
|
|
|
|
|
## 🚀 Usage |
|
|
|
|
|
```python |
|
|
from datasets import load_dataset |
|
|
|
|
|
# Load specific discriminative subset |
|
|
frontier = load_dataset("jatinganhotra/SWE-bench_Verified-discriminative", split="frontier") |
|
|
challenging = load_dataset("jatinganhotra/SWE-bench_Verified-discriminative", split="challenging") |
|
|
``` |
|
|
|
|
|
Created by [Jatin Ganhotra](https://jatinganhotra.dev) | Last Updated: June 19 2025 |
