Add metrics table (#4)
Browse files- Add metrics table (3b6f59382786863689dd809020d3476a42f3e5da)
- Re-order metrics table (a0201006deffec20c4f482019a506455d90ccc15)
Co-authored-by: Adarsh <[email protected]>
README.md
CHANGED
|
@@ -111,6 +111,67 @@ Please refer to [Qwen Documentation](https://qwen.readthedocs.io/en/latest/deplo
|
|
| 111 |
Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts**.
|
| 112 |
We advise adding the `rope_scaling` configuration only when processing long contexts is required.
|
| 113 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
## Training Data
|
| 115 |
|
| 116 |
- **Data Collection and Filtering:** Our code data is sourced from a combination of publicly available datasets (e.g., Code available on <https://github.com>), and additional synthetic data generated at IBM Quantum. We exclude code that is older than 2023.
|
|
|
|
| 111 |
Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts**.
|
| 112 |
We advise adding the `rope_scaling` configuration only when processing long contexts is required.
|
| 113 |
|
| 114 |
+
### Comparison of Qiskit models across benchmarks
|
| 115 |
+
|
| 116 |
+
<table
|
| 117 |
+
style="
|
| 118 |
+
display: inline-table;
|
| 119 |
+
border-collapse: separate;
|
| 120 |
+
border-spacing: 0;
|
| 121 |
+
font-family: Inter, -apple-system, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
|
| 122 |
+
box-shadow: 0 6px 18px rgba(12, 20, 29, 0.06);
|
| 123 |
+
border-radius: 12px;
|
| 124 |
+
overflow: hidden;
|
| 125 |
+
table-layout: auto;
|
| 126 |
+
box-sizing: border-box;
|
| 127 |
+
margin: 16px 0;
|
| 128 |
+
"
|
| 129 |
+
>
|
| 130 |
+
<thead>
|
| 131 |
+
<tr>
|
| 132 |
+
<th style="text-align:left; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
|
| 133 |
+
Model
|
| 134 |
+
</th>
|
| 135 |
+
<th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
|
| 136 |
+
QiskitHumanEval
|
| 137 |
+
</th>
|
| 138 |
+
<th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
|
| 139 |
+
QiskitHumanEval-Hard
|
| 140 |
+
</th>
|
| 141 |
+
<th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
|
| 142 |
+
HumanEval
|
| 143 |
+
</th>
|
| 144 |
+
</tr>
|
| 145 |
+
</thead>
|
| 146 |
+
<tbody>
|
| 147 |
+
<tr style="background:#f7fafc;">
|
| 148 |
+
<td style="padding:12px 16px; font-weight:700; color:#07102a;">Qwen2.5-Coder-14B-Qiskit</td>
|
| 149 |
+
<td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">25.16</td>
|
| 150 |
+
<td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">49.01</td>
|
| 151 |
+
<td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">91.46</td>
|
| 152 |
+
</tr>
|
| 153 |
+
<tr style="background:#ffffff;">
|
| 154 |
+
<td style="padding:12px 16px; color:#0f172a;">mistral-small-3.2-24b-qiskit</td>
|
| 155 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">20.53</td>
|
| 156 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">40.39</td>
|
| 157 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">77.49</td>
|
| 158 |
+
</tr>
|
| 159 |
+
<tr style="background:#ffffff;">
|
| 160 |
+
<td style="padding:12px 16px; color:#0f172a;">granite-3.3-8b-qiskit</td>
|
| 161 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">14.56</td>
|
| 162 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">27.15</td>
|
| 163 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">62.80</td>
|
| 164 |
+
</tr>
|
| 165 |
+
<tr style="background:#fbfdff;">
|
| 166 |
+
<td style="padding:12px 16px; color:#0f172a;">granite-3.2-8b-qiskit</td>
|
| 167 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">9.93</td>
|
| 168 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">24.50</td>
|
| 169 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">57.31</td>
|
| 170 |
+
</tr>
|
| 171 |
+
</tbody>
|
| 172 |
+
</table>
|
| 173 |
+
|
| 174 |
+
|
| 175 |
## Training Data
|
| 176 |
|
| 177 |
- **Data Collection and Filtering:** Our code data is sourced from a combination of publicly available datasets (e.g., Code available on <https://github.com>), and additional synthetic data generated at IBM Quantum. We exclude code that is older than 2023.
|