cbjuan tidealwari commited on
Commit
ba692d5
·
verified ·
1 Parent(s): f1922f7

Add metrics table (#4)

Browse files

- Add metrics table (3b6f59382786863689dd809020d3476a42f3e5da)
- Re-order metrics table (a0201006deffec20c4f482019a506455d90ccc15)


Co-authored-by: Adarsh <[email protected]>

Files changed (1) hide show
  1. README.md +61 -0
README.md CHANGED
@@ -111,6 +111,67 @@ Please refer to [Qwen Documentation](https://qwen.readthedocs.io/en/latest/deplo
111
  Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts**.
112
  We advise adding the `rope_scaling` configuration only when processing long contexts is required.
113
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
114
  ## Training Data
115
 
116
  - **Data Collection and Filtering:** Our code data is sourced from a combination of publicly available datasets (e.g., Code available on <https://github.com>), and additional synthetic data generated at IBM Quantum. We exclude code that is older than 2023.
 
111
  Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts**.
112
  We advise adding the `rope_scaling` configuration only when processing long contexts is required.
113
 
114
+ ### Comparison of Qiskit models across benchmarks
115
+
116
+ <table
117
+ style="
118
+ display: inline-table;
119
+ border-collapse: separate;
120
+ border-spacing: 0;
121
+ font-family: Inter, -apple-system, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
122
+ box-shadow: 0 6px 18px rgba(12, 20, 29, 0.06);
123
+ border-radius: 12px;
124
+ overflow: hidden;
125
+ table-layout: auto;
126
+ box-sizing: border-box;
127
+ margin: 16px 0;
128
+ "
129
+ >
130
+ <thead>
131
+ <tr>
132
+ <th style="text-align:left; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
133
+ Model
134
+ </th>
135
+ <th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
136
+ QiskitHumanEval
137
+ </th>
138
+ <th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
139
+ QiskitHumanEval-Hard
140
+ </th>
141
+ <th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
142
+ HumanEval
143
+ </th>
144
+ </tr>
145
+ </thead>
146
+ <tbody>
147
+ <tr style="background:#f7fafc;">
148
+ <td style="padding:12px 16px; font-weight:700; color:#07102a;">Qwen2.5-Coder-14B-Qiskit</td>
149
+ <td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">25.16</td>
150
+ <td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">49.01</td>
151
+ <td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">91.46</td>
152
+ </tr>
153
+ <tr style="background:#ffffff;">
154
+ <td style="padding:12px 16px; color:#0f172a;">mistral-small-3.2-24b-qiskit</td>
155
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">20.53</td>
156
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">40.39</td>
157
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">77.49</td>
158
+ </tr>
159
+ <tr style="background:#ffffff;">
160
+ <td style="padding:12px 16px; color:#0f172a;">granite-3.3-8b-qiskit</td>
161
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">14.56</td>
162
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">27.15</td>
163
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">62.80</td>
164
+ </tr>
165
+ <tr style="background:#fbfdff;">
166
+ <td style="padding:12px 16px; color:#0f172a;">granite-3.2-8b-qiskit</td>
167
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">9.93</td>
168
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">24.50</td>
169
+ <td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">57.31</td>
170
+ </tr>
171
+ </tbody>
172
+ </table>
173
+
174
+
175
  ## Training Data
176
 
177
  - **Data Collection and Filtering:** Our code data is sourced from a combination of publicly available datasets (e.g., Code available on <https://github.com>), and additional synthetic data generated at IBM Quantum. We exclude code that is older than 2023.