SOMA benchmark runs (73 stored, auto-refresh 30s)
| version / task | reps | mean weighted |
resolved | mean steps |
|---|
| baseline | 28 runs | 145,679 | 10/10 | |
| django__django-11551 | reps 1,2,3 | 176,776 | 1/1 | 34 |
| django__django-11603 | reps 1,2,3 | 145,587 | 1/1 | 28 |
| django__django-13516 | reps 1,2,3 | 90,670 | 1/1 | 21 |
| django__django-13821 | reps 1,2,3 | 177,598 | 1/1 | 22 |
| django__django-13964 | reps 1,2 | 209,324 | 1/1 | 40 |
| django__django-14373 | reps 1,2 | 88,389 | 1/1 | 23 |
| django__django-15103 | reps 1,2,3 | 180,407 | 1/1 | 36 |
| django__django-15375 | reps 1,2,3 | 43,202 | 1/1 | 13 |
| django__django-15863 | reps 1,2,3 | 281,493 | 1/1 | 32 |
| sympy__sympy-19637 | reps 1,2,3 | 65,458 | 1/1 | 15 |
| r26bash600 | 45 runs | 84,785 | 39/39 | |
| django__django-11551 | reps 1,2,3,4,6 | 124,741 | 4/4 | 40 |
| django__django-11603 | reps 1,2,3,4,5,6 | 95,185 | 4/4 | 33 |
| django__django-13516 | reps 1,2,3,6 | 61,162 | 3/3 | 29 |
| django__django-13821 | reps 1,2,3,5 | 119,752 | 3/3 | 26 |
| django__django-13964 | reps 1,2,3,5 | 101,760 | 4/4 | 27 |
| django__django-14373 | reps 1,2,3,5 | 57,754 | 4/4 | 20 |
| django__django-15103 | reps 1,2,3,5 | 89,351 | 4/4 | 27 |
| django__django-15375 | reps 1,2,3,4,5 | 59,612 | 5/5 | 20 |
| django__django-15863 | reps 1,2,3,4,6 | 77,872 | 4/4 | 29 |
| sympy__sympy-19637 | reps 1,2,3,4 | 53,489 | 4/4 | 17 |