Overview
Live Dashboard of PRs collections and SWE Tasks Generation
Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.
Total PRs collected
8,818
822 unique repos · 1h 0 / 24h -580,464
Total valid SWE
83
32 unique repos · 1h +18 / 24h -49,882
Overall success rate
2.9%
Valid SWE / processed 2,819
Mean difficulty_score
6.95
median 7.0, count 83
Language Progress
| Language | PRs collected | Last 1h | Last 24h | Valid SWE | Last 1h | Last 24h | Processed | Success rate |
|---|
| Cc | 621 | 0 | -31,736 | 14 | 0 | -9,695 | 441 | |
| C++cpp | 1,326 | 0 | -46,662 | 4 | +1 | -4,035 | 296 | |
| Gogo | 1,621 | 0 | -130,482 | 13 | +3 | -8,082 | 325 | |
| Javajava | 1,165 | 0 | -87,864 | 12 | +3 | -4,005 | 408 | |
| JavaScriptjs | 890 | 0 | -38,792 | 10 | +3 | -7,127 | 339 | |
| Pythonpy | 1,140 | 0 | -104,628 | 14 | +7 | -5,044 | 478 | |
| Rustrust | 787 | 0 | -71,078 | 11 | 0 | -5,494 | 255 | |
| TypeScriptts | 1,268 | 0 | -69,222 | 5 | +1 | -6,400 | 277 | |
Run Parameters
| Language | Eval model (OPENAI) | Completion model (ANTHROPIC) | Concurrency | min_source_files | max_source_files |
|---|
| C | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| C++ | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Go | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Java | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| JavaScript | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Python | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Rust | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| TypeScript | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
Failure Reason Breakdown
click to expand
| Language | Processed | Valid SWE | Failed | trivial_pr | validation | infra_error | timeout | workflow_error | Other |
|---|
| C | 441 | 14 | 427 | 227 | 3 | 175 | 28 | 0 | 0 |
| C++ | 296 | 4 | 292 | 158 | 7 | 93 | 31 | 5 | 0 |
| Go | 325 | 13 | 312 | 135 | 1 | 143 | 41 | 0 | 0 |
| Java | 408 | 12 | 396 | 200 | 1 | 172 | 32 | 3 | 0 |
| JavaScript | 339 | 10 | 329 | 175 | 55 | 88 | 5 | 6 | 0 |
| Python | 478 | 14 | 464 | 203 | 4 | 218 | 47 | 3 | 0 |
| Rust | 255 | 11 | 244 | 118 | 1 | 88 | 44 | 1 | 0 |
| TypeScript | 277 | 5 | 272 | 138 | 3 | 96 | 38 | 2 | 0 |
trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.
validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).
infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).
timeout: processing timed out (per-PR total timeout or Claude Code session timeout).
workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).
fix.patch Complexity
click to expand
| Language | Valid SWE Count | Avg fix.patch lines | Avg fix.patch hunks | Avg fix.patch files |
|---|
| C | 14 | 139.50 | 10.71 | 3.71 |
| C++ | 4 | 70.50 | 6.75 | 2.75 |
| Go | 13 | 84.54 | 7.69 | 3.15 |
| Java | 12 | 64.58 | 10.42 | 4.83 |
| JavaScript | 10 | 87.00 | 4.50 | 2.40 |
| Python | 14 | 140.57 | 6.64 | 2.21 |
| Rust | 11 | 248.45 | 9.45 | 3.91 |
| TypeScript | 5 | 163.80 | 17.00 | 8.40 |
Metric Definitions
click to expand
Difficulty score (difficulty_score)
Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.
The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.
Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.
Tag generation and display
tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.
The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.
fix.patch statistics
Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.
Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.
difficulty_label Distribution
click to expand
| Language | easy / medium / hard | easy | medium | hard |
|---|
| C | 0 / 8 / 6 | 0 | 8 | 6 |
| C++ | 0 / 1 / 3 | 0 | 1 | 3 |
| Go | 0 / 8 / 5 | 0 | 8 | 5 |
| Java | 0 / 7 / 5 | 0 | 7 | 5 |
| JavaScript | 0 / 7 / 3 | 0 | 7 | 3 |
| Python | 0 / 6 / 8 | 0 | 6 | 8 |
| Rust | 0 / 4 / 7 | 0 | 4 | 7 |
| TypeScript | 0 / 2 / 3 | 0 | 2 | 3 |
difficulty_score Overview
click to expand
| Language | count | min | p25 | median | mean | p75 | max |
|---|
| C | 14 | 4.7 | 5.7 | 6.8 | 6.93 | 8.5 | 8.9 |
| C++ | 4 | 5.2 | 6.7 | 7.8 | 7.50 | 8.6 | 9.3 |
| Go | 13 | 3.7 | 4.2 | 5.8 | 6.17 | 7.5 | 9.4 |
| Java | 12 | 5.7 | 6.3 | 6.8 | 7.24 | 8.0 | 9.7 |
| JavaScript | 10 | 4.3 | 4.8 | 5.7 | 6.16 | 7.1 | 8.9 |
| Python | 14 | 4.4 | 5.5 | 7.8 | 7.13 | 8.4 | 9.7 |
| Rust | 11 | 4.6 | 6.5 | 8.7 | 7.82 | 9.1 | 9.2 |
| TypeScript | 5 | 5.0 | 6.0 | 7.4 | 7.02 | 7.8 | 8.9 |
Global Top Tags
click to expand
library38 (45.8%)
backend25 (30.1%)
frontend10 (12.0%)
cli9 (10.8%)
logging4 (4.8%)
tui4 (4.8%)
missing-api3 (3.6%)
error-handling3 (3.6%)
incomplete-validation3 (3.6%)
static-analysis3 (3.6%)
api3 (3.6%)
react3 (3.6%)
python2 (2.4%)
testing2 (2.4%)
unknown-defect2 (2.4%)
podio2 (2.4%)
schema-evolution2 (2.4%)
false-positive2 (2.4%)
missing-feature2 (2.4%)
missing-implementation2 (2.4%)
httpx2 (2.4%)
api-compatibility-fix1 (1.2%)
build-system1 (1.2%)
cache-coherency1 (1.2%)
canfigger1 (1.2%)
checksum-computation-error1 (1.2%)
config-parsing1 (1.2%)
incomplete-deduplication-logic1 (1.2%)
libbpf1 (1.2%)
memory-management1 (1.2%)
Per-Language Tag Distribution
click to expand
C c
library12 (85.7%)
logging3 (21.4%)
backend2 (14.3%)
missing-api2 (14.3%)
api-compatibility-fix1 (7.1%)
build-system1 (7.1%)
cache-coherency1 (7.1%)
canfigger1 (7.1%)
checksum-computation-error1 (7.1%)
config-parsing1 (7.1%)
error-handling1 (7.1%)
incomplete-deduplication-logic1 (7.1%)
incomplete-validation1 (7.1%)
libbpf1 (7.1%)
memory-management1 (7.1%)
missing-backend1 (7.1%)
missing-cleanup-api1 (7.1%)
missing-function-implementation1 (7.1%)
missing-state-check1 (7.1%)
module-lifecycle1 (7.1%)
C++ cpp
library4 (100.0%)
podio2 (50.0%)
schema-evolution2 (50.0%)
incorrect-logic1 (25.0%)
missing-auto-detection1 (25.0%)
dynamic-window-management1 (25.0%)
egl1 (25.0%)
Go go
backend6 (46.2%)
library5 (38.5%)
static-analysis3 (23.1%)
cli2 (15.4%)
false-positive2 (15.4%)
configurable-leeway1 (7.7%)
context1 (7.7%)
http1 (7.7%)
incorrect-parsing1 (7.7%)
insecure-default1 (7.7%)
lifecycle-state-handling1 (7.7%)
missing-caching1 (7.7%)
missing-context-propagation1 (7.7%)
regex1 (7.7%)
structtag1 (7.7%)
tls1 (7.7%)
token1 (7.7%)
webhooks1 (7.7%)
whitespace-handling1 (7.7%)
wrong-function-resolution1 (7.7%)
Java java
backend9 (75.0%)
library3 (25.0%)
error-handling2 (16.7%)
api1 (8.3%)
api-response1 (8.3%)
conditional-field-inclusion1 (8.3%)
dependency-migration1 (8.3%)
dropwizard1 (8.3%)
incorrect-message1 (8.3%)
ledger1 (8.3%)
missing-error-handling1 (8.3%)
missing-error-mapping1 (8.3%)
missing-model1 (8.3%)
missing-parameter-support1 (8.3%)
missing-response-mapping1 (8.3%)
payment-api1 (8.3%)
payment-validation1 (8.3%)
rest-api1 (8.3%)
unused-dependency1 (8.3%)
java-streams1 (8.3%)
JavaScript js
frontend5 (50.0%)
react3 (30.0%)
cli2 (20.0%)
library2 (20.0%)
missing-feature2 (20.0%)
ccda1 (10.0%)
dependency-api-change1 (10.0%)
incomplete-diagnostics1 (10.0%)
incomplete-validation1 (10.0%)
missing-field-extraction1 (10.0%)
missing-prop-support1 (10.0%)
postgres1 (10.0%)
tar1 (10.0%)
trust-store1 (10.0%)
backend1 (10.0%)
platform-compatibility1 (10.0%)
unix-sockets1 (10.0%)
missing-refresh-mechanism1 (10.0%)
vscode1 (10.0%)
missing-field-display1 (10.0%)
Python py
library10 (71.4%)
backend2 (14.3%)
cli2 (14.3%)
missing-implementation2 (14.3%)
httpx2 (14.3%)
autosubmit1 (7.1%)
gui1 (7.1%)
incomplete-matching1 (7.1%)
incomplete-path-sanitization1 (7.1%)
incorrect-calculation1 (7.1%)
logging1 (7.1%)
missing-gui-config1 (7.1%)
missing-sensitive-data-redaction1 (7.1%)
path-sanitization1 (7.1%)
status-priority-handling1 (7.1%)
string-matching1 (7.1%)
template-matching1 (7.1%)
workflow1 (7.1%)
magql1 (7.1%)
missing-docstring-extraction1 (7.1%)
Rust rust
frontend4 (36.4%)
tui4 (36.4%)
cli3 (27.3%)
backend2 (18.2%)
library2 (18.2%)
budget1 (9.1%)
claude-code1 (9.1%)
command-routing1 (9.1%)
glyph-rendering-update1 (9.1%)
mcp1 (9.1%)
missing-functionality1 (9.1%)
missing-hook-implementation1 (9.1%)
missing-push-fallback1 (9.1%)
missing-validation-warning1 (9.1%)
missing-writer1 (9.1%)
process-management1 (9.1%)
python1 (9.1%)
testing1 (9.1%)
time-formatting1 (9.1%)
ui-redesign1 (9.1%)
TypeScript ts
backend3 (60.0%)
api2 (40.0%)
api-contract-update1 (20.0%)
eth-rpc-adapter1 (20.0%)
frontend1 (20.0%)
fullstack1 (20.0%)
incomplete-validation1 (20.0%)
infinite-cache-growth1 (20.0%)
optional-field-handling1 (20.0%)
polkadot-api1 (20.0%)
webrtc1 (20.0%)
missing-data-propagation1 (20.0%)
Global Top Bug Classes
click to expand
Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.
missing-api3 (3.6%)
incomplete-validation3 (3.6%)
unknown-defect2 (2.4%)
false-positive2 (2.4%)
missing-feature2 (2.4%)
missing-implementation2 (2.4%)
api-compatibility-fix1 (1.2%)
cache-coherency1 (1.2%)
checksum-computation-error1 (1.2%)
incomplete-deduplication-logic1 (1.2%)
missing-backend1 (1.2%)
missing-cleanup-api1 (1.2%)
missing-function-implementation1 (1.2%)
missing-state-check1 (1.2%)
module-lifecycle1 (1.2%)
toolchain-incompatibility1 (1.2%)
incorrect-logic1 (1.2%)
missing-auto-detection1 (1.2%)
schema-evolution1 (1.2%)
dynamic-window-management1 (1.2%)
configurable-leeway1 (1.2%)
incorrect-parsing1 (1.2%)
insecure-default1 (1.2%)
lifecycle-state-handling1 (1.2%)
missing-caching1 (1.2%)
missing-context-propagation1 (1.2%)
whitespace-handling1 (1.2%)
wrong-function-resolution1 (1.2%)
missing-existence-check1 (1.2%)
missing-backoff-logic1 (1.2%)
Per-Language Bug-Class Distribution
click to expand
Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.
C c14 tagged
missing-api2 (14.3%)
api-compatibility-fix1 (7.1%)
cache-coherency1 (7.1%)
checksum-computation-error1 (7.1%)
incomplete-deduplication-logic1 (7.1%)
incomplete-validation1 (7.1%)
missing-backend1 (7.1%)
missing-cleanup-api1 (7.1%)
missing-function-implementation1 (7.1%)
missing-state-check1 (7.1%)
module-lifecycle1 (7.1%)
toolchain-incompatibility1 (7.1%)
unknown-defect1 (7.1%)
C++ cpp4 tagged
incorrect-logic1 (25.0%)
missing-auto-detection1 (25.0%)
schema-evolution1 (25.0%)
dynamic-window-management1 (25.0%)
Go go13 tagged
false-positive2 (15.4%)
configurable-leeway1 (7.7%)
incorrect-parsing1 (7.7%)
insecure-default1 (7.7%)
lifecycle-state-handling1 (7.7%)
missing-caching1 (7.7%)
missing-context-propagation1 (7.7%)
whitespace-handling1 (7.7%)
wrong-function-resolution1 (7.7%)
missing-existence-check1 (7.7%)
missing-backoff-logic1 (7.7%)
wrong-default1 (7.7%)
Java java12 tagged
conditional-field-inclusion1 (8.3%)
dependency-migration1 (8.3%)
incorrect-message1 (8.3%)
missing-error-handling1 (8.3%)
missing-error-mapping1 (8.3%)
missing-model1 (8.3%)
missing-parameter-support1 (8.3%)
missing-response-mapping1 (8.3%)
unused-dependency1 (8.3%)
missing-api-feature1 (8.3%)
missing-api1 (8.3%)
duplicate-identifier1 (8.3%)
JavaScript js10 tagged
missing-feature2 (20.0%)
dependency-api-change1 (10.0%)
incomplete-diagnostics1 (10.0%)
incomplete-validation1 (10.0%)
missing-field-extraction1 (10.0%)
missing-prop-support1 (10.0%)
platform-compatibility1 (10.0%)
missing-refresh-mechanism1 (10.0%)
missing-field-display1 (10.0%)
Python py14 tagged
missing-implementation2 (14.3%)
incomplete-matching1 (7.1%)
incomplete-path-sanitization1 (7.1%)
incorrect-calculation1 (7.1%)
missing-gui-config1 (7.1%)
missing-sensitive-data-redaction1 (7.1%)
status-priority-handling1 (7.1%)
missing-docstring-extraction1 (7.1%)
missing-testing-helpers1 (7.1%)
incomplete-aggregation1 (7.1%)
metrics-calculation-error1 (7.1%)
api-refactor1 (7.1%)
api-refactoring1 (7.1%)
Rust rust11 tagged
glyph-rendering-update1 (9.1%)
missing-functionality1 (9.1%)
missing-hook-implementation1 (9.1%)
missing-push-fallback1 (9.1%)
missing-validation-warning1 (9.1%)
missing-writer1 (9.1%)
time-formatting1 (9.1%)
ui-redesign1 (9.1%)
unknown-defect1 (9.1%)
wrong-command-scope1 (9.1%)
wrong-keybinding1 (9.1%)
TypeScript ts5 tagged
api-contract-update1 (20.0%)
incomplete-validation1 (20.0%)
infinite-cache-growth1 (20.0%)
optional-field-handling1 (20.0%)
missing-data-propagation1 (20.0%)