Overview
Live Dashboard of PRs collections and SWE Tasks Generation
Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.
Total PRs collected
8,818
822 unique repos · 1h 0 / 24h -580,464
Total valid SWE
109
43 unique repos · 1h +1 / 24h -49,856
Overall success rate
3.8%
Valid SWE / processed 2,837
Mean difficulty_score
7.15
median 7.4, count 109
Language Progress
| Language | PRs collected | Last 1h | Last 24h | Valid SWE | Last 1h | Last 24h | Processed | Success rate |
|---|
| Cc | 621 | 0 | -31,736 | 15 | 0 | -9,694 | 442 | |
| C++cpp | 1,326 | 0 | -46,662 | 4 | 0 | -4,035 | 296 | |
| Gogo | 1,621 | 0 | -130,482 | 22 | +1 | -8,073 | 325 | |
| Javajava | 1,165 | 0 | -87,864 | 14 | 0 | -4,003 | 408 | |
| JavaScriptjs | 890 | 0 | -38,792 | 10 | 0 | -7,127 | 346 | |
| Pythonpy | 1,140 | 0 | -104,628 | 25 | 0 | -5,033 | 478 | |
| Rustrust | 787 | 0 | -71,078 | 11 | 0 | -5,494 | 255 | |
| TypeScriptts | 1,268 | 0 | -69,222 | 8 | 0 | -6,397 | 287 | |
Run Parameters
| Language | Eval model (OPENAI) | Completion model (ANTHROPIC) | Concurrency | min_source_files | max_source_files |
|---|
| C | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| C++ | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Go | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Java | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| JavaScript | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Python | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Rust | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| TypeScript | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
Failure Reason Breakdown
click to expand
| Language | Processed | Valid SWE | Failed | trivial_pr | validation | infra_error | timeout | workflow_error | Other |
|---|
| C | 442 | 15 | 427 | 227 | 4 | 175 | 28 | 0 | 0 |
| C++ | 296 | 4 | 292 | 158 | 7 | 93 | 31 | 5 | 0 |
| Go | 325 | 22 | 303 | 135 | 1 | 143 | 41 | 0 | 0 |
| Java | 408 | 14 | 394 | 200 | 1 | 172 | 32 | 3 | 0 |
| JavaScript | 346 | 10 | 336 | 180 | 139 | 3 | 0 | 14 | 0 |
| Python | 478 | 25 | 453 | 203 | 4 | 218 | 47 | 3 | 0 |
| Rust | 255 | 11 | 244 | 118 | 1 | 88 | 44 | 1 | 0 |
| TypeScript | 287 | 8 | 279 | 138 | 12 | 96 | 38 | 3 | 0 |
trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.
validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).
infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).
timeout: processing timed out (per-PR total timeout or Claude Code session timeout).
workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).
fix.patch Complexity
click to expand
| Language | Valid SWE Count | Avg fix.patch lines | Avg fix.patch hunks | Avg fix.patch files |
|---|
| C | 15 | 134.07 | 10.80 | 3.80 |
| C++ | 4 | 70.50 | 6.75 | 2.75 |
| Go | 22 | 70.05 | 7.55 | 3.27 |
| Java | 14 | 63.07 | 9.79 | 4.71 |
| JavaScript | 10 | 87.00 | 4.50 | 2.40 |
| Python | 25 | 193.60 | 8.48 | 2.88 |
| Rust | 11 | 248.45 | 9.45 | 3.91 |
| TypeScript | 8 | 251.25 | 12.25 | 5.88 |
Metric Definitions
click to expand
Difficulty score (difficulty_score)
Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.
The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.
Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.
Tag generation and display
tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.
The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.
fix.patch statistics
Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.
Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.
difficulty_label Distribution
click to expand
| Language | easy / medium / hard | easy | medium | hard |
|---|
| C | 0 / 9 / 6 | 0 | 9 | 6 |
| C++ | 0 / 1 / 3 | 0 | 1 | 3 |
| Go | 0 / 10 / 12 | 0 | 10 | 12 |
| Java | 0 / 8 / 6 | 0 | 8 | 6 |
| JavaScript | 0 / 7 / 3 | 0 | 7 | 3 |
| Python | 0 / 8 / 17 | 0 | 8 | 17 |
| Rust | 0 / 4 / 7 | 0 | 4 | 7 |
| TypeScript | 0 / 2 / 6 | 0 | 2 | 6 |
difficulty_score Overview
click to expand
| Language | count | min | p25 | median | mean | p75 | max |
|---|
| C | 15 | 4.7 | 5.8 | 6.6 | 6.89 | 8.3 | 8.9 |
| C++ | 4 | 5.2 | 6.7 | 7.8 | 7.50 | 8.6 | 9.3 |
| Go | 22 | 3.7 | 5.5 | 7.2 | 6.75 | 8.0 | 9.4 |
| Java | 14 | 5.5 | 6.0 | 6.8 | 7.26 | 8.2 | 9.7 |
| JavaScript | 10 | 4.3 | 4.8 | 5.7 | 6.16 | 7.1 | 8.9 |
| Python | 25 | 4.4 | 6.2 | 8.1 | 7.58 | 9.0 | 9.7 |
| Rust | 11 | 4.6 | 6.5 | 8.7 | 7.82 | 9.1 | 9.2 |
| TypeScript | 8 | 5.0 | 7.1 | 7.5 | 7.29 | 7.9 | 8.9 |
Global Top Tags
click to expand
library46 (42.2%)
backend37 (33.9%)
frontend13 (11.9%)
cli12 (11.0%)
logging5 (4.6%)
incomplete-validation4 (3.7%)
tui4 (3.7%)
missing-api3 (2.8%)
error-handling3 (2.8%)
python3 (2.8%)
testing3 (2.8%)
unknown-defect3 (2.8%)
static-analysis3 (2.8%)
mcp3 (2.8%)
api3 (2.8%)
missing-feature3 (2.8%)
react3 (2.8%)
parsing2 (1.8%)
podio2 (1.8%)
schema-evolution2 (1.8%)
false-positive2 (1.8%)
configuration2 (1.8%)
autosubmit2 (1.8%)
httpx2 (1.8%)
missing-implementation2 (1.8%)
missing-functionality2 (1.8%)
api-compatibility-fix1 (0.9%)
build-system1 (0.9%)
cache-coherency1 (0.9%)
canfigger1 (0.9%)
Per-Language Tag Distribution
click to expand
C c
library12 (80.0%)
logging3 (20.0%)
backend2 (13.3%)
missing-api2 (13.3%)
api-compatibility-fix1 (6.7%)
build-system1 (6.7%)
cache-coherency1 (6.7%)
canfigger1 (6.7%)
checksum-computation-error1 (6.7%)
config-parsing1 (6.7%)
display1 (6.7%)
error-handling1 (6.7%)
frontend1 (6.7%)
incomplete-deduplication-logic1 (6.7%)
incomplete-validation1 (6.7%)
libbpf1 (6.7%)
memory-management1 (6.7%)
missing-backend1 (6.7%)
missing-cleanup-api1 (6.7%)
missing-function-implementation1 (6.7%)
C++ cpp
library4 (100.0%)
podio2 (50.0%)
schema-evolution2 (50.0%)
dynamic-window-management1 (25.0%)
egl1 (25.0%)
incorrect-logic1 (25.0%)
missing-auto-detection1 (25.0%)
Go go
backend12 (54.5%)
library7 (31.8%)
cli3 (13.6%)
static-analysis3 (13.6%)
false-positive2 (9.1%)
bitcoin1 (4.5%)
clock-drift-tolerance1 (4.5%)
configurable-leeway1 (4.5%)
configuration1 (4.5%)
context1 (4.5%)
data-format-inconsistency1 (4.5%)
event-processing1 (4.5%)
header-sync1 (4.5%)
http1 (4.5%)
ibc1 (4.5%)
incorrect-parsing1 (4.5%)
insecure-default1 (4.5%)
jwt1 (4.5%)
lifecycle-state-handling1 (4.5%)
mcp1 (4.5%)
Java java
backend10 (71.4%)
library4 (28.6%)
error-handling2 (14.3%)
api1 (7.1%)
api-response1 (7.1%)
conditional-field-inclusion1 (7.1%)
cose1 (7.1%)
dependency-migration1 (7.1%)
dropwizard1 (7.1%)
duplicate-identifier1 (7.1%)
incorrect-message1 (7.1%)
java-streams1 (7.1%)
ledger1 (7.1%)
missing-api1 (7.1%)
missing-api-feature1 (7.1%)
missing-error-handling1 (7.1%)
missing-error-mapping1 (7.1%)
missing-feature1 (7.1%)
missing-metric-implementation1 (7.1%)
missing-model1 (7.1%)
JavaScript js
frontend5 (50.0%)
react3 (30.0%)
cli2 (20.0%)
library2 (20.0%)
missing-feature2 (20.0%)
backend1 (10.0%)
ccda1 (10.0%)
dependency-api-change1 (10.0%)
incomplete-diagnostics1 (10.0%)
incomplete-validation1 (10.0%)
missing-field-display1 (10.0%)
missing-field-extraction1 (10.0%)
missing-prop-support1 (10.0%)
missing-refresh-mechanism1 (10.0%)
pivotaltracker1 (10.0%)
platform-compatibility1 (10.0%)
postgres1 (10.0%)
tar1 (10.0%)
trust-store1 (10.0%)
unix-sockets1 (10.0%)
Python py
library15 (60.0%)
backend6 (24.0%)
cli4 (16.0%)
autosubmit2 (8.0%)
httpx2 (8.0%)
logging2 (8.0%)
missing-implementation2 (8.0%)
api-refactor1 (4.0%)
api-refactoring1 (4.0%)
argparse1 (4.0%)
aws-bedrock1 (4.0%)
bulk-evaluation1 (4.0%)
command-responsibility-misplacement1 (4.0%)
configuration1 (4.0%)
couchbase1 (4.0%)
elicitation-support1 (4.0%)
event-binding1 (4.0%)
graphql1 (4.0%)
gui1 (4.0%)
incomplete-aggregation1 (4.0%)
Rust rust
frontend4 (36.4%)
tui4 (36.4%)
cli3 (27.3%)
backend2 (18.2%)
library2 (18.2%)
budget1 (9.1%)
claude-code1 (9.1%)
command-routing1 (9.1%)
glyph-rendering-update1 (9.1%)
mcp1 (9.1%)
missing-functionality1 (9.1%)
missing-hook-implementation1 (9.1%)
missing-push-fallback1 (9.1%)
missing-validation-warning1 (9.1%)
missing-writer1 (9.1%)
process-management1 (9.1%)
python1 (9.1%)
testing1 (9.1%)
time-formatting1 (9.1%)
ui-redesign1 (9.1%)
TypeScript ts
backend4 (50.0%)
frontend3 (37.5%)
api2 (25.0%)
api-contract-update1 (12.5%)
architecture-boundary-violation1 (12.5%)
checkpoint-recording-failure1 (12.5%)
embedding1 (12.5%)
eth-rpc-adapter1 (12.5%)
fullstack1 (12.5%)
incomplete-validation1 (12.5%)
infinite-cache-growth1 (12.5%)
missing-data-propagation1 (12.5%)
monolithic-component1 (12.5%)
nextjs1 (12.5%)
optional-field-handling1 (12.5%)
polkadot-api1 (12.5%)
refactoring1 (12.5%)
webrtc1 (12.5%)
Global Top Bug Classes
click to expand
Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.
incomplete-validation4 (3.7%)
missing-api3 (2.8%)
unknown-defect3 (2.8%)
missing-feature3 (2.8%)
false-positive2 (1.8%)
missing-implementation2 (1.8%)
missing-functionality2 (1.8%)
api-compatibility-fix1 (0.9%)
cache-coherency1 (0.9%)
checksum-computation-error1 (0.9%)
incomplete-deduplication-logic1 (0.9%)
missing-backend1 (0.9%)
missing-cleanup-api1 (0.9%)
missing-function-implementation1 (0.9%)
missing-state-check1 (0.9%)
module-lifecycle1 (0.9%)
toolchain-incompatibility1 (0.9%)
ui-flicker1 (0.9%)
dynamic-window-management1 (0.9%)
incorrect-logic1 (0.9%)
missing-auto-detection1 (0.9%)
schema-evolution1 (0.9%)
clock-drift-tolerance1 (0.9%)
configurable-leeway1 (0.9%)
data-format-inconsistency1 (0.9%)
incorrect-parsing1 (0.9%)
insecure-default1 (0.9%)
lifecycle-state-handling1 (0.9%)
missing-backoff-logic1 (0.9%)
missing-caching1 (0.9%)
Per-Language Bug-Class Distribution
click to expand
Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.
C c15 tagged
missing-api2 (13.3%)
api-compatibility-fix1 (6.7%)
cache-coherency1 (6.7%)
checksum-computation-error1 (6.7%)
incomplete-deduplication-logic1 (6.7%)
incomplete-validation1 (6.7%)
missing-backend1 (6.7%)
missing-cleanup-api1 (6.7%)
missing-function-implementation1 (6.7%)
missing-state-check1 (6.7%)
module-lifecycle1 (6.7%)
toolchain-incompatibility1 (6.7%)
ui-flicker1 (6.7%)
unknown-defect1 (6.7%)
C++ cpp4 tagged
dynamic-window-management1 (25.0%)
incorrect-logic1 (25.0%)
missing-auto-detection1 (25.0%)
schema-evolution1 (25.0%)
Go go22 tagged
false-positive2 (9.1%)
clock-drift-tolerance1 (4.5%)
configurable-leeway1 (4.5%)
data-format-inconsistency1 (4.5%)
incorrect-parsing1 (4.5%)
insecure-default1 (4.5%)
lifecycle-state-handling1 (4.5%)
missing-backoff-logic1 (4.5%)
missing-caching1 (4.5%)
missing-config-toggle1 (4.5%)
missing-configuration1 (4.5%)
missing-context-propagation1 (4.5%)
missing-existence-check1 (4.5%)
missing-fallback1 (4.5%)
missing-message-type-handling1 (4.5%)
Java java14 tagged
conditional-field-inclusion1 (7.1%)
dependency-migration1 (7.1%)
duplicate-identifier1 (7.1%)
incorrect-message1 (7.1%)
missing-api1 (7.1%)
missing-api-feature1 (7.1%)
missing-error-handling1 (7.1%)
missing-error-mapping1 (7.1%)
missing-feature1 (7.1%)
missing-metric-implementation1 (7.1%)
missing-model1 (7.1%)
missing-parameter-support1 (7.1%)
missing-response-mapping1 (7.1%)
unused-dependency1 (7.1%)
JavaScript js10 tagged
missing-feature2 (20.0%)
dependency-api-change1 (10.0%)
incomplete-diagnostics1 (10.0%)
incomplete-validation1 (10.0%)
missing-field-display1 (10.0%)
missing-field-extraction1 (10.0%)
missing-prop-support1 (10.0%)
missing-refresh-mechanism1 (10.0%)
platform-compatibility1 (10.0%)
Python py25 tagged
missing-implementation2 (8.0%)
api-refactor1 (4.0%)
api-refactoring1 (4.0%)
command-responsibility-misplacement1 (4.0%)
elicitation-support1 (4.0%)
incomplete-aggregation1 (4.0%)
incomplete-matching1 (4.0%)
incomplete-path-sanitization1 (4.0%)
incorrect-calculation1 (4.0%)
incorrect-combination-logic1 (4.0%)
input-validation1 (4.0%)
metrics-calculation-error1 (4.0%)
missing-authentication1 (4.0%)
missing-cli-feature1 (4.0%)
missing-docstring-extraction1 (4.0%)
Rust rust11 tagged
glyph-rendering-update1 (9.1%)
missing-functionality1 (9.1%)
missing-hook-implementation1 (9.1%)
missing-push-fallback1 (9.1%)
missing-validation-warning1 (9.1%)
missing-writer1 (9.1%)
time-formatting1 (9.1%)
ui-redesign1 (9.1%)
unknown-defect1 (9.1%)
wrong-command-scope1 (9.1%)
wrong-keybinding1 (9.1%)
TypeScript ts8 tagged
api-contract-update1 (12.5%)
architecture-boundary-violation1 (12.5%)
checkpoint-recording-failure1 (12.5%)
incomplete-validation1 (12.5%)
infinite-cache-growth1 (12.5%)
missing-data-propagation1 (12.5%)
monolithic-component1 (12.5%)
optional-field-handling1 (12.5%)