Overview
Live Dashboard of PRs collections and SWE Tasks Generation
Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.
Total PRs collected
8,818
822 unique repos · 1h 0 / 24h -580,464
Total valid SWE
100
39 unique repos · 1h +4 / 24h -49,865
Overall success rate
3.5%
Valid SWE / processed 2,822
Mean difficulty_score
7.04
median 7.1, count 100
Language Progress
| Language | PRs collected | Last 1h | Last 24h | Valid SWE | Last 1h | Last 24h | Processed | Success rate |
|---|
| Cc | 621 | 0 | -31,736 | 15 | +1 | -9,694 | 441 | |
| C++cpp | 1,326 | 0 | -46,662 | 4 | 0 | -4,035 | 296 | |
| Gogo | 1,621 | 0 | -130,482 | 19 | +1 | -8,076 | 325 | |
| Javajava | 1,165 | 0 | -87,864 | 14 | 0 | -4,003 | 408 | |
| JavaScriptjs | 890 | 0 | -38,792 | 10 | 0 | -7,127 | 342 | |
| Pythonpy | 1,140 | 0 | -104,628 | 20 | +2 | -5,038 | 478 | |
| Rustrust | 787 | 0 | -71,078 | 11 | 0 | -5,494 | 255 | |
| TypeScriptts | 1,268 | 0 | -69,222 | 7 | 0 | -6,398 | 277 | |
Run Parameters
| Language | Eval model (OPENAI) | Completion model (ANTHROPIC) | Concurrency | min_source_files | max_source_files |
|---|
| C | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| C++ | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Go | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Java | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| JavaScript | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Python | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Rust | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| TypeScript | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
Failure Reason Breakdown
click to expand
| Language | Processed | Valid SWE | Failed | trivial_pr | validation | infra_error | timeout | workflow_error | Other |
|---|
| C | 441 | 15 | 426 | 227 | 3 | 175 | 28 | 0 | 0 |
| C++ | 296 | 4 | 292 | 158 | 7 | 93 | 31 | 5 | 0 |
| Go | 325 | 19 | 306 | 135 | 1 | 143 | 41 | 0 | 0 |
| Java | 408 | 14 | 394 | 200 | 1 | 172 | 32 | 3 | 0 |
| JavaScript | 342 | 10 | 332 | 179 | 103 | 38 | 0 | 12 | 0 |
| Python | 478 | 20 | 458 | 203 | 4 | 218 | 47 | 3 | 0 |
| Rust | 255 | 11 | 244 | 118 | 1 | 88 | 44 | 1 | 0 |
| TypeScript | 277 | 7 | 270 | 138 | 3 | 96 | 38 | 2 | 0 |
trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.
validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).
infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).
timeout: processing timed out (per-PR total timeout or Claude Code session timeout).
workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).
fix.patch Complexity
click to expand
| Language | Valid SWE Count | Avg fix.patch lines | Avg fix.patch hunks | Avg fix.patch files |
|---|
| C | 15 | 134.07 | 10.80 | 3.80 |
| C++ | 4 | 70.50 | 6.75 | 2.75 |
| Go | 19 | 71.79 | 7.26 | 3.05 |
| Java | 14 | 63.07 | 9.79 | 4.71 |
| JavaScript | 10 | 87.00 | 4.50 | 2.40 |
| Python | 20 | 185.95 | 7.30 | 2.45 |
| Rust | 11 | 248.45 | 9.45 | 3.91 |
| TypeScript | 7 | 278.86 | 13.29 | 6.57 |
Metric Definitions
click to expand
Difficulty score (difficulty_score)
Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.
The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.
Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.
Tag generation and display
tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.
The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.
fix.patch statistics
Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.
Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.
difficulty_label Distribution
click to expand
| Language | easy / medium / hard | easy | medium | hard |
|---|
| C | 0 / 9 / 6 | 0 | 9 | 6 |
| C++ | 0 / 1 / 3 | 0 | 1 | 3 |
| Go | 0 / 10 / 9 | 0 | 10 | 9 |
| Java | 0 / 8 / 6 | 0 | 8 | 6 |
| JavaScript | 0 / 7 / 3 | 0 | 7 | 3 |
| Python | 0 / 8 / 12 | 0 | 8 | 12 |
| Rust | 0 / 4 / 7 | 0 | 4 | 7 |
| TypeScript | 0 / 2 / 5 | 0 | 2 | 5 |
difficulty_score Overview
click to expand
| Language | count | min | p25 | median | mean | p75 | max |
|---|
| C | 15 | 4.7 | 5.8 | 6.6 | 6.89 | 8.3 | 8.9 |
| C++ | 4 | 5.2 | 6.7 | 7.8 | 7.50 | 8.6 | 9.3 |
| Go | 19 | 3.7 | 5.5 | 6.7 | 6.55 | 7.8 | 9.4 |
| Java | 14 | 5.5 | 6.0 | 6.8 | 7.26 | 8.2 | 9.7 |
| JavaScript | 10 | 4.3 | 4.8 | 5.7 | 6.16 | 7.1 | 8.9 |
| Python | 20 | 4.4 | 5.8 | 8.0 | 7.32 | 8.5 | 9.7 |
| Rust | 11 | 4.6 | 6.5 | 8.7 | 7.82 | 9.1 | 9.2 |
| TypeScript | 7 | 5.0 | 6.7 | 7.4 | 7.16 | 7.7 | 8.9 |
Global Top Tags
click to expand
library46 (46.0%)
backend29 (29.0%)
frontend13 (13.0%)
cli11 (11.0%)
logging5 (5.0%)
tui4 (4.0%)
missing-api3 (3.0%)
error-handling3 (3.0%)
incomplete-validation3 (3.0%)
python3 (3.0%)
testing3 (3.0%)
unknown-defect3 (3.0%)
static-analysis3 (3.0%)
api3 (3.0%)
missing-feature3 (3.0%)
react3 (3.0%)
podio2 (2.0%)
schema-evolution2 (2.0%)
false-positive2 (2.0%)
mcp2 (2.0%)
autosubmit2 (2.0%)
httpx2 (2.0%)
missing-implementation2 (2.0%)
api-compatibility-fix1 (1.0%)
build-system1 (1.0%)
cache-coherency1 (1.0%)
canfigger1 (1.0%)
checksum-computation-error1 (1.0%)
config-parsing1 (1.0%)
incomplete-deduplication-logic1 (1.0%)
Per-Language Tag Distribution
click to expand
C c
library12 (80.0%)
logging3 (20.0%)
backend2 (13.3%)
missing-api2 (13.3%)
api-compatibility-fix1 (6.7%)
build-system1 (6.7%)
cache-coherency1 (6.7%)
canfigger1 (6.7%)
checksum-computation-error1 (6.7%)
config-parsing1 (6.7%)
error-handling1 (6.7%)
incomplete-deduplication-logic1 (6.7%)
incomplete-validation1 (6.7%)
libbpf1 (6.7%)
memory-management1 (6.7%)
missing-backend1 (6.7%)
missing-cleanup-api1 (6.7%)
missing-function-implementation1 (6.7%)
missing-state-check1 (6.7%)
module-lifecycle1 (6.7%)
C++ cpp
library4 (100.0%)
podio2 (50.0%)
schema-evolution2 (50.0%)
dynamic-window-management1 (25.0%)
egl1 (25.0%)
incorrect-logic1 (25.0%)
missing-auto-detection1 (25.0%)
Go go
backend9 (47.4%)
library7 (36.8%)
cli3 (15.8%)
static-analysis3 (15.8%)
false-positive2 (10.5%)
bitcoin1 (5.3%)
configurable-leeway1 (5.3%)
configuration1 (5.3%)
context1 (5.3%)
header-sync1 (5.3%)
http1 (5.3%)
incorrect-parsing1 (5.3%)
insecure-default1 (5.3%)
lifecycle-state-handling1 (5.3%)
mcp1 (5.3%)
missing-backoff-logic1 (5.3%)
missing-caching1 (5.3%)
missing-config-toggle1 (5.3%)
missing-configuration1 (5.3%)
missing-context-propagation1 (5.3%)
Java java
backend10 (71.4%)
library4 (28.6%)
error-handling2 (14.3%)
api1 (7.1%)
api-response1 (7.1%)
conditional-field-inclusion1 (7.1%)
cose1 (7.1%)
dependency-migration1 (7.1%)
dropwizard1 (7.1%)
duplicate-identifier1 (7.1%)
incorrect-message1 (7.1%)
java-streams1 (7.1%)
ledger1 (7.1%)
missing-api1 (7.1%)
missing-api-feature1 (7.1%)
missing-error-handling1 (7.1%)
missing-error-mapping1 (7.1%)
missing-feature1 (7.1%)
missing-metric-implementation1 (7.1%)
missing-model1 (7.1%)
JavaScript js
frontend5 (50.0%)
react3 (30.0%)
cli2 (20.0%)
library2 (20.0%)
missing-feature2 (20.0%)
backend1 (10.0%)
ccda1 (10.0%)
dependency-api-change1 (10.0%)
incomplete-diagnostics1 (10.0%)
incomplete-validation1 (10.0%)
missing-field-display1 (10.0%)
missing-field-extraction1 (10.0%)
missing-prop-support1 (10.0%)
missing-refresh-mechanism1 (10.0%)
pivotaltracker1 (10.0%)
platform-compatibility1 (10.0%)
postgres1 (10.0%)
tar1 (10.0%)
trust-store1 (10.0%)
unix-sockets1 (10.0%)
Python py
library15 (75.0%)
cli3 (15.0%)
autosubmit2 (10.0%)
backend2 (10.0%)
httpx2 (10.0%)
logging2 (10.0%)
missing-implementation2 (10.0%)
api-refactor1 (5.0%)
api-refactoring1 (5.0%)
aws-bedrock1 (5.0%)
bulk-evaluation1 (5.0%)
event-binding1 (5.0%)
graphql1 (5.0%)
gui1 (5.0%)
incomplete-aggregation1 (5.0%)
incomplete-matching1 (5.0%)
incomplete-path-sanitization1 (5.0%)
incorrect-calculation1 (5.0%)
incorrect-combination-logic1 (5.0%)
indexing1 (5.0%)
Rust rust
frontend4 (36.4%)
tui4 (36.4%)
cli3 (27.3%)
backend2 (18.2%)
library2 (18.2%)
budget1 (9.1%)
claude-code1 (9.1%)
command-routing1 (9.1%)
glyph-rendering-update1 (9.1%)
mcp1 (9.1%)
missing-functionality1 (9.1%)
missing-hook-implementation1 (9.1%)
missing-push-fallback1 (9.1%)
missing-validation-warning1 (9.1%)
missing-writer1 (9.1%)
process-management1 (9.1%)
python1 (9.1%)
testing1 (9.1%)
time-formatting1 (9.1%)
ui-redesign1 (9.1%)
TypeScript ts
backend3 (42.9%)
frontend3 (42.9%)
api2 (28.6%)
api-contract-update1 (14.3%)
architecture-boundary-violation1 (14.3%)
eth-rpc-adapter1 (14.3%)
fullstack1 (14.3%)
incomplete-validation1 (14.3%)
infinite-cache-growth1 (14.3%)
missing-data-propagation1 (14.3%)
monolithic-component1 (14.3%)
nextjs1 (14.3%)
optional-field-handling1 (14.3%)
polkadot-api1 (14.3%)
refactoring1 (14.3%)
webrtc1 (14.3%)
Global Top Bug Classes
click to expand
Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.
missing-api3 (3.0%)
incomplete-validation3 (3.0%)
unknown-defect3 (3.0%)
missing-feature3 (3.0%)
false-positive2 (2.0%)
missing-implementation2 (2.0%)
api-compatibility-fix1 (1.0%)
cache-coherency1 (1.0%)
checksum-computation-error1 (1.0%)
incomplete-deduplication-logic1 (1.0%)
missing-backend1 (1.0%)
missing-cleanup-api1 (1.0%)
missing-function-implementation1 (1.0%)
missing-state-check1 (1.0%)
module-lifecycle1 (1.0%)
toolchain-incompatibility1 (1.0%)
ui-flicker1 (1.0%)
dynamic-window-management1 (1.0%)
incorrect-logic1 (1.0%)
missing-auto-detection1 (1.0%)
schema-evolution1 (1.0%)
configurable-leeway1 (1.0%)
incorrect-parsing1 (1.0%)
insecure-default1 (1.0%)
lifecycle-state-handling1 (1.0%)
missing-backoff-logic1 (1.0%)
missing-caching1 (1.0%)
missing-config-toggle1 (1.0%)
missing-configuration1 (1.0%)
missing-context-propagation1 (1.0%)
Per-Language Bug-Class Distribution
click to expand
Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.
C c15 tagged
missing-api2 (13.3%)
api-compatibility-fix1 (6.7%)
cache-coherency1 (6.7%)
checksum-computation-error1 (6.7%)
incomplete-deduplication-logic1 (6.7%)
incomplete-validation1 (6.7%)
missing-backend1 (6.7%)
missing-cleanup-api1 (6.7%)
missing-function-implementation1 (6.7%)
missing-state-check1 (6.7%)
module-lifecycle1 (6.7%)
toolchain-incompatibility1 (6.7%)
unknown-defect1 (6.7%)
ui-flicker1 (6.7%)
C++ cpp4 tagged
dynamic-window-management1 (25.0%)
incorrect-logic1 (25.0%)
missing-auto-detection1 (25.0%)
schema-evolution1 (25.0%)
Go go19 tagged
false-positive2 (10.5%)
configurable-leeway1 (5.3%)
incorrect-parsing1 (5.3%)
insecure-default1 (5.3%)
lifecycle-state-handling1 (5.3%)
missing-backoff-logic1 (5.3%)
missing-caching1 (5.3%)
missing-config-toggle1 (5.3%)
missing-configuration1 (5.3%)
missing-context-propagation1 (5.3%)
missing-existence-check1 (5.3%)
missing-fallback1 (5.3%)
unbounded-accumulation1 (5.3%)
unknown-defect1 (5.3%)
whitespace-handling1 (5.3%)
Java java14 tagged
conditional-field-inclusion1 (7.1%)
dependency-migration1 (7.1%)
duplicate-identifier1 (7.1%)
incorrect-message1 (7.1%)
missing-api1 (7.1%)
missing-api-feature1 (7.1%)
missing-error-handling1 (7.1%)
missing-error-mapping1 (7.1%)
missing-feature1 (7.1%)
missing-metric-implementation1 (7.1%)
missing-model1 (7.1%)
missing-parameter-support1 (7.1%)
missing-response-mapping1 (7.1%)
unused-dependency1 (7.1%)
JavaScript js10 tagged
missing-feature2 (20.0%)
dependency-api-change1 (10.0%)
incomplete-diagnostics1 (10.0%)
incomplete-validation1 (10.0%)
missing-field-display1 (10.0%)
missing-field-extraction1 (10.0%)
missing-prop-support1 (10.0%)
missing-refresh-mechanism1 (10.0%)
platform-compatibility1 (10.0%)
Python py20 tagged
missing-implementation2 (10.0%)
api-refactor1 (5.0%)
api-refactoring1 (5.0%)
incomplete-aggregation1 (5.0%)
incomplete-matching1 (5.0%)
incomplete-path-sanitization1 (5.0%)
incorrect-calculation1 (5.0%)
incorrect-combination-logic1 (5.0%)
metrics-calculation-error1 (5.0%)
missing-docstring-extraction1 (5.0%)
missing-gui-config1 (5.0%)
missing-index-mapping1 (5.0%)
missing-module-infrastructure1 (5.0%)
missing-sensitive-data-redaction1 (5.0%)
missing-testing-helpers1 (5.0%)
Rust rust11 tagged
glyph-rendering-update1 (9.1%)
missing-functionality1 (9.1%)
missing-hook-implementation1 (9.1%)
missing-push-fallback1 (9.1%)
missing-validation-warning1 (9.1%)
missing-writer1 (9.1%)
time-formatting1 (9.1%)
ui-redesign1 (9.1%)
unknown-defect1 (9.1%)
wrong-command-scope1 (9.1%)
wrong-keybinding1 (9.1%)
TypeScript ts7 tagged
api-contract-update1 (14.3%)
architecture-boundary-violation1 (14.3%)
incomplete-validation1 (14.3%)
infinite-cache-growth1 (14.3%)
missing-data-propagation1 (14.3%)
monolithic-component1 (14.3%)
optional-field-handling1 (14.3%)