SWE-gen/instance
Updated 2026-07-21 00:17:10 BJT · next 2026-07-21 01:17:11 BJT
Overview

Live Dashboard of PRs collections and SWE Tasks Generation

Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.

Total PRs collected
8,818
822 unique repos · 1h 0 / 24h -580,464
Total valid SWE
51
20 unique repos · 1h +25 / 24h -49,914
Overall success rate
1.8%
Valid SWE / processed 2,819
Mean difficulty_score
6.82
median 6.6, count 51

Language Progress

LanguagePRs collectedLast 1hLast 24hValid SWELast 1hLast 24hProcessedSuccess rate
Cc6210-31,73613+5-9,696441
2.9%
C++cpp1,3260-46,66220-4,037296
0.7%
Gogo1,6210-130,4828+3-8,087325
2.5%
Javajava1,1650-87,8646+5-4,011408
1.5%
JavaScriptjs8900-38,7925+3-7,132339
1.5%
Pythonpy1,1400-104,6287+3-5,051478
1.5%
Rustrust7870-71,0787+3-5,498255
2.7%
TypeScriptts1,2680-69,2223+3-6,402277
1.1%

Run Parameters

LanguageEval model (OPENAI)Completion model (ANTHROPIC)Concurrencymin_source_filesmax_source_files
CQwen3.6-35B-A3Bclaude-sonnet-4-616210
C++Qwen3.6-35B-A3Bclaude-sonnet-4-616210
GoQwen3.6-35B-A3Bclaude-sonnet-4-616210
JavaQwen3.6-35B-A3Bclaude-sonnet-4-616210
JavaScriptQwen3.6-35B-A3Bclaude-sonnet-4-616210
PythonQwen3.6-35B-A3Bclaude-sonnet-4-616210
RustQwen3.6-35B-A3Bclaude-sonnet-4-616210
TypeScriptQwen3.6-35B-A3Bclaude-sonnet-4-616210

Failure Reason Breakdown

click to expand
LanguageProcessedValid SWEFailedtrivial_prvalidationinfra_errortimeoutworkflow_errorOther
C4411342822731752800
C++29622941587933150
Go325831713511434100
Java408640220011723230
JavaScript339533417411104360
Python478747120342184730
Rust25572481181884410
TypeScript27732741383963820

trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.

validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).

infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).

timeout: processing timed out (per-PR total timeout or Claude Code session timeout).

workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).

fix.patch Complexity

click to expand
LanguageValid SWE CountAvg fix.patch linesAvg fix.patch hunksAvg fix.patch files
C13144.5410.773.77
C++247.504.502.50
Go862.383.752.00
Java681.0011.334.83
JavaScript535.003.602.00
Python7154.145.432.00
Rust7329.1411.144.29
TypeScript390.3315.009.67

Metric Definitions

click to expand

Difficulty score (difficulty_score)

Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.

The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.

Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.

Tag generation and display

tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.

The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.

fix.patch statistics

Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.

Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.

difficulty_label Distribution

click to expand
Languageeasy / medium / hardeasymediumhard
C
0 / 7 / 6
076
C++
0 / 0 / 2
002
Go
0 / 6 / 2
062
Java
0 / 3 / 3
033
JavaScript
0 / 4 / 1
041
Python
0 / 4 / 3
043
Rust
0 / 2 / 5
025
TypeScript
0 / 2 / 1
021

difficulty_score Overview

click to expand
Languagecountminp25medianmeanp75max
C134.75.87.07.088.78.9
C++27.27.78.28.258.89.3
Go83.74.14.85.256.17.5
Java65.75.97.27.539.29.7
JavaScript54.75.25.35.666.07.1
Python74.44.86.26.718.59.7
Rust76.47.68.98.319.29.2
TypeScript35.05.56.06.136.77.4

Global Top Tags

click to expand
library25 (49.0%)
backend14 (27.5%)
cli7 (13.7%)
frontend5 (9.8%)
logging4 (7.8%)
static-analysis3 (5.9%)
react3 (5.9%)
missing-api2 (3.9%)
error-handling2 (3.9%)
schema-evolution2 (3.9%)
false-positive2 (3.9%)
api2 (3.9%)
missing-feature2 (3.9%)
api-compatibility-fix1 (2.0%)
build-system1 (2.0%)
cache-coherency1 (2.0%)
canfigger1 (2.0%)
config-parsing1 (2.0%)
incomplete-validation1 (2.0%)
libbpf1 (2.0%)
memory-management1 (2.0%)
missing-cleanup-api1 (2.0%)
module-lifecycle1 (2.0%)
parsing1 (2.0%)
python1 (2.0%)
qmk1 (2.0%)
testing1 (2.0%)
toolchain-incompatibility1 (2.0%)
unknown-defect1 (2.0%)
missing-function-implementation1 (2.0%)

Per-Language Tag Distribution

click to expand

C c

library12 (92.3%)
logging3 (23.1%)
missing-api2 (15.4%)
api-compatibility-fix1 (7.7%)
backend1 (7.7%)
build-system1 (7.7%)
cache-coherency1 (7.7%)
canfigger1 (7.7%)
config-parsing1 (7.7%)
incomplete-validation1 (7.7%)
libbpf1 (7.7%)
memory-management1 (7.7%)
missing-cleanup-api1 (7.7%)
module-lifecycle1 (7.7%)
parsing1 (7.7%)
python1 (7.7%)
qmk1 (7.7%)
testing1 (7.7%)
toolchain-incompatibility1 (7.7%)
unknown-defect1 (7.7%)

C++ cpp

library2 (100.0%)
schema-evolution2 (100.0%)
missing-auto-detection1 (50.0%)
podio1 (50.0%)

Go go

library4 (50.0%)
static-analysis3 (37.5%)
backend2 (25.0%)
cli2 (25.0%)
false-positive2 (25.0%)
context1 (12.5%)
incorrect-parsing1 (12.5%)
insecure-default1 (12.5%)
missing-context-propagation1 (12.5%)
regex1 (12.5%)
structtag1 (12.5%)
tls1 (12.5%)
whitespace-handling1 (12.5%)
wrong-function-resolution1 (12.5%)
configurable-leeway1 (12.5%)
webhooks1 (12.5%)

Java java

backend6 (100.0%)
api1 (16.7%)
missing-response-mapping1 (16.7%)
ledger1 (16.7%)
unused-dependency1 (16.7%)
error-handling1 (16.7%)
incorrect-message1 (16.7%)
missing-model1 (16.7%)
payment-api1 (16.7%)
missing-error-handling1 (16.7%)
payment-validation1 (16.7%)
api-response1 (16.7%)
conditional-field-inclusion1 (16.7%)

JavaScript js

frontend3 (60.0%)
react3 (60.0%)
library2 (40.0%)
missing-feature2 (40.0%)
dependency-api-change1 (20.0%)
tar1 (20.0%)
missing-prop-support1 (20.0%)
ccda1 (20.0%)
missing-field-extraction1 (20.0%)

Python py

library4 (57.1%)
cli2 (28.6%)
autosubmit1 (14.3%)
backend1 (14.3%)
gui1 (14.3%)
incomplete-path-sanitization1 (14.3%)
incorrect-calculation1 (14.3%)
missing-gui-config1 (14.3%)
path-sanitization1 (14.3%)
status-priority-handling1 (14.3%)
workflow1 (14.3%)
incomplete-matching1 (14.3%)
string-matching1 (14.3%)
logging1 (14.3%)
missing-sensitive-data-redaction1 (14.3%)
missing-implementation1 (14.3%)
template-matching1 (14.3%)

Rust rust

cli3 (42.9%)
backend2 (28.6%)
command-routing1 (14.3%)
frontend1 (14.3%)
library1 (14.3%)
missing-functionality1 (14.3%)
missing-validation-warning1 (14.3%)
process-management1 (14.3%)
tui1 (14.3%)
ui-redesign1 (14.3%)
validation1 (14.3%)
wrong-command-scope1 (14.3%)
budget1 (14.3%)
missing-writer1 (14.3%)
claude-code1 (14.3%)
missing-hook-implementation1 (14.3%)
mcp1 (14.3%)
missing-push-fallback1 (14.3%)

TypeScript ts

backend2 (66.7%)
api1 (33.3%)
api-contract-update1 (33.3%)
frontend1 (33.3%)
infinite-cache-growth1 (33.3%)
polkadot-api1 (33.3%)
eth-rpc-adapter1 (33.3%)
optional-field-handling1 (33.3%)

Global Top Bug Classes

click to expand

Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.

missing-api2 (3.9%)
false-positive2 (3.9%)
missing-feature2 (3.9%)
api-compatibility-fix1 (2.0%)
cache-coherency1 (2.0%)
incomplete-validation1 (2.0%)
missing-cleanup-api1 (2.0%)
module-lifecycle1 (2.0%)
toolchain-incompatibility1 (2.0%)
unknown-defect1 (2.0%)
missing-function-implementation1 (2.0%)
incomplete-deduplication-logic1 (2.0%)
missing-backend1 (2.0%)
missing-state-check1 (2.0%)
missing-auto-detection1 (2.0%)
schema-evolution1 (2.0%)
incorrect-parsing1 (2.0%)
insecure-default1 (2.0%)
missing-context-propagation1 (2.0%)
whitespace-handling1 (2.0%)
wrong-function-resolution1 (2.0%)
configurable-leeway1 (2.0%)
missing-response-mapping1 (2.0%)
unused-dependency1 (2.0%)
incorrect-message1 (2.0%)
missing-model1 (2.0%)
missing-error-handling1 (2.0%)
conditional-field-inclusion1 (2.0%)
dependency-api-change1 (2.0%)
missing-prop-support1 (2.0%)

Per-Language Bug-Class Distribution

click to expand

Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.

C c13 tagged

missing-api2 (15.4%)
api-compatibility-fix1 (7.7%)
cache-coherency1 (7.7%)
incomplete-validation1 (7.7%)
missing-cleanup-api1 (7.7%)
module-lifecycle1 (7.7%)
toolchain-incompatibility1 (7.7%)
unknown-defect1 (7.7%)
missing-function-implementation1 (7.7%)
incomplete-deduplication-logic1 (7.7%)
missing-backend1 (7.7%)
missing-state-check1 (7.7%)

C++ cpp2 tagged

missing-auto-detection1 (50.0%)
schema-evolution1 (50.0%)

Go go8 tagged

false-positive2 (25.0%)
incorrect-parsing1 (12.5%)
insecure-default1 (12.5%)
missing-context-propagation1 (12.5%)
whitespace-handling1 (12.5%)
wrong-function-resolution1 (12.5%)
configurable-leeway1 (12.5%)

Java java6 tagged

missing-response-mapping1 (16.7%)
unused-dependency1 (16.7%)
incorrect-message1 (16.7%)
missing-model1 (16.7%)
missing-error-handling1 (16.7%)
conditional-field-inclusion1 (16.7%)

JavaScript js5 tagged

missing-feature2 (40.0%)
dependency-api-change1 (20.0%)
missing-prop-support1 (20.0%)
missing-field-extraction1 (20.0%)

Python py7 tagged

incomplete-path-sanitization1 (14.3%)
incorrect-calculation1 (14.3%)
missing-gui-config1 (14.3%)
status-priority-handling1 (14.3%)
incomplete-matching1 (14.3%)
missing-sensitive-data-redaction1 (14.3%)
missing-implementation1 (14.3%)

Rust rust7 tagged

missing-functionality1 (14.3%)
missing-validation-warning1 (14.3%)
ui-redesign1 (14.3%)
wrong-command-scope1 (14.3%)
missing-writer1 (14.3%)
missing-hook-implementation1 (14.3%)
missing-push-fallback1 (14.3%)

TypeScript ts3 tagged

api-contract-update1 (33.3%)
infinite-cache-growth1 (33.3%)
optional-field-handling1 (33.3%)