Live Dashboard of PRs collections and SWE Tasks Generation
Live state of GitHub PR collection and verifiable SWE-Bench task generation across 8 languages. Detailed per-language analysis is collapsed below — click any section to expand.
Language Progress
| Language | PRs collected | Last 1h | Last 24h | Valid SWE | Last 1h | Last 24h | Processed | Success rate |
|---|---|---|---|---|---|---|---|---|
| Cc | 621 | -31,736 | -31,736 | 8 | -9,701 | -9,701 | 441 | |
| C++cpp | 1,326 | -46,662 | -46,662 | 2 | -4,037 | -4,037 | 296 | |
| Gogo | 1,621 | -130,482 | -130,482 | 5 | -8,090 | -8,090 | 325 | |
| Javajava | 1,165 | -87,864 | -87,864 | 0 | -4,017 | -4,017 | 408 | |
| JavaScriptjs | 890 | -38,792 | -38,792 | 1 | -7,136 | -7,136 | 339 | |
| Pythonpy | 1,140 | -104,628 | -104,628 | 3 | -5,055 | -5,055 | 478 | |
| Rustrust | 787 | -71,078 | -71,078 | 3 | -5,502 | -5,502 | 255 | |
| TypeScriptts | 1,268 | -69,222 | -69,222 | 0 | -6,405 | -6,405 | 277 |
Run Parameters
| Language | Eval model (OPENAI) | Completion model (ANTHROPIC) | Concurrency | min_source_files | max_source_files |
|---|---|---|---|---|---|
| C | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| C++ | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Go | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Java | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| JavaScript | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Python | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| Rust | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
| TypeScript | Qwen3.6-35B-A3B | claude-sonnet-4-6 | 16 | 2 | 10 |
Failure Reason Breakdown
click to expand
| Language | Processed | Valid SWE | Failed | trivial_pr | validation | infra_error | timeout | workflow_error | Other |
|---|---|---|---|---|---|---|---|---|---|
| C | 441 | 8 | 433 | 227 | 3 | 175 | 28 | 0 | 0 |
| C++ | 296 | 2 | 294 | 158 | 7 | 93 | 31 | 5 | 0 |
| Go | 325 | 5 | 320 | 135 | 1 | 143 | 41 | 0 | 0 |
| Java | 408 | 0 | 408 | 200 | 1 | 172 | 32 | 3 | 0 |
| JavaScript | 339 | 1 | 338 | 174 | 1 | 110 | 47 | 6 | 0 |
| Python | 478 | 3 | 475 | 203 | 4 | 218 | 47 | 3 | 0 |
| Rust | 255 | 3 | 252 | 118 | 1 | 88 | 44 | 1 | 0 |
| TypeScript | 277 | 0 | 277 | 138 | 3 | 96 | 38 | 2 | 0 |
trivial_pr: the PR was judged by the LLM as too trivial (e.g. only config, docs, or dependency-version changes) and unsuitable as a SWE task.
validation: validation failed after task generation (the NOP agent did not return reward=0, or the ORACLE agent did not return reward=1).
infra_error: infrastructure error (Docker build failure, network timeout, insufficient disk space, etc.).
timeout: processing timed out (per-PR total timeout or Claude Code session timeout).
workflow_error: workflow error (PR metadata fetch failure, worktree creation failure, patch generation failure, etc.).
fix.patch Complexity
click to expand
| Language | Valid SWE Count | Avg fix.patch lines | Avg fix.patch hunks | Avg fix.patch files |
|---|---|---|---|---|
| C | 8 | 155.75 | 9.25 | 3.88 |
| C++ | 2 | 47.50 | 4.50 | 2.50 |
| Go | 5 | 76.40 | 3.60 | 2.00 |
| Java | 0 | 0.00 | 0.00 | 0.00 |
| JavaScript | 1 | 115.00 | 1.00 | 1.00 |
| Python | 3 | 6.67 | 2.33 | 1.67 |
| Rust | 3 | 257.67 | 11.33 | 3.33 |
| TypeScript | 0 | 0.00 | 0.00 | 0.00 |
Metric Definitions
click to expand
Difficulty score (difficulty_score)
Reads each valid task directory's solution/fix.patch, tests/, and instruction.md, scored statically with zero API calls by src/swegen/scoring.py.
The current formula uses log-scale continuous scoring to avoid mid-sized patches becoming hard too early. Weights: patch_scope 38%, logic_complexity 32%, context_breadth 15%, test_complexity 10%, instruction_complexity 5%.
Label thresholds: easy <= 4.0, medium <= 7.0, hard > 7.0.
Tag generation and display
tags are not computed live by the dashboard; they are generated by the LLM from PR information when swegen builds the task, and written to [metadata].tags in task.toml.
The prompt asks for tags in four parts: programming language, project layer/domain, framework/library or specific topic, and a domain-independent bug class (e.g. missing-fallback, incomplete-validation). The dashboard reads existing task.toml files, counts each language's tag occurrences and share, and treats the 4th tag as the bug class for the Bug-Class panels below.
fix.patch statistics
Patch stats come from each valid task's solution/fix.patch, filtering code files by language extension, consistent with the code-only stats in upload_march_swe_to_hf.py.
Avg fix.patch lines counts added/removed lines in code-file diffs; Avg fix.patch hunks counts @@ hunks; Avg fix.patch files counts the code files involved.
difficulty_label Distribution
click to expand
| Language | easy / medium / hard | easy | medium | hard |
|---|---|---|---|---|
| C | 0 / 6 / 2 | 0 | 6 | 2 |
| C++ | 0 / 0 / 2 | 0 | 0 | 2 |
| Go | 0 / 4 / 1 | 0 | 4 | 1 |
| Java | no data | 0 | 0 | 0 |
| JavaScript | 0 / 1 / 0 | 0 | 1 | 0 |
| Python | 0 / 3 / 0 | 0 | 3 | 0 |
| Rust | 0 / 1 / 2 | 0 | 1 | 2 |
| TypeScript | no data | 0 | 0 | 0 |
difficulty_score Overview
click to expand
| Language | count | min | p25 | median | mean | p75 | max |
|---|---|---|---|---|---|---|---|
| C | 8 | 5.3 | 5.8 | 6.4 | 6.60 | 7.1 | 8.7 |
| C++ | 2 | 7.2 | 7.7 | 8.2 | 8.25 | 8.8 | 9.3 |
| Go | 5 | 4.2 | 4.2 | 5.5 | 5.42 | 5.7 | 7.5 |
| Java | 0 | 0.0 | 0.0 | 0.0 | 0.00 | 0.0 | 0.0 |
| JavaScript | 1 | 6.0 | 6.0 | 6.0 | 6.00 | 6.0 | 6.0 |
| Python | 3 | 4.4 | 4.8 | 5.3 | 5.30 | 5.8 | 6.2 |
| Rust | 3 | 6.4 | 7.5 | 8.7 | 8.10 | 8.9 | 9.2 |
| TypeScript | 0 | 0.0 | 0.0 | 0.0 | 0.00 | 0.0 | 0.0 |
Global Top Tags
click to expand
Per-Language Tag Distribution
click to expand
Global Top Bug Classes
click to expand
Bug class is the 4th tag in task.toml -> [metadata].tags: a domain-independent label describing the defect mechanism (e.g. missing-fallback, incomplete-validation, off-by-one-error). Generated by the LLM during swegen create and backfilled into legacy 3-tag tasks via swegen backfill-tags.
Per-Language Bug-Class Distribution
click to expand
Top bug classes per mainstream language. Counts are over tasks whose task.toml already carries a 4-tag entry; tasks still on the legacy 3-tag schema do not contribute until the backfill catches up.