Viewing a saved revision (read-only)
yAT8hKNnJajGsXzB8CNVN
Created from react-d3-starter template
DeepSWE 1.1 — Model Effort × Cost × Resolution
An interactive D3 + React visualization of the DeepSWE 1.1 benchmark snapshot, plotting resolution rate against cost per task for each model × effort configuration, colored by model.
Datasets
data/github_copilot_deepswe_1_1_digitized.csv
- Size: 1950 bytes
- Rows: 32 (plus header row)
- Apparent subject: A single benchmark ("DeepSWE 1.1") with one row per
model×effortconfiguration, reporting the benchmark's resolution rate and the dollar cost per task, along with a provenance tag for the value (value_source).
Columns
| Column | Type (Munzner) | Notes |
|---|---|---|
benchmark |
Nominal | Cardinality 1 ("DeepSWE 1.1"). All rows share this value, so it carries no analytical signal; useful only as a title constant. |
model |
Nominal | Cardinality 8. Names like copilot-fusion, gpt-6.1-sol, gpt-6-astra, etc. No ordering implied by the string values. |
effort |
Ordinal | Cardinality 5 (STANDARD, MEDIUM, HIGH, XHIGH, MAX). These are an explicit ordered scale of effort tiers, so treat them as ordinal rather than nominal. Missing in 19 of 32 rows (~59%). An empty effort appears to represent "no effort tier reported / default", not a tier named "": do not coerce blanks into an ordinal position. |
resolution_rate_pct |
Quantitative | Percentage, 52.2–78.1, mean ≈ 68.1, 0 missing. Real-valued; 28 distinct values across 32 rows (so some ties exist). |
cost_per_task_usd |
Quantitative | US dollars per task, 0.08–13.98, mean ≈ 3.63, 0 missing. 31 distinct values across 32 rows. |
value_source |
Nominal | Cardinality 3 (published_snapshot, screenshot_ranking, estimated_from_plot). A data-quality / provenance flag. Note that these denote how the number was captured (verbatim publication, OCR of a ranking image, or estimated off a plot), which is a caveat on measurement noise, not a dimension to plot. |
Missing values
effort: 19 missing (blank strings). All other columns are complete.- Parsing assumption: an empty
effortcell is retained asnull/""and rendered as the label "(unspecified)" in tooltips. It is excluded from the ordinal sort position and simply sorts to the end.
Parsing / typing assumptions
- The CSV header is the first record of the file and may,
in principle, contain quoted cells with embedded newlines. All parsing is
done with D3's
csv()which uses a compliant RFC 4180 parser; we never split lines manually. - Exact parsed column keys are accessed verbatim:
benchmark,model,effort,resolution_rate_pct,cost_per_task_usd,value_source. No renaming of the raw keys occurs before+-coercion of the numeric fields. resolution_rate_pctandcost_per_task_usdare coerced with+d.x. Rows where either coercion yieldsNaNare dropped (defensive; the profiler reports no missing values for these columns, so this is a guard, not an expected filter).
Munzner type summary
- Nominal:
benchmark,model,value_source - Ordinal:
effort - Quantitative:
resolution_rate_pct,cost_per_task_usd - Temporal: none
- Geographic (coordinates or identifiers): none
Relationships and join keys
- This project contains one tabular dataset. There is no second dataset to join to, so no cross-dataset key validation is required.
- Grain / key within the dataset: the combination
(benchmark, model, effort)is the natural candidate key, but it is not guaranteed unique becauseeffortis missing for many rows — multiple rows for the samemodelcan share anulleffort, which would collide under that key. We therefore do not treat(benchmark, model, effort)as a firm key and we do not join on it. Instead every row is rendered as its own point, identified by its index in the parsed file. Points that overlap (same rounded x/y) are separated only by hover/click tooltips, not by a join. benchmarkis constant, so it is not a usable join key here.- No identifiers appear in more than one unusable form; no reconciliation between "similarly named" columns is possible or attempted.
Visualization Mappings
Chart type: scatter plot (one point per row), because the analysis question is "how does benchmark accuracy trade against dollar cost across model/effort configurations?" — a bivariate relationship over two quantitative fields.
Fields and encodings
| Field | Encoding | Channel |
|---|---|---|
cost_per_task_usd |
x position | Quantitative position (log scale) |
resolution_rate_pct |
y position | Quantitative position |
model |
hue | Nominal color |
effort |
tooltip label + ordinal legend note | — |
value_source |
tooltip label | — |
- Log x-scale: cost spans 0.08 to 13.98 (≈175×), so a linear scale
crowds low-cost points. A log scale makes the spread readable and is
appropriate for ratio-scaled dollar values. Cost is strictly positive
(min 0.08), so log is safe; the domain is padded to
[0.05, 20]. - Color: 8 models mapped to a categorical color scheme. Model names
are compared across the x-axis, so hue is the natural choice; the
ordinal
effortis deliberately not used as a color channel to avoid stacking two dimensions on one channel. - Radius: constant (fixed dot size) to avoid implying a spurious third
quantitative encoding (
sizeroles were listed in the profiler but adding a size channel would double-map information).
Interactions
- Hover: enlarges the point and shows an accessible tooltip with
model,effort(or "(unspecified)"),resolution_rate_pct,cost_per_task_usd, andvalue_source. - Click: pins the selection, drawing a focus ring; keyboard users can also cycle through points with Tab and activate with Enter/Space.
- Both interactions are pointer- and keyboard-accessible; the SVG exposes
role="img"with a live-region summary for assistive tech.
Responsive behavior: the container is measured with a
ResizeObserver; the SVG, scales and axes are re-rendered on any resize.
Rationale for excluding the other roles: benchmark is constant and
cannot be encoded; value_source is a data-quality caveat (surfaced in
the tooltip rather than as a visual channel) since making provenance a
color would compete with model identity and could be misread as a
measurement dimension.
Files
index.html— page shell and import map (React + D3 from CDN).index.jsx— mounts<App />.App.jsx— loads the CSV, parses, and renders the scatter plot.viz.js— D3 render entry point (axes, points, tooltip, interactions).useContainerDimensions.js— container size tracking.styles.css— page + tooltip styles.data/github_copilot_deepswe_1_1_digitized.csv— the dataset (read-only).
Getting started
The chart loads /data/github_copilot_deepswe_1_1_digitized.csv at runtime.
Edit viz.js to change the encodings; the preview updates live.
Caveats
- 19 of 32 rows have no
effortvalue; configurations with a blank effort are shown with an "(unspecified)" tooltip label and are not assigned an ordinal rank. value_sourcemixes published snapshots with values that were OCR'd from ranking screenshots or estimated from plots. Rows markedestimated_from_plotcarry more measurement uncertainty thanpublished_snapshotrows; this is a caveat, not corrected for in the chart.- Because the natural key is not provably unique (see above), overlapping points may represent distinct rows; use hover/click to read them.