Created from react-d3-starter template
DeepSWE 1.1 — Model Effort × Cost × Resolution
An interactive D3 + React visualization of the DeepSWE 1.1 benchmark snapshot, plotting resolution rate against cost per task for each model × effort configuration, colored by model.
Datasets
data/github_copilot_deepswe_1_1_digitized.csv
- Size: 1950 bytes
- Rows: 32 (plus header row)
- Apparent subject: A single benchmark ("DeepSWE 1.1") with one row per
model×effortconfiguration, reporting the benchmark's resolution rate and the dollar cost per task, along with a provenance tag for the value (value_source).
Columns
| Column | Type (Munzner) | Notes |
|---|---|---|
benchmark |
Nominal | Cardinality 1 ("DeepSWE 1.1"). All rows share this value, so it carries no analytical signal; useful only as a title constant. |
model |
Nominal | Cardinality 8. Names like copilot-fusion, gpt-6.1-sol, gpt-6-astra, etc. No ordering implied by the string values. |
effort |
Ordinal | Cardinality 5 (STANDARD, MEDIUM, HIGH, XHIGH, MAX). These are an explicit ordered scale of effort tiers, so treat them as ordinal rather than nominal. Missing in 19 of 32 rows (~59%). An empty effort appears to represent "no effort tier reported / default", not a tier named "": do not coerce blanks into an ordinal position. |
resolution_rate_pct |
Quantitative | Percentage, 52.2–78.1, mean ≈ 68.1, 0 missing. Real-valued; 28 distinct values across 32 rows (so some ties exist). |
cost_per_task_usd |
Quantitative | US dollars per task, 0.08–13.98, mean ≈ 3.63, 0 missing. 31 distinct values across 32 rows. |
value_source |
Nominal | Cardinality 3 (published_snapshot, screenshot_ranking, estimated_from_plot). A data-quality / provenance flag. Note that these denote how the number was captured (verbatim publication, OCR of a ranking image, or estimated off a plot), which is a caveat on measurement noise, not a dimension to plot. |
Missing values
effort: 19 missing (blank strings). All other columns are complete.- Parsing assumption: an empty
effortcell is retained asnull/""and rendered as the label "(unspecified)" in tooltips. It is excluded from the ordinal sort position and simply sorts to the end.
Parsing / typing assumptions
- The CSV header is the first record of the file and may,
in principle, contain quoted cells with embedded newlines. All parsing is
done with D3's
csv()which uses a compliant RFC 4180 parser; we never split lines manually. - Exact parsed column keys are accessed verbatim:
benchmark,model,effort,resolution_rate_pct,cost_per_task_usd,value_source. No renaming of the raw keys occurs before+-coercion of the numeric fields. resolution_rate_pctandcost_per_task_usdare coerced with+d.x. Rows where either coercion yieldsNaNare dropped (defensive; the profiler reports no missing values for these columns, so this is a guard, not an expected filter).
Munzner type summary
- Nominal:
benchmark,model,value_source - Ordinal:
effort - Quantitative:
resolution_rate_pct,cost_per_task_usd - Temporal: none
- Geographic (coordinates or identifiers): none
Relationships and join keys
- This project contains one tabular dataset. There is no second dataset to join to, so no cross-dataset key validation is required.
- Grain / key within the dataset: the combination
(benchmark, model, effort)is the natural candidate key, but it is not guaranteed unique becauseeffortis missing for many rows — multiple rows for the samemodelcan share anulleffort, which would collide under that key. We therefore do not treat(benchmark, model, effort)as a firm key and we do not join on it. Instead every row is rendered as its own point, identified by its index in the parsed file. Points that overlap (same rounded x/y) are separated only by hover/click tooltips, not by a join. benchmarkis constant, so it is not a usable join key here.- No identifiers appear in more than one unusable form; no reconciliation between "similarly named" columns is possible or attempted.
Visualization Mappings
Chart type: scatter plot (one point per row), because the analysis question is "how does benchmark accuracy trade against dollar cost across model/effort configurations?" — a bivariate relationship over two quantitative fields.
Fields and encodings
| Field | Encoding | Channel |
|---|---|---|
cost_per_task_usd |
x position | Quantitative position (log scale) |
resolution_rate_pct |
y position | Quantitative position |
model |
hue | Nominal color |
effort |
tooltip label + ordinal legend note | — |
value_source |
tooltip label | — |
- Log x-scale: cost spans 0.08 to 13.98 (≈175×), so a linear scale
crowds low-cost points. A log scale makes the spread readable and is
appropriate for ratio-scaled dollar values. Cost is strictly positive
(min 0.08), so log is safe; the domain is padded to
[0.05, 20]. - Color: 8 models mapped to a categorical color scheme. Model names
are compared across the x-axis, so hue is the natural choice; the
ordinal
effortis deliberately not used as a color channel to avoid stacking two dimensions on one channel. - Radius: constant (fixed dot size) to avoid implying a spurious third
quantitative encoding (
sizeroles were listed in the profiler but adding a size channel would double-map information).
Interactions
- Hover: enlarges the point and shows an accessible tooltip with
model,effort(or "(unspecified)"),resolution_rate_pct,cost_per_task_usd, andvalue_source. - Click: pins the selection, drawing a focus ring; keyboard users can also cycle through points with Tab and activate with Enter/Space.
- Both interactions are pointer- and keyboard-accessible; the SVG exposes
role="img"with a live-region summary for assistive tech.
Responsive behavior: the container is measured with a
ResizeObserver; the SVG, scales and axes are re-rendered on any resize.
Rationale for excluding the other roles: benchmark is constant and
cannot be encoded; value_source is a data-quality caveat (surfaced in
the tooltip rather than as a visual channel) since making provenance a
color would compete with model identity and could be misread as a
measurement dimension.
Visualization Ideas
No project brief was provided, so these ideas are framed around the existing chart's goal. That goal is to help readers compare the cost vs. resolution trade-off across model × effort configurations in DeepSWE 1.1. All ideas use only the six existing columns. Each loads the CSV at runtime with d3.csv(), coerces the numeric fields, and treats blank effort as "(unspecified)".
Pareto Frontier Overlay
- Main question: Which configurations are not beaten on both cost and resolution?
- Fields:
cost_per_task_usd,resolution_rate_pct,model. - Encoding: Keep the scatter and add a step line through non-dominated points. Fade dominated points.
- Interaction: Toggle the frontier on and off. Label the frontier points.
- Implementation: Sort by cost and keep a running maximum of resolution rate. Draw with
d3.line().curve(d3.curveStepAfter).
Effort Ladder per Model (Slope Chart)
- Main question: How much does each step up in effort improve resolution?
- Fields:
model,effort,resolution_rate_pct. - Encoding: x is ordinal effort and y is resolution rate. One colored line per model.
- Interaction: Hover a line to highlight its model.
- Implementation: Use
scalePointoverEFFORT_ORDER. Models without an effort value are omitted or shown in a separate "(unspecified)" column.
Cost Multiplier Bars by Effort
- Main question: How much more does higher effort cost than the model's cheapest configuration?
- Fields:
model,effort,cost_per_task_usd. - Encoding: Grouped bars on a log y-axis. Each bar is cost divided by that model's minimum cost.
- Interaction: Tooltip with the raw dollar value.
- Implementation: Use
d3.rollupby model to find the minimum, then divide.
Model Ranking Bump Chart
- Main question: Does the model ranking change from one effort tier to the next?
- Fields:
model,effort,resolution_rate_pct. - Encoding: Rank on y and effort on x, with lines connecting each model's ranks.
- Interaction: Click a model to pin its line.
- Implementation: Rank within each effort tier using
d3.rankor a sort. Skip tiers where a model has no row.
Value-Source Provenance Scatter
- Main question: Are the conclusions robust to the data-capture method?
- Fields:
value_source,cost_per_task_usd,resolution_rate_pct. - Encoding: Marker shape or stroke style by source (hollow for
estimated_from_plot). - Interaction: Checkbox filters per source.
- Implementation: Use
d3.symboltypes and a Set of enabled sources.
Small Multiples by Model
- Main question: What does each model's own cost/resolution trajectory look like?
- Fields:
model,cost_per_task_usd,resolution_rate_pct,effort. - Encoding: An 8-panel grid with shared axes. Points are labeled by effort initial.
- Interaction: Hovering a point in one panel highlights it in a ghosted overview.
- Implementation: Use
d3.groupwith a CSS grid, keeping scales shared across panels.
Efficiency Ranking (Points per Dollar)
- Main question: Which configuration gives the most resolution per dollar?
- Fields:
resolution_rate_pct,cost_per_task_usd,model,effort. - Encoding: Horizontal sorted bars of rate divided by cost, colored by model.
- Interaction: A toggle between log and linear scale, because cheap configurations dominate this ratio.
- Implementation: Derive the ratio in the client. Label it as a derived value.
Cost Strip Plot by Model
- Main question: How widely does cost vary within each model?
- Fields:
model,cost_per_task_usd. - Encoding: A log-x dot strip per model row, with a range line from minimum to maximum.
- Interaction: Hover for the effort and rate.
- Implementation: Use
scaleBandfor rows andscaleLogfor x.
Resolution Range Dumbbells
- Main question: How far apart are each model's worst and best configurations?
- Fields:
model,resolution_rate_pct. - Encoding: One dumbbell per model from minimum to maximum rate, sorted by maximum.
- Interaction: Click a dumbbell to list that model's configurations.
- Implementation: Use
d3.extentper group.
Effort × Model Heatmap of Resolution
- Main question: Where in the model/effort grid is resolution highest?
- Fields:
model,effort,resolution_rate_pct. - Encoding: Cell color from a sequential scale, with the value printed in each cell. Empty cells stay blank.
- Interaction: Tooltip with cost and source.
- Implementation: Include an "(unspecified)" column. Use
scaleBandon both axes.
Effort × Model Heatmap of Cost
- Main question: Where are the expensive configurations?
- Fields:
model,effort,cost_per_task_usd. - Encoding: Log-scaled sequential color.
- Interaction: Linked highlighting with the resolution heatmap (idea 10).
- Implementation: Use
scaleSequentialLogwith a shared hover state.
Marginal Return Arrows
- Main question: Does the next effort step pay off?
- Fields:
model,effort,cost_per_task_usd,resolution_rate_pct. - Encoding: Arrows between consecutive effort points on the scatter. Green for gains and red for drops.
- Interaction: Label each arrow with Δrate and Δcost.
- Implementation: Sort the points of each model by effort rank and skip those with a null effort.
Budget Slider with Best Pick
- Main question: Given a spending cap, what is the best achievable resolution?
- Fields:
cost_per_task_usd,resolution_rate_pct,model,effort. - Encoding: A vertical cost-cap line on the scatter. Points over budget are dimmed and the best affordable point is ringed.
- Interaction: A slider on a log scale.
- Implementation: Filter by cap and take the
d3.greatestby rate.
Quadrant Scatter with Median Splits
- Main question: Which configurations are both cheaper and more accurate than typical?
- Fields:
cost_per_task_usd,resolution_rate_pct. - Encoding: Dashed median lines divide the plot into four quadrants. The upper-left quadrant is tinted.
- Interaction: Hover a quadrant to count the points in it.
- Implementation: Use
d3.median. Draw rects behind the points.
Voronoi Neighborhood Explorer
- Main question: Which configurations are closest in cost and resolution to a selected one?
- Fields:
cost_per_task_usd,resolution_rate_pct,model. - Encoding: Visible Voronoi cells tinted by model. The pinned point's nearest neighbors are connected.
- Interaction: Click to pin and show the 3 nearest neighbors.
- Implementation: The existing Delaunay can use
delaunay.neighbors(i).
Ridgeline of Resolution by Effort
- Main question: How does the distribution of resolution change with effort level?
- Fields:
effort,resolution_rate_pct. - Encoding: Stacked density curves per effort tier, plus rug ticks for the actual points.
- Interaction: Hover to show the count and the mean.
- Implementation: Use a KDE with a wide bandwidth and show n per tier. Some tiers have few rows.
Rank-Order Lollipop with Cost Annotation
- Main question: How do the models stack up at their best, and what does that cost?
- Fields:
model,resolution_rate_pct,cost_per_task_usd. - Encoding: Lollipops of each model's best rate, with the cost as a text label or a secondary bar.
- Interaction: Syncs with the Rankings panel.
- Implementation: Reuse the
modelsmemo and itsbestfield.
Parallel Coordinates
- Main question: How do effort, cost, resolution and source relate for each row?
- Fields:
effort,cost_per_task_usd,resolution_rate_pct,value_source,model. - Encoding: Four axes (one ordinal, one log, one linear, one nominal) with lines colored by model.
- Interaction: Brush any axis to filter.
- Implementation: Use
d3.brushYper axis and a band scale for the categorical axes.
Data Completeness Matrix
- Main question: Which model/effort combinations exist, and which are missing or uncertain?
- Fields:
model,effort,value_source. - Encoding: A grid with one cell per model × effort. Fill by source color and leave empty cells hatched.
- Interaction: Click a cell to jump to the point in the scatter.
- Implementation: Use
d3.group. Treat a blank effort as its own column.
Cost–Resolution Scatter with Uncertainty Halos
- Main question: How much should we trust the position of each point?
- Fields:
value_source,cost_per_task_usd,resolution_rate_pct. - Encoding: Halo radius or opacity grows for
estimated_from_plotand shrinks forpublished_snapshot. The radii are an illustrative ordinal scheme, not measured error. - Interaction: A "show uncertainty" toggle. A legend states that the halos are qualitative.
- Implementation: Map the source to a radius with
scaleOrdinal. Render halos as circles beneath the points.
Files
index.html— page shell and import map (React + D3 from CDN).index.jsx— mounts<App />.App.jsx— loads the CSV, parses, and renders the scatter plot.viz.js— D3 render entry point (axes, points, tooltip, interactions).useContainerDimensions.js— container size tracking.styles.css— page + tooltip styles.data/github_copilot_deepswe_1_1_digitized.csv— the dataset (read-only).
Getting started
The chart loads /data/github_copilot_deepswe_1_1_digitized.csv at runtime.
Edit viz.js to change the encodings; the preview updates live.
Caveats
- 19 of 32 rows have no
effortvalue; configurations with a blank effort are shown with an "(unspecified)" tooltip label and are not assigned an ordinal rank. value_sourcemixes published snapshots with values that were OCR'd from ranking screenshots or estimated from plots. Rows markedestimated_from_plotcarry more measurement uncertainty thanpublished_snapshotrows; this is a caveat, not corrected for in the chart.- Because the natural key is not provably unique (see above), overlapping points may represent distinct rows; use hover/click to read them.
Comments
Sign in to join the discussion.