Editor Enhancements

[kalmdown]

Course is great! A few wishlist items for editor…

1. Something Observable does that is very helpful to learning is showing matching braces, brackets, parens, etc.
2. Would be nice if editor could compare data to code and see if there were capitalization or spelling mismatches.
3. Text selection is hard to see - dif between highlight and background is not dif enough.
4. Font in your videos makes a ligature of =>, may help to either use same font in editor or not use ligatures.
5. “I don’t what is wrong” help - opens a side panel and suggests likely solutions paths - “Have you looked at the Console?”, “Are you seeing…”, “You may not have…”

[curran]

@kalmdown Thank you so much for writing these up! It is incredibly useful

@kalmdown

showing matching braces, brackets, parens, etc.

Great idea! This may well be within reach. Currently VizHub uses CodeMirror version 5 as the editor. I hope to upgrate it to CodeMirror 6, which would feel better overall I think.

@kalmdown

compare data to code and see if there were capitalization or spelling mismatches.

This feels like an extremely challenging problem that has never been solved before. If you know of any editors or environments that do this, please let me know!

@kalmdown

Text selection is hard to see - dif between highlight and background is not dif enough.

Totally fixable. Will consider changing the colors to be more pronounced.

@kalmdown

Font in your videos makes a ligature of =>, may help to either use same font in editor or not use ligatures.

Yeah this is a funny one. What do you think of the ligatures? I got them to work a long time ago and thought it was really cool. I still think it’s really cool. However, loading that custom built font added something like an extra 300KB to the page load, and at the time I was trying to fight against bloat in the software. I researched quite a lot about the “system font” concept, where no additional fonts are loaded. I did end up adopting a system font stack for VizHub. The appearance may be nicer with custom fonts, and honestly I’m still torn as to whether loading additional design-oriented fonts would be worth it, for VizHub in particulary. There are lots of tradeoffs, but at the moment my feeling is that I want the software to be as lightweight as possible, for example working well on mobile devices, and loading custom fonts would harm that goal I think.

@kalmdown

“I don’t what is wrong” help - opens a side panel and suggests likely solutions paths - “Have you looked at the Console?”, “Are you seeing…”, “You may not have…”

A very cool idea! Indeed, I could imagine building a little unobtrusive “help” icon somewhere on the page that pops up an information panel of sorts that just has some text on it, like you said “Check the console for errors.” etc. This could also suggest to search here on the forum, as there is at this points lots of actual content here, resulting from when folks got stuck and reached out, and it often does contain solutions.

Thank you so much for posting these ideas. I will consider them.

[kalmdown]

I think catching all misspellings, miscapitalizations is difficult. But catching any is helpful. In the case you helped me with all it would have taken is comparing column headers vs my code. By taking it in small steps - 1st data col headers, next code within project, then… a nice tool could be built up over time. Another way to do it would be having a “Are you stuck?” button on the editor, and start creating tools that help them debug.

A good project for students might be for them to catalog the errors that req them to Google/SO them in a spreadsheet (given to them at beginning of the course) and use that as the foundation for classifying errors, how often they happen and whether they are worth creating specialized tools to reduce them. At a min you can see issues learners have with your course, in future can grow to be a tool to help learners understand debugging better.

I suggest you not use a font with ligatures as each specific character has meaning and people without background in font design will not realize they are two characters.

[curran]

@kalmdown

all it would have taken is comparing column headers vs my code

That’s all? To sketch out what a solution might look like, which would automatically suggest corrections in spelling errors like the issue I helped you with:

• Write some code that detects if any CSV files are fetched at runtime, when the viz runs.
  • How to know if any CSV files are fetched at runtime? We’ll need to create a proxy for fetch that runs inside the iframe runner, then messages back to the VizHub app when a CSV is loaded. This proxy would need to extract the header row of the CSV file.
• When the runtime gets the message that a CSV file has been loaded, it would store the column names internally somehow.
• After getting the message that a CSV file has been loaded, the editor should examine the file that you currently have open and extract all string literals.
  • How to extract all string literals? We’ll need to use a full JavaScript parser to parse the code into an abstract syntax tree, then traverse that tree to isolate string literals.
• Once we have the list of column name and the list of string literals, we must compare all pairs of these (which would take N * M time where N is the number of columns and M is the number of string literals in the code).
  • How to compare each pair of strings? We would probably want to compute some sort of edit distance. Which edit distance metric to use? We could choose between Levenshtein distance, Longest common subsequence, Hamming distance, Damerau–Levenshtein distance, or Jaro distance. We’d need to research each of those distance metrics to see which ones are appropriate, then research to see if there are any JavaScript libraries available that implement those distance metrics.
  • Because this step is computationally expensive, we would want to make this run in a Web Worker. It should also be debounced.
• Once we know the edit distances between all pairs of strings, we need to come up with a heuristic for when to show a warning. If two strings are substantially different, that’s fine. If two strings are exactly the same, that’s fine. If two strings have a small edit distance (in your case ‘RGB hex value’ vs. ‘RGB Hex value’), then the editor should show a warning. How small should that threshold be? Let’s say, if they differ by 1 or 2 characters, we show the warning.
• What about false positives? What if, for example, there were a menu where we wanted the label to say “RGB Hex value”, but the field in the data was “RGB hex value”? This would be valid code, and yet the editor would show a warning anyway. Is this acceptable behavior? No one likes to see warnings if they know everything is correct. So maybe we would want to invent some way of supressing the warning, or design it such that it’s very unobtrusive.
• Once we know what warning to show, we’d need to figure out how to integrate with CodeMirror to build the UI that shows the warning inside the editor. How should this warning look? A small icon in the sidebar with a tooltip that pops up? An inline warning inside the editor itself? There are all sorts of design tradeoffs and the design itself of how to show the warning would be substantial effort.

@kalmdown

I suggest you not use a font with ligatures as each specific character has meaning and people without background in font design will not realize they are two characters.

That’s a great point. I do agree with that.

[kalmdown]

You seem to have it worked out! Is it worth 1st only looking for capitalization errors? Would that be a smaller effort?

Can this be cached, so when further editing is done evaluated code doesn’t get reevaluated?

@curran

Once we have the list of column name and the list of string literals, we must compare all pairs of these (which would take N * M time where N is the number of columns and M is the number of string literals in the code).

In the case below it seems that the label would be inside a string, not being used as a variable… In the interest of making the an MVP would it be easier to ignore data?

@curran

What about false positives? What if, for example, there were a menu where we wanted the label to say “RGB Hex value”, but the field in the data was “RGB hex value”? This would be valid code, and yet the editor would show a warning anyway. Is this acceptable behavior? No one likes to see warnings if they know everything is correct. So maybe we would want to invent some way of suppressing the warning, or design it such that it’s very unobtrusive.