We've performed a usability study on the new System.Text.Json APIs which will
replace JSON.NET as the default JSON experience in .NET Core 3.0:
- We focused on the primary "getting started" scenarios to validate that we picked the right type- and method names.
- We simulated what we believe to be common error cases to validate that our exception messages are understandable and actionable.
- Since we were short on time, we couldn't use the recruiting provided by the UX lab but instead brought in our own developers, which we recruited from the .NET engineering team
This has worked out well for us as we were able to get short term feedback which resulted in significant improvements to the overall API shape and usability. In the future, we should do more API focused UX studies 😊
| # | Description | Code |
|---|---|---|
| 1 | Serialize and pretty print | Link |
| 2 | Deserialize with trailing comma | Link |
| 3 | Deserialize with custom property names | Link |
| 4 | Read with DOM | Link |
| 5 | Pretty print an existing JSON file | Link |
We expect developers to...
| # | Description | Result |
|---|---|---|
| 1 | ...assume the APIs are similar/identical to JSON.NET | Validated |
| 2 | ...complete tasks 1-4 successfully | Validated |
| 3 | ...struggle/unable to complete the task 5, because it requires working with two different components which is very different from JSON.NET | Invalidated |
| 4 | ...not look at documentation for most of their coding as IntelliSense is good enough | Invalidated |
| 5 | ...be happy with the experience of using the API | Validated |
| 6 | ...plan on using System.Text.Json for their next JSON task |
Partially Validated |
- Everyone completed scenarios #1-#3, 2 didn't complete #4, 6 didn't complete #5
- The reason was always "out of time". Every person who started a scenario successfully finished it.
- We feel this validates that we picked the right scope for a 60 minute study of people from various backgrounds
- Overall people liked the new API
- Key value prop for most people was the fact that "it's now built in"
- Very few mentioned perf and the ones that did thought that it probably wouldn't make a difference in their scenarios.
- Most people said they would prefer
System.Text.Jsonfor new APIs, but would stick to JSON.NET in code that already uses it. - [Action] None, this is in line with what we expected.
- Virtually everyone tried to google for sample code in the documentation
- Everyone found our blog post and online documentation
- People new to .NET were confused that JSON.NET and
System.Text.Jsonweren't the same thing - Everyone complained that the current docs are lacking, especially sample code.
- [Action] Docs are work in progress. We plan on shipping with a fully documented API surface and sample code.
- Virtually everyone found the key type relatively quickly
- Some people were confused when types were static, rather than something that one can new up
- [Action] None. This is by-design as we try to reduce the number of allocations.
- Virtually everyone had trouble selecting the correct method on
JsonSerializer- We chose a very complicated naming convention
- [Action] We simplified it by using names that people in the study told us were looking for.
- Most of our error messages were well received because they told people what
the problem is & how they can fix them.
- One error message was unclear because we reported an unexpected character when we could have reported that we didn't expect a comment.
- [Action] We've reworded the error message.
- We had one error message that was generated by a lower layer and provided an action item that doesn't make sense when people receive it when using the higher layer component.
- [Action] We've reworded the error message to be more generic so it makes sense for both kind of consumers.
- Virtually everyone used the new Roslyn completion feature which will also show
types from namespaces not imported yet.
- However, virtually nobody realized that they were doing it.
- This caused confusion when people found other APIs that happen to include Json in them, specifically the old data contract serialization.
- [Action] Talk to the Roslyn IDE team to see if this something more
people will run into. This could prove to be especially problematic for
technologies that share type names (e.g.
UserControlin WinForms and WPF) and both are referenced by the project.
- Optional parameters and
async/awaitare usability cliffs- Many were confused by optional parameters and tried to pass them in, even though they didn't have to.
- [Action] Talk to the Roslyn IDE team to see if we can visualize them better in VS.
- [Action] Try to avoid heavy use of optional parameters for mainline APIs.
- Async is a very steep cliff. People have no idea what to do in their code to make async work. While IntelliSense shows the usage by prefixing it with await, people didn't notice that.
- [Action] Unclear; this is a broader issue.
- Virtually nobody used F1 to get help for APIs
- Surprise. I expected more people to use F1.
- Virtually everyone used F12 to explore the API ("metadata view")
- Surprise. I expected most people to explore the type via the IntelliSense completion
- Nobody used Object Browser to explorer the APIs
- Surprise. I expected at least some people to.
I can confirm I never use F1 nor Object Browser. Google All The Time.