from Matt Drance:
Struggling to explain to a friend why there’s no software equivalent to NTSB crash ratings. “It’s complicated” is honest, but not acceptable
The explanation is pretty straightforward. Cars are pretty uniform. They have changed a lot, but fundamentally, they are used the same as they have been for over a century, they fail in very consistent ways, and we've thus been able to make some good well-defined tests to determine how safe a car is in the event of a crash.
Software isn't like that. Web applications may have started to follow some consistent structure, and we have some patterns we follow pretty commonly (like talking over HTTP), but the fact remains that the apps and services are super un-uniform.
That poses a real difficulty if you want to develop some standardized tests. It would be pretty easy to develop some standardized tests for an app that only spoke HTTP and used a RESTful protocol to talk to one specific web browser using JSON, but real life is more complicated than that. You have a variety of different clients, some of which are web browsers (all communicating with servers using their own specific dialects), some of which are native apps (whose implementation languages are incredibly varied and hard to look into), and you might even have some embedded apps somewhere in there. A test that would produce data about the security of software in general that would be of any value would be so onerous that it'd probably take years to run through all the different test examples that the test would have, let alone the difficulty in writing such a test.
Cars are easier. They are all generally the same shape and structure. They're all driven on roads using the tires. They are all subject to the same limitations of being a car with wheels on the road, and so we have figured out a lot of common ways cars can get into accidents and we can score cars on specific criteria to ensure they strike a good balance between being safe and being practical for everyday use.
For determining security of an online service and its companion apps, the best you can really do is get a human who can look at the design and try to guess at what the vulnerabilities might be. There are some best practices we've put together over the years (use SSL, store passwords as a salted hash so that if retrieved, no one knows them, protect your data centers from physical attack, etc.) and these best practices help us a lot, but they're never as slick and reproducible as the process we use to test cars, and I'm not sure they could be.