Spurred by recent events (https://news.ycombinator.com/item?id=8244700), this is a quick set of jotted-down thoughts about the state of "Semantic" Versioning, and why we should be fighting the good fight against it.
For a long time in the history of software, version numbers indicated the relative progress and change in a given piece of software. A major release (1.x.x) was major, a minor release (x.1.x) was minor, and a patch release was just a small patch. You could evaluate a given piece of software by name + version, and get a feeling for how far away version 2.0.1 was from version 2.8.0.
But Semantic Versioning (henceforth, SemVer), as specified at http://semver.org/, changes this to prioritize a mechanistic understanding of a codebase over a human one. Any "breaking" change to the software must be accompanied with a new major version number. It's alright for robots, but bad for us.
SemVer tries to compress a huge amount of information — the nature of the change, the percentage of users that will be affected by the change, the severity of the change (Is it easy to fix my code? Or do I have to rewrite everything?) — into a single number. And unsurprisingly, it's impossible for that single number to contain enough meaningful information.
If your package has a minor change in behavior that will "break" for 1% of your users, is that a breaking change? Does that change if the number of affected users is 10%? or 20? How about if instead, it's only a small number of users that will have to change their code, but the change for them will be difficult? — a common event with deprecated unpopular features. Semantic versioning treats all of these scenarios in the same way, even though in a perfect world the consumers of your codebase should be reacting to them in quite different ways.
Breaking changes are no fun, and we should strive to avoid them when possible. To the extent that SemVer encourages us to avoid changing our public API, it's all for the better. But to the extent that SemVer encourages us to pretend like minor changes in behavior aren't happening all the time; and that it's safe to blindly update packages — it needs to be re-evaluated.
Some pieces of software are like icebergs: a small surface area that's visible, and a mountain of private code hidden beneath. For those types of packages, something like SemVer can be helpful. But much of the code on the web, and in repositories like npm, isn't code like that at all — there's a lot of surface area, and minor changes happen frequently.
Ultimately, SemVer is a false promise that appeals to many developers — the promise of pain-free, don't-have-to-think-about-it, updates to dependencies. But it simply isn't true. Node doesn't follow SemVer, Rails doesn't do it, Python doesn't do it, Ruby doesn't do it, jQuery doesn't (really) do it, even npm doesn't follow SemVer. There's a distinction that can be drawn here between large packages and tiny ones — but that only goes to show how inappropriate it is for a single number to "define" the compatibility of any large body of code. If you've ever had trouble reconciling your npm dependencies, then you know that it's a false promise. If you've ever depended on a package that attempted to do SemVer, you've missed out on getting updates that probably would have been lovely to get, because of a minor change in behavior that almost certainly wouldn't have affected you.
If at this point you're hopping on one foot and saying — wait a minute, Node is 0.x.x — SemVer allows pre-1.0 packages to change anything at any time! You're right! And you're also missing the forest for the trees! Keeping a system that's in heavy production use at pre-1.0 levels for many years is effectively the same thing as not using SemVer in the first place.
The responsible way to upgrade isn't to blindly pull in dependencies and assume that all is well just because a version number says so — the responsible way is to set aside five or ten minutes, every once in a while, to go through and update your dependencies, and make any minor changes that need to be made at that time. If an important security fix happens in a version that also contains a breaking change for your app — you still need to adjust your app to get the fix, right?
SemVer is woefully inadequate as a scheme that determines compatibility between two pieces of code — even a textual changelog is better. Perhaps a better automated compatibility scheme is possible. One based on matching type signatures against a public API, or comparing the runs of a project's public test suite — imagine a package manager that ran the test suite of the version you're currently using against the code of the version you'd like to upgrade to, and told you exactly what wasn't going to work. But SemVer isn't that. SemVer is pretty close to the most reductive compatibility check you would be able to dream up if you tried.
If you pretend like SemVer is going to save you from ever having to deal with a breaking change — you're going to be disappointed. It's better to keep version numbers that reflect the real state and progress of a project, use descriptive changelogs to mark and annotate changes in behavior as they occur, avoid creating breaking changes in the first place whenever possible, and responsibly update your dependencies instead of blindly doing so.
Basically, Romantic Versioning, not Semantic Versioning.
All that said, okay, okay, fine — Underscore 1.7.0 can be Underscore 2.0.0. Uncle.
(typed in haste, excuse any grammar-os, will correct later)
Those documented idiomatic "features" are just convention. Those conventions assume everything out there uses SemVer, and uses it correctly (even though the OP's point is that determining breaking vs non-breaking is not always an easy thing).
The tools don't know that minor bumps are supposed to be non-breaking, you tell them to freely update to what you think/hope will be non-breaking by specifying the version constraints. The tools don't misunderstand, they just do what you tell them to. But the users might, and that's kind of the point. Doing perfect SemVer, ie: never ever breaking anything on minor (or patch) bumps; managing to get important fixes that only exist behind a major bump out to users since so many, through convention, have specified to never take a major bump, etc.
Making the assumptions that SemVer insists on is stifling on both ends. Minor bumps do break things sometimes, even patch bumps sometimes change expected behaviors and break things, so users need to be more careful than just "lock the major, take all minor and patch bumps", and devs need to do so much extra work: in deciding whether to save things up for a big major bump, whether to shoehorn improvments back to previous majors, when to stop supporting a previous major or minor.
Many many packages already do say "the version numbers of my package have a different meaning". Linux kernel, anthing with CalVer, basically all modern browsers (when's the last time Chrome or Firefox didn't have a .0 minor?) Some Python libraries effectively only bump the major and patch, either because breaking changes are either super-rare or practically unavoidable and users are expect to read the docs. Some packages basically always have minor version locked in real world usage because they're unreliable about not breaking things. Yes, changing it on the fly might cause issues, but it can be done, maybe after a major bump, after all that's what it's for!
And we did't even get into the reproducibility argument yet, and that no one should be leaving even patch level updates to be applied on the fly at deploy time. Which comes back to OP: set aside time to do upgrades on a regular basis, read the docs for stuf you use, bump versions according to how those rules apply to your usage, test thoroughly, be prepared to have to make fixes no matter the size/type of the version bump, then lock those versions for consistent behavior throughout the workflow.