Tom Preston-Werner, the creator of the SemVer version numbering standard, published an article a little bit ago titled Major Version Numbers are Not Sacred. The article is worth a read, but the basic argument (as best I can summarize it) is that we shouldn't be afraid of incrementing the major version number of our SemVer-adhering software projects to indicate breaking changes.
On the one hand, I agree with him: if SemVer is to be meaningful, then we obviously need to ensure that we increment the major version number any time we release a breaking change. Even if the breaking change isn't a big, sexy, marketable change. Even if it's just a change to a corner of the API that few, if any, people use. You still need to increment the major version number to indicate the backwards compatibility break. If you don't, then you're simply not adhering to SemVer. And there's nothing specifically wrong about not adhering to SemVer, but you shouldn't then claim to be adhering.
On the other hand, he then goes on to argue (if I understand him correctly) that we should therefore be willing to increment the major version number willy-nilly. And I very much disagree with him there. It is specifically because SemVer ties API breakage to the major version number that SemVer-adhering projects should be hesitant about major version bumps. Not because the version number matters, but because backwards compatibility matters.
Of course, if your project is experimental, still in alpha, just for fun, etc. then this obviously doesn't apply. But for serious projects out of alpha/beta that are intended for real use, backwards compatibility matters a lot.