You're right, headphones can expose loudspeaker issues rather e asily. Especially those to do with transient response!
However, there are certain problems with headphones.
They might need a special amp.
Finding one with a flat frequency response is not easy. Frequency anomalies seem far easier to identify with headphones.
Headphones just don't have the same visual appeal that loudspeakers can have.
Headphones also don't really work well in family environments where headphones might be seen as a particularly selfish indulgence?
They're often also seen as a luxury item.
Whilst they can remove possible room issues there might be problems with finding a good fit for every head etc.
For some, the lack of tweaking options might be an issue.
For me, it's one of comfort. I guess I'm rather fidgety and don't like being tied down in one place for too long.