You are correct about that, and I'm at least as guilty as any of intolerance at times; but as @bdp24 points out, it was a really flawed point of view on Schubert's part to dismiss the joy that others found in types of music that he didn't care for.
It struck me as surprising for someone who was a cultured man, but then we humans are nothing if not a collection of contradictions.