The OP is slapping paint with a 12 inch wide brush.
Yes, ... most tube amps have higher output impedances that SS amps. My amp, an ARC Ref 150 SE, has an output impedance of roughly .76 ohms off the 8 ohm taps. By contrast, most SS amps have output impedances that are much, much lower, ... some close to zero.
And as other posters have noted, most speakers were voiced to be driven by a low output impedance SS amp. So, if a tube amp has a high'ish output impedance, ... yes, ... a speaker that was voiced to be driven by a SS amp, will not produce a linear FR if driven by a high'ish output impedance tube amp.
In the case of my ARC Ref 150 SE, I calculated that the FR variation between the low impedance point of my speakers (say 4 ohms in the bass region) and the high point (say 23 ohms at the 2.2K Hz mid-to-tweeter x-over point) is roughly 1.6 db. It's an Ohm's Law thing.
But that does not end the story, ...ergo my point about the 12 inch brush. As don_c55 notes, a particular tube amp/speaker combo may just sound wonderful. Just because ...
The OP should check Ralph Karsten's (Atmasphere's) posts over the years. He speaks at great length about why some tube amps sound so good, despite being "tone controls."
Just my humble opinion.
BIF