Phono Stage Gain - how does it impact sound?

Currently (not a word used to show that I never heard of Ohm's Law) I'm using three moving iron cartridges. 1mV, 3mV and 5mV. My phono amp has three settings for 'sensitivity' (ie gain), expecting respective inputs of 1.3mV, 2.6mV and 5.2mV. Now I can use the pre-amp's volume control and make everything work on the medium 2.6mV input setting, but it sounds like sh*t. Better I change the phono amp's gain to match the input, where it sounds delightful. I'm not sufficiently well-versed in electronics to explain that, but I'm sure someone here will.

@dogberry makes sense to me.

Shouldn't sound like sh*t when you use the cartridge with 3mV output and the phono set for 2.6mV.  But anyway, as for audphile, the results depend in part also upon the downstream equipment.  I am surprised that the 2.6mV input does not work decently also with the 5mV cartridge, if that's what you are saying.

Sensitivity ought not affect frequency response. I presume that's mechanical cartridge break-in (or you just dont like the response of either the cart or the phono stage, which, after all, is an equalizer).

I presume your question as asked is rhetorical mostly. I'm happy to explain why its a ridiculously tough task, and therefore will have a huge impact.

G

Ran it all day while working today and added another 7-8hrs.
Sounds amazing, there’s just no other word for it. Relaxed, zero fatigue listening to it all day on and off. I believe the bass is somewhat coming around and improving with more play time on the cartridge. Threw on few classical records, Dire Straits and some new Steely Dan (Analog Productions “Everything Must Go” 45rpm). It requires 8-10 more volume clicks than my benchmark DAC does but I run the benchmark into the XP-12 with a pair of XLRs.
As I said I’m not too worried, I’ll make it work. Worst case I’ll try a higher gain phono stage and see if it makes any difference. But for now I’m going to be patient until I have around 40-50hrs on this cartridge. 
Thanks all!!!