Weak Link in Vinyl Playback

@owl9113 I’m just going by the manufacturer’s specifications for your cartridge, AT-VM95ML which is not that much different than the Shibata-tipped model (except it is a lot cheaper). The impedance loading at the phono stage should be 47 kilOhms and the capacitance loading should be between 100 and 200 picoFarads. These are fairly standard values. Your cartridge puts out 3.5 milliVolts, which is higher than the standard 2.5 mV, you should be fine at a gain of ‘1’ (36 dB of gain, although Ifi allows you to move up to a gain of ‘2’ if the system is not producing enough volume. However, as the Ifi manual states, on p. 18 in their FAQ “How do I Know which Cartridge Setting is Right for Me” that you may well not be able to match the volume of CDs or downloads because vinyl recordings are less ‘loud’ since they encompass a greater dynamic range than CDs or downloads [due to the ‘loudness wars of the 80’s, etc.] (contra @nlitworld above, and from my experience, vinyl recordings are quieter, but the needles bounce more).
Regarding the Schiit products, if I had it to do over again, I might choose the Freya S, I still pick up a gain of 4 (12 dB), but I don’t have the heat and distortion from the tubes, not to mention the ability to stack other items on top, like a Schiit DAC or an Ifi phono stage. The difference for me, between a gain of 1 (0 dB) and a gain of 4 (12 dB) was about a 1/4 turn on the dial, maybe a bit more. If you’re maxed out at a gain of 1 on Saga, you should expect to get the same volume at 2 or 3:00 that you’re now getting at 5:00 (considering the volume knob as a clock face. It sounds like that is what you’re looking for.

Edit: The Crutchfield add cited various capacitance loadings, but I could find no reference in the Ifi manual. Again, the capacitance issue, while critical if not correct, should be ‘close enough’ at this point in your audio journey. You can keep it in mind when you decide to upgrade from the Ifi Zen in your system now, but I still am with Herb on this one, your immediate problem in getting satisfactory performance from your system is the lack of gain at the preamp. Scratch where you itch, you’ll feel better for it.

@owl9113, @jasonbourne71 makes a point about passive volume controls. They interact with the output impedance of your sources, in this case the DAC and your phono stage, both of which have to drive your amplifier thru the passive control, which can really increase the source impedance that the amplifier sees (this is bad, in case there's any question). Its highly unlikely that both have the same output impedance! Because of that its very possible that the phono section is working alright, but is placed in a bad light simply on this account. So its worth trying to run the line stage active rather than passive.

@atmasphere @jasonbourne71 I actually just turned off the active mode for the first time last week, prior to posting which is why I said I wasn’t using it. I switched it back on for now - thanks for the clarification. That seems to further the inability of the Saga to run gain being the problem.

@oldrooney I have no allegiance to Schiit or anything, but the Freya S does seem to get legitimately good reviews and measures well. If there’s anything else in this price range with similar performance, I’m certainly open to hearing it.

It seems to me that you are looking for greater dynamics. To achieve this you need to increase the gain. When you increased the gain on the phonostage from 1 to 2 the system sounded better. Gain 2 is for cartridges with less that 2mV. Gain 1 is for cartridges greater than 2mV. You are stuck in the middle. Now if the loading of the MM gain 1 is preferred then you need a MM cartridge having greater output, say 3.5-5.0 mV.  

DACs output 2 or greater V. Passive preamps work well with that output. With vinyl an active preamp is often preferred. 

@mesch looks like my cartridge is rated at 3.5 mV, so level 1 is correct. I suppose I could always run the preamp active for vinyl, passive for digital. This would still apply to the upgraded preamp I was considering as well.