... switch 2 on the left side is the 47ohm resistive load, which is the only dip I have in the down position.
That’s the problem, or at least most of it!!
The 47K load that is recommended for the cartridge means 47,000 ohms, not 47 ohms (which would only be suitable for some low output moving coil cartridges).
When you put left channel switches 1 through 4 and right channel switches 7 through 10 to the up position you’ll be applying the proper 47,000 ohm load to the cartridge, and I’m sure you’ll notice a dramatic improvement.
Regards,
-- Al