Why not? Consider the case to be an ideal current amp with the appropriate series resistance added so the input impedance is 5Ω. In this gedanken world the ideal voltage amp and the ideal current amp sound identical.
It seems to me that you are still conflating virtual ground and actual ground as the same thing! As I said before this leads to confusion.
So the 'Why not?' is the same as before: Because that 5 Ohms is a virtual 5 Ohms instead of a real 5 Ohms. The cartridge is not loaded at an actual 5 Ohms. 'Virtual Ground', again, is opamp parlance for a point in the circuit that exists at the same potential as ground but isn't actually ground.
I recommend that you read up on opamp operation since this seems to be the hanging point. Here's a short tutorial opamp virtual ground.
If you don't want to do that, just keep in mind that 'virtual ground' isn't the same as actual ground. So the cartridge would not be loaded at an actual 5 Ohms even though the virtual ground is 5 Ohms.