Thanks for the responses.
Can I tell? No, actually I don't. But I just want to do it as correct as I can. You never know. i don't want to find out in five years I've been listening to a sound that could have been a lot better.
Almarg makes a god point - the phono stage does go through the line stage. I don't have a manual, and on the manufacturer's site it just says: "Line Stage phase inverting, Phono Stage phase correct" which sounds pretty ambiguous to me. I've sent them a mail. unless someone here knows the Conrad-Johnson PV12a and knows what it does?
Why does it hum when reversing the wires? No idea. I use a Technics SL1200MK2 with standard tonearm, if that helps. I see a black ground wire going in the bottom of the tonearm, but I don't know where it is connected to. I suspect the ground is connected to the inside of the tonearm (I can't check since the outside of the tonearm isn't conductive). The green wire (R-) is connected to the cartridge body, I can see that. But as far as I can make out, the cartridge body is not connected to the tonearm / ground. I can't figure out why switching the red/green and blue/white would cause a big hum, but it does.