Dave, based on the info in the manual that I referred to earlier, about XLR pin 2 being shorted to ground when the RCA inputs are selected, I suspect that the only thing that switch does is to put that specific short in place when the RCAs are selected, and remove that short from the circuit when XLRs are selected. I suspect that XLR pin 1 is always connected to either circuit ground or chassis ground, regardless of the position of that switch.
And regardless of which of those grounds pin 1 is connected to, it should serve as a suitable point for connecting the sub's ground (the black wire). Certainly if pin 1 is connected to circuit ground, and most likely if it is connected to chassis ground (although in that case there is a slight possibility of some hum resulting).
What I'm envisioning is that XLR pins 2 and 3, through which the balanced pair of signals would be supplied if the XLR connectors were being used, are routed to the + and - inputs of a differential receiver circuit. With the center pin of the RCA connector being connected directly to one of those inputs (the one corresponding to XLR pin 3), and the switch providing a ground (0 volts) to the other input of the differential receiver (the one corresponding to XLR pin 2) when the RCA input is selected.
You can verify what I'm envisioning, with respect to the grounds, by disconnecting the RCA cables and measuring the resistance between the RCA ground sleeves and pin 1 of the XLR connectors. If XLR pin 1 is connected to circuit ground, that will measure 0 ohms, or at most a tiny fraction of an ohm, regardless of the position of the switch. If it is connected to chassis ground, you may get a fluctuating reading similar to what you previously reported when you measured between the RCA ground sleeve and AC safety ground. In that event, take a measurement between the AC safety ground pin of the power plug and XLR pin 1. If that reading is 0 ohms or very close to it, regardless of the position of the switch, it would confirm that pin 1 is connected to chassis ground, and you would also be good to go.
Best regards,
-- Al