The 4V refers to the DIFFERENCE between the non-inverted and inverted signals that are provided to pins 2 and 3, respectively, of the XLR connector. Each of those signals individually has a maximum amplitude of 2V. By adapting the XLR output to an RCA input, you would only be using one of those signals (the non-inverted one), so the RCA input would see a maximum amplitude of 2V.
Something to keep in mind, however: Most XLR-to-RCA adapters, and perhaps some adapter cables, short the inverted signal on XLR pin 3 to ground (XLR pin 1). Many components won't have a problem with that, but some will (if it is done on an XLR output; it is no problem on an XLR input). Use an adapter or adapter cable that leaves pin 3 unconnected. Talk to the supplier to be sure that is what they will provide.
Another approach to consider would be to just use the RCA output, via a y-adapter that would allow you to connect it to two destinations. That should work fine as long as the total of the two cable lengths is not extremely long.
Regards,
-- Al