It depends upon whether the center pin or the outer ring is causing the problem. If the outer ring is loose, you can squeeze it down a little with a pair of pliers. If the center pin is not making a good connection, you might be able to slightly bend it, so that it will contact the inside of the jack better. Try not to break it off. You might have to do this periodically, because the pin will want to center itself after a while.
It is also possible that the connections of the cable to the RCA plug might be causing it, depending on how you are applying pressure on them to make it work. If you can get a look at the connections under the barrel of the RCA plug, you might get some indicator of whether the problem may lie in that area.
Either way, this is not a real big deal, and should be fairly easily remedied.