Go over bad tubes and replace those.
That's pretty much an idea
What happens if you tap the tubes? that is how you check for microphonics. With nothing playing, turn on the amp and let it warm up. tap lightly on each tube with your fingernail. If sound comes out of the speaker, that tube has microphonics. Often tubes with microphonics are seen to be a non-problem and are resold. I don't tolerate microphonics because the problem you are vulerable to is as you describe.
If, OTOH, tapping on the tubes does not cause noise to come out of the speakers, then your problem isn't microphonics in your tubes. In that case I'd say you have a loosely connected ground loop and tapping makes the connection come and go.
Sadly, that is very common behavior for noisy small-signal tubes in high-sensitivity slots (preamps). Especially for older tubes with bigger plates. Phantom whistling, rustling leaves, comes & goes, etc. You either have to move the tube to a lower sensitivity slot (for example, move it from an input position to the output / follower slot, or move it to a power amp if it uses the same tube type), or replace it with a more tightly screened low noise tube. New production tubes (like Russian Gold Lions) are fantastic for tricky slots because they can be easily selected for very low noise and tight matches.
The Herbie’s halos look really cool but in my experience do NOTHING for this problem. Don’t even bother. Same for isolation bases and all manner of such tweaks.
Before you start replacing tubes, I suggest you pull the tube(s) and clean the contacts. I had a similar experience with a Brimar NOS 12AU7. It was so bad that walking around the room triggered noise. But after using high quality contact cleaner and spending some time wigging the tub in its position all is working good again.