If you are going to invite people into your house you should treat them like guest. If you don't want them looking at, touching, breathing on, moving or otherwise interacting with your high fidelity system, then keep your system in a locked room. If that is not possible, then you need to accept that there could be an "incident". What the OP described is not a big deal. Even if the person had damaged the loudspeakers the proper response of the host should be, "Don't worry, it's not a problem."
Tradesmen are a different type of problem. IMO, you should be in the room with them at all times.
Tradesmen are a different type of problem. IMO, you should be in the room with them at all times.