Aside from getting burned, there is no reason why you should avoid contact with the glass envelope of a tube. Tubes do not get so hot that oils from your hand could lead to failure. The best reason for not touching the glass when changing tubes, involves tubes with a base (i.e., not entirely glass)--you do not want to pull tubes by the glass because that might lead to separation of the base from the glass envelope.
The whole idea of making a fast swap suggests some sort of quick comparison of tubes. Of course such a comparison is not entirely practical because it will take some time for the new tubes to come up to good working order (a few minutes). While it is important to get tubes you like into your gear, one should be mindful of the wear and tear on tube sockets that constant swapping causes. An old technician told me that most tube sockets are good for about 30 in and out cycles before the socket needs re-tensioning or even replacement. In the old days (before neurotic swapping) that would mean 30 or more years of service).