I have several integrated tube amps and run them all at full volume. Your report that you can hear the audible difference is exactly what is predicted. Indeed, the volume control is an area where cost or design can result in compromises that harm sound quality. A good volume control knob will have an excellent contact a full clockwise to jumper the resistors, pots, or whatever is used to control volume out of the circuit. I have toyed with the idea of going into the amps and actually wiring around the volume control circuits but I have found that the full volume setting works fine. My amps generally have a resistive ladder which is better but even with it I generally have less noise at full volume.
Using an integrated like has a nice advantage. You now have an amp with multiple inputs. I use a spare set of inputs for HT bypass.
Of course you aren't damaging anything. You are using your amp as an amp.
Jerry