The first thing I'd suggest you do is buy a sound level meter. Ones for home use are relatively inexpensive ($20 to $30) or you can download an app for your smartphone for even less, or free. This will give you a ballpark idea of just how loud you listen. (One's person's "loud" may be someone else's "medium", etc.) For example, my "loud" these days is around 85 dB average.
Then, look up the sensitivity of your speakers. A lot of modern speakers have ratings in the upper 80s -- for example my Ohms are 88 dB -- it takes one watt of power to produce that sound level. Small bookshelf speakers tend to be a bit less sensitive, while many larger speakers, like horn systems, tend to be much more sensitive -- they will play much louder on that one watt of power.
Even if one watt of power gives more volume than your average listening level, you still need more power than that from your amp. Music is dynamic with loud, short duration peaks above the average playback level-- such as a drum strike or orchestra crescendo.
Unless you are a head-banger that likes your music at an average volume of 100 dB or more, there usually isn't much concern about having too much power, though some do get carried away on this issue. I happen to be in the camp that is more concerned about the sound quality of an amp rather than whether it has a zillion watts.