Absolutely not, I enjoy tinkering just like I like working on my own cars. I have brand new loudspeakers and I already have the interfaces apart cooking up modifications to make them work the way I want.
Being an audiophile is frequently called a "hobby." For most it is just a leisurely pastime. For people like lewm who is not at all scared to attack his equipment with a soldering iron and designs his own amplifiers it is a hobby for sure.
I do wish everyone who loves music could afford a first class system. There is a lot of excellent relatively inexpensive gear out there now so it is easier than ever. One huge advantage of digital reproduction that is not frequently discussed is the availability of high quality inexpensive sources It is much easier and cheaper to build a 1st class digital system than an analog one. I have heard $50 DACs do an amazing job considering to cost. The one area this all falls apart is with loudspeakers. But, in saving a lot of money on the front end give you more to spend on speakers....