This is a hobby that people have spent rich amounts of money on, not a rich mans hobby. It is no different from any other hobby where people either with the cash, or through sacrifice have spent huge sums ... and just like other hobbies, that does not mean the best outcomes.
Yes, listen, but please, not at audio shows. Audio shows are an exercise in how poor you can make really expensive equipment sound. It is impossible in a poorly acoustic treated room to have "great" sound. That the audio press waxes poetic about these "awesome" systems at trade-shows and how wonderful they sound, is good evidence they are either blissfully or wilfully ignorant ... or selective in their hearing.
Your local quality dealer is usually (not always) a better place, as they often have properly set up rooms. Find out when they are not busy and visit then. Audio clubs are great, though seem to be waning.
Educate yourself about acoustics and room treatment. This will put you far ahead of most people who identify as audiophiles.
Personally, jump in. Yes, jump in. Give yourself a starting budget, and keep your system really basic. Streaming DAC, Amplifier, Speakers, acoustic treatment ... and set aside $50-100 for a basic microphone and learn how to use REW, which again will put you way ahead of most "audiophiles". Formulas work pretty good in regular rooms for speaker placement, but nothing like being able to measure and optimize.
$1000 on a basic streaming DAC, used amplifier, used speakers, and some basic cables can get you quite a nice sounding system. If you are handy with basic tools and can build your own acoustic panels, then for another $100-200, you can put together a half decent set of a few critical panels. Pass on the turntable to start. You will blow that whole budget just to get something decent.
I am going to say it again. Learn about acoustics and room treatment. Audiophiles who don’t are always chasing their tales with the latest tweak that "totally improved the system" ..... I mean really, how often can "totally improve the system"? Ask yourself this, what is more likely going to change how you perceive the sound, something like acoustics that makes an a change so significant it is easily measured with $100 of equipment, or something that no vendor seems to be able supply any measurable evidence to support? Yes amplifier/speaker cable/speaker combinations can have subtle differences in how they sound, but unless you are dropping serious money to start, it is not something to worry about yet.
One last point. You rarely listen to "equipment". Unless you are listening to a piece of equipment, in your own listening space, with your own speakers, and all your other own equipment in the chain, then what you are listening too is mainly speakers and how they interact with the room for the exact space you are sitting in, overlaid with some equipment that has some indeterminate impact on the overall sound. With the exception of things like a turntable that can have fairly unique sonic signatures, picking out subtle differences and assigning them to one piece of equipment is ... difficult (read near impossible). Switching between a tube and solid state amp may be noticeable as well because one usually adds enough to the signal to be noticeable. Whether you like what it adds, that is a question only you can decide.