Thanks everyone for the advice. There are indeed a lot of things to look at that could help out, even though I still feel there's something related to air pressure and very fast transients that's particularly bugging my sensitive ears, and that it's coming mainly from the amplifier. I've had 3 amplifiers that were fine (Audio Analogue Puccini SE is the one I haven't mentioned) and two that were not, all other things being equal. But certainly working on the rest might make more amplifiers work for me.
I'll see if I find a picture of my main listening room, it's temporarily not used as such, so I can't take one now. I do have panels in first and second reflection points, as well as some extra panels and bass traps. But I tried the amps mostly in another room, which was less treated, so there might be something to that also.
My interconnects are mostly Morrow Cables. I know they are not the smoothest and especially the ones containing a part of silver could be particularly problematic. But I didn't find that changing them to other interconnects (QED for instance) helped, other than losing a lot of definition and clarity. I will try to find some good cables to try out. I use Isotek Initium and Premier power cables mostly because they're the only ones doing a Swiss version. No power conditioning, just an audiophile power strip without any filtering. The power cords do make the amps have more clarity and rhythm, and I believe the transients are faster with those cables. I did also try with normal cables, but they didn't completely solve the issue. Perhaps I should change my power socket (I'm renting, so wanted to avoid) at home so I could try other power cables, such as the Shunyata.
Thanks for the recommendations again. I would love to try Pass Labs or First Watt amps, and they are on top of my endgame list, but I would like to avoid class A because of the heat and electricity consumption. They are also currently out of my budget unfortunately (ideally max 2000$, perhaps stretching to 3000$). Because of the current use case, an integrated with auto standby would be better than having to manually power on and power off various components (e.g. when watching TV or putting background music). Some other recommendations were also too expensive. But if it's correct as @audioman58 states that moving to higher end amplifiers would help, that would be very positive news.
From the ones listed, Naim is the one closes to my budget and easiest to test around here. Hegel would also be a possibility, even though stretching the budget a bit and I don't like paying for a streamer and a DAC I won't use. Perhaps a used one would make sense as proposed by @curiousjim, even though I'm not sure I need to go as far up the line as H360.
I also checked the Hafler amps, somewhat limited supply here in Europe and I'm not sure I want to go the vintage route for now.
Any recommendations regarding Naim, could the Nait 5SI be ok or should I move up the line or go with an older model?