Replacing speaker is the right approach in this case. N804 aren't that big even for your small listening room, so you may search for similar but with less fatigue.
My thoughts would be towards Vandersteen 3A-sig or Aerial 7T
My thoughts would be towards Vandersteen 3A-sig or Aerial 7T