Subs?

Two subs are always better than one. They extend the soundstage. 

If you are looking for realistic sound then the subs will be turned now to a whisper... if they are not then the sound field will become compromised and broken up by the extended bass interfering. 

It really depends on what you are looking for... if those hip hop cars where you can feel them go by the bass vibrating your car... then go big. If you're looking for accurate sound then smaller. 

IMO, too much money is spent on main speakers that try to accomplish too much with respect to quality and quantity of bass.  Too often the result is boomy, ill-defined bass unless you spend a lot of money to get it right.  Your 804s are well-suited to provide enjoyable, balanced sound from around 40Hz up.  Take the win.  Adding subs will enhance your experience and improve on what you hear at the higher frequencies.

IME, two subs are way better than one and even more can be better yet.  Google "Swarm Subwoofer System" and "AudioKinesis" to read about the potential benefits of using an array of subs.  Not everyone can either afford an array of subs or get away with having 3 or 4 subs in their room.  However, if you can swing two subs I suspect you would be happier with the result compared to only using one.  I will let others weigh in on which brands/models of subs to consider. 

My Aerial SW-12 subs are plenty impactful, musical, and adjustable for me so I never needed to try others.  I have had good luck with them in the front of the room, set up for stereo, using line-level connections, and dialed in with a light touch. I have tried a passive high-pass filter to relieve the amplifiers of the chore of driving the lowest bass, and to relieve my main speakers from having to reproduce the lowest bass.  In my case it was a wash and not a noticable improvement but for others using a high pass filter has been beneficial.  Good luck.

A sub can be just as important as a tweeter and choice of sub will depend on the type of speaker you have.

Take a look at (or listen to) to the DB series B&W subs. For just music I'd take one of those over a JL all day long.

+1 on two subs, so size might be predicated on space available and cost.  That being said, ideally I would take two 12" subs over two 10" subs.  I am a REL fan, based on experience, sound  and reliability.  There are many reports here of JL reliability issues.  I have heard the B/W subs and would choose REL over them.  I also replace two Paradigm subs with REL and again they sounded much better.  I am sure a swarm sounds great but even for my excessiveness not practical.  IMHO, get the best REL subs you can afford.  You may hear that REL subs do not have some of the sophistication in adjustment (variable phase and remote control), but I have absolutely no issues with dialing them in.  They add air pressure, bass presence and depth, and augment the soundstage if placed and adjusted properly.

Matching brands is not the leading concern either.