So much Bach, where do I start?

Everybody who loves Bach is going to have their own favorites for recorded versions. My choice for the complete Brandenburg concertos is a relatively new recording by the Boston Baroque (Martin Pearlman, Director) on the Telarc label. It features period instruments, and the recording quality is superb. Another version which is also excellent is the recording by Tafelmusik on the Sony label. If you like violin, I recommend the solo and double violin concertos, recorded on the Harmonia Mundi label by Andre Manze, Richard Podger, and the Academy of Ancient Music. The Orchestral Suites are also well worth owning, and the version by Christopher Hogwood and the Academy of Ancient Music on the London label is excellent. The Well-Tempered Clavier, a group of some 24 preludes and fugues, is also a great work. There are two good recordings of WTC, one on harpsichord by Davitt Moroney (Harmonia Mundi) and the other on piano by Andras Schiff (London, 2 CD's). I also like the Goldberg variations, with the most famous recording by Glenn Gould. There are 2 more recent versions which also are commendable by Andras Schiff (London) and Ton Koopman (Erato). Bach was also a master of choral music, and no collection is complete without his "St. Matthew's Passion" (try the John Eliot Gardner version on Deutsche Gramophone with the Monteverdi Choir), and the "Mass in B Minor" (2 good versions: John Eliot Gardiner with Monteverdi Choir on the DG label; and the version directed by Robert King with the King's Consort on the Hyperion label.)

Yes Brandenburg Concertos are very good, too many good versions to mention, I have Pinnock/Archiv label.

What you really want though going by your description are
Vivaldi concertos, which can make Bach sound stodgy by comparison. Get the entire sets, not just "4 seasons".
Great sets include, Opus 3 L'estro Armonico, Opus 4 La
Stravaganza, Opus 8 Il Cimento (includes 4 seasons), Opus
9 La Cetra. Each opus contains 12 concertos, exciting and
passionate as baroque music gets. Many good cheap sets available by Pinnock, Hogwood, Marriner and others.

This has exploded :-)
Well, I'll see what's at the local record shops, and I'm sure if I look I can find a specialty store. I dug around my mother's house and turned up an ADD recording of the Brandenburg Concertos on Sonata, they are missing sleeves though. I hadn't even considered Bachs's choral pieces.
Megasam, I'll take a look at Vivaldi. I also found the four seasons with Pinnock, I quite like Winter.
I became interested in Bach in large part due to the book "Godel, Escher, Bach" by Douglas Hofstadler (sp?). I wanted to look into some of the compostion techniques he talks about, backwards themes, hiding his name, etc.

For Vivaldi orchestral try the Musici version with C. Scimone on Philips Classics, vols 1&2, and enjoy!

i read 'goedel, escher, bach' more than ten years ago - a fascinating book, but not an easy one. with regard to the composition techniques, hofstaedter - for the most part and as far as i remember - refers to bach's 'art of fugue'. in fact, 'goedel, escher, bach' inspired me to get a recording of this fascinating piece (kenneth gilbert on DG/archiv). however, this might not be the kind of baroque music you are looking for. not very easy to listen to, but still enjoyable (harpsicord, though).
for a better understanding of hofstaedter's work it might also help, if you read up a little on canons, fugues, contrapunctus etc. (i have to admit, i only have a very basic understanding of these topics).