how big a speaker can you put in an 11 x 11 room?

I agree the main issue is how far you can get from the walls. I have a small room myself, and one set of rear-ported speakers (Soliloquy 5.3i) clearly overwhelmed it. A much larger rear-ported speaker does a lot better! (Usher CP 6311). So I guess it's the speaker design too. Go for front ported if you can

Since one wall of your room isn't really a wall, but an opening into a much larger room, I wouldn't consider your room to be small. Much depends on the layout of the speakers and your listening position in the 11x11 area where the system will reside.

You should keep the speakers as small as possible and add a sub later only if needed.

I use OHM 100s and Dynaudio monitors in two 12X12 rooms in my house.

The largely omni OHMS work best. They are small footprint full range floorstanders. Being largely omnidirectional, they tend to fill teh room with sound rather than blast it at you, which is of benefit in a small room. They are also less fussy about location for smooth response than box designs in a square room.

OHM Micro Walsh Talls have an even smaller footprint than the 100s and are recommended for nearfield listening conditions as you will have in a room that size.

They can be auditioned in-house for several months. You really have nothing to loose.

www.ohmspeakers.com for more info.

Many British monitors work well in a room that size. Believe me I know, I have a 12' X 11' room.
Much has been discussed here on Audiogon about small rooms. Do a search, you will find plenty of info.
Monitors definately work best, sub not likely needed.
A partial list:

Linn
Spendor
Neat
PMC
Joseph Audio
ERA Designs
Aerial
Audio Note

All of the above make nice monitors that will work

The problem in a small room is low frequency response. A small sealed monitor will work because there won't be a lot of bass. Otherwise, a full-range transmission line speaker will work. You will have bass, although you should have bass traps in a room that small regardless of the speakers, and a TL speaker will be easier to place than a ported full-range speaker. The Salk Sound SongTower or Veracity QWT would be good choices.