how big a speaker can you put in an 11 x 11 room?

Will you be able to place the speakers properly away from the walls and still sit far enough away from them for the separate drivers to cohere? You can't sit but so close to physically large multi-driver systems and not hear the individual drivers as individuals drivers.

I'm not even going to mention that your room is a square and the problems that will present for smooth bass response.

I agree the main issue is how far you can get from the walls. I have a small room myself, and one set of rear-ported speakers (Soliloquy 5.3i) clearly overwhelmed it. A much larger rear-ported speaker does a lot better! (Usher CP 6311). So I guess it's the speaker design too. Go for front ported if you can

Since one wall of your room isn't really a wall, but an opening into a much larger room, I wouldn't consider your room to be small. Much depends on the layout of the speakers and your listening position in the 11x11 area where the system will reside.

You should keep the speakers as small as possible and add a sub later only if needed.

I use OHM 100s and Dynaudio monitors in two 12X12 rooms in my house.

The largely omni OHMS work best. They are small footprint full range floorstanders. Being largely omnidirectional, they tend to fill teh room with sound rather than blast it at you, which is of benefit in a small room. They are also less fussy about location for smooth response than box designs in a square room.

OHM Micro Walsh Talls have an even smaller footprint than the 100s and are recommended for nearfield listening conditions as you will have in a room that size.

They can be auditioned in-house for several months. You really have nothing to loose.

www.ohmspeakers.com for more info.

Many British monitors work well in a room that size. Believe me I know, I have a 12' X 11' room.
Much has been discussed here on Audiogon about small rooms. Do a search, you will find plenty of info.
Monitors definately work best, sub not likely needed.
A partial list:

Linn
Spendor
Neat
PMC
Joseph Audio
ERA Designs
Aerial
Audio Note

All of the above make nice monitors that will work