The posters above have it right: Echo Audio and its neighborhood (esp. Powell's) would be a great area to spend some time; Kurt's terrific, and there are a ton of good places to eat nearby and in the adjacent Pearl and West End districts. Stereotypes is also very good, and it's next door to Nostrana which serves very good Italian food. Out in Beaverton, there's Chelsea Audio & Visual, where you can listen to your heart's content. You'll need a car to get there, but what Gertrude Stein once said about Oakland is also true for that part of Beaverton: there's no there there. Fred's House of Stereo on Hawthorne has been there a long time and is also in a fun locale.
And if all your listening gives rise to a temptation to buy, remember that there's no sales tax in Oregon. Enjoy your trip!