I've always thought second hand/thrift stores were great for picking up stuff cheap. The trick (at least in my area) is to hit up the ones in the nicer areas of town, the selection is usually better and it hasn't been picked through as much.
Edit: Just to add, the day of the week, and even season matters. Some places tend to put out new stuff at certain times, some just toss stuff on the shelves as they get it. So sometimes you can go one day and make out like a bandit, other days there's squat. You just have to visit a place a few times before you get a feel how they do things. Also mornings are usually better than afternoons.
Also garage sales. Same basic idea applies, the garage sales in the nicer areas usually have nicer stuff. The earlier you can hit them up the better selection you'll find, but you can often get better deals near the end of the day when they just want to get rid of something. (I bought 2 decent dressers, 2 shelving units, and a small desk for $40 last summer just because the guy didn't want to have to drag it all back into his house.)