Is SE Washington DC really all that bad?

It's always been my dream to live in Washington DC. I don't know, something about the city draws me in and I find myself spending a good amount of my time on Google Maps/Earth just looking around. I feel like I know the neighborhoods, and I've looked at apartment listings and surprise, surprise (not really) it's expensive! I live in southern California and it is rediculously expensive but it seems DC beats that. All the cheaper apartments I can find are SE, which I've heard is probably not the best place to move into. I know to stay out of Anacostia but is it really all that bad? I'm desperate. I might have to end up living there for a while before I can afford someplace else (which is possibly never).