Amanda, I think it's more important to love what you are doing than retain a few readers that don't appreciate the changes you've made. Most likely, because you are more passionate about the direction your blog is going now, you'll regain new and "stickier" followers because they will better appreciate you.
It's tough, I hate when I lose readers. They never make it known to me why they leave either (guess I'm not controversial enough!). I just have to believe that just like there is a better fit blogger for them, there is a better fit reader for me. You know?
I think unfortunately a lot of people follow to get followers and never come back to visit again and probably clear you off their list next time they do a clean out so the fact that you are losing followers doesn't mean you are doing anything wrong. I would focus on your regular readers and what they enjoy. You could always do a poll or something to get feedback from readers regarding what aspects of the blog they like and don't like.
Gaining readers and comments is great for visibility in the search engines. As far as followers, sometimes bloggers follow others through RSS feeds.
But, every blogger loses some followers; if visitor stats and comments are growing, I wouldn't worry about it.