It depends if you want them to be at your site or not. I want traffic and would prefer discussions of any kind take place on the blog where the question originated. Not only does this get the reader involved by coming back and responding, it also may get other readers involved.
It's really personal preference.
If it's a new commenter, I usually use the comment email responder plugin (wordpress) to reply but only for the first time.
It all depends. Sometimes I'll reply in a blog entry a day or 2 later. Often, I leave a comment in my comments area for them. I can't say I've ever gone to their blog and left a comment about what they said on mine. My usual way is to comment after their comment on my blog. I feel like if they read it and left a comment, they'll check for more comments.
Mom Bloggers for Social Good is a global coalition of 2000+ mom bloggers who currently span over twenty countries and who care about spreading the good news about the amazing work non-profit organizations and NGOs are doing around the world. Want to make a difference?Please join us!