It depends if you want them to be at your site or not. I want traffic and would prefer discussions of any kind take place on the blog where the question originated. Not only does this get the reader involved by coming back and responding, it also may get other readers involved.
It's really personal preference.
If it's a new commenter, I usually use the comment email responder plugin (wordpress) to reply but only for the first time.
It all depends. Sometimes I'll reply in a blog entry a day or 2 later. Often, I leave a comment in my comments area for them. I can't say I've ever gone to their blog and left a comment about what they said on mine. My usual way is to comment after their comment on my blog. I feel like if they read it and left a comment, they'll check for more comments.