Do you reply to all of your comments, some of your comment or none of your comments? I am of the mindset that it would be rude NOT to reply, but feel like if someone's just saying "cute kid," they probably don't expect a reply.
Which brings us to my second thought, do you reply by email or in the comment section of your blog?
I comment on their blog, and if they left a comment to me that needed my answer I answer it on their blog, but occasionally I live a response comment on my blog too, only if the post has some hot discussions LOL so I know people are coming back to check that :)
I do not reply to all my comments. If they ask a question then I reply through email. No one that I comment on replies either. I never thought they were rude. Sometimes it would just take to much time to respond. But I will visit their blog and leave a comment about their post.