The time it takes to load my blog and someone else's page is one of my biggest pet peeves. A site has to have really great content for me to sit through the few minutes it takes to load the site.
You may not realize how slow your site is on your own computer, because your site is already in your cache. You can go to http://sitereportcard.com/, enter your URL, then click on Load Time Check, and see how many seconds it takes for the 1.44MB T1 connection speed only. (Most people have at least a 1.44MB T1 connection speed, the others listed are for dial-up.)
Are you turned off when it takes forever for a site to load? Do you care?