If those sites don't want spiders, they can just specify that in robots.txt, which Google honors, right?

From the point of view of the law, that might not matter, but when there's a standardized way to make clear to bots that they're not welcome and you didn't bother to implement it, you'll look pretty silly if you complain.