Because most of the good sites won't let you crawl them any more, unless you're ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

Animats on Nov 13, 2024 | parent | context | favorite | on: Two upstart search engines are teaming up to take ...

Because most of the good sites won't let you crawl them any more, unless you're Google.

immibis on Nov 13, 2024 [–]

Crawl them anyway. Adversarial interoperability is the name of the game.

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact